Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

gt;> interested to drive this further. So we will just "reassign" it to them. >> >> Thanks for letting us know. >> >> >> -Matthias >> >> On 6/20/18 2:51 PM, Jeyhun Karimov wrote: >> > Hi Matthias, all, >> > >> > Curr

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

f `through()` and `to()`, can you add > >> the different behavior using different overloads? It's not clear from > >> the KIP what the semantics are. > >> > >> > >> -Matthias > >> > >> On 11/17/17 3:27 PM, Jeyhun Karimov wrote: > >>>

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

gt;>>>>>> Even then its difficult already: I never ran a topology with > caching > >>>>>>>> but I > >>>>>>>> am not even 100% sure what the record Context means behind > >>>>>>>> a materialized KTable

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

bout it at all, but only specify num.partitions. > >> > >> 3. Details: in Produced we do not have specs for specifying the > >> num.partitions or should we repartition or not. So it is still not > >> clear to > >> me how we would make use of that to achi

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

p it that way until #6. In addition, the > >>>> RecordContext > >>>> fields (topic, offset, etc) are really orthogonal to the key-value > >>>> payloads > >>>> themselves, so I think separating them into this object is a > >>>> clean

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

gt; wrote: > > > >> Hi Jeyhun, thanks, looks good. > >> Do we need to remove the line that says: > >> > >>- on-demand commit() feature > >> > >> Cheers, > >> Damian > >> > >> On Tue, 31 Oct 2017 at 23:07 J

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

ser don't want to manage topic manually, thus, it's still an > >> internal topic and Streams create the topic name automatically as for > >> all other internal topics. However, users gets some more control about > >> topic parameters like number of partitions (we should

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

7 10:19 PM, Jan Filipiak wrote: > >> -1 non binding > >> > >> I don't get the motivation. > >> In 80% of my DSL processors there is no such thing as a reasonable > >> RecordContext. > >> After a join the record I am processing belongs to at least 2 top

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

> > Maybe, we could use `through()` to create a topics in the back ground > > and picky back all to `Produced`. > > > > > > -Matthias > > > > On 11/4/17 4:16 PM, Ted Yu wrote: > > > API is given without much javadoc on the role / meaning of method >

[DISCUSS] KIP-221: Repartition Topic Hints in Streams

Dear community, I would like to initiate discussion on KIP-221 [1] based on issue [2]. Please feel free to comment. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Repartition+Topic+Hints+in+Streams [2] https://issues.apache.org/jira/browse/KAFKA-6037 Cheers, Jeyhun

[VOTE] KIP-159: Introducing Rich functions to Streams

Dear community, It seems the discussion for KIP-159 [1] converged finally. I would like to initiate voting for the particular KIP. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-159%3A+Introducing+Rich+functions+to+Streams Cheers, Jeyhun

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

re saying, but I don't agree that > > > but also we need a commit() method > > I would just not provide `commit()` at DSL level and close the > corresponding Jira as "not a problem" or similar. > > > -Matthias > > On 10/27/17 3:42 PM, Jeyhun Karimov wrote:

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

` is for a specific record. > > To me, this does not seem to be a sound API design if we follow this path. > > > -Matthias > > > > On 10/26/17 10:41 PM, Jeyhun Karimov wrote: > > Hi, > > > > Thanks for your suggestions. > > > > I have some co

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

; On Mon, Oct 23, 2017 at 1:40 PM, Matthias J. Sax <matth...@confluent.io> > wrote: > > > Fair point. This is a long discussion and I totally forgot that we > > discussed this. > > > > Seems I changed my opinion about including KAFKA-3907... > > > > Happ

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

l I am not even sure if we should fix KAFKA-3907 at all. > Manual commits are something DSL users should not worry about -- and if > one really needs this, an advanced user can still insert a dummy > `transform` to request a commit from there. > > -Matthias > > > On 10/18/1

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

r context not a record context > logically: when you call that function, it means we would commit the state > of the whole task up to this processed record, not only that single record > itself. > > > Guozhang > > On Mon, Oct 16, 2017 at 9:19 AM, Jeyhun Karimov &l

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

.processor.internals to > o.a.k.processor. > > 6. I'm not clear why we want to move `commit()` from ProcessorContext to > RecordContext? Conceptually I think it would better staying in the > ProcessorContext. Do you find this not doable in the internal > implementations? > > &

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

public RecordContext recordContext() { > return this.recordContext(); > > Can you explain a bit about the above implementation ? > >void commit () { > throw new UnsupportedOperationException("commit() is not supported in > this context"); > > Is the exc

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

Dear community, I updated the related KIP [1]. Please feel free to comment. Cheers, Jeyhun [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-159%3A+Introducing+Rich+functions+to+Streams On Fri, Sep 22, 2017 at 12:20 AM Jeyhun Karimov <je.kari...@gmail.com> wrote: >

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

gt; Damian > > On Thu, 21 Sep 2017 at 15:23 Jeyhun Karimov <je.kari...@gmail.com> wrote: > > > Hi all, > > > > Thanks a lot for your comments. For the single interface (RichXXX and > > XXXWithKey) solution, I have already submitted a PR but probably it is &

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

;bbej...@gmail.com> > wrote: > > > > > > > +1 for me on collapsing the Rich and ValueWithKey interfaces > > > into 1 > > > > interface. > > > > > > > > Thanks, > > > > Bill > > > > > > > > On

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

gt; lines of option 2 where we collapse the Rich and ValueWithKey etc > interfaces into 1 interface that has all of the arguments. I think we then > only need to add one additional overload for each operator? > > Thanks, > Damian > > On Wed, 13 Sep 2017 at 10:59 Jeyhun K

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

lease feel free to comment on this. [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=73637757 Cheers, Jeyhun On Fri, Jul 21, 2017 at 2:00 AM Jeyhun Karimov <je.kari...@gmail.com> wrote: > Hi Matthias, Damian, all, > > Thanks for your comments and sorry for su

Re: [VOTE]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Eno Thereska <eno.there...@gmail.com> > > wrote: > > > > > > > +1 (non-binding) Thanks. > > > > > > > > Eno > > > > > On 6 Jul 2017, at 21:49, Gwen Shapira <g...@confluent.io> wrote: > > > > > > > > >

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

<matth...@confluent.io> wrote: > I would not block this KIP with regard to DSL refactoring. IMHO, we can > just finish this one and the DSL refactoring will help later on to > reduce the number of overloads. > > -Matthias > > On 7/7/17 5:28 AM, Jeyhun Karimov wrote: >

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

text() method from > > > InternalProcessorContext to ProcessorContext. > > > > > > In the KIP you have an example showing: > > > richMapper.init((RecordContext) processorContext); > > > But the interface is: > > > public interface RichValueMa

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

gt; > > > > > > > I have one concern with regard to backward compatibility. You suggest > > to > > > > use RecrodContext as base interface for ProcessorContext. This will > > > > break compatibility. > > > > > > > > I think, we s

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

for ProcessorContext. This will > > break compatibility. > > > > I think, we should just have two independent interfaces. Our own > > ProcessorContextImpl class would implement both. This allows us to cast > > it to `RecordContext` and thus limit the visible sc

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

> >> upgrade guidance. > >> > >> Regarding the scope I'm still trying to solicit opinions regarding > >> ReducerWithKey and InitializerWithKey; to me they are not necessarily > to be > >> included. > >> > >> > >> Guozhang > >> &g

Re: Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

ated() or similar? So that users don't have to read the docs > to know it isn't the creation timestamp for instance. > Cheers, > Michał > > > On 04/06/17 01:24, Jeyhun Karimov wrote: > > Hi Matthias, > > Thanks for comments. > > - why do you only consider get()

[VOTE]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Dear all, I would like to start the vote on KIP-149 [1]. Cheers, Jeyhun [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner -- -Cheers Jeyhun

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

to address this issue as well or continue as it is? Cheers, Jeyhun On Wed, Jun 14, 2017 at 11:29 PM Guozhang Wang <wangg...@gmail.com> wrote: > LGTM. Thanks! > > > Guozhang > > On Tue, Jun 13, 2017 at 2:20 PM, Jeyhun Karimov <je.kari...@gmail.com> > wrote: >

Re: Handling 2 to 3 Million Events before Kafka

Hi, With kafka you can increase overall throughput by increasing the number of nodes in a cluster. I had a similar issue, where we needed to ingest vast amounts of data to streaming system. In our case, kafka was a bottleneck, because of disk I/O. To solve it, we implemented (simple) distributed

[jira] [Commented] (KAFKA-4829) Improve logging of StreamTask commits

[ https://issues.apache.org/jira/browse/KAFKA-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049834#comment-16049834 ] Jeyhun Karimov commented on KAFKA-4829: --- [~guozhang] I would suggest similar model to SQL Server's

[jira] [Assigned] (KAFKA-3907) Better support for user-specific committing in the Streams DSL

[ https://issues.apache.org/jira/browse/KAFKA-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-3907: - Assignee: Jeyhun Karimov > Better support for user-specific committing in the Stre

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

formerCommon: > > > > I think changing the interface to extend from a new interface is not > binary > > compatible though source compatible, i.e. users still need to recompile > > their code though no need to make code changes. We may need to mention > that > > in the

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

6/6/17 5:15 PM, Guozhang Wang wrote: > > Thanks for the comprehensive summary! > > > > Personally I'd prefer the option of passing RecordContext as an > additional > > parameter into he overloaded function. But I'm also open to other > arguments > > if there are

[jira] [Comment Edited] (KAFKA-3826) Sampling on throughput / latency metrics recording in Streams

[ https://issues.apache.org/jira/browse/KAFKA-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046497#comment-16046497 ] Jeyhun Karimov edited comment on KAFKA-3826 at 6/12/17 12:32 PM

[jira] [Commented] (KAFKA-3826) Sampling on throughput / latency metrics recording in Streams

[ https://issues.apache.org/jira/browse/KAFKA-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046497#comment-16046497 ] Jeyhun Karimov commented on KAFKA-3826: --- [~guozhang] I think KAFKA-4829 also can be related

[jira] [Assigned] (KAFKA-4653) Improve test coverage of RocksDBStore

[ https://issues.apache.org/jira/browse/KAFKA-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4653: - Assignee: Jeyhun Karimov > Improve test coverage of RocksDBSt

[jira] [Assigned] (KAFKA-4658) Improve test coverage InMemoryKeyValueLoggedStore

[ https://issues.apache.org/jira/browse/KAFKA-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4658: - Assignee: Jeyhun Karimov > Improve test coverage InMemoryKeyValueLoggedSt

[jira] [Assigned] (KAFKA-4659) Improve test coverage of CachingKeyValueStore

[ https://issues.apache.org/jira/browse/KAFKA-4659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4659: - Assignee: Jeyhun Karimov > Improve test coverage of CachingKeyValueSt

[jira] [Assigned] (KAFKA-4655) Improve test coverage of CompositeReadOnlySessionStore

[ https://issues.apache.org/jira/browse/KAFKA-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4655: - Assignee: Jeyhun Karimov > Improve test coverage of CompositeReadOnlySessionSt

[jira] [Assigned] (KAFKA-4661) Improve test coverage UsePreviousTimeOnInvalidTimestamp

[ https://issues.apache.org/jira/browse/KAFKA-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4661: - Assignee: Jeyhun Karimov > Improve test coverage UsePreviousTimeOnInvalidTimest

Re: KIP-160 - Augment KStream.print() to allow users pass in extra parameters in the printed string

I agree with Matthias's comment. Constructing RecordContext with more metadata seems more feasible for me. Cheers, Jeyun On Mon, Jun 5, 2017 at 7:47 AM Matthias J. Sax wrote: > Not with the scope of the current discussion. > > So far, we discuss to add `RecordContext`,

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

ly a little bit undecided what the better option might be. > >> Curious to hear what other think about this trade off. > >> > >> > >> -Matthias > >> > >> > >> On 6/1/17 6:13 PM, Jeyhun Karimov wrote: > >>> Hi Guozhang, >

Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

le new > method returning KeyContext not be sufficient? > - for backward compatibility, we will also need a new interface and > cannot just extend the existing one > > > > -Matthias > > On 5/29/17 4:55 PM, Jeyhun Karimov wrote: > > Dear community, > > > > I

[jira] [Commented] (KAFKA-4325) Improve processing of late records for window operations

[ https://issues.apache.org/jira/browse/KAFKA-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035688#comment-16035688 ] Jeyhun Karimov commented on KAFKA-4325: --- [~mjsax], would this jira requre KIP? > Impr

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

. Cheers, Jeyhun On Fri, Jun 2, 2017 at 2:27 AM Guozhang Wang <wangg...@gmail.com> wrote: > Does this KIP subsume this ticket as well? > https://issues.apache.org/jira/browse/KAFKA-4125 > > On Sat, May 20, 2017 at 9:05 AM, Jeyhun Karimov <je.kari...@gmail.com> >

[DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

Dear community, I want to share KIP-165 [1] based on issue KAFKA-4304 [2]. I would like to get your comments. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-165%3A+Extend+Interactive+Queries+for+return+latest+update+timestamp+per+key [2]

[jira] [Comment Edited] (KAFKA-4304) Extend Interactive Queries for return latest update timestamp per key

[ https://issues.apache.org/jira/browse/KAFKA-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027964#comment-16027964 ] Jeyhun Karimov edited comment on KAFKA-4304 at 5/28/17 10:34 PM: - >F

[jira] [Assigned] (KAFKA-4304) Extend Interactive Queries for return latest update timestamp per key

[ https://issues.apache.org/jira/browse/KAFKA-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4304: - Assignee: Jeyhun Karimov > Extend Interactive Queries for return latest update timest

[jira] [Comment Edited] (KAFKA-4304) Extend Interactive Queries for return latest update timestamp per key

[ https://issues.apache.org/jira/browse/KAFKA-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027964#comment-16027964 ] Jeyhun Karimov edited comment on KAFKA-4304 at 5/28/17 10:25 PM: - >F

[jira] [Commented] (KAFKA-4304) Extend Interactive Queries for return latest update timestamp per key

[ https://issues.apache.org/jira/browse/KAFKA-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027964#comment-16027964 ] Jeyhun Karimov commented on KAFKA-4304: --- >From the user perspective, one would qu

[jira] [Commented] (KAFKA-4304) Extend Interactive Queries for return latest update timestamp per key

[ https://issues.apache.org/jira/browse/KAFKA-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027958#comment-16027958 ] Jeyhun Karimov commented on KAFKA-4304: --- [~mjsax] I completely forgot this issue. Sorry for super

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

if we duplicate > functionality. > > For this reason, it seems to be better to got with the > `#valueMapper(ValueMapper mapper, RecordContext context)` approach. > > WDYT? > > > > -Matthias > > On 5/27/17 11:00 AM, Jeyhun Karimov wrote: > > Hi, > >

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

ut we can't do this for ValueTransformer (and similar). > > Btw: This reminds me about KIP-159: with regard to the RichFunction we > might need a supplier pattern, too. (I'll comment on the other thread, > too.) > > > -Matthias > > On 5/28/17 5:45 AM, Jeyhun Karimov wrote: > &g

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

te and also allows to register > punctuations. Both those features will not be available via withKey() > interfaces. > > -Matthias > > > On 5/27/17 1:25 PM, Jeyhun Karimov wrote: > > Hi Matthias, > > > > Thanks for your comments. > > > > I t

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

t/complete API, it might be a good idea. Any thoughts? > > > -Matthias > > > On 5/24/17 3:47 PM, Matthias J. Sax wrote: > > Jeyhun, > > > > I was just wondering if you did look into the key-deep-copy idea we > > discussed. I am curious to see what the impact might be. &g

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

would allow us to drop "init()". This might even > allow to use Lambdas and we could keep the name RichFunction as we > preserve the nature of being a function. > > > -Matthias > > On 5/24/17 12:13 PM, Jeyhun Karimov wrote: > > Hi Michal, > > > > Thank

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

>> function. Sounds more like the Processor/Transformer kind of thing >> semantically, rather than a function. >> >> The KIP says there are multiple use-cases for this but doesn't enumerate >> any - I think some examples would be useful, otherwise that section sounds >

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

, so I'm asking all this to better > understand the rationale for init() and close(). > > Thanks, > Michał > > On 20/05/17 17:05, Jeyhun Karimov wrote: > > Dear community, > > As we discussed in KIP-149 [DISCUSS] thread [1], I would like to initiate > KIP for rich func

[DISCUSS]: KIP-159: Introducing Rich functions to Streams

Dear community, As we discussed in KIP-149 [DISCUSS] thread [1], I would like to initiate KIP for rich functions (interfaces) [2]. I would like to get your comments. [1]

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

Noll <mich...@confluent.io> > wrote: > > > Thanks for your work and for driving this, Jeyhun! :-) > > > > -Michael > > > > > > On Tue, Apr 25, 2017 at 11:11 PM, Jeyhun Karimov <je.kari...@gmail.com> > > wrote: > > > > > Dear a

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

do this even in parallel) -- we > don't want to slow you down :) But it make the discussion and code > review easier, if we separate both IMHO. > > > -Matthias > > > On 5/19/17 2:25 AM, Jeyhun Karimov wrote: > > Hi Damian, > > > > Thanks for your comments. I t

[jira] [Commented] (KAFKA-4785) Records from internal repartitioning topics should always use RecordMetadataTimestampExtractor

[ https://issues.apache.org/jira/browse/KAFKA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017575#comment-16017575 ] Jeyhun Karimov commented on KAFKA-4785: --- [~mjsax] Thanks for reminding. I am on it > Records f

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Also, i don't like the abstract classes: RichValueMapper, RichValueJoiner, > RichInitializer etc. Why can't they not just be interfaces? Generally we > should provide users with Intefaces to code against, not classes. > > Thanks, > Damian > > On Fri, 19 May 2017 at 00:50 Jeyhun Karim

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

> > > Overall, I like the KIP a lot! > > > -Matthias > > > On 5/16/17 7:03 AM, Jeyhun Karimov wrote: > > Hi, > > > > Thanks for your comments. > > > > I think supporting Lambdas for `withKey` and `AbstractRichFunction` > >> don't go togeth

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

s to get Lambdas for `withKey` interfaces too much because we have > already so many overlaods. On the other hand, I do see value in > supporting Lambdas for `withKey`. > > Depending on what we want to support, it might make sense to > include/exclude RichFunctions from this KIP -- and th

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

hat about KStream-KTable joins? You don't have overlaods added for > > >> them. Why? (Even if I still hope that we don't need to add any new > > >> overloads) > > >> > > >> - Why do we need `AbstractRichFunction`? > > >> > &

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

d the new > overlaod. > > And yes, we need to do one check -- but this happens when building the > topology. At runtime (I mean after KafkaStream#start() we don't need any > check as we will always use `ValueMapperWithKey`) > > > -Matthias > > > On 5/9/17 2:55 AM, Je

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

One correction: and in runtime (inside processor) we still have to check it is ValueMapper > or ValueMapperWithKey before wrapping it into the rich function. this will be in compile time in API level. Cheers, Jeyhun On Tue, May 9, 2017 at 11:55 AM Jeyhun Karimov <je.kari...@gma

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

chFunctions). > > > > This approach should not effect lambdas (or do I miss something?) and > > might be cleaner, as we could have one more top level interface > > `RichFunction` with methods `init()` and `close()` and also interfaces > > for `RichValueMapper` etc. (thu

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

like that would require subclassing RichFunction? That's a > bit of an inconvenience IMO. > > Cheers, > > Michal > > On 07/05/17 01:29, Jeyhun Karimov wrote: > > Hi, > > Thanks for comments. I extended PR and KIP to include rich functions. I > will still have to evalu

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

gt; > We do have a performance system test. Maybe it's worth for you to apply > > the deep copy strategy and run the test. It's very basic performance > > test only, but might give some insight. If you want to do this, look > > into folder "tests" for general test se

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

terface that > > would allow users to access record metadata (like timestamp, offset, > > partition etc) within DSL. This would be a similar concept. Thus, I am > > wondering, if it would make sense to enlarge the scope of this KIP by > > that? WDYT? > > > > &g

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

gned. If the key type is a mutable object (eg. byte[]), > it can still be mutated. (eg. key[0] = 0). But I'm not really sure there's > much that can be done about that. > > Mathieu > > > On Mon, May 1, 2017 at 5:39 PM, Jeyhun Karimov <je.kari...@gmail.com> > wrote: > &

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

t; > I just want to add my voice that, I too, have wished for access to the > > record key during a mapValues or similar operation. > > > > On the other hand, the number of compile failures that would need to be > > fixed from this change is unfortunate. :-) But at least it

[DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Dear community, I want to share KIP-149 [1] based on issues KAFKA-4218 [2], KAFKA-4726 [3], KAFKA-3745 [4]. The related PR can be found at [5]. I would like to get your comments. [1]

[jira] [Comment Edited] (KAFKA-4785) Records from internal repartitioning topics should always use RecordMetadataTimestampExtractor

[ https://issues.apache.org/jira/browse/KAFKA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989838#comment-15989838 ] Jeyhun Karimov edited comment on KAFKA-4785 at 4/29/17 6:08 PM: [~mjsax

[jira] [Commented] (KAFKA-3745) Consider adding join key to ValueJoiner interface

[ https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989988#comment-15989988 ] Jeyhun Karimov commented on KAFKA-3745: --- [~sree2k] is there an update on issue? > Consider add

[jira] [Commented] (KAFKA-4785) Records from internal repartitioning topics should always use RecordMetadataTimestampExtractor

[ https://issues.apache.org/jira/browse/KAFKA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989838#comment-15989838 ] Jeyhun Karimov commented on KAFKA-4785: --- [~mjsax], as it is discussed in KAFKA-4218 , we can group

[jira] [Commented] (KAFKA-4218) Enable access to key in ValueTransformer

[ https://issues.apache.org/jira/browse/KAFKA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989837#comment-15989837 ] Jeyhun Karimov commented on KAFKA-4218: --- [~mjsax], thanks for your comments. I absolutely agree

[jira] [Assigned] (KAFKA-4218) Enable access to key in ValueTransformer

[ https://issues.apache.org/jira/browse/KAFKA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4218: - Assignee: Jeyhun Karimov > Enable access to key in ValueTransfor

[jira] [Commented] (KAFKA-4785) Records from internal repartitioning topics should always use RecordMetadataTimestampExtractor

[ https://issues.apache.org/jira/browse/KAFKA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989557#comment-15989557 ] Jeyhun Karimov commented on KAFKA-4785: --- [~mjsax] Do you think KAFKA-4144 is blocking for this issue

[jira] [Assigned] (KAFKA-4785) Records from internal repartitioning topics should always use RecordMetadataTimestampExtractor

[ https://issues.apache.org/jira/browse/KAFKA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4785: - Assignee: Jeyhun Karimov > Records from internal repartitioning topics should always

Re: Splitting tasks in streams?

PM Eno Thereska <eno.there...@gmail.com> wrote: > Hi Jeyhun, > > You make a good observation and I think a discussion/contribution around > this would be very much appreciated by the community. Are you thinking of a > KIP perhaps? > > Eno > > > On 28 Apr 201

Splitting tasks in streams?

Hi community, I have a question regarding with streams library. Currently, in kafka-streams we run the whole topology in one instance and there can be several topologies or tasks in a single node. However, there can be use-cases with very complex topologies with costly operators. So, when we

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

gt; > On Tue, 2017-02-28 at 08:59 +, Jeyhun Karimov wrote: > > Dear community, > > > > I'd like to start the vote for KIP-123: > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=6871 > > 4788 > > > > > > Cheers, > >

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983515#comment-15983515 ] Jeyhun Karimov commented on KAFKA-4144: --- [~mjsax] I am sorry, I didn't read your email carefully. My

[jira] [Updated] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov updated KAFKA-4144: -- Status: Reopened (was: Closed) > Allow per stream/table timestamp extrac

[jira] [Closed] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov closed KAFKA-4144. - > Allow per stream/table timestamp extrac

[jira] [Resolved] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov resolved KAFKA-4144. --- Resolution: Fixed Reviewer: Matthias J. Sax > Allow per stream/table timest

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

> wrote: > > > > Thanks Jeyhun. > > > > Can you also update the KIP accordingly. It must contain all changes to > > public API. Thus, list all parameters that get deprecated and newly > > added. And add a sentence about backward compatibility. > > >

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

sides the minor comments mentioned above. > > > Guozhang > > > On Mon, Mar 6, 2017 at 3:14 PM, Jeyhun Karimov <je.kari...@gmail.com> > wrote: > > > Sorry I was late. I will update javadocs in related methods to emphasize > > that TimestampExtractor is statele

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

gt; > On Wed, Mar 1, 2017 at 6:09 AM, Michael Noll <mich...@confluent.io> > > wrote: > > > > > >> +1 (non-binding) > > >> > > >> Thanks for the KIP! > > >> > > >> On Wed, Mar 1, 2017 at 1:49 PM, Bill Bejeck <bbej...@gmail.com&g

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

. If the join is not allowed, how do you prevent that join from > happening? Do you throw an exception? > > Thanks > Eno > > > > On 28 Feb 2017, at 10:04, Jeyhun Karimov <je.kari...@gmail.com> wrote: > > > > Hi Eno, > > > > Thanks for feedback. I thin

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

time vs. wallclock time) in their > processing inadvertently. Before this KIP, all the relevant topics have one > time stamp extractor so that issue does not come up. > > What will be the behavior if times mismatch, e.g., for joins? > > Thanks > Eno > > > On 22 Feb 2017,

[VOTE] KIP-123: Allow per stream/table timestamp extractor

Dear community, I'd like to start the vote for KIP-123: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788 Cheers, Jeyhun -- -Cheers Jeyhun

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

verloads with independent optional > parameters. > > :-) > > > > eg. stream(), table(), globalTable(), addSource(), could all accept a > > "TopicReference" parameter that can be built like: > > > TopicReference("my-topic").keySerde(...).value

  1   2   >