Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

2018-06-20 Thread Jeyhun Karimov
hrough()` and `to()`, can you add > >> the different behavior using different overloads? It's not clear from > >> the KIP what the semantics are. > >> > >> > >> -Matthias > >> > >> On 11/17/17 3:27 PM, Jeyhun Karimov wrote: > >>>

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

2018-10-04 Thread Jeyhun Karimov
are >> interested to drive this further. So we will just "reassign" it to them. >> >> Thanks for letting us know. >> >> >> -Matthias >> >> On 6/20/18 2:51 PM, Jeyhun Karimov wrote: >> > Hi Matthias, all, >> > >> >

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-09-21 Thread Jeyhun Karimov
gt; > On Thu, Sep 14, 2017 at 10:37 PM, Ted Yu wrote: > > > > > +1 > > > > > > One interface is cleaner. > > > > > > On Thu, Sep 14, 2017 at 7:26 AM, Bill Bejeck > wrote: > > > > > > > +1 for me on collapsing the Rich and

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-09-21 Thread Jeyhun Karimov
21 Sep 2017 at 15:23 Jeyhun Karimov wrote: > > > Hi all, > > > > Thanks a lot for your comments. For the single interface (RichXXX and > > XXXWithKey) solution, I have already submitted a PR but probably it is > > outdated (when the KIP first proposed), I need to

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-09-22 Thread Jeyhun Karimov
Dear community, I updated the related KIP [1]. Please feel free to comment. Cheers, Jeyhun [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-159%3A+Introducing+Rich+functions+to+Streams On Fri, Sep 22, 2017 at 12:20 AM Jeyhun Karimov wrote: > Hi Damian, > > Thanks for t

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-09-22 Thread Jeyhun Karimov
tion("commit() is not supported in > this context"); > > Is the exception going to be replaced with real code in the PR ? > > Cheers > > > On Fri, Sep 22, 2017 at 9:28 AM, Jeyhun Karimov > wrote: > > > Dear community, > > > > I updated the rela

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-10-16 Thread Jeyhun Karimov
want to move `commit()` from ProcessorContext to > RecordContext? Conceptually I think it would better staying in the > ProcessorContext. Do you find this not doable in the internal > implementations? > > > Guozhang > > > > On Fri, Sep 22, 2017 at 1:09 PM, Ted Yu

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-10-18 Thread Jeyhun Karimov
u call that function, it means we would commit the state > of the whole task up to this processed record, not only that single record > itself. > > > Guozhang > > On Mon, Oct 16, 2017 at 9:19 AM, Jeyhun Karimov > wrote: > > > Hi, > > > > Thanks for the

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-10-23 Thread Jeyhun Karimov
hould fix KAFKA-3907 at all. > Manual commits are something DSL users should not worry about -- and if > one really needs this, an advanced user can still insert a dummy > `transform` to request a commit from there. > > -Matthias > > > On 10/18/17 5:39 AM, Jeyhun Karimov wrote

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-10-26 Thread Jeyhun Karimov
17 at 1:40 PM, Matthias J. Sax > wrote: > > > Fair point. This is a long discussion and I totally forgot that we > > discussed this. > > > > Seems I changed my opinion about including KAFKA-3907... > > > > Happy to hear what others think. > > > >

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-10-27 Thread Jeyhun Karimov
> > To me, this does not seem to be a sound API design if we follow this path. > > > -Matthias > > > > On 10/26/17 10:41 PM, Jeyhun Karimov wrote: > > Hi, > > > > Thanks for your suggestions. > > > > I have some comments, to make sure that th

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-10-31 Thread Jeyhun Karimov
't agree that > > > but also we need a commit() method > > I would just not provide `commit()` at DSL level and close the > corresponding Jira as "not a problem" or similar. > > > -Matthias > > On 10/27/17 3:42 PM, Jeyhun Karimov wrote: > > Hi Matth

[VOTE] KIP-159: Introducing Rich functions to Streams

2017-11-01 Thread Jeyhun Karimov
Dear community, It seems the discussion for KIP-159 [1] converged finally. I would like to initiate voting for the particular KIP. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-159%3A+Introducing+Rich+functions+to+Streams Cheers, Jeyhun

[DISCUSS] KIP-221: Repartition Topic Hints in Streams

2017-11-04 Thread Jeyhun Karimov
Dear community, I would like to initiate discussion on KIP-221 [1] based on issue [2]. Please feel free to comment. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Repartition+Topic+Hints+in+Streams [2] https://issues.apache.org/jira/browse/KAFKA-6037 Cheers, Jeyhun

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

2017-11-04 Thread Jeyhun Karimov
the back ground > > and picky back all to `Produced`. > > > > > > -Matthias > > > > On 11/4/17 4:16 PM, Ted Yu wrote: > > > API is given without much javadoc on the role / meaning of method > > > parameters. > > > > > > Can y

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

2017-11-06 Thread Jeyhun Karimov
wrote: > >> -1 non binding > >> > >> I don't get the motivation. > >> In 80% of my DSL processors there is no such thing as a reasonable > >> RecordContext. > >> After a join the record I am processing belongs to at least 2 topics. > &g

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

2017-11-06 Thread Jeyhun Karimov
't want to manage topic manually, thus, it's still an > >> internal topic and Streams create the topic name automatically as for > >> all other internal topics. However, users gets some more control about > >> topic parameters like number of partitions (w

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-11-06 Thread Jeyhun Karimov
yhun, thanks, looks good. > >> Do we need to remove the line that says: > >> > >>- on-demand commit() feature > >> > >> Cheers, > >> Damian > >> > >> On Tue, 31 Oct 2017 at 23:07 Jeyhun Karimov > wrote: > >> &g

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

2017-11-07 Thread Jeyhun Karimov
ep it that way until #6. In addition, the > >>>> RecordContext > >>>> fields (topic, offset, etc) are really orthogonal to the key-value > >>>> payloads > >>>> themselves, so I think separating them into this object is a > >>>>

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

2017-11-17 Thread Jeyhun Karimov
ponsible for managing them, including the topic name. For this KIP's > >> purpose, though, users would not care about the topic name. I.e. as a > >> user > >> I still want to make it be an internal topic so that I do not need to > >> worry > >> a

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

2017-11-18 Thread Jeyhun Karimov
: I never ran a topology with > caching > >>>>>>>> but I > >>>>>>>> am not even 100% sure what the record Context means behind > >>>>>>>> a materialized KTable with Caching? Topic and Partition are > pr

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-06-13 Thread Jeyhun Karimov
, Guozhang Wang wrote: > > Thanks for the comprehensive summary! > > > > Personally I'd prefer the option of passing RecordContext as an > additional > > parameter into he overloaded function. But I'm also open to other > arguments > > if there are sth. that I have

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-06-14 Thread Jeyhun Karimov
think changing the interface to extend from a new interface is not > binary > > compatible though source compatible, i.e. users still need to recompile > > their code though no need to make code changes. We may need to mention > that > > in the upgrade path if we want to keep i

Re: Handling 2 to 3 Million Events before Kafka

2017-06-21 Thread Jeyhun Karimov
Hi, With kafka you can increase overall throughput by increasing the number of nodes in a cluster. I had a similar issue, where we needed to ingest vast amounts of data to streaming system. In our case, kafka was a bottleneck, because of disk I/O. To solve it, we implemented (simple) distributed

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-06-27 Thread Jeyhun Karimov
address this issue as well or continue as it is? Cheers, Jeyhun On Wed, Jun 14, 2017 at 11:29 PM Guozhang Wang wrote: > LGTM. Thanks! > > > Guozhang > > On Tue, Jun 13, 2017 at 2:20 PM, Jeyhun Karimov > wrote: > > > Thanks for the comment Matthias. After all

[VOTE]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-06-27 Thread Jeyhun Karimov
Dear all, I would like to start the vote on KIP-149 [1]. Cheers, Jeyhun [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner -- -Cheers Jeyhun

Re: Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

2017-06-27 Thread Jeyhun Karimov
) or similar? So that users don't have to read the docs > to know it isn't the creation timestamp for instance. > Cheers, > Michał > > > On 04/06/17 01:24, Jeyhun Karimov wrote: > > Hi Matthias, > > Thanks for comments. > > - why do you only cons

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-07-04 Thread Jeyhun Karimov
rade guidance. > >> > >> Regarding the scope I'm still trying to solicit opinions regarding > >> ReducerWithKey and InitializerWithKey; to me they are not necessarily > to be > >> included. > >> > >> > >> Guozhang > >> > >> > &

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-07-04 Thread Jeyhun Karimov
Context. This will > > break compatibility. > > > > I think, we should just have two independent interfaces. Our own > > ProcessorContextImpl class would implement both. This allows us to cast > > it to `RecordContext` and thus limit the visible scope. > > > > &

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-07-06 Thread Jeyhun Karimov
g: > richMapper.init((RecordContext) processorContext); > But the interface is: > public interface RichValueMapper { > VR apply(final V value, final RecordContext recordContext); > } > i.e., there is no init(...), besides as above this wouldn't make sense. > > Thanks,

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-07-07 Thread Jeyhun Karimov
rnalProcessorContext to ProcessorContext. > > > > > > In the KIP you have an example showing: > > > richMapper.init((RecordContext) processorContext); > > > But the interface is: > > > public interface RichValueMapper { > > > VR apply(final V value, fi

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-07-20 Thread Jeyhun Karimov
wrote: > I would not block this KIP with regard to DSL refactoring. IMHO, we can > just finish this one and the DSL refactoring will help later on to > reduce the number of overloads. > > -Matthias > > On 7/7/17 5:28 AM, Jeyhun Karimov wrote: > > I am following the related

Re: [VOTE]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-07-21 Thread Jeyhun Karimov
(non-binding) Thanks. > > > > > > > > Eno > > > > > On 6 Jul 2017, at 21:49, Gwen Shapira wrote: > > > > > > > > > > +1 > > > > > > > > > > On Wed, Jul 5, 2017 at 9:25 AM Matthias J. Sax < > > matth...@confl

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-09-13 Thread Jeyhun Karimov
lease feel free to comment on this. [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=73637757 Cheers, Jeyhun On Fri, Jul 21, 2017 at 2:00 AM Jeyhun Karimov wrote: > Hi Matthias, Damian, all, > > Thanks for your comments and sorry for super-late update. > > Sur

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-09-13 Thread Jeyhun Karimov
Rich and ValueWithKey etc > interfaces into 1 interface that has all of the arguments. I think we then > only need to add one additional overload for each operator? > > Thanks, > Damian > > On Wed, 13 Sep 2017 at 10:59 Jeyhun Karimov wrote: > > > Dear all, > > > &g

Operator order

2016-06-18 Thread Jeyhun Karimov
Hi community, Is there a way in Kafka Streams to change the order of operators in runtime? For example, I have operators Source->A->B->C->D->E->Sink and I want to forward some tuples from A to E, from B to Sink and etc. As far as I know, the stream execution graph is computed in compile time and

Re: Operator order

2016-06-18 Thread Jeyhun Karimov
> you? Do you mean you would dynamically change the edge between A->B and > A->sink ? I guess, this would be a very special pattern and I doubt that > any library or system can offer this. > > -Matthias > > On 06/18/2016 05:33 PM, Jeyhun Karimov wrote: > > Hi co

Reduce function Null checks

2016-06-19 Thread Jeyhun Karimov
Hi community, When using, reduce(Reducer,Reducer, KeyValueMapper,String) function in KTable, the NullPointerExeption is thrown. In specified function, below call is made: return reduce(adder, subtractor, selector, null, null, name); and afterwards, in reduce(Reducer,Reducer, KeyValueMapp

Re: Operator order

2016-06-20 Thread Jeyhun Karimov
r operators will run in the same thread (it's more or less just > > another chained method call), thus, it should not be too large. > > Furthermore, it should never the required to write tagged record to > > Kafka -- thus, it would only be some main memory overhead. But you would >

Parallelisation factor in kafka streams

2016-07-06 Thread Jeyhun Karimov
Hi community, How can I set parallelisation factor in kafka streams? Is it related with the number of partitions within topics? Cheers Jeyhun -- -Cheers Jeyhun

Re: Parallelisation factor in kafka streams

2016-07-06 Thread Jeyhun Karimov
um.stream.threads" (default value is 1). > > See > > http://docs.confluent.io/3.0.0/streams/developer-guide.html#optional-configuration-parameters > > > -Matthias > > On 07/06/2016 06:11 PM, Jeyhun Karimov wrote: > > Hi community, > > > > How can I set

Re: Parallelisation factor in kafka streams

2016-07-06 Thread Jeyhun Karimov
t;? For sure, you do not > need to restart the Kafka Brokers. And for a Streams application, there > is no cluster. Streams applications are regular Java applications and > can run anywhere (not necessarily on the same machines as Kafka Brokers). > > -Matthias > > On 07/06/2016

[VOTE] KIP-123: Allow per stream/table timestamp extractor

2017-02-28 Thread Jeyhun Karimov
Dear community, I'd like to start the vote for KIP-123: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788 Cheers, Jeyhun -- -Cheers Jeyhun

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-28 Thread Jeyhun Karimov
r > processing inadvertently. Before this KIP, all the relevant topics have one > time stamp extractor so that issue does not come up. > > What will be the behavior if times mismatch, e.g., for joins? > > Thanks > Eno > > > On 22 Feb 2017, at 09:21, Jeyhun Karimov

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-28 Thread Jeyhun Karimov
how do you prevent that join from > happening? Do you throw an exception? > > Thanks > Eno > > > > On 28 Feb 2017, at 10:04, Jeyhun Karimov wrote: > > > > Hi Eno, > > > > Thanks for feedback. I think you mean [1]. In this KIP we do not consider > >

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

2017-03-06 Thread Jeyhun Karimov
ll > > wrote: > > > > > >> +1 (non-binding) > > >> > > >> Thanks for the KIP! > > >> > > >> On Wed, Mar 1, 2017 at 1:49 PM, Bill Bejeck > wrote: > > >> > > >>> +1 > > >>> > > >

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

2017-03-23 Thread Jeyhun Karimov
omments mentioned above. > > > Guozhang > > > On Mon, Mar 6, 2017 at 3:14 PM, Jeyhun Karimov > wrote: > > > Sorry I was late. I will update javadocs in related methods to emphasize > > that TimestampExtractor is stateless. > > > > Cheers, > > Je

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

2017-03-24 Thread Jeyhun Karimov
; Can you also update the KIP accordingly. It must contain all changes to > > public API. Thus, list all parameters that get deprecated and newly > > added. And add a sentence about backward compatibility. > > > > > > -Matthias > > > > On 3/23/17 3:16 AM, J

[DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-14 Thread Jeyhun Karimov
Dear community, I want to share the KIP-123 [1] which is based on issue KAFKA-4144 [2]. You can check the PR in [3]. I would like to get your comments. [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788 [2] https://issues.apache.org/jira/browse/KAFKA-4144 [3] https://g

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-16 Thread Jeyhun Karimov
ke sense to introduce a builder-style API rather than > >> adding a mix of new method overloads with independent optional > parameters. > >> :-) > >> > >> eg. stream(), table(), globalTable(), addSource(), could all accept a > >> "TopicReference"

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-22 Thread Jeyhun Karimov
dent optional > parameters. > > :-) > > > > eg. stream(), table(), globalTable(), addSource(), could all accept a > > "TopicReference" parameter that can be built like: > > > TopicReference("my-topic").keySerde(...).valueSerde(...).autoOffsetReset(...).

Streams support for Serdes

2016-09-19 Thread Jeyhun Karimov
Hi community, When using kafka-streams with POJO data types we write our own de/serializers. However I think if we have built-in Serdes support for Tuple-n data types (ex:Serdes.Tuple2) we may easily use Tuples and built-in Serdes can help to reduce the development cycle. Please correct me if I am

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

2017-04-25 Thread Jeyhun Karimov
-28 at 08:59 +0000, Jeyhun Karimov wrote: > > Dear community, > > > > I'd like to start the vote for KIP-123: > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=6871 > > 4788 > > > > > > Cheers, > > Jeyhun > -- > >

Splitting tasks in streams?

2017-04-28 Thread Jeyhun Karimov
Hi community, I have a question regarding with streams library. Currently, in kafka-streams we run the whole topology in one instance and there can be several topologies or tasks in a single node. However, there can be use-cases with very complex topologies with costly operators. So, when we wan

Re: Splitting tasks in streams?

2017-04-28 Thread Jeyhun Karimov
17 at 5:42 PM Eno Thereska wrote: > Hi Jeyhun, > > You make a good observation and I think a discussion/contribution around > this would be very much appreciated by the community. Are you thinking of a > KIP perhaps? > > Eno > > > On 28 Apr 2017, at 16:13, Jey

[DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-01 Thread Jeyhun Karimov
Dear community, I want to share KIP-149 [1] based on issues KAFKA-4218 [2], KAFKA-4726 [3], KAFKA-3745 [4]. The related PR can be found at [5]. I would like to get your comments. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMappe

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-01 Thread Jeyhun Karimov
nt to add my voice that, I too, have wished for access to the > > record key during a mapValues or similar operation. > > > > On the other hand, the number of compile failures that would need to be > > fixed from this change is unfortunate. :-) But at least it would all be >

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-03 Thread Jeyhun Karimov
utable object (eg. byte[]), > it can still be mutated. (eg. key[0] = 0). But I'm not really sure there's > much that can be done about that. > > Mathieu > > > On Mon, May 1, 2017 at 5:39 PM, Jeyhun Karimov > wrote: > > > Thanks for comments. > > > &

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-05 Thread Jeyhun Karimov
would allow users to access record metadata (like timestamp, offset, > > partition etc) within DSL. This would be a similar concept. Thus, I am > > wondering, if it would make sense to enlarge the scope of this KIP by > > that? WDYT? > > > > > > > > -Matthi

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-06 Thread Jeyhun Karimov
. Maybe it's worth for you to apply > > the deep copy strategy and run the test. It's very basic performance > > test only, but might give some insight. If you want to do this, look > > into folder "tests" for general test setup, and into > > "tests/k

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-07 Thread Jeyhun Karimov
ike that would require subclassing RichFunction? That's a > bit of an inconvenience IMO. > > Cheers, > > Michal > > On 07/05/17 01:29, Jeyhun Karimov wrote: > > Hi, > > Thanks for comments. I extended PR and KIP to include rich functions. I > will still have to

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-09 Thread Jeyhun Karimov
> > This approach should not effect lambdas (or do I miss something?) and > > might be cleaner, as we could have one more top level interface > > `RichFunction` with methods `init()` and `close()` and also interfaces > > for `RichValueMapper` etc. (thus, no abstract classes requir

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-09 Thread Jeyhun Karimov
One correction: and in runtime (inside processor) we still have to check it is ValueMapper > or ValueMapperWithKey before wrapping it into the rich function. this will be in compile time in API level. Cheers, Jeyhun On Tue, May 9, 2017 at 11:55 AM Jeyhun Karimov wrote: > Hi, >

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-11 Thread Jeyhun Karimov
> And yes, we need to do one check -- but this happens when building the > topology. At runtime (I mean after KafkaStream#start() we don't need any > check as we will always use `ValueMapperWithKey`) > > > -Matthias > > > On 5/9/17 2:55 AM, Jeyhun Karimov wrote: > &g

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-15 Thread Jeyhun Karimov
#x27;t have overlaods added for > > >> them. Why? (Even if I still hope that we don't need to add any new > > >> overloads) > > >> > > >> - Why do we need `AbstractRichFunction`? > > >> > > >> - What about interfaces In

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-16 Thread Jeyhun Karimov
erloaded > methods to get Lambdas for `withKey` interfaces too much because we have > already so many overlaods. On the other hand, I do see value in > supporting Lambdas for `withKey`. > > Depending on what we want to support, it might make sense to > include/exclude RichFunctions from

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-18 Thread Jeyhun Karimov
() -- if you don't want those, you would implement a > >> different interface (ie, ValueMapperWithKey) > >> > >> As an alternative, we could argue, that it is sufficient to support > >> Lambdas for the "plain" API only, but not for any "extended API". For &

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-19 Thread Jeyhun Karimov
7;t like the abstract classes: RichValueMapper, RichValueJoiner, > RichInitializer etc. Why can't they not just be interfaces? Generally we > should provide users with Intefaces to code against, not classes. > > Thanks, > Damian > > On Fri, 19 May 2017 at 00:50 Jeyhun Karim

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-20 Thread Jeyhun Karimov
llel) -- we > don't want to slow you down :) But it make the discussion and code > review easier, if we separate both IMHO. > > > -Matthias > > > On 5/19/17 2:25 AM, Jeyhun Karimov wrote: > > Hi Damian, > > > > Thanks for your comments. I think providing t

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

2017-05-20 Thread Jeyhun Karimov
gt; > Thanks for your work and for driving this, Jeyhun! :-) > > > > -Michael > > > > > > On Tue, Apr 25, 2017 at 11:11 PM, Jeyhun Karimov > > wrote: > > > > > Dear all, > > > > > > I am closing this vote now. The KIP got acce

[DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-05-20 Thread Jeyhun Karimov
Dear community, As we discussed in KIP-149 [DISCUSS] thread [1], I would like to initiate KIP for rich functions (interfaces) [2]. I would like to get your comments. [1] http://search-hadoop.com/m/Kafka/uyzND1PMjdk2CslH12?subj=Re+DISCUSS+KIP+149+Enabling+key+access+in+ValueTransformer+ValueMappe

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-05-23 Thread Jeyhun Karimov
gt; but maybe I'm just lacking in imagination, so I'm asking all this to better > understand the rationale for init() and close(). > > Thanks, > Michał > > On 20/05/17 17:05, Jeyhun Karimov wrote: > > Dear community, > > As we discussed in KIP-149 [DISCUSS] thre

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-05-24 Thread Jeyhun Karimov
ing that requires >> initialisation and closing (which implies holding state) as being a >> function. Sounds more like the Processor/Transformer kind of thing >> semantically, rather than a function. >> >> The KIP says there are multiple use-cases for this but doesn'

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-05-27 Thread Jeyhun Karimov
t;init()". This might even > allow to use Lambdas and we could keep the name RichFunction as we > preserve the nature of being a function. > > > -Matthias > > On 5/24/17 12:13 PM, Jeyhun Karimov wrote: > > Hi Michal, > > > > Thanks for your comments. I s

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-27 Thread Jeyhun Karimov
thoughts? > > > -Matthias > > > On 5/24/17 3:47 PM, Matthias J. Sax wrote: > > Jeyhun, > > > > I was just wondering if you did look into the key-deep-copy idea we > > discussed. I am curious to see what the impact might be. > > > > > >

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-28 Thread Jeyhun Karimov
lso allows to register > punctuations. Both those features will not be available via withKey() > interfaces. > > -Matthias > > > On 5/27/17 1:25 PM, Jeyhun Karimov wrote: > > Hi Matthias, > > > > Thanks for your comments. > > > > I tested the deep copy approa

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-28 Thread Jeyhun Karimov
this for ValueTransformer (and similar). > > Btw: This reminds me about KIP-159: with regard to the RichFunction we > might need a supplier pattern, too. (I'll comment on the other thread, > too.) > > > -Matthias > > On 5/28/17 5:45 AM, Jeyhun Karimov wrote:

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-05-28 Thread Jeyhun Karimov
if we duplicate > functionality. > > For this reason, it seems to be better to got with the > `#valueMapper(ValueMapper mapper, RecordContext context)` approach. > > WDYT? > > > > -Matthias > > On 5/27/17 11:00 AM, Jeyhun Karimov wrote: > > Hi, > > > > Thank

[DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

2017-05-29 Thread Jeyhun Karimov
Dear community, I want to share KIP-165 [1] based on issue KAFKA-4304 [2]. I would like to get your comments. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-165%3A+Extend+Interactive+Queries+for+return+latest+update+timestamp+per+key [2] https://issues.apache.org/jira/browse/KAFKA-4304

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-06-01 Thread Jeyhun Karimov
. Cheers, Jeyhun On Fri, Jun 2, 2017 at 2:27 AM Guozhang Wang wrote: > Does this KIP subsume this ticket as well? > https://issues.apache.org/jira/browse/KAFKA-4125 > > On Sat, May 20, 2017 at 9:05 AM, Jeyhun Karimov > wrote: > > > Dear community, > > > > As we disc

Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

2017-06-03 Thread Jeyhun Karimov
ntext not be sufficient? > - for backward compatibility, we will also need a new interface and > cannot just extend the existing one > > > > -Matthias > > On 5/29/17 4:55 PM, Jeyhun Karimov wrote: > > Dear community, > > > > I want to share KIP-165 [1] based

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-06-05 Thread Jeyhun Karimov
;> they do, we can support them easily as the above implementations; > >>> > >>> Similarly for "ReducerWithKey", it can be implemented as `Aggregator >> V, > >>> V>`, and people who needs key access can just use `aggregate` directly. > >&g

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-06-05 Thread Jeyhun Karimov
but for joins and aggregations. > >> > >> > >> On the other hand, as I mentioned in an older email, I am personally > >> fine to break the pure functional interface, and add > >> > >> - interface WithRecordContext with method `open(RecordContext)`

Re: KIP-160 - Augment KStream.print() to allow users pass in extra parameters in the printed string

2017-06-05 Thread Jeyhun Karimov
I agree with Matthias's comment. Constructing RecordContext with more metadata seems more feasible for me. Cheers, Jeyun On Mon, Jun 5, 2017 at 7:47 AM Matthias J. Sax wrote: > Not with the scope of the current discussion. > > So far, we discuss to add `RecordContext`, but the context object we

[jira] [Assigned] (KAFKA-4661) Improve test coverage UsePreviousTimeOnInvalidTimestamp

2017-06-09 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4661: - Assignee: Jeyhun Karimov > Improve test coverage UsePreviousTimeOnInvalidTimest

[jira] [Assigned] (KAFKA-4655) Improve test coverage of CompositeReadOnlySessionStore

2017-06-10 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4655: - Assignee: Jeyhun Karimov > Improve test coverage of CompositeReadOnlySessionSt

[jira] [Assigned] (KAFKA-4659) Improve test coverage of CachingKeyValueStore

2017-06-10 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4659: - Assignee: Jeyhun Karimov > Improve test coverage of CachingKeyValueSt

[jira] [Assigned] (KAFKA-4656) Improve test coverage of CompositeReadOnlyKeyValueStore

2017-06-10 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4656: - Assignee: Jeyhun Karimov > Improve test coverage of CompositeReadOnlyKeyValueSt

[jira] [Assigned] (KAFKA-4658) Improve test coverage InMemoryKeyValueLoggedStore

2017-06-10 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4658: - Assignee: Jeyhun Karimov > Improve test coverage InMemoryKeyValueLoggedSt

[jira] [Assigned] (KAFKA-4653) Improve test coverage of RocksDBStore

2017-06-10 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4653: - Assignee: Jeyhun Karimov > Improve test coverage of RocksDBSt

[jira] [Commented] (KAFKA-3826) Sampling on throughput / latency metrics recording in Streams

2017-06-12 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16046497#comment-16046497 ] Jeyhun Karimov commented on KAFKA-3826: --- [~guozhang] I think KAFKA-4829 also ca

[jira] [Comment Edited] (KAFKA-3826) Sampling on throughput / latency metrics recording in Streams

2017-06-12 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16046497#comment-16046497 ] Jeyhun Karimov edited comment on KAFKA-3826 at 6/12/17 12:3

[jira] [Assigned] (KAFKA-3907) Better support for user-specific committing in the Streams DSL

2017-06-14 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-3907: - Assignee: Jeyhun Karimov > Better support for user-specific committing in the Stre

[jira] [Commented] (KAFKA-4829) Improve logging of StreamTask commits

2017-06-14 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16049834#comment-16049834 ] Jeyhun Karimov commented on KAFKA-4829: --- [~guozhang] I would suggest similar m

[jira] [Commented] (KAFKA-3839) Handling null keys in KTable.groupBy

2016-06-14 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330865#comment-15330865 ] Jeyhun Karimov commented on KAFKA-3839: --- I can take this task as a sta

[jira] [Work started] (KAFKA-3839) Handling null keys in KTable.groupBy

2016-06-15 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3839 started by Jeyhun Karimov. - > Handling null keys in KTable.grou

[jira] [Commented] (KAFKA-3839) Handling null keys in KTable.groupBy

2016-06-17 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336310#comment-15336310 ] Jeyhun Karimov commented on KAFKA-3839: --- It seems that groupby operator is

[jira] [Assigned] (KAFKA-3836) KStreamReduce and KTableReduce should not pass nulls to Deserializers

2016-06-17 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-3836: - Assignee: Jeyhun Karimov > KStreamReduce and KTableReduce should not pass nulls

[jira] [Assigned] (KAFKA-3825) Allow users to specify different types of state stores in Streams DSL

2016-06-19 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-3825: - Assignee: Jeyhun Karimov > Allow users to specify different types of state stores

[jira] [Commented] (KAFKA-3825) Allow users to specify different types of state stores in Streams DSL

2016-06-26 Thread Jeyhun Karimov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350110#comment-15350110 ] Jeyhun Karimov commented on KAFKA-3825: --- [~guozhang] I am starting on this i

  1   2   >