Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-05-05 Thread Guozhang Wang
Hi Ewen, Personally I'm thinking to remove deprecated APIs after one minor release, so yes, 0.12.0.0 would be the preferred timeline. Guozhang On Thu, Apr 6, 2017 at 8:40 PM, Ewen Cheslack-Postava wrote: > I agree there's a potential issue that Eno raises, but probably it's not > worth worryi

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-04-06 Thread Ewen Cheslack-Postava
I agree there's a potential issue that Eno raises, but probably it's not worth worrying about trying to enforce it. It seems like we have the right default for long term and for most cases a custom timestamp extractor should only be needed for legacy data that isn't putting a timestamp in the recor

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-28 Thread Matthias J. Sax
Eno, I think this problem is out-of-scope and also present in the current setting. We cannot avoid that a custom timestamp extractor uses and if-else branch and returns different timestamps for different topic. That is possible even right now. Furthermore, the TimestampExtractor interface states:

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-28 Thread Jeyhun Karimov
Hi Eno, Thanks for clarification. I think it is by definition allowed. So if we want to join a stream that uses wallclock time with a stream that uses an event time, then we can assign the first one a timestamp extractor that returns system clock, and for the second stream we can assign timestamp

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-28 Thread Eno Thereska
Hi Jeyhun, I mean something slightly different. In your motivation you say "joining multiple streams/tables that require different timestamp extraction methods". I wan to understand the scope of this. Is it allowed to have a stream that uses wallclock time join a stream that uses event time? (I

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-28 Thread Jeyhun Karimov
Hi Eno, Thanks for feedback. I think you mean [1]. In this KIP we do not consider the situations you mentioned. So, either we can extend the KIP and solve mentioned issues or submit 2 PRs incrementally. [1] https://issues.apache.org/jira/browse/KAFKA-4785 Cheers, Jeyhun On Tue, Feb 28, 2017 a

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-28 Thread Eno Thereska
Hi Jeyhun, Thanks for the KIP, sorry I'm coming a bit late to the discussion. One thing I'd like to understand is whether we can avoid situations where the user is mixing different times (event time vs. wallclock time) in their processing inadvertently. Before this KIP, all the relevant topics

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-22 Thread Jeyhun Karimov
Dear community, I would like to get further feedbacks on this KIP (if any). Cheers Jeyhun On Wed, Feb 15, 2017 at 2:36 AM Matthias J. Sax wrote: > Mathieu, > > I personally agree with your observation, and we have plans to submit a > KIP like this. If you want to drive this discussion feel fre

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-16 Thread Jeyhun Karimov
Hi Matthias, Done. On Thu, Feb 16, 2017 at 7:24 PM Matthias J. Sax wrote: > Jeyhun, > > can you please add the KIP to this table: > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-KIPsunderdiscussion > > and to this list: > https://cwiki

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-16 Thread Matthias J. Sax
Jeyhun, can you please add the KIP to this table: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-KIPsunderdiscussion and to this list: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams Thanks! -Matthias On 2/14/17 5:36 PM,

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-14 Thread Matthias J. Sax
Mathieu, I personally agree with your observation, and we have plans to submit a KIP like this. If you want to drive this discussion feel free to start the KIP by yourself! Having said that, for this KIP we might want to focus the discussion the the actual feature that gets added: allowing to spe

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-14 Thread Mathieu Fenniak
Hi Jeyhun, This KIP might not be the appropriate time, but my first thought reading it is that it might make sense to introduce a builder-style API rather than adding a mix of new method overloads with independent optional parameters. :-) eg. stream(), table(), globalTable(), addSource(), could a

[DISCUSS] KIP-123: Allow per stream/table timestamp extractor

2017-02-14 Thread Jeyhun Karimov
Dear community, I want to share the KIP-123 [1] which is based on issue KAFKA-4144 [2]. You can check the PR in [3]. I would like to get your comments. [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788 [2] https://issues.apache.org/jira/browse/KAFKA-4144 [3] https://g