Thanks Thomas,
I'm also wary of changing the existing semantics of punctuate, for
backward compatibility reasons, although I like the conceptual
simplicity of that option.
Adding a new method to me feels safer but, in a way, uglier. I added
this to the KIP now as option (C).
The TimestampExtractor mechanism is actually more flexible, as it allows
you to return any value, you're not limited to event time or system time
(although I don't see an actual use case where you might need anything
else then those two). Hence I also proposed the option to allow users
to, effectively, decide what "stream time" is for them given the
presence or absence of messages, much like they can decide what msg time
means for them using the TimestampExtractor. What do you think about
that? This is probably most flexible but also most complicated.
All comments appreciated.
Cheers,
Michal
On 03/04/17 19:23, Thomas Becker wrote:
Although I fully agree we need a way to trigger periodic processing
that is independent from whether and when messages arrive, I'm not sure
I like the idea of changing the existing semantics across the board.
What if we added an additional callback to Processor that can be
scheduled similarly to punctuate() but was always called at fixed, wall
clock based intervals? This way you wouldn't have to give up the notion
of stream time to be able to do periodic processing.
On Mon, 2017-04-03 at 10:34 +0100, Michal Borowiecki wrote:
Hi all,
I have created a draft for KIP-138: Change punctuate semantics
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-138%3A+Change+
punctuate+semantics>
.
Appreciating there can be different views on system-time vs event-
time
semantics for punctuation depending on use-case and the importance of
backwards compatibility of any such change, I've left it quite open
and
hope to fill in more info as the discussion progresses.
Thanks,
Michal
--
Tommy Becker
Senior Software Engineer
O +1 919.460.4747
tivo.com
________________________________
This email and any attachments may contain confidential and privileged material
for the sole use of the intended recipient. Any review, copying, or
distribution of this email (or any attachments) by others is prohibited. If you
are not the intended recipient, please contact the sender immediately and
permanently delete this email and any attachments. No employee or agent of TiVo
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by
email. Binding agreements with TiVo Inc. may only be made by a signed written
agreement.
