That sound correct, Colin. At runtime (we just merged an improvement this week, cf KIP-353), Kafka Streams synchronizes different topics based on record timestamps. Records are buffered internally before processed and we `pause()` partitions for which the number of records in the buffer exceeds a configurable threshold (`buffered.records.per.partition` parameter).
Timestamp based synchronization (ie, message choosing) is essential for DSL semantics. A custom MessageChosser might break DSL operator semantics. Having said this, there might be use cases for with timestamps based synchronization is not desired. However, I would assume that this would be a Processor API level feature, not a DSL level feature. Hence, offering something similar to MessageChooser interface at Processor API level that is leveraged at runtime might make sense. For this case, the DSL would plug-in its timestamp based synchronization strategy. Take this with a grain of salt though. I have not thought this through and it might actually not be possible to express the needed timestamp synchronization with a MessageChooser interface. -Matthias On 9/10/18 10:54 AM, Colin McCabe wrote: > Hmm. My understanding is that streams doesn't need anything like this since > streams pauses topics when it doesn't need more data from them. (Matthias, > can you confirm?) > > best, > Colin > > > On Mon, Aug 20, 2018, at 06:01, Thomas Becker wrote: >> I agree with Jan. A strategy interface for choosing processing order is >> nice, and would hopefully be a step towards getting this in streams. >> >> -Tommy >> >> On Mon, 2018-08-20 at 12:52 +0200, Jan Filipiak wrote: >> >> On 20.08.2018 00:19, Matthias J. Sax wrote: >> >> @Nick: A KIP is only accepted if it got 3 binding votes, ie, votes from >> >> committers. If you close the vote before that, the KIP would not be >> >> accepted. Note that committers need to pay attention to a lot of KIPs >> >> and it can take a while until people can look into it. Thanks for your >> >> understanding. >> >> >> @Jan: Can you give a little bit more context on your concerns? It's >> >> unclear why you mean atm. >> >> Just saying that we should peek at the Samza approach, it's a much more >> >> powerful abstraction. We can ship a default MessageChooser >> >> that looks at the topics priority. >> >> @Adam: anyone can vote :) >> >> >> >> >> -Matthias >> >> >> On 8/19/18 9:58 AM, Adam Bellemare wrote: >> >> While I am not sure if I can or can’t vote, my question re: Jan’s >> comment is, “should we be implementing it as Samza does?” >> >> >> I am not familiar with the drawbacks of the current approach vs how >> samza does it. >> >> >> On Aug 18, 2018, at 5:06 PM, >> n...@afshartous.com<mailto:n...@afshartous.com> wrote: >> >> >> >> I only saw one vote on KIP-349, just checking to see if anyone else >> would like to vote before closing this out. >> >> -- >> >> Nick >> >> >> >> On Aug 13, 2018, at 9:19 PM, >> n...@afshartous.com<mailto:n...@afshartous.com> wrote: >> >> >> >> Hi All, >> >> >> Calling for a vote on KIP-349 >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics >> >> >> -- >> >> Nick >> >> >> >> >> >> >> >> >> >> ________________________________ >> >> This email and any attachments may contain confidential and privileged >> material for the sole use of the intended recipient. Any review, >> copying, or distribution of this email (or any attachments) by others is >> prohibited. If you are not the intended recipient, please contact the >> sender immediately and permanently delete this email and any >> attachments. No employee or agent of TiVo Inc. is authorized to conclude >> any binding agreement on behalf of TiVo Inc. by email. Binding >> agreements with TiVo Inc. may only be made by a signed written >> agreement.
signature.asc
Description: OpenPGP digital signature
