Re: committing offset metadata in kafka streams

Sure. Sorry I was not clear. Thank you! lör 10 mars 2018 kl. 00:54 skrev Matthias J. Sax : > If there is only one partition by task, processing order is guaranteed. > > For default partitions grouper, it depends on your program. If you read > from multiple topics and

Re: committing offset metadata in kafka streams

If there is only one partition by task, processing order is guaranteed. For default partitions grouper, it depends on your program. If you read from multiple topics and join/merge them, a task gets multiple partitions (from different topics) assigned. -Matthias On 3/9/18 2:42 PM, Stas Chizhov

Re: committing offset metadata in kafka streams

> Also note, that the processing order might slightly differ if you process the same data twice Is this still a problem when default partition grouper is used (with 1 partition per task)? Thank you, Stanislav. 2018-03-09 3:19 GMT+01:00 Matthias J. Sax : > Thanks

Re: committing offset metadata in kafka streams

Thanks for the explanation. Not sure if setting the metadata you want to get committed in punctuation() would be sufficient. But I would think about it in more details if we get a KIP for this. It's correct that flushing and committing offsets is correlated. But it's not related to punctuation.

Re: committing offset metadata in kafka streams

Thank you, Matthias! We currently do use kafka consumer and store event time highwatermarks as offset metadata. This is used during backup procedure, which is to create a snapshot of the target storage with all events up to certain timestamp and no other. As for the API - I guess being able to

Re: committing offset metadata in kafka streams

You are correct. This is not possible atm. Note, that commits happen "under the hood" and users cannot commit explicitly. Users can only "request" as commit -- this implies that Kafka Streams will commit as soon as possible -- but when `context#commit()` returns, the commit is not done yet (it

committing offset metadata in kafka streams

Hi, There seems to be no way to commit custom metadata along with offsets from within Kafka Streams. Are there any plans to expose this functionality or have I missed something? Best regards, Stanislav.