Re: Kafka streams exactly_once auto commit timeout transaction issue

2019-02-20 Thread Guozhang Wang
Glad to hear that. I think it worth adding an FAQ entry as it seems to be a common scenarios that users forgot to config on the final consumption stage. Guozhang On Tue, Feb 19, 2019 at 4:28 AM Xander Uiterlinden wrote: > Thanks for your reply. I figured out what was wrong, and it turned out

Re: Kafka streams exactly_once auto commit timeout transaction issue

2019-02-19 Thread Xander Uiterlinden
Thanks for your reply. I figured out what was wrong, and it turned out to be a stupid mistake at my end as I did not use a consumer with isolation level "read_committed" to verify. Xander On Fri, Feb 8, 2019 at 8:58 PM Guozhang Wang wrote: > Hello Xander, > > Upon committing the state with

Re: Kafka streams exactly_once auto commit timeout transaction issue

2019-02-08 Thread Guozhang Wang
Hello Xander, Upon committing the state with `exactly_once`, Streams will commit the transaction by going through the commit protocol (details can be found here [1]). So I guess the following happened in time: 1) one record gets read in. 2) processing the record by traversing the topology, not

Kafka streams exactly_once auto commit timeout transaction issue

2019-02-06 Thread Xander Uiterlinden
Hi, I'm trying to get a fairly simple example of using Kafka Streams with exactly once processing to work. I defined a setup where messages are being read from an input topic and two streams transform and output the result to their own output topic. In normal conditions this works fine, i.e. when