Re: At Least Once semantics for Kafka Streams

Ah OK! Thanks! On Mon, Feb 6, 2017, 3:09 PM Eno Thereska wrote: > Oh, by "other" I meant the original one you started discussing: > COMMIT_INTERVAL_MS_CONFIG. > > Eno > > On 6 Feb 2017, at 09:28, Mahendra Kariya > wrote: > > > > Thanks Eno! >

Re: At Least Once semantics for Kafka Streams

Oh, by "other" I meant the original one you started discussing: COMMIT_INTERVAL_MS_CONFIG. Eno > On 6 Feb 2017, at 09:28, Mahendra Kariya wrote: > > Thanks Eno! > > I am just wondering what is this other commit parameter? > > On Mon, Feb 6, 2017, 12:52 PM Eno

Re: At Least Once semantics for Kafka Streams

Thanks Eno! I am just wondering what is this other commit parameter? On Mon, Feb 6, 2017, 12:52 PM Eno Thereska wrote: > Hi Mahendra, > > That is a good question. Streams uses consumers and that config applies to > consumers. However, in streams we always set

Re: At Least Once semantics for Kafka Streams

Hi Mahendra, That is a good question. Streams uses consumers and that config applies to consumers. However, in streams we always set enable.auto.commit to false, and manage commits using the other commit parameter. That way streams has more control on when offsets are committed. Eno > On 6

Re: At Least Once semantics for Kafka Streams

I have another follow up question regarding configuration. There is a config for enable.auto.commit for consumers. Does this apply to Kafka streams? If yes, how is the behavior different when the value of this config is true vs false? More generally, which of the consumer configs

Re: At Least Once semantics for Kafka Streams

Ah OK! Thanks a lot for this clarification. it will only commit the offsets if the value of COMMIT_INTERVAL_MS_CONFIG > has > passed. >

Re: At Least Once semantics for Kafka Streams

It controls the minimum frequency at which the offsets are committed. The StreamThread runs in a loop that looks something like this: while(true) records = consumer.poll(..) for each record task = findTask(record) task.process(..) end maybeCommit() end This is grossly

Re: At Least Once semantics for Kafka Streams

Hi Mahendra, The commit is done on the same thread as the processing, so only offsets that have been fully processed by the topology will be committed. Thanks, Damian On Fri, 3 Feb 2017 at 08:40 Mahendra Kariya wrote: > Thanks a lot for this Matthias. > > I have a

Re: At Least Once semantics for Kafka Streams

Thanks a lot for this Matthias. I have a follow up question. There is a COMMIT_INTERVAL_MS_CONFIG config for streams. This confuses things a little bit. If the value of this config is set to, say 100 ms, does it mean that the offset will be committed after 100 ms? If yes, then how does at least

Re: At Least Once semantics for Kafka Streams

Hi, yes, all examples have at-least-once semantics because this is the only "mode" Kafka Streams supports -- you cannot "disable" it. (btw: we are currently working on exactly-once for Streams that you will be able to turn off/on). There is not much documentation about how it work internally,

At Least Once semantics for Kafka Streams

Hey All, I am new to Kafka streams. From the documentation , it is pretty much clear that streams support at least once semantics. But I couldn't find details about how this is supported. I am interested in knowing