Re: Whey does the window final result is not emitted after the window has elapsed?

Guozhang: Yes, my case is a real production scenario. I am monitoring on-line live-broadcast classes. I need to have a summary of each 5-minute period for one class. Each class has a classroom Id. I report class activity data to a Kafka topic. Classroom id is used to partition these data. Here

Re: Programmatic method of setting consumer groups offsets

I’ll look into assign - I think the issue I am having is often I will want to do these offset resets after the my consumer group has crashed. I guess what may be happening is that kafka still thinks there are active consumers even though they have crashed.

Re: Programmatic method of setting consumer groups offsets

I’ll look into assign - I think the issue I am having is often I will want to do these offset resets after the my consumer group has crashed. I guess what may be happening is that kafka still thinks there are active consumers even though they have crashed.

Re: Programmatic method of setting consumer groups offsets

Adding on to Ryanne's point, if subscribe() isn't giving your consumer all of the partitions for a topic, that implies there are still active consumers running for that group. The consumer groups CLI command does not allow you to modify offsets for consumer groups that have active consumers. I

Re: Programmatic method of setting consumer groups offsets

Nimi, try using the low-level consumer API via assign() instead of subscribe(), and then commit() each partition. I suspect you are trying to commit partitions that the consumer is not subscribed to. Ryanne On Thu, Jan 3, 2019, 6:13 PM Nimi Wariboko Hi, > > Is there a guide or API that is

Programmatic method of setting consumer groups offsets

Hi, Is there a guide or API that is implemented that can modify stored offsets in Kafka? For example, I'm looking to set a consumer groups stored offsets in Kafka to a specific value. I know there is the `kafka-consumer-groups` CLI command, but I'm looking for a way to do so from an

Re: Kafka Streams 2.1.0, 3rd time data lose investigation

Hi Nitay, I'm sorry to hear of these troubles; it sounds frustrating. No worries about spamming the list, but it does sound like this might be worth tracking as a bug report in Jira. Obviously, we do not expect to lose data when instances come and go, regardless of the frequency, and we do have

Re: Why do I get an IllegalStateException when accessing record metadata?

I see. When updating the FAQ, it should be clear what you mean. Your current proposal was unclear to me, and thus, it might be unclear to other users, too. -Matthias On 1/3/19 9:13 PM, Eric Lalonde wrote: > > >> On Jan 2, 2019, at 6:31 AM, Matthias J. Sax > >

Re: Why do I get an IllegalStateException when accessing record metadata?

> On Jan 2, 2019, at 6:31 AM, Matthias J. Sax wrote: > > Thanks for reporting this. Feel free to edit the Wiki with the FAQ directly. > > What is unclear to me: what do you mean by "the state store [...] was > errantly scoped to the TransformerProvider, not the Transformer" ? > > I would

Re: Whey does the window final result is not emitted after the window has elapsed?

Hello Jingguo, Is this case (i.e. you only have data over 57 minutes, and no new data afterwards) a real production scenario? In stream processing we usually expect the input data stream in continuously, and I'm curious to learn your use case better and why it would not have further data after a

Re: Whey does the window final result is not emitted after the window has elapsed?

Hi, Matthias I am doing a 5-minute tumbling window analysis over a 57-minute data flow. And I want only one final result per window. So I need suppress. The 57-minute period can be divided into about 12 windows. The results of the first 11 windows can be delivered downstream. But the final result

Re: High end-to-end latency with processing.guarantee=exactly_once

Also, occasionally I see errors like this. Is it related to this issue? [2018-12-30 18:21:00,552] ERROR (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:131) task [0_4] Error sending record (key value id: "\227\236\022L\205\356\375\373\373\304\241\301n\250H\367" timestamp

Re: High end-to-end latency with processing.guarantee=exactly_once

Hi Matthias, I get these errors even on reprocessing, when data is flowing full throttle through the system. Can you help me understand how to tune this behavior, if possible? I appreciate that it's aggressive, but it seems to be so extremely aggressive that I get these errors constantly. Just