Re: KTable in Compact Topic takes too long to be updated

2020-02-20 Thread John Roesler
Aha! Thanks, Renato, that's very clear. I think there's a couple of ways you can model this, but one thing that applies to all of them is that you should consider the `max.task.idle.ms` configuration option. If you set it higher than `max.poll.interval.ms`, then Streams will be able to ensure

Re: KTable in Compact Topic takes too long to be updated

2020-02-19 Thread Renato Melo
Hi John, Thank you for your reply. Let me clarify. I used the word aggregate, but we are using aggregate functions. Our case is a relationship whole-part between messageA and message1, 2, n. Like order and order items. So translating our case, messageA is the order and message1 and 2 are

Re: KTable in Compact Topic takes too long to be updated

2020-02-19 Thread John Roesler
Hi Renato, Can you describe a little more about the nature of the join+aggregation logic? It sounds a little like the KTable represents the result of aggregating messages from the KStream? If that's the case, the operation you probably wanted was like: > KStream.groupBy().aggregate() which

KTable in Compact Topic takes too long to be updated

2020-02-19 Thread Renato Melo
Hi Kafka Community, Please take a look into my use case: Fist message1 1. We have a KStream joined to a KTable(Compact Topic). 2. We received a message1 from the KStream, aggregates the message1 to the joined messageA from KTable. 3. We pushes back the messageA with aggregated message1 into