Re: Tracking when a batch of messages has arrived?

I don't - it would require fetching all messages and iterating over them just to count them, which is expensive. I know the counts after they have been sent. On Sun, Dec 4, 2016 at 9:34 PM, Marko Bonaći wrote: > Do you know in advance (when sending the first message)

Re: Tracking when a batch of messages has arrived?

Do you know in advance (when sending the first message) how many messages that batch is going to have? Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support Sematext | Contact

Re: Tracking when a batch of messages has arrived?

Hey Apurva, I am including the batch_id inside the messages. Could you give me an example of what you mean by custom control messages with a control topic please? On Sat, Dec 3, 2016 at 12:35 AM, Apurva Mehta wrote: > That should work, though it sounds like you may be

Re: Tracking when a batch of messages has arrived?

That should work, though it sounds like you may be interested in : https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging If you can include the 'batch_id' inside your messages, and define custom control messages with a control topic, then you

Tracking when a batch of messages has arrived?

Heya, I need to send a group of messages, which are all related, and then process those messages, only when all of them have arrived. Here is how I'm planning to do this. Is this the right way, and can any improvements be made to this? 1) Send a message to a topic called batch_start, with a