We have a situation where we are streaming data in a Kafka topic to be 
processed. The topic has multiple partitions. From time to time we need to add 
control messages to the stream and this needs to be coherent with the data 
steam (e.g. "replay from point X to this new location"). Because of the way 
Kafka partitions it seems hard to get these control messages to every consumer, 
no matter how they have been assigned, in the right place in the queues. The 
way I see it there are two options:

1) Have the producer put these control messages on every partition. We could 
get the number of partitions and manually post the same message on every 
partition, which will guarantee it gets read in the right place.
2) Reserve a special partition (e.g. 0) to be a control one, and have all 
consumers subscribe to that one. I don't think this gives the guarantees we 
want though.

The other option is to use a separate queue for these, but then we'd need some 
special logic to ensure they have their effect at the right point in the queue.

Has anyone else implemented anything like this? Am I barking up the wrong tree?


Ben Young
Principal Software Engineer - Risk
Risk & Compliance
T:  +44 (0) 1865 578140
E: ben.yo...@fisglobal.com
FIS | Empowering the Financial World
FIS Systems Limited * Registered in England and Wales No. 1937159 * Registered 
Office: Level 39, 25 Canada Square, London E14 5LQ

The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.

Reply via email to