So if the stream is: A:1:FOO A:3:BAR A:3:BAZ
Then A:3* must be processed after A:1 but A:3:BAR and A:3:BAZ can be processed in any order? I don’t think there is a way to do that with topics. -Dave From: Andre Mermegas <and...@gmail.com> Reply-To: "users@kafka.apache.org" <users@kafka.apache.org> Date: Wednesday, September 2, 2020 at 4:06 PM To: "users@kafka.apache.org" <users@kafka.apache.org> Subject: [External] key and subkey sequential processing across competing consumer groups? Hi All, New to kafka and would love some feedback on how to think about a solution for this kind of flow: So, sequencing must be maintained for events of type X:Y:PAYLOAD Where X is a grouping mechanism and all X must be processed sequentially e.g. account number, (ok i use key here to keep these in sequence) But Y is a secondary grouping and distribution variable, that will fan out to competing consumers for each type of Y e.g. where y=A it goes to a topic of competing consumers and where y=B it goes out to a different topic of competing consumers, etc..Each competing consumer group is configured to handle only Ys of its type. All Xs must consumed sequentially, across distributed consumer groups even those fanned out with a subkey of Y How do I keep it sequential processing FIFO across? I know I can use a key to sequence by X in a topic partition for sequential processing, but it breaks down from there for me as I also need to sequence by Y. Is there a good built in way to handle this type of problem? Am I thinking about it wrong? I hope this is somewhat clear? thanks for any input! looks something like this maybe: [cid:ii_kelvcjxy1] -- Thanks, -Andre This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
