Hi. I have the following problem : we are building a system that will generate any number of different events published to different topics
Events have an associated client and a client can express interest in these events. When they do, then for each event we will execute a callback to a remote (http) endpoint It is important that we so not overload the remote endpoint say after a crash and there are a metric ton of unprocessed events waiting to be processed So i'd like all events for a client to end up on the same cosumer and do the throttling there Is it possible to do this in kafka ? I know that i can specify the client as a key when producing the events so that events for a client for that topic all end up on the same partition but because we have a topic per event type and lots of events then the messages will still be processed by many backends Creating a new topic that contains all the events from all the other topics would work i guess. But this feels inelegant and then you are storing data twice Can anyone offer any guidance?
