I have a question basically on how it would be the best way to implement
something within Kafka Streams. The thing I would like to do: "dynamically
update the subscription pattern of the source topics.
The reasoning behind this (in my project):
meta data about the source topics is evented on an other kafka topic, that
should be tracked by the kafka streams topology, and depending on that meta
data specific source topics should be added, or removed from the kafka
Currently I track the "meta data topic" as "global state", so that every
processor can actually access it to fetch the meta data (this meta data for
instance also describes whether or not a specific topic pattern should be
tracked by the stream processor) - so consider this as some kind of
"configuration" stream about the source topics.
So now it comes,
Is there any way I could (from a running topology) update the kafka
So that I'm able to replace the source topic pattern while the topology is
I don't think there currently is a way to do this, but as under the hood it
is just a kafka consumer, my believe is that it should be possible somehow
I was thinking about the PartitionAssigner ... if I could get my hands on
that one, maybe I could dynamically configure it to only allow specific
Or directly alter the subscription on the underlying consumer?
I don't know all the nifty details about the Kafka Streams internals, so it
would be nice if someone could direct me in the right direction to achieve