Mark Tranter created KAFKA-5009:
-----------------------------------
Summary: Globally Unique internal topic names when using Joins
Key: KAFKA-5009
URL: https://issues.apache.org/jira/browse/KAFKA-5009
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Mark Tranter
We are using multiple different Kafka Streams applications on the back of a
single kafka cluster. This allows each micro-service in our enterprise to
consume & process kafka data to suit its own needs.
Currently when joining streams, an internal topic is created & named using
KStreamsBuilder.newName(prefix);
```
String thisWindowStreamName = topology.newName(WINDOWED_NAME);
String otherWindowStreamName = topology.newName(WINDOWED_NAME);
String joinThisName = rightOuter ? topology.newName(OUTERTHIS_NAME)
: topology.newName(JOINTHIS_NAME);
String joinOtherName = leftOuter ?
topology.newName(OUTEROTHER_NAME) : topology.newName(JOINOTHER_NAME);
String joinMergeName = topology.newName(MERGE_NAME);
```
This prefix is a constant, and internally an incrementor is used to generate a
unique (per KStreamBuilder instance) topic name for the topic.
In situations where multiple KStreamBuilders are in use (for example, multiple
different Kafka Streams applications) we are seeing collisions in topic names.
Perhaps the join() methods should take a topic prefix overload to allow
developers to provide unique names for these topics. Similar to the way other
stateful processors work when having to provide a store name.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
