Stef Noten created KAFKA-20358:
----------------------------------
Summary: Allow KTable/KStream and GlobalKTable on same topic
Key: KAFKA-20358
URL: https://issues.apache.org/jira/browse/KAFKA-20358
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 4.2.0
Reporter: Stef Noten
In a single streams app, we sometimes want to consume a topic in two ways:
# as a GlobalKTable: for global read-only reference data
# as a KTable or KStream, to react on changes on that same topic
Kafka Streams currently prevents this by a validation check (in
InternalTopologyBuilder.validateTopicNotAlreadyRegistered):
{code:java}
var builder = new StreamsBuilder();
builder.table("input", Consumed.as("input-as-table"));
builder.globalTable("input", Consumed.as("input-as-globaltable"));
builder.build(); {code}
Result:
{code:java}
Topic input has already been registered by another source. {code}
However, there is nothing that conceptually prevents this from working, since
GlobalKTables do not participate in the app's consumer group and are handled by
a dedicated subtopology and GlobalStreamsThread which does not need to commit
any offsets. So it's only the validation rule that blocks this, requiring an
annoying workaround to mirror the topic to a another one with a different name:
{code:java}
var myTable = builder.table("my-topic");
myTable.toStream().to("my-topic-global-copy");
var myGlobalTable = builder.globalTable("my-topic-global-copy");{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
