Peter James Pringle created KAFKA-13915:
-------------------------------------------
Summary: Kafka streams should validate that the repartition topics
are not created with cleanup.policy compact
Key: KAFKA-13915
URL: https://issues.apache.org/jira/browse/KAFKA-13915
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 2.8.1
Reporter: Peter James Pringle
Add sanity validation on streams start up that *repartition* topics are not
setup with *cleanup.policy* of {*}compact{*}.
In enterprise envs automated creation of kafka streams intermediate topics is
not always possible due to policy restrictions and as a result it is done
manually which is prone to user misconfiguration.
In several cases we have found the repartition topics have been incorrectly
setup following the changelog topic setup with compact enabled. The result
being that a compacted repartition topic will result in data loss if more that
one value is mapped to the new key.
Example:
{{Original data: (coffee, drink), (tea, drink), (beer, drink)}}
Repartition by type i.e. drink:
Expected:
{{(drink, coffee), (drink, tea), (drink, beer)}}
With compaction the following is possible:
Actual
{{(drink, beer); }}
coffee and tea are lost.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
