Shanthoosh Venkataraman created SAMZA-2431:
----------------------------------------------
Summary: Fix the checkpoint and change-log topic-creation workflow.
Key: SAMZA-2431
URL: https://issues.apache.org/jira/browse/SAMZA-2431
Project: Samza
Issue Type: Bug
Reporter: Shanthoosh Venkataraman
Assignee: Shanthoosh Venkataraman
Symptom: Checkpoint and changelog kafka topics are expected to be created for a
samza job with cleanup.policy set to compact. This ensures that all the data in
the metadata topics are log-compacted. However, we observed that checkpoint &
changelog topics for hundreds of production samza jobs were created with
cleanup.policy set to delete leading to strong-possibility of message-l0ss.
Cause:
- *Checkpoint:* The control-flow in KafkaStreamSpec to build checkpoint spec
swallows the essential kafka-topic configuration rather passes empty
configuration bag to kafka-broker(via kafka-rest). This issue was introduced in
SAMZA-2339.
- *Changelog:* There change-log topic configurations are incorrectly generated
when the RocksDB store TTL is set to -1 by the user.
Changes: Change the topic-creation control-flow for above metadata topics to
generate correct topic-configurations.
Tests: Added unit tests to validate the expected topic configuration bag.
API Changes: None
Upgrade Instructions: None
Usage Instructions: None
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
