shanthoosh opened a new pull request #1251: SAMZA-2431: Fix the checkpoint and changelog topic auto-creation. URL: https://github.com/apache/samza/pull/1251 Symptom: Checkpoint and changelog kafka topics are expected to be created for a samza job with cleanup.policy set to compact. This ensures that all the data in the metadata topics are log-compacted. However, we observed that checkpoint & changelog topics for hundreds of production samza jobs were created with cleanup.policy set to delete leading to strong-possibility of message-l0ss. Cause: - *Checkpoint:* The control-flow in KafkaStreamSpec to build checkpoint spec swallows the essential kafka-topic configuration rather passes empty configuration bag to kafka-broker(via kafka-rest). This was introduced in SAMZA-2339. - *Changelog:* There change-log topic configurations are incorrectly generated when the RocksDB store TTL is set to -1 by the user. This was introduced in SAMZA-1929. Changes: Fix the topic-creation control-flow for the metadata topics and generate the correct topic-configurations. Tests: Added unit tests to validate the expected topic configuration bag. API Changes: None Upgrade Instructions: None Usage Instructions: None
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
