jia-gao opened a new pull request #1581: URL: https://github.com/apache/samza/pull/1581
1. Upgrade Kafka version in dependency-versions.gradle 2. Issue: StreamApplicationIntegrationTestHarness is flaky because in the test there is a Kafka consumer subscribes to input topic and consumes from it then, the same consumer subscribes to the changelog topic. During this change, Kafka needs to revoke the previous assignments and do rebalancing which causes the changelog topic is not assigned to any member. Change: create a separate consumer of a different consumer group to consume changelog topic, so it can avoid Kafka consumer group rebalance 3. Issue: Apache Kafka 2.4 introduces sticky partition strategy:https://www.confluent.io/blog/apache-kafka-producer-improvements-sticky-partitioner/ which causes TestZkLocalApplicationRunner tests flaky. The reason is that in some of the tests, Kafka events published to an input topic with 5 partitions with DefaultPartitioner. With the sticky partition strategy, some of the partitions could be empty. The tests bring up some stream processors to process them and wait for processors to consume messages. However, a stream processor doesn't have any messages to process if it consumes from partitions that have no messages In such a case, the test will be stuck and timeout. Change: publish Kafka events to all partitions in a round-robin way to make sure all partitions have messages Tests: /gradlew clean build API Changes: None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
