dajac opened a new pull request, #23207: URL: https://github.com/apache/kafka/pull/23207
When appending records, we create a new batch if no batch exists. KAFKA-19760 introduced a flush of any existing batch when appending large records, to maximize our chances of compressing the records under the max.message.bytes. When these two things happen in the same append operation, we flush an empty batch. Flushing an empty batch is written to fail and revert the coordinator state. This would be harmless, except some group coordinator operations update the coordinator state directly without replay. Upon appending their records and triggering the empty batch flush, their state changes are then reverted while their records are written. The group coordinator's in-memory state diverges from the on-disk state and subsequent writes for the group can be invalid for the on-disk state. eg. we may have a consumer group downgraded to a classic group, followed by consumer group records which is invalid. Do not flush empty batches when appending large records. Backport of #22969 to 4.2. The test is adapted to the `CoordinatorRuntime` API of this branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
