dajac opened a new pull request, #23209: URL: https://github.com/apache/kafka/pull/23209
When appending records, we create a new batch if no batch exists. KAFKA-19760 introduced a flush of any existing batch when appending large records, to maximize our chances of compressing the records under the max.message.bytes. When these two things happen in the same append operation, we flush an empty batch. Flushing an empty batch is written to fail and revert the coordinator state. This would be harmless, except some group coordinator operations update the coordinator state directly without replay. Upon appending their records and triggering the empty batch flush, their state changes are then reverted while their records are written. The group coordinator's in-memory state diverges from the on-disk state and subsequent writes for the group can be invalid for the on-disk state. eg. we may have a consumer group downgraded to a classic group, followed by consumer group records which is invalid. Do not flush empty batches when appending large records. Backport of #22969 to 4.0. `CoordinatorContext.batchEpoch` does not exist on this branch, so the test cannot assert that a single batch was created. Instead, it uses a non-replaying write operation which applies its in-memory changes directly and verifies that they are not reverted by the flush of an empty batch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
