Chia-Ping Tsai created KAFKA-20036:
--------------------------------------
Summary: Handle LogCleaner segment overflow caused by compression
level changes
Key: KAFKA-20036
URL: https://issues.apache.org/jira/browse/KAFKA-20036
Project: Kafka
Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
from: [https://github.com/apache/kafka/pull/15516#discussion_r2649600413]
The {{LogCleaner}} can fail with {{java.lang.IllegalArgumentException}} when
the compacted data exceeds the segment size limit.
This issue occurs because the cleaner rebuilds batches using the default
compression level. If the data was originally ingested with a higher
compression level, the re-compressed batch size increases, causing the
cleaner's size estimation to be inaccurate and leading to an overflow.
To address this without major architectural changes (like splitting segments
during cleaning), we should introduce a safety margin or adaptive logic for
partitions that encounter this overflow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
