sanghyeok An created KAFKA-21083:
------------------------------------
Summary: Return the effective batch size limit in
MESSAGE_TOO_LARGE responses to improve producer batch splitting
Key: KAFKA-21083
URL: https://issues.apache.org/jira/browse/KAFKA-21083
Project: Kafka
Issue Type: Improvement
Reporter: sanghyeok An
Assignee: sanghyeok An
*Description*
When a producer sends a record batch that exceeds the effective broker-side
batch size limit, the broker rejects it with MESSAGE_TOO_LARGE. This limit
applies to the record batch, after compression if enabled.
The producer currently splits the rejected batch using the configured
batch.size. If an already-split batch is rejected again, it reduces the split
target to half the estimated batch size, bounded below by the largest record
size.
This approach allows progressively smaller batches, but the producer does not
know the limit enforced by the broker. Consequently, split batches may still
exceed that limit, resulting in additional rejected requests, decompression,
batch rebuilding, and retransmission.
*Proposed improvement*
Include the effective maximum allowed record batch size in the
MESSAGE_TOO_LARGE response for the affected partition. The value should reflect
the applicable topic-level max.message.bytes, including the broker default when
no topic override is configured.
The producer could use this value to choose a split target that respects both
its configured batching target and the reported limit, reducing the need to
discover an acceptable size through repeated failures.
For example, with a producer batch.size of 8 MiB and an effective broker limit
of 1 MiB, the producer could use the reported limit immediately when splitting,
instead of requiring further rejections to progressively reduce the target.
{*}Considerations{*}{*}{*}
* Preserve the existing splitting behavior when the response does not provide
a limit. (for compatiability for old version broker and producer)
* Treat the reported limit as a hint: configuration changes and compression
estimation errors can still cause subsequent rejections.
* Preserve failure handling for a single-record batch that exceeds the limit;
batch splitting does not divide an individual record.
*Related work*
KIP-126 identified obtaining the broker’s maximum message size as a future
direction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
