[
https://issues.apache.org/jira/browse/KAFKA-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054763#comment-16054763
]
ASF GitHub Bot commented on KAFKA-5032:
---------------------------------------
GitHub user apurvam opened a pull request:
https://github.com/apache/kafka/pull/3374
KAFKA-5032: Update the docs for message size configs across the board
Before 0.11, we used to have limits for maximum message size on the
producer, broker, and consumer side.
From 0.11 onward, these limits apply to record batches as a whole. This
patch updates the documentation of the configs to make this explicit.
A separate patch will have more extensive upgrade notes to tie all the
changes together in one narrative.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apurvam/kafka KAFKA-5032-message-size-docs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3374.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3374
----
commit b8e1379a54d21141a22694b2aa6d422709bfb89f
Author: Apurva Mehta <[email protected]>
Date: 2017-06-19T21:20:03Z
Change references to 'message' in the size options to 'record batch', since
everything is written and read in batches in the current version.
----
> Think through implications of max.message.size affecting record batches in
> message format V2
> --------------------------------------------------------------------------------------------
>
> Key: KAFKA-5032
> URL: https://issues.apache.org/jira/browse/KAFKA-5032
> Project: Kafka
> Issue Type: Sub-task
> Components: clients, core, producer
> Reporter: Ismael Juma
> Assignee: Apurva Mehta
> Priority: Critical
> Labels: documentation, exactly-once
> Fix For: 0.11.0.0
>
>
> It's worth noting that the new behaviour for uncompressed messages is the
> same as the existing behaviour for compressed messages.
> A few things to think about:
> 1. Do the producer settings max.request.size and batch.size still make sense
> and do we need to update the documentation? My conclusion is that things are
> still fine, but we may need to revise the docs.
> 2. (Seems like we don't need to do this) Consider changing default max
> message set size to include record batch overhead. This is currently defined
> as:
> {code}
> val MessageMaxBytes = 1000000 + MessageSet.LogOverhead
> {code}
> We should consider changing it to (I haven't thought it through though):
> {code}
> val MessageMaxBytes = 1000000 + DefaultRecordBatch.RECORD_BATCH_OVERHEAD
> {code}
> 3. When a record batch is too large, we throw RecordTooLargeException, which
> is confusing because there's also a RecordBatchTooLargeException. We should
> consider renaming these exceptions to make the behaviour clearer.
> 4. We should consider deprecating max.message.bytes (server config) and
> message.max.bytes (topic config) in favour of configs that make it clear that
> we are talking about record batches instead of individual messages.
> Part of the work in this JIRA is working out what should be done for 0.11.0.0
> and what can be done later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
