[
https://issues.apache.org/jira/browse/KAFKA-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053101#comment-16053101
]
Ismael Juma edited comment on KAFKA-5465 at 6/18/17 9:06 AM:
-------------------------------------------------------------
Jason has a PR in progress for this, but it's a little tricky and we'll
probably need some time to review and discuss. It's also late in the process,
so we have to be particularly careful not to introduce regressions.
It's worth pointing out that all of the following need to be true for this to
happen:
1. Pre KIP-74 consumer (KIP-74 was introduced in 0.10.1.0)
2. Broker configured to use message format v2. Even though this is the default,
the upgrade instructions recommend that users only switch to the new message
format after most consumers have been upgraded to avoid the loss of zero-copy.
3. Broker's max message size is larger than the consumer's max partition bytes.
The reason is that we check the size of the record batch before and after
conversion in the broker. Hmm, thinking about it, it seems like this is also a
change in behaviour for produce requests. If one sends 2 uncompressed messages,
each under the max message size, we could reject after conversion because we
create a single batch for all the uncompressed messages in a single produce
request.
4. partition max bytes is less than the v2 record batch size. For this to
happen with the Java producer, the producer's max request size (default 1 MB
for the Java producer) must be larger than the consumer's max partition size
(default 1 MB). Note that I didn't mention the producer's batch size. The
reason is that we allow a single record to be sent if it's larger than the
batch size, but smaller than max request size.
Given the above, it seems to me that users with default settings for the Java
clients won't run into the problem. It would be good to understand if it's the
same for non Java clients (cc [~edenhill [~dana.powers]). The workaround of
setting the broker message format version to v1 also seems straightforward (and
it's what the upgrade instructions recommend if consumers that don't support
the new message format are still being used).
I will proceed with RC1. If we agree to include this, we can spin RC2.
was (Author: ijuma):
Jason has a PR in progress for this, but it's a little tricky and we'll
probably need some time to review and discuss. It's also late in the process,
so we have to be particularly careful not to introduce regressions.
It's worth pointing out that all of the following need to be true for this to
happen:
1. Pre KIP-74 consumer (KIP-74 was introduced in 0.10.1.0)
2. Broker configured to use message format v2. Even though this is the default,
the upgrade instructions recommend that users only switch to the new message
format after most consumers have been upgraded to avoid the loss of zero-copy.
3. Broker's max message size is larger than the consumer's max partition bytes.
The reason is that we check the size of the record batch before and after
conversion in the broker. Hmm, thinking about it, it seems like this is a
change in behaviour for produce requests. If one sends 2 uncompressed messages,
each under the max message size, we could reject after conversion because we
create a single batch for all the uncompressed messages in a single produce
request.
4. partition max bytes is less than the v2 record batch size. For this to
happen with the Java producer, the producer's max request size (default 1 MB
for the Java producer) must be larger than the consumer's max partition size
(default 1 MB). Note that I didn't mention the producer's batch size. The
reason is that we allow a single record to be sent if it's larger than the
batch size, but smaller than max request size.
Given the above, it seems to me that users with default settings for the Java
clients won't run into the problem. It would be good to understand if it's the
same for non Java clients (cc [~edenhill [~dana.powers]). The workaround of
setting the broker message format version to v1 also seems straightforward (and
it's what the upgrade instructions recommend if consumers that don't support
the new message format are still being used).
I will proceed with RC1. If we agree to include this, we can spin RC2.
> FetchResponse v0 does not return any messages when max_bytes smaller than v2
> message set
> -----------------------------------------------------------------------------------------
>
> Key: KAFKA-5465
> URL: https://issues.apache.org/jira/browse/KAFKA-5465
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.11.0.0
> Reporter: Dana Powers
> Assignee: Jason Gustafson
> Priority: Blocker
> Fix For: 0.11.0.1
>
>
> In prior releases, when consuming uncompressed messages, FetchResponse v0
> will return a message if it is smaller than the max_bytes sent in the
> FetchRequest. In 0.11.0.0 RC0, when messages are stored as v2 internally, the
> response will be empty unless the full MessageSet is smaller than max_bytes.
> In some configurations, this may cause old consumers to get stuck on large
> messages where previously they were able to make progress one message at a
> time.
> For example, when I produce 10 5KB messages using ProduceRequest v0 and then
> attempt FetchRequest v0 with partition max bytes = 6KB (larger than a single
> message but smaller than all 10 messages together), I get an empty message
> set from 0.11.0.0. Previous brokers would have returned a single message.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
