[
https://issues.apache.org/jira/browse/KUDU-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved KUDU-2296.
---------------------------------
Resolution: Fixed
Fix Version/s: 1.7.0
commit dc3543300ebbe8121ec0a96b7076f69b46dc9868
Author: Joe McDonnell <joemcdonn...@cloudera.com>
Date: Tue Feb 13 13:53:33 2018 -0800
KUDU-2296: Fix deserialization of messages larger than 64MB
Protobuf's CodedInputStream has a 64MB total byte limit by
default. When trying to deserialize messages larger than
this, ParseMessage() hits this limit and mistakenly
think that the packet is too short. This issue is dormant
due to Kudu's default rpc_max_message_size of 50MB.
However, Impala will be using a larger value for
rpc_max_message_size and requires this fix.
The fix is to override the default 64MB limit by calling
CodedInputStream::SetTotalByteLimit() with the buffer's
size.
Change-Id: I57d3f3ca6ec0aa8be0e67e6a13c4b560c9d2c63a
> Kudu RPC cannot deserialize messages larger than 64MB
> -----------------------------------------------------
>
> Key: KUDU-2296
> URL: https://issues.apache.org/jira/browse/KUDU-2296
> Project: Kudu
> Issue Type: Bug
> Components: rpc
> Affects Versions: 1.6.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Fix For: 1.7.0
>
>
> Impala has been testing Kudu RPC with a larger value for
> rpc_max_message_size. I noticed that when the message size exceeds 64MB,
> rpc::serialization::ParseMessage() hits this condition:
> {code:java}
> if (PREDICT_FALSE(!in.Skip(main_msg_len))) {
> return Status::Corruption(
> StringPrintf("Invalid packet: data too short, expected %d byte
> main_msg", main_msg_len),
> KUDU_REDACT(buf.ToDebugString()));
> }
> {code}
> The actual buffer is the appropriate size. What is happening is that protobuf
> imposes a 64MB total byte limit by default. Once a message exceeds that, the
> Skip() call will return false when trying to go past the 64MB limit. The
> deserialization code can get around this by setting the total byte limit with
> CodedInputSTream::SetTotalByteLimit().
> This should not impact existing systems at the moment, because the default
> value for rpc_max_message_size is 50MB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
