Repository: kudu Updated Branches: refs/heads/master cbd34fa85 -> 0396cca5b
KUDU-2296: Fix deserialization of messages larger than 64MB Protobuf's CodedInputStream has a 64MB total byte limit by default. When trying to deserialize messages larger than this, ParseMessage() hits this limit and mistakenly think that the packet is too short. This issue is dormant due to Kudu's default rpc_max_message_size of 50MB. However, Impala will be using a larger value for rpc_max_message_size and requires this fix. The fix is to override the default 64MB limit by calling CodedInputStream::SetTotalByteLimit() with the buffer's size. Change-Id: I57d3f3ca6ec0aa8be0e67e6a13c4b560c9d2c63a Reviewed-on: http://gerrit.cloudera.org:8080/9312 Tested-by: Kudu Jenkins Reviewed-by: Todd Lipcon <t...@apache.org> Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/a7f5e838 Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/a7f5e838 Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/a7f5e838 Branch: refs/heads/master Commit: a7f5e8388e8e63b681200ccd2e8423edf76e3dc8 Parents: cbd34fa Author: Joe McDonnell <joemcdonn...@cloudera.com> Authored: Tue Feb 13 13:53:33 2018 -0800 Committer: Todd Lipcon <t...@apache.org> Committed: Tue Feb 13 22:34:25 2018 +0000 ---------------------------------------------------------------------- src/kudu/rpc/serialization.cc | 4 ++++ 1 file changed, 4 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kudu/blob/a7f5e838/src/kudu/rpc/serialization.cc ---------------------------------------------------------------------- diff --git a/src/kudu/rpc/serialization.cc b/src/kudu/rpc/serialization.cc index 4103989..b6d6393 100644 --- a/src/kudu/rpc/serialization.cc +++ b/src/kudu/rpc/serialization.cc @@ -119,6 +119,10 @@ Status ParseMessage(const Slice& buf, << "Got mis-sized buffer: " << KUDU_REDACT(buf.ToDebugString()); CodedInputStream in(buf.data(), buf.size()); + // Protobuf enforces a 64MB total bytes limit on CodedInputStream by default. + // Override this default with the actual size of the buffer to allow messages + // larger than 64MB. + in.SetTotalBytesLimit(buf.size(), -1); in.Skip(kMsgLengthPrefixLength); uint32_t header_len;
