Joe McDonnell created KUDU-2296:

             Summary: Kudu RPC cannot deserialize messages larger than 64MB
                 Key: KUDU-2296
             Project: Kudu
          Issue Type: Bug
          Components: rpc
    Affects Versions: 1.6.0
            Reporter: Joe McDonnell
            Assignee: Joe McDonnell

Impala has been testing Kudu RPC with a larger value for rpc_max_message_size. 
I noticed that when the message size exceeds 64MB, 
rpc::serialization::ParseMessage() hits this condition:
if (PREDICT_FALSE(!in.Skip(main_msg_len))) {
  return Status::Corruption(
      StringPrintf("Invalid packet: data too short, expected %d byte main_msg", 
The actual buffer is the appropriate size. What is happening is that protobuf 
imposes a 64MB total byte limit by default. Once a message exceeds that, the 
Skip() call will return false when trying to go past the 64MB limit. The 
deserialization code can get around this by setting the total byte limit with 

This should not impact existing systems at the moment, because the default 
value for rpc_max_message_size is 50MB.

This message was sent by Atlassian JIRA

Reply via email to