Jonathan Eagles commented on TEZ-3914:

Essentially, the underlying serializer/deserializer for protobuf messages is 
the CodedOutputStream/CodedInputStream. When messages are written either with 
the writeDelimitedTo (read parallel is parseDelimitedFrom) or writeMessageNoTag 
(parallel is readMessage), first the serialized size is written and then the 
serialized message is written. In addition to this, Tez prepends the message 
type to handle message dispatching. To protect itself from remote messages, 
CodedInputStream has a default serialized size limit of 64 MB. When using the 
parseDelimitedFrom java.io.InputStream method, new underlying deserializer 
CodedInputStream is created using the default message size limit. This prevents 
larger than 64MB Tez recovery messages from being properly parsed and prevents 
further messages after the large message from being parsed as well since any 
Exception is treated the same as EOF.

There are a few possible solutions to this.
1) Ensure all recovery messages are bounded in size less than 64 MB
2) Ensure messages larger than 64 MB can be successfully read
3) Ensure large messages are non-fatal and skip them.

This jira takes aproach 2 since recovery data is not lost. In addition, this 
jira replaces the higher level writeDelimitedTo/parseDelimitedFrom with the 
writeMessageNoTag/readMessage. This allows control over the maximum message 
size as required. In addition, recover writes and reads are now more efficient 
as serializing/deserializing streams (CodedInputStream/CodedOutputStream) are 
now reused.

> Recovering a large DAG fails to size limit exceeded
> ---------------------------------------------------
>                 Key: TEZ-3914
>                 URL: https://issues.apache.org/jira/browse/TEZ-3914
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Major
>         Attachments: TEZ-3914.001.patch, TEZ-3914.002.patch, 
> TEZ-3914.003.patch
> A large message will be failed to parse and will be treated as recovery file 
> EOF.
> {noformat}
> 2018-04-16 15:33:59,807 WARN  [Thread-2] app.RecoveryParser 
> (RecoveryParser.java:parseRecoveryData(771)) - Corrupt data found when trying 
> to read next event
> com.google.protobuf.InvalidProtocolBufferException: Protocol message was too 
> large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase 
> the size limit.
> {noformat}

This message was sent by Atlassian JIRA

Reply via email to