[
https://issues.apache.org/jira/browse/TEZ-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450873#comment-17450873
]
Rajesh Balamohan commented on TEZ-4142:
---------------------------------------
setSizeLimit will just adjust the threshold and not allocate any buffer. Hence
it isn't needed to make it a configurable val. 2 GB is fairly large enough
value for most large payloads and is intentionally capped by CodedInputStream
itself.
> TezUtils.createConfFromByteString on Configuration larger than 32MB throws
> com.google.protobuf.CodedInputStream exception
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: TEZ-4142
> URL: https://issues.apache.org/jira/browse/TEZ-4142
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Turner Eagles
> Assignee: Jonathan Turner Eagles
> Priority: Major
> Fix For: 0.10.0, 0.9.3
>
> Attachments: TEZ-4142-branch-0.9.001.patch, TEZ-4142.001.patch
>
>
> {code}
> Status: Failed
> Vertex failed, vertexName=Map 1, vertexId=vertex_1586321981777_24335_1_00,
> diagnostics=[Vertex vertex_1586321981777_24335_1_00 [Map 1] killed/failed due
> to:INIT_FAILURE, Fail to create InputInitializerManager,
> org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class
> with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
> at
> org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)
> ...
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
> at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
> at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.<init>(DAGProtos.java:19294)
> at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.<init>(DAGProtos.java:19258)
> at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto$1.parsePartialFrom(DAGProtos.java:19360)
> at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto$1.parsePartialFrom(DAGProtos.java:19355)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.parseFrom(DAGProtos.java:19552)
> at org.apache.tez.common.TezUtils.createConfFromByteString(TezUtils.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:92)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
