[
https://issues.apache.org/jira/browse/TEZ-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17451058#comment-17451058
]
Brahma Reddy Battula commented on TEZ-4142:
-------------------------------------------
Yes, this will be used only while reading and Default value in
codedinputstream is 64MB, I dn't think we can have payload with 2GB. Suggested
value is smaller from java doc also and it's application specific to handle
this.
https://www.javadoc.io/static/com.google.protobuf/protobuf-java/2.2.0/com/google/protobuf/CodedInputStream.html#setSizeLimit(int)
> TezUtils.createConfFromByteString on Configuration larger than 32MB throws
> com.google.protobuf.CodedInputStream exception
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: TEZ-4142
> URL: https://issues.apache.org/jira/browse/TEZ-4142
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Turner Eagles
> Assignee: Jonathan Turner Eagles
> Priority: Major
> Fix For: 0.10.0, 0.9.3
>
> Attachments: TEZ-4142-branch-0.9.001.patch, TEZ-4142.001.patch
>
>
> {code}
> Status: Failed
> Vertex failed, vertexName=Map 1, vertexId=vertex_1586321981777_24335_1_00,
> diagnostics=[Vertex vertex_1586321981777_24335_1_00 [Map 1] killed/failed due
> to:INIT_FAILURE, Fail to create InputInitializerManager,
> org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class
> with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
> at
> org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)
> ...
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
> at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
> at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.<init>(DAGProtos.java:19294)
> at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.<init>(DAGProtos.java:19258)
> at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto$1.parsePartialFrom(DAGProtos.java:19360)
> at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto$1.parsePartialFrom(DAGProtos.java:19355)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.parseFrom(DAGProtos.java:19552)
> at org.apache.tez.common.TezUtils.createConfFromByteString(TezUtils.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:92)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
