[
https://issues.apache.org/jira/browse/IMPALA-13764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-13764.
-------------------------------------
Assignee: (was: Quanlong Huang)
Resolution: Duplicate
> JniFrontend.createExecRequest hits OOM of JVM array limit
> ---------------------------------------------------------
>
> Key: IMPALA-13764
> URL: https://issues.apache.org/jira/browse/IMPALA-13764
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Quanlong Huang
> Priority: Critical
>
> When there are lots of files to scan in a query, the TExecRequest might need
> a huge byte array to serialize which could hit OOM of exceeding the JVM array
> limit (2GB):
> {noformat}
> I0215 09:22:04.852778 322082 jni-util.cc:321]
> b04ef5f1a668be58:5831dd9200000000] java.lang.OutOfMemoryError: Requested
> array size exceeds VM limit
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:197)
> at
> org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:204)
> at
> org.apache.impala.thrift.TScanRangeLocation$TScanRangeLocationStandardScheme.write(TScanRangeLocation.java:503)
> at
> org.apache.impala.thrift.TScanRangeLocation$TScanRangeLocationStandardScheme.write(TScanRangeLocation.java:448)
> at
> org.apache.impala.thrift.TScanRangeLocation.write(TScanRangeLocation.java:391)
> at
> org.apache.impala.thrift.TScanRangeLocationList$TScanRangeLocationListStandardScheme.write(TScanRangeLocationList.java:468)
> at
> org.apache.impala.thrift.TScanRangeLocationList$TScanRangeLocationListStandardScheme.write(TScanRangeLocationList.java:402)
> at
> org.apache.impala.thrift.TScanRangeLocationList.write(TScanRangeLocationList.java:342)
> at
> org.apache.impala.thrift.TScanRangeSpec$TScanRangeSpecStandardScheme.write(TScanRangeSpec.java:485)
> at
> org.apache.impala.thrift.TScanRangeSpec$TScanRangeSpecStandardScheme.write(TScanRangeSpec.java:413)
> at
> org.apache.impala.thrift.TScanRangeSpec.write(TScanRangeSpec.java:355)
> at
> org.apache.impala.thrift.TPlanExecInfo$TPlanExecInfoStandardScheme.write(TPlanExecInfo.java:512)
> at
> org.apache.impala.thrift.TPlanExecInfo$TPlanExecInfoStandardScheme.write(TPlanExecInfo.java:425)
> at org.apache.impala.thrift.TPlanExecInfo.write(TPlanExecInfo.java:366)
> at
> org.apache.impala.thrift.TQueryExecRequest$TQueryExecRequestStandardScheme.write(TQueryExecRequest.java:1915)
> at
> org.apache.impala.thrift.TQueryExecRequest$TQueryExecRequestStandardScheme.write(TQueryExecRequest.java:1711)
> at
> org.apache.impala.thrift.TQueryExecRequest.write(TQueryExecRequest.java:1516)
> at
> org.apache.impala.thrift.TExecRequest$TExecRequestStandardScheme.write(TExecRequest.java:2920)
> at
> org.apache.impala.thrift.TExecRequest$TExecRequestStandardScheme.write(TExecRequest.java:2567)
> at org.apache.impala.thrift.TExecRequest.write(TExecRequest.java:2240)
> at org.apache.thrift.TSerializer.serialize(TSerializer.java:84)
> at
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:180){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
