Quanlong Huang created IMPALA-13764:
---------------------------------------
Summary: JniFrontend.createExecRequest hits OOM of JVM array limit
Key: IMPALA-13764
URL: https://issues.apache.org/jira/browse/IMPALA-13764
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Quanlong Huang
Assignee: Quanlong Huang
When there are lots of files to scan in a query, the TExecRequest might need a
huge byte array to serialize which could hit OOM of exceeding the JVM array
limit (2GB):
{noformat}
I0215 09:22:04.852778 322082 jni-util.cc:321]
b04ef5f1a668be58:5831dd9200000000] java.lang.OutOfMemoryError: Requested array
size exceeds VM limit
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:197)
at
org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:204)
at
org.apache.impala.thrift.TScanRangeLocation$TScanRangeLocationStandardScheme.write(TScanRangeLocation.java:503)
at
org.apache.impala.thrift.TScanRangeLocation$TScanRangeLocationStandardScheme.write(TScanRangeLocation.java:448)
at
org.apache.impala.thrift.TScanRangeLocation.write(TScanRangeLocation.java:391)
at
org.apache.impala.thrift.TScanRangeLocationList$TScanRangeLocationListStandardScheme.write(TScanRangeLocationList.java:468)
at
org.apache.impala.thrift.TScanRangeLocationList$TScanRangeLocationListStandardScheme.write(TScanRangeLocationList.java:402)
at
org.apache.impala.thrift.TScanRangeLocationList.write(TScanRangeLocationList.java:342)
at
org.apache.impala.thrift.TScanRangeSpec$TScanRangeSpecStandardScheme.write(TScanRangeSpec.java:485)
at
org.apache.impala.thrift.TScanRangeSpec$TScanRangeSpecStandardScheme.write(TScanRangeSpec.java:413)
at
org.apache.impala.thrift.TScanRangeSpec.write(TScanRangeSpec.java:355)
at
org.apache.impala.thrift.TPlanExecInfo$TPlanExecInfoStandardScheme.write(TPlanExecInfo.java:512)
at
org.apache.impala.thrift.TPlanExecInfo$TPlanExecInfoStandardScheme.write(TPlanExecInfo.java:425)
at org.apache.impala.thrift.TPlanExecInfo.write(TPlanExecInfo.java:366)
at
org.apache.impala.thrift.TQueryExecRequest$TQueryExecRequestStandardScheme.write(TQueryExecRequest.java:1915)
at
org.apache.impala.thrift.TQueryExecRequest$TQueryExecRequestStandardScheme.write(TQueryExecRequest.java:1711)
at
org.apache.impala.thrift.TQueryExecRequest.write(TQueryExecRequest.java:1516)
at
org.apache.impala.thrift.TExecRequest$TExecRequestStandardScheme.write(TExecRequest.java:2920)
at
org.apache.impala.thrift.TExecRequest$TExecRequestStandardScheme.write(TExecRequest.java:2567)
at org.apache.impala.thrift.TExecRequest.write(TExecRequest.java:2240)
at org.apache.thrift.TSerializer.serialize(TSerializer.java:84)
at
org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:180){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
