Quanlong Huang created IMPALA-13782:
---------------------------------------
Summary: Avoid hitting OOM of exceeding JVM array limit
Key: IMPALA-13782
URL: https://issues.apache.org/jira/browse/IMPALA-13782
Project: IMPALA
Issue Type: Epic
Components: Catalog, Frontend
Reporter: Quanlong Huang
Assignee: Quanlong Huang
JVM has a limit on the array length depending on the implementation, e.g. it's
Integer.MAX_VALUE - 8 in OpenJDK.
{code:java}
/**
* The maximum size of array to allocate.
* Some VMs reserve some header words in an array.
* Attempts to allocate larger arrays may result in
* OutOfMemoryError: Requested array size exceeds VM limit
*/
private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8; {code}
https://github.com/openjdk/jdk8u/blob/4c5094095fade410e23e6a071c93c7e4fdc1dade/jdk/src/share/classes/java/io/ByteArrayOutputStream.java#L102
We have seen such kinds of OutOfMemoryError in many places during Thrift
serialization, i.e. converting a Thrift object into a byte array (byte[]). This
epic tracks these issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
