GitHub user liufengdb opened a pull request:
https://github.com/apache/spark/pull/19460
[SPARK-22222][core] Fix the ARRAY_MAX in BufferHolder and add a test
## What changes were proposed in this pull request?
We should not break the assumption that the length of the allocated byte
array is word rounded:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java#L170
So we want to use `Integer.MAX_VALUE - 15` instead of `Integer.MAX_VALUE -
8` as the upper bound of an allocated byte array.
## How was this patch tested?
Since the Spark unit test JVM has less than 1GB heap, here we run the test
code as a submit job, so it can run on a JVM has 4GB memory.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liufengdb/spark fix_array_max
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19460.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19460
----
commit 92a6d2d53aea02042d47888e99df5a4f2167cd1f
Author: Feng Liu <[email protected]>
Date: 2017-10-09T17:43:39Z
init
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
