GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/22705
[SPARK-25704][CORE][WIP] Allocate a bit less than Int.MaxValue
JVMs don't you allocate arrays of length exactly Int.MaxValue, so leave
a little extra room. This is necessary when reading blocks >2GB off
the network (for remote reads or for cache replication).
WIP because I'm still running tests on a real cluster
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark SPARK-25704
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22705.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22705
----
commit cb07badcd853da0e4083b7e02bdfdf86c9d295f1
Author: Imran Rashid <irashid@...>
Date: 2018-10-12T01:54:34Z
[SPARK-25704][CORE] Allocate a bit less than Int.MaxValue
JVMs don't you allocate arrays of length exactly Int.MaxValue, so leave
a little extra room. This is necessary when reading blocks >2GB off
the network (for remote reads or for cache replication).
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
