Mridul Muralidharan created SPARK-1476:
------------------------------------------
Summary: 2GB limit in spark for blocks
Key: SPARK-1476
URL: https://issues.apache.org/jira/browse/SPARK-1476
Project: Spark
Issue Type: Bug
Components: Spark Core
Environment: all
Reporter: Mridul Muralidharan
Priority: Critical
The underlying abstraction for blocks in spark is a ByteBuffer : which limits
the size of the block to 2GB.
This has implication not just for managed blocks in use, but also for shuffle
blocks (memory mapped blocks are limited to 2gig, even though the api allows
for long), ser-deser via byte array backed outstreams (SPARK-1391), etc.
This is a severe limitation for use of spark when used on non trivial datasets.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
