[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967720#comment-13967720
]
Patrick Wendell edited comment on SPARK-1476 at 4/13/14 3:41 AM:
-----------------------------------------------------------------
This says it's a "severe limitation" - but why not just use more, smaller
blocks? I think Spark's design assumes in various places that block's are not
extremely large. Think of it like e.g. the HDFS block size... it can't be
arbitrary large. The answer here might be to use multiple blocks in the case of
something like a shuffle where the size can get really large.
was (Author: pwendell):
This says it's a "severe limitation" - but why not just use more, smaller
blocks? I think Spark's design assumes in various places that block's are not
extremely large. Think of it like e.g. the HDFS block size... it can't be
arbitrary large.
> 2GB limit in spark for blocks
> -----------------------------
>
> Key: SPARK-1476
> URL: https://issues.apache.org/jira/browse/SPARK-1476
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Environment: all
> Reporter: Mridul Muralidharan
> Priority: Critical
> Fix For: 1.1.0
>
>
> The underlying abstraction for blocks in spark is a ByteBuffer : which limits
> the size of the block to 2GB.
> This has implication not just for managed blocks in use, but also for shuffle
> blocks (memory mapped blocks are limited to 2gig, even though the api allows
> for long), ser-deser via byte array backed outstreams (SPARK-1391), etc.
> This is a severe limitation for use of spark when used on non trivial
> datasets.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
