[
https://issues.apache.org/jira/browse/SPARK-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012296#comment-14012296
]
Mridul Muralidharan edited comment on SPARK-1956 at 5/29/14 11:43 AM:
----------------------------------------------------------------------
We have not taken a call on when to contribute them back - particularly since
it is in context of large body of other work which is still ongoing.
Two things though :
a) shuffle consolidation in master is broken : please dont use it for anything
nontrivial(executor failures, speculative execution, etc).
b) Going forward, the default shuffle is not scalable - we need shuffle
consolidation, and possibly a better variant : but that would be a future
enhancement.
Btw, iirc there was some umbrella jira about shuffle consolidation from while
back .. not sure if it was cleaned up or not. We were planning to use that (if
exists) when we contribute back.
was (Author: mridulm80):
We have not taken a call on when to contribute them back - particularly since
it is in context of large body of other work which is still ongoing.
Two things though :
a) shuffle consolidation in master is broken : please dont use it for anything
nontrivial(executor failures, speculative execution, etc).
b) Going forward, the default shuffle is not scalable - we need shuffle
consolidation, and possibly a better variant : but that would be a future
enhancement.
> Enable shuffle consolidation by default
> ---------------------------------------
>
> Key: SPARK-1956
> URL: https://issues.apache.org/jira/browse/SPARK-1956
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
> Affects Versions: 1.0.0
> Reporter: Sandy Ryza
>
> The only drawbacks are on ext3, and most everyone has ext4 at this point. I
> think it's better to aim the default at the common case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
