[
https://issues.apache.org/jira/browse/SPARK-10730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-10730.
-------------------------------
Resolution: Not A Problem
[~patrizio.munzi] please first read
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
I don't think this qualifies as a problem, not with this much information.
Spark schedules according to many heuristics, and it's not necessarily ideal to
spread work over machines. In your case you're working with a local collection
that's been sent to the cluster; it may be optimal to execute locally.
> spark receivers not evenly distributed among all cluster slaves
> ---------------------------------------------------------------
>
> Key: SPARK-10730
> URL: https://issues.apache.org/jira/browse/SPARK-10730
> Project: Spark
> Issue Type: Bug
> Reporter: Patrizio Munzi
>
> I'm running a yarn cluster of 8 executors and the spark receivers are not
> evenly distributed among all cluster slaves.
> Sometimes they all get executed in the same machine!!
> I also tried to reinforce the concept you added after the last same bug
> submission, that is running a dummy job to ensure all slaves have joined the
> cluster.
> Here's my code:
> log.info("Starting a dummy job to ensure all slaves have
> registered.");
> final int dummyListLength = 1000;
> final List<Integer> dummyList = new ArrayList<Integer>();
> for(int i = 1; i <= dummyListLength; i++) {
> dummyList.add(i);
> }
> jssc.sparkContext()
> .parallelize(dummyList, dummyListLength)
> .mapToPair(new DummyPairFunction())
> .reduceByKey(new DummyReduceByKeyFunction())
> .collect();
> log.info("Dummy job completed.");
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
