Yin Huai created SPARK-10087:
--------------------------------
Summary: In some cases, all reducers are scheduled to the same
executor
Key: SPARK-10087
URL: https://issues.apache.org/jira/browse/SPARK-10087
Project: Spark
Issue Type: Bug
Components: Scheduler
Affects Versions: 1.5.0
Reporter: Yin Huai
Priority: Critical
In some cases, when spark.shuffle.reduceLocality.enabled is enabled, we are
scheduling all reducers to the same executor (the cluster has plenty of
resources). Changing spark.shuffle.reduceLocality.enabled to false resolve the
problem.
Here is a little bit more information. For one of my query, all 200 reducers
were scheduled to the same reducer and every reducer has about 800 KB input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
