Andrew Or created SPARK-6920:
--------------------------------
Summary: Be more explicit about references to "executor" and
"task" in Spark on Mesos
Key: SPARK-6920
URL: https://issues.apache.org/jira/browse/SPARK-6920
Project: Spark
Issue Type: Bug
Components: Documentation, Mesos
Affects Versions: 1.0.0
Reporter: Andrew Or
In both Spark and Mesos, the terms "executor" and "task" mean different things
depending on the context. In the past, this has caused a great deal of
confusion, to both the user and the developer of Spark.
The consequences are real. For instance, the fine-grained
`MesosSchedulerBackend` code incorrectly uses `spark.tasks.cpus` as the number
of cores to grant the Mesos executor [0]. This is a result of conflating the
Mesos executor with the Spark executor, and the confusion is reflected in the
comment, where the "executor" here refers to the Mesos executor, not the Spark
executor.
{code}
// If the executor doesn't exist yet, subtract CPU for executor
{code}
We really need to be explicit about our references to these terms such that
this part of the Spark code can progress as quickly as other modules.
[0]
https://github.com/apache/spark/blob/6de282e2de3cb69f9b746d03fde581429248824a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala#L238
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
