Josh Rosen created SPARK-9144:
---------------------------------
Summary: Remove DAGScheduler.runLocallyWithinThread and
spark.localExecution.enabled
Key: SPARK-9144
URL: https://issues.apache.org/jira/browse/SPARK-9144
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Josh Rosen
Assignee: Josh Rosen
Spark has an option called {{spark.localExecution.enabled}}; according to the
docs:
{quote}
Enables Spark to run certain jobs, such as first() or take() on the driver,
without sending tasks to the cluster. This can make certain jobs execute very
quickly, but may require shipping a whole partition of data to the driver.
{quote}
This feature ends up adding quite a bit of complexity to DAGScheduler,
especially in the {{runLocallyWithinThread}} method, but as far as I know
nobody uses this feature (I searched the mailing list and haven't seen any
recent mentions of the configuration nor stacktraces including the runLocally
method). As a step towards scheduler complexity reduction, I propose that we
remove this feature and all code related to it for Spark 1.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
