[
https://issues.apache.org/jira/browse/SPARK-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-9144:
------------------------------
Component/s: Scheduler
> Remove DAGScheduler.runLocallyWithinThread and spark.localExecution.enabled
> ---------------------------------------------------------------------------
>
> Key: SPARK-9144
> URL: https://issues.apache.org/jira/browse/SPARK-9144
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler, Spark Core
> Reporter: Josh Rosen
> Assignee: Josh Rosen
>
> Spark has an option called {{spark.localExecution.enabled}}; according to the
> docs:
> {quote}
> Enables Spark to run certain jobs, such as first() or take() on the driver,
> without sending tasks to the cluster. This can make certain jobs execute very
> quickly, but may require shipping a whole partition of data to the driver.
> {quote}
> This feature ends up adding quite a bit of complexity to DAGScheduler,
> especially in the {{runLocallyWithinThread}} method, but as far as I know
> nobody uses this feature (I searched the mailing list and haven't seen any
> recent mentions of the configuration nor stacktraces including the runLocally
> method). As a step towards scheduler complexity reduction, I propose that we
> remove this feature and all code related to it for Spark 1.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
