[
https://issues.apache.org/jira/browse/SPARK-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632075#comment-14632075
]
Apache Spark commented on SPARK-9144:
-------------------------------------
User 'JoshRosen' has created a pull request for this issue:
https://github.com/apache/spark/pull/7484
> Remove DAGScheduler.runLocallyWithinThread and spark.localExecution.enabled
> ---------------------------------------------------------------------------
>
> Key: SPARK-9144
> URL: https://issues.apache.org/jira/browse/SPARK-9144
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Reporter: Josh Rosen
> Assignee: Josh Rosen
>
> Spark has an option called {{spark.localExecution.enabled}}; according to the
> docs:
> {quote}
> Enables Spark to run certain jobs, such as first() or take() on the driver,
> without sending tasks to the cluster. This can make certain jobs execute very
> quickly, but may require shipping a whole partition of data to the driver.
> {quote}
> This feature ends up adding quite a bit of complexity to DAGScheduler,
> especially in the {{runLocallyWithinThread}} method, but as far as I know
> nobody uses this feature (I searched the mailing list and haven't seen any
> recent mentions of the configuration nor stacktraces including the runLocally
> method). As a step towards scheduler complexity reduction, I propose that we
> remove this feature and all code related to it for Spark 1.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
