[
https://issues.apache.org/jira/browse/SPARK-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14029416#comment-14029416
]
Thomas Graves commented on SPARK-1740:
--------------------------------------
On yarn when I run pyspark it kills the executors after a single action.
Perhaps this is caused by this same issue.
This could be a much bigger deal on yarn since when it kills the executors, it
has to go back to the resource manager to get more containers. This is an
aweful lot of thrashing of containers and could cause major headaches.
> Pyspark cancellation kills unrelated pyspark workers
> ----------------------------------------------------
>
> Key: SPARK-1740
> URL: https://issues.apache.org/jira/browse/SPARK-1740
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.0.0
> Reporter: Aaron Davidson
>
> PySpark cancellation calls SparkEnv#destroyPythonWorker. Since there is one
> python worker per process, this would seem like a sensible thing to do.
> Unfortunately, this method actually destroys a python daemon, and all
> associated workers, which generally means that we can cause failures in
> unrelated Pyspark jobs.
> The severity of this bug is limited by the fact that the Pyspark daemon is
> easily recreated, so the tasks will succeed after being restarted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
