Aaron Davidson created SPARK-1700:
-------------------------------------
Summary: PythonRDD leaks socket descriptors during cancellation
Key: SPARK-1700
URL: https://issues.apache.org/jira/browse/SPARK-1700
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 0.9.0, 1.0.0
Reporter: Aaron Davidson
Assignee: Aaron Davidson
Sockets from Spark to Python workers are not cleaned up over the duration of a
job, causing the total number of opened file descriptors to grow to around the
number of partitions in the job. Usually these go away if the job is
successful, but in the case of cancellation (and possibly exceptions, though I
haven't investigated), the socket file descriptors remain indefinitely.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
