Kay Ousterhout created SPARK-9326:
-------------------------------------
Summary: Spark never closes the lock file used to prevent
concurrent downloads
Key: SPARK-9326
URL: https://issues.apache.org/jira/browse/SPARK-9326
Project: Spark
Issue Type: Bug
Affects Versions: 1.4.0
Reporter: Kay Ousterhout
Assignee: Kay Ousterhout
Priority: Minor
A lock file is used to ensure multiple executors running on the same machine
don't download the same file concurrently. Spark never closes these lock files
(we release the lock, but releasing the lock does not close the underlying
file). In theory, if an executor fetched a large number of files, this could
eventually lead to an issue with too many open files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
