xwebber created SPARK-19167:
-------------------------------
Summary: .inprogress file can't be renamed if driver exit when OOM
occured in AsynchronousListenerBus thread
Key: SPARK-19167
URL: https://issues.apache.org/jira/browse/SPARK-19167
Project: Spark
Issue Type: Bug
Reporter: xwebber
I was running a spark application and OOM occured in AsynchronousListenerBus
thread. This will cause Utils.tryOrStopSparkContext(sparkContext) to do
sc.stop(). And then my user code catch the exception that sc is stopping, and
this will cause spark driver shut down. but at this time the sc.stop in
AsynchronousListenerBus thread hasn't call _cleaner.foreach(_.stop()) yet. This
will cause the file left on disk forever. can it be solved?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
