[
https://issues.apache.org/jira/browse/SPARK-19167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-19167.
-------------------------------
Resolution: Invalid
If you're asking a question -- use [email protected]. Do you have a
suggested change?
I think that in the case of OOM, some bets may be off. Cleanup is best effort.
You can try calling sc.stop() too because that's good practice, and because it
should block until the other stop() call finishes anyway.
> .inprogress file can't be renamed if driver exit when OOM occured in
> AsynchronousListenerBus thread
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-19167
> URL: https://issues.apache.org/jira/browse/SPARK-19167
> Project: Spark
> Issue Type: Bug
> Reporter: xwebber
>
> I was running a spark application and OOM occured in AsynchronousListenerBus
> thread. This will cause Utils.tryOrStopSparkContext(sparkContext) to do
> sc.stop(). And then my user code catch the exception that sc is stopping, and
> this will cause spark driver shut down. but at this time the sc.stop in
> AsynchronousListenerBus thread hasn't call _cleaner.foreach(_.stop()) yet.
> This will cause the file left on disk forever. can it be solved?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
