[
https://issues.apache.org/jira/browse/SPARK-19167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827704#comment-15827704
]
xwebber commented on SPARK-19167:
---------------------------------
I think this is a core issue that should be fixed. We have meet serveral
scenes that this issue affect our application including not able to create new
sc in the same process and SPARK_YARN_MODE missing because of the "old" sc.stop
haven't finished and set "SparkContext already stopped." too early. I suggest
we make sc.stop() blocked until other sc.stop() call finishes anyway.
> .inprogress file can't be renamed if driver exit when OOM occured in
> AsynchronousListenerBus thread
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-19167
> URL: https://issues.apache.org/jira/browse/SPARK-19167
> Project: Spark
> Issue Type: Bug
> Reporter: xwebber
>
> I was running a spark application and OOM occured in AsynchronousListenerBus
> thread. This will cause Utils.tryOrStopSparkContext(sparkContext) to do
> sc.stop(). And then my user code catch the exception that sc is stopping, and
> this will cause spark driver shut down. but at this time the sc.stop in
> AsynchronousListenerBus thread hasn't call _cleaner.foreach(_.stop()) yet.
> This will cause the file left on disk forever. can it be solved?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
