xwebber created SPARK-19167: ------------------------------- Summary: .inprogress file can't be renamed if driver exit when OOM occured in AsynchronousListenerBus thread Key: SPARK-19167 URL: https://issues.apache.org/jira/browse/SPARK-19167 Project: Spark Issue Type: Bug Reporter: xwebber
I was running a spark application and OOM occured in AsynchronousListenerBus thread. This will cause Utils.tryOrStopSparkContext(sparkContext) to do sc.stop(). And then my user code catch the exception that sc is stopping, and this will cause spark driver shut down. but at this time the sc.stop in AsynchronousListenerBus thread hasn't call _cleaner.foreach(_.stop()) yet. This will cause the file left on disk forever. can it be solved? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org