hi devs, In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use deleteExternalTmpPath to clean temporary directories after Job committed and cancel deleteOnExit if succeeded.But sometimes (e.g., when speculative task is enabled), temporary directories may be left uncleaned. This is happened if there are still some tasks running after we called deleteExternalTmpPath. Thus I tried to add JobCleaned Status to clean temporary directories. The JobCleaned Status is happened with all stages has been cleaned in a job, thus it is a good way to do some job cleanings.
JIRA: https://issues.apache.org/jira/browse/SPARK-31438 PR: https://github.com/apache/spark/pull/28280 There have been some discuss in https://github.com/apache/spark/pull/28129. This PR has been submitted for about 3 months. Any one could review this? Thanks, Jacky Lee -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
