Jackey Lee created SPARK-31346:
----------------------------------
Summary: Add new configuration to verify temporary directory
cleaned
Key: SPARK-31346
URL: https://issues.apache.org/jira/browse/SPARK-31346
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.1.0, 3.0.1
Reporter: Jackey Lee
In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use
deleteExternalTmpPath to clean temporary directories after Job committed and
cancel deleteOnExit if succeeded. But sometimes (e.g., when speculative task is
enabled), temporary directories may be left uncleaned. This is happened if
there are still some tasks running after we called deleteExternalTmpPath. Thus
it maybe necessary to keep deleteOnExit, even if temporary directory has
already deleted, to verify the temporary directories cleaned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
