[
https://issues.apache.org/jira/browse/SPARK-31346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jackey Lee updated SPARK-31346:
-------------------------------
Description: In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use
deleteExternalTmpPath to clean temporary directories after Job committed and
cancel deleteOnExit if succeeded. But sometimes (e.g., when speculative task is
enabled), temporary directories may be left uncleaned. This is happened if
there are still some tasks running after we called deleteExternalTmpPath. Thus
it maybe necessary to keep deleteOnExit, even if temporary directory has
already deleted, to make sure the temporary directories cleaned. (was: In
InsertIntoHiveTable and InsertIntoHiveDirCommand, we use deleteExternalTmpPath
to clean temporary directories after Job committed and cancel deleteOnExit if
succeeded. But sometimes (e.g., when speculative task is enabled), temporary
directories may be left uncleaned. This is happened if there are still some
tasks running after we called deleteExternalTmpPath. Thus it maybe necessary to
keep deleteOnExit, even if temporary directory has already deleted, to verify
the temporary directories cleaned.)
Summary: Add new configuration to make sure temporary directory cleaned
(was: Add new configuration to verify temporary directory cleaned)
> Add new configuration to make sure temporary directory cleaned
> --------------------------------------------------------------
>
> Key: SPARK-31346
> URL: https://issues.apache.org/jira/browse/SPARK-31346
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Jackey Lee
> Priority: Major
>
> In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use
> deleteExternalTmpPath to clean temporary directories after Job committed and
> cancel deleteOnExit if succeeded. But sometimes (e.g., when speculative task
> is enabled), temporary directories may be left uncleaned. This is happened if
> there are still some tasks running after we called deleteExternalTmpPath.
> Thus it maybe necessary to keep deleteOnExit, even if temporary directory has
> already deleted, to make sure the temporary directories cleaned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
