[
https://issues.apache.org/jira/browse/SPARK-21822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lufei updated SPARK-21822:
--------------------------
Description:
When insert Hive Table is finished, it is better to clean out the tmpLocation
dir(the temp directories like
".hive-staging_hive_2017-08-19_10-56-01_540_5448395226195533570-9/-ext-10000"
or "/tmp/hive/..." for an old spark version).
Otherwise, when lots of spark job are executed, millions of temporary
directories are left in HDFS. And these temporary directories can only be
deleted by the maintainer through the shell script.
was:
When insert Hive Table is finished, it is better to clean out the tmpLocation
dir(the temp directorys like
".hive-staging_hive_2017-08-19_10-56-01_540_5448395226195533570-9/-ext-10000"
or "/tmp/hive/..." for an old spark version).
Otherwise, when lots of spark job are executed, millions of temporary
directories are left in HDFS. And these temporary directories can only be
deleted by the maintainer through the shell script.
> When insert Hive Table is finished, it is better to clean out the tmpLocation
> dir
> ---------------------------------------------------------------------------------
>
> Key: SPARK-21822
> URL: https://issues.apache.org/jira/browse/SPARK-21822
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: lufei
>
> When insert Hive Table is finished, it is better to clean out the tmpLocation
> dir(the temp directories like
> ".hive-staging_hive_2017-08-19_10-56-01_540_5448395226195533570-9/-ext-10000"
> or "/tmp/hive/..." for an old spark version).
> Otherwise, when lots of spark job are executed, millions of temporary
> directories are left in HDFS. And these temporary directories can only be
> deleted by the maintainer through the shell script.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
