[
https://issues.apache.org/jira/browse/IMPALA-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18085911#comment-18085911
]
Joe McDonnell commented on IMPALA-12389:
----------------------------------------
There are other changes needed to avoid accumulating trash:
# When we drop the unique database, Hive will move non-external tables to the
trash. This means that deleting the database directory itself using -skipTrash
doesn't help for some of the largest usages of trash space. To fix this, we
should delete the database directory using -skipTrash BEFORE dropping the
database.
# Some tests use directories outside the unique database. In particular,
Iceberg tests that use create_iceberg_table_from_directory() are creating the
table outside the unique database.
# Once we accumulate trash, the "hdfs -expunge -immediate" command can help
recover, but it can be very slow. A quick Google search recommends increasing
dfs.block.invalidate.limit to allow more blocks to be deleted in a single
heartbeat.
> Use -skipTrash to avoid accumulating files in the trash
> -------------------------------------------------------
>
> Key: IMPALA-12389
> URL: https://issues.apache.org/jira/browse/IMPALA-12389
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.3.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> When tests use unique databases, there is some logic to clean up the database
> directory when the test is done. This logic uses delete_file_dir. On some
> filesystem types like S3, this is implemented with HadoopFsCommandLineClient.
> {noformat}
> def delete_file_dir(self, path, recursive=False):
> """Delete the file or directory given by the specified path. Recursive
> must be true
> for directories."""
> rm_command = ['-rm', path]
> if recursive:
> rm_command = ['-rm', '-r', path]
> (status, stdout, stderr) = self._hadoop_fs_shell(rm_command)
> return status == 0{noformat}
> We should change this to add "-skipTrash" to avoid accumulating files in the
> trash directory. This is particularly important for S3, because the trash
> directory is currently not cleaned up by
> bin/jenkins/release_cloud_resources.sh.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
