[
https://issues.apache.org/jira/browse/IMPALA-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18086127#comment-18086127
]
ASF subversion and git services commented on IMPALA-12389:
----------------------------------------------------------
Commit 3aca44ba5e51f38a6850cb981ec90501cf8e0566 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3aca44ba5 ]
IMPALA-12389: Use -skipTrash to avoid accumulating trash
The default behavior for deleting files on Hadoop is to
move them to a trash folder. The trash folder can be
aged out, but Impala's developer environment sets the
trash to live a long time. This is a problem, because the
trash contents will continue to accumulate.
This combines multiple changes to avoid accumulating trash:
1. This changes HadoopFsCommandLineClient's delete_file_dir
to use -skipTrash to avoid accumulating the trash for
this case. This helps on non-HDFS test environments.
2. This changes the unique_database fixture to delete the
database directory before dropping the database. Non-external
tables deleted as part of DROP DATABASE .. CASCADE are
moved to the trash. Deleting the database directory ourselves
avoids sending these files to the trash.
3. "hdfs dfs -expunge -immediate" can recover the disk space, but
it is very slow. This increases the dfs.block.invalidate.limit
to allow HDFS to delete more blocks in a single heartbeat.
To support this change, there were other test-only changes:
- This updates a few tests that placed tables outside of the
unique_database. In particular, Iceberg tests using
create_iceberg_table_from_directory() were putting tables
outside the database.
- TestHdfsEncryption and TestHdfsPermissions used WebHDFS-style
paths without the leading slash. This is harmless, but this
cleans them up to use normal paths. This is safe, because the
delegating client converts it internally when using the WebHDFS
client.
Testing:
- Ran tests locally and examined the trash directory
Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8
Reviewed-on: http://gerrit.cloudera.org:8080/20386
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
> Use -skipTrash to avoid accumulating files in the trash
> -------------------------------------------------------
>
> Key: IMPALA-12389
> URL: https://issues.apache.org/jira/browse/IMPALA-12389
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.3.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> When tests use unique databases, there is some logic to clean up the database
> directory when the test is done. This logic uses delete_file_dir. On some
> filesystem types like S3, this is implemented with HadoopFsCommandLineClient.
> {noformat}
> def delete_file_dir(self, path, recursive=False):
> """Delete the file or directory given by the specified path. Recursive
> must be true
> for directories."""
> rm_command = ['-rm', path]
> if recursive:
> rm_command = ['-rm', '-r', path]
> (status, stdout, stderr) = self._hadoop_fs_shell(rm_command)
> return status == 0{noformat}
> We should change this to add "-skipTrash" to avoid accumulating files in the
> trash directory. This is particularly important for S3, because the trash
> directory is currently not cleaned up by
> bin/jenkins/release_cloud_resources.sh.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
