This is an automated email from the ASF dual-hosted git repository. michaelsmith pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/impala.git
commit 874e4fa117bdccfb8784c1987e5e3bf1ef4fbc1d Author: Laszlo Gaal <[email protected]> AuthorDate: Wed Jun 26 00:28:26 2024 +0200 IMPALA-13222: Clean up .Trash and temp files at the end of S3 test runs Remove the .Trash directory for HDFS, and temporary files left in /tmp and in /other from the S3 bucket used for an S3 test run. Deletion happens using AWSCLI after the minicluster is shut down. Files are deleted only from selected refixes (subdirectories) so that the cleanup logic is safe to use for private buckets, or the regular bucket for private-s3-parameterized runs, impala-test-uswest2-3 too, where other files may exist besides the ones generated for a test run. Tested by running an S3 build then checking the contents of the test bucket. Change-Id: I60a23394de8a67768a0b5b4c9c9576ee6a24348e Reviewed-on: http://gerrit.cloudera.org:8080/21585 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- bin/jenkins/release_cloud_resources.sh | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/bin/jenkins/release_cloud_resources.sh b/bin/jenkins/release_cloud_resources.sh index a03f95579..d6154e52c 100755 --- a/bin/jenkins/release_cloud_resources.sh +++ b/bin/jenkins/release_cloud_resources.sh @@ -43,8 +43,12 @@ if [[ "${TARGET_FILESYSTEM}" == "s3" ]]; then -region "${S3GUARD_DYNAMODB_REGION}" echo "Done cleaning up s3guard" fi - # Remove the test warehouse - echo "Removing test warehouse from s3://${S3_BUCKET}${TEST_WAREHOUSE_DIR} ..." - aws s3 rm --recursive --quiet s3://${S3_BUCKET}${TEST_WAREHOUSE_DIR} - echo "Done removing test warehouse" + + echo "Removing temporary data files and test warehouse..." + # Add temp files and HDFS .Trash to the delete list + for del_dir in ${TEST_WAREHOUSE_DIR} /other /tmp /user/jenkins/.Trash ; do + echo "Removing s3://${S3_BUCKET}${del_dir}" + aws s3 rm --recursive --quiet s3://${S3_BUCKET}${del_dir} + done + echo "Done cleaning the test bucket s3://${S3_BUCKET}" fi
