Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/22378 )
Change subject: IMPALA-13677: Add startup cleanup for remote scratch ...................................................................... Patch Set 3: (2 comments) Thanks Daniel for the review. http://gerrit.cloudera.org:8080/#/c/22378/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/22378/2//COMMIT_MSG@21 PS2, Line 21: This design assumes one Impala daemon per host. > Could this be extended to multiple impalads per host, for example by checki It’s possible to extend support for this in the future. I think this patch is for a quick improvement for the most common scenario, which is one impalad per host. Discussed with Abhishek before, in the real use, this seems typically the case. For clusters with multiple impalads per host, we can set the cleanup flag to false for now. Added some comments for the flag and commit message to clarify this http://gerrit.cloudera.org:8080/#/c/22378/2/be/src/runtime/tmp-file-mgr.cc File be/src/runtime/tmp-file-mgr.cc: http://gerrit.cloudera.org:8080/#/c/22378/2/be/src/runtime/tmp-file-mgr.cc@882 PS2, Line 882: return Status(GetHdfsErrorMsg("HDFS create path failed: ", path_)); > Will this delete the directory itself or only its contents? It will delete the host-level dir. For example, the path_ is "hdfs://xxx/impala-scratch", then it will try to delete the dir of "hdfs://xxx/impala-scratch/hostname" if it exists. During the tests, it seems no trouble to recreate this host-level hdfs dir later in the upload process. -- To view, visit http://gerrit.cloudera.org:8080/22378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadd49b7384d52bac5ddab4e86cd9f39dc2c88e1b Gerrit-Change-Number: 22378 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]> Gerrit-Comment-Date: Fri, 24 Jan 2025 09:40:39 +0000 Gerrit-HasComments: Yes
