Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 )
Change subject: acid: Filter unwanted files based on ACID state. ...................................................................... Patch Set 15: (3 comments) Did a bit of restructuring to avoid a potential performance problem: we now filter the FileStatuses before converting them to FileDescriptors, which has the advantage of not needing to fetch locations for files which are actually superceded or invalid. That should make REFRESH faster on post-compaction tables. It also made the call signature into AcidUtils simpler -- just a single method call to do the filtering, rather than separate calls to do the ValidWrite filtering vs the BaseData filtering. Extended the tests with SHOW FILES for post-compaction. Extended unit tests for a few other cases that Csaba pointed out. http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java File fe/src/main/java/org/apache/impala/util/AcidUtils.java: http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@48 PS14, Line 48: n Transa > Good point. I'll see if I can add a test case for this one. Ended up restructing the code a bit and adding a test case for this one, both e2e (by a SHOW FILES command) and here. http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@105 PS14, Line 105: > Good point. Mind if I add a TODO to support upgraded tables? I'm not sure h Added some unit test cases to catch this, and left a TODO for an e2e. http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@170 PS14, Line 170: * @param stats the FileStatuses obtained from recursively listing the directory : * @param baseDir the base dire > We will go to this branch with any file if getMaxBaseId() didn't found any got a bit restructured. LMK what you think -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 15 Gerrit-Owner: Sudhanshu Arora <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Sudhanshu Arora <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Yongzhi Chen <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Wed, 22 May 2019 23:31:34 +0000 Gerrit-HasComments: Yes
