Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14547 )
Change subject: WIP IMPALA-9045: Filter base directories of open/aborted compactions ...................................................................... Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14547/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/14547/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@581 PS1, Line 581: ValidTxnList validTxnList = writeIds != null ? loadValidTxns(client) : null; > Write ids are local to the tables, so it was reasonable to store them in th Yeah, it is hard for me to imagine such a fast compaction, but it is possible that they will become faster in the future. The current patch is already a big improvement, so it may be the best to create a follow up Jira and add some TODOs. Another unsolved issue with caching table metadata + compactions is that files in old deltas can be deleted, as we do not have a read lock on them. I think that in the long run we will need a more complex solution on Impala and/or HMS side to achieve both performance and correctness. -- To view, visit http://gerrit.cloudera.org:8080/14547 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6 Gerrit-Change-Number: 14547 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Tue, 29 Oct 2019 14:56:27 +0000 Gerrit-HasComments: Yes
