Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13215 )
Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition ...................................................................... Patch Set 6: (5 comments) http://gerrit.cloudera.org:8080/#/c/13215/6/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/13215/6/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@388 PS6, Line 388: public static long getHighestWriteId(String validWriteIds) This method is no longer used. Let's remove it until we have a use http://gerrit.cloudera.org:8080/#/c/13215/6/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java File fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java: http://gerrit.cloudera.org:8080/#/c/13215/6/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@111 PS6, Line 111: public long getWriteId() { maybe we can implement these as default methods in the FeTable interface? http://gerrit.cloudera.org:8080/#/c/13215/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/13215/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@924 PS6, Line 924: //TODO put the writeID load here for now. Can you rephrase this to explain the _future_ state, not the current state? http://gerrit.cloudera.org:8080/#/c/13215/6/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/13215/6/fe/src/main/java/org/apache/impala/catalog/Table.java@287 PS6, Line 287: LOG.warn("Could not get valid writeIds for: " + tblFullName, e); shouldn't we be passing this through as a tableloadingexception of some kind? it seems bad to silently drop the information. http://gerrit.cloudera.org:8080/#/c/13215/6/fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java File fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java: http://gerrit.cloudera.org:8080/#/c/13215/6/fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java@99 PS6, Line 99: <= MetastoreShim.getHighestWriteId(t.getValidWriteIds())); Instead can we assert on t.getValidWriteIds().isWriteIdValid(t.getWriteId())? (we may need to add a shim class for ValidWriteIdList btw -- ie make a stub implementation in the hive-2-shims source directly that has the same method names, but throws UnsupportedOperationException) -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Sudhanshu Arora <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Yongzhi Chen <[email protected]> Gerrit-Comment-Date: Tue, 07 May 2019 08:12:04 +0000 Gerrit-HasComments: Yes
