Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16206 )
Change subject: IMPALA-9964: Fill file descriptors properly in setFileMetadataFromFS ...................................................................... Patch Set 4: (7 comments) http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3150 PS4, Line 3150: loadAndSet > If I read the code correctly loadAndSet() only differs from loadAndGet() in Ah, it looks we changed this recently in IMPALA-9778. In that case, I think we should just rename the method loadAndSet() to load() since we don't have a loadAndGet() method anymore. http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3152 PS4, Line 3152: HdfsPartition part = builder.getOldInstance(); I think it would be more readable if you can you add a comment here which suggests we need to use the oldInstance since the HdfsPartition.Builder creates a new HdfsPartition object from its oldInstance while partToPartialInfoMap uses the old HdfsPartition as key. http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java: http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@1005 PS4, Line 1005: encodedFileDescriptors_ = partition.encodedFileDescriptors_; Does this need to copy the encoded insert and delete FileDescriptors as well? http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java File fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java@115 PS4, Line 115: HdfsPartitions nit, please change this to HdfsPartition.Builders http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java@144 PS4, Line 144: load may be rename this to loadInternal and loadAndSet to load() http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java File fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java: http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java@563 PS4, Line 563: executeHiveSql("insert into " + getTestFullAcidTblName() + " select * from " : + "functional_orc_def.alltypes"); : executeHiveSql("delete from " + getTestFullAcidTblName() : + " where id % 2 = 0"); Its a little weird that we execute hiveSql in the try catch block for Impala jdbc client. Can we replace this try-final block with: executeImpalaSql("create table " + getTestFullAcidTblName() + " like functional_orc_def.alltypes"); executeHiveSql("insert into " + getTestFullAcidTblName() + " select * from functional_orc_def.alltypes"); executeHiveSql("delete from " + getTestFullAcidTblName() + " where id % 2 = 0"); http://gerrit.cloudera.org:8080/#/c/16206/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java@590 PS4, Line 590: TGetPartialCatalogObjectResponse response = sendRequest(request); I think it would be useful to add the following asserts here to make sure that the loaded table has correct number of files. Assert.assertEquals(24, response.getTable_info().getPartitionsSize()); for (TPartialPartitionInfo part : response.getTable_info().getPartitions()) { Assert.assertEquals(0, part.file_descriptors.size()); Assert.assertEquals(2, part.insert_file_descriptors.size()); Assert.assertEquals(2, part.delete_file_descriptors.size()); } -- To view, visit http://gerrit.cloudera.org:8080/16206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2927171cf426597c86766fb83d565c5e57025c73 Gerrit-Change-Number: 16206 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Fri, 17 Jul 2020 17:11:37 +0000 Gerrit-HasComments: Yes
