Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/10450 )
Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations. ...................................................................... Patch Set 2: (2 comments) > (2 comments) > > Just interested in this optimization. May I ask some questions? > > Looks like we are optimizing the case when partitionsToUpdate != > null and partitions were neither dropped, created. Can we optimize > the case that partitionsToUpdate != null and some partitions are > dropped? For example when an INSERT OVERWRITE statement updates the > majority of the partitions and only drops few of them. Will this case not cause introduce inconsistency between HMS and Impala ? http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1409 PS2, Line 1409: size() == 0 > nit: can be simplified by isEmpty() OK http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1410 PS2, Line 1410: partitionsToUpdateFileMdByPath = getPartitionsByPath(partitionsToUpdate); : loadMetadataAndDiskIds(partitionsToUpdateFileMdByPath, true); > Looks like the original codes perform the same as these two lines. Since dr The new change behaves very much like the old code except for the case when dirtyPartitions exist in that case there is an overhead of dropping and loading the dirty partitions from Metastore. -- To view, visit http://gerrit.cloudera.org:8080/10450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a Gerrit-Change-Number: 10450 Gerrit-PatchSet: 2 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Comment-Date: Sat, 19 May 2018 18:14:42 +0000 Gerrit-HasComments: Yes
