Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11227 )
Change subject: IMPALA-7047. Refreshing partitions should not make an RPC per file ...................................................................... Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/11227/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/11227/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1256 PS1, Line 1256: * Updates the file metadata of an unpartitioned HdfsTable. Maybe we should add that this is optimized for incremental load. (Full load for unpartitioned tables seem to be taking loadAllPartitions() -> loadMetadataAndDiskIDs()). http://gerrit.cloudera.org:8080/#/c/11227/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1264 PS1, Line 1264: resetPartitions(); : org.apache.hadoop.hive.metastore.api.Table msTbl = getMetaStoreTable(); : Preconditions.checkNotNull(msTbl); : setPrototypePartition(msTbl.getSd()); : HdfsPartition part = createPartition(msTbl.getSd(), null, new FsPermissionCache()); Do we need all this resetting and creating stuff for incremental updates? Can't we just do refreshPartitionFileMetadata(oldPartition) or something like that? (or directly call it from the load()) http://gerrit.cloudera.org:8080/#/c/11227/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2068 PS1, Line 2068: refreshedPartition.setFileDescriptors(oldPartition.getFileDescriptors()); same here, can't we directly call refreshpartitionFileMetadata(oldPartition)? (after updating it's hmsPartition) -- To view, visit http://gerrit.cloudera.org:8080/11227 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2051b96599206164aaa06ecbdf64374c46eda956 Gerrit-Change-Number: 11227 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Vuk Ercegovac <[email protected]> Gerrit-Comment-Date: Thu, 16 Aug 2018 22:40:53 +0000 Gerrit-HasComments: Yes
