Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-4847: Simplify HdfsTable block metadata loading code ......................................................................
IMPALA-4847: Simplify HdfsTable block metadata loading code This commit is a part of ground work for the upcoming multi threaded block metadata loading patches. The patch for IMPALA-4172 introduced code that groups the block location requests for partition directories that reside under the table directory into a single call to the NN in order to reduce the number of RPCs. However, it turns out that the hdfs client library internally makes one RPC per directory thus defeating the purpose of optimization. Also, this made the code unnecessarily complex since we need to map each file to its corresponding partition at runtime. This patch undos that optimization and makes HDFS calls per partition, which is much easier to understand. This also helps the upcoming patch on multi threaded block metadata loading since there is much less shared state when loading multiple partitions in parallel. Change-Id: I963d647bd2ba11e3843c6ef2ac6be113c74280bf Reviewed-on: http://gerrit.cloudera.org:8080/7652 Reviewed-by: Bharath Vissapragada <[email protected]> Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java 2 files changed, 60 insertions(+), 180 deletions(-) Approvals: Impala Public Jenkins: Verified Bharath Vissapragada: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/7652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I963d647bd2ba11e3843c6ef2ac6be113c74280bf Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Dimitris Tsirogiannis <[email protected]> Gerrit-Reviewer: Impala Public Jenkins
