Bharath Vissapragada has uploaded a new change for review. http://gerrit.cloudera.org:8080/7652
Change subject: IMPALA-4847: Simplify HdfsTable block metadata loading code ...................................................................... IMPALA-4847: Simplify HdfsTable block metadata loading code This commit is a part of ground work for the upcoming multi threaded block metadata loading patches. The patch for IMPALA-4172 introduced code that groups the block location requests for partition directories that reside under the table directory into a single call to the NN in order to reduce the number of RPCs. However, it turns out that the hdfs client library internally makes one RPC per directory thus defeating the purpose of optimization. Also, this made the code unnecessarily complex since we need to map each file to its corresponding partition at runtime. This patch undos that optimization and makes HDFS calls per partition, which is much easier to understand. This also helps the upcoming patch on multi threaded block metadata loading since there is much less shared state when loading multiple partitions in parallel. Change-Id: I963d647bd2ba11e3843c6ef2ac6be113c74280bf --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java 1 file changed, 36 insertions(+), 105 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/7652/1 -- To view, visit http://gerrit.cloudera.org:8080/7652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I963d647bd2ba11e3843c6ef2ac6be113c74280bf Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada <[email protected]>
