Bharath Vissapragada has uploaded a new patch set (#2).

Change subject: IMPALA-4847: Simplify HdfsTable block metadata loading code
......................................................................

IMPALA-4847: Simplify HdfsTable block metadata loading code

This commit is a part of ground work for the upcoming multi
threaded block metadata loading patches.

The patch for IMPALA-4172 introduced code that groups the block
location requests for partition directories that reside under the
table directory into a single call to the NN in order to reduce the
number of RPCs. However, it turns out that the hdfs client library
internally makes one RPC per directory thus defeating the
purpose of optimization. Also, this made the code unnecessarily
complex since we need to map each file to its corresponding partition
at runtime.

This patch undos that optimization and makes HDFS calls per partition,
which is much easier to understand. This also helps the upcoming patch
on multi threaded block metadata loading since there is much less shared
state when loading multiple partitions in parallel.

Change-Id: I963d647bd2ba11e3843c6ef2ac6be113c74280bf
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
1 file changed, 60 insertions(+), 154 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/7652/2
-- 
To view, visit http://gerrit.cloudera.org:8080/7652
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I963d647bd2ba11e3843c6ef2ac6be113c74280bf
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bhara...@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com>

Reply via email to