Dimitris Tsirogiannis has posted comments on this change. Change subject: IMPALA-4029: Reduce memory requirements for storing file metadata ......................................................................
Patch Set 6: (9 comments) http://gerrit.cloudera.org:8080/#/c/6406/5/fe/src/main/java/org/apache/impala/catalog/DiskIdMapper.java File fe/src/main/java/org/apache/impala/catalog/DiskIdMapper.java: Line 46: // the storage ID of a particular disk is unique across all the nodes in the cluster. > ? Oops, sorry missed that one. Done http://gerrit.cloudera.org:8080/#/c/6406/6/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java: Line 112: ListMap<TNetworkAddress> hostIndex, Reference<Long> unknownDiskIds) > numUnknown...? Done Line 275: * File Block metadata > provide more information (such as the fact that it's used in conjunction wi Done Line 283: * Constructs the metadata of a file block from its block location metadata > "Constructs an FbFileBlock..."? Done Line 291: ListMap<TNetworkAddress> hostIndex, Reference<Long> unknownDiskIds) > numUnknown...? Done Line 305: boolean isReplicaCached = cachedHosts.contains(loc.getHosts()[i]); > doesn't guava have some kind of arrayutils.contains(loc.getcachedhosts(), l Not in Guava, ArrayUtils is a class in Apache Commons. Line 327: * using 'fbb' and returns the offset in the underlying buffer where the encoded file > "in the underlying...starts": fine to leave out, that's implied by fb seman Agreed but maybe it's good to explicitly mention it for future readers that may not be familiar with FB semantics. http://gerrit.cloudera.org:8080/#/c/6406/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: Line 287: Reference<Long> unknownDiskIds = new Reference<Long>(); > numUnknown...? Unfortunately not, generics don't work with primitive types. Miss C++ :)? Line 741: * Helper method to load the partition file metadata from scratch. This method is > is this from a rebase? Yes, this is from the performance improvements of REFRESH. -- To view, visit http://gerrit.cloudera.org:8080/6406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I483d3cadc9d459f71a310c35a130d073597b0983 Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dimitris Tsirogiannis <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Dimitris Tsirogiannis <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Marcel Kornacker <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-HasComments: Yes
