Mostafa Mokhtar has posted comments on this change. Change subject: IMPALA-4172/IMPALA-3653: Improvements to block metadata loading ......................................................................
Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/5148/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: Line 368: if (!pathMd.filteredParitionPaths.contains(fileStatus.getPath())) continue; > Looks like this contains() and the indexOf() below can become very expensiv Tried out the patch and the ArrayList is very inefficient and causing a huge regression. Stack Trace Sample Count Percentage(%) org.apache.impala.catalog.HdfsTable.loadBlockMetadata(FileSystem, Path, HdfsTable$FilteredPartitionPathsMd, Map) 19,323 99.974 java.util.ArrayList.contains(Object) 9,831 50.864 java.util.ArrayList.indexOf(Object) 9,458 48.934 -- To view, visit http://gerrit.cloudera.org:8080/5148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ie127658172e6e70dae441374530674a4ac9d5d26 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Mostafa Mokhtar <[email protected]> Gerrit-HasComments: Yes
