Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18220 )
Change subject: IMPALA-11107: Allow specifying footer size in IssueFooterRanges ...................................................................... IMPALA-11107: Allow specifying footer size in IssueFooterRanges FOOTER_SIZE was a constant of 100KB in HdfsScanner::IssueFooterRanges, an estimate based on Parquet format. Other scanner subclasses such as HdfsOrcScanner expect that footer size is much lower at 16KB (orc::DIRECTORY_SIZE_GUESS in ORC lib). This patch adds footer_size_estimate as a parameter so different file formats can ask for different footer range sizes. Having a more precise initial range can also help reduce waste in the data cache. Testing: - Pass core tests. - Manually verify that 'IoReadSkippedBytes' reduced after this patch. Change-Id: Ib0a5aab48324bf9e78fb0bb0bf6f649d87e89dfc Reviewed-on: http://gerrit.cloudera.org:8080/18220 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java 7 files changed, 39 insertions(+), 21 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib0a5aab48324bf9e78fb0bb0bf6f649d87e89dfc Gerrit-Change-Number: 18220 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]>
