[
https://issues.apache.org/jira/browse/IMPALA-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095494#comment-18095494
]
ASF subversion and git services commented on IMPALA-15157:
----------------------------------------------------------
Commit 234fb8592bfa7bc8cae415a769789b39f5aba5ca in impala's branch
refs/heads/master from zhangyifan27
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=234fb8592 ]
IMPALA-15157: Avoid decompressing a page that is fully skipped at a page
boundary
During late materialization / row-level skipping, when the rows to skip
end exactly on a Parquet page boundary (e.g. an entire row group is
filtered out by row-level predicates), BaseScalarColumnReader::
SkipRowsInternal() would fall out of the page-skipping loop and then
decompress that final page via ReadCurrentDataPage() only to then skip
all of its values.
This changes the loop guard from '>' to '>=' so a page whose values are
all skipped is discarded as a whole compressed page (counted in
NumPagesSkippedByLateMaterialization) without decompressing it. When the
skip lands exactly on the page boundary we short-circuit and return
before reading the next page header. Query results are unchanged; this
only removes the unnecessary decompression and in-page value skipping.
Testing:
- Updated and extended parquet-late-materialization-unique-db.test to
cover skipping to a page boundary (whole-page discard) and in-page
value skipping.
Assisted-by: Claude Opus 4.8
Change-Id: I18f0e69c267c9bf041ff9f6a3d677239ae43971a
Reviewed-on: http://gerrit.cloudera.org:8080/24566
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Parquet scanner reads non-existent page header when skipping to a page
> boundary
> -------------------------------------------------------------------------------
>
> Key: IMPALA-15157
> URL: https://issues.apache.org/jira/browse/IMPALA-15157
> Project: IMPALA
> Issue Type: Improvement
> Reporter: YifanZhang
> Priority: Major
>
> {{BaseScalarColumnReader::SkipRowsInternal()}} (in
> {{{}be/src/exec/parquet/parquet-column-readers.cc{}}}) can attempt to read a
> page header that does not exist when late materialization / row-level
> skipping needs to skip a set of rows that ends exactly on a Parquet page
> boundary — for example, when an entire row group is filtered out by row-level
> predicates and the rows to skip run to the end of the column chunk.
> *Root cause*
> The whole-page-skip loop discarded pages only while the number of rows to
> skip was _strictly greater_ than the values buffered in the current page:
> {code:java}
> while (num_rows > num_buffered_values_) {
> ...
> if (!col_chunk_reader_.SkipPageData().ok() || !AdvanceNextPageHeader()) {
> return false;
> }
> } {code}
> When {{num_rows == num_buffered_values_}} (the skip ends exactly at a page
> boundary), the loop exited and the code proceeded to decompress the current
> page and skip values in-page. In the boundary case where nothing remains
> after the current page, this path still calls {{AdvanceNextPageHeader()}} for
> a page header that may not exist, and needlessly decompresses a page that is
> going to be fully skipped.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
