[
https://issues.apache.org/jira/browse/IMPALA-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YifanZhang updated IMPALA-15157:
--------------------------------
Description:
When late materialization / row-level skipping skips a set of rows that ends
exactly on a Parquet page boundary,
{{BaseScalarColumnReader::SkipRowsInternal()}} exits the page-skipping loop and
calls {{{}ReadCurrentDataPage(){}}}, decompressing that final page just to skip
all of its values. The whole page could instead be discarded without
decompression.
*Root cause*
The whole-page-skip loop discarded pages only while the number of rows to skip
was _strictly greater_ than the values buffered in the current page:
{code:java}
while (num_rows > num_buffered_values_) {
...
if (!col_chunk_reader_.SkipPageData().ok() || !AdvanceNextPageHeader()) {
return false;
}
} {code}
When {{num_rows == num_buffered_values_}} (the skip ends exactly at a page
boundary), the loop exited and the code proceeded to decompress the current
page and skip values in-page. In the boundary case where nothing remains after
the current page, this path still calls {{AdvanceNextPageHeader()}} for a page
header that may not exist, and needlessly decompresses a page that is going to
be fully skipped.
was:
{{BaseScalarColumnReader::SkipRowsInternal()}} (in
{{{}be/src/exec/parquet/parquet-column-readers.cc{}}}) can attempt to read a
page header that does not exist when late materialization / row-level skipping
needs to skip a set of rows that ends exactly on a Parquet page boundary — for
example, when an entire row group is filtered out by row-level predicates and
the rows to skip run to the end of the column chunk.
*Root cause*
The whole-page-skip loop discarded pages only while the number of rows to skip
was _strictly greater_ than the values buffered in the current page:
{code:java}
while (num_rows > num_buffered_values_) {
...
if (!col_chunk_reader_.SkipPageData().ok() || !AdvanceNextPageHeader()) {
return false;
}
} {code}
When {{num_rows == num_buffered_values_}} (the skip ends exactly at a page
boundary), the loop exited and the code proceeded to decompress the current
page and skip values in-page. In the boundary case where nothing remains after
the current page, this path still calls {{AdvanceNextPageHeader()}} for a page
header that may not exist, and needlessly decompresses a page that is going to
be fully skipped.
> Parquet scanner unnecessarily decompresses a fully-skipped page at a page
> boundary
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-15157
> URL: https://issues.apache.org/jira/browse/IMPALA-15157
> Project: IMPALA
> Issue Type: Improvement
> Reporter: YifanZhang
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> When late materialization / row-level skipping skips a set of rows that ends
> exactly on a Parquet page boundary,
> {{BaseScalarColumnReader::SkipRowsInternal()}} exits the page-skipping loop
> and calls {{{}ReadCurrentDataPage(){}}}, decompressing that final page just
> to skip all of its values. The whole page could instead be discarded without
> decompression.
> *Root cause*
> The whole-page-skip loop discarded pages only while the number of rows to
> skip was _strictly greater_ than the values buffered in the current page:
> {code:java}
> while (num_rows > num_buffered_values_) {
> ...
> if (!col_chunk_reader_.SkipPageData().ok() || !AdvanceNextPageHeader()) {
> return false;
> }
> } {code}
> When {{num_rows == num_buffered_values_}} (the skip ends exactly at a page
> boundary), the loop exited and the code proceeded to decompress the current
> page and skip values in-page. In the boundary case where nothing remains
> after the current page, this path still calls {{AdvanceNextPageHeader()}} for
> a page header that may not exist, and needlessly decompresses a page that is
> going to be fully skipped.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
