[
https://issues.apache.org/jira/browse/IMPALA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118557#comment-18118557
]
ASF subversion and git services commented on IMPALA-3841:
---------------------------------------------------------
Commit 820e108c83ebc653a9b21fb54039586768f3b74e in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=820e108c8 ]
IMPALA-15374: Roll back current_row_ when undoing a level read ahead
Collection column readers drive their children through the
non-batched interface, so a child always has one level read ahead.
When the scanner skips rows for such a child,
BaseScalarColumnReader::SkipRows() undoes that read ahead but did not
roll back 'current_row_', which NextLevels() had already incremented
for the level being un-read. SkipTopLevelRows() then counted from a
starting point one too high, so the counter drifted by one per
skipped range within a row group.
Which symptom this produces depends on the page index. Without it the
values stay aligned but LastProcessedRow() is too high, so a
collection reader that fills the file position slot reports positions
that are too large:
set parquet_late_materialization_threshold=1;
select file__position, int_array, id from complextypestbl
where id % 2 = 0;
returned positions 1, 4, 7, 0 instead of 1, 3, 5, 0; 7 is past the
end of the seven row file. With the page index the rows to skip are
computed as 'skip_row_id - LastProcessedRow()', so the reader skips
one row too few and falls behind the columns it is read with,
silently pairing rows with the wrong collection value:
set batch_size=4;
select c_custkey, count(o.o_orderkey), min(o.o_orderkey)
from customer_nested_multiblock_multipage c left join c.c_orders o
where c_custkey > 280 and c_custkey % 9 = 2 group by c_custkey;
returned customer 289's orders for customer 290, and none for
customer 299.
Roll the counter back under the same condition NextLevels() uses to
advance it, before 'rep_level_' is invalidated. Levels read ahead at
the end of a row group never advanced it and have 'rep_level_' ==
ROW_GROUP_END, so they are left alone.
Broken since IMPALA-3841, which added the read ahead undo along with
late materialization for collections. Only readers driven through the
non-batched interface are affected: top level scalar readers never
read a level ahead, and struct and VARIANT readers are still
excluded.
Testing:
- Added QueryTest/parquet-late-materialization-collections.test with
a query per symptom, run with late materialization and the page
index on and off. Both fail without the fix.
- Ran test_parquet_late_materialization.py, test_nested_types.py and
test_parquet_stats.py.
- customer_nested_multiblock_multipage is now loaded during dataload.
Change-Id: Icaf0aab770d87505513b0168f52f96480eda3202
Assisted-by: Claude Fable 5.1 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24886
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Avoid materializing nested collections if top-level predicates already
> disqualify the row.
> ------------------------------------------------------------------------------------------
>
> Key: IMPALA-3841
> URL: https://issues.apache.org/jira/browse/IMPALA-3841
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.5.0, Impala 2.6.0
> Reporter: Alexander Behm
> Assignee: Xuebin Su
> Priority: Critical
> Labels: complextype, nested_types, parquet, performance
> Fix For: Impala 5.0.0
>
>
> Today, we fully materialize a row before evaluating the top-level conjuncts
> when scanning Parquet. This includes materializing nested collections. We
> should avoid materializing nested collections if top-level conjuncts already
> discard the row. Our recent move to column-wise materialization makes this
> improvement feasible (IMPALA-2736).
> To illustrate the problem, consider this query:
> {code}
> select * from customer c, c.orders o where c.id = 10
> {code}
> Even though we have a very selective predicate on the top-level customer, our
> scanner will still fully materialize all orders of all customers. The
> non-matches will be filtered, but we still pay the cost of materializing the
> orders.
> The proposed improvement is to avoid materializing the orders of
> non-qualifying customers.
> The improvement will several things:
> * Analyze and separate the top-level conjuncts into those that can be
> evaluated before materializing the nested collections and those that require
> nested collections to be materialized. In particular, we need to be careful
> with our auto-generated !empty() predicates on nested collections.
> * Add a new SkipValues() or similar interface to the Parquet column readers
> to advances the scanner without actually materializing values. If possible,
> we should skip entire blocks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
