[
https://issues.apache.org/jira/browse/IMPALA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118012#comment-18118012
]
ASF subversion and git services commented on IMPALA-3841:
---------------------------------------------------------
Commit c3483c2946bee72b75960788bf3bbd13d2b9a7b7 in impala's branch
refs/heads/master from Aleksandr Efimov
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c3483c294 ]
IMPALA-15387: Reset next_levels_consumed_ for each row group
ComplexColumnReader::Reset() did not clear next_levels_consumed_, added
by IMPALA-3841. In a new row group the parent collection skipped its
nested child in the first NextLevels(), treated itself as empty and read
levels again, so a scalar sibling read its levels twice. From the first
page boundary on, that sibling returned the next row's value.
Testing:
- Added a two-level query over customer_nested_multiblock_multipage to
parquet-late-materialization-unique-db.test. Expected result computed
with pyarrow; a build without the fix returns 11908,9964,29547222652.
Change-Id: I7a1fe611da5dfebd99fa62f694a719675822935f
Assisted-by: Claude Opus 5 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24910
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Avoid materializing nested collections if top-level predicates already
> disqualify the row.
> ------------------------------------------------------------------------------------------
>
> Key: IMPALA-3841
> URL: https://issues.apache.org/jira/browse/IMPALA-3841
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.5.0, Impala 2.6.0
> Reporter: Alexander Behm
> Assignee: Xuebin Su
> Priority: Critical
> Labels: complextype, nested_types, parquet, performance
> Fix For: Impala 5.0.0
>
>
> Today, we fully materialize a row before evaluating the top-level conjuncts
> when scanning Parquet. This includes materializing nested collections. We
> should avoid materializing nested collections if top-level conjuncts already
> discard the row. Our recent move to column-wise materialization makes this
> improvement feasible (IMPALA-2736).
> To illustrate the problem, consider this query:
> {code}
> select * from customer c, c.orders o where c.id = 10
> {code}
> Even though we have a very selective predicate on the top-level customer, our
> scanner will still fully materialize all orders of all customers. The
> non-matches will be filtered, but we still pay the cost of materializing the
> orders.
> The proposed improvement is to avoid materializing the orders of
> non-qualifying customers.
> The improvement will several things:
> * Analyze and separate the top-level conjuncts into those that can be
> evaluated before materializing the nested collections and those that require
> nested collections to be materialized. In particular, we need to be careful
> with our auto-generated !empty() predicates on nested collections.
> * Add a new SkipValues() or similar interface to the Parquet column readers
> to advances the scanner without actually materializing values. If possible,
> we should skip entire blocks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
