[
https://issues.apache.org/jira/browse/IMPALA-15387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118011#comment-18118011
]
ASF subversion and git services commented on IMPALA-15387:
----------------------------------------------------------
Commit c3483c2946bee72b75960788bf3bbd13d2b9a7b7 in impala's branch
refs/heads/master from Aleksandr Efimov
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c3483c294 ]
IMPALA-15387: Reset next_levels_consumed_ for each row group
ComplexColumnReader::Reset() did not clear next_levels_consumed_, added
by IMPALA-3841. In a new row group the parent collection skipped its
nested child in the first NextLevels(), treated itself as empty and read
levels again, so a scalar sibling read its levels twice. From the first
page boundary on, that sibling returned the next row's value.
Testing:
- Added a two-level query over customer_nested_multiblock_multipage to
parquet-late-materialization-unique-db.test. Expected result computed
with pyarrow; a build without the fix returns 11908,9964,29547222652.
Change-Id: I7a1fe611da5dfebd99fa62f694a719675822935f
Assisted-by: Claude Opus 5 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24910
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Parquet: two nested levels return the next row's values in later row groups
> ---------------------------------------------------------------------------
>
> Key: IMPALA-15387
> URL: https://issues.apache.org/jira/browse/IMPALA-15387
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Critical
> Labels: correctness
>
> Since IMPALA-3841, reading two nested levels can return a scalar field of the
> next top-level row, silently. It starts at the first page boundary of the
> second or a later row group. We hit it on production data.
> {code:sql}
> select count(*), count(o.o_orderkey), sum(o.o_orderkey)
> from customer_nested_multiblock_multipage c
> left join c.c_orders o
> left join o.o_lineitems l;
> -- expected (pyarrow): 11908, 11808, 34964524398
> -- actual: 11908, 9964, 29547222652
> {code}
> One level ({{left join c.c_orders o}}) is correct.
> Cause: ComplexColumnReader::Reset() does not clear next_levels_consumed_. The
> first NextLevels() of c_orders in a new row group skips o_lineitems, treats
> c_orders as empty and reads levels again, so o_orderkey reads its levels
> twice. At the next page boundary it drops one value and is one row ahead from
> then on.
> Fix: clear next_levels_consumed_ in Reset().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
