Aleksandr Efimov created IMPALA-15387:
-----------------------------------------
Summary: Parquet: two nested levels return the next row's values
in later row groups
Key: IMPALA-15387
URL: https://issues.apache.org/jira/browse/IMPALA-15387
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Aleksandr Efimov
Since IMPALA-3841, reading two nested levels can return a scalar field of
the next top-level row, silently. It starts at the first page boundary of
the second or a later row group. We hit it on production data.
{code:sql}
select count(*), count(o.o_orderkey), sum(o.o_orderkey)
from customer_nested_multiblock_multipage c
left join c.c_orders o
left join o.o_lineitems l;
-- expected (pyarrow): 11908, 11808, 34964524398
-- actual: 11908, 9964, 29547222652
{code}
One level ({{left join c.c_orders o}}) is correct.
Cause: ComplexColumnReader::Reset() does not clear next_levels_consumed_.
The first NextLevels() of c_orders in a new row group skips o_lineitems,
treats c_orders as empty and reads levels again, so o_orderkey reads its
levels twice. At the next page boundary it drops one value and is one row
ahead from then on.
Fix: clear next_levels_consumed_ in Reset().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
