Hello Xuebin Su, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24885
to look at the new patch set (#2).
Change subject: IMPALA-15373: Fix FILE__POSITION for complex readers in later
row groups
......................................................................
IMPALA-15373: Fix FILE__POSITION for complex readers in later row groups
ReadFilePositionNonBatched() returns 'row_group_first_row_ +
LastProcessedRow() + 1', but 'row_group_first_row_' was only assigned in
BaseScalarColumnReader::Reset(). The file position slot is attached to
(*column_readers)[0] whatever kind of reader that is, so when that is a
complex reader (collection, struct or VARIANT) the offset stayed 0 and
the second and later row groups reported row-group-relative positions.
This also mis-applies Iceberg position deletes, which match on
FILE__POSITION.
A regression from IMPALA-11780, which made 'current_row_'
row-group-relative and added the compensating term only on the scalar
path.
Pass the offset to ComplexColumnReader::Reset() as well, mirroring
InitScalarColumns(). complex_readers_ holds every complex reader,
including nested ones and the one CreateCountingReader() builds, so all
of them are covered. LastProcessedRow() stays row-group-relative, so
skip_row_id and the page index arithmetic are unaffected. ORC does not
share the bug.
Testing:
- Two expected results in virtual-column-file-position-parquet.test
encoded the old behaviour and are corrected to the true row indexes.
- Added regression queries over all row groups of two different files.
- The struct and VARIANT paths are fixed by the same change but have no
regression test, because no checked-in file has both a top level
struct or VARIANT column and more than one row group.
- Ran test_scanners.py -k test_virtual_column_file_position_parquet.
Change-Id: Ieb186c5727ae2a0cd53fb0c68ac540b8cd9fa8c7
Assisted-by: Claude Fable 5.1 (Claude Code)
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.h
M be/src/exec/parquet/parquet-complex-column-reader.h
M
testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-parquet.test
4 files changed, 34 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/24885/2
--
To view, visit http://gerrit.cloudera.org:8080/24885
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ieb186c5727ae2a0cd53fb0c68ac540b8cd9fa8c7
Gerrit-Change-Number: 24885
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Xuebin Su <[email protected]>
