Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24885 )
Change subject: IMPALA-15373: Fix FILE__POSITION for complex readers in later row groups ...................................................................... IMPALA-15373: Fix FILE__POSITION for complex readers in later row groups ReadFilePositionNonBatched() returns 'row_group_first_row_ + LastProcessedRow() + 1', but 'row_group_first_row_' was only assigned in BaseScalarColumnReader::Reset(). The file position slot is attached to (*column_readers)[0] whatever kind of reader that is, so when that is a complex reader (collection, struct or VARIANT) the offset stayed 0 and the second and later row groups reported row-group-relative positions. This also mis-applies Iceberg position deletes, which match on FILE__POSITION. A regression from IMPALA-11780, which made 'current_row_' row-group-relative and added the compensating term only on the scalar path. Pass the offset to ComplexColumnReader::Reset() as well, mirroring InitScalarColumns(). complex_readers_ holds every complex reader, including nested ones and the one CreateCountingReader() builds, so all of them are covered. LastProcessedRow() stays row-group-relative, so skip_row_id and the page index arithmetic are unaffected. ORC does not share the bug. Testing: - Two expected results in virtual-column-file-position-parquet.test encoded the old behaviour and are corrected to the true row indexes. - Added regression queries over all row groups of two different files. - The struct and VARIANT paths are fixed by the same change but have no regression test, because no checked-in file has both a top level struct or VARIANT column and more than one row group. - Ran test_scanners.py -k test_virtual_column_file_position_parquet. Change-Id: Ieb186c5727ae2a0cd53fb0c68ac540b8cd9fa8c7 Assisted-by: Claude Fable 5.1 (Claude Code) Reviewed-on: http://gerrit.cloudera.org:8080/24885 Reviewed-by: Zoltan Borok-Nagy <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-complex-column-reader.h M testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-parquet.test 4 files changed, 34 insertions(+), 10 deletions(-) Approvals: Zoltan Borok-Nagy: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/24885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ieb186c5727ae2a0cd53fb0c68ac540b8cd9fa8c7 Gerrit-Change-Number: 24885 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Xuebin Su <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
