[
https://issues.apache.org/jira/browse/IMPALA-11780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118110#comment-18118110
]
ASF subversion and git services commented on IMPALA-11780:
----------------------------------------------------------
Commit a45095e75da4437cb1bfeb098149b2fa1e6249af in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a45095e75 ]
IMPALA-15373: Fix FILE__POSITION for complex readers in later row groups
ReadFilePositionNonBatched() returns 'row_group_first_row_ +
LastProcessedRow() + 1', but 'row_group_first_row_' was only assigned in
BaseScalarColumnReader::Reset(). The file position slot is attached to
(*column_readers)[0] whatever kind of reader that is, so when that is a
complex reader (collection, struct or VARIANT) the offset stayed 0 and
the second and later row groups reported row-group-relative positions.
This also mis-applies Iceberg position deletes, which match on
FILE__POSITION.
A regression from IMPALA-11780, which made 'current_row_'
row-group-relative and added the compensating term only on the scalar
path.
Pass the offset to ComplexColumnReader::Reset() as well, mirroring
InitScalarColumns(). complex_readers_ holds every complex reader,
including nested ones and the one CreateCountingReader() builds, so all
of them are covered. LastProcessedRow() stays row-group-relative, so
skip_row_id and the page index arithmetic are unaffected. ORC does not
share the bug.
Testing:
- Two expected results in virtual-column-file-position-parquet.test
encoded the old behaviour and are corrected to the true row indexes.
- Added regression queries over all row groups of two different files.
- The struct and VARIANT paths are fixed by the same change but have no
regression test, because no checked-in file has both a top level
struct or VARIANT column and more than one row group.
- Ran test_scanners.py -k test_virtual_column_file_position_parquet.
Change-Id: Ieb186c5727ae2a0cd53fb0c68ac540b8cd9fa8c7
Assisted-by: Claude Fable 5.1 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24885
Reviewed-by: Zoltan Borok-Nagy <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Wrong FILE__POSITION values for multi row group Parquet files when page
> filtering is used
> -----------------------------------------------------------------------------------------
>
> Key: IMPALA-11780
> URL: https://issues.apache.org/jira/browse/IMPALA-11780
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.2.0
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
> Fix For: Impala 4.3.0
>
>
> Impala generates wrong values for the FILE__POSITION column when the Parquet
> file contains multiple row groups and page filtering is being used.
> We are using the value of 'current_row_' to populate the file position slot:
> https://github.com/apache/impala/blob/8dbc8b733d3712035f8615f477f74c66ec9d63ea/be/src/exec/parquet/parquet-column-readers.cc#L906
> Problem is that 'current_row_' denotes the index of the row within the row
> group and not withing the file. We cannot change 'current_row_' as page
> filtering depends on its value, and the page index also uses the row
> group-based indexes of the rows, not the file indexes.
> The value of FILE__POSITION is critical for Iceberg V2 tables as position
> delete files store file positions of the deleted rows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
