[
https://issues.apache.org/jira/browse/IMPALA-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105415#comment-18105415
]
ASF subversion and git services commented on IMPALA-15191:
----------------------------------------------------------
Commit 80c57fa9b63e0ac285b324bac54cc60af0055f3a in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=80c57fa9b ]
IMPALA-15191: Fix _row_id virtual column in USING clause
_row_id and _last_updated_sequence_number are Iceberg V3 syntactic-sugar
virtual columns that are expanded during analysis to COALESCE()
expressions by the mandatory IcebergVirtualColumnRewriteRule, so they
never reach the planner or backend.
That rewrite runs only once, between the first analyze() and the
re-analysis that follows expr rewriting. A JOIN ... USING (col) clause,
however, is re-generated into an on-clause on every analysis pass
(TableRef.analyzeJoin()), and reset() discards the previously built
on-clause when a USING clause is present. As a result the rewritten
COALESCE() predicate produced in the first pass was thrown away during
re-analysis and replaced by a raw equi-predicate referencing the
ICEBERG_ROW_ID virtual column. That column then reached the scan node,
where it hit DCHECK(false) in HdfsScanPlanNode.
This patch expands the syntactic-sugar virtual columns into their
COALESCE() equivalents at the point where the USING clause builds the
equi-join predicate, instead of relying on the one-shot expr rewrite.
Testing:
* New e2e regression tests
Change-Id: Ia1aef8e4030dc7ea25f38ca78219cde217bc3d84
Assisted-by: Claude Opus 4.8 (1M context)
Reviewed-on: http://gerrit.cloudera.org:8080/24596
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Join using _row_id hits DCHECK
> ------------------------------
>
> Key: IMPALA-15191
> URL: https://issues.apache.org/jira/browse/IMPALA-15191
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Michael Smith
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: iceberg
>
> With the following setup
> {code:java}
> create table fk (_row_id int primary key) stored as kudu;
> insert into fk values (1);
> create table fi (i int) stored as iceberg tblproperties('format-version'='3');
> insert into fi values (10), (11);{code}
> running
> {code:java}
> select i from fi join fk using (_row_id);{code}
> triggers the DCHECK at
> [https://github.com/apache/impala/blob/master/be/src/exec/hdfs-scan-node-base.cc#L1122.]
> If I continue on {{_row_id}} - which I think is wrong because
> {{{}_row_id{}}}'s not supposed to be sent to the backend - the results are
> wrong.
> I've run into the same failure in a more complicated scenario with {{{}on
> alias1._row_id = alias2._row_id{}}}, but I haven't figured out a more
> isolated reproduction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
