[
https://issues.apache.org/jira/browse/IMPALA-12327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy resolved IMPALA-12327.
----------------------------------------
Fix Version/s: Impala 4.3.0
Resolution: Fixed
> Iceberg V2 operator wrong results in PARTITIONED mode
> -----------------------------------------------------
>
> Key: IMPALA-12327
> URL: https://issues.apache.org/jira/browse/IMPALA-12327
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
> Fix For: Impala 4.3.0
>
>
> The Iceberg delete node tries to do mini merge-joins between data records and
> delete records. This works in DISTRIBUTED mode, and most of the time in
> PARTITIONED mode as well. The Iceberg delete node had the wrong assumption
> that if the rows in a row batch belong to the same file, and come in
> ascending order, we don't need to update the IcebergDeleteState which tracks
> the state of the probing.
> But when PARTITIONED mode is used, we cannot rely on ascending row order, not
> even inside row batches, not even when the previous file path is the same as
> the current one.
> This is because files with multiple blocks can be processed by multiple hosts
> in parallel, then the rows are getting hash-exchanged based on their file
> paths. Then the exchange-receiver at the LHS coalesces the row batches from
> multiple senders, hence the row IDs getting unordered.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
