[
https://issues.apache.org/jira/browse/IMPALA-11388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798769#comment-17798769
]
ASF subversion and git services commented on IMPALA-11388:
----------------------------------------------------------
Commit 012996a06b9978200bf061a9d26a3b248bcb605a in impala's branch
refs/heads/master from Gabor Kaszab
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=012996a06 ]
IMPALA-12597: Basic Equality delete read support for Iceberg tables
In general, applying equality deletes is similar to how position
deletes are applied to data files: using a LEFT ANTI JOIN where the
SCAN for the data rows is on the left side while the SCAN for the
delete rows is on the right side of the JOIN. The difference is the
virtual columns and the conjuncts being used.
For equality deletes the data sequence number of a delete file has to
be greater than the data sequence number of the data file being
investigated. This information is added as a virtual column to the
scans and a conjunct is created in the JOIN node to check the relation.
The equality delete fields from the delete files are checked agains the
respective columns of the data SCANS.
This patch makes it possible for Impala to read Iceberg tables with
basic equality delete files. The Iceberg spec gives great flexibility
for engines for writing equality deletes, however in practice Flink,
one of the engines that write EQ-deletes supports only a subset of the
use cases. This patch focuses on reading the EQ-deletes written by
Flink.
The restrictions are the following:
- All equality delete files in a table should have the same equality
field ID list.
- For partitioned Iceberg tables it is expected that the partition
values are also written into the equality delete files.
- Tables with equality deletes shouldn't have partition or schema
evolution.
- Floating point equality columns aren't supported.
- If a malformed equality delete file doesn't have some of the equality
field IDs then Parquet reader will fill those missing fields with
NULLs. As a side effect this will drop the rows from the result where
the corresponding data columns have a null value.
See IMPALA-11388 epic Jira for more details.
Testing:
- Checked if the existing functional_parquet.iceberg_v2_delete_equality
table can be read successfully.
- Added new test tables so that E2E tests can validate correctness.
Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad
Reviewed-on: http://gerrit.cloudera.org:8080/20753
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add support for equality-based deletes
> --------------------------------------
>
> Key: IMPALA-11388
> URL: https://issues.apache.org/jira/browse/IMPALA-11388
> Project: IMPALA
> Issue Type: Epic
> Components: Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Gabor Kaszab
> Priority: Major
> Labels: impala-iceberg
>
> Iceberg V2 adds support for row-level modifications.
> One way to implement this is via equality based delete files:
> https://iceberg.apache.org/spec/#equality-delete-files
> https://iceberg.apache.org/spec/#scan-planning
> We could implement this via doing ANTI HASH JOIN between data and delete
> files. Similarly to what we do for Hive full ACID tables:
> https://github.com/apache/impala/blob/f5fc08573352d0a1943296209791a4db17268086/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L1729-L1735
> The complexity comes when different delete files use different set of
> columns. In that case we will need multiple ANTI HASH JOINs on top of each
> other.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
