[
https://issues.apache.org/jira/browse/IMPALA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy resolved IMPALA-15433.
----------------------------------------
Resolution: Fixed
> [DOCS] Document Iceberg V3 table support (row lineage, deletion vectors, DML)
> -----------------------------------------------------------------------------
>
> Key: IMPALA-15433
> URL: https://issues.apache.org/jira/browse/IMPALA-15433
> Project: IMPALA
> Issue Type: Documentation
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
> Fix For: Impala 5.0.0
>
>
> Impala 5.0 supports Iceberg format-version 3 tables, but docs/ never mentions
> V3: a grep for "V3", "row lineage", "deletion vector" and _row_id under docs/
> finds nothing. Add a concept (suggested id iceberg_v3) in
> docs/topics/impala_iceberg.xml next to iceberg_v2, covering:
> - Creating V3 tables: CREATE TABLE ... STORED BY ICEBERG
> TBLPROPERTIES('format-version'='3'), upgrading an existing table with ALTER
> TABLE ... SET TBLPROPERTIES('format-version'='3'), and ALTER TABLE ...
> CONVERT TO ICEBERG TBLPROPERTIES('format-version'='3') (IMPALA-15071;
> accepted values are 1, 2 and 3). CREATE TABLE LIKE keeps the
> source table's format version (IMPALA-14836).
> - Row lineage (IMPALA-14592, IMPALA-14825): the virtual columns _row_id and
> _last_updated_sequence_number (BIGINT), which exist only on V3 tables, are
> not returned by SELECT *, need qualifying when several V3 tables are in
> scope, and lose to a real column of the same name. Underneath are the hidden
> columns _file_row_id and
> _file_last_updated_sequence_number plus the virtual column
> ICEBERG__FIRST__ROW__ID; the definitions are COALESCE(_file_row_id,
> ICEBERG__FIRST__ROW__ID + FILE__POSITION) and
> COALESCE(_file_last_updated_sequence_number,
> ICEBERG__DATA__SEQUENCE__NUMBER). The hidden columns can be selected but not
> altered, dropped, inserted into or reused
> as column names, and COMPUTE STATS skips them.
> - How the values change: INSERT assigns new row ids; UPDATE and MERGE keep
> _row_id and assign a new _last_updated_sequence_number; OPTIMIZE preserves
> both.
> - DML on V3 (IMPALA-14588 DELETE, IMPALA-14913 UPDATE, IMPALA-14937 MERGE):
> row-level changes write deletion vectors in Puffin files instead of Parquet
> position delete files, at most one per data file. They show up in the files
> metadata table as content=1, file_format='PUFFIN'. The Parquet rule still
> applies: write.delete.format.default
> must be PARQUET.
> - The upgrade restriction: on a V3 table that still has V2 position delete
> files, DELETE, UPDATE and MERGE fail with "... is not allowed on Iceberg
> format version 3 table '<t>' as it has existing version 2 position delete
> file(s) ... Run 'OPTIMIZE TABLE <t>'".
> Text elsewhere that belongs to the same work:
> - impala_iceberg.xml: iceberg_delete (~line 600) and iceberg_update (~line
> 699) say "against Iceberg V2 tables", and the iceberg_update limitation "Only
> writes position delete files" doesn't hold for V3.
> - impala_iceberg.xml iceberg_v2 (~line 258) still tells the reader to set
> 'format-version'='2' to get a V2 table, which has been the default since the
> Iceberg 1.5 upgrade.
> - impala_dml.xml:88: "For Iceberg V2 tables, Impala supports the DELETE ...
> UPDATE ... MERGE ..." — add V3.
> - impala_langref_unsupported.xml:314: "Impala does not provide virtual
> columns." Wrong since 4.2 (INPUT__FILE__NAME), and more so now. Consider
> adding a list of the Iceberg virtual columns: INPUT__FILE__NAME,
> FILE__POSITION, PARTITION__SPEC__ID, ICEBERG__PARTITION__SERIALIZED,
> ICEBERG__DATA__SEQUENCE__NUMBER, ICEBERG__FIRST__ROW__ID,
> plus _row_id and _last_updated_sequence_number on V3.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
