Zoltán Borók-Nagy created IMPALA-15433:
------------------------------------------
Summary: [DOCS] Document Iceberg V3 table support (row lineage,
deletion vectors, DML)
Key: IMPALA-15433
URL: https://issues.apache.org/jira/browse/IMPALA-15433
Project: IMPALA
Issue Type: Documentation
Reporter: Zoltán Borók-Nagy
Assignee: Zoltán Borók-Nagy
Impala 5.0 supports Iceberg format-version 3 tables, but docs/ never mentions
V3: a grep for "V3", "row lineage", "deletion vector" and _row_id under docs/
finds nothing. Add a concept (suggested id iceberg_v3) in
docs/topics/impala_iceberg.xml next to iceberg_v2, covering:
- Creating V3 tables: CREATE TABLE ... STORED BY ICEBERG
TBLPROPERTIES('format-version'='3'), upgrading an existing table with ALTER
TABLE ... SET TBLPROPERTIES('format-version'='3'), and ALTER TABLE ... CONVERT
TO ICEBERG TBLPROPERTIES('format-version'='3') (IMPALA-15071; accepted values
are 1, 2 and 3). CREATE TABLE LIKE keeps the
source table's format version (IMPALA-14836).
- Row lineage (IMPALA-14592, IMPALA-14825): the virtual columns _row_id and
_last_updated_sequence_number (BIGINT), which exist only on V3 tables, are not
returned by SELECT *, need qualifying when several V3 tables are in scope, and
lose to a real column of the same name. Underneath are the hidden columns
_file_row_id and
_file_last_updated_sequence_number plus the virtual column
ICEBERG__FIRST__ROW__ID; the definitions are COALESCE(_file_row_id,
ICEBERG__FIRST__ROW__ID + FILE__POSITION) and
COALESCE(_file_last_updated_sequence_number, ICEBERG__DATA__SEQUENCE__NUMBER).
The hidden columns can be selected but not altered, dropped, inserted into or
reused
as column names, and COMPUTE STATS skips them.
- How the values change: INSERT assigns new row ids; UPDATE and MERGE keep
_row_id and assign a new _last_updated_sequence_number; OPTIMIZE preserves both.
- DML on V3 (IMPALA-14588 DELETE, IMPALA-14913 UPDATE, IMPALA-14937 MERGE):
row-level changes write deletion vectors in Puffin files instead of Parquet
position delete files, at most one per data file. They show up in the files
metadata table as content=1, file_format='PUFFIN'. The Parquet rule still
applies: write.delete.format.default
must be PARQUET.
- The upgrade restriction: on a V3 table that still has V2 position delete
files, DELETE, UPDATE and MERGE fail with "... is not allowed on Iceberg format
version 3 table '<t>' as it has existing version 2 position delete file(s) ...
Run 'OPTIMIZE TABLE <t>'".
Text elsewhere that belongs to the same work:
- impala_iceberg.xml: iceberg_delete (~line 600) and iceberg_update (~line 699)
say "against Iceberg V2 tables", and the iceberg_update limitation "Only writes
position delete files" doesn't hold for V3.
- impala_iceberg.xml iceberg_v2 (~line 258) still tells the reader to set
'format-version'='2' to get a V2 table, which has been the default since the
Iceberg 1.5 upgrade.
- impala_dml.xml:88: "For Iceberg V2 tables, Impala supports the DELETE ...
UPDATE ... MERGE ..." — add V3.
- impala_langref_unsupported.xml:314: "Impala does not provide virtual
columns." Wrong since 4.2 (INPUT__FILE__NAME), and more so now. Consider adding
a list of the Iceberg virtual columns: INPUT__FILE__NAME, FILE__POSITION,
PARTITION__SPEC__ID, ICEBERG__PARTITION__SERIALIZED,
ICEBERG__DATA__SEQUENCE__NUMBER, ICEBERG__FIRST__ROW__ID,
plus _row_id and _last_updated_sequence_number on V3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
