Zoltán Borók-Nagy created IMPALA-11484:
------------------------------------------
Summary: Create SCAN plan for Iceberg V2 position delete tables
Key: IMPALA-11484
URL: https://issues.apache.org/jira/browse/IMPALA-11484
Project: IMPALA
Issue Type: New Feature
Components: Frontend
Reporter: Zoltán Borók-Nagy
Iceberg position delete files store the full URIs and and file positions of
rows that are deleted. Therefore we can do an ANTI HASH JOIN between data files
and delete files to retrieve only the active rows.
For the data file rows we need to get the virtual columns INPUT_FILE_NAME and
FILE_POSITION, while in the delete files we need to retrieve the columns
'file_path' and 'pos': https://iceberg.apache.org/spec/#position-delete-files
Since the data files are in table schema, and the delete files are in a
different schema, we need to create a virtual table for the delete files with
the corresponding schema.
Iceberg tells us which delete files must be applied to which data files, i.e.
if a data file doesn't have a corresponding delete file, the content can be
just UNION'ed with the output of the ANTI HASH JOIN.
See more information in the design doc:
https://docs.google.com/document/d/1WF_UOanQ61RUuQlM4LaiRWI0YXpPKZ2VEJ8gyJdDyoY/edit#heading=h.5gc49pcc2543
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
