ebyhr opened a new pull request, #18408: URL: https://github.com/apache/iceberg/pull/18408
IncrementalFileCleanup can delete a deletion vector (DV) Puffin file that is still referenced by live DVs, which corrupts the table. A single Puffin file can hold the DVs of many data files, and each DV is its own manifest entry with the Puffin file's location. When one of those DVs is replaced, the old entry becomes DELETED and still points at the shared Puffin file. When the snapshot that wrote that entry expires, findFilesToDelete adds entry.file().location() to the delete set without checking whether live entries still reference the path. The other DVs in that file then fail to read. ReachableFileCleanup does this check, but IncrementalFileCleanup is the default for tables with only main. Reproduction (Trino, v3 table): ```sql CREATE TABLE t (id integer) WITH (format = 'PARQUET', format_version = 3); -- 10 inserts of 100 rows each -> 10 data files INSERT INTO t SELECT x FROM UNNEST(sequence(1, 100)) t(x); -- repeat for 101-200 through 901-1000 -- writes all 10 DVs into one Puffin file DELETE FROM t WHERE id % 2 = 0; -- replaces the DV of the first data file only; the other 9 stay in the shared file DELETE FROM t WHERE id % 5 = 0 AND id <= 100; INSERT INTO t VALUES 1001; ALTER TABLE t EXECUTE expire_snapshots(retention_threshold => '0s'); SELECT count(*) FROM t; ``` The SELECT fails with `ICEBERG_CANNOT_OPEN_SPLIT`. The cause is a `FileNotFoundException` for the `dv-<query id>-<uuid>.puffin` file that `expire_snapshots` just deleted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
