karubian opened a new issue, #19035:
URL: https://github.com/apache/iceberg/issues/19035

   ### Apache Iceberg version
   
   1.12.0 (latest release)
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug 🐞
   
   `rewrite_position_delete_files` can drop position delete entries whose data 
file is still live, when a manifest partition summary wrongly excludes that 
data file's partition. The rewrite then removes the input delete file, so the 
deleted rows reappear for every reader that plans the data file, such as an 
unfiltered scan.
   
   Such a summary violates the spec, since its bounds must cover the manifest's 
partition values, [Manifest 
Lists](https://iceberg.apache.org/spec/#manifest-lists), but Iceberg Java wrote 
them before 1.11.0. In #15128, a manifest filter or merge that copies entries 
on a table with a binary partition field records bounds from a byte array the 
manifest reader reuses: 
[PartitionData](https://github.com/apache/iceberg/blob/apache-iceberg-1.10.0/core/src/main/java/org/apache/iceberg/PartitionData.java#L137-L139),
 
[filter](https://github.com/apache/iceberg/blob/apache-iceberg-1.10.0/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L486-L536),
 
[merge](https://github.com/apache/iceberg/blob/apache-iceberg-1.10.0/core/src/main/java/org/apache/iceberg/ManifestMergeManager.java#L171-L188).
 #15127 fixed new writes in 1.11.0, but later snapshots keep the wrong summary 
as long as they reuse the manifest, so a 1.12.0 rewrite still drops the deletes.
   
   On Spark 3.5.3 with a Hadoop catalog, the first five statements on Iceberg 
1.10.0, then the Spark session stopped and started again with the 1.12.0 
runtime on the same warehouse:
   
   ```sql
   -- Iceberg 1.10.0
   CREATE TABLE t (id bigint NOT NULL, data binary) USING iceberg PARTITIONED 
BY (data)
     TBLPROPERTIES ('format-version' = '2', 'write.delete.mode' = 
'merge-on-read');
   INSERT INTO t VALUES (1, X'e3bcd1'), (2, X'e3bcd1'), (3, X'bcd1');
   DELETE FROM t WHERE data = X'bcd1';  -- metadata delete; the manifest 
summary becomes [bcd1, bcd1d1]
   DELETE FROM t WHERE id = 1;          -- one position delete for the e3bcd1 
data file
   SELECT * FROM t;                     -- (2, X'e3bcd1')
   
   -- Iceberg 1.12.0, same warehouse
   CALL system.rewrite_position_delete_files(table => 't', options => 
map('rewrite-all', 'true'));
   -- rewritten_delete_files_count 1, added_delete_files_count 0
   SELECT * FROM t;                     -- (1, X'e3bcd1'), (2, X'e3bcd1')
   ```
   
   Row 1 is back and the table has no delete file left. `rewrite-all` makes the 
planner pick the single small delete file. A filtered read, `WHERE data = 
X'e3bcd1'`, returns nothing before and after the rewrite, because the wrong 
summary still prunes the manifest. Running every statement on 1.10.0 gives the 
same result. Running every statement on 1.12.0 writes a right summary and the 
rewrite keeps the delete.
   
   The runner keeps only the position delete entries whose `file_path` appears 
in a `data_files` scan filtered to the file group's partition: 
[SparkRewritePositionDeleteRunner](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkRewritePositionDeleteRunner.java#L113-L149).
 That scan skips manifests by their partition summaries, 
[BaseFilesTable](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/core/src/main/java/org/apache/iceberg/BaseFilesTable.java#L59-L87)
 and 
[ManifestEvaluator](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java#L245-L265),
 as [Scan Planning](https://iceberg.apache.org/spec/#scan-planning) allows. A 
live data file in a skipped manifest looks gone, so its entries are dropped as 
dangling, although the spec deletes a row when "there is an entry in a position 
delete file for the row's file and posit
 ion": [Position Delete 
Files](https://iceberg.apache.org/spec/#position-delete-files). Only file 
groups whose data file sits in a skipped manifest are affected. 
`rewrite_manifests` repairs filtered reads when it rewrites the manifests, 
since new manifests get summaries from their entries, but it leaves a single 
data manifest as it is, as in this table, and a delete the rewrite has dropped 
is missing from every later snapshot; rolling back to an earlier snapshot, or 
adding the old delete file back while it still exists, restores it. 
`RemoveDanglingDeleteFilesAction` and `RemoveDanglingDeletesSparkAction` plan 
unfiltered scans, 
[core](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/core/src/main/java/org/apache/iceberg/actions/RemoveDanglingDeleteFilesAction.java#L172-L173)
 and 
[Spark](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RemoveDanglingDeletesSparkAction.java#L89-L90),
 and keep these d
 eletes. The runner on main is the same for Spark 3.5, 4.0, 4.1 and 4.2, and 
the same filter decides which entries survive when it converts v2 position 
deletes to deletion vectors on a table upgraded to v3.
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to