yadavay-amzn opened a new issue, #17203:
URL: https://github.com/apache/iceberg/issues/17203
### Apache Iceberg version
main (development)
### Query engine
Spark
### Please describe the bug 🐞
`RemoveDanglingDeletesSparkAction` cleans up dangling deletion vectors (v3,
PUFFIN) but does **not** remove dangling **v2 file-scoped position-delete
files** (PARQUET). A file-scoped position delete whose referenced data file has
been removed, but whose sequence number is `>=` the partition's minimum data
sequence number, is never cleaned up and lingers in table metadata indefinitely.
**Root cause (on `main`):**
- `findDanglingDvs()` performs the correct referenced-data-file left-join,
but restricts it to PUFFIN:
```java
// spark/.../actions/RemoveDanglingDeletesSparkAction.java
Dataset<Row> dvs =
loadMetadataTable(table, MetadataTableType.DELETE_FILES)
.where(col("file_format").equalTo(FileFormat.PUFFIN.name())); //
PUFFIN only
```
v2 PARQUET position deletes are therefore excluded from the
referenced-file check.
- `findDanglingDeletes()` only removes position deletes whose sequence
number is *below* the partition minimum:
```java
col("data_file.content").equalTo("1")
.and(col("sequence_number").$less(col("min_data_sequence_number")))
```
A file-scoped position delete whose referenced data file is gone, but
whose `sequence_number >= min_data_sequence_number` (because another live data
file in the same partition keeps the minimum low), is not caught by either
method.
**Result:** such position-delete files are unremovable via
`remove_dangling_deletes`, bloating table metadata and adding scan-planning
overhead.
**Steps to reproduce:**
1. Format-version-2, partitioned table. Append two data files `A` and `B` to
the same partition (sequence 1).
2. Add a file-scoped position-delete file referencing only `A` (sequence 2).
3. Remove/overwrite `A` while `B` remains (sequence 3). The partition
minimum data sequence number stays 1 (from `B`).
4. Run `remove_dangling_deletes`. Observe: the position-delete file
referencing the now-absent `A` is **not** removed (its sequence 2 >= partition
min 1, and it is not PUFFIN).
**Expected behavior:** a file-scoped position-delete file whose only
referenced data file is absent should be detected as dangling and removed, the
same way dangling DVs are.
**Proposed fix (I can contribute):** generalize the referenced-data-file
left-join used for DVs to also cover v2 file-scoped position deletes - deriving
the referenced data file from the delete file's `lower_bounds`/`upper_bounds`
for the `file_path` field when they are equal - while guarding so that only
single-data-file (file-scoped) deletes are flagged, never partition-scoped
deletes that may still cover live data. I would include a reproducer test.
### Willingness to contribute
- [X] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
