Neuw84 opened a new issue, #18620:
URL: https://github.com/apache/iceberg/issues/18620
### Apache Iceberg version
1.12.0 (iceberg-spark-runtime-4.1_2.13, Spark 4.1.3)
### Query engine
Spark
### Please describe the bug 🐞
An equality delete keyed on a field nested inside a struct removes **every**
row when read through Iceberg's Spark reader, while Iceberg core's generic
reader applies it correctly.
**Repro**
1. `CREATE TABLE t (id int, st struct<k int, s string>, payload bigint)
USING iceberg TBLPROPERTIES ('format-version'='2',
'write.delete.mode'='merge-on-read')`
2. Insert 6000 rows with `st = named_struct('k', id % 50, 's', concat('s',
id % 7))`; every struct and key is non-null.
3. With the Java API write one Parquet equality delete file: delete schema
`table.schema().select("st.k")`, `equalityFieldIds = [field id of st.k]`, two
records (built with `GenericRecord.create(deleteSchema)` and a populated child
`st`) whose `st.k` is 5 and 10; commit with `newRowDelta().addDeletes(...)`.
4. Expected: 5760 rows remain (the rows with `st.k` not in {5, 10}).
- `IcebergGenerics.read(table).build()` returns 5760 rows -- correct.
- `spark.table("t")` / `SELECT id FROM t` returns 0 rows -- wrong.
The delete file is well formed: equality ids `[4]` match the data's `st.k`
field id 4, the delete schema is `{2: st: optional struct<4: k: optional
int>}`, record count 2, and the generic reader agrees with the independently
computed expected set.
**Where to look**
The Spark reader's delete-apply path (`DeleteFilter.applyEqDeletes` with
`StructProjection` over an `InternalRowWrapper`-wrapped Spark row) for an
equality key that is a field inside a struct: the projected nested key
apparently compares equal for every row, so every row matches the delete
predicate. Core's `GenericDeleteFilter` path is not affected.
Found while building a native Iceberg scan for Spark that applies the same
delete correctly (5760 rows) by reading the nested key through its ancestor
struct.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
