Neuw84 opened a new issue, #18618: URL: https://github.com/apache/iceberg/issues/18618
### Apache Iceberg version 1.12.0 (iceberg-spark-runtime-4.1_2.13, Spark 4.1.3) ### Query engine Spark (also affects any engine that loads equality deletes through `BaseDeleteLoader` with the generic Parquet reader) ### Please describe the bug 🐞 An equality delete file keyed on a `fixed(L)` column deletes almost nothing on read: N distinct keys behave like a single key. **Repro** 1. v2 table `(id long, f fixed(4))`, `write.delete.mode=merge-on-read`; insert 1000 rows with a distinct `f` per row (e.g. the 4 big-endian bytes of `id`). 2. Write one Parquet equality delete file on `f` with 143 distinct keys (`id` = 0, 7, 14, ..., 994) using `GenericFileWriterFactory` (or `SparkFileWriterFactory`), commit it with `newRowDelta().addDeletes(...)`. 3. `SELECT id FROM t`: 999 rows remain instead of 857. Same with the vectorized and the row-based Spark reader. Loading the delete file directly with `BaseDeleteLoader.loadEqualityDeletes(...)` returns a `StructLikeSet` of **size 1**, and `set.contains(<one of the real keys>)` is `false`. **Root cause** - `org.apache.iceberg.data.parquet.GenericParquetReaders$FixedReader.read(byte[] reuse)` fills and returns the caller's reuse buffer (`column.nextBinary().toByteBuffer().duplicate().get(reuse); return reuse;`), so every record's fixed value is the same `byte[]` instance, mutated as the file is iterated. - `BaseDeleteLoader.readEqDeletes` passes the records (through a flyweight `InternalRecordWrapper`) to `StructLikeSet.add`, which goes through `StructLikeWrapper.copyFor(s)` -> `set(s)`, which keeps only the `StructLike` reference and does not copy field values. All N delete records end up aliasing one array, so after the file is read they all compare equal to the last value and the set collapses to one element. `int`/`long`/`string` keys are not affected because their values are fresh immutable objects per row; `fixed` (and any reused-buffer binary value) is. **Possible fix** Return a fresh array in `FixedReader.read` (or do not reuse for delete loading), or deep-copy field values when building the equality-delete `StructLikeSet` in `readEqDeletes`. Found while building a native (vectorized) Iceberg scan for Spark that loads equality deletes through Iceberg's own loader; we now read the delete file ourselves and copy each key as it is read, which deletes the spec-correct rows. ### Willingness to contribute - [x] I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
