mhatipog opened a new issue, #19064:
URL: https://github.com/apache/iceberg/issues/19064
### Apache Iceberg version
1.12.0 (also reproduces on 1.10.1 and 1.4.3)
### Query engine
Spark
### Please describe the bug 🐞
On ORC tables, `IS NULL` on a struct field misses rows where the parent
struct is null, in reads and in `DELETE`:
```sql
CREATE TABLE t (id INT, st STRUCT<x: INT>) USING iceberg
TBLPROPERTIES ('write.format.default' = 'orc');
INSERT INTO t VALUES (1, NULL), (2, named_struct('x', 5));
SELECT id FROM t WHERE st.x IS NULL; -- no rows, expected 1
DELETE FROM t WHERE st.x IS NULL; -- row 1 is not deleted
```
The same table in Parquet returns and deletes row 1.
ORC only stores a struct field where all of its ancestors are present, and
`hasNull` reflects the field's own nulls. Two places read that as "no nulls":
- `OrcMetrics` records `null_value_count = 0` when `hasNull()` is false, so
`IS NULL` prunes the file.
- `ExpressionToSearchArgument` pushes `isNull` on the nested column to ORC,
which skips the row group based on the same statistics. Spark fixes this in its
own ORC reader in SPARK-60362.
Trino reading the same table returns the same wrong result, since it prunes
on the manifest counts.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]