andygrove opened a new issue, #7050:
URL: https://github.com/apache/datafusion-comet/issues/7050
### Describe the bug
`docs/source/user-guide/latest/iceberg-writes.md` says this about float and
double fields nested under a nullable struct:
> On Iceberg 1.9+, manifest `value_counts` / `null_value_counts` for
float/double columns nested under a nullable struct count rows whose parent
struct is null (they come from the parquet footer), while iceberg-java's
writer-tracked counts do not. Both counts inflate by the same amount, so the
derived null ratios and `IS NULL` / `IS NOT NULL` pruning decisions are
unaffected.
The `IS NULL` half is wrong. When every present struct has a value for the
field, iceberg-java records `null_value_counts = 0` for it even though the
struct is NULL in some rows. Iceberg's planner then prunes the file for `WHERE
s.x IS NULL` and loses every NULL-struct row. That is an iceberg-java bug,
apache/iceberg#17561, with a fix in apache/iceberg#17560. The native writer's
footer counts include those rows, so its files are kept and the query returns
the right rows. `IS NOT NULL` decisions do agree.
### Steps to reproduce
With Comet disabled, on `main` at c230bf4e98:
```sql
CREATE TABLE t (id INT, s STRUCT<x: DOUBLE>) USING iceberg;
INSERT INTO t SELECT id, IF(id % 2 = 0, named_struct('x', CAST(id AS
DOUBLE)), NULL) FROM range(10);
SELECT count(*) FROM t WHERE s.x IS NULL;
```
| Iceberg | `null_value_counts` of `s.x` in each file | `count(*)` |
| --- | --- | --- |
| 1.5.2 (Spark 3.4.3) and 1.8.1 (Spark 3.5.9) | 1 | 5 |
| 1.11.0 (Spark 4.1.3) | 0 | 0 |
`FLOAT` behaves like `DOUBLE`. With an `INT` field, every version records 1
and returns 5.
### Expected behavior
The bullet says that the native counts are the footer's, that iceberg-java's
leave out the NULL-struct rows because of apache/iceberg#17561, and that
iceberg-java's counts can prune files for `IS NULL` that the native counts
keep. That is also why #6659 should stay closed: matching iceberg-java's counts
would copy the wrong result.
### Additional context
Found in a nested-type audit of the native writers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
