andygrove opened a new issue, #6934:
URL: https://github.com/apache/datafusion-comet/issues/6934
### What is the problem the feature request solves?
Since #6543, which fixed #6504 and #6546, the native Iceberg scan falls back
to Spark when a projected struct, list or map column has a nested field that
some schema in the table's history lacks or names differently. `CometScanRule`
finds these fields with `IcebergReflection.nestedFieldsAddedOrRenamed`. The
fallback was needed because iceberg-rust matched a data file's nested fields to
the table's by position, not by field id (apache/iceberg-rust#2617). A file
written before a nested field was added failed the scan with `Incorrect number
of arrays for StructArray fields`, and a nested field renamed since the file
was written read back as NULL.
The check has to use the table's schema history, because a `FileScanTask`
doesn't say which schema wrote its file. So once a table adds or renames a
nested field, every scan that projects that column falls back for good, even
after compaction has rewritten all the old files.
apache/iceberg-rust#3255 fixed the root cause. It merged on 2026-10-11 as
`050ddfda`, after the pin that #6851 moves to (`1f3bc34`).
`RecordBatchTransformer` now builds a plan for each file that matches nested
children by field id through structs, lists and maps. It fills absent optional
children with nulls and applies renames and reorders. The same bump picks up
apache/iceberg-rust#3384, which fixes #6790.
### Describe the potential solution
After #6851 merges:
- Move the `iceberg` and `iceberg-storage-opendal` pin to iceberg-rust
`main`. That was `b154603a` on 2026-10-13, 14 commits past `1f3bc34`.
iceberg-rust is still on arrow and parquet 59.2, and the lock change is only
the three iceberg-rust crates.
- Replace `nestedFieldsAddedOrRenamed` with a narrower check for the one
shape that still differs (below). That check would fall back when some schema
in the table's history has a projected struct, at any depth, that shares none
of its requested children with the current schema. If #6725 lands first,
"requested" means the children that the pruned read asks for.
- Turn the four fallback tests in `CometIcebergNativeSuite` ("schema
evolution - nested field ... falls back") into native reads that compare with
Spark. The self-join query reads the table twice, so it needs a check that
allows two native scans. Keep "dropped, moved, or promoted stays native", and
add fallback tests for the shapes the narrower check keeps.
- Narrow the limitation bullet in
`docs/source/user-guide/latest/iceberg.md`, and update the comments that cite
apache/iceberg-rust#2617 in `CometScanRule` and `CometIcebergNativeSuite`.
- Close #6790 with a test. The same range has apache/iceberg-rust#3384,
which applies a name mapping to nested fields. A table made with
`SparkActions.snapshotTable`, which sets the mapping, reads its struct, list
and map fields correctly alongside a top-level column at `b154603a`. At
`1f3bc34` they come back NULL.
What still differs at `b154603a` is a struct whose current children were all
added after a data file was written. For example, a file is written with `s
STRUCT<a: INT>` holding `{a: 1}` and NULL, and then `ALTER TABLE` adds `s.b`
and drops `s.a`. iceberg-rust's projection selects no leaf under `s` for that
file, and the result depends on what else the scan reads. With the #6504 check
turned off, on Spark 4.1 and Iceberg 1.11:
| Shape, for a row whose old file has a non-null value | Spark | Native |
| --- | --- | --- |
| top-level struct, read alone (`SELECT s`) | NULL | struct of NULLs |
| top-level struct, read with another column (`SELECT id, s`) | NULL | NULL |
| list of such structs, read alone | NULL | list of structs of NULLs |
| such a struct inside a struct whose other fields are in the file | struct
of NULLs | NULL |
So `SELECT count(*) ... WHERE s IS NULL` counts both of that file's rows on
Spark, and only the NULL one natively. apache/iceberg-rust#3378 (open) is about
the NULL when another column is read, and treats that NULL as the bug. Spark
also returns NULL for a top-level struct, though. It returns a struct of NULLs
only for a nested one, so an upstream fix that always returns a struct would
break the case that matches today. A map whose value struct has this shape
fails in Iceberg Java's own reader (`IllegalArgumentException: ... is not in
the store`), so there is nothing to compare it with.
### Additional context
Checked in a scratch build of #6851 with the pin moved to `b154603a`, on
Spark 4.1 (Iceberg 1.11):
- It compiles with no Comet changes, and the 875 `datafusion-comet` lib
tests pass.
- With the #6504 check in place, the 14 Iceberg suites that don't need MinIO
give 434 passed, 0 failed and 1 canceled, the same as at `1f3bc34`.
- With the check turned off, every query in the four fallback tests matches
Spark. Against `1f3bc34`, the queries on added fields fail and the renamed
fields read NULL.
Other changes in `1f3bc34..b154603a` to check during the bump:
- apache/iceberg-rust#3070 makes a case-insensitive name lookup fail on an
ambiguous name instead of picking one of the matches. iceberg-rust binds
equality-delete predicates with the task's `case_sensitive` flag, and Comet
sets that to `false` on every `FileScanTask`. From reading the code, an
equality delete on a table whose column names differ only by case could now
fail natively. Comet binds residual predicates case-sensitively, so those don't
change.
- apache/iceberg-rust#3303 changes `strip_metadata_from_schema` to normalize
list and map field names. Neither Comet nor iceberg-rust's reader and writer
call it.
- apache/iceberg-rust#3243 fixes truncate literals for decimal and binary in
predicate projection. Comet's native writer truncates whole arrays, which it
doesn't change.
- apache/iceberg-rust#3364 decodes short `BYTE_ARRAY` decimal row-group
bounds. Comet can't reach that until #6854 pushes decimal predicates down.
- apache/iceberg-rust#3334 rejects a boolean JSON literal for a non-boolean
type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
