comphead opened a new issue, #7019:
URL: https://github.com/apache/datafusion-comet/issues/7019
### What is the problem the feature request solves?
With nested schema pruning (#6725), every task of a native Iceberg scan
reads the pruned scan schema, with the partition sources and equality-delete
keys the tasks need appended at its top level. When one of those fields is
nested inside a struct the query pruned away, appending it at the top level
could clash with a top-level column of the same name (`Invalid schema: multiple
fields for name region`). So `IcebergReflection.nativeTaskSchemas` gives up on
the pruned schema, and the scan reads its full schema instead. That has two
costs:
- The scan decodes every leaf of its projected nested columns, which is the
cost #6724 describes. For example, a table partitioned by `s.region` gets no
pruning for `SELECT s.a FROM t`.
- A `VERSION AS OF` read can fall back to Spark where `main` reads it
natively. The full schema of such a read can hold a nested field that an older
data file lacks, which the nested-evolution check rejects:
```sql
CREATE TABLE t (id INT, c INT, s STRUCT<k: STRING, a: INT>) USING iceberg
PARTITIONED BY (s.k);
INSERT INTO t VALUES (1, 100, named_struct('k', 'east', 'a', 10));
ALTER TABLE t ADD COLUMN s.b INT;
INSERT INTO t VALUES (2, 200, named_struct('k', 'east', 'a', 20, 'b', 99));
-- take snapshot S here
ALTER TABLE t DROP COLUMN c;
ALTER TABLE t DROP COLUMN s.b;
SELECT id, c, s.a FROM t VERSION AS OF S;
```
With #6725 this falls back with `s.b (added)`. `main` appends `s.k` at the
top level of the scan schema and reads it natively, matching Spark, as
@andygrove checked in
https://github.com/apache/datafusion-comet/pull/6725#discussion_r4277126004.
The case is in the `CometIcebergNativeSuite` test "schema evolution - nested
field added to a struct falls back".
### Describe the potential solution
Merge the missing field into its pruned parent struct instead of appending
it at the top level, so the example's pruned task schema holds `s{k, a}`. The
scan would keep its pruned read, and the nested-evolution check would walk a
schema without `s.b`. At Comet's current iceberg-rust pin, nested fields are
matched by position (apache/iceberg-rust#2617), so the merged field has to keep
its place among the struct's other fields. The batch adaptation after
iceberg-rust already drops nested fields the query does not read.
### Additional context
- #6934 would make the `VERSION AS OF` example native again through the full
schema, once iceberg-rust matches nested fields by id and the nested-evolution
check narrows. That read still decodes every leaf of `s`.
- An equality-delete key nested in a struct fails natively for another
reason (#6782), so this would help partition sources first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
