andygrove opened a new issue, #6790:
URL: https://github.com/apache/datafusion-comet/issues/6790
### Describe the bug
On an Iceberg table whose data files have no Iceberg field ids and are read
through a name mapping (`schema.name-mapping.default`, which the `migrate`,
`snapshot` and `add_files` procedures set), the native Iceberg scan returns
NULL for nested fields whenever the query also reads a top-level primitive
column. That covers struct fields, list element fields and map value fields.
Spark returns the values, and nothing fails, so the wrong results are silent.
iceberg-rust applies the mapping in `apply_name_mapping_to_arrow_schema`
(`crates/iceberg/src/arrow/reader/projection.rs`), which assigns field ids to
the file's top-level Arrow fields only. Nested fields get no id, so the
field-id projection never selects their leaves. When the query also reads a
top-level primitive, the projection reads only that column, and
`RecordBatchTransformer` fills the struct, list or map column with NULL as
though the file didn't have it. When the query reads only nested fields, no
leaf matches at all, the reader falls back to reading every column, and the
values come out right. Iceberg Java's `ApplyNameMapping` visitor applies the
mapping at every level of the schema.
The iceberg-rust code is the same at Comet's current pin, at bb1e4a4 (the
1.1 pin), and on iceberg-rust's main, so the 1.1.0 release has the bug too.
apache/iceberg-rust#1845 tracks nested name mapping there as a missing feature.
### Steps to reproduce
```scala
// Parquet files written without Iceberg field ids
spark.sql("""SELECT CAST(id AS INT) AS id,
named_struct('a', CAST(id AS INT), 'b', CAST(-id AS INT)) AS s,
array(named_struct('x', CAST(id AS INT))) AS items,
map('k', named_struct('v', CAST(id AS INT))) AS m
FROM range(50)""").coalesce(1).write.parquet(dataPath)
spark.sql("""CREATE TABLE cat.db.t (id INT, s STRUCT<a: INT, b: INT>,
items ARRAY<STRUCT<x: INT>>, m MAP<STRING, STRUCT<v: INT>>) USING
iceberg""")
spark.sql(s"CREATE TABLE src USING parquet LOCATION '$dataPath'")
SparkTableUtil.importSparkTable(spark, TableIdentifier("src"), table,
stagingDir)
// importSparkTable sets no mapping. migrate, snapshot and add_files set
this one.
table.updateProperties()
.set("schema.name-mapping.default",
NameMappingParser.toJson(MappingUtil.create(table.schema())))
.commit()
spark.sql("REFRESH TABLE cat.db.t")
```
On main at 7d294535e8 (Spark 4.1, Iceberg 1.11):
| Query | Spark | Comet |
| --- | --- | --- |
| `SELECT id, s.a, s.b FROM t WHERE id < 3 ORDER BY id` | `[0,0,0] [1,1,-1]
[2,2,-2]` | `[0,null,null] [1,null,null] [2,null,null]` |
| `SELECT id, items.x, m['k'].v FROM t WHERE id < 3 ORDER BY id` |
`[0,[0],0] [1,[1],1] [2,[2],2]` | `[0,null,null] [1,null,null] [2,null,null]` |
| `SELECT s.a, s.b FROM t ORDER BY s.a LIMIT 3` | `[0,0] [1,-1] [2,-2]` |
`[0,0] [1,-1] [2,-2]` |
Spark 3.4 with Iceberg 1.5.2 gives the same NULLs.
### Expected behavior
The native scan returns the same values as Spark. The fix belongs in
iceberg-rust, where `apply_name_mapping_to_arrow_schema` would apply the
mapping to nested fields as Iceberg Java does. Until Comet's pin has that,
`CometScanRule` could fall back when the table has a name mapping and the scan
reads a struct, list or map column. Comet can't tell at planning time which
data files lack field ids, so that guard would also catch tables whose files
were all rewritten with ids after the migration.
### Additional context
Found while reviewing #6725, which doesn't change it: the results are the
same with its nested schema pruning on or off. Comet's existing migrated-table
tests use `SparkTableUtil.importSparkTable`, which sets no name mapping, so no
test reads a nested column through one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
