andygrove opened a new issue, #6192:
URL: https://github.com/apache/datafusion-comet/issues/6192
### Describe the bug
With `spark.sql.parquet.fieldId.read.enabled=true`, the native scan reads a
nested struct by position whenever the requested struct has the same field
names, in the same order and with the same types, as the file's struct but
different field ids. Spark resolves those fields by id. Schema evolution that
drops a nested column and adds it back under the same name produces exactly
this, since the new column gets a new id, and Comet then returns the dropped
column's old values where Spark returns null.
### Steps to reproduce
On `main` at 67803a7a4 with the default Spark 4.1 profile and Comet enabled:
```scala
import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
spark.conf.set("spark.sql.parquet.fieldId.read.enabled", "true")
def withId(id: Int) = new MetadataBuilder().putLong("parquet.field.id",
id).build()
def xy(xId: Int, yId: Int) = new StructType()
.add("x", LongType, true, withId(xId))
.add("y", LongType, true, withId(yId))
spark.createDataFrame(
spark.sparkContext.parallelize(Seq(Row(Row(1L, 2L))), 1),
new StructType().add("s", xy(1, 2), true, withId(10)))
.write.mode("overwrite").parquet("/tmp/relabel")
// x was dropped and added back, so it now carries id 3
spark.read
.schema(new StructType().add("s", xy(3, 2), true, withId(10)))
.parquet("/tmp/relabel")
.show()
```
| File → requested | Spark | Comet |
| --- | --- | --- |
| `s<x (id 1), y (id 2)>` → `s<x (id 3), y (id 2)>` | `{null, 2}` | `{1, 2}`
|
| `s<x (id 1), y (id 2)>` → `s<x (id 2), y (id 1)>` | `{2, 1}` | `{1, 2}` |
| The same two shapes inside `array<struct>` or a map value | read by id |
read by position |
| Root `x (id 1), y (id 2)` → `x (id 2), y (id 1)` | `{2, 1}` | `{2, 1}` |
`CometNativeScanExec` is in the plan for every row, checked against Spark
4.1.3.
### Expected behavior
The same answers as Spark. Nested fields that carry a field id are matched
by id, so a requested id the file does not have reads as null.
### Additional context
`is_pure_structural_narrowing` declines any target that carries field ids
([schema_adapter.rs#L122-L126](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/native/core/src/parquet/schema_adapter.rs#L122-L126)),
so these columns always get a `CometCastColumnExpr`. Its `evaluate` then takes
the relabel shortcut
([cast_column.rs#L278](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/native/core/src/parquet/cast_column.rs#L278))
because `types_differ_only_in_field_names`
([cast_column.rs#L41](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/native/core/src/parquet/cast_column.rs#L41))
compares names, types and nullability but not `PARQUET:field_id`. The file's
array comes back relabelled in place.
The shortcut dates to #3536, and 1.0.0 already sent field ids to the native
scan, so 1.0.0 is probably affected too. I haven't run it. The native Iceberg
scan is not affected, because it runs the same adapter with `use_field_id` off
([iceberg_scan.rs#L233](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/native/core/src/execution/operators/iceberg_scan.rs#L233)).
#5654 fixes it. Its once-per-file `FieldMapping` only allows the relabel
shortcut when the mapping is positional, and every shape above matches Spark on
that branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
