andygrove opened a new issue, #5305:
URL: https://github.com/apache/datafusion-comet/issues/5305
### Describe the bug
`ParquetWriterExec` builds the output Parquet schema by renaming the fields
of the incoming Arrow schema:
```rust
let fields: Vec<_> = input_schema.fields().iter().enumerate()
.map(|(i, field)|
Arc::new(field.as_ref().clone().with_name(&column_names[i])))
.collect();
```
So two pieces of schema information come from whatever the Comet scan/serde
produced, not from the Catalyst schema Spark intends to write:
1. **Nullability.** Spark writes each column `required` or `optional`
according to the Catalyst `StructField.nullable` of the *target* schema. Comet
inherits the Arrow field's nullability, which can differ — a column Spark would
mark `required` may come out `optional`, or vice versa. That changes the file's
schema as seen by every reader, and a spuriously `required` column is the
dangerous direction.
2. **Field IDs.** `spark.sql.parquet.fieldId.write.enabled` makes Spark emit
Parquet field IDs from the Catalyst metadata. Comet emits none, so
`ParquetFieldIdIOSuite`-style round trips and any consumer keyed on field ID
(Iceberg, Delta) lose them silently.
### Expected behavior
Take nullability and field IDs from the Catalyst schema.
`WriteJobDescription.dataColumns` is already plumbed through to
`CometWriteFilesExec` for column names (#5293), so the attribute list with its
`nullable` flag and metadata is available at the same point — it needs to reach
the native side as a serialized schema rather than just names.
### Additional context
Found while reviewing #5293. Same family as #3425 (INT96) and #3427 (footer
metadata keys): the written file is readable but is not byte-comparable with
what Spark writes, which matters most for files other engines read — see #4658
and the plan to write Iceberg data files natively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
