andygrove opened a new issue, #6141:
URL: https://github.com/apache/datafusion-comet/issues/6141
### Describe the bug
On a format-version 1 table, a partition spec can mix a live field with a
`void` field whose source column has since been dropped (iceberg-java keeps a
dropped V1 partition field as a `void` transform). The native Iceberg writer
fails every task on such a spec, and the query does not fall back.
`CometLocationGenerator::try_new`
(`native/core/src/execution/operators/iceberg_partition_path.rs:66-70`) skips
resolving the partition type only when the spec is entirely `void`
(`is_unpartitioned()`). For a mixed spec it calls
`PartitionSpec::partition_type(schema)`, which errors with "No column with
source column id" because the `void` field's source column is gone. The
partition-value computation hits the same resolution.
The fixes for #5691 and #5693 cover the all-`void` spec only.
This was found by reading the code and has not been reproduced yet.
### Steps to reproduce
```sql
CREATE TABLE t (id INT, a STRING, b STRING) USING iceberg
PARTITIONED BY (a, b) TBLPROPERTIES ('format-version'='1');
ALTER TABLE t DROP PARTITION FIELD b;
ALTER TABLE t DROP COLUMN b;
-- with spark.comet.iceberg.write.enabled=true
INSERT INTO t VALUES (1, 'x');
```
### Expected behavior
The insert succeeds, as it does with iceberg-java. Either the native writer
resolves the partition type without the `void` field's source column (a `void`
field never contributes a value), or the gate declines the native write when a
`void` field's source column is missing from the schema.
### Additional context
Found in an audit of the native Iceberg write path before enabling it by
default. Part of #5649. Related: #5691, #5693.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
