jordepic commented on code in PR #5361:
URL: https://github.com/apache/datafusion-comet/pull/5361#discussion_r3787169498
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeWrite.scala:
##########
@@ -163,12 +159,12 @@ object CometIcebergNativeWrite extends
CometOperatorSerde[IcebergWriteExec] with
"custom location provider unsupported"),
requireFormatVersionAtMostTwo,
requireNoEncryptionPrefix,
- requireSupportedMetricsModes,
requireNoBloomFilterColumnsEnabled,
requireRowGroupCheckMinRecordCountAtDefault,
requireRowGroupCheckMaxRecordCountAtDefault,
requireParquetPageVersionDefault,
requireShredVariantsDisabled,
+ requireParseableCompressionLevel,
Review Comment:
You're right — a `uuid` column passed detection and then failed the task
inside `decorate_batch_with_field_ids` (Arrow has no `Utf8 ->
FixedSizeBinary(16)` cast). Added a type-based trigger rule: detection now
walks the write schema (nested fields included) and declines any `uuid` column
with a fall-back reason. Pinned by a new detection test that evolves a `uuid`
column onto a table through the Iceberg API (Spark DDL can't declare one) and
asserts the write falls back and still succeeds through iceberg-java.
`fixed(N)` is pinned too: a new round-trip test writes exact-N-byte values
through the native path and compares row values and `readable_metrics` bounds
against a JVM-written twin — `Binary -> FixedSizeBinary(4)` casts fine as long
as lengths match, and a wrong-length value fails the task the same way the JVM
writer rejects it. I audited the remaining Iceberg types: `uuid` is the only
Spark-writable type with a cast gap (`time` / `timestamp_ns` can't be written
through Spark's Iceberg integration at all, and the V3-only types are behind
the format-version gate).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
