andygrove opened a new issue, #5694:
URL: https://github.com/apache/datafusion-comet/issues/5694
### Describe the bug
Writing to a table partitioned on a `timestamptz` column panics in
iceberg-rust while generating the data file's partition path:
```
org.apache.comet.CometNativeException: called `Option::unwrap()` on a `None`
value
at core::option::unwrap_failed
at
iceberg::spec::values::temporal::timestamptz::microseconds_to_datetimetz
at <iceberg::spec::values::datum::Datum as core::fmt::Display>::fmt
at <iceberg::spec::values::datum::Datum>::to_human_string
at <iceberg::spec::transform::Transform>::to_human_string
at
<iceberg::spec::partition::PartitionSpec>::partition_to_path::{closure#0}
at <iceberg::spec::partition::PartitionSpec>::partition_to_path
at <...DefaultLocationGenerator as
...LocationGenerator>::generate_location
at <...RollingFileWriter<ParquetWriterBuilder, DefaultLocationGenerator,
...>>
```
`microseconds_to_datetimetz` unwraps a `None` when converting the partition
value to its human-readable form for the directory name, so the task dies with
a Rust panic crossing the JNI boundary rather than an error.
### Steps to reproduce
Spark 4.1.3, Iceberg 1.11.0, `spark.comet.iceberg.write.enabled=true` plus
the Iceberg Spark SQL test setup from `dev/diffs/iceberg/1.11.0.diff`:
```
./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 -DflinkVersions=
-DkafkaVersions= \
:iceberg-spark:iceberg-spark-4.1_2.13:test --tests '*TestSparkDataFile*'
-Pquick=true -x javadoc
```
Failing cases:
- `TestSparkDataFile.testValueConversionPartitionedTable`
- `TestSparkDataFile.testValueConversionWithEmptyStats`
Both write to a table whose partition spec covers a wide set of column
types, including `timestamptz`.
### Expected behavior
The partition path is generated for `timestamptz` partition values the same
way iceberg-java generates it, and the write succeeds. Any value the conversion
genuinely cannot represent produces an error rather than a panic.
### Additional context
Found by turning the two Iceberg write flags on by default in #5677.
The panic is in iceberg-rust rather than in Comet's own code, so the fix may
need to go upstream. Worth checking whether the value reaching
`microseconds_to_datetimetz` is itself wrong (a unit or epoch mismatch on the
way in) before concluding the conversion is at fault.
Part of #5649.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
