jackylee-ch opened a new pull request, #1037: URL: https://github.com/apache/paimon-rust/pull/1037
Variant `DATE`, `TIMESTAMP` and `TIMESTAMP_NTZ` values were serialized as the raw backing integer. `GenericVariant::to_json` emitted a date as `"19723"` and a timestamp as `"1704067200000000"`, and the variant→VARCHAR extraction (`cast_variant_to_string`) and the DataFusion `CAST(variant AS STRING)` / `variant_get(.., 'string')` path did the same. The output is silently wrong rather than an error, so a Variant column written by Spark/Flink/Paimon (or assembled internally) round-trips to JSON/string as a bare integer. The three paths also would have diverged the moment one was fixed. Java `GenericVariantUtil` renders `DATE` via `LocalDate.ofEpochDay(..)` and `TIMESTAMP`/`TIMESTAMP_NTZ` via an ISO formatter (`ISO_LOCAL_DATE` + `' '` + `ISO_LOCAL_TIME`), anchoring NTZ to UTC and the zoned `TIMESTAMP` to the caller's zone with a `+HH:MM` offset. This adds shared `format_variant_date` / `format_variant_timestamp` helpers and routes all three paths through them, so they can no longer disagree. `DATE` and `TIMESTAMP_NTZ` match Java exactly; the fractional part uses 0/3/6 digits like Java's ISO time formatter, consistent with the existing `partition_utils` timestamp formatting. One deliberate divergence: `GenericVariant::to_json` takes no `ZoneId`, so a local-zoned `TIMESTAMP` is rendered in UTC rather than a session zone. Matching Java's zone-aware rendering would require threading a zone through the public API and is left as a follow-up. Non-finite float rendering is already handled and is unchanged here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
