sdf-jkl opened a new issue, #11945: URL: https://github.com/apache/arrow-rs/issues/11945
## Is your feature request related to a problem or challenge? While implementing Spark-compatible Variant functions in [datafusion-variant](https://github.com/datafusion-contrib/datafusion-variant), we need to serialize timestamp values using the SQL session time zone or an explicit `to_json` option. Arrow's scalar `VariantToJson` methods and batch `variant_to_json` kernel currently expose no time-zone option and serialize instant timestamps in UTC, including timestamps nested in arrays and objects. For example, this was executed against Arrow 60.0.0: ```rust use arrow::array::{ArrayRef, TimestampMicrosecondArray}; use parquet_variant_compute::{cast_to_variant, variant_to_json}; use parquet_variant_json::VariantToJson; let input = TimestampMicrosecondArray::from(vec![Some(0)]).with_timezone("UTC"); let variant = cast_to_variant(&input)?; let scalar = variant.value(0).to_json_string()?; let batch = variant_to_json(&ArrayRef::from(variant))?; assert_eq!(scalar, "\"1970-01-01T00:00:00+00:00\""); assert_eq!(batch.value(0), scalar); ``` Both assertions pass. There is no parameter to request `America/Los_Angeles`, which should render the same instant as `"1969-12-31T16:00:00-08:00"` using Arrow's existing RFC 3339 formatting. ## Describe the solution you'd like Add time-zone-aware serialization options for the scalar and batch APIs, retaining UTC as the default for existing callers. - Apply the chosen zone to `TimestampMicros` and `TimestampNanos`, including values nested in Variant arrays and objects. - Leave timestamps without a time zone, dates, and times unchanged. - Resolve named-zone offsets for each instant, including daylight-saving transitions. - Keep streaming JSON, JSON strings, `serde_json::Value`, and batch results consistent. The caller would resolve SQL session settings and options; Arrow would perform the serialization. Tests should cover UTC, fixed offsets, named zones across daylight-saving transitions, both timestamp precisions, nested values, and unchanged timestamp-without-time-zone values. ## Describe alternatives you've considered A downstream recursive serializer would duplicate Arrow's existing handling of Variant values. Rewriting JSON strings afterward loses the distinction between timestamp values and ordinary strings with the same contents. ## Additional context The regular `arrow-json` writer already honors a timestamp array's declared time zone through [`ArrayFormatter`](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/arrow-cast/src/display.rs#L755-L795). With Arrow 60.0.0, epoch `0` in an array annotated with `America/Los_Angeles` produced `"1969-12-31T16:00:00-08:00"`. Variant timestamps do not carry that display-zone annotation, so their serializer needs a caller-supplied zone. Arrow's existing `arrow_array::timezone::Tz` provides reusable timezone handling; the timestamp formatting helper itself is currently private. Current `main` at `5746b443f7` still [formats instant timestamps directly with `to_rfc3339`](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/parquet-variant-json/src/to_json.rs#L184-L191), and the [batch kernel](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/parquet-variant-compute/src/to_json.rs#L27-L45) calls the same serializer without options. Spark's [Variant JSON writer](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala#L340-L341) passes the configured zone to `VariantVal.toJson`. Its [JSON options](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala#L112-L113) use the explicit time-zone option or the supplied default. These Spark sources were inspected, not executed locally; matching Spark's other timestamp formatting choices is outside this request. A search of open and closed Arrow issues and PRs found no matching request. Related work: [#10102](https://github.com/apache/arrow-rs/issues/10102) concerns temporal casting semantics, while [#11402](https://github.com/apache/arrow-rs/pull/11402) reduced JSON serialization allocations without adding a time-zone option. Spark's [#59561](https://github.com/apache/spark/pull/59561) concerns additional string/display paths; its `to_json` path already passes the configured zone. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
