chenhao-db opened a new pull request, #59561:
URL: https://github.com/apache/spark/pull/59561
### What changes were proposed in this pull request?
Make variant-to-string conversion use the session time zone in all paths:
- `ToStringBase`: add a `VariantType` case to the interpreted path and the
codegen path. It calls `VariantVal.toJson(zoneId)`. Before, both paths fell
through to `VariantVal.toString`, which always uses UTC. This fixes a variant
nested in a struct, array, or map under `CAST(... AS STRING)`. It also fixes
`ToPrettyString`, which `Dataset.show()` uses.
- `HiveResult`: add the session `zoneId` to `TimeFormatters`, and use it for
variants in `toHiveString`. This fixes the spark-sql CLI and Thrift server
output.
### Why are the changes needed?
A top-level `cast(v as string)` uses the cast's time zone, through
`VariantGet.cast` and `VariantCastArgs`. A nested variant ignores that zone and
always prints UTC. So the same value prints with a different offset at top
level and when nested. `to_json` already uses the session time zone.
### Does this PR introduce _any_ user-facing change?
Yes. Variant timestamps now print in the session time zone, not in UTC, in
these places:
- a variant nested in a struct, array, or map that is cast to string
- `Dataset.show()`
- spark-sql CLI and Thrift server output
The format does not change: it is still JSON with a `+HH:MM` offset. With
`spark.sql.session.timeZone=America/Los_Angeles`:
```sql
SELECT cast(named_struct('a', cast(TIMESTAMP'2020-01-01 00:00:00Z' AS
VARIANT)) AS STRING);
-- Before: {"2020-01-01 00:00:00+00:00"}
-- After: {"2019-12-31 16:00:00-08:00"}
```
### How was this patch tested?
- New test in `VariantExpressionSuite`. It covers struct, array, and map
casts and `ToPrettyString`, in UTC and in `America/Los_Angeles`, with codegen
on and off.
- New test in `HiveResultSuite`. It covers a top-level variant and a variant
nested in an array.
- Both new tests fail without the fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
