Chenhao Li created SPARK-60301:
----------------------------------
Summary: Spark doesn't respect session timezone when casting
nested variant to string
Key: SPARK-60301
URL: https://issues.apache.org/jira/browse/SPARK-60301
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.4, 4.1.3, 4.2.0
Reporter: Chenhao Li
**Top-level variant** (`cast(v as string)`): Spark uses the cast's time zone.
In `Cast.scala`, a cast from `VariantType` calls `VariantGet.cast(v, to,
castArgs)`. `castArgs` is a `VariantCastArgs` that carries the cast's
`timeZoneId` and `zoneId`.
**Variant nested in a struct, array, or map** (`cast(struct(v) as string)`):
Spark ignores the cast's time zone and always uses UTC.
- In `ToStringBase.scala`, `castToString` sends the leaf type to
`TypeApiOps(from, zoneId)`. `TypeApiOps.apply` has no `VariantType` case, so it
returns `None`.
- `castToStringDefault` has no `VariantType` case either. It falls through to
`case _ => o => UTF8String.fromString(o.toString)`.
- The codegen path, `castToStringCode`, also has no `VariantType` case. It
falls through to `UTF8String.fromString(String.valueOf($c))`.
- Both paths call `VariantVal.toString()`, which is `toJson(ZoneOffset.UTC)`.
- `Variant.toJson` prints a timestamp with `.atZone(zoneId)` and a `+HH:MM`
offset. Here `zoneId` is UTC.
The nested behavior comes from a fall-through to `toString`, not from a
deliberate choice.
**Example:** the session time zone is `America/Los_Angeles`, and a variant
holds the instant `2020-01-01 00:00:00 UTC`.
- `cast(named_struct('a', v) as string)` prints the timestamp in UTC,
approximately `\{2020-01-01 00:00:00+00:00}`.
- A top-level `cast(v as string)` uses the cast's time zone (here LA).
The same variant value therefore prints with a different offset at top level
than when it is nested.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
