hhr293 opened a new pull request, #59394:
URL: https://github.com/apache/spark/pull/59394
### What changes were proposed in this pull request?
`Cast.needsTimeZone` recurses into ARRAY/MAP/STRUCT only when the target has
the same shape as
the source, so a cast that collapses a complex type to a single STRING falls
through to the
`case _ => false` fallback. `castToString` (and `castToStringCode`) does
recurse into those
containers and formats every LTZ timestamp leaf with the session `zoneId`,
so the classification
contradicts the implementation it is documented to mirror:
| cast | before | after |
| --- | --- | --- |
| `TIMESTAMP -> STRING` | true | true |
| `ARRAY<TIMESTAMP> -> ARRAY<STRING>` | true | true |
| `ARRAY<TIMESTAMP> -> STRING` | **false** | true |
| `MAP<INT, TIMESTAMP> -> STRING` | **false** | true |
| `STRUCT<t: TIMESTAMP> -> STRING` | **false** | true |
| `ARRAY<ARRAY<TIMESTAMP>> -> ARRAY<STRING>` | **false** | true |
| `ARRAY<TIMESTAMP_NTZ> -> STRING` | false | false |
This PR adds one case to `Cast.needsTimeZone`: a cast whose target is STRING
needs the session
time zone when the source type recursively contains an LTZ timestamp leaf.
The nested
`ARRAY<ARRAY<TIMESTAMP>> -> ARRAY<STRING>` shape is covered by the existing
complex-to-complex
recursion, which now bottoms out in the new case. TIMESTAMP_NTZ, DATE and
TIME render on a fixed
grid and stay zone-independent, as in the scalar cases.
A Python UDT is included because `castToString` renders it through its
`sqlType`; a Scala UDT is
not, because it goes through the user's `stringifyValue`.
### Why are the changes needed?
Three consumers of the predicate are affected.
1. `Cast.resolved` only demands a `timeZoneId` when `needsTimeZone` is true,
so the cast was
reported as resolved without one and then failed in
`TimeZoneAwareExpression.zoneId`:
```scala
val c = Cast(Literal.create(Seq(0L), ArrayType(TimestampType)), StringType)
c.resolved // true
c.eval(null) // java.util.NoSuchElementException: None.get
```
2. `Cast.canonicalized` replaces the zone with `null` when `!needsTimeZone`,
so two casts carrying
different session time zones compared equal.
3. Because `CacheManager` looks up cached plans with `sameResult`, a result
cached under one
session time zone was served to a query running under another, returning
a stale rendering.
With a table holding `ts TIMESTAMP` and `ats ARRAY<TIMESTAMP>` at the
same instant:
```scala
// q = SELECT CAST(<col> AS STRING) AS v FROM t
spark.conf.set("spark.sql.session.timeZone", "UTC")
val cached = sql(q); cached.cache(); cached.collect()
spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
sql(q).collect()
```
```
col = ts (needsTimeZone = true) -> [1969-12-31 16:00:00] correct,
cache not reused
col = ats (needsTimeZone = false) -> [[1970-01-01 00:00:00]] wrong, UTC
rendering reused
col = ats, after unpersist -> [[1969-12-31 16:00:00]] correct
```
The classification has had no complex-to-STRING case since branch-3.5, so
this is not a
regression.
### Does this PR introduce _any_ user-facing change?
Yes. A query that renders a complex type containing an LTZ timestamp as a
string no longer
receives a result cached under a different session time zone, as shown above.
The stricter classification also means such a cast now requires a
`timeZoneId` to be resolved.
`ResolveTimeZone` assigns the session zone unconditionally, so plans built
by the analyzer are
unaffected. Reporting these casts as zone-dependent additionally makes
`Cast.contextIndependentFoldable` false for them, which can forgo a V2
pushdown or a constant
folding opportunity that was previously taken with a zone-dependent value.
### How was this patch tested?
New tests:
* `CastSuiteBase` - a positive/negative matrix for `needsTimeZone` over
ARRAY/MAP/STRUCT, nested
containers, LTZ nanos, VARIANT and a Python UDT, each also checked one
level further out as
`ARRAY<...> -> ARRAY<STRING>`; TIMESTAMP_NTZ, DATE, TIME and an NTZ-backed
Python UDT as the
negative controls. Run by `CastSuite`, `CastWithAnsiOnSuite` and
`TryCastSuite`.
* `CastSuiteBase` - a complex-to-string cast is unresolved without a zone,
resolved with one, not
semantically equal across two zones, and renders differently per zone
(`checkEvaluation` covers
the interpreted and the codegen path).
* `CachedTableSuite` - the cross-time-zone cache case above, over
`ARRAY<TIMESTAMP>`,
`ARRAY<ARRAY<TIMESTAMP>>` and a scalar `TIMESTAMP` control.
Verified that each new test fails without the production change: reverting
the classification
turns both the `ARRAY<TIMESTAMP>` and the `ARRAY<ARRAY<TIMESTAMP>>` cache
cases red while the
scalar control stays green, and dropping only the Python UDT branch fails
exactly that entry of
the matrix.
Regression: `CastSuite` + `CastWithAnsiOnSuite` + `TryCastSuite` (276 tests)
and the full
`CachedTableSuite` (107 tests) pass.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Co-authored-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
