hhr293 commented on PR #58424: URL: https://github.com/apache/spark/pull/58424#issuecomment-5749694551
Hi @sunchao, thanks for the detailed repro! Fixed. Root cause: `Cast.needsTimeZone` does not recurse through complex-to-STRING casts, so cases such as `CAST(ARRAY<TIMESTAMP> AS STRING)` incorrectly report `needsTimeZone=false` even though formatting the nested timestamp depends on the captured time zone. The fix extends `isRepeatableExpression` to walk the source type tree with `DataType.existsRecursively` and reject a complex-to-STRING cast if any nested type is time-zone-sensitive according to `Cast.needsTimeZone`. For the regression test, I reproduced the reported case directly: two projections of the same Parquet `FileIndex` are analyzed under `UTC` and `America/Los_Angeles`, then joined on the formatted STRING value. The query diverges with the rewrite on/off before the fix and agrees after it. I also added coverage for ARRAY / MAP / STRUCT cases, with `TIMESTAMP_NTZ` as the positive control that still allows the rewrite. This also looks like a more general gap in Spark's time-zone classification for complex-to-STRING casts, rather than something specific to this rule, so it may be worth addressing separately in Spark core. `sql/testOnly *RewriteSelfJoinInequalityToAggregateSuite` passes **41/41**. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
