fengruotj commented on PR #11286: URL: https://github.com/apache/paimon/pull/11286#issuecomment-7033636011
@JingsongLi Thanks for the repro. **[P1] Preserve the epoch value with Spark legacy timestamp mapping.** I confirmed the eight-hour shift with `spark-sql` on Spark 3.5.9, using an `Asia/Shanghai` JVM and a `UTC` SQL session. I also checked an ordinary `TIMESTAMP_LTZ` column and an element in `ARRAY<TIMESTAMP_LTZ>`. With legacy mapping enabled, both read eight hours early on the base commit and on this PR. With legacy mapping disabled, both preserved the stored epoch. This matches the historical behavior. Paimon 0.8 mapped both Paimon `TimestampType` and `LocalZonedTimestampType` to Spark `TimestampType`. The newer mapping distinguishes them on Spark 3.4+, and `spark.paimon.legacy-timestamp-mapping.enabled=true` restores the old mapping for workloads that depend on it. The [Paimon docs](https://paimon.apache.org/docs/master/spark/data-types/#timestamps) describe the legacy conversion as correct for Paimon `TimestampType` only. On Spark 3.3 and earlier, the connector always selects that conversion, so readers may need to adjust for the JVM time-zone offset manually when reading LTZ values. I confirmed that setting the option to `false` made no difference on Spark 3.2.4 or 3.3.4. **[P2] Coordinate the StarRocks reader before upgrading its Paimon dependency. ** Got it. This change looks like it has compatibility implications for several downstream readers. I'll hold off on moving it forward for now and leave the PR open for discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
