fengruotj commented on PR #11286:
URL: https://github.com/apache/paimon/pull/11286#issuecomment-7033636011

   @JingsongLi Thanks for the repro. 
   
   **[P1] Preserve the epoch value with Spark legacy timestamp mapping.** 
   I confirmed the eight-hour shift with `spark-sql` on Spark 3.5.9, using an 
`Asia/Shanghai` JVM and a `UTC` SQL session. I also checked an ordinary 
`TIMESTAMP_LTZ` column and an element in `ARRAY<TIMESTAMP_LTZ>`. With legacy 
mapping enabled, both read eight hours early on the base commit and on this PR. 
With legacy mapping disabled, both preserved the stored epoch.
   
   This matches the historical behavior. Paimon 0.8 mapped both Paimon 
`TimestampType` and `LocalZonedTimestampType` to Spark `TimestampType`. The 
newer mapping distinguishes them on Spark 3.4+, and 
`spark.paimon.legacy-timestamp-mapping.enabled=true` restores the old mapping 
for workloads that depend on it. The [Paimon 
docs](https://paimon.apache.org/docs/master/spark/data-types/#timestamps) 
describe the legacy conversion as correct for Paimon `TimestampType` only. On 
Spark 3.3 and earlier, the connector always selects that conversion, so readers 
may need to adjust for the JVM time-zone offset manually when reading LTZ 
values. I confirmed that setting the option to `false` made no difference on 
Spark 3.2.4 or 3.3.4.
   
   **[P2] Coordinate the StarRocks reader before upgrading its Paimon 
dependency. ** 
   Got it. This change looks like it has compatibility implications for several 
downstream readers. I'll hold off on moving it forward for now and leave the PR 
open for discussion.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to