fengruotj commented on PR #11286: URL: https://github.com/apache/paimon/pull/11286#issuecomment-7033350707
> Using the snapshot instant for both system tables addresses the JVM-dependent results in [#11219](https://github.com/apache/paimon/issues/11219). All 13 Core snapshot and tag tests passed on a clean integration with master. I found two reader compatibility issues with the type change. > > **[P1] Preserve the epoch value with Spark legacy timestamp mapping.** I committed a row, created its tag, and joined `$snapshots` to `$tags` on Spark 3.5.9 with an Asia/Shanghai JVM and a UTC SQL session. With `spark.paimon.legacy-timestamp-mapping.enabled=false`, both `unix_millis(commit_time)` values matched the stored `1795146612121`. Setting it to `true` returned `1795117812121` for both, eight hours early. On master, the legacy-mapping query returned the correct instant. `AbstractSparkInternalRow.getTimestampMicros` calls the converter without the Paimon type. The legacy branch calls `Timestamp.toSQLTimestamp()`, applying the JVM zone to the new epoch-backed value. Spark 3.2 and 3.3 always use that branch. Please pass through epoch micros for `TIMESTAMP_LTZ` and exercise both mapping settings in the SQL regression. > > **[P2] Coordinate the StarRocks reader before upgrading its Paimon dependency.** StarRocks reads `$snapshots.commit_time` for unpartitioned table freshness and converts it using `commitTime.toLocalDateTime().atZone(ZoneId.systemDefault()).toInstant()`. I ran that conversion with an Asia/Shanghai JVM and both Paimon timestamp representations. A commit at `2026-11-17T06:31:24.761Z` produced the same value before this change and `2026-11-16T22:31:24.761Z` afterward. The earlier timestamp can affect materialized-view refresh and staleness decisions. Please document the type transition and track a StarRocks reader update that uses epoch milliseconds for LTZ and retains the existing conversion for older Paimon schemas. [StarRocks consumer](https://github.com/StarRocks/starrocks/blob/ee983a26ea12e033846f53369da3ea941389a020/fe/fe-core/src/main/java/com/starrocks/connector/paimon/PaimonMetadata.java#L1127-L1163) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
