JingsongLi commented on PR #10160: URL: https://github.com/apache/paimon/pull/10160#issuecomment-5832119205
Reviewed the write conversion and the SQL regression as a persisted-data correctness fix. When Spark supplies `java.sql.Date` with `spark.sql.datetime.java8API.enabled=false`, Spark's own `fromJavaDate` is the inverse of its hybrid-calendar conversion; the generic Paimon epoch-millisecond conversion was not. The `LocalDate` branch stays unchanged. The test exercises both Spark Java 8 API settings, a year-1000 date, the day before and the day of the 1582 cutover, a modern date, equality filtering, and ordering. Local verification on the PR head: Spark 3.5 `org.apache.paimon.spark.sql.DDLTest` passed 40/40 tests, including the new case; `git diff --check` passed. CI is green for Spark 3/4, Scala 2.12/2.13, JDK 8/11 Core and integrations, and Flink/Spark interoperability. I found no code blocker. Release note: this corrects future Spark writes. Rows already persisted with the old shifted epoch day remain shifted and need an explicit backfill or rewrite if affected; this change does not reinterpret existing files on read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
