JingsongLi opened a new pull request, #10508: URL: https://github.com/apache/paimon/pull/10508
## What is the purpose of the change? Align type-change paths already supported by PyPaimon with Java and verify them through Native reads. The existing Arrow cast reads timestamp physical ticks as BIGINT and treats BIGINT as timestamp ticks, so the results use the wrong units. **Depends on https://github.com/apache/paimon-rust/pull/1107.** This PR is a draft until the Rust fix is merged because Native CI deliberately builds paimon-rust from upstream `main`. ## Brief change log - Use epoch seconds for TIMESTAMP/TIMESTAMP_LTZ → BIGINT, including floor division before the epoch and Java's LTZ local-date-time round trip. - Use Java's signed-long intermediate multiplication for BIGINT → ordinary TIMESTAMP and retain NULLs for Arrow storage overflow. - Apply these conversions in the existing shared schema-evolution helper, which is already used by scalar, recursive nested and leaf-projection readers. - Add 30 unit cases for sliced/chunked arrays, NULLs, Java boundary values and Arrow units. - Add 43 REST Native parity cases for append, primary-key and data-evolution tables, covering mixed schema versions, single-value file statistics, predicate-only fields, NULLs, ROW/ARRAY/MAP evolution and nested/MAP key projections. Native checks explicitly forbid Python read/plan fallback. ## Verifying this change - Reproduced the old Native result `42001ms → 42001`, where Java and the corrected Python path return `42`. - Rebuilt the Native extension using the dependent Rust branch. - `schema_evolution_cast_test.py` + `native_schema_evolution_cast_test.py`: 166 passed with Native plan/read/write/commit enabled (296 Native plans/reads and 176 Native writes/commits). - Existing schema-evolution, PK reads, MAP key projection and nested projection suites: 112 passed, 21 subtests passed with Native enabled. - Final nested epoch/MAP key projection checks: 3 passed across all table modes. - Changed-file Flake8, `git diff --check` and Python 3.6 grammar parsing passed. - Java `CastExecutorTest#testTimestampToNumeric+testTimestampToNumericPreEpoch`: 2 passed. Direct Java CastExecutors calls confirmed the overflow golden values. No new Python API, dependency or unsupported Python type-conversion capability is introduced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
