JingsongLi opened a new pull request, #10572: URL: https://github.com/apache/paimon/pull/10572
## Dependency Requires https://github.com/apache/paimon-rust/pull/1128. This PR is draft until the Rust changes are merged. Native CI continues to install Rust `main`; it is not pinned to a PR branch. ## Purpose Route Python's existing old-file/schema-evolution conversions through the same Rust core implementation as Native table readers, with Java semantics as the reference. ## Changes - Delegate scalar and nested schema casts to the core Arrow transport API, including chunked/empty arrays and field-ID reconciliation. - Use the core runtime read schema for Native and Python read adapters. This permits Java cast-produced NULLs for logical NOT NULL fields without changing persisted constraints. - Preserve valid read-to-write roundtrips through the existing core writer normalizer. Actual visible forbidden NULL children are rejected. Python Parquet retains nullable transport fields for hidden NULL children underneath NULL ROW parents, after core validation. - Reject TIME precision changes Java cannot execute while accepting equivalent precision spellings. - Add REST append/PK/data-evolution regressions for constructed strings, LargeList layouts, scalar Java casts, type aliases/defaults, NOT NULL overflow and read-to-write constraints. Extend the non-UTC Native CI checks. CAST parsing, numeric/temporal rules, nested visibility and writer value validation remain in Rust core; the Python changes transport types/arrays and integrate the returned schemas. ## Verification - Native acceptance: 639 passed, 2 skipped, 16 subtests; 788 Native plans and 773 Native reads exercised, with guards preventing Python read fallback. - Existing Python/fallback regressions: 244 passed, 1 skipped, 13 subtests. - Native timestamp subset in separate Asia/Shanghai and America/New_York processes: 25 passed, 1 skipped in each zone. - Flake8, Apache license-header check and Python 3.6 grammar checks passed. Runtime tests used Python 3.13 with the paired editable Rust binding. - The paired Rust PR passed 4,045 core tests, 368 self-contained DataFusion tests, and the production evolution path's 1,562 actual-Java scalar cases per supported fixture zone. The extreme millisecond timestamp case is covered directly and in append/data-evolution reads; its PK input test is skipped because the existing Python PK writer cannot materialize that out-of-datetime-range value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
