JingsongLi opened a new pull request, #10301: URL: https://github.com/apache/paimon/pull/10301
### Purpose Enable native writes for tables with `data-file.external-paths`, using the Rust implementation in https://github.com/apache/paimon-rust/pull/981. **Merge order:** merge the Rust PR first, then rerun Native CI here. This PR is a draft while that dependency is pending. CI continues building `apache/paimon-rust@main`. - Remove the external-path fallback from native writer selection for supported Parquet append, primary-key and data-evolution tables. - Resolve literal Java path strings for Python file I/O without decoding escaped partition characters. Preserve the original external path in persisted metadata and share physical-path and sidecar resolution between readers, writers and commit cleanup. - Delete external index sidecars along with their data files when aborting native commit messages through the Python committer. - Match Java's positive 32-bit external-path weights and trailing-comma handling. - Document native external writes. Retain the existing Blob writer fallback. ### Tests With the companion Rust build and all five native test flags enabled: - **1593 passed, 2 skipped, 74 subtests passed** across affected plan/read/write/update, data-directory, external-path, Blob, split-serialization and commit-conflict suites. - 75 new parameterized cases cover five external-path strategies; append/PK/data-evolution; partitioned and unpartitioned tables; Python and native writers/planners/readers; batch and stream writes; changed destinations; escaped partition values; updates/upserts/deletes; historical reads; abort cleanup; and existing Blob fallback. - Native execution counters: 1453 plans, 2023 reads, 1014 writes, 5 commits; updates exercised row-id, grouped, predicate, upsert and incremental routes. - Flake8 for all changed Python files and `git diff --check` passed. Local test environment: Python 3.13 and PyArrow 18.1.0. GitHub CI supplies the Linux/Python 3.11 Native matrix after the Rust dependency merges. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
