JingsongLi opened a new pull request, #10202: URL: https://github.com/apache/paimon/pull/10202
### Purpose Enable PyPaimon to use the Parquet capabilities already implemented in Rust core and remove obsolete native writer/read gates. Depends on https://github.com/apache/paimon-rust/pull/970. This PR is draft until that change reaches Rust main. Native CI now builds `apache/paimon-rust@main` and continues to print the resolved commit for provenance; it does not pin the companion topic branch. - Enable native batch/stream writers for ordinary Parquet data-evolution tables, first-row/partial-update/aggregation merge engines, primary-key value statistics, batch-boundary row rolling, and compatible nested/fixed-size binary Arrow input. - Keep fallback for unsupported MAP shared-shredding and data-evolution row sidecars, along with existing format/Blob/bucket constraints. - Defer Python-only merge capability validation until a read actually falls back to Python. Invalid configurations still fail eagerly; unsupported Python merge options cannot bypass validation through raw splits, streaming readers, iterator APIs or parallel dispatch. - Share canonical partition-path resolution among append/PK, data-evolution and chunk-shuffle planners. This lets Python readers locate Java/Rust escaped partition directories while retaining legacy Python paths and explicit external paths. - Keep implementation-specific assertions on their Python lane and make Ray fixture batches explicit so batch-boundary rolling still yields deterministic file groups. ### Tests Against the locally rebuilt companion Rust tree, with native plan/read/write/update/commit all enabled: - Affected Native, data-evolution, merge-engine, Parquet page-index, Variant, nested-input, Blob, sequence and Ray suites: 1,686 passed plus 186 subtests. The run identified an unpartitioned path-helper regression, fixed and rechecked below. The remaining three cases require the locally unavailable optional Vortex package. - Final path-helper/group-stats/new native-capability regression run: 47 passed plus 17 subtests. - Scanner, deletion-vector-path and sequence regression run: 218 passed. - Rust Python write binding tests: 29 passed. - Flake8 for every changed Python file and `git diff --check` passed. The new capability tests assert the native writer is selected and native reads do not fall back; check results through both readers where their capabilities overlap; inspect primary-key value statistics; cover sliced nested/null input, repeated commits, sequence-group aggregation, delete filtering, partitioned delete/history reads and chunk shuffle. GitHub Native CI must be rerun after apache/paimon-rust#970 merges because this PR intentionally follows Rust main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
