JingsongLi opened a new pull request, #10354: URL: https://github.com/apache/paimon/pull/10354
### Purpose Enable the existing native writer/committer wrappers for two Paimon bucket modes and repair the metadata scans needed by pending files. Business logic remains in the Rust core; the Python binding API is unchanged. Depends on apache/paimon-rust#1017. This is a draft until that Rust change is merged: the native CI lane continues to install Rust `main`, with no fork or commit pin. ### Changes - Enable native cross-partition primary-key writes, including all four merge engines, full composite keys, migration deletes, batch/stream writes and restart bootstrap. - Enable ordinary native Parquet postpone writes for deduplicate tables without BLOB columns. Other postpone merge engines, BLOB writes and the separate fixed-bucket builder keep their Python route. - Allow native publication of postpone commit messages. - Make overwrite, commit-conflict scans and partition-only deletes include pending buckets. Ordinary reads still expose only real buckets. This follows Java's separation between read and commit scans. - Document the supported scope and single-writer ownership requirement. - Check postpone directory/naming behavior without assuming an Avro suffix, since native postpone files are Parquet. ### Validation 32 new native end-to-end cases cover all merge engines, composite-key bootstrap, A → B → A batch migrations, row input and row kinds, checkpoints, both Python and native REST committers, overwrite, abort, pending-file deletion, file key bounds/counts and physical arrival order. Tests assert the selected native writer and prevent fallback on the exercised native read/commit paths. With a local build of apache/paimon-rust#1017: - Broader native writer/commit/update regression: 483 passed initially; two existing Avro-only assertions were corrected, then all 253 affected native tests passed (plus 20 subtests). - Ordinary Python writer/commit/update regression: 221 passed (plus 20 subtests). - Flake8 and `git diff --check` passed. No compaction, video support or additional format implementation is included. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
