JingsongLi opened a new pull request, #10353: URL: https://github.com/apache/paimon/pull/10353
### Purpose Enable PyPaimon Native Parquet writes for primary-key dynamic-bucket tables (`bucket=-1`, partition columns included in the primary key). Depends on https://github.com/apache/paimon-rust/pull/1016. This PR is draft until that Rust change reaches `main`; Native CI intentionally continues to build Rust `main`, without pinning a fork or weakening capability tests. The previous gate rejected all dynamic-bucket writes. Simply removing it was incorrect: Python rejected the Java/Rust HASH replacement protocol, bucket-local HASH paths were not resolved correctly, and the Rust writer needed max-buckets and recovery fixes. Changes: - Enable the existing Rust core writer through the thin Native bridge. No bucket assignment, hashing, or persistent index work is duplicated in the Python binding. - Align HASH manifest merging with Java `BucketedCombiner`: an ADD replaces the previous complete HASH index for that partition/bucket. Explicit DELETE stale checks and competing additions within one commit remain validated. - As in Java, independent writers must have one owner per bucket. New-only HASH messages do not contain a concurrency baseline; retaining an old index is not by itself a conflict. - Use shared bucket-index path resolution for Python HASH reads, writes and abort, including legacy `table/index` files and explicit external paths. - Reject custom `bucket-key` for dynamic writes, matching Java SchemaValidation. - Cover Python/Rust restarts, both committers, stream checkpoints, max-buckets, missing indexes, overwrite, abort ownership, external paths and legacy layouts. - Update the existing dynamic/cross-partition scan tests to the current Rust batch-builder API, including their capability guard, so those tests execute instead of silently skipping. ### Tests Using a locally rebuilt Rust binding containing the companion core changes: - 390 tests and 33 subtests passed with native planning, reading, writing and committing enabled. - Native counters: 243 plans, 207 reads, 196 writes, 54 commits. - 60 Python-path tests and 11 subtests passed without Native forcing. - Flake8 for all changed Python files; `git diff --check`. The Native run covers `native_dynamic_write_test`, `native_plan_dynamic_bucket_test`, the `tests/write` directory, `native_write_test`, `native_write_capabilities_test`, `index_manifest_write_test`, `native_commit_test`, `file_store_commit_test`, and `index_file_handler_test`. Native-specific tests assert the writer backend and guard against read/commit fallback where applicable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
