kerwin-zk opened a new pull request, #8781: URL: https://github.com/apache/paimon/pull/8781
### Purpose Daft previously let independent sink tasks write the same primary-key bucket. For fixed-bucket tables this could produce overlapping sequence numbers, while dynamic-bucket tables used task-local bucket assignments that could place the same key in different buckets. Both cases could silently return stale or duplicate primary keys. This change: - writes each fixed `(partition, bucket)` group through one Paimon writer; - coordinates dynamic bucket assignment through persistent, Java-compatible `HASH` indexes and restores mappings across commits; - pins one base snapshot per Daft job and adds optimistic conflict checks for concurrent index changes and overwrite remapping; - makes regular PyPaimon dynamic-bucket writes restore the same persistent mapping; - fails fast for primary-key write modes that do not yet have safe global ownership, and preserves files when commit success is uncertain; - adds Java/Python hash goldens and bidirectional `HASH` index interoperability coverage. The group-based Daft path intentionally favors correctness: all rows for one `(partition, bucket)` are materialized by one worker. Fixed-bucket tables with very few or highly skewed buckets can therefore have a single-worker memory/CPU hotspot; dynamic buckets are bounded by their target row count. ### Tests - `pytest` focused Daft/write/commit suites: 87 passed, 1 skipped - Java writes dynamic `HASH` index -> Python restores and updates the same composite key - Python writes dynamic `HASH` index -> Java restores and updates the same composite key - changed Python files: flake8 and py_compile - `bash -n paimon-python/dev/run_mixed_tests.sh` - Maven checkstyle and Spotless through the targeted non-fast-build test flow - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
