SteNicholas opened a new issue, #541: URL: https://github.com/apache/paimon-cpp/issues/541
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found no existing issue tracking the full dynamic-bucket primary-key table lifecycle. Related work: #493 tracks batch reads; #494 introduced resolved-bucket-mode handling for index manifests; PR #499 implements batch reads with Java compatibility coverage. PR #499 is still open at the time of filing, so its actual landing should be tracked as part of this issue. ### Motivation Sub-issue of #519: **Core Feature Alignment — Support dynamic-bucket primary-key tables**. Paimon Java primary-key tables can use `bucket = -1` to grow the number of buckets as keys arrive. Paimon C++ recognizes `BucketMode::HASH_DYNAMIC`, but the table write path still rejects non-positive bucket counts outside postpone mode, and `BucketIdCalculator` rejects dynamic primary-key bucket calculation. The existing write restore path also supplies no dynamic-bucket index. Batch reading can reuse bucket IDs recorded in manifests, which is the scope of PR #499. Full support additionally needs stateful key routing, Java-compatible bucket-index persistence and recovery, and streaming reads that account for newly created buckets. This issue targets `HASH_DYNAMIC` primary-key tables, including partitioned tables whose primary keys contain all partition fields. Cross-partition upserts remain a separate roadmap item in #519. ### Solution Implement this in focused stages while preserving the existing fixed-bucket and append-table behavior: - [ ] **Complete batch-read support.** Reuse #493 / #494 / PR #499, track the batch-read PR through landing, and verify batch planning, predicate filtering, snapshot selection, and deletion-vector lookup across multiple partitions and buckets. Keep `HASH_DYNAMIC` distinct from append-table `BUCKET_UNAWARE` even though both configure `bucket = -1`. - [ ] **Add a stateful dynamic bucket assigner.** Restore the Java-compatible primary-key-hash-to-bucket mapping, reuse a key's assigned bucket for subsequent upserts and deletes, and allocate buckets for new keys according to the supported Java dynamic-bucket options, including `dynamic-bucket.target-row-num`. Define assignment ownership and supported writer topology consistently with Java. - [ ] **Persist and restore bucket indexes.** Implement compatible hash-index file reading/writing and manifest handling. Connect them to `RestoreFiles::DynamicBucketIndex()` and `FileSystemWriteRestore`, restoring only mappings visible in the selected committed snapshot. Keep dynamic-bucket indexes and deletion-vector indexes correctly scoped by partition and bucket. - [ ] **Integrate dynamic routing with writes and commits.** Add the stateful routing path to table writes and support insert, upsert, delete, flush, prepare-commit, commit, abort, and restart. Publish data, bucket-index changes, and applicable deletion-vector changes consistently in snapshots; a failed or retried commit must not lose or reassign committed keys. - [ ] **Preserve correctness through compaction and cleanup.** Retain valid key routing as data files are compacted or retired, and retain index files needed by supported snapshots and tags. Apply existing writer/concurrency constraints and explicitly validate unsupported combinations. - [ ] **Support streaming reads.** Discover newly appearing buckets while planning successive snapshots and preserve the existing supported incremental/changelog and deletion-vector semantics. - [ ] **Document the support matrix.** Describe supported reads and writes, Java-compatible options, assignment ownership, recovery behavior, and remaining restrictions. Update the current statement that primary-key dynamic-bucket writes are unsupported. Useful implementation entry points: - [Bucket mode definition](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/table/bucket_mode.h) and [resolution](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/table/bucket_mode.cpp). - [Table read/write construction](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/table/table.cpp) and [bucket calculation](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/bucket/bucket_id_calculator.cpp). - [RestoreFiles](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/operation/restore_files.h) and [filesystem write restore](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/operation/file_system_write_restore.h). - [Index manifest handling](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/manifest/index_manifest_file_handler.cpp) and the existing primary-key write/commit and deletion-vector paths. ### Anything else? Acceptance and validation: - [ ] Add focused assigner, index-file, restore, and commit unit tests, plus table-level integration tests. - [ ] Cover unpartitioned and partitioned tables, multiple buckets, bucket growth, repeated keys, hash collisions, upserts, deletes, and writer restart from committed snapshots. - [ ] Cover commit/abort/retry failures and compaction, ensuring that committed keys retain consistent routing and no duplicate live primary keys appear across buckets. - [ ] Cover deletion vectors enabled and disabled and the existing applicable primary-key merge modes. - [ ] Verify batch and streaming reads across snapshots that introduce new buckets, and confirm that fixed-bucket and append-unaware tables retain their existing behavior. - [ ] Cross-read Java-written tables from C++ and C++-written tables from Java, including bucket-index files, restored assignments, updates/deletes, and deletion vectors. - [ ] Update documentation and keep the parent roadmap item unchecked until the agreed full lifecycle is supported and validated. The existing batch-read PR is an initial delivery of this roadmap item. Its completion alone does not complete dynamic-bucket writes and streaming reads. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
