SteNicholas opened a new issue, #541:
URL: https://github.com/apache/paimon-cpp/issues/541

   ### Search before asking
   
   - [x] I searched in the 
[issues](https://github.com/apache/paimon-cpp/issues) and found no existing 
issue tracking the full dynamic-bucket primary-key table lifecycle.
   
   Related work: #493 tracks batch reads; #494 introduced resolved-bucket-mode 
handling for index manifests; PR #499 implements batch reads with Java 
compatibility coverage. PR #499 is still open at the time of filing, so its 
actual landing should be tracked as part of this issue.
   
   ### Motivation
   
   Sub-issue of #519: **Core Feature Alignment — Support dynamic-bucket 
primary-key tables**.
   
   Paimon Java primary-key tables can use `bucket = -1` to grow the number of 
buckets as keys arrive. Paimon C++ recognizes `BucketMode::HASH_DYNAMIC`, but 
the table write path still rejects non-positive bucket counts outside postpone 
mode, and `BucketIdCalculator` rejects dynamic primary-key bucket calculation. 
The existing write restore path also supplies no dynamic-bucket index.
   
   Batch reading can reuse bucket IDs recorded in manifests, which is the scope 
of PR #499. Full support additionally needs stateful key routing, 
Java-compatible bucket-index persistence and recovery, and streaming reads that 
account for newly created buckets.
   
   This issue targets `HASH_DYNAMIC` primary-key tables, including partitioned 
tables whose primary keys contain all partition fields. Cross-partition upserts 
remain a separate roadmap item in #519.
   
   ### Solution
   
   Implement this in focused stages while preserving the existing fixed-bucket 
and append-table behavior:
   
   - [ ] **Complete batch-read support.** Reuse #493 / #494 / PR #499, track 
the batch-read PR through landing, and verify batch planning, predicate 
filtering, snapshot selection, and deletion-vector lookup across multiple 
partitions and buckets. Keep `HASH_DYNAMIC` distinct from append-table 
`BUCKET_UNAWARE` even though both configure `bucket = -1`.
   - [ ] **Add a stateful dynamic bucket assigner.** Restore the 
Java-compatible primary-key-hash-to-bucket mapping, reuse a key's assigned 
bucket for subsequent upserts and deletes, and allocate buckets for new keys 
according to the supported Java dynamic-bucket options, including 
`dynamic-bucket.target-row-num`. Define assignment ownership and supported 
writer topology consistently with Java.
   - [ ] **Persist and restore bucket indexes.** Implement compatible 
hash-index file reading/writing and manifest handling. Connect them to 
`RestoreFiles::DynamicBucketIndex()` and `FileSystemWriteRestore`, restoring 
only mappings visible in the selected committed snapshot. Keep dynamic-bucket 
indexes and deletion-vector indexes correctly scoped by partition and bucket.
   - [ ] **Integrate dynamic routing with writes and commits.** Add the 
stateful routing path to table writes and support insert, upsert, delete, 
flush, prepare-commit, commit, abort, and restart. Publish data, bucket-index 
changes, and applicable deletion-vector changes consistently in snapshots; a 
failed or retried commit must not lose or reassign committed keys.
   - [ ] **Preserve correctness through compaction and cleanup.** Retain valid 
key routing as data files are compacted or retired, and retain index files 
needed by supported snapshots and tags. Apply existing writer/concurrency 
constraints and explicitly validate unsupported combinations.
   - [ ] **Support streaming reads.** Discover newly appearing buckets while 
planning successive snapshots and preserve the existing supported 
incremental/changelog and deletion-vector semantics.
   - [ ] **Document the support matrix.** Describe supported reads and writes, 
Java-compatible options, assignment ownership, recovery behavior, and remaining 
restrictions. Update the current statement that primary-key dynamic-bucket 
writes are unsupported.
   
   Useful implementation entry points:
   
   - [Bucket mode 
definition](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/table/bucket_mode.h)
 and 
[resolution](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/table/bucket_mode.cpp).
   - [Table read/write 
construction](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/table/table.cpp)
 and [bucket 
calculation](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/bucket/bucket_id_calculator.cpp).
   - 
[RestoreFiles](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/operation/restore_files.h)
 and [filesystem write 
restore](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/operation/file_system_write_restore.h).
   - [Index manifest 
handling](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/manifest/index_manifest_file_handler.cpp)
 and the existing primary-key write/commit and deletion-vector paths.
   
   ### Anything else?
   
   Acceptance and validation:
   
   - [ ] Add focused assigner, index-file, restore, and commit unit tests, plus 
table-level integration tests.
   - [ ] Cover unpartitioned and partitioned tables, multiple buckets, bucket 
growth, repeated keys, hash collisions, upserts, deletes, and writer restart 
from committed snapshots.
   - [ ] Cover commit/abort/retry failures and compaction, ensuring that 
committed keys retain consistent routing and no duplicate live primary keys 
appear across buckets.
   - [ ] Cover deletion vectors enabled and disabled and the existing 
applicable primary-key merge modes.
   - [ ] Verify batch and streaming reads across snapshots that introduce new 
buckets, and confirm that fixed-bucket and append-unaware tables retain their 
existing behavior.
   - [ ] Cross-read Java-written tables from C++ and C++-written tables from 
Java, including bucket-index files, restored assignments, updates/deletes, and 
deletion vectors.
   - [ ] Update documentation and keep the parent roadmap item unchecked until 
the agreed full lifecycle is supported and validated.
   
   The existing batch-read PR is an initial delivery of this roadmap item. Its 
completion alone does not complete dynamic-bucket writes and streaming reads.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to