yunfengzhou-hub opened a new pull request, #11015:
URL: https://github.com/apache/paimon/pull/11015

   ### Purpose
   
   For a fixed-bucket table the sink shuffles records by bucket, so each bucket 
is written by a single writer subtask and write parallelism is capped by the 
bucket count. When the bucket count cannot be rescaled, writer subtasks beyond 
that count stay idle.
   
   This PR adds an opt-in `sink.striped-bucket-write.enabled` that lets a 
fixed-bucket table raise write parallelism beyond the bucket count without 
rescaling the table. It applies to both primary-key and append-only 
fixed-bucket tables.
   
   - Primary-key tables keep key locality: records sharing a primary key still 
land on the same writer, so each writer produces sorted runs over a 
hash-partitioned subset of the bucket's keys.
   - Append-only tables distribute records across the writers round-robin.
   - `enabled` is the only switch. The stripe count is derived from the writer 
parallelism and bucket count, so users normally only decide whether to turn it 
on.
   - `sink.striped-bucket-write.writers-per-bucket` optionally overrides the 
stripe count; `-1` lets every writer subtask write every bucket. It only takes 
effect when enabled.
   
   Striped write requires `write-only=true` with a dedicated compaction job, 
and rejects `deletion-vectors.enabled`. Multiple writers per bucket cannot run 
inline compaction or maintain a per-bucket deletion-vector index without 
conflicting commits, so these misconfigurations are rejected when the sink is 
built. Dynamic, unaware and postpone bucket modes ignore the option.
   
   Changing writer parallelism or writers-per-bucket when restoring from a 
savepoint loses data, because each bucket's writer states are redistributed by 
bucket and states for the same bucket overwrite each other. This is documented; 
start a fresh job rather than rescaling a striped-write savepoint.
   
   ### Tests
   
   - `StripedBucketWriteChannelComputerTest`: single-writer-per-bucket routing 
equals the default computer, each bucket fans to exactly K disjoint channels, 
same key always maps to the same channel, balanced distribution, auto stripe 
count with ceil when not divisible, and partitioned tables.
   - `StripedBucketWriteSinkITCase`: end-to-end primary-key upsert dedup and 
append-only correctness, writer parallelism expansion, guard rejection of 
write-only-off and deletion-vectors, disabled path keeping the bucket cap, 
writers-per-bucket ignored when disabled and rejected when zero, and 
dynamic/unaware modes ignoring the option.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to