yunfengzhou-hub opened a new pull request, #11015: URL: https://github.com/apache/paimon/pull/11015
### Purpose For a fixed-bucket table the sink shuffles records by bucket, so each bucket is written by a single writer subtask and write parallelism is capped by the bucket count. When the bucket count cannot be rescaled, writer subtasks beyond that count stay idle. This PR adds an opt-in `sink.striped-bucket-write.enabled` that lets a fixed-bucket table raise write parallelism beyond the bucket count without rescaling the table. It applies to both primary-key and append-only fixed-bucket tables. - Primary-key tables keep key locality: records sharing a primary key still land on the same writer, so each writer produces sorted runs over a hash-partitioned subset of the bucket's keys. - Append-only tables distribute records across the writers round-robin. - `enabled` is the only switch. The stripe count is derived from the writer parallelism and bucket count, so users normally only decide whether to turn it on. - `sink.striped-bucket-write.writers-per-bucket` optionally overrides the stripe count; `-1` lets every writer subtask write every bucket. It only takes effect when enabled. Striped write requires `write-only=true` with a dedicated compaction job, and rejects `deletion-vectors.enabled`. Multiple writers per bucket cannot run inline compaction or maintain a per-bucket deletion-vector index without conflicting commits, so these misconfigurations are rejected when the sink is built. Dynamic, unaware and postpone bucket modes ignore the option. Changing writer parallelism or writers-per-bucket when restoring from a savepoint loses data, because each bucket's writer states are redistributed by bucket and states for the same bucket overwrite each other. This is documented; start a fresh job rather than rescaling a striped-write savepoint. ### Tests - `StripedBucketWriteChannelComputerTest`: single-writer-per-bucket routing equals the default computer, each bucket fans to exactly K disjoint channels, same key always maps to the same channel, balanced distribution, auto stripe count with ceil when not divisible, and partitioned tables. - `StripedBucketWriteSinkITCase`: end-to-end primary-key upsert dedup and append-only correctness, writer parallelism expansion, guard rejection of write-only-off and deletion-vectors, disabled path keeping the bucket cap, writers-per-bucket ignored when disabled and rejected when zero, and dynamic/unaware modes ignoring the option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
