JingsongLi commented on PR #11015: URL: https://github.com/apache/paimon/pull/11015#issuecomment-6854246912
**[P2] Preserve append fan-out for low-cardinality bucket keys** In [StripedBucketWriteChannelComputer.java:135–138](https://github.com/apache/paimon/blob/9a981431ae708d33a57e3e94ae080ccb5e89e29e/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/StripedBucketWriteChannelComputer.java#L135-L138), the append stripe depends only on the bucket key. Consequently, arbitrarily many append rows sharing a bucket key still go to one writer, even when their other columns differ. For a low-cardinality or hot tenant/status key, enabling striping can leave the original write bottleneck unchanged. I reproduced this on `9a981431` with JDK 8 / Flink 1.20, using an actual Flink job through `FlinkSinkBuilder`, followed by a table read. The table has columns `(k INT, v INT)`, no primary key, and these options: ```properties bucket=2 bucket-key=k file.format=avro write-only=true bucket-append-ordered=false sink.striped-bucket-write.enabled=true sink.striped-bucket-write.writers-per-bucket=2 sink.parallelism=4 ``` A single source emits 2,000 rows: `(k = i % 2, v = i)` for `i = 0..1999`. The real bucket extractor maps the two keys to different buckets. I also ran a control with the same input and storage/commit path, replacing only the routing policy with round-robin counters indexed by bucket number: | Routing policy | Channels used by bucket 0 | Channels used by bucket 1 | Data files per bucket | Rows read back | | --- | --- | --- | --- | --- | | Current bucket-key hash | `{0}` | `{2}` | `1, 1` | 2,000 | | Per-bucket round-robin control | `{0, 3}` | `{1, 2}` | `2, 2` | 2,000 | Thus, the current route uses only two of the four writers for this input. Both runs retain all rows; the issue is ineffective parallelism expansion. The updated documentation explicitly describes same-bucket-key writer locality, and the hash route does fan out higher-cardinality inputs. However, unordered append does not need primary-key merge locality, and the latest change replaces the earlier fixed-key `testAppendAlternatingBucketsFanOut` scenario with 64 distinct keys per bucket, leaving this previously covered case unsupported. Could we preserve that append fan-out with bounded routing state—for example, counters indexed only by fixed bucket number, with a single-stripe bypass—and restore the low-cardinality regression test? That keeps state at `O(bucket count)` without retaining entries for every partition. The existing 14 routing tests, 15 sink integration tests, and savepoint integration test all passed locally (30 tests total). The additional probe above exposes the missing case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
