JingsongLi commented on PR #11015:
URL: https://github.com/apache/paimon/pull/11015#issuecomment-6854246912

   **[P2] Preserve append fan-out for low-cardinality bucket keys**
   
   In 
[StripedBucketWriteChannelComputer.java:135–138](https://github.com/apache/paimon/blob/9a981431ae708d33a57e3e94ae080ccb5e89e29e/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/StripedBucketWriteChannelComputer.java#L135-L138),
 the append stripe depends only on the bucket key. Consequently, arbitrarily 
many append rows sharing a bucket key still go to one writer, even when their 
other columns differ. For a low-cardinality or hot tenant/status key, enabling 
striping can leave the original write bottleneck unchanged.
   
   I reproduced this on `9a981431` with JDK 8 / Flink 1.20, using an actual 
Flink job through `FlinkSinkBuilder`, followed by a table read. The table has 
columns `(k INT, v INT)`, no primary key, and these options:
   
   ```properties
   bucket=2
   bucket-key=k
   file.format=avro
   write-only=true
   bucket-append-ordered=false
   sink.striped-bucket-write.enabled=true
   sink.striped-bucket-write.writers-per-bucket=2
   sink.parallelism=4
   ```
   
   A single source emits 2,000 rows: `(k = i % 2, v = i)` for `i = 0..1999`. 
The real bucket extractor maps the two keys to different buckets. I also ran a 
control with the same input and storage/commit path, replacing only the routing 
policy with round-robin counters indexed by bucket number:
   
   | Routing policy | Channels used by bucket 0 | Channels used by bucket 1 | 
Data files per bucket | Rows read back |
   | --- | --- | --- | --- | --- |
   | Current bucket-key hash | `{0}` | `{2}` | `1, 1` | 2,000 |
   | Per-bucket round-robin control | `{0, 3}` | `{1, 2}` | `2, 2` | 2,000 |
   
   Thus, the current route uses only two of the four writers for this input. 
Both runs retain all rows; the issue is ineffective parallelism expansion.
   
   The updated documentation explicitly describes same-bucket-key writer 
locality, and the hash route does fan out higher-cardinality inputs. However, 
unordered append does not need primary-key merge locality, and the latest 
change replaces the earlier fixed-key `testAppendAlternatingBucketsFanOut` 
scenario with 64 distinct keys per bucket, leaving this previously covered case 
unsupported.
   
   Could we preserve that append fan-out with bounded routing state—for 
example, counters indexed only by fixed bucket number, with a single-stripe 
bypass—and restore the low-cardinality regression test? That keeps state at 
`O(bucket count)` without retaining entries for every partition.
   
   The existing 14 routing tests, 15 sink integration tests, and savepoint 
integration test all passed locally (30 tests total). The additional probe 
above exposes the missing case.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to