LuciferYang opened a new issue, #10103: URL: https://github.com/apache/paimon/issues/10103
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master ### Compute Engine Flink (dynamic-bucket table write) ### Minimal reproduce step Write a dynamic-bucket table with `dynamic-bucket.max-buckets` set to a positive value smaller than the writer parallelism, for example `dynamic-bucket.max-buckets = 2` with parallelism 4. Buckets are owned by `bucket % numAssigners == assignId % numAssigners`, so the assigner whose id reaches or exceeds the cap owns no bucket at all under that cap. Send it a record on a fresh partition. ### What doesn't meet your expectations? The `dynamic-bucket.max-buckets` value is only validated against `[1, MAX_DYNAMIC_BUCKETS]`, never against the writer parallelism. When it is smaller, an assigner that owns no admissible bucket fails in a way that hides the cause: - Loaded path (`HashBucketAssigner` / `PartitionIndex.assign`): on a fresh partition no bucket can be created under the cap, so the final random pick runs over an empty bucket list and throws `java.lang.IllegalArgumentException: list is empty`, which says nothing about the misconfiguration. - Overwrite path (`SimpleHashBucketAssigner`): its `loadNewBucket()` fails to find an owned bucket under the cap, but the return value is ignored, so `currentBucket` silently stays at the default 0. That bucket belongs to another assigner, and the assigner writes all of its records there with no error. Expected: a `max-buckets` value that leaves an assigner with no bucket is reported clearly, naming the option, instead of an opaque `list is empty` or a silent write to a wrong bucket. ### Anything else? Fix direction: `PartitionIndex.assign` throws a message naming `dynamic-bucket.max-buckets` when its bucket list is empty; `SimpleHashBucketAssigner` rejects at construction a cap that does not admit this assigner's smallest owned bucket (`assignId % numAssigners`). An assigner that already owns buckets beyond a later-lowered cap keeps working. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
