polyzos opened a new issue, #3249: URL: https://github.com/apache/paimon/issues/3249
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation When creating an Append for Message Queue table, as depicted in the screenshot here:  we can notice the following: 1. 5 buckets are specified, but unless data comes in the bucket is not created 2. If you check the Kafka partitions; partition 3 has keys 2, 3 and 4 3. These keys though end up in different buckets 4. Paimon does a shuffle, even though the parallelism is the same because it doesn't do 1-1 mapping Because it is a Kafka-like message queue functionality, some users are confused, as they expect the same partitioning to happen and overall have a 1-1 mapping, between a Kafka partition and a paimon bucket. At the same time, I believe this is a really good enhancement and should also allow to remove the shuffle between the operators, thus improving performance. ### Solution _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
