robertwb opened a new issue, #22129: URL: https://github.com/apache/beam/issues/22129
### What would you like to happen? By default, Beam automatically chooses the number of shards to write, but there are cases where these shards are either significantly larger or smaller than desired. Users can manually set the number of shards, but this comes at a(n often significant) performance cost as it forces all data to be reshuffled, and also requires guessing at the total output size to determine the correct number of shards. While creating two many small shards may be difficult to rectify (distinct bundles cannot write to the same shard), writing multiple shards in a single bundle is a straightforward way to enforce an upper bound on shard sizes. ### Issue Priority Priority: 2 ### Issue Component Component: io-py-common -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
