cozos commented on PR #958: URL: https://github.com/apache/beam/pull/958#issuecomment-1329966457
Thanks for the quick reply many years after the PR was made @robertwb! Can you shine some light on why a side input with many shards is slow? The original comment says: > The benefit we gain here from reshuffling is that we are able to reduce the read latency that comes from oversharding for a large collection of small elements [i.e. gcs paths](through prefetching and skipping empty shards) I am inferring that this is on Dataflow and that before being broadcast, a side input has to be assembled from many different shards from many different machines, over the network. And a shuffle is faster because... its somehow faster than the "collect side input from shards" process. And then once the side input is nicely collected on a contiguous block of memory on the same machine, it can be broadcasted in a pipelined manner via prefetching, thus decreasing read latency on the ptransforms consuming the side inputs... Is that what he meant? Or is it something else? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
