Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19788
I think we can do this better, to make it a purely server-side
optimization. The shuffle protocol can already fetch multiple blocks in one
request, i.e. the `OpenBlocks` request.
The `OneForOneBlockFetcher`(client side) sends `OpenBlocks` request to
`NettyBlockRpcServer`(serve side), and use a callback to process the fetched
data. The server side can detect if the requested blocks are continuous shuffle
blocks, and then optimize it into one IO request.
Then this is totally compatible for the shuffle service.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
