cloud-fan commented on issue #22173: [SPARK-24355] Spark external shuffle server improvement to better handle block fetch requests. URL: https://github.com/apache/spark/pull/22173#issuecomment-571202152 When I say sync mode, I mean a thread that handles fetch request has to finish reading the shuffle blocks and writing the response to channel, before handling the next requests, although the channel writing is done by another thread pool. Previously it's fully async: the thread can handle the next request once it finishes reading the shuffle blocks of the current request. That said, the throughput of handling fetch requests is reduced now. Anyway let me come up with a microbenchmark first.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
