tgravescs commented on issue #22173: [SPARK-24355] Spark external shuffle server improvement to better handle block fetch requests. URL: https://github.com/apache/spark/pull/22173#issuecomment-571193949 sorry I'm not understanding quite what you are saying and I think perhaps you mean fetch requests in "async" mode? There are now 2 thread pools, the fetch requests go to the new thread pool, but the response does still have to go back through the original evenLoop Group, so the fetches are somewhat async now and like the comment for the "flush" function mentioned (see the comment: https://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java#L118). it does add in an await() call there to help with some throttling so it didn't take up all the threads again and perhaps that is causing some slowness if the event loop channel its registered to is also processing some incoming events in between. If can get more information from your workload to see what the threads are doing or have test case to reproduce that would be very helpful
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
