Victsm commented on issue #22173: [SPARK-24355] Spark external shuffle server improvement to better handle block fetch requests. URL: https://github.com/apache/spark/pull/22173#issuecomment-578323514 When we worked on the original fix in #21402, we were holding a wrong view of how Netty handles both event loop groups. We thought both the chunk fetch requests and the control plane RPCs are going to become tasks placed in the task queues of threads inside the server I/O event loop group. This was the reason we used `sync` or `await` in the patch, so that we are able to limit the # `WriteAndFlushTask` placed by chunk fetch requests in the task queues of threads in server I/O event loop group, which would allow the control plane RPCs to be picked up by the I/O threads in time to avoid timeout. More recently, when working in SPARK-30512, we get a closer and much more correct view of how Netty handles these 2 event loop groups (the chunk fetch request dedicated one and the I/O event loop group). As mentioned by @otterc, Netty should have already processed the data plane RPCs and control plane RPCs separately after we set up the dedicated event loop for chunk fetch requests. Thus, we don't really need to throttle the # tasks placed by chunk fetch requests in the task queues of threads in I/O event loop group. We have an internal stress testing framework that can help to validate if the original timeout issue still occurs after removing `await`. We will do the test to validate things are working as expected. If so, we may take the benefit of this fix without generating the performance regression.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
