otterc commented on issue #22173: [SPARK-24355] Spark external shuffle server improvement to better handle block fetch requests. URL: https://github.com/apache/spark/pull/22173#issuecomment-577936282 I think `await` does't provide any benefit and could be removed. When the chunk fetch event loop runs ``` channel.writeAndFlush(result) ``` This adds a `WriteAndFlushTask` in the pendingQueue of the default server-IO thread registered with that channel. The code in `NioEventLoop.run()` itself throttles the number of tasks that can be run at a time from its pending queue. Here is the code: ``` final long ioStartTime = System.nanoTime(); try { processSelectedKeys(); } finally { // Ensure we always run tasks. final long ioTime = System.nanoTime() - ioStartTime; runAllTasks(ioTime * (100 - ioRatio) / ioRatio); } } ``` Here it records how much time it took to perform the IO operations, that is, execute `processSelectedKeys()`. `runAllTasks`, which is the method that processes the tasks from pendingQueue, will be performed for the same amount of time. `runAllTasks()` does process 64 tasks and then checks the time. ``` // Check timeout every 64 tasks because nanoTime() is relatively expensive. // XXX: Hard-coded value - will make it configurable if it is really a problem. if ((runTasks & 0x3F) == 0) { lastExecutionTime = ScheduledFutureTask.nanoTime(); if (lastExecutionTime >= deadline) { break; } } ``` This ensures that the default server-IO thread always gets time to process the ready channels. Its not always busy processing `WriteAndFlushTask`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
