tustvold commented on PR #9605: URL: https://github.com/apache/arrow-datafusion/pull/9605#issuecomment-1997942978
So yes in an ideal world all CPU computation would be spawned to rayon or a similar blocking threadpool as in this PR. However, unfortunately this isn't the way DF has been implemented. Instead the compromise we use in IOx is to just spawn the IO off to a separate runtime, and accept that the runtime DF is running in will not have good poll latencies. Therefore rather than spawning the CPU bound work, we instead want to spawn the IO of the multipart upload. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
