stuhood commented on issue #13692: URL: https://github.com/apache/datafusion/issues/13692#issuecomment-2580867595
> My concern here is that this thread switch (i.e. the tokio worker gets a new thread from the pool) is potentially expensive, esp. in a hot loop. It's also a potential synchronization point, because the thread pool is shared across all workers. So while this may be Ok if you do that occasionally, I don't think this is gonna yield good results if you do that e.g. for every record batch. Sorry, but to be 1000% clear: `block_in_place` _does not_ context switch. It cannot. The closure it runs _cannot_ be moved to another thread, because it doesn't require `Send`/etc. That generally makes it more convenient to use as well. The reason that additional threads might be spawned is not for the task which is running a line of code which hit a `block_in_place`: rather, so that _other_ tasks that are currently running on the executor do not starve. So: `block_in_place` does not / cannot cause a context switch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
