stuhood commented on issue #13692: URL: https://github.com/apache/datafusion/issues/13692#issuecomment-2578874963
(take with a grain of salt because I haven't worked with `block_in_place` inside of hot loops) > The performance implications definitely concern me, I have a nagging suspicion `block_in_place` spawns a thread... `block_in_place` doesn't spawn a thread for the work that it scopes. As mentioned in the description, it can't: it doesn't place any lifetime/`Send` bounds on the closure that is running, and so the closure cannot be moved to another thread. But it _is_ possible that using `block_in_place` in enough positions will cause the `tokio` runtime to need to spin up more threads on the runtime (subject to the [`worker_threads` and `max_blocking_threads`](https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.worker_threads) limits). To the extent that you saw a slowdown with `block_in_place`, I expect that the `tokio` maintainers would like to know about it: because last I checked, it was used internally in a few different `tokio` codepaths to wrap syscalls which didn't have non-blocking variants. Additionally, as mentioned in its docstring, `block_in_place` can _remove_ parallelism if it isn't used in concert with additional calls to `spawn`: for example, if you have a call to `join` like: ```rust let (res1, res2) = join!( async { task::block_in_place(...); }, async { // something async ... }, ); ``` ... then the two async blocks may end up executing sequentially, because the entire task running the `join` is blocked. To fix it, you would `spawn` more tasks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
