mailmindlin opened a new issue, #26262:
URL: https://github.com/apache/datafusion/issues/26262
### Describe the bug
`FFI_TableProvider::new` acceps a tokio `Handle`, and
`FFI_ExecutionPlan::execute` and record batch stream polling enter that
runtime. Other FFI_TableProvider methods (`scan`, `insert_into`, `delete_from`,
`update` and `truncate`) don't, and their futures are polled by the consumer's
executor which might not have an active tokio runtime.
As a result, any provider code that needs a tokio runtime during planning
fails when called through FFI. I hit this using `tokio::time::sleep` in
`scan()` to back off when an upstream server was overloaded. The sleep panics
with:
> there is no reactor running, must be called from the context of a Tokio
1.x runtime
The same applies to `tokio::spawn`, tokio I/O, or anything else that relies
on `Handle::current()`.
### To Reproduce
1. Implement a `TableProvider` whose `scan` uses tokio, for example:
```rust
async fn scan(&self, /* ... */) -> Result<Arc<dyn ExecutionPlan>> {
// Anything that needs the tokio runtime
tokio::time::sleep(Duration::from_millis(100)).await;
}
```
2. Wrap it with `FFI_TableProvider::new(provider, true,
Some(runtime.handle().clone()), ...)` and load it from a separate library or
consumer, such as datafusion-python.
3. Run a query against the table. Planning calls `scan` through the FFI
boundary, and the sleep panics because the provider's runtime isn't active on
the thread polling the future.
### Expected behavior
When a runtime is passed to `FFI_TableProvider::new`, the provider's async
methods should run inside it the same way `execute` and stream polling already
do.
Provider code should be able to use tokio timers, `spawn` and I/O.
### Additional context
_No response_
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
