yihua opened a new issue, #712:
URL: https://github.com/apache/hudi-rs/issues/712
Reading a table over object storage through the DataFusion FFI table
provider panics instead of returning rows. Reading the same table through the
Python Table API works, and reading a local-filesystem table through the FFI
provider works, so it takes both the FFI boundary and a remote store to
reproduce.
```python
ctx = datafusion.SessionContext()
ctx.register_table("t",
HudiDataFusionDataSource("s3://hudi-demo/cow/v6_complexkeygen_hivestyle"))
ctx.sql("SELECT count(*) FROM t").collect()
```
```
thread '<unnamed>' panicked:
there is no reactor running, must be called from the context of a Tokio 1.x
runtime
tokio::time::sleep
reqwest::async_impl::client::Client::execute_request
object_store::client::http::connection::HttpClient::execute
<object_store::aws::client::S3Client as ...::GetClient>::get_request
...
datafusion_ffi::table_provider::scan_fn_wrapper::{{closure}}
async_ffi::LocalBorrowingFfiFuture<T>::new::poll_fn
```
`scan_fn_wrapper` returns the provider's `scan()` as an `FfiFuture`, and the
consumer polls it on its own executor, which has no tokio reactor. The S3
client arms a request timeout with `tokio::time::sleep`, which needs one. A
local store never asks for a timer, which is why only remote stores fail.
The `runtime: Option<Handle>` argument of
`FFI_TableProvider::new_with_ffi_codec` looks like the answer but is not:
datafusion-ffi only forwards that handle to the `FFI_ExecutionPlan` it returns
and never enters it around the scan future itself. Passing
`Some(rt().handle().clone())` there leaves the panic exactly where it was
(verified). The fix has to put the scan body on a runtime this crate controls,
for example spawning it on `rt()` and awaiting the join handle.
Reproduced against MinIO on `main` at eac90708 (arrow 57, DataFusion 52,
object_store 0.12, datafusion-python 52.3.0) and, identically, on the arrow 58
/ DataFusion 54 / object_store 0.13 branch in #711, so this is not tied to a
dependency version.
Nothing covers this combination today: the `hudi-table-api/python`
integration app reads S3 through the Table API rather than the FFI provider,
and #711 adds FFI tests that use a local table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
