yihua opened a new pull request, #829:
URL: https://github.com/apache/hudi-rs/pull/829
## Description
closes #827
Querying a Lance-based Hudi table through `HudiDataFusionDataSource` aborted
the Python interpreter with `there is no reactor running` from Lance's I/O
scheduler. datafusion-python polls the provider's stream across the
`datafusion-ffi` boundary on its own tokio runtime, and tokio's current-runtime
context belongs to each compiled copy, so Lance, which spawns I/O tasks, found
no runtime in the binding's copy; the panic surfaced inside an `extern "C"`
poll and could not unwind. Parquet reads never ask for a runtime, which is why
only Lance was affected.
The binding now passes its own runtime handle to `FFI_TableProvider`, which
datafusion-ffi enters around execution and every stream poll.
This covers execution and polling, which is where the Lance reader does its
I/O. datafusion-ffi does not enter the runtime around `scan`, so planning that
itself needs a runtime (cloud-storage listing, or a Lance table whose commit
metadata carries no schema) is still exposed; that is pre-existing and separate
from this fix.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`test_datafusion_ffi_reads_a_lance_table` reads the Lance fixture through
the FFI provider and compares the rows with `HudiTable.read()`. Without the fix
the test run aborts the pytest process; with it the whole Python suite passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
