timsaucer opened a new pull request, #13937: URL: https://github.com/apache/datafusion/pull/13937
## Which issue does this PR close? Closes #13827 Closes #13851 ## Rationale for this change During testing with `delta-kernel-rs` and `datafusion-table-providers` it was discovered that there are multiple cases where table providers are spawning processes using tokio. Since these modules may be running in a different thread than the main executable, they do not have the tokio runtime. Here we are optionally adding in a tokio runtime reference so we an enter the runtime during calls to the record batch stream. In a recent change to DataFusion core, there has been a required change in the API for operating over the FFI boundary. With this change, it will break when someone attempts to load a module from DF43 into DF44. This PR addresses this by adding a `version()` function that will return the major version for which the module was compiled against. This will allow users to check for compatibility before attempting to use modules across the boundary. ## What changes are included in this PR? We add a reference to the tokio runtime that is entered during calls to the record batch stream so we can spawn async processes during execution without panicing. We add a single `version()` function at the root of the FFI API. We add an integration test that builds a module and loads it from another executable to test the FFI boundary. This has both a synchronous and an asynchronous table provider. ## Are these changes tested? These are tested against datafusion-python with a python wrapper around delta-kernel-rs. Additionally an integration test is added to this repository. ## Are there any user-facing changes? This exposes a new `version()` function in the FFI module and adds an new method for creating `FFI_TableProvider` with the runtime. The existing `From<>` implementation still remains if the user is not spawning via tokio. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
