naikaayush opened a new pull request, #26944: URL: https://github.com/apache/datafusion/pull/26944
## Which issue does this PR close? https://github.com/apache/datafusion/issues/19634 Related discussion: https://github.com/apache/datafusion-iceberg/pull/45#discussion_r4370619490 opening this as a draft to discuss the API direction ## Rationale for this change schema providers backed by remote catalogs may need async io when adding or removing tables. currently, `SchemaProvider` register_table and deregister_table methods are sync, so we cannot await those operations directly this proposes making both methods async to work with remote catalogs, including Iceberg REST catalogs ## What changes are included in this PR? - Make SchemaProvider::register_table and deregister_table async. - Propagate async through SessionContext::register_table, deregister_table, and register_batch, including SQL DDL callers. - Release the session state lock before awaiting schema providers. - Update the FFI bridge to return futures for registration and removal. - Update implementations, callers, tests, examples, benchmarks, and docs The change touches 152 files; most edits propagate async and add await at existing call sites. ## What is the testing strategy for this PR? add a public API test using a schema provider that yields during registration and removal. it checks that the session lock is released, the futures remain Send, and CREATE/REPLACE/DROP operations work. Add a cross-library FFI test covering registration, querying registered tables, error propagation, and removal. Local validation passed: - Workspace check across all targets. - Clippy across all targets and features with warnings denied. - Catalog and session-context tests. - FFI unit and integration tests. - Documentation tests, Rust formatting, and documentation formatting. ## Are there any user-facing changes? Yes. This is a breaking Rust API and FFI change. `SchemaProvider` implementations must make register_table and deregister_table async. Callers must await these methods and the SessionContext registration/removal helpers, including register_batch. FFI consumers must rebuild against the updated interface -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
