This is an automated email from the ASF dual-hosted git repository. timsaucer pushed a commit to branch feat/ffi-with-extensions in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
commit 53453ea5bb7662f334bf8c2a098f3bafa6d383ca Author: Tim Saucer <[email protected]> AuthorDate: Fri Aug 7 14:19:06 2026 -0400 Document and test the context-outlives-DataFrame contract A DataFrame does not keep its SessionContext alive. FFI components hold a weak task-context provider, so operations that reach an FFI codec after the context is collected fail with a clean out-of-scope error rather than crashing. Lock that behavior in with a test and document the ownership contract in the FFI guide and with_extensions docstring. Co-Authored-By: Claude Fable 5 <[email protected]> --- docs/source/contributor-guide/ffi.md | 7 +++++++ .../python/tests/_test_three_library_query_planner.py | 19 +++++++++++++++++++ python/datafusion/context.py | 5 +++++ 3 files changed, 31 insertions(+) diff --git a/docs/source/contributor-guide/ffi.md b/docs/source/contributor-guide/ffi.md index 17c58826..a51e979e 100644 --- a/docs/source/contributor-guide/ffi.md +++ b/docs/source/contributor-guide/ffi.md @@ -327,6 +327,13 @@ cache bound components, and do not retain the context passed in. Catalogs are sh with the source context, so registrations made during binding are not rolled back on failure. +The returned context is the strong owner of every installed component's task-context +provider, and dependent objects do not extend its lifetime. A `DataFrame`, logical +plan, or capsule can outlive the context, but any operation that reaches an FFI codec +after the context is collected fails with `TaskContextProvider went out of scope over +FFI boundary`. Keep the context alive for as long as objects derived from it are in +use. + `MyPlannerExtension` in [`datafusion-ffi-query-planner-example`] is a complete Rust implementation of this protocol, including extracting the task-context provider from the supplied context and constructing a Python `SessionExtensionComponents`. diff --git a/examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py b/examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py index 5846661f..2fde3e2d 100644 --- a/examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py +++ b/examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py @@ -286,6 +286,25 @@ def test_with_extensions_failure_leaves_source_usable(): assert batches[0].column(0).to_pylist() == [0, 1, 2, 3, 4, 5] +def test_dataframe_outliving_context_fails_cleanly(): + """A DataFrame does not keep its SessionContext alive. FFI components + resolve the task context through a weak reference, so using the + DataFrame after dropping the context raises a clean error instead of + crashing. This locks in the documented ownership contract: the context + must outlive DataFrames that depend on FFI codecs.""" + config = SessionConfig().with_extension(PlannerConfig(max_rows=2)) + ctx = SessionContext(config).with_extensions( + ProviderCodecsExtension(), MyPlannerExtension() + ) + ctx.register_table("numbers", MyTableProvider(1, 6, 1)) + df = ctx.sql('SELECT "A" FROM numbers ORDER BY "A"') + del ctx + gc.collect() + + with pytest.raises(Exception, match="went out of scope"): + df.collect() + + def test_composed_codecs_with_query_planner(): """A second pair of codecs installed on top of the provider codecs composes with them instead of replacing them. The extra codecs diff --git a/python/datafusion/context.py b/python/datafusion/context.py index ce6bbd53..19b3191c 100644 --- a/python/datafusion/context.py +++ b/python/datafusion/context.py @@ -1880,6 +1880,11 @@ class SessionContext: registering tables or otherwise mutating the context during binding is not rolled back on failure. + The returned context is the strong owner of the installed components' + task-context providers. Keep it alive for as long as DataFrames or + plans derived from it are in use; FFI operations after the context is + collected raise an error. + Args: extensions: One or more objects implementing ``__datafusion_session_extension__`` (see --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
