This is an automated email from the ASF dual-hosted git repository. timsaucer pushed a commit to branch feat/distributed-extensions-example in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
commit a06e53b808e79984ef4e35a5bed7b650b1933c53 Author: Tim Saucer <[email protected]> AuthorDate: Wed Sep 9 14:18:12 2026 -0400 Wire the distributed example into CI, and say which example is which Three maturin builds and three test invocations for #1719, plus the documentation change that keeping three example trees requires. The plan had been to retire `datafusion-ffi-query-planner-example` and fold it into the new engine. That is now off, on evidence from building the engine: roughly fifteen of its forty-seven tests cover planner *layering*, and `dfx_engine`'s planner structurally cannot delegate to a `fallback`. Delegating hands physical planning back to the host, which returns opaque `ForeignExecutionPlan` nodes the engine can neither serialize nor split — a stage-splitting planner has to plan for itself. So the new example has nothing for those tests to nest, and deleting the crate would delete real coverage of the most subtle part of #1679's contract. Three trees then, with distinct jobs, which the guide now states up front rather than leaving a reader to infer: `examples/distributed` is the worked example and the place to start; `datafusion-ffi-example` is the capsule-protocol test bed, one of every hook exercised hard; and `datafusion-ffi-query-planner-example` is the planner-composition test bed. The guide's "three roles in a query" section described only the latter two. Two stale claims fixed while in there. `examples/README.md` linked three `sql-on-*.py` files that do not exist. The planner example's README said its planner "owns no serializable types of its own and deliberately uses only built-in physical nodes", which stopped being true when `DistributedExec` was added — and the sentence mattered, because owning a node is exactly why that library ships its codec and planner as one bundle. The `actionlint` pre-commit hook needs Docker and could not run here; the workflow files are otherwise lint-clean and parse as YAML. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> --- .github/workflows/build.yml | 35 ++++++++++++++++++++++ .github/workflows/test.yml | 9 ++++++ docs/source/extension-guide/index.md | 27 +++++++++++++---- examples/README.md | 24 ++++++++------- .../datafusion-ffi-query-planner-example/README.md | 4 +-- 5 files changed, 81 insertions(+), 18 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index d7af9b66..4ed5782b 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -206,6 +206,38 @@ jobs: args: --out dist rustup-components: rust-std + # The three libraries of the distributed example. Built in dependency + # order for readability only; they are independent cdylibs. + - name: Build distributed example storage library + if: matrix.python-tag == 'abi3' + uses: PyO3/maturin-action@v1 + with: + target: x86_64-unknown-linux-gnu + manylinux: "2_28" + working-directory: examples/distributed/storage-library + args: --out dist + rustup-components: rust-std + + - name: Build distributed example UDF library + if: matrix.python-tag == 'abi3' + uses: PyO3/maturin-action@v1 + with: + target: x86_64-unknown-linux-gnu + manylinux: "2_28" + working-directory: examples/distributed/udf-library + args: --out dist + rustup-components: rust-std + + - name: Build distributed example engine library + if: matrix.python-tag == 'abi3' + uses: PyO3/maturin-action@v1 + with: + target: x86_64-unknown-linux-gnu + manylinux: "2_28" + working-directory: examples/distributed/engine-library + args: --out dist + rustup-components: rust-std + - name: Archive wheels uses: actions/upload-artifact@v7 with: @@ -220,6 +252,9 @@ jobs: path: | examples/datafusion-ffi-example/dist/* examples/datafusion-ffi-query-planner-example/dist/* + examples/distributed/storage-library/dist/* + examples/distributed/udf-library/dist/* + examples/distributed/engine-library/dist/* # ============================================ # Build - Linux ARM64 diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 047b3503..c1efe0ba 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -127,6 +127,15 @@ jobs: uv run --no-project pytest python/tests/_test*.py cd ../datafusion-ffi-query-planner-example uv run --no-project pytest python/tests/_test*.py + # The distributed example. Its tests spawn worker processes with + # `sys.executable`, so they need the same interpreter the wheels + # were installed into -- which `uv run` gives them. + cd ../distributed/storage-library + uv run --no-project pytest python/tests/_test*.py + cd ../udf-library + uv run --no-project pytest python/tests/_test*.py + cd ../engine-library + uv run --no-project pytest python/tests/_test*.py - name: Run tpchgen-cli to create 1 Gb dataset if: matrix.wheel-tag == 'abi3' diff --git a/docs/source/extension-guide/index.md b/docs/source/extension-guide/index.md index 91d42a3c..dce944de 100644 --- a/docs/source/extension-guide/index.md +++ b/docs/source/extension-guide/index.md @@ -59,11 +59,27 @@ this section only makes sense once they are distinct in your head: the codecs that serialize them. - **A planner library** — owns a query planner and the configuration it needs. -The worked examples in this repository use two separate crates, -[`datafusion-ffi-example`] and [`datafusion-ffi-query-planner-example`], so -each role has a distinct shared-library identity. A real library may play more -than one role; keeping them separate in the examples is what makes the -boundaries observable. +A real library may play more than one role. Keeping them in separate crates is +what makes the boundaries observable, because each one is then a distinct +shared library and the FFI conversions are real rather than same-library +downcasts. + +The examples in this repository are three trees, and it is worth knowing which +one answers your question: + +- [`examples/distributed`] is the **worked example**, and the place to start. + Three libraries — functions, tables, and an engine — cooperate on one query + whose leaf stage runs in separate worker processes. It is also the only + example whose plans genuinely leave the process, so it is where the codecs + encode durable metadata rather than tokens. +- [`datafusion-ffi-example`] is the **capsule-protocol test bed**: one of every + hook, exercised hard. Read it to see the shape of a getter, not to see a + library designed well. +- [`datafusion-ffi-query-planner-example`] is the **planner-composition test + bed**: what happens when more than one library contributes a query planner, + and how `fallback` nests them. The distributed example cannot cover this — + a planner that rewrites the plan into stages has to plan for itself, so it + has no use for a fallback. The session owns the codecs used for the exchange and supplies them to the foreign planner. That is what lets the planner decode provider-owned objects, @@ -134,3 +150,4 @@ checklist [`datafusion-ffi-example`]: https://github.com/apache/datafusion-python/tree/main/examples/datafusion-ffi-example [`datafusion-ffi-query-planner-example`]: https://github.com/apache/datafusion-python/tree/main/examples/datafusion-ffi-query-planner-example +[`examples/distributed`]: https://github.com/apache/datafusion-python/tree/main/examples/distributed diff --git a/examples/README.md b/examples/README.md index 7bbb45dc..e38932ae 100644 --- a/examples/README.md +++ b/examples/README.md @@ -51,23 +51,25 @@ Here is a direct link to the file used in the examples: ### Rust FFI Extensions -- [Table providers, functions, and codecs](./datafusion-ffi-example/) -- [Independent query planner and planner configuration](./datafusion-ffi-query-planner-example/) +Start with the worked example; the other two are focused test beds that +exercise one part of the protocol hard rather than reading as a tutorial. -These two crates form a three-library interoperability example with -`datafusion-python`. They are separate shared libraries so the tests exercise real FFI -type and codec boundaries rather than same-library Rust downcasts. +- [**Three libraries in one distributed query**](./distributed/) — a UDF + library, a table provider with its own scan node, and a toy engine that + splits the plan and runs each partition in a separate process. Read this one + first. +- [Capsule protocol conformance](./datafusion-ffi-example/) — table providers, + catalogs, functions, config, and codecs, one of each. +- [Query planner composition](./datafusion-ffi-query-planner-example/) — what + happens when more than one library contributes a planner, and how they nest. + +Each is a separate shared library, so the tests exercise real FFI type and +codec boundaries rather than same-library Rust downcasts. ### Substrait Support - [Serialize query plans using Substrait](./substrait.py) -### Executing SQL against DataFrame Libraries (Experimental) - -- [Executing SQL on Polars](./sql-on-polars.py) -- [Executing SQL on Pandas](./sql-on-pandas.py) -- [Executing SQL on cuDF](./sql-on-cudf.py) - ## TPC-H Examples Within the subdirectory `tpch` there are 22 examples that reproduce queries in diff --git a/examples/datafusion-ffi-query-planner-example/README.md b/examples/datafusion-ffi-query-planner-example/README.md index af128886..e7c105e2 100644 --- a/examples/datafusion-ffi-query-planner-example/README.md +++ b/examples/datafusion-ffi-query-planner-example/README.md @@ -76,8 +76,8 @@ ctx.register_udf(provider_udf) ctx.set_query_planner(MyQueryPlanner()) ``` -`MyPlannerConfig` is transferred through the foreign session. `MyQueryPlanner` reads `ffi_query_planner.max_rows`, creates the plan with `DefaultPhysicalPlanner`, and adds a built-in `GlobalLimitExec`. The test changes the setting with `SET` and verifies the new row limit. +`MyPlannerConfig` is transferred through the foreign session. `MyQueryPlanner` reads `ffi_query_planner.max_rows`, creates the plan with `DefaultPhysicalPlanner`, adds a built-in `GlobalLimitExec`, and wraps the result in a `DistributedExec` of its own. The test changes the setting with `SET` and verifies the new row limit. -The provider's codec chain is attached to the planner when it is installed and is also used to decode the returned physical plan in `datafusion-python`. Extension codecs compose: each `with_logical_extension_codec` / `with_physical_extension_codec` call appends to the session's codec chain, and each payload records which codec wrote it, so several libraries can install codecs on the same session and the order between them does not affect decoding. This planner owns no serializable types [...] +The provider's codec chain is attached to the planner when it is installed and is also used to decode the returned physical plan in `datafusion-python`. Extension codecs compose: each `with_logical_extension_codec` / `with_physical_extension_codec` call appends to the session's codec chain, and each payload records which codec wrote it, so several libraries can install codecs on the same session and the order between them does not affect decoding. This planner does own a node of its own [...] For the limits behind that choice — how the codec chain dispatches, which node kinds survive the boundary, and what a derived context shares with the context it came from — see the [Extension Guide](https://datafusion.apache.org/python/extension-guide/index.html). --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
