This is an automated email from the ASF dual-hosted git repository. HyukjinKwon pushed a commit to branch fix-ci-actions in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
commit 8bfb59f6ba408b853bd83f7afd3047ae78079a79 Author: Hyukjin Kwon <[email protected]> AuthorDate: Tue Aug 25 13:00:06 2026 +0900 [SPARK-58983] Run the WASM packer against the reference pyspark in CI The `wasm-udf` job's "Packer end-to-end" step pointed SPARK_CONNECT_WASM_PACKER_PATH at ./python, which put the Rust-backed `pyspark` skin on the packer's import path. `pyspark_wasm_udf.pack` needs `pyspark.cloudpickle` and `pyspark.sql.types._parse_datatype_json_value`, and must emit a reference `DataType` that executors unpickle -- none of which the skin provides. Isolate `pyspark_wasm_udf` in a `.packer` dir so `import pyspark` resolves to the reference pyspark installed earlier in the job. Also refresh the pack.py docstring to say we use the reference `pyspark.cloudpickle`. Co-authored-by: Isaac <[email protected]> --- .github/workflows/rust.yml | 13 +++++++++++-- python/pyspark_wasm_udf/pack.py | 8 ++++---- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index d693d59..8d22123 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -122,6 +122,15 @@ jobs: cargo build -p examples --bin wasm_udf_macro --features examples/wasm-udf-macro - name: Packer end-to-end (Rust -> cloudpickle command) + # The packer runs `python -m pyspark_wasm_udf.pack`, which imports the + # reference `pyspark` (installed above) for `pyspark.cloudpickle` and + # `pyspark.sql.types._parse_datatype_json_value`, and produces a + # reference `DataType` the executors unpickle. Put *only* the + # `pyspark_wasm_udf` package on the packer path (not ./python, whose + # `pyspark` skin is Rust-backed and would shadow the reference). env: - SPARK_CONNECT_WASM_PACKER_PATH: ${{ github.workspace }}/python - run: cargo test -p apache-spark-connect --features wasm-udf -- --ignored packer_produces + SPARK_CONNECT_WASM_PACKER_PATH: ${{ github.workspace }}/.packer + run: | + mkdir -p .packer + cp -r python/pyspark_wasm_udf .packer/ + cargo test -p apache-spark-connect --features wasm-udf -- --ignored packer_produces diff --git a/python/pyspark_wasm_udf/pack.py b/python/pyspark_wasm_udf/pack.py index 9b3c305..a38a7cb 100644 --- a/python/pyspark_wasm_udf/pack.py +++ b/python/pyspark_wasm_udf/pack.py @@ -35,10 +35,10 @@ generically with :func:`pyspark.sql.types._parse_datatype_json_value`, so any Spark type is supported without a token table. The command mirrors ``pyspark.sql.connect.expressions.PythonUDF``: -``cloudpickle.dumps((func, output_type))``. We use the standalone PyPI -``cloudpickle`` (byte-for-byte the same as ``pyspark.cloudpickle``) and register -:mod:`pyspark_wasm_udf` for pickling **by value**, so executors need not have -this package installed. +``cloudpickle.dumps((func, output_type))``. We use the bundled +``pyspark.cloudpickle`` (the same cloudpickle the reference client ships) and +register :mod:`pyspark_wasm_udf` for pickling **by value**, so executors need not +have this package installed. """ import base64 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
