This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new f0ff832d [SPARK-61141][CONNECT] Run Rust WASM UDFs once per Arrow batch
f0ff832d is described below

commit f0ff832d3e4ef887751f8af31b7a55a59282ed7f
Author: Tim Meehan <[email protected]>
AuthorDate: Sun Nov 8 10:37:24 2026 +0900

    [SPARK-61141][CONNECT] Run Rust WASM UDFs once per Arrow batch
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61141](https://issues.apache.org/jira/browse/SPARK-61141): 
`#[spark_wasm_udf]` functions whose arguments and result are all `i32`, `i64`, 
`f32`, `f64`, `bool`, `String`, `Vec<u8>`, or one `Option` of those are now 
called once per Arrow batch instead of once per row.
    
    - **Macro** (`spark-connect-macros`): for such a function it also emits a 
`spark_batch_udf_<fn>` export, and the generated `udf::<fn>_udf()` builder 
calls `.with_batch_entrypoint(...)`. Other signatures (arrays, nested 
`Option`s) are unchanged. The batch export names use a prefix of their own, so 
they cannot collide with the per-row `spark_udf_<fn>` exports.
    - **Build runtime** (`spark-connect-build`): adds the batch ABI to the 
runtime compiled into every module. The host writes a descriptor and the Arrow 
buffers of each argument column (validity, values, offsets) into a scratch 
buffer (`spark_batch_scratch`). The export evaluates every row and returns a 
descriptor of the result's Arrow buffers, or an error message.
    - **Packer** (`wasm_packer.py`): adds the `WasmArrowUDF` runner. It casts 
each argument to the declared Arrow type, copies the buffers in, calls the 
batch export once, and builds the result array from the returned buffers. Each 
Python worker process keeps one wasmtime engine and compiles each module once, 
keyed by its hash. Each runner gets its own wasmtime instance, which is 
recreated after a trap.
    - **Client** (`wasm_udf.rs`): adds 
`UserDefinedFunction::with_batch_entrypoint`, which sets the eval type to 
`SQL_SCALAR_ARROW_UDF` (250). The eval type decides which runner is packed: the 
batch entrypoint goes into the UDF only under eval type 250. Eval type 250 
without a batch entrypoint, or with zero arguments, is an error on the client.
    - Tests in a new `tests/wasm-udf` crate, excluded from the workspace like 
the WASM UDF examples, plus docs and CI.
    
    ### Why are the changes needed?
    
    Spark's Python worker calls a `SQL_ARROW_BATCHED_UDF` once per row. For a 
WASM UDF, each call encodes the row's arguments, enters the module, and decodes 
the result, which costs about 100 microseconds per row in the measurements 
below. With one call per batch, these UDFs cost about the same as a Python 
`arrow_udf` doing the same work with `pyarrow.compute`.
    
    Measured end to end against a local Spark 4.3.0 Connect server (`--master 
local[1]`, default 10,000-row Arrow batches), in ns per row:
    
    | UDF | built-in expression | per row (today) | per batch (this PR) | 
Python `arrow_udf` (`pyarrow.compute`) |
    |---|---:|---:|---:|---:|
    | `add_one (i64)` | 1.2 / 1.6 | 100,235 | 123.5 | 131.4 |
    | `shout (String)` | 109.2 | 103,467 | 301.9 | 304.2 |
    
    - **Method:** each case runs `range(n, 1 
partition).select(f(id)).agg(sum)`, with `sum(length(...))` for the string 
cases. The cost per row is the difference between the run times of two row 
counts divided by the difference in rows, so query, worker and module start-up 
costs cancel out. Each row count takes the best of 3 runs after one warm-up run.
    - **Built-ins:** `id + 1` and `concat(upper(cast(id as string)), '!')`.
    - **Runs:** the figures are from the first of two consecutive runs. In the 
second run every figure was within 15%, except the built-in `id + 1` (a 1-2 ns 
figure), so both of its runs are shown. Second run: 102,251 / 125.9 / 127.3 for 
`add_one`, and 101.2 / 97,310 / 303.5 / 294.6 for `shout`.
    - **Machine:** Intel Xeon Platinum 8559C, 4 cores / 8 threads. OpenJDK 
21.0.12.1, Python 3.12.3, pyspark 4.3.0, pyarrow 26.0.0, wasmtime 50.0.1, rustc 
1.99.1.
    
    <details>
    <summary>How to reproduce (the bench and the script are not part of this 
PR)</summary>
    
    Start a Spark 4.3.0 Connect server whose Python workers have `pyarrow` and 
`wasmtime`, for example from a `pyspark==4.3.0` install:
    
    ```bash
    PYSPARK_PYTHON=<python with pyarrow and wasmtime> \
        "$SPARK_HOME/sbin/start-connect-server.sh" --master 'local[1]'
    ```
    
    Put the three bench files in `examples/wasm-udf-bench/`, add 
`"examples/wasm-udf-bench"` to the root `Cargo.toml` `exclude` list, and run 
the bench and then the `arrow_udf` script (needs `pyspark[connect]==4.3.0` and 
`pyarrow`):
    
    ```bash
    rustup target add wasm32-unknown-unknown
    SPARK_CONNECT_PYTHON=<python with pyspark> \
        cargo run --release --manifest-path examples/wasm-udf-bench/Cargo.toml
    python arrow_udf_bench.py
    ```
    
    `examples/wasm-udf-bench/Cargo.toml`:
    
    ```toml
    [workspace]
    
    [package]
    name = "wasm-udf-bench"
    version = "0.0.0"
    edition = "2021"
    publish = false
    
    [dependencies]
    spark-connect-macros = { package = "apache-spark-connect-macros", path = 
"../../crates/spark-connect-macros" }
    
    [target.'cfg(not(target_arch = "wasm32"))'.dependencies]
    spark-connect = { package = "apache-spark-connect", path = 
"../../crates/spark-connect", features = ["wasm-udf"] }
    
    [build-dependencies]
    spark-connect-build = { package = "apache-spark-connect-build", path = 
"../../crates/spark-connect-build" }
    spark-connect-macros = { package = "apache-spark-connect-macros", path = 
"../../crates/spark-connect-macros" }
    ```
    
    `examples/wasm-udf-bench/build.rs`:
    
    ```rust
    fn main() {
        // Compiles this crate's `#[spark_wasm_udf]` functions to wasm32 and 
embeds
        // the module (exposed to the crate as the `WASM_UDFS_MODULE` env var).
        spark_connect_build::embed_wasm_udf("src/main.rs");
    }
    ```
    
    `examples/wasm-udf-bench/src/main.rs`:
    
    ```rust
    //! Rust UDFs per row vs per Arrow batch, end to end through a Spark 
Connect server.
    //!
    //! Each case runs `range(n, 1 partition).select(f(id)).agg(sum)`. The cost 
per row is the
    //! difference between two row counts divided by the difference in rows, so 
query, worker and
    //! module start-up costs cancel out. Best of 3 per row count, after one 
warm-up run.
    //!
    //!     rustup target add wasm32-unknown-unknown
    //!     SPARK_CONNECT_PYTHON=<python with pyspark> \
    //!         cargo run --release --manifest-path 
examples/wasm-udf-bench/Cargo.toml
    
    use spark_connect_macros::spark_wasm_udf;
    
    #[spark_wasm_udf]
    mod udfs {
        pub fn add_one(x: i64) -> i64 {
            x + 1
        }
    
        pub fn shout(s: String) -> String {
            let mut t = s.to_uppercase();
            t.push('!');
            t
        }
    
        pub fn half_if_even(x: Option<i64>) -> Option<i64> {
            x.filter(|v| v % 2 == 0).map(|v| v / 2)
        }
    }
    
    #[cfg(not(target_arch = "wasm32"))]
    fn main() -> Result<(), Box<dyn std::error::Error>> {
        use spark_connect::functions::{col, expr, length, sum};
        use spark_connect::row::Value;
        use spark_connect::udf::eval_type;
        use spark_connect::{Column, SparkSession, SparkSessionBuilder};
        use std::time::Instant;
    
        type BoxErr = Box<dyn std::error::Error>;
    
        let spark: SparkSession = SparkSessionBuilder::default()
            .remote("sc://127.0.0.1:15002/")
            .get_or_create()?;
    
        // Seconds to collect sum(f(id)) (or sum(length(f(id)))) over n ids in 
one partition.
        let run = |f: &dyn Fn() -> Result<Column, BoxErr>,
                   sum_len: bool,
                   n: i64|
         -> Result<(f64, i64), BoxErr> {
            let total = if sum_len {
                sum(length(col("x")))
            } else {
                sum(col("x"))
            };
            let df = spark
                .range_full(0, n, 1, Some(1))?
                .select([f()?.alias("x")])
                .agg(vec![total.expression().clone()]);
            let start = Instant::now();
            let rows = df.collect()?;
            let secs = start.elapsed().as_secs_f64();
            match rows[0].get(0) {
                Some(Value::Long(v)) => Ok((secs, *v)),
                other => Err(format!("unexpected result {other:?}").into()),
            }
        };
    
        let ns_per_row = |name: &str,
                          f: &dyn Fn() -> Result<Column, BoxErr>,
                          sum_len: bool,
                          expected: &dyn Fn(i64) -> i64,
                          n1: i64,
                          n2: i64|
         -> Result<(), BoxErr> {
            run(f, sum_len, n1)?;
            let mut best = [f64::MAX; 2];
            for (k, n) in [n1, n2].into_iter().enumerate() {
                for _ in 0..3 {
                    let (secs, v) = run(f, sum_len, n)?;
                    if v != expected(n) {
                        return Err(
                            format!("{name}: sum {v} for {n} rows, expected 
{}", expected(n)).into(),
                        );
                    }
                    best[k] = best[k].min(secs);
                }
            }
            let ns = (best[1] - best[0]) / (n2 - n1) as f64 * 1e9;
            println!(
                "{name:22} {ns:10.1} ns/row   ({n1:>10} rows {:7.3} s, {n2:>10} 
rows {:7.3} s)",
                best[0], best[1]
            );
            Ok(())
        };
    
        let add_one_sum = |n: i64| n * (n - 1) / 2 + n;
        let shout_len = |n: i64| (0..n).map(|i| i.to_string().len() as i64 + 
1).sum::<i64>();
        let half_sum = |n: i64| (0..n).filter(|i| i % 2 == 0).map(|i| i / 
2).sum::<i64>();
        let id_str = || col("id").cast_str("string");
    
        // Same answers on both paths, including null results.
        let n = 25_000;
        let (_, per_row) = run(
            &|| {
                Ok(udf::half_if_even_udf()
                    .with_eval_type(eval_type::SQL_ARROW_BATCHED_UDF)
                    .call(vec![col("id")])?)
            },
            false,
            n,
        )?;
        let (_, batch) = run(&|| Ok(udf::half_if_even(col("id"))?), false, n)?;
        assert_eq!((per_row, batch), (half_sum(n), half_sum(n)), 
"half_if_even");
    
        ns_per_row(
            "builtin id + 1",
            &|| Ok(expr("id + 1")),
            false,
            &add_one_sum,
            2_000_000,
            20_000_000,
        )?;
        ns_per_row(
            "add_one per row",
            &|| {
                Ok(udf::add_one_udf()
                    .with_eval_type(eval_type::SQL_ARROW_BATCHED_UDF)
                    .call(vec![col("id")])?)
            },
            false,
            &add_one_sum,
            10_000,
            50_000,
        )?;
        ns_per_row(
            "add_one per batch",
            &|| Ok(udf::add_one(col("id"))?),
            false,
            &add_one_sum,
            2_000_000,
            20_000_000,
        )?;
        ns_per_row(
            "builtin shout",
            &|| Ok(expr("concat(upper(cast(id as string)), '!')")),
            true,
            &shout_len,
            1_000_000,
            5_000_000,
        )?;
        ns_per_row(
            "shout per row",
            &|| {
                Ok(udf::shout_udf()
                    .with_eval_type(eval_type::SQL_ARROW_BATCHED_UDF)
                    .call(vec![id_str()])?)
            },
            true,
            &shout_len,
            10_000,
            50_000,
        )?;
        ns_per_row(
            "shout per batch",
            &|| Ok(udf::shout(id_str())?),
            true,
            &shout_len,
            1_000_000,
            5_000_000,
        )?;
        Ok(())
    }
    ```
    
    `arrow_udf_bench.py`:
    
    ```python
    """Python arrow_udf baselines for the WASM UDF bench, measured the same way.
    
    Each case runs `range(n, 1 partition).select(f(id)).agg(sum)`. The cost per 
row is the
    difference between two row counts divided by the difference in rows, so 
query, worker and
    start-up costs cancel out. Best of 3 per row count, after one warm-up run.
    
        python arrow_udf_bench.py   # Spark Connect server on 
sc://localhost:15002
    """
    
    import time
    
    import pyarrow as pa
    import pyarrow.compute as pc
    from pyspark.sql import SparkSession
    from pyspark.sql.functions import arrow_udf, col, length
    from pyspark.sql.functions import sum as sum_
    
    spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
    
    arrow_udf("long")
    def add_one(x: pa.Array) -> pa.Array:
        return pc.add(x, 1)
    
    arrow_udf("string")
    def shout(s: pa.Array) -> pa.Array:
        return pc.binary_join_element_wise(pc.utf8_upper(s), "!", "")
    
    def run(f, sum_len, n):
        df = spark.range(0, n, 1, 1).select(f().alias("x"))
        df = df.agg(sum_(length(col("x"))) if sum_len else sum_(col("x")))
        start = time.perf_counter()
        value = df.collect()[0][0]
        return time.perf_counter() - start, value
    
    def ns_per_row(name, f, sum_len, expected, n1, n2):
        run(f, sum_len, n1)
        best = [float("inf")] * 2
        for k, n in enumerate((n1, n2)):
            for _ in range(3):
                secs, v = run(f, sum_len, n)
                assert v == expected(n), f"{name}: sum {v} for {n} rows, 
expected {expected(n)}"
                best[k] = min(best[k], secs)
        ns = (best[1] - best[0]) / (n2 - n1) * 1e9
        print(
            f"{name:22} {ns:10.1f} ns/row   ({n1:>10} rows {best[0]:7.3f} s, "
            f"{n2:>10} rows {best[1]:7.3f} s)"
        )
    
    def add_one_sum(n):
        return n * (n - 1) // 2 + n
    
    def shout_len(n):
        return sum(len(str(i)) + 1 for i in range(n))
    
    ns_per_row(
        "add_one arrow_udf", lambda: add_one(col("id")), False, add_one_sum, 
2_000_000, 20_000_000
    )
    ns_per_row(
        "shout arrow_udf",
        lambda: shout(col("id").cast("string")),
        True,
        shout_len,
        1_000_000,
        5_000_000,
    )
    ```
    
    </details>
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, compared with released versions:
    
    - **Batch path:** a `#[spark_wasm_udf]` function with a batch-capable 
signature (above) is now sent as a scalar Arrow UDF (`SQL_SCALAR_ARROW_UDF`) 
and called once per Arrow batch instead of once per row. Results are identical, 
including nulls in and out.
    - **Per-row path kept:** functions with arrays or nested `Option`s, and 
UDFs built by hand with `wasm_udf::udf(...)`, stay per row. To force a 
batch-capable UDF per row, use 
`udf::<fn>_udf().with_eval_type(eval_type::SQL_ARROW_BATCHED_UDF)`.
    - **New API:** `UserDefinedFunction::with_batch_entrypoint(entrypoint)`.
    - **Spark versions:** eval type 250 needs Spark 4.1 or later. The client 
already requires Spark 4.3.0 or later.
    - **Mistyped inputs:** on the batch path, each argument column is converted 
to the declared type with a safe Arrow cast, because Spark passes a column with 
its own type, e.g. a `LongType` column into an `i32` argument.
      - Values that convert exactly (wider integers in range, whole floats) 
work.
      - An out-of-range integer, or a float with a fractional part into an 
integer argument, now raises `ArrowInvalid`, e.g. `Float value 1.500000 was 
truncated converting to int32`.
      - On the per-row path an out-of-range integer raised `struct.error`, and 
a fractional float was truncated with `int()`.
    - **New error messages on the batch path:**
      - A null in an argument that is not an `Option`: ``WASM UDF 
'spark_batch_udf_<fn>': argument `x` is null in row 1 of the batch, but its 
type is not an Option; declare it as Option<T> to accept nulls``.
      - A module whose batch export takes a different number of columns than 
the UDF declares (only possible with a hand-built `udf(...)`): `WASM UDF 
'<entrypoint>': the batch has 1 columns, but the function takes 2`.
      - Arguments of different lengths: `WASM UDF '<entrypoint>' got arguments 
of different lengths: [3, 2]`.
      - A batch whose input buffers exceed 2 GiB: `WASM UDF '<entrypoint>' got 
a batch of N bytes, more than the 2147483647 a WASM module can take; lower 
spark.sql.execution.arrow.maxRecordsPerBatch`.
      - On the client:
        - `WASM UDF '<name>' has eval type SQL_SCALAR_ARROW_UDF, which needs a 
batch entrypoint; set one with with_batch_entrypoint`.
        - `WASM UDF '<name>' has no arguments, so it cannot be called once per 
batch`.
        - A hand-built `udf(...)` given a batch entrypoint and a type the batch 
path can't carry: `WASM UDF '<name>' has type 'array:i64', so it cannot be 
called once per batch; call it per row with 
with_eval_type(SQL_ARROW_BATCHED_UDF)`.
    
    ### How was this patch tested?
    
    Added tests:
    
    - `tests/wasm-udf` (new; fixture UDFs compiled with the macro and runtime). 
These tests drive the real runners with `pyarrow` arrays, the way Spark's 
worker does:
      - Only batch-capable signatures get the batch runner, and 
`with_eval_type(SQL_ARROW_BATCHED_UDF)` forces the per-row one.
      - For every supported type, the batch runner returns the same values as 
the per-row runner and as the expected values: nulls in and out, two arguments 
of mixed types, a batch of only empty strings, and empty batches. All runners 
of a module share one compiled module.
      - Sliced input (including bitmap bit offsets), chunked input, 
`large_string`/`large_binary`, and wider integer columns are accepted; an 
overflow raises.
      - One runner is called with 2, then 50,000, then 3 rows. The 50,000-row 
batch grows the scratch buffer and the module's memory during the call.
      - Errors name their cause, and the next batch succeeds. Covered: a null 
in a non-`Option` argument, including one whose parameter is `_` and so is 
named by position (`` `#1` ``), both column-count mismatches, arguments of 
different lengths (including an empty one), a batch export the module lacks, a 
batch over 2 GiB, a missing `wasmtime` (which raises the install hint), and a 
panic in user code that leaves a `RefCell` borrowed.
      - Every runner test also checks that the Python process exits without 
`Exception ignored` tracebacks on stderr. Without the exit hooks that drop the 
cached wasmtime engine and modules, those objects print these when a process 
that used the batch runner exits normally.
      - The runner tests, including the clean-exit check, also pass with 
wasmtime 17.0.0.
      - The fixture includes functions named `scratch` and `batch_scratch`, so 
a colliding export name fails the build.
    - `tests/wasm-udf/tests/e2e_wasm_udf.rs`: against a live server, covers a 
batch UDF with nulls, a `String` UDF, a two-argument UDF, an array UDF that 
stays per row, and `spark.udf().register(...)` followed by a SQL call.
    - `spark-connect-macros`: unit test of which types the batch path accepts.
    - `wasm_udf.rs`: unit tests that the eval type decides whether the spec 
carries the batch entrypoint, and that eval type 250 is rejected without a 
batch entrypoint, without arguments, or with a type the batch path can't carry 
(an array argument, an `Option` of an array, a nested `Option` return).
    - CI: the `wasm-udf` job also runs the macro unit tests and the runner 
tests. The `connect-e2e` job adds the wasm32 target, `wasmtime` and the 
packer's `pyspark`, and runs the e2e test.
    
    Ran locally (Spark 4.3.0 Connect server, Java 21, Python 3.12 with pyspark 
4.3.0, pyarrow 26.0.0, wasmtime 50.0.1):
    
    ```bash
    cargo test -p apache-spark-connect --features wasm-udf                      
                # 703 passed, 27 ignored
    cargo test -p apache-spark-connect --features wasm-udf -- --ignored 
packer_produces         # 1 passed
    cargo test -p apache-spark-connect-macros                                   
                # 1 passed
    cargo test --manifest-path tests/wasm-udf/Cargo.toml -- --ignored           
                # 4 passed
    SPARK_REMOTE=sc://localhost:15002 cargo test --manifest-path 
tests/wasm-udf/Cargo.toml --test e2e_wasm_udf  # 1 passed
    cargo test -p apache-spark-connect --release --lib                          
                # 461 passed
    cargo test -p apache-spark-connect --release --no-run
    cargo test -p apache-spark-connect --release --features wasm-udf --no-run
    cargo fmt --all --check
    cargo fmt --manifest-path tests/wasm-udf/Cargo.toml --check
    cargo build -p apache-spark-connect-macros -p apache-spark-connect-build
    cargo build --manifest-path examples/wasm-udfs/Cargo.toml
    cargo build --manifest-path examples/wasm-udf-inline/Cargo.toml
    cargo build -p examples --bin wasm_udf_macro --features 
examples/wasm-udf-macro
    python scripts/check_doc_snippets.py
    ```
    
    `cargo clippy --workspace --all-targets` reports the same diagnostics as on 
master, and none in the changed files.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #262 from tdcmeehan/wasm-udf-arrow-batch.
    
    Lead-authored-by: Tim Meehan <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 .github/workflows/build_python_connect.yml |  12 +-
 .github/workflows/rust.yml                 |  11 +-
 .gitignore                                 |   3 +-
 AGENTS.md                                  |  16 +-
 Cargo.toml                                 |   9 +-
 crates/spark-connect-build/src/lib.rs      | 286 ++++++++++++++++++++++++++++-
 crates/spark-connect-macros/src/lib.rs     | 106 ++++++++++-
 crates/spark-connect/src/wasm_packer.py    | 201 +++++++++++++++++++-
 crates/spark-connect/src/wasm_udf.rs       | 169 +++++++++++++++--
 docs/architecture.md                       |   2 +-
 docs/udfs.md                               |  19 +-
 tests/wasm-udf/Cargo.toml                  |  35 ++++
 tests/wasm-udf/build.rs                    |  20 ++
 tests/wasm-udf/src/lib.rs                  |  86 +++++++++
 tests/wasm-udf/tests/e2e_wasm_udf.rs       |  72 ++++++++
 tests/wasm-udf/tests/runners.rs            | 242 ++++++++++++++++++++++++
 16 files changed, 1233 insertions(+), 56 deletions(-)

diff --git a/.github/workflows/build_python_connect.yml 
b/.github/workflows/build_python_connect.yml
index 408ce82c..8f471c3e 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -314,10 +314,11 @@ jobs:
           distribution: temurin
           java-version: "17"
 
-      - name: Install Rust toolchain
+      - name: Install Rust toolchain (+ wasm32 target)
         run: |
           rustup toolchain install stable --profile minimal
           rustup default stable
+          rustup target add wasm32-unknown-unknown
 
       - name: Cache cargo & target
         uses: actions/cache@v4
@@ -334,10 +335,10 @@ jobs:
           sudo apt-get update
           sudo apt-get install -y protobuf-compiler
 
-      - name: Install Python dependencies (server-side UDF workers)
+      - name: Install Python dependencies (server-side UDF workers, WASM UDF 
packer)
         run: |
           python -m pip install --upgrade pip
-          python -m pip install 'pandas>=2.2,<3.0' pyarrow numpy
+          python -m pip install 'pandas>=2.2,<3.0' pyarrow numpy wasmtime 
"pyspark==${SPARK_VERSION}"
 
       - name: Cache Spark ${{ env.SPARK_VERSION }} distribution
         uses: actions/cache@v4
@@ -393,6 +394,11 @@ jobs:
             --test e2e_coverage --test e2e_linear_svc --test e2e_integration \
             -- --test-threads=1
 
+      - name: Rust WASM UDF e2e test (against the live server)
+        env:
+          SPARK_REMOTE: sc://localhost:15002
+        run: cargo test --manifest-path tests/wasm-udf/Cargo.toml --test 
e2e_wasm_udf
+
   # ---------------------------------------------------------------------------
   # E2E performance benchmark: our client vs reference, same 4.3.0 server.
   # ---------------------------------------------------------------------------
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index 82c00df7..3248e349 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -151,11 +151,13 @@ jobs:
         # The client builds the UDF command by running the packer embedded in 
the
         # crate as `python -c` (needs `pyspark` for its vendored cloudpickle + 
the
         # pure-Python type parser); `wasmtime` is what the executors need at 
run time
-        # and lets a follow-up job exercise the module directly.
-        run: python -m pip install --upgrade cloudpickle pyspark wasmtime
+        # and, with `pyarrow`, lets the runner tests exercise the module 
directly.
+        run: python -m pip install --upgrade cloudpickle pyspark wasmtime 
pyarrow
 
       - name: Feature-gated unit tests
-        run: cargo test -p apache-spark-connect --features wasm-udf
+        run: |
+          cargo test -p apache-spark-connect --features wasm-udf
+          cargo test -p apache-spark-connect-macros
 
       - name: Build the macro/build helpers and WASM UDF crates + examples
         run: |
@@ -171,3 +173,6 @@ jobs:
         # `DataType` the executors unpickle. No separate `pyspark_wasm_udf` 
package
         # is needed on any path now, so just run the ignored end-to-end test.
         run: cargo test -p apache-spark-connect --features wasm-udf -- 
--ignored packer_produces
+
+      - name: WASM UDF runners (pyarrow batches, no server)
+        run: cargo test --manifest-path tests/wasm-udf/Cargo.toml -- --ignored
diff --git a/.gitignore b/.gitignore
index a5e476e9..addb8a66 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,9 +16,10 @@
 # under the License.
 
 /target
-# The workspace-excluded WASM example crates build to their own target/ dirs.
+# The workspace-excluded WASM crates build to their own target/ dirs.
 /examples/wasm-udfs/target
 /examples/wasm-udf-inline/target
+/tests/wasm-udf/target
 **/*.rs.bk
 Cargo.lock
 __pycache__/
diff --git a/AGENTS.md b/AGENTS.md
index 07d0be7e..4810ae4d 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -287,13 +287,17 @@ Do not rely on the previous version's suite. When bumping 
to vX.Y.Z:
 - The packer is a Python script **embedded in the crate**
   (`crates/spark-connect/src/wasm_packer.py`, pulled in with `include_str!`). 
The
   Rust client runs it as `python -c <src>` (so it executes as `__main__`), 
which
-  makes cloudpickle serialize the `WasmScalarUDF` runner **by value** into the 
UDF
-  command. Because it runs as `__main__` there is **no** separate 
`pyspark_wasm_udf`
-  pip package: the client needs only a Python interpreter with `pyspark` (for 
its
-  vendored `cloudpickle` + `pyspark.sql.types._parse_datatype_json_value`), 
and the
-  executors need only `wasmtime`. If the packer's ABI/codec changes, keep it
+  makes cloudpickle serialize the runner (`WasmScalarUDF` per row, 
`WasmArrowUDF`
+  per Arrow batch) **by value** into the UDF command. Because it runs as 
`__main__`
+  there is **no** separate `pyspark_wasm_udf` pip package: the client needs 
only a
+  Python interpreter with `pyspark` (for its vendored `cloudpickle` +
+  `pyspark.sql.types._parse_datatype_json_value`), and the executors need only
+  `wasmtime`. If the packer's ABI/codec changes, keep it
   byte-compatible with `spark_connect::wasm_udf::AbiType` and the build-time 
runtime
-  in `apache-spark-connect-build`.
+  in `apache-spark-connect-build`. The tests that check this are outside the
+  workspace: `cargo test --manifest-path tests/wasm-udf/Cargo.toml -- 
--ignored`
+  (Python with `pyspark`, `pyarrow` and `wasmtime`), formatted with
+  `cargo fmt --manifest-path tests/wasm-udf/Cargo.toml`.
 - Registration mirrors pyspark: `spark.udf().register("name", &udf)` (a
   session-side `UdfRegistration` handle), then callable from SQL.
 - Removing Python on the client entirely would mean pre-baking the cloudpickled
diff --git a/Cargo.toml b/Cargo.toml
index 5fe753a8..68f28243 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -29,16 +29,17 @@ members = [
 exclude = [
     "examples/wasm-udfs",
     "examples/wasm-udf-inline",
+    "tests/wasm-udf",
 ]
 # `pyspark-rs` (the CPython extension cdylib, built via maturin) and the
 # Rust-native `examples` crate both link the pure-Rust core directly; both are
 # workspace members but excluded from `default-members` so `cargo test`/`cargo 
build`
 # stay clean. Build them explicitly (`cargo build -p pyspark-rs` / `-p 
examples`).
 #
-# The `examples/wasm-udfs` and `examples/wasm-udf-inline` crates are excluded 
from
-# the workspace entirely because they require the `wasm32-unknown-unknown` 
target,
-# which may not be installed on every machine (so `cargo test --workspace` 
stays
-# green without it). Build them explicitly with
+# The `examples/wasm-udfs`, `examples/wasm-udf-inline` and `tests/wasm-udf` 
crates
+# are excluded from the workspace entirely because they require the
+# `wasm32-unknown-unknown` target, which may not be installed on every machine
+# (so `cargo test --workspace` stays green without it). Build them explicitly 
with
 # `cargo build --manifest-path examples/wasm-udfs/Cargo.toml` (etc) in CI or 
when
 # the target is installed locally.
 default-members = [
diff --git a/crates/spark-connect-build/src/lib.rs 
b/crates/spark-connect-build/src/lib.rs
index c1d22e03..0a5b824a 100644
--- a/crates/spark-connect-build/src/lib.rs
+++ b/crates/spark-connect-build/src/lib.rs
@@ -112,10 +112,11 @@ pub fn embed_wasm_udf(src_file: impl AsRef<Path>) -> 
PathBuf {
     wasm_out
 }
 
-/// The runtime prepended to the wasm build: the exported allocator and the
-/// length-prefixed binary codec (`Abi`) the macro's export wrappers call. Must
-/// stay byte-compatible with `spark_connect::wasm_udf::AbiType` and the Python
-/// runner's codec in the embedded packer (`spark-connect/src/wasm_packer.py`).
+/// The runtime prepended to the wasm build: the exported allocator, the
+/// length-prefixed binary codec (`Abi`) the macro's per-row export wrappers 
call,
+/// and the Arrow-buffer codec (`batch`) of its batch exports. Must stay
+/// byte-compatible with `spark_connect::wasm_udf::AbiType` and the Python
+/// runners in the embedded packer (`spark-connect/src/wasm_packer.py`).
 const WASM_RUNTIME: &str = r#"
 #[allow(dead_code)]
 pub mod spark_wasm_rt {
@@ -198,6 +199,283 @@ pub mod spark_wasm_rt {
         unsafe { std::slice::from_raw_parts_mut(p, out.len()) 
}.copy_from_slice(&out);
         ((p as u64) << 32) | (out.len() as u64)
     }
+
+    thread_local! {
+        static SCRATCH: std::cell::RefCell<Vec<u8>> = 
std::cell::RefCell::new(Vec::new());
+        static KEEP: std::cell::RefCell<Vec<Vec<u8>>> = 
std::cell::RefCell::new(Vec::new());
+    }
+
+    // Per-row exports are `spark_udf_<fn>`, so batch names use another prefix 
to never collide.
+    #[no_mangle]
+    pub extern "C" fn spark_batch_scratch(len: u32) -> u32 {
+        SCRATCH.with(|s| {
+            let mut s = s.borrow_mut();
+            if s.len() < len as usize {
+                s.resize(len as usize, 0);
+            }
+            s.as_mut_ptr() as u32
+        })
+    }
+
+    // Batch ABI of the `spark_batch_udf_<fn>` exports, little-endian u32 
words. Input: [rows,
+    // columns, then per column: validity ptr (0: no nulls), values ptr, 
values len, offsets ptr
+    // (0: fixed width), offsets len]. Output, valid until the next call: 
[status (0 ok, 1 error),
+    // null count, validity ptr, validity len, values ptr, values len, offsets 
ptr, offsets len],
+    // with the error message at (validity ptr, validity len).
+    pub mod batch {
+        use super::KEEP;
+
+        pub struct Col {
+            validity: Option<&'static [u8]>,
+            values: &'static [u8],
+            offsets: Option<&'static [u8]>,
+        }
+
+        #[inline]
+        fn bit(bytes: &[u8], i: usize) -> bool {
+            (bytes[i >> 3] >> (i & 7)) & 1 == 1
+        }
+
+        impl Col {
+            #[inline]
+            fn is_valid(&self, i: usize) -> bool {
+                self.validity.map_or(true, |v| bit(v, i))
+            }
+            #[inline]
+            fn offset(&self, i: usize) -> usize {
+                let o = self.offsets.expect("variable-width column without 
offsets");
+                i32::from_le_bytes(o[i * 4..i * 4 + 4].try_into().unwrap()) as 
usize
+            }
+            #[inline]
+            fn bytes(&self, i: usize) -> &'static [u8] {
+                &self.values[self.offset(i)..self.offset(i + 1)]
+            }
+        }
+
+        pub trait FromCol: Sized {
+            fn get_valid(col: &Col, i: usize) -> Self;
+            // None: a null in an argument that isn't an Option.
+            #[inline]
+            fn get(col: &Col, i: usize) -> Option<Self> {
+                if col.is_valid(i) { Some(Self::get_valid(col, i)) } else { 
None }
+            }
+        }
+
+        macro_rules! prim_in { ($t:ty, $n:expr) => {
+            impl FromCol for $t {
+                #[inline]
+                fn get_valid(col: &Col, i: usize) -> Self {
+                    <$t>::from_le_bytes(col.values[i * $n..i * $n + 
$n].try_into().unwrap())
+                }
+            }
+        }}
+        prim_in!(i32, 4); prim_in!(i64, 8); prim_in!(f32, 4); prim_in!(f64, 8);
+
+        impl FromCol for bool {
+            #[inline]
+            fn get_valid(col: &Col, i: usize) -> Self { bit(col.values, i) }
+        }
+        impl FromCol for String {
+            #[inline]
+            fn get_valid(col: &Col, i: usize) -> Self {
+                String::from_utf8(col.bytes(i).to_vec()).unwrap()
+            }
+        }
+        impl FromCol for Vec<u8> {
+            #[inline]
+            fn get_valid(col: &Col, i: usize) -> Self { col.bytes(i).to_vec() }
+        }
+        impl<T: FromCol> FromCol for Option<T> {
+            #[inline]
+            fn get_valid(col: &Col, i: usize) -> Self { Some(T::get_valid(col, 
i)) }
+            #[inline]
+            fn get(col: &Col, i: usize) -> Option<Self> {
+                Some(if col.is_valid(i) { Some(T::get_valid(col, i)) } else { 
None })
+            }
+        }
+
+        pub struct Out {
+            n: usize,
+            row: usize,
+            validity: Vec<u8>,
+            null_count: u32,
+            values: Vec<u8>,
+            offsets: Vec<u8>,
+        }
+
+        impl Out {
+            fn new(n: usize) -> Self {
+                Out {
+                    n,
+                    row: 0,
+                    validity: vec![0xFF; (n + 7) / 8],
+                    null_count: 0,
+                    values: Vec::new(),
+                    offsets: Vec::new(),
+                }
+            }
+            #[inline]
+            fn set_null(&mut self) {
+                let r = self.row;
+                self.validity[r >> 3] &= !(1u8 << (r & 7));
+                self.null_count += 1;
+            }
+            #[inline]
+            fn start_var(&mut self) {
+                if self.offsets.is_empty() {
+                    self.offsets.reserve(4 * (self.n + 1));
+                    self.offsets.extend_from_slice(&0i32.to_le_bytes());
+                }
+            }
+            #[inline]
+            fn end_var(&mut self) {
+                // Lossless: a wasm32 Vec holds at most i32::MAX bytes.
+                let end = self.values.len() as i32;
+                self.offsets.extend_from_slice(&end.to_le_bytes());
+            }
+            #[inline]
+            fn bits(&mut self) {
+                if self.values.is_empty() {
+                    self.values = vec![0u8; (self.n + 7) / 8];
+                }
+            }
+            fn finish(self) -> u32 {
+                let nulls = self.null_count;
+                let (vp, vl) = if nulls == 0 { (0, 0) } else { 
keep(self.validity) };
+                let (xp, xl) = keep(self.values);
+                let (op, ol) = keep(self.offsets);
+                descriptor(&[0, nulls, vp, vl, xp, xl, op, ol])
+            }
+        }
+
+        pub trait ToCol: Default {
+            fn push(&self, out: &mut Out);
+            #[inline]
+            fn push_null(out: &mut Out) {
+                out.set_null();
+                Self::default().push(out);
+            }
+        }
+
+        macro_rules! prim_out { ($t:ty) => {
+            impl ToCol for $t {
+                #[inline]
+                fn push(&self, out: &mut Out) {
+                    out.values.extend_from_slice(&self.to_le_bytes());
+                    out.row += 1;
+                }
+            }
+        }}
+        prim_out!(i32); prim_out!(i64); prim_out!(f32); prim_out!(f64);
+
+        impl ToCol for bool {
+            #[inline]
+            fn push(&self, out: &mut Out) {
+                out.bits();
+                if *self {
+                    let r = out.row;
+                    out.values[r >> 3] |= 1u8 << (r & 7);
+                }
+                out.row += 1;
+            }
+        }
+        impl ToCol for String {
+            #[inline]
+            fn push(&self, out: &mut Out) { push_bytes(out, self.as_bytes()) }
+        }
+        impl ToCol for Vec<u8> {
+            #[inline]
+            fn push(&self, out: &mut Out) { push_bytes(out, self) }
+        }
+        impl<T: ToCol> ToCol for Option<T> {
+            #[inline]
+            fn push(&self, out: &mut Out) {
+                match self {
+                    Some(v) => v.push(out),
+                    None => T::push_null(out),
+                }
+            }
+            #[inline]
+            fn push_null(out: &mut Out) { T::push_null(out) }
+        }
+
+        #[inline]
+        fn push_bytes(out: &mut Out, b: &[u8]) {
+            out.start_var();
+            out.values.extend_from_slice(b);
+            out.end_var();
+            out.row += 1;
+        }
+
+        fn keep(b: Vec<u8>) -> (u32, u32) {
+            if b.is_empty() {
+                return (0, 0);
+            }
+            let (p, l) = (b.as_ptr() as u32, b.len() as u32);
+            KEEP.with(|k| k.borrow_mut().push(b));
+            (p, l)
+        }
+
+        fn descriptor(words: &[u32]) -> u32 {
+            let mut d = Vec::with_capacity(words.len() * 4);
+            for w in words {
+                d.extend_from_slice(&w.to_le_bytes());
+            }
+            keep(d).0
+        }
+
+        fn error(msg: String) -> u32 {
+            let (mp, ml) = keep(msg.into_bytes());
+            descriptor(&[1, 0, mp, ml, 0, 0, 0, 0])
+        }
+
+        fn raw(p: usize, l: usize) -> &'static [u8] {
+            if l == 0 { &[] } else { unsafe { core::slice::from_raw_parts(p as 
*const u8, l) } }
+        }
+
+        fn words(p: usize, n: usize) -> impl Fn(usize) -> usize {
+            let b = raw(p, 4 * n);
+            move |k| u32::from_le_bytes(b[4 * k..4 * k + 
4].try_into().unwrap()) as usize
+        }
+
+        // `f` evaluates row `i`; Err(arg) means a null in argument `arg`, 
which isn't an Option.
+        pub fn run<F>(desc: u32, args: &[&str], mut f: F) -> u32
+        where
+            F: FnMut(&[Col], usize, &mut Out) -> Result<(), usize>,
+        {
+            KEEP.with(|k| k.borrow_mut().clear());
+            let head = words(desc as usize, 2);
+            let (n, ncols) = (head(0), head(1));
+            if ncols != args.len() {
+                return error(format!(
+                    "the batch has {ncols} columns, but the function takes {}",
+                    args.len()
+                ));
+            }
+            let w = words(desc as usize + 8, 5 * ncols);
+            let cols: Vec<Col> = (0..ncols)
+                .map(|a| {
+                    let b = 5 * a;
+                    Col {
+                        validity: if w(b) == 0 { None } else { Some(raw(w(b), 
(n + 7) / 8)) },
+                        values: raw(w(b + 1), w(b + 2)),
+                        offsets: if w(b + 3) == 0 { None } else { Some(raw(w(b 
+ 3), w(b + 4))) },
+                    }
+                })
+                .collect();
+            let mut out = Out::new(n);
+            for i in 0..n {
+                if let Err(a) = f(&cols, i, &mut out) {
+                    return error(format!(
+                        "argument `{}` is null in row {i} of the batch, but 
its type is not an \
+                         Option; declare it as Option<T> to accept nulls",
+                        args[a]
+                    ));
+                }
+            }
+            out.finish()
+        }
+    }
 }
 "#;
 
diff --git a/crates/spark-connect-macros/src/lib.rs 
b/crates/spark-connect-macros/src/lib.rs
index 66a17201..01a75852 100644
--- a/crates/spark-connect-macros/src/lib.rs
+++ b/crates/spark-connect-macros/src/lib.rs
@@ -6,8 +6,8 @@
 use proc_macro::TokenStream;
 use quote::{format_ident, quote};
 use syn::{
-    parse_macro_input, FnArg, GenericArgument, Item, ItemFn, ItemMod, 
PathArguments, ReturnType,
-    Type,
+    parse_macro_input, FnArg, GenericArgument, Item, ItemFn, ItemMod, Pat, 
PathArguments,
+    ReturnType, Type,
 };
 
 /// Turn a module of plain Rust functions into Spark UDFs backed by 
WebAssembly.
@@ -25,6 +25,12 @@ use syn::{
 ///   signature **inferred** from the Rust signature and the crate's own 
compiled
 ///   wasm module embedded.
 ///
+/// A function whose arguments and result are all `i32`, `i64`, `f32`, `f64`,
+/// `bool`, `String`, `Vec<u8>`, or one `Option` of those also gets a batch
+/// export, and its builder sends it as a scalar Arrow UDF that Spark calls 
once
+/// per Arrow batch. 
`.with_eval_type(spark_connect::udf::eval_type::SQL_ARROW_BATCHED_UDF)`
+/// on the builder calls it once per row instead, like functions with other 
types.
+///
 /// ```ignore
 /// use spark_connect_macros::spark_wasm_udf;
 ///
@@ -82,6 +88,19 @@ fn abi_type(ty: &Type) -> Result<proc_macro2::TokenStream, 
syn::Error> {
     ))
 }
 
+fn batch_supported(ty: &Type) -> bool {
+    if let Some(inner) = generic_inner(ty, "Option") {
+        return generic_inner(inner, "Option").is_none() && 
batch_supported(inner);
+    }
+    if let Some(inner) = generic_inner(ty, "Vec") {
+        return path_ident(inner).as_deref() == Some("u8");
+    }
+    matches!(
+        path_ident(ty).as_deref(),
+        Some("i32" | "i64" | "f32" | "f64" | "bool" | "String")
+    )
+}
+
 /// The last path segment ident of a plain type path (`i64`, `String`, ...).
 fn path_ident(ty: &Type) -> Option<String> {
     match ty {
@@ -109,6 +128,8 @@ fn generic_inner<'a>(ty: &'a Type, wrapper: &str) -> 
Option<&'a Type> {
 struct Udf {
     name: syn::Ident,
     export: syn::Ident,
+    batch_export: Option<syn::Ident>,
+    arg_names: Vec<String>,
     arg_tys: Vec<Type>,
     ret_ty: Type,
 }
@@ -127,10 +148,17 @@ fn collect_udf(func: &ItemFn) -> Result<Udf, syn::Error> {
         ));
     }
     let name = func.sig.ident.clone();
+    let mut arg_names = Vec::new();
     let mut arg_tys = Vec::new();
-    for input in &func.sig.inputs {
+    for (i, input) in func.sig.inputs.iter().enumerate() {
         match input {
-            FnArg::Typed(pt) => arg_tys.push((*pt.ty).clone()),
+            FnArg::Typed(pt) => {
+                arg_names.push(match &*pt.pat {
+                    Pat::Ident(p) => p.ident.to_string(),
+                    _ => format!("#{i}"),
+                });
+                arg_tys.push((*pt.ty).clone());
+            }
             FnArg::Receiver(r) => {
                 return Err(syn::Error::new_spanned(
                     r,
@@ -155,9 +183,16 @@ fn collect_udf(func: &ItemFn) -> Result<Udf, syn::Error> {
         }
     };
     let export = format_ident!("spark_udf_{}", name);
+    let batch_export = arg_tys
+        .iter()
+        .chain([&ret_ty])
+        .all(batch_supported)
+        .then(|| format_ident!("spark_batch_udf_{}", name));
     Ok(Udf {
         name,
         export,
+        batch_export,
+        arg_names,
         arg_tys,
         ret_ty,
     })
@@ -192,6 +227,8 @@ fn expand(mut module: ItemMod) -> 
Result<proc_macro2::TokenStream, syn::Error> {
         let Udf {
             name,
             export,
+            batch_export,
+            arg_names,
             arg_tys,
             ret_ty,
         } = u;
@@ -211,6 +248,30 @@ fn expand(mut module: ItemMod) -> 
Result<proc_macro2::TokenStream, syn::Error> {
                 crate::spark_wasm_rt::finish(__o)
             }
         });
+        if let Some(batch_export) = batch_export {
+            let vars: Vec<syn::Ident> = (0..arg_tys.len())
+                .map(|i| format_ident!("__a{i}"))
+                .collect();
+            let gets = arg_tys.iter().zip(&vars).enumerate().map(|(i, (ty, 
v))| {
+                quote! {
+                    let #v: #ty =
+                        <#ty as 
crate::spark_wasm_rt::batch::FromCol>::get(&__cols[#i], __i)
+                            .ok_or(#i)?;
+                }
+            });
+            wrappers.push(quote! {
+                #[cfg(target_arch = "wasm32")]
+                #[no_mangle]
+                pub extern "C" fn #batch_export(__desc: u32) -> u32 {
+                    crate::spark_wasm_rt::batch::run(__desc, 
&[#(#arg_names),*], |__cols, __i, __out| {
+                        #(#gets)*
+                        let __r: #ret_ty = #name(#(#vars),*);
+                        crate::spark_wasm_rt::batch::ToCol::push(&__r, __out);
+                        ::core::result::Result::Ok(())
+                    })
+                }
+            });
+        }
     }
     if let Some((_, content)) = &mut module.content {
         for w in wrappers {
@@ -231,14 +292,20 @@ fn expand(mut module: ItemMod) -> 
Result<proc_macro2::TokenStream, syn::Error> {
         let Udf {
             name,
             export,
+            batch_export,
             arg_tys,
             ret_ty,
+            ..
         } = u;
         let name_str = name.to_string();
         let export_str = export.to_string();
         let builder = format_ident!("{}_udf", name);
         let arg_abis: Vec<_> = 
arg_tys.iter().map(abi_type).collect::<Result<_, _>>()?;
         let ret_abi = abi_type(ret_ty)?;
+        let with_batch = batch_export.as_ref().map(|b| {
+            let b = b.to_string();
+            quote!(.with_batch_entrypoint(#b))
+        });
         let params: Vec<syn::Ident> = (0..arg_tys.len())
             .map(|i| format_ident!("__a{i}"))
             .collect();
@@ -262,6 +329,7 @@ fn expand(mut module: ItemMod) -> 
Result<proc_macro2::TokenStream, syn::Error> {
                     ::std::vec![#(#arg_abis),*],
                     #ret_abi,
                 )
+                #with_batch
             }
 
             #[doc = #call_doc]
@@ -291,3 +359,33 @@ fn expand(mut module: ItemMod) -> 
Result<proc_macro2::TokenStream, syn::Error> {
         }
     })
 }
+
+#[cfg(test)]
+mod tests {
+    use super::batch_supported;
+
+    #[test]
+    fn batch_supported_accepts_scalars_and_one_option_level_only() {
+        for (ty, expected) in [
+            ("i32", true),
+            ("i64", true),
+            ("f32", true),
+            ("f64", true),
+            ("bool", true),
+            ("String", true),
+            ("std::string::String", true),
+            ("Vec<u8>", true),
+            ("Option<i64>", true),
+            ("Option<String>", true),
+            ("Option<Vec<u8>>", true),
+            ("Vec<i64>", false),
+            ("Vec<String>", false),
+            ("Vec<Option<u8>>", false),
+            ("Option<Vec<i64>>", false),
+            ("Option<Option<i64>>", false),
+        ] {
+            let parsed: syn::Type = syn::parse_str(ty).unwrap();
+            assert_eq!(batch_supported(&parsed), expected, "{ty}");
+        }
+    }
+}
diff --git a/crates/spark-connect/src/wasm_packer.py 
b/crates/spark-connect/src/wasm_packer.py
index bc0de100..3bcf9384 100644
--- a/crates/spark-connect/src/wasm_packer.py
+++ b/crates/spark-connect/src/wasm_packer.py
@@ -18,19 +18,22 @@
 
 The Rust client runs this source as ``python -c <this>`` (so it executes as
 ``__main__``), feeding a JSON spec on stdin and reading the raw cloudpickled
-``command`` bytes from stdout. Because the ``WasmScalarUDF`` runner is defined
-here in ``__main__``, cloudpickle serializes it **by value** automatically -- 
so
+``command`` bytes from stdout. Because the runners (``WasmScalarUDF``, called
+per row, and ``WasmArrowUDF``, called per Arrow batch) are defined here in
+``__main__``, cloudpickle serializes them **by value** automatically -- so
 neither a separate ``pyspark_wasm_udf`` pip package on the client nor this 
module
 on the executors is required. Executors need only the ``wasmtime`` package (and
-Spark's own Python worker) to unpickle and run the ``(WasmScalarUDF, 
return_type)``
-tuple. The only client-side requirement is a Python interpreter with 
``pyspark``
-(for its vendored ``cloudpickle`` and the pure-Python type parser).
+Spark's own Python worker, with the ``pyarrow`` its Arrow UDF eval types need) 
to
+unpickle and run the ``(runner, return_type)`` tuple. The only client-side
+requirement is a Python interpreter with ``pyspark`` (for its vendored
+``cloudpickle`` and the pure-Python type parser).
 
 Input JSON (stdin)::
 
     {
       "wasm_b64":    "<base64 of the .wasm module>",
       "entrypoint":  "add_one",
+      "batch_entrypoint": null,                       # or the batch export
       "arg_types":   ["i64", "array:string", ...],   # ABI descriptors
       "ret_type":    "i64",
       "output_type": "long"                           # Spark type-JSON value
@@ -46,6 +49,8 @@ The binary ABI (little-endian, mirroring 
``spark_connect::wasm_udf::AbiType``):
 
 The module exports ``spark_udf_alloc(len)->ptr`` / 
``spark_udf_dealloc(ptr,len)``
 and each UDF as ``fn(args_ptr, args_len) -> (ptr << 32 | len)`` of the result.
+A batch entrypoint instead takes the Arrow buffers of a whole batch; its ABI is
+documented with the runtime in ``apache-spark-connect-build``.
 """
 
 import base64
@@ -242,7 +247,185 @@ class WasmScalarUDF:
         return value
 
 
-# --- packing: (WasmScalarUDF, return_type) -> cloudpickled command bytes -----
+# --- the picklable, per-batch callable ---------------------------------------
+
+
+def _arrow_type(desc: str) -> Any:
+    import pyarrow as pa
+
+    kind, inner = _split(desc)
+    return {
+        "i32": pa.int32(),
+        "i64": pa.int64(),
+        "f32": pa.float32(),
+        "f64": pa.float64(),
+        "bool": pa.bool_(),
+        "string": pa.string(),
+        "binary": pa.binary(),
+    }[inner if kind == "option" else desc]
+
+
+# Scratch offsets are passed as wasm i32s, which wasmtime wraps silently.
+_MAX_BATCH_BYTES = 2**31 - 1
+
+
+class WasmArrowUDF(WasmScalarUDF):
+    """A per-batch callable for a scalar Arrow UDF 
(``SQL_SCALAR_ARROW_UDF``)."""
+
+    def _ensure(self):
+        if self._rt is not None:
+            return self._rt
+        import atexit
+        import hashlib
+
+        try:
+            import wasmtime
+        except ImportError as exc:  # pragma: no cover - depends on executor 
env
+            raise ImportError(
+                "The 'wasmtime' package is required to run WASM UDFs on the "
+                "Spark executors. Install it in the workers' Python env."
+            ) from exc
+
+        # Lives on `sys` because this class is unpickled by value per task, so 
its
+        # module globals don't outlive the task, while the worker process does.
+        # Dropped at exit because wasmtime's finalizers fail during 
interpreter teardown.
+        if not hasattr(sys, "_spark_connect_wasm_udf_engine"):
+            sys._spark_connect_wasm_udf_engine = wasmtime.Engine()
+            sys._spark_connect_wasm_udf_modules = {}
+            atexit.register(delattr, sys, "_spark_connect_wasm_udf_engine")
+            atexit.register(delattr, sys, "_spark_connect_wasm_udf_modules")
+        engine = sys._spark_connect_wasm_udf_engine
+        modules = sys._spark_connect_wasm_udf_modules
+        key = hashlib.sha256(self.wasm).hexdigest()
+        if key not in modules:
+            modules[key] = wasmtime.Module(engine, self.wasm)
+        store = wasmtime.Store(engine)
+        exports = wasmtime.Instance(store, modules[key], []).exports(store)
+        entry = exports.get(self.entrypoint)
+        if entry is None:
+            raise ValueError(f"WASM module does not export 
'{self.entrypoint}'")
+        self._rt = {
+            "store": store,
+            "memory": exports["memory"],
+            "scratch": exports["spark_batch_scratch"],
+            "entry": entry,
+            "scratch_ptr": 0,
+            "scratch_len": 0,
+        }
+        return self._rt
+
+    def __call__(self, *arrays: Any) -> Any:
+        import pyarrow as pa
+
+        if len(arrays) != len(self.arg_types):
+            raise ValueError(
+                f"WASM UDF '{self.entrypoint}' expected {len(self.arg_types)} "
+                f"argument(s), got {len(arrays)}"
+            )
+        lengths = [len(a) for a in arrays]
+        if len(set(lengths)) > 1:
+            raise ValueError(
+                f"WASM UDF '{self.entrypoint}' got arguments of different 
lengths: {lengths}"
+            )
+        out_type = _arrow_type(self.ret_type)
+        n = lengths[0]
+        if n == 0:
+            return pa.array([], type=out_type)
+
+        cols = []
+        for desc, a in zip(self.arg_types, arrays):
+            if isinstance(a, pa.ChunkedArray):
+                a = a.combine_chunks()
+            expected = _arrow_type(desc)
+            if a.type != expected:
+                a = a.cast(expected)
+            if a.offset:
+                a = pa.concat_arrays([a])
+            bufs = a.buffers()
+            validity = bufs[0] if a.null_count else None
+            if len(bufs) == 3:
+                cols.append((validity, bufs[2], bufs[1]))
+            else:
+                cols.append((validity, bufs[1], None))
+
+        size = (8 + 20 * len(cols) + 7) & ~7
+        placed = []
+        for col in cols:
+            slots = []
+            for buf in col:
+                if buf is None or buf.size == 0:
+                    slots.append((0, None))
+                else:
+                    slots.append((size, buf))
+                    size = (size + buf.size + 7) & ~7
+            placed.append(slots)
+        if size > _MAX_BATCH_BYTES:
+            raise ValueError(
+                f"WASM UDF '{self.entrypoint}' got a batch of {size} bytes, 
more than "
+                f"the {_MAX_BATCH_BYTES} a WASM module can take; lower "
+                "spark.sql.execution.arrow.maxRecordsPerBatch"
+            )
+
+        rt = self._ensure()
+        import wasmtime
+
+        try:
+            return self._call(rt, n, placed, size, out_type)
+        except (wasmtime.Trap, wasmtime.WasmtimeError):
+            # A trap can leave the module's state broken (e.g. a RefCell left 
borrowed).
+            self._rt = None
+            raise
+
+    def _call(self, rt: dict, n: int, placed: list, size: int, out_type: Any) 
-> Any:
+        import ctypes
+
+        import pyarrow as pa
+
+        store, memory = rt["store"], rt["memory"]
+        if size > rt["scratch_len"]:
+            rt["scratch_ptr"] = rt["scratch"](store, size) & 0xFFFFFFFF
+            rt["scratch_len"] = size
+        sp = rt["scratch_ptr"]
+        base = ctypes.addressof(memory.data_ptr(store).contents)
+
+        words = [n, len(placed)]
+        for (vo, vb), (xo, xb), (oo, ob) in placed:
+            words += [
+                sp + vo if vb else 0,
+                sp + xo if xb else 0,
+                xb.size if xb else 0,
+                sp + oo if ob else 0,
+                ob.size if ob else 0,
+            ]
+            for o, b in ((vo, vb), (xo, xb), (oo, ob)):
+                if b is not None:
+                    ctypes.memmove(base + sp + o, b.address, b.size)
+        desc = struct.pack(f"<{len(words)}I", *words)
+        ctypes.memmove(base + sp, desc, len(desc))
+
+        out = rt["entry"](store, sp) & 0xFFFFFFFF
+        base = ctypes.addressof(memory.data_ptr(store).contents)
+        status, nulls, vp, vl, xp, xl, op, ol = struct.unpack(
+            "<8I", ctypes.string_at(base + out, 32)
+        )
+        if status != 0:
+            message = ctypes.string_at(base + vp, vl).decode("utf-8", 
"replace")
+            raise ValueError(f"WASM UDF '{self.entrypoint}': {message}")
+
+        def copy_out(ptr: int, size: int) -> Any:
+            buf = pa.allocate_buffer(size)
+            if size:
+                ctypes.memmove(buf.address, base + ptr, size)
+            return buf
+
+        buffers = [copy_out(vp, vl) if vp else None]
+        if out_type.num_buffers == 3:
+            buffers.append(copy_out(op, ol))
+        buffers.append(copy_out(xp, xl))
+        return pa.Array.from_buffers(out_type, n, buffers, null_count=nulls)
+
+
+# --- packing: (runner, return_type) -> cloudpickled command bytes ------------
 
 
 def build_command(spec: dict) -> bytes:
@@ -256,9 +439,11 @@ def build_command(spec: dict) -> bytes:
     from pyspark.sql.types import _parse_datatype_json_value
 
     wasm = base64.b64decode(spec["wasm_b64"])
-    runner = WasmScalarUDF(
+    batch_entrypoint = spec.get("batch_entrypoint")
+    runner_class = WasmArrowUDF if batch_entrypoint else WasmScalarUDF
+    runner = runner_class(
         wasm,
-        spec["entrypoint"],
+        batch_entrypoint or spec["entrypoint"],
         spec["arg_types"],
         spec["ret_type"],
     )
diff --git a/crates/spark-connect/src/wasm_udf.rs 
b/crates/spark-connect/src/wasm_udf.rs
index ef36f3db..beb970c6 100644
--- a/crates/spark-connect/src/wasm_udf.rs
+++ b/crates/spark-connect/src/wasm_udf.rs
@@ -5,7 +5,7 @@
 //! Spark Connect has no native "run this Rust code" UDF type. What it *does*
 //! have is the [`PythonUDF`](crate::udf) path: a cloudpickled `(callable,
 //! return_type)` tuple shipped inside a `CommonInlineUserDefinedFunction` and
-//! executed row-by-row on the Spark executors by a Python worker.
+//! executed on the Spark executors by a Python worker.
 //!
 //! This module leverages that path to run **Rust** UDFs, distributed on the
 //! cluster, without touching the Spark server:
@@ -13,21 +13,27 @@
 //! 1. The user compiles their Rust function to a WebAssembly module.
 //! 2. [`udf`] embeds the `.wasm` bytes and the signature and runs a packer 
script
 //!    **embedded in this crate** as `python -c` (so it executes as 
`__main__`),
-//!    which uses `cloudpickle` to serialize a `WasmScalarUDF` runner **by 
value**
+//!    which uses `cloudpickle` to serialize a runner **by value**
 //!    into the `command`. Running it as `__main__` is what makes cloudpickle 
embed
 //!    the runner by value, so there is no separate `pyspark_wasm_udf` pip 
package to
 //!    install on the client and the executors need only the `wasmtime` 
package.
 //! 3. On each executor the runner instantiates the module with `wasmtime` and
-//!    invokes the exported entrypoint once per input row.
+//!    invokes the exported entrypoint once per Arrow batch (with a batch
+//!    entrypoint, see [`UserDefinedFunction::with_batch_entrypoint`]) or once 
per
+//!    input row.
 //!
 //! # ABI
 //!
-//! Arguments and the result cross the WASM boundary as a little-endian,
-//! length-prefixed byte buffer (see [`AbiType`]): the runner encodes the row's
-//! arguments into linear memory (via the module's exported `spark_udf_alloc`),
-//! calls the entrypoint `fn(ptr, len) -> packed_ptr_len`, and decodes the
-//! result. This supports scalars, `bool`, strings, binary, arrays, and
-//! nullability — not just numbers.
+//! For a per-row entrypoint, arguments and the result cross the WASM boundary
+//! as a little-endian, length-prefixed byte buffer (see [`AbiType`]): the 
runner
+//! encodes the row's arguments into linear memory (via the module's exported
+//! `spark_udf_alloc`), calls the entrypoint `fn(ptr, len) -> packed_ptr_len`, 
and
+//! decodes the result. This supports scalars, `bool`, strings, binary, arrays,
+//! and nullability — not just numbers.
+//!
+//! A batch entrypoint, which the 
[`spark_wasm_udf`](https://docs.rs/apache-spark-connect-macros)
+//! macro generates for the signatures its docs list, instead takes the Arrow
+//! buffers of a whole batch and returns the result's Arrow buffers.
 //!
 //! # API
 //!
@@ -66,7 +72,7 @@ use crate::column::Column;
 use crate::expression::ExpressionNode;
 use crate::types::DataType;
 use crate::udf::{eval_type, CommonInlineUserDefinedFunctionExpression, 
PythonUDFPayload};
-use spark_connect_core::error::Result;
+use spark_connect_core::error::{Result, SparkError};
 
 /// The type of one WASM UDF argument or its result, describing how the value 
is
 /// encoded across the WASM boundary and which Spark SQL type it maps to.
@@ -161,11 +167,27 @@ impl AbiType {
             AbiType::Nullable(inner) => inner.to_spark_json(),
         }
     }
+
+    fn batch_supported(&self) -> bool {
+        match self {
+            AbiType::I32
+            | AbiType::I64
+            | AbiType::F32
+            | AbiType::F64
+            | AbiType::Bool
+            | AbiType::Str
+            | AbiType::Binary => true,
+            AbiType::Array(_) => false,
+            AbiType::Nullable(inner) => {
+                !matches!(**inner, AbiType::Array(_) | AbiType::Nullable(_))
+            }
+        }
+    }
 }
 
 /// The packer script, embedded at compile time. Run as `python -c <this>` (so 
it
-/// executes as `__main__`), which makes cloudpickle serialize the 
`WasmScalarUDF`
-/// runner by value automatically — no `pyspark_wasm_udf` pip package is 
needed on
+/// executes as `__main__`), which makes cloudpickle serialize the runner by 
value
+/// automatically — no `pyspark_wasm_udf` pip package is needed on
 /// the client (only a Python interpreter with `pyspark`) or on the executors 
(only
 /// `wasmtime`).
 const PACKER_SRC: &str = include_str!("wasm_packer.py");
@@ -204,7 +226,6 @@ impl PythonPacker {
     /// stdin and returning the raw `command` bytes from stdout. Running it as 
`-c`
     /// (i.e. `__main__`) is what lets cloudpickle embed the runner by value.
     fn run(&self, spec_json: &str) -> Result<Vec<u8>> {
-        use spark_connect_core::error::SparkError;
         let mut cmd = Command::new(&self.python_exe);
         cmd.arg("-c")
             .arg(PACKER_SRC)
@@ -280,7 +301,8 @@ fn pathsep() -> &'static str {
 ///   Spark output type.
 ///
 /// The returned [`UserDefinedFunction`] is callable on columns via
-/// [`UserDefinedFunction::call`].
+/// [`UserDefinedFunction::call`], and calls `entrypoint` once per row unless
+/// given a batch entrypoint with 
[`UserDefinedFunction::with_batch_entrypoint`].
 pub fn udf(
     name: impl Into<String>,
     wasm_module: impl Into<Vec<u8>>,
@@ -298,6 +320,7 @@ pub fn udf(
         // Arrow-optimized Python UDF: func is still called once per row
         // (see pyspark worker `wrap_arrow_batch_udf`), but IO is Arrow.
         eval_type: eval_type::SQL_ARROW_BATCHED_UDF,
+        batch_entrypoint: None,
         python_ver: default_python_ver(),
         packer: PythonPacker::default(),
     }
@@ -314,6 +337,7 @@ pub struct UserDefinedFunction {
     ret_type: AbiType,
     deterministic: bool,
     eval_type: i32,
+    batch_entrypoint: Option<String>,
     python_ver: String,
     packer: PythonPacker,
 }
@@ -327,12 +351,25 @@ impl UserDefinedFunction {
     }
 
     /// Override the Python evaluation type (default
-    /// [`SQL_ARROW_BATCHED_UDF`](eval_type::SQL_ARROW_BATCHED_UDF)).
+    /// [`SQL_ARROW_BATCHED_UDF`](eval_type::SQL_ARROW_BATCHED_UDF), or
+    /// [`SQL_SCALAR_ARROW_UDF`](eval_type::SQL_SCALAR_ARROW_UDF) after
+    /// [`with_batch_entrypoint`](Self::with_batch_entrypoint)). Only
+    /// `SQL_SCALAR_ARROW_UDF` uses the batch entrypoint, so setting
+    /// `SQL_ARROW_BATCHED_UDF` calls the UDF once per row.
     pub fn with_eval_type(mut self, value: i32) -> Self {
         self.eval_type = value;
         self
     }
 
+    /// Call `entrypoint`, a batch export generated by `#[spark_wasm_udf]`, 
once per
+    /// Arrow batch, by sending the UDF as a scalar Arrow UDF
+    /// ([`SQL_SCALAR_ARROW_UDF`](eval_type::SQL_SCALAR_ARROW_UDF)).
+    pub fn with_batch_entrypoint(mut self, entrypoint: impl Into<String>) -> 
Self {
+        self.batch_entrypoint = Some(entrypoint.into());
+        self.eval_type = eval_type::SQL_SCALAR_ARROW_UDF;
+        self
+    }
+
     /// Override the Python version tag sent to the server (`"major.minor"`).
     pub fn with_python_ver(mut self, value: impl Into<String>) -> Self {
         self.python_ver = value.into();
@@ -351,20 +388,48 @@ impl UserDefinedFunction {
     }
 
     /// Build the JSON spec handed to the Python packer.
-    fn build_spec(&self) -> String {
+    fn build_spec(&self) -> Result<String> {
+        let batch_entrypoint = if self.eval_type == 
eval_type::SQL_SCALAR_ARROW_UDF {
+            let Some(entrypoint) = &self.batch_entrypoint else {
+                return Err(SparkError::value_msg(format!(
+                    "WASM UDF '{}' has eval type SQL_SCALAR_ARROW_UDF, which 
needs a batch \
+                     entrypoint; set one with with_batch_entrypoint",
+                    self.name
+                )));
+            };
+            if self.arg_types.is_empty() {
+                return Err(SparkError::value_msg(format!(
+                    "WASM UDF '{}' has no arguments, so it cannot be called 
once per batch",
+                    self.name
+                )));
+            }
+            let mut types = self.arg_types.iter().chain([&self.ret_type]);
+            if let Some(ty) = types.find(|t| !t.batch_supported()) {
+                return Err(SparkError::value_msg(format!(
+                    "WASM UDF '{}' has type '{}', so it cannot be called once 
per batch; \
+                     call it per row with 
with_eval_type(SQL_ARROW_BATCHED_UDF)",
+                    self.name,
+                    ty.descriptor()
+                )));
+            }
+            Some(entrypoint)
+        } else {
+            None
+        };
         let spec = serde_json::json!({
             "wasm_b64": 
base64::engine::general_purpose::STANDARD.encode(&self.wasm_module),
             "entrypoint": self.entrypoint,
+            "batch_entrypoint": batch_entrypoint,
             "arg_types": self.arg_types.iter().map(|t| 
t.descriptor()).collect::<Vec<_>>(),
             "ret_type": self.ret_type.descriptor(),
             "output_type": self.ret_type.to_spark_json(),
         });
-        spec.to_string()
+        Ok(spec.to_string())
     }
 
     /// Build the cloudpickle-compatible `command` bytes by invoking the 
packer.
     fn build_command(&self) -> Result<Vec<u8>> {
-        self.packer.run(&self.build_spec())
+        self.packer.run(&self.build_spec()?)
     }
 
     /// Build the [`PythonUDFPayload`] carrying the pickled command.
@@ -564,10 +629,13 @@ mod tests {
         );
     }
 
+    fn spec_of(f: &UserDefinedFunction) -> serde_json::Value {
+        serde_json::from_str(&f.build_spec().unwrap()).unwrap()
+    }
+
     #[test]
     fn build_spec_encodes_signature_and_wasm() {
-        let f = sample();
-        let spec: serde_json::Value = 
serde_json::from_str(&f.build_spec()).unwrap();
+        let spec = spec_of(&sample());
         assert_eq!(spec["entrypoint"], "add_one");
         assert_eq!(spec["arg_types"], serde_json::json!(["i64"]));
         assert_eq!(spec["ret_type"], "i64");
@@ -575,6 +643,67 @@ mod tests {
         assert_eq!(spec["wasm_b64"], "AGFzbQEA");
     }
 
+    #[test]
+    fn eval_type_decides_whether_the_spec_carries_the_batch_entrypoint() {
+        let batch = sample().with_batch_entrypoint("spark_batch_udf_add_one");
+        let per_row = batch
+            .clone()
+            .with_eval_type(eval_type::SQL_ARROW_BATCHED_UDF);
+        assert_eq!(batch.eval_type, eval_type::SQL_SCALAR_ARROW_UDF);
+        assert_eq!(
+            spec_of(&batch)["batch_entrypoint"],
+            "spark_batch_udf_add_one"
+        );
+        assert_eq!(per_row.eval_type, eval_type::SQL_ARROW_BATCHED_UDF);
+        assert_eq!(
+            spec_of(&per_row)["batch_entrypoint"],
+            serde_json::Value::Null
+        );
+    }
+
+    #[test]
+    fn scalar_arrow_eval_type_without_a_usable_batch_entrypoint_is_rejected() {
+        let no_entrypoint = 
sample().with_eval_type(eval_type::SQL_SCALAR_ARROW_UDF);
+        let no_args = udf("now", vec![0x00], "now", vec![], AbiType::I64)
+            .with_batch_entrypoint("spark_batch_udf_now");
+        let array_arg = udf(
+            "sum",
+            vec![0x00],
+            "sum",
+            vec![AbiType::Array(Box::new(AbiType::I64))],
+            AbiType::I64,
+        )
+        .with_batch_entrypoint("spark_batch_udf_sum");
+        let nested_option_ret = udf(
+            "wrap",
+            vec![0x00],
+            "wrap",
+            vec![AbiType::I64],
+            
AbiType::Nullable(Box::new(AbiType::Nullable(Box::new(AbiType::I64)))),
+        )
+        .with_batch_entrypoint("spark_batch_udf_wrap");
+        let option_array_arg = udf(
+            "first",
+            vec![0x00],
+            "first",
+            vec![AbiType::Nullable(Box::new(AbiType::Array(Box::new(
+                AbiType::I64,
+            ))))],
+            AbiType::I64,
+        )
+        .with_batch_entrypoint("spark_batch_udf_first");
+        for (f, expected) in [
+            (no_entrypoint, "needs a batch entrypoint"),
+            (no_args, "has no arguments"),
+            (array_arg, "has type 'array:i64'"),
+            (nested_option_ret, "has type 'option:option:i64'"),
+            (option_array_arg, "has type 'option:array:i64'"),
+        ] {
+            let err = f.to_payload().unwrap_err().message();
+            assert!(err.contains(expected), "{err}");
+        }
+    }
+
     /// End-to-end packer test. Requires only a Python interpreter with 
`pyspark`
     /// importable (for its vendored `cloudpickle` + the pure-Python type 
parser);
     /// the packer script is embedded in the crate. Point `python_exe` at that
diff --git a/docs/architecture.md b/docs/architecture.md
index abbc4f77..cb8663de 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -211,7 +211,7 @@ executed with `wasmtime`, so nothing Rust-specific is 
needed server-side.
 flowchart LR
     RUSTFN["#[spark_wasm_udf] fn"] -->|"build.rs -> wasm32"| WASM[".wasm 
module (embedded)"]
     WASM -->|"cloudpickled into a PythonUDF"| SRV["Spark executors"]
-    SRV -->|"wasmtime runs it per row"| RES["UDF result column"]
+    SRV -->|"wasmtime runs it per batch or row"| RES["UDF result column"]
 ```
 
 See [Rust UDFs via WebAssembly](udfs.md) for the full story.
diff --git a/docs/udfs.md b/docs/udfs.md
index 580cd2fa..e5e91e8b 100644
--- a/docs/udfs.md
+++ b/docs/udfs.md
@@ -87,7 +87,7 @@ cargo run
 
 ## Supported types
 
-Arguments and results cross the WASM boundary with a length-prefixed binary ABI
+Arguments and results cross the WASM boundary in a binary ABI
 (`spark_connect::wasm_udf::AbiType`), inferred from the Rust signature:
 
 | Rust        | Spark SQL type       |
@@ -104,6 +104,10 @@ Arguments and results cross the WASM boundary with a 
length-prefixed binary ABI
 
 These nest arbitrarily - e.g. `Vec<Option<String>>` -> `ArrayType(StringType, 
nullable)`.
 
+A UDF whose arguments and result are all scalars, `String`, `Vec<u8>`, or one
+`Option` of those is sent as a scalar Arrow UDF and called once per Arrow 
batch;
+any other UDF is called once per row.
+
 ## How it works
 
 ```text
@@ -115,7 +119,8 @@ client       udf::add_one(col) builds a standard PythonUDF: 
a tiny Python runner
              `wasmtime` - nothing to pre-deploy.
 
 executors    the Python worker instantiates the module with wasmtime and 
invokes
-             the exported entrypoint once per input row over the binary ABI.
+             the exported entrypoint once per Arrow batch, or once per input 
row
+             for signatures with arrays or nested Options.
 ```
 
 The user never writes WASM, never touches the ABI, and never loads a `.wasm`
@@ -147,6 +152,16 @@ udf::add_one_udf()
     .call(vec![col("id")])?;
 ```
 
+To call a UDF once per row instead of once per batch, set its eval type:
+
+```{.rust .no-run}
+use spark_connect::udf::eval_type;
+
+udf::add_one_udf()
+    .with_eval_type(eval_type::SQL_ARROW_BATCHED_UDF)
+    .call(vec![col("id")])?;
+```
+
 If you'd rather load a prebuilt module and spell out the types yourself, the
 lower-level factory mirrors `pyspark.sql.functions.udf`:
 
diff --git a/tests/wasm-udf/Cargo.toml b/tests/wasm-udf/Cargo.toml
new file mode 100644
index 00000000..85989538
--- /dev/null
+++ b/tests/wasm-udf/Cargo.toml
@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Fixture UDFs and the tests that run them: through the runners the packer 
ships
+# (needs Python with pyspark, pyarrow and wasmtime), and end to end on a Spark
+# Connect server.
+[package]
+name = "wasm-udf-tests"
+version = "0.0.0"
+edition = "2021"
+publish = false
+
+[dependencies]
+spark-connect-macros = { package = "apache-spark-connect-macros", path = 
"../../crates/spark-connect-macros" }
+
+[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
+spark-connect = { package = "apache-spark-connect", path = 
"../../crates/spark-connect", features = ["wasm-udf"] }
+
+[build-dependencies]
+spark-connect-build = { package = "apache-spark-connect-build", path = 
"../../crates/spark-connect-build" }
+spark-connect-macros = { package = "apache-spark-connect-macros", path = 
"../../crates/spark-connect-macros" }
diff --git a/tests/wasm-udf/build.rs b/tests/wasm-udf/build.rs
new file mode 100644
index 00000000..14858482
--- /dev/null
+++ b/tests/wasm-udf/build.rs
@@ -0,0 +1,20 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+fn main() {
+    spark_connect_build::embed_wasm_udf("src/lib.rs");
+}
diff --git a/tests/wasm-udf/src/lib.rs b/tests/wasm-udf/src/lib.rs
new file mode 100644
index 00000000..c65ff83e
--- /dev/null
+++ b/tests/wasm-udf/src/lib.rs
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use spark_connect_macros::spark_wasm_udf;
+
+#[spark_wasm_udf]
+mod fixtures {
+    use std::cell::RefCell;
+
+    thread_local! {
+        static STATE: RefCell<()> = const { RefCell::new(()) };
+    }
+
+    pub fn sub_i32(x: i32, y: Option<i32>) -> Option<i32> {
+        y.map(|y| x.wrapping_sub(y))
+    }
+
+    pub fn sub_i64(x: i64, y: Option<i64>) -> Option<i64> {
+        y.map(|y| x.wrapping_sub(y))
+    }
+
+    pub fn sub_f32(x: f32, y: Option<f32>) -> Option<f32> {
+        y.map(|y| x - y)
+    }
+
+    pub fn sub_f64(x: f64, y: Option<f64>) -> Option<f64> {
+        y.map(|y| x - y)
+    }
+
+    pub fn and_not(x: bool, y: Option<bool>) -> Option<bool> {
+        y.map(|y| x && !y)
+    }
+
+    pub fn concat(x: String, y: Option<String>) -> Option<String> {
+        y.map(|y| x + &y)
+    }
+
+    pub fn concat_bytes(x: Vec<u8>, y: Option<Vec<u8>>) -> Option<Vec<u8>> {
+        y.map(|y| [x, y].concat())
+    }
+
+    pub fn label(name: Option<String>, n: i64) -> String {
+        format!("{}#{n}", name.unwrap_or_default())
+    }
+
+    pub fn keep_first(x: i64, _: i64) -> i64 {
+        x
+    }
+
+    pub fn sum(xs: Vec<i64>) -> i64 {
+        xs.iter().sum()
+    }
+
+    // Panics while `STATE` is borrowed, which stays borrowed in the trapped 
instance.
+    pub fn non_negative(x: i64) -> i64 {
+        STATE.with(|s| {
+            let _borrow = s.borrow_mut();
+            assert!(x >= 0, "negative input");
+            x
+        })
+    }
+
+    // Their exports would collide with `spark_udf_<fn>` ones under prefixes 
like
+    // `spark_udf_batch_<fn>` and `spark_udf_scratch`.
+    pub fn scratch(x: i64) -> i64 {
+        x
+    }
+
+    pub fn batch_scratch(x: i64) -> i64 {
+        x
+    }
+}
diff --git a/tests/wasm-udf/tests/e2e_wasm_udf.rs 
b/tests/wasm-udf/tests/e2e_wasm_udf.rs
new file mode 100644
index 00000000..754d3ff0
--- /dev/null
+++ b/tests/wasm-udf/tests/e2e_wasm_udf.rs
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! WASM UDFs on a live Spark Connect server, whose workers need `pyarrow` and
+//! `wasmtime`. Run with `SPARK_REMOTE` set; skipped otherwise.
+
+use spark_connect::column::col;
+use spark_connect::row::Value;
+use spark_connect::session::SparkSession;
+use wasm_udf_tests::udf;
+
+const TABLE: &str =
+    "SELECT * FROM VALUES (5L, 2L, 'a', array(1L, 2L)), (7L, NULL, NULL, 
array(3L)) \
+                     AS t(x, y, s, xs) ORDER BY x";
+
+#[test]
+fn wasm_udfs_run_on_the_server() {
+    let Ok(url) = std::env::var("SPARK_REMOTE") else {
+        return;
+    };
+    let spark = SparkSession::builder()
+        .remote(&url)
+        .get_or_create()
+        .expect("session");
+    let rows = spark
+        .sql(TABLE)
+        .unwrap()
+        .select([
+            udf::sub_i64(col("x"), col("y")).unwrap(),
+            udf::label(col("s"), col("x")).unwrap(),
+            udf::sum(col("xs")).unwrap(),
+        ])
+        .collect()
+        .unwrap();
+    let rows: Vec<&[Value]> = rows.iter().map(|r| r.values()).collect();
+    assert_eq!(
+        rows,
+        [
+            [Value::Long(3), Value::String("a#5".into()), Value::Long(3)],
+            [Value::Null, Value::String("#7".into()), Value::Long(3)],
+        ]
+    );
+
+    spark
+        .udf()
+        .register("wasm_label", &udf::label_udf())
+        .unwrap();
+    let rows = spark
+        .sql(&format!("SELECT wasm_label(s, x) FROM ({TABLE})"))
+        .unwrap()
+        .collect()
+        .unwrap();
+    let labels: Vec<&Value> = rows.iter().map(|r| &r.values()[0]).collect();
+    assert_eq!(
+        labels,
+        [&Value::String("a#5".into()), &Value::String("#7".into())]
+    );
+}
diff --git a/tests/wasm-udf/tests/runners.rs b/tests/wasm-udf/tests/runners.rs
new file mode 100644
index 00000000..952d2482
--- /dev/null
+++ b/tests/wasm-udf/tests/runners.rs
@@ -0,0 +1,242 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The runners the packer ships to the executors, driven with pyarrow arrays 
the
+//! way Spark's Python worker drives them. Needs a Python with `pyspark`, 
`pyarrow`
+//! and `wasmtime` (`$SPARK_CONNECT_PYTHON`, as for the packer). Run with
+//! `cargo test --manifest-path tests/wasm-udf/Cargo.toml -- --ignored`.
+
+use std::io::Write;
+use std::process::{Command, Stdio};
+
+use spark_connect::udf::eval_type;
+use spark_connect::wasm_udf::{self, AbiType, PythonPacker, 
UserDefinedFunction};
+use wasm_udf_tests::udf;
+
+const PRELUDE: &str = r#"
+import copy
+import pickle
+import sys
+
+import pyarrow as pa
+import wasmtime
+
+
+def raises(exc, needle, f, *args):
+    try:
+        f(*args)
+    except exc as e:
+        assert needle in str(e), (needle, str(e))
+    else:
+        raise AssertionError(f"expected {exc.__name__}")
+"#;
+
+/// Runs `check` in Python with `udfs[name]` bound to each UDF's unpickled 
runner.
+fn run_python(udfs: &[(&str, UserDefinedFunction)], check: &str) {
+    let mut script = format!("{PRELUDE}\nudfs = {{}}\n");
+    for (name, f) in udfs {
+        let command = f.to_payload().expect("packer should succeed").command;
+        let hex: String = command.iter().map(|b| format!("{b:02x}")).collect();
+        script += &format!("udfs[{name:?}] = 
pickle.loads(bytes.fromhex({hex:?}))[0]\n");
+    }
+    script += check;
+
+    let mut child = Command::new(PythonPacker::default().python_exe)
+        .arg("-")
+        .stdin(Stdio::piped())
+        .stdout(Stdio::piped())
+        .stderr(Stdio::piped())
+        .spawn()
+        .expect("spawn python");
+    child
+        .stdin
+        .take()
+        .unwrap()
+        .write_all(script.as_bytes())
+        .unwrap();
+    let output = child.wait_with_output().unwrap();
+    let stderr = String::from_utf8_lossy(&output.stderr);
+    assert!(output.status.success(), "{stderr}");
+    assert!(!stderr.contains("Exception ignored"), "{stderr}");
+}
+
+fn per_row(f: UserDefinedFunction) -> UserDefinedFunction {
+    f.with_eval_type(eval_type::SQL_ARROW_BATCHED_UDF)
+}
+
+#[test]
+#[ignore]
+fn only_batch_capable_signatures_get_the_batch_runner() {
+    run_python(
+        &[
+            ("scalars", udf::sub_i64_udf()),
+            ("array", udf::sum_udf()),
+            ("forced_per_row", per_row(udf::sub_i64_udf())),
+        ],
+        r#"
+assert type(udfs["scalars"]).__name__ == "WasmArrowUDF"
+assert type(udfs["array"]).__name__ == "WasmScalarUDF"
+assert type(udfs["forced_per_row"]).__name__ == "WasmScalarUDF"
+"#,
+    );
+}
+
+#[test]
+#[ignore]
+fn batch_runner_returns_what_the_per_row_runner_returns() {
+    let fixtures = [
+        ("sub_i32", udf::sub_i32_udf()),
+        ("sub_i64", udf::sub_i64_udf()),
+        ("sub_f32", udf::sub_f32_udf()),
+        ("sub_f64", udf::sub_f64_udf()),
+        ("and_not", udf::and_not_udf()),
+        ("concat", udf::concat_udf()),
+        ("concat_bytes", udf::concat_bytes_udf()),
+        ("label", udf::label_udf()),
+    ];
+    let row_names: Vec<String> = fixtures.iter().map(|(n, _)| 
format!("{n}_row")).collect();
+    let mut udfs = Vec::new();
+    for ((name, f), row_name) in fixtures.into_iter().zip(&row_names) {
+        udfs.push((row_name.as_str(), per_row(f.clone())));
+        udfs.push((name, f));
+    }
+    run_python(
+        &udfs,
+        r##"
+cases = [
+    ("sub_i32", pa.int32(), [pa.array([5, -7, 2**31 - 1, 0], pa.int32()), 
pa.array([2, None, -1, 0], pa.int32())], [3, None, -2**31, 0]),
+    ("sub_i64", pa.int64(), [pa.array([5, -7, 0], pa.int64()), pa.array([2, 
None, 0], pa.int64())], [3, None, 0]),
+    ("sub_f32", pa.float32(), [pa.array([1.5, 2.0], pa.float32()), 
pa.array([0.25, None], pa.float32())], [1.25, None]),
+    ("sub_f64", pa.float64(), [pa.array([1.5, -2.0]), pa.array([None, 0.5])], 
[None, -2.5]),
+    ("and_not", pa.bool_(),
+     [pa.array([True, True, False, True, False, True, True, False, True]),
+      pa.array([False, True, False, None, True, False, False, False, None])],
+     [True, False, False, None, False, True, True, False, None]),
+    ("concat", pa.string(), [pa.array(["a", "", "hé"]), pa.array(["b", None, 
"!"])], ["ab", None, "hé!"]),
+    ("concat", pa.string(), [pa.array(["", ""]), pa.array(["", ""])], ["", 
""]),
+    ("concat_bytes", pa.binary(), [pa.array([b"\x00", b"", b"xy"]), 
pa.array([b"\xff", None, b""])], [b"\x00\xff", None, b"xy"]),
+    ("label", pa.string(), [pa.array(["a", None]), pa.array([1, -2], 
pa.int64())], ["a#1", "#-2"]),
+    ("sub_i64", pa.int64(), [pa.array([], pa.int64()), pa.array([], 
pa.int64())], []),
+    ("concat", pa.string(), [pa.array([], pa.string()), pa.array([], 
pa.string())], []),
+]
+for name, out_type, args, expected in cases:
+    got = udfs[name](*args)
+    assert got.type == out_type, (name, got.type)
+    assert got.to_pylist() == expected, (name, got.to_pylist(), expected)
+    rows = zip(*(a.to_pylist() for a in args))
+    assert [udfs[name + "_row"](*row) for row in rows] == expected, name
+assert len(sys._spark_connect_wasm_udf_modules) == 1
+"##,
+    );
+}
+
+#[test]
+#[ignore]
+fn batch_runner_accepts_sliced_chunked_large_and_wider_inputs() {
+    run_python(
+        &[
+            ("sub_i32", udf::sub_i32_udf()),
+            ("sub_i64", udf::sub_i64_udf()),
+            ("and_not", udf::and_not_udf()),
+            ("concat", udf::concat_udf()),
+            ("concat_bytes", udf::concat_bytes_udf()),
+        ],
+        r#"
+x = pa.array([100, 1, 2, 3, 4, 5, 6, 7, 8, 9], pa.int64()).slice(3)
+y = pa.array([None, 0, 0, 1, None, 1, 1, 1, 1, None], pa.int64()).slice(3)
+assert udfs["sub_i64"](x, y).to_pylist() == [2, None, 4, 5, 6, 7, None]
+x = pa.array([False] * 5 + [True, True, False, True, True, False]).slice(5)
+y = pa.array([True] * 5 + [False, None, True, True, False, False]).slice(5)
+assert udfs["and_not"](x, y).to_pylist() == [True, None, False, False, True, 
False]
+x, y = pa.array(["skip", "a", "bb"]).slice(1), pa.array(["z", "1", 
None]).slice(1)
+assert udfs["concat"](x, y).to_pylist() == ["a1", None]
+
+for n in (2, 50_000, 3):
+    x = ["ab" * (i % 17) for i in range(n)]
+    y = [None if i % 7 == 3 else str(i) for i in range(n)]
+    expected = [None if b is None else a + b for a, b in zip(x, y)]
+    assert udfs["concat"](pa.array(x), pa.array(y)).to_pylist() == expected, n
+
+x = pa.chunked_array([[5], [7, 9]], pa.int64())
+y = pa.chunked_array([[1, None], [2]], pa.int64())
+assert udfs["sub_i64"](x, y).to_pylist() == [4, None, 7]
+
+x, y = pa.array(["a", "b"], pa.large_string()), pa.array(["1", None], 
pa.large_string())
+assert udfs["concat"](x, y).to_pylist() == ["a1", None]
+x, y = pa.array([b"a"], pa.large_binary()), pa.array([b"1"], pa.large_binary())
+assert udfs["concat_bytes"](x, y).to_pylist() == [b"a1"]
+
+assert udfs["sub_i32"](pa.array([5, 7], pa.int64()), pa.array([2, 3], 
pa.int64())).to_pylist() == [3, 4]
+raises(pa.ArrowInvalid, "not in range", udfs["sub_i32"], pa.array([2**40]), 
pa.array([1]))
+"#,
+    );
+}
+
+#[test]
+#[ignore]
+fn batch_runner_errors_name_the_cause_and_the_next_batch_succeeds() {
+    const MODULE: &[u8] = include_bytes!(env!("WASM_UDFS_MODULE"));
+    let one_column = wasm_udf::udf(
+        "sub_i64",
+        MODULE,
+        "spark_udf_sub_i64",
+        vec![AbiType::I64],
+        AbiType::Nullable(Box::new(AbiType::I64)),
+    )
+    .with_batch_entrypoint("spark_batch_udf_sub_i64");
+    run_python(
+        &[
+            ("sub_i64", udf::sub_i64_udf()),
+            ("one_column", one_column),
+            (
+                "missing",
+                
udf::sub_i64_udf().with_batch_entrypoint("spark_batch_udf_missing"),
+            ),
+            ("keep_first", udf::keep_first_udf()),
+            ("non_negative", udf::non_negative_udf()),
+        ],
+        r#"
+f = udfs["sub_i64"]
+ok = [pa.array([3], pa.int64()), pa.array([1], pa.int64())]
+
+raises(ValueError, "argument `x` is null in row 1", f, pa.array([1, None]), 
pa.array([1, 1]))
+assert f(*ok).to_pylist() == [2]
+raises(ValueError, "argument `#1` is null in row 0", udfs["keep_first"], 
ok[0], pa.array([None], pa.int64()))
+raises(ValueError, "got arguments of different lengths: [3, 2]", f, 
pa.array([1, 2, 3]), pa.array([1, 2]))
+raises(ValueError, "got arguments of different lengths: [0, 1]", f, 
pa.array([], pa.int64()), ok[1])
+assert f(*ok).to_pylist() == [2]
+raises(ValueError, "does not export 'spark_batch_udf_missing'", 
udfs["missing"], *ok)
+raises(ValueError, "expected 2 argument(s), got 1", f, ok[0])
+assert f(*ok).to_pylist() == [2]
+raises(ValueError, "the batch has 1 columns, but the function takes 2", 
udfs["one_column"], ok[0])
+
+big = pa.Array.from_buffers(pa.int64(), 2**27, [None, 
pa.allocate_buffer(2**30)])
+raises(ValueError, "maxRecordsPerBatch", f, big, big)
+assert f(*ok).to_pylist() == [2]
+
+saved, sys.modules["wasmtime"] = sys.modules["wasmtime"], None
+try:
+    raises(ImportError, "The 'wasmtime' package is required", copy.copy(f), 
*ok)
+finally:
+    sys.modules["wasmtime"] = saved
+
+g = udfs["non_negative"]
+raises(wasmtime.Trap, "unreachable", g, pa.array([1, -1], pa.int64()))
+assert g(pa.array([4], pa.int64())).to_pylist() == [4]
+"#,
+    );
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to