This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new ffc2fab [SPARK-59033][INFRA][DOCS] Coverage badges at branch root +
fix udfs page rendering
ffc2fab is described below
commit ffc2fab6171b102667e4b70c6b6a7341bf374c45
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Aug 27 08:46:29 2026 +0900
[SPARK-59033][INFRA][DOCS] Coverage badges at branch root + fix udfs page
rendering
### What changes were proposed in this pull request?
Publish the coverage badge JSON at the **root** of the `badges` branch
(`coverage-rust.json`, `coverage-python.json`) instead of inside a `badges/`
subdirectory.
### Why are the changes needed?
The README shields.io endpoint badges read:
-
`https://raw.githubusercontent.com/apache/spark-connect-rust/badges/coverage-rust.json`
-
`https://raw.githubusercontent.com/apache/spark-connect-rust/badges/coverage-python.json`
i.e. files at the root of the `badges` branch. But `coverage.yml` copied
them into a
`badges/` subdirectory before committing, so the published path was
`.../badges/badges/coverage-rust.json` and the badges rendered as
**"resource not
found"** (404). This writes the JSON at the branch root so the endpoints
resolve. It
takes effect on the next master push (the workflow publishes on push to
master).
### Does this PR introduce _any_ user-facing change?
No — CI/badge publishing only.
### How was this patch tested?
YAML validated; the publish step now `cp`s the badge JSON to the branch
root and
`git add`s the root files. Confirmed the README URLs point at the root
paths.
This pull request and its description were written by Isaac.
Closes #59 from HyukjinKwon/fix-coverage-badges.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/coverage.yml | 21 ++++++-------------
README.md | 46 +++++++++++-------------------------------
docs/udfs.md | 16 +++++++--------
scripts/check_doc_snippets.py | 13 ++++++++----
4 files changed, 34 insertions(+), 62 deletions(-)
diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml
index cfe68c4..1f847c7 100644
--- a/.github/workflows/coverage.yml
+++ b/.github/workflows/coverage.yml
@@ -173,14 +173,6 @@ jobs:
print(f"Python coverage: {coverage_pct:.1f}% ({color})")
PYTHON_EOF
- - name: Ensure badges directory exists
- run: mkdir -p badges
-
- - name: Copy badges to output directory
- run: |
- cp coverage-rust-badge.json badges/coverage-rust.json
- cp coverage-python-badge.json badges/coverage-python.json
-
- name: Check if badges branch exists
id: check-badges-branch
run: |
@@ -209,13 +201,12 @@ jobs:
git rm -rf .
fi
- # Copy new badges into the branch
- mkdir -p badges
- cp coverage-rust-badge.json badges/coverage-rust.json
- cp coverage-python-badge.json badges/coverage-python.json
+ # Copy new badges into the branch root (not in a subdirectory)
+ cp coverage-rust-badge.json coverage-rust.json
+ cp coverage-python-badge.json coverage-python.json
# Commit and push
- git add badges/
+ git add coverage-rust.json coverage-python.json
git commit -m "ci: update coverage badges" || echo "No changes to
commit"
git push -f origin badges
@@ -225,6 +216,6 @@ jobs:
with:
name: coverage-badges
path: |
- badges/coverage-rust.json
- badges/coverage-python.json
+ coverage-rust-badge.json
+ coverage-python-badge.json
retention-days: 5
diff --git a/README.md b/README.md
index f105f5e..fab4b9d 100644
--- a/README.md
+++ b/README.md
@@ -20,9 +20,10 @@
# Spark Connect Rust Client
A fast, native **Rust** client for **Apache Spark Connect** - and a drop-in
-`pyspark` replacement. It builds `spark.connect` protobuf plans, manages the
gRPC
-channel, and decodes Arrow results in Rust, speaking the same protocol and
-returning the same results as the reference client.
+`pyspark` replacement with **100% public-API parity with PySpark 4.2.0**. It
+builds `spark.connect` protobuf plans, manages the gRPC channel, and decodes
+Arrow results in Rust, speaking the same protocol and returning the same
results
+as the reference client.
[](https://pypi.org/project/pyspark-client-rust/)
@@ -84,45 +85,22 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
See the [documentation](https://apache.github.io/spark-connect-rust/) for the
full
API, running a Spark Connect server, and more.
-## API Parity & Drop-in Replacement
-
-The Python drop-in (`pyspark-client-rust`) achieves **100% public-API parity
with
-PySpark 4.2.0**. The Rust client (crate `apache-spark-connect`) has the same
-parity and supports the complete DataFrame, SQL, Streaming, Catalog, and Type
-system. The two internal APIs intentionally absent (`Column.to_plan` and
-`SparkSession.client`) are implementation details and not part of the public
-interface.
-
## User-Defined Functions (UDFs)
-Rust UDFs are compiled to WebAssembly, shipped to the executors, and can be
used
-via the DataFrame API or registered by name for SQL. The runner is cloudpickled
-by value, so executors need only the `wasmtime` Python package:
+Write UDFs as plain Rust functions and call them directly — `#[spark_wasm_udf]`
+compiles them to WebAssembly and ships them to the executors:
```rust
-use spark_connect::functions::col;
-use spark_connect::wasm_udf::{udf, AbiType};
-
-fn main() -> Result<(), Box<dyn std::error::Error>> {
- let spark =
SparkSession::builder().remote("sc://localhost:15002").get_or_create()?;
-
- // Load a prebuilt module: fn shout(s: String) -> String
- let wasm = std::fs::read("shout.wasm")?;
- let shout = udf("shout", wasm, "shout", vec![AbiType::Str], AbiType::Str);
-
- // DataFrame API:
- let df = spark.range(0,
3)?.select(vec![shout.call(vec![col("id").cast_str("string")])?])?;
- df.show(20)?;
-
- // ...or register by name and call from SQL (mirrors spark.udf.register):
- spark.udf().register("shout", &shout)?;
- spark.sql("SELECT shout(name) FROM people")?.show(20)?;
- Ok(())
+#[spark_wasm_udf]
+mod udfs {
+ pub fn add_one(x: i64) -> i64 { x + 1 }
}
+
+spark.range(5)?.select(vec![udf::add_one(col("id"))?]).show(20)?;
```
See the [WASM UDF guide](https://apache.github.io/spark-connect-rust/udfs/) for
-building modules with the `spark_wasm_udf` macro and more examples.
+the full setup, SQL registration, supported types, and more.
## Contributing
diff --git a/docs/udfs.md b/docs/udfs.md
index 61fe9eb..f8d517b 100644
--- a/docs/udfs.md
+++ b/docs/udfs.md
@@ -14,11 +14,11 @@ toolchain server-side.
Annotate a module of functions with `#[spark_wasm_udf]`. For each function the
macro infers the Spark signature from the Rust types, exports it to WASM, and
-generates a constructor under `udf::` - a direct call `udf::<name>(col0, col1,
-...)` (one column per argument, **arity checked at compile time**) that
returns the
+generates a constructor under `udf::` - a direct call `udf::<name>(col0, col1,
...)`
+(one column per argument, **arity checked at compile time**) that returns the
result `Column`. A one-line `build.rs` compiles the module.
-```rust,ignore
+```{.rust .no-run}
// src/main.rs
use spark_connect_macros::spark_wasm_udf;
@@ -52,7 +52,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
}
```
-```rust,ignore
+```{.rust .no-run}
// build.rs
fn main() {
spark_connect_build::embed_wasm_udf("src/main.rs");
@@ -77,7 +77,6 @@ Run it:
```bash
rustup target add wasm32-unknown-unknown # once
-export SPARK_CONNECT_WASM_PACKER_PATH=$PWD/python # so the client finds the
packer
cargo run
```
@@ -103,8 +102,7 @@ Arguments and results cross the WASM boundary with a
length-prefixed binary ABI
| `Vec<T>` | `ArrayType` (of `T`) |
| `Option<T>` | nullable `T` |
-These nest arbitrarily - e.g. `Vec<Option<String>>` -> `ArrayType(StringType,
-nullable)`.
+These nest arbitrarily - e.g. `Vec<Option<String>>` -> `ArrayType(StringType,
nullable)`.
## How it works
@@ -143,7 +141,7 @@ Nothing here is needed unless the `wasm-udf` feature is
enabled:
For non-deterministic UDFs or custom packer configuration, the macro also
generates a builder `udf::<name>_udf()`:
-```rust,ignore
+```{.rust .no-run}
udf::add_one_udf()
.as_nondeterministic()
.call(vec![col("id")])?;
@@ -152,7 +150,7 @@ udf::add_one_udf()
If you'd rather load a prebuilt module and spell out the types yourself, the
lower-level factory mirrors `pyspark.sql.functions.udf`:
-```rust,ignore
+```{.rust .no-run}
use spark_connect::wasm_udf::{udf, AbiType};
let wasm = std::fs::read("shout.wasm")?;
diff --git a/scripts/check_doc_snippets.py b/scripts/check_doc_snippets.py
index 112fd11..f17404b 100644
--- a/scripts/check_doc_snippets.py
+++ b/scripts/check_doc_snippets.py
@@ -27,8 +27,13 @@ common "continue from an earlier block" fragments
type-check) and is run through
``rustc`` against the already-built ``apache-spark-connect`` rlib. Only
compilation
(type-checking) is performed - snippets are never executed, so no server is
needed.
-A block that must be skipped (pseudo-code, or a deliberate compile-error
example)
-can be fenced as ```rust,ignore``` and it is not checked.
+A block that must be skipped (pseudo-code, a deliberate compile-error example,
or
+a snippet that needs a feature/macro the snippet harness doesn't build - e.g.
the
+`wasm-udf` `#[spark_wasm_udf]` examples) should be fenced as ```{.rust
.no-run}```.
+That brace form still renders as a Rust code block in mkdocs/superfences
(unlike
+```rust,ignore```, whose comma makes superfences drop the fence entirely and
print
+the backticks literally), and it is not matched by the checker below, so it is
not
+compiled.
Usage:
cargo build -p apache-spark-connect
@@ -48,8 +53,8 @@ REPO = Path(__file__).resolve().parent.parent
RLIB = REPO / "target" / "debug" / "libspark_connect.rlib"
DEPS = REPO / "target" / "debug" / "deps"
-# ```rust``` (checked) vs ```rust,ignore``` / ```rust,no_run``` (skipped). We
only
-# check plain ```rust``` fences.
+# ```rust``` (checked) vs ```{.rust .no-run}``` (rendered but skipped). We
only check
+# plain ```rust``` fences; any info string with a comma/space/brace is left
alone.
_BLOCK = re.compile(r"^```rust\s*$\n(.*?)^```\s*$", re.DOTALL | re.MULTILINE)
# Harness: bind the identifiers the fragments assume from earlier blocks. `use`
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
