This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 633f1c0 [SPARK-58990] Add a MkDocs documentation site and slim the
README (#53)
633f1c0 is described below
commit 633f1c08bfa2a443e961594e801ea07e5d6f6d3c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Tue Aug 25 19:09:18 2026 +0900
[SPARK-58990] Add a MkDocs documentation site and slim the README (#53)
Introduce a documentation site (published to GitHub Pages) so the README can
stay minimal and the guides live in one browsable place.
- MkDocs Material site (mkdocs.yml + docs/): landing page, installation,
quickstart, configuration, DataFrames, columns & functions, SQL, reading &
writing, streaming, catalog, types, Rust UDFs (WASM), a diagram-driven
Architecture page (Mermaid), compatibility, examples, troubleshooting, and
contributing. Rust-first; the Python side is documented as a drop-in for
the
pyspark-client PyPI package.
- Every Rust snippet in the guides is compile-verified against the crate API
(range arity, DataFrame-returning transformations vs Result-returning
actions,
lit()/lit_string() literals, comparison methods, agg via
expression().clone(),
reader/writer/streaming/catalog/type signatures).
- Custom "Slate & rust-ember" theme (docs/stylesheets/extra.css),
light/dark.
- .github/workflows/docs.yml: builds and publishes via `mkdocs gh-deploy`
to the
gh-pages branch that .asf.yaml already serves (ghp_branch: gh-pages),
using only
first-party actions/* per the ASF Actions policy.
- Move the internal design/CI/testing notes out of the published site to
dev/design/, and point agents at them from AGENTS.md.
- Move the generated parity ledger out of the docs tree to dev/parity/
(inventory.csv) and repoint scripts/gen_parity_ledger.py --out; it is a
dev
tracking artifact, not consumed by CI. Add a human-facing Compatibility
page.
- Slim README.md to a minimal landing that points at the docs site.
Co-authored-by: Isaac <[email protected]>
---
.github/workflows/build_python_connect.yml | 4 +-
.github/workflows/docs.yml | 62 +++++++
.github/workflows/lint.yml | 4 +-
.github/workflows/rust.yml | 4 +-
AGENTS.md | 15 ++
README.md | 180 +++++----------------
.../ACCEPTANCE.md => dev/design/acceptance.md | 6 +-
.../ARCHITECTURE.md => dev/design/architecture.md | 56 +++----
docs/design/CI.md => dev/design/ci.md | 34 ++--
.../design/official-tests.md | 8 +-
.../design/test-harness.md | 2 +-
{docs => dev}/parity/inventory.csv | 0
docs/architecture.md | 160 ++++++++++++++++++
docs/catalog.md | 117 ++++++++++++++
docs/columns-and-functions.md | 161 ++++++++++++++++++
docs/compatibility.md | 59 +++++++
docs/configuration.md | 78 +++++++++
docs/contributing.md | 51 ++++++
docs/data-sources.md | 106 ++++++++++++
docs/dataframes.md | 154 ++++++++++++++++++
docs/examples.md | 93 +++++++++++
docs/index.md | 111 +++++++++++++
docs/installation.md | 91 +++++++++++
docs/quickstart.md | 72 +++++++++
docs/sql.md | 158 ++++++++++++++++++
docs/streaming.md | 115 +++++++++++++
docs/stylesheets/extra.css | 161 ++++++++++++++++++
docs/troubleshooting.md | 66 ++++++++
docs/types.md | 127 +++++++++++++++
docs/udfs.md | 160 ++++++++++++++++++
mkdocs.yml | 105 ++++++++++++
scripts/gen_parity_ledger.py | 11 +-
32 files changed, 2327 insertions(+), 204 deletions(-)
diff --git a/.github/workflows/build_python_connect.yml
b/.github/workflows/build_python_connect.yml
index 1f5f1f0..e8fe9c8 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -26,9 +26,9 @@ name: Python Connect Parity (Spark 4.2.0)
on:
push:
- branches: [master, main]
+ branches: [master]
pull_request:
- branches: [master, main]
+ branches: [master]
workflow_dispatch:
env:
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
new file mode 100644
index 0000000..3780f71
--- /dev/null
+++ b/.github/workflows/docs.yml
@@ -0,0 +1,62 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Docs
+
+on:
+ push:
+ branches: [master]
+ paths:
+ - "docs/**"
+ - "mkdocs.yml"
+ - ".github/workflows/docs.yml"
+ workflow_dispatch:
+
+# `mkdocs gh-deploy` builds the site and pushes it to the `gh-pages` branch,
+# which ASF INFRA serves as GitHub Pages (see `.asf.yaml`: `ghp_branch:
+# gh-pages`). Only first-party `actions/*` are used, per the ASF Actions
policy.
+permissions:
+ contents: write
+
+concurrency:
+ group: docs-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ build-and-deploy:
+ name: Build and deploy the docs site
+ runs-on: ubuntu-latest
+ # Only ever publish from the canonical repository's master branch (covers
+ # both push and workflow_dispatch); forks and non-master refs never deploy.
+ if: github.repository == 'apache/spark-connect-rust' && github.ref ==
'refs/heads/master'
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+
+ - name: Set up Python
+ uses: actions/setup-python@v5
+ with:
+ python-version: "3.12"
+
+ - name: Install MkDocs Material
+ run: pip install "mkdocs-material==9.*"
+
+ - name: Build and deploy to the gh-pages branch
+ run: mkdocs gh-deploy --force --no-history
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
index 9569a6c..ea0949a 100644
--- a/.github/workflows/lint.yml
+++ b/.github/workflows/lint.yml
@@ -21,9 +21,9 @@ name: Lint
on:
push:
- branches: [master, main]
+ branches: [master]
pull_request:
- branches: [master, main]
+ branches: [master]
workflow_dispatch:
jobs:
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index bb1d7e0..70538ce 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -21,9 +21,9 @@ name: Rust CI
on:
push:
- branches: [master, main]
+ branches: [master]
pull_request:
- branches: [master, main]
+ branches: [master]
jobs:
rust-build-and-test:
diff --git a/AGENTS.md b/AGENTS.md
index c106092..47f170c 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -12,3 +12,18 @@ Security model: [SECURITY.md](./SECURITY.md)
Agents that scan this repository should consult `SECURITY.md`
for the project's threat model, in-scope / out-of-scope
declarations, and known non-findings before reporting issues.
+
+## Design and internals
+
+Maintainer- and agent-facing internals live under
+[`dev/design/`](./dev/design) (not published on the user docs site):
+
+- `dev/design/architecture.md` - system design, parity strategy, no-stub rule
+- `dev/design/ci.md` - the CI pipeline (golden-proto tests, parity gate)
+- `dev/design/official-tests.md` - the official Apache Spark Connect test
suite approach
+- `dev/design/test-harness.md` - the official-test harness
+- `dev/design/acceptance.md` - acceptance criteria / definition of done
+
+The API parity ledger is generated by `scripts/gen_parity_ledger.py` into
+`dev/parity/inventory.csv`. User-facing documentation lives in `docs/` and is
+published to GitHub Pages.
diff --git a/README.md b/README.md
index ffbfc20..480b18b 100644
--- a/README.md
+++ b/README.md
@@ -19,173 +19,73 @@
# Spark Connect Rust Client
-A Rust rewrite of the PySpark **Spark Connect** client
(`pyspark.sql.connect.*`),
-exposed through PyO3 as a drop-in `pyspark` package. It speaks the same Spark
-Connect gRPC/protobuf protocol and returns the same results, so existing
-Spark Connect code runs unchanged against the same server - with the plan
-building, transport, and Arrow decoding done in Rust.
+A fast, native **Rust** client for **Apache Spark Connect** - and a drop-in
+`pyspark` replacement. It builds `spark.connect` protobuf plans, manages the
gRPC
+channel, and decodes Arrow results in Rust, speaking the same protocol and
+returning the same results as the reference client.
-**Supported Spark version:** Apache Spark **4.2.0 and later**. The crate and
wheel
-version tracks the Spark release it targets (starting at `4.2.0`), so the
version
-number tells you which Spark it speaks.
+[](https://pypi.org/project/pyspark-client-rust/)
+
+
-## Why
+## ๐ Documentation
-The reference Spark Connect client is pure Python: it builds protobuf plans,
-manages the gRPC channel, and decodes Arrow results in Python. This project
moves
-that work into Rust while keeping a byte-for-byte compatible Python surface,
so it
-can be a **drop-in replacement** - same imports, same API, same server.
+**Full documentation lives at
+[apache.github.io/spark-connect-rust](https://apache.github.io/spark-connect-rust/)**
+- installation, quickstart, the DataFrame / Columns / SQL / Reading &
+Writing / Streaming / Catalog / Types API, [Rust UDFs via
+WebAssembly](https://apache.github.io/spark-connect-rust/udfs/), and the
+[architecture](https://apache.github.io/spark-connect-rust/architecture/).
-## Architecture
+## Install
-A Cargo workspace of four crates plus a Python skin:
-
-| Component | Path | Responsibility |
-|---|---|---|
-| `apache-spark-connect-proto` | `crates/spark-connect-proto` | gRPC/protobuf
codegen for `spark.connect.*` |
-| `apache-spark-connect-core` | `crates/spark-connect-core` | Transport:
channel, retries, reattach, artifacts, errors |
-| `apache-spark-connect` | `crates/spark-connect` | DataFrame API: session,
dataframe, column, functions, plan, group, catalog, window, readwriter,
streaming, udf, types |
-| `pyspark-rs` | `crates/pyspark-rs` | PyO3 bindings - builds the `_pyspark`
extension module |
-| Python skin | `python/pyspark` | The drop-in `pyspark` package (+ vendored
`pyspark.pandas`, `cloudpickle`, `pyspark.testing`) |
-
-See [`docs/design/ARCHITECTURE.md`](docs/design/ARCHITECTURE.md) for detail.
-
-## Installation
-
-`pyspark-client-rust` is a **complete drop-in** for the reference `pyspark`
-Spark Connect client: install it and existing Spark Connect code runs unchanged
-โ same imports, same API, same server.
+**Python** - a faster, drop-in replacement for the
+[`pyspark-client`](https://pypi.org/project/pyspark-client/) PyPI package
+(uninstall any existing `pyspark` / `pyspark-client` first):
```bash
pip install pyspark-client-rust
```
-### Build from source
+Your Spark Connect code then runs unchanged; use it exactly like
+[PySpark](https://spark.apache.org/docs/latest/api/python/).
-Requires a Rust toolchain, `protobuf-compiler`, and Python 3.9+.
+**Rust** - the native crate:
-```bash
-# Build the wheel (mixed Python/Rust layout via maturin) and install it:
-maturin build --release --out dist
-pip install dist/pyspark_client_rust-*.whl
-```
-
-For local development without maturin, build the extension and copy it into
the skin:
-
-```bash
-cargo build -p pyspark-rs --release
-cp target/release/lib_pyspark.dylib python/pyspark/_pyspark.so # .so/.dylib
per platform
-PYTHONPATH=python python -c "from pyspark.sql import SparkSession; ..."
+```toml
+[dependencies]
+apache-spark-connect = "4.2"
```
-## Usage
-
-Identical to the reference Spark Connect client:
-
-```python
-from pyspark.sql import SparkSession, functions as sf
-
-spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
-
-df = (
- spark.range(0, 1_000_000)
- .select((sf.col("id") * 2).alias("x"))
- .filter(sf.col("x") % 3 == 0)
-)
-print(df.count())
-df.groupBy((sf.col("x") % 10).alias("k")).agg(sf.sum("x"), sf.avg("x")).show()
-```
-
-### Rust usage (native)
-
-The pure-Rust API is synchronous and mirrors PySpark's surface:
+## Quickstart (Rust)
```rust
-use spark_connect::session::SparkSession;
-use spark_connect::functions as f;
+use spark_connect::{SparkSession, functions as f, lit};
-fn main() -> Result<()> {
+fn main() -> Result<(), Box<dyn std::error::Error>> {
let spark = SparkSession::builder()
.remote("sc://localhost:15002")
.get_or_create()?;
-
+
let df = spark
- .range(0, 1_000_000)?
- .select(vec![f::col("id") * 2])?
- .filter(f::col("id") % 3 == 0)?;
-
- println!("Count: {}", df.count()?);
- df.show(10)?;
-
+ .range(1_000_000)?
+ .select(vec![(f::col("id") * lit(2)).alias("x")])
+ .filter((f::col("x") % lit(3)).eq(lit(0)));
+
+ println!("count = {}", df.count()?);
+ df.show(20)?;
Ok(())
}
```
-### Rust UDFs via WebAssembly
-
-Python UDFs run as normal. In addition, a Rust function can be compiled to
-WebAssembly and shipped as a UDF: the `.wasm` bytes are embedded in a
cloudpickled
-Python shim and executed on the worker via `wasmtime`, so no Rust toolchain is
-needed server-side. See `python/pyspark_wasm_udf/` and the example crates
-`examples/wasm-udf-inline/` (single-file) and `examples/wasm-udfs/` +
-`examples/src/wasm_udf_macro.rs` (reusable crate).
-
-## Getting started with Spark Connect server
-
-Run a Spark Connect server locally for development:
-
-1. [Download Spark 4.2.0](https://spark.apache.org/downloads.html) and unzip it
-2. Set `SPARK_HOME` to the unzipped directory
-3. Start the server with:
-
-```bash
-$SPARK_HOME/sbin/start-connect-server.sh \
- --packages
"org.apache.spark:spark-connect_2.13:4.2.0,io.delta:delta-spark_2.13:4.2.0" \
- --conf "spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp" \
- --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
- --conf
"spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"
-```
-
-The server listens on `sc://localhost:15002` by default.
-
-### Sample data
-
-Sample datasets used by the examples live in `examples/datasets/` (people.csv,
-employees.json, kv1.txt, etc.). The reader/deltalake examples load them by a
path
-relative to the **server's** working directory (e.g. `./datasets/people.csv`),
so
-start the Spark Connect server from the `examples/` directory (where
`./datasets`
-resolves), or adjust the path in the example to point at the data.
-
-## Continuous integration
-
-- **Build & test** - Rust build/test via `cargo test`
-- **Golden-proto tests** - Validate the plan/expression protos this client
builds
- (`plan.rs` and friends) byte-for-byte against the reference client. This is
what
- guards plan-building correctness.
-- **Parity gate** - Runs the official Apache Spark connect test suite against
this
- client. The suite drives the standard pyspark public API but routes
plan-building
- through *upstream* pyspark (via `rust_transport_plugin.py`), so it exercises
our
- **transport and Arrow result paths** end-to-end against a real server โ not
our
- own `plan.rs` (that is the golden-proto tests' job). Together the two gates
cover
- plan-building and transport respectively.
-- **Benchmark** - End-to-end performance vs reference client
-- **Lint** - `rustfmt --check`, `clippy`, and `ruff` for Python tooling
-
-## Development
-
-```bash
-cargo fmt --all # format Rust
-cargo clippy --workspace # lint Rust
-ruff check scripts/ # lint our Python tooling
-ruff format scripts/
-```
+See the [documentation](https://apache.github.io/spark-connect-rust/) for the
full
+API, running a Spark Connect server, and more.
-`ruff` is intentionally scoped to `scripts/`; the `python/pyspark` tree is
largely
-vendored from Apache Spark and kept byte-compatible with upstream.
+## Contributing
-Remaining parity work is tracked mechanically in the parity ledger at
-`docs/parity/inventory.csv`; the official Apache Spark connect test suite is
the
-authoritative gate.
+Issues are tracked in ASF JIRA under
+[SPARK](https://issues.apache.org/jira/browse/SPARK) (GitHub Issues are
disabled).
+See the [contributing
guide](https://apache.github.io/spark-connect-rust/contributing/).
## License
diff --git a/docs/design/ACCEPTANCE.md b/dev/design/acceptance.md
similarity index 95%
rename from docs/design/ACCEPTANCE.md
rename to dev/design/acceptance.md
index 84c5e7f..f43b700 100644
--- a/docs/design/ACCEPTANCE.md
+++ b/dev/design/acceptance.md
@@ -10,10 +10,10 @@ as the pure-Python `pyspark-client` replacement, must pass
the **same** tests.
1. Builds Spark from source (SBT) and starts a Connect server via
`./sbin/start-connect-server.sh` (with protobuf + avro jars).
-2. `pip install pyspark*client-*.tar.gz` โ the pure-Python client package.
+2. `pip install pyspark*client-*.tar.gz` - the pure-Python client package.
**This is what we replace.** Then installs grpcio 1.76, protobuf 6.33.5,
pandas 2.3.3, pyarrow (implied), scipy, plotly, mlflow, torch,
scikit-learn, โฆ
-3. **Deletes `pyspark.zip` and `py4j` zip** so there is *no* JVM/Py4J path โ
the
+3. **Deletes `pyspark.zip` and `py4j` zip** so there is *no* JVM/Py4J path -
the
client is pure and talks only gRPC. Our package must likewise need no JVM.
4. Runs, with `SPARK_CONNECT_TESTING_REMOTE=sc://localhost`:
- `./python/run-tests --parallelism=1 --python-executables=python3
--modules pyspark-connect,pyspark-ml-connect`
@@ -25,7 +25,7 @@ as the pure-Python `pyspark-client` replacement, must pass
the **same** tests.
## Scope
**371 connect test files** under `python/pyspark/**/tests/connect/**` (SQL,
-client, arrow, pandas, streaming, ml, resource). Most are `test_parity_*` โ
they
+client, arrow, pandas, streaming, ml, resource). Most are `test_parity_*` -
they
run the classic PySpark test body over a Connect session, so passing them means
behavioral parity with classic PySpark, not just API shape.
diff --git a/docs/design/ARCHITECTURE.md b/dev/design/architecture.md
similarity index 72%
rename from docs/design/ARCHITECTURE.md
rename to dev/design/architecture.md
index a720bae..f77c02b 100644
--- a/docs/design/ARCHITECTURE.md
+++ b/dev/design/architecture.md
@@ -1,6 +1,6 @@
<!-- SPDX-License-Identifier: Apache-2.0 -->
-# Spark Connect Rust Client โ Architecture & Plan
+# Spark Connect Rust Client - Architecture & Plan
A Rust rewrite of the PySpark **Spark Connect** client, exposed to Python as a
drop-in replacement for the `pyspark` package. Goal: **full API parity**
@@ -9,29 +9,29 @@ hot paths (plan building, gRPC transport, Arrow decode) in
Rust.
## Source of truth
-- **Protocol**: Spark Connect protobuf `v4.1.0` โ vendored under
+- **Protocol**: Spark Connect protobuf `v4.1.0` - vendored under
`crates/spark-connect-proto/proto/`, SHA pinned in `proto/SPARK_SHA.txt`.
Developed and validated against Apache Spark 4.2.0 (wire-compatible).
- **Python reference**: `python/pyspark/sql/connect/**` (+ `sql/types.py`,
`errors/exceptions/**`). This is the code we mirror line-by-line.
- **Prior art (same author)**: `spark-connect-ruby`, `spark-connect-scala3`,
- `pyspark-client-wasm` โ proven module decomposition we follow.
+ `pyspark-client-wasm` - proven module decomposition we follow.
## Parity ledger (the "don't miss anything" mechanism)
`scripts/gen_parity_ledger.py` AST-parses the reference source and emits
-`docs/parity/inventory.csv`: one row per public class / function / method
-(2,255 items across 45 modules), each with a `status` (`todo` โ `wip` โ
-`done` โ `verified`) preserved across regenerations. Every implemented symbol
+`dev/parity/inventory.csv`: one row per public class / function / method
+(2,255 items across 45 modules), each with a `status` (`todo` -> `wip` ->
+`done` -> `verified`) preserved across regenerations. Every implemented symbol
flips its row; parity is reached when all rows are `verified`. Surface size:
| Area | Items | Notes |
|---|---|---|
-| `functions/builtin.py` | 576 | SQL functions โ mostly mechanical expr
builders |
-| `plan.py` | 343 (106 cls) | Logical plan โ proto `Relation`/`Command` |
+| `functions/builtin.py` | 576 | SQL functions - mostly mechanical expr
builders |
+| `plan.py` | 343 (106 cls) | Logical plan -> proto `Relation`/`Command` |
| `sql/types.py` | 239 (49 cls) | `DataType` hierarchy + parsing/JSON |
| `dataframe.py` | 147 | The `DataFrame` API |
-| `expressions.py` | 123 (25 cls) | `Column` expression tree โ proto
`Expression` |
+| `expressions.py` | 123 (25 cls) | `Column` expression tree -> proto
`Expression` |
| `client/core.py` | 91 | gRPC transport, config, analyze/execute |
| everything else | ~700 | catalog, readwriter, group, window, udf/udtf,
streaming, observation, errors |
@@ -54,8 +54,8 @@ python/
Rationale for the split: `spark-connect` is a usable pure-Rust client in its
own
right (like the Ruby/Scala3 clients); `pyspark-rs` + `python/pyspark` is the
-thin compatibility skin. Python-only concerns that *must* stay in Python โ
-cloudpickle UDF serialization, pandas/pyarrow interop โ live behind PyO3 calls
+thin compatibility skin. Python-only concerns that *must* stay in Python -
+cloudpickle UDF serialization, pandas/pyarrow interop - live behind PyO3 calls
into the interpreter, not reimplemented in Rust.
## Runtime & async model
@@ -74,21 +74,21 @@ we build `toPandas()` / `createDataFrame(pandas)` without
re-serializing.
## UDF strategy (full compatibility)
Python/pandas UDFs are **serialized, not executed** on the client:
-1. `cloudpickle.dumps((func, return_type))` โ called via PyO3 into CPython.
+1. `cloudpickle.dumps((func, return_type))` - called via PyO3 into CPython.
2. Wrap into `CommonInlineUserDefinedFunction` with `PythonUDF { command,
output_type, eval_type, python_ver }`. `eval_type` distinguishes
SQL_BATCHED,
SQL_ARROW_BATCHED, pandas scalar/grouped-map/cogrouped-map, mapInPandas,
- mapInArrow, UDTF, etc. โ matching `pyspark.util.PythonEvalType`.
+ mapInArrow, UDTF, etc. - matching `pyspark.util.PythonEvalType`.
3. The server's Python worker unpickles and runs it. No client-side execution.
This means byte-for-byte parity requires matching the pickle protocol version,
-the command layout, and the eval-type enum exactly โ tracked as ledger items.
+the command layout, and the eval-type enum exactly - tracked as ledger items.
-## Definition of done (ENFORCED โ 100%, no stubs)
+## Definition of done (ENFORCED - 100%, no stubs)
Per explicit user requirement: **100% coverage, zero stubs, no deferred
-followups.** A ledger item is complete only when it reaches `verified` โ backed
-by a passing golden-proto match and/or the official connect test โ never merely
+followups.** A ledger item is complete only when it reaches `verified` - backed
+by a passing golden-proto match and/or the official connect test - never merely
`done`. Nothing ships as a stub, including reattach, `UserDefinedType`,
TLS/secure
channels, artifacts, and Python/pandas UDFs (all eval types). `scripts/
audit_no_stubs.sh` fails the build on any TODO/`unimplemented!`/stub/"for now"/
@@ -102,20 +102,20 @@ as open ledger `wip` rows and must be finished before
their phases count as done
## Build order (phases)
-1. **P0 Foundation** โ
โ workspace, proto codegen (offline), parity ledger.
-2. **P1 Transport** โ `spark-connect-core`: channel builder (`sc://` URL
parse),
+1. **P0 Foundation** โ
- workspace, proto codegen (offline), parity ledger.
+2. **P1 Transport** - `spark-connect-core`: channel builder (`sc://` URL
parse),
gRPC client, config RPC, analyze/execute, retries, reattach, error mapping.
-3. **P2 Types** โ `DataType` hierarchy + proto <-> type, DDL/JSON parse.
-4. **P3 Expressions & Column** โ expression tree โ proto `Expression`;
operators.
-5. **P4 Plan** โ logical plan nodes โ `Relation`/`Command`.
-6. **P5 DataFrame + Session** โ end-to-end: `spark.range/sql/createDataFrame`,
+3. **P2 Types** - `DataType` hierarchy + proto <-> type, DDL/JSON parse.
+4. **P3 Expressions & Column** - expression tree -> proto `Expression`;
operators.
+5. **P4 Plan** - logical plan nodes -> `Relation`/`Command`.
+6. **P5 DataFrame + Session** - end-to-end: `spark.range/sql/createDataFrame`,
transformations, `collect/show/toPandas`. First runnable vertical slice.
-7. **P6 functions/builtin** โ the 576 SQL functions.
+7. **P6 functions/builtin** - the 576 SQL functions.
8. **P7 readwriter / catalog / group / window / observation**.
-9. **P8 UDF / UDTF** โ cloudpickle path, all eval types.
-10. **P9 Streaming** โ readwriter, query, listener.
-11. **P10 Python skin** โ `python/pyspark/**`, import-compat, error classes.
-12. **P11 Hardening** โ retries/reattach edge cases, artifacts, TLS/auth,
+9. **P8 UDF / UDTF** - cloudpickle path, all eval types.
+10. **P9 Streaming** - readwriter, query, listener.
+11. **P10 Python skin** - `python/pyspark/**`, import-compat, error classes.
+12. **P11 Hardening** - retries/reattach edge cases, artifacts, TLS/auth,
Databricks `x-databricks-*` headers, parity verification pass.
## Building
diff --git a/docs/design/CI.md b/dev/design/ci.md
similarity index 79%
rename from docs/design/CI.md
rename to dev/design/ci.md
index 40d3dad..a5b830d 100644
--- a/docs/design/CI.md
+++ b/dev/design/ci.md
@@ -8,8 +8,8 @@ This document describes the GitHub Actions CI pipeline for the
Spark Connect Rus
The CI consists of two main workflows:
-1. **Rust CI** (`.github/workflows/rust.yml`) โ Build and test the pure-Rust
crates
-2. **Python Connect Tests** (`.github/workflows/build_python_connect.yml`) โ
Build the PyO3 wheel and run Spark Connect integration tests
+1. **Rust CI** (`.github/workflows/rust.yml`) - Build and test the pure-Rust
crates
+2. **Python Connect Tests** (`.github/workflows/build_python_connect.yml`) -
Build the PyO3 wheel and run Spark Connect integration tests
Both workflows run on every push to main/master and on all pull requests.
@@ -21,7 +21,7 @@ This workflow validates the pure-Rust implementation:
- **Build**: Compiles `spark-connect`, `spark-connect-core`, and
`spark-connect-proto` crates (pyspark-rs is excluded from default cargo
operations because it requires maturin + Python environment)
- **Test**: Runs `cargo test` on `spark-connect` and `spark-connect-core`
-- **No-stub audit**: Runs `scripts/audit_no_stubs.sh` to enforce the "100%
coverage, zero stubs" rule (see [ARCHITECTURE.md](./ARCHITECTURE.md))
+- **No-stub audit**: Runs `scripts/audit_no_stubs.sh` to enforce the "100%
coverage, zero stubs" rule (see [Design notes](architecture.md))
### Key steps
@@ -45,8 +45,8 @@ This workflow builds the PyO3 extension and runs the official
Apache Spark Conne
The job is matrix-tested across Python versions (3.9, 3.11) to ensure broad
compatibility.
**Expected behavior**:
-- Many tests will fail initially โ this is intentional and documents the work
remaining
-- Each failure corresponds to a gap in parity (see `docs/parity/inventory.csv`)
+- Many tests will fail initially - this is intentional and documents the work
remaining
+- Each failure corresponds to a gap in parity (see `dev/parity/inventory.csv`)
- The workflow remains stable (doesn't flake) because it's deterministic
(pinned Spark version)
- As features are implemented and marked `verified` in the parity ledger, test
failures will turn green
- **Final goal**: 100% of Connect test suite passing
@@ -107,10 +107,10 @@ These logs are available in the workflow run artifacts
for debugging failures.
### Expected test modules
The workflow currently runs:
-- `test_connect_column` โ Column and expression operations
-- `test_connect_functions` โ SQL functions
-- `test_connect_dataframe` โ DataFrame operations
-- `test_connect_udf` โ Python and pandas UDFs
+- `test_connect_column` - Column and expression operations
+- `test_connect_functions` - SQL functions
+- `test_connect_dataframe` - DataFrame operations
+- `test_connect_udf` - Python and pandas UDFs
Additional modules can be added as parity improves (e.g.,
`test_connect_readwriter`, `test_connect_udf_batch`, etc.).
@@ -119,7 +119,7 @@ Additional modules can be added as parity improves (e.g.,
`test_connect_readwrit
Each failing test is a **gap in parity**. The fix process:
1. **Identify the gap**: Read the test failure and corresponding official code
-2. **Locate the ledger row**: Find the symbol in `docs/parity/inventory.csv`
+2. **Locate the ledger row**: Find the symbol in `dev/parity/inventory.csv`
3. **Implement the feature**: Code the missing functionality in the Rust crate
4. **Update ledger status**: Mark the row `done` (or `verified` if a golden
proto is included)
5. **Verify locally**: Run `scripts/run_official_connect_tests.py` with
`--target-pyspark ./python`
@@ -127,7 +127,7 @@ Each failing test is a **gap in parity**. The fix process:
## Continuous improvement
-The parity ledger (`docs/parity/inventory.csv`) is the source of truth:
+The parity ledger (`dev/parity/inventory.csv`) is the source of truth:
- **todo**: Not started
- **wip**: In progress, not testable yet
- **done**: Implemented, test(s) passing but not yet officially verified
@@ -167,9 +167,9 @@ To run the Connect test suite locally:
## References
-- [ARCHITECTURE.md](./ARCHITECTURE.md) โ System design, parity strategy,
no-stub rule
-- [ACCEPTANCE.md](./ACCEPTANCE.md) โ Acceptance criteria and definition of done
-- [pyproject.toml](../../pyproject.toml) โ Maturin build configuration
--
[scripts/run_official_connect_tests.py](../../scripts/run_official_connect_tests.py)
โ Test runner utility
-- [scripts/audit_no_stubs.sh](../../scripts/audit_no_stubs.sh) โ No-stub audit
gate
-- [Apache Spark
Workflow](https://github.com/apache/spark/blob/master/.github/workflows/build_python_connect.yml)
โ Reference implementation
+- [Design notes](architecture.md) - System design, parity strategy, no-stub
rule
+- [Acceptance criteria](acceptance.md) - Acceptance criteria and definition of
done
+-
[pyproject.toml](https://github.com/apache/spark-connect-rust/blob/master/pyproject.toml)
- Maturin build configuration
+-
[scripts/run_official_connect_tests.py](https://github.com/apache/spark-connect-rust/blob/master/scripts/run_official_connect_tests.py)
- Test runner utility
+-
[scripts/audit_no_stubs.sh](https://github.com/apache/spark-connect-rust/blob/master/scripts/audit_no_stubs.sh)
- No-stub audit gate
+- [Apache Spark
Workflow](https://github.com/apache/spark/blob/master/.github/workflows/build_python_connect.yml)
- Reference implementation
diff --git a/docs/design/OFFICIAL_TESTS.md b/dev/design/official-tests.md
similarity index 96%
rename from docs/design/OFFICIAL_TESTS.md
rename to dev/design/official-tests.md
index 9a83da7..cf7f22f 100644
--- a/docs/design/OFFICIAL_TESTS.md
+++ b/dev/design/official-tests.md
@@ -169,7 +169,7 @@ The script sets these for test execution:
The module name might not match the actual file. Check:
1. Module is listed in `tests/official/connect_test_modules.txt`
2. Module exists under `<spark>/python/pyspark/**/tests/connect/**/`
-3. Module name matches test file (e.g., `test_connect_basic` โ
`test_connect_basic.py`)
+3. Module name matches test file (e.g., `test_connect_basic` ->
`test_connect_basic.py`)
### "Connection refused" to Spark Connect server
@@ -193,21 +193,21 @@ Check environment differences:
3. **Dependencies**: All versions in `requirements-connect-test.txt` installed
4. **Connect server**: Same configuration as local testing
-## Integration with docs/design/ACCEPTANCE.md
+## Integration with acceptance.md
This gate is part of the broader **Acceptance Criteria**:
- **Baseline**: Validates official tests pass with reference client
- **Gating**: Required before shipping our package
- **Cadence**: Runs in CI on every merge
-See `docs/design/ACCEPTANCE.md` for the full acceptance criteria framework.
+See `acceptance.md` for the full acceptance criteria framework.
## Files
- `tests/official/connect_test_modules.txt` - Manifest of all 371 test modules
- `scripts/run_official_connect_tests.py` - Script to execute tests
- `requirements-connect-test.txt` - Pip dependencies
-- `docs/design/OFFICIAL_TESTS.md` - This document
+- `official-tests.md` - This document
## References
diff --git a/docs/design/OFFICIAL_TESTS_HARNESS.md b/dev/design/test-harness.md
similarity index 98%
rename from docs/design/OFFICIAL_TESTS_HARNESS.md
rename to dev/design/test-harness.md
index f418961..b4ef3ec 100644
--- a/docs/design/OFFICIAL_TESTS_HARNESS.md
+++ b/dev/design/test-harness.md
@@ -51,7 +51,7 @@ and correctness starts high because the protos are the
reference's.
**Coverage boundary (read this before claiming "the official suite validates
parity").**
Because plan-building is routed through *upstream* pyspark
(`rust_transport_plugin.py`),
-this harness exercises our **transport + Arrow result** paths โ not our own
plan builder
+this harness exercises our **transport + Arrow result** paths - not our own
plan builder
(`spark-connect::plan` and the expression/function encoders). Those never run
under the
official suite. Plan-building correctness is instead covered by the
**golden-proto tests**,
which assert the protos *our* client emits are byte-for-byte identical to the
reference's.
diff --git a/docs/parity/inventory.csv b/dev/parity/inventory.csv
similarity index 100%
rename from docs/parity/inventory.csv
rename to dev/parity/inventory.csv
diff --git a/docs/architecture.md b/docs/architecture.md
new file mode 100644
index 0000000..8ce951c
--- /dev/null
+++ b/docs/architecture.md
@@ -0,0 +1,160 @@
+# Architecture
+
+**One protocol, one Rust core, two front-ends.** This project reimplements the
+Spark Connect *client* - plan building, gRPC transport, and Arrow result
decoding
+- in Rust, and exposes it both as a native Rust library and, through PyO3, as a
+drop-in `pyspark` package.
+
+## The one big idea
+
+A Spark Connect client does three things: it **builds a protobuf plan** from
your
+DataFrame/SQL operations, **sends it over gRPC** to a Spark Connect server, and
+**decodes the Arrow result** the server streams back. The reference client does
+all three in Python.
+
+Here, those three jobs live in Rust crates. On top of that core sit two thin
+front-ends that share it: a synchronous **Rust API** shaped like PySpark, and a
+**`pyspark` Python package** (a PyO3 extension) that is byte-for-byte
compatible
+with the reference client. Same `spark.connect` protocol, same server, same
+results.
+
+## End-to-end picture
+
+```mermaid
+flowchart TD
+ subgraph front["Front-ends"]
+ PY["User PySpark code (unchanged)"]
+ SKIN["python/pyspark - drop-in package"]
+ EXT["_pyspark - PyO3 extension (crates/pyspark-rs)"]
+ RS["User Rust code"]
+ end
+
+ subgraph core["Rust core"]
+ API["apache-spark-connect - DataFrame / Column / functions / SQL"]
+ PLAN["Plan builder - spark.connect protobuf"]
+ XPORT["apache-spark-connect-core - gRPC channel, retries, reattach,
artifacts"]
+ ARROW["Arrow result decode"]
+ end
+
+ SERVER["Spark Connect server (Spark 4.2+, :15002)"]
+ OUT["Rows / Arrow - optionally DataFusion or Polars (Rust); pandas
(Python)"]
+
+ PY --> SKIN --> EXT --> API
+ RS --> API
+ API --> PLAN --> XPORT
+ XPORT -->|"ExecutePlan (gRPC/HTTP2)"| SERVER
+ SERVER -.->|"Arrow IPC batches"| XPORT
+ XPORT --> ARROW --> OUT
+ OUT -.-> PY
+ OUT -.-> RS
+```
+
+Everything below the front-ends is shared: the Python package doesn't
+reimplement anything the Rust API doesn't already do - it calls straight into
the
+same core through the `_pyspark` extension.
+
+## The crates
+
+A Cargo workspace of library crates, plus a Python skin:
+
+| Component | Path | Responsibility |
+|---|---|---|
+| `apache-spark-connect-proto` | `crates/spark-connect-proto` | Generated
`spark.connect.*` gRPC/protobuf types |
+| `apache-spark-connect-core` | `crates/spark-connect-core` | Transport:
channel, retries, reattach, artifacts, errors |
+| `apache-spark-connect` | `crates/spark-connect` | The DataFrame API:
session, dataframe, column, functions, plan, group, catalog, window,
readwriter, streaming, types |
+| `pyspark-rs` | `crates/pyspark-rs` | PyO3 bindings - builds the `_pyspark`
extension module |
+| Python skin | `python/pyspark` | The drop-in `pyspark` package (+ vendored
`pyspark.pandas`, `cloudpickle`, `pyspark.testing`) |
+
+```mermaid
+flowchart LR
+ PROTO["spark-connect-proto"] --> CORE["spark-connect-core"]
+ CORE --> SC["spark-connect (DataFrame API)"]
+ SC --> PYO3["pyspark-rs (PyO3)"]
+ PYO3 --> SKIN["python/pyspark skin"]
+ SC --> RUSTAPP["your Rust app"]
+```
+
+## A query's lifecycle
+
+When you call an action such as `count()` or `show()`, the lazily-built plan is
+serialized and sent to the server; results stream back as Arrow batches.
+
+```mermaid
+sequenceDiagram
+ participant App as Your code (Rust or pyspark)
+ participant API as spark-connect
+ participant Core as spark-connect-core
+ participant Srv as Spark Connect server
+
+ App->>API: range(...).select(...).filter(...)
+ Note over API: builds a spark.connect Plan (lazy)
+ App->>API: count() / show() (action)
+ API->>Core: execute(plan)
+ Core->>Srv: ExecutePlan(plan) [gRPC/HTTP2]
+ Srv-->>Core: ExecutePlanResponse* (Arrow IPC batches)
+ Note over Core,Srv: ReattachExecute resumes a broken stream
+ Core->>API: decoded Arrow batches
+ API-->>App: rows / count / DataFrame
+```
+
+Transformations are **lazy** - they only build up the plan. Only an *action*
+triggers `ExecutePlan`. If the response stream breaks mid-flight, the core's
+reattachable-execute iterator resumes from the last response it saw, so long
+results survive transient disconnects.
+
+## Two front-ends, one core
+
+=== "Rust (native)"
+
+ ```rust
+ use spark_connect::{SparkSession, functions as f};
+
+ let spark = SparkSession::builder()
+ .remote("sc://localhost:15002")
+ .get_or_create()?;
+ spark.range(100)?.select(vec![f::col("id")]).show(20)?;
+ ```
+
+=== "Python (drop-in)"
+
+ ```python
+ from pyspark.sql import SparkSession, functions as sf
+
+ spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
+ spark.range(0, 100).select(sf.col("id")).show(20)
+ ```
+
+Both paths build the *same* protobuf plan through the *same* Rust core and get
+the *same* Arrow results back. The Python package is a compatibility skin over
+`pyspark-rs`; it does not fork or reimplement the client logic.
+
+## Rust UDFs on the executors
+
+Beyond the client, a Rust function can be compiled to WebAssembly and run as a
+Spark UDF **on the executors** - shipped inside a standard `PythonUDF` and
+executed with `wasmtime`, so nothing Rust-specific is needed server-side.
+
+```mermaid
+flowchart LR
+ RUSTFN["#[spark_wasm_udf] fn"] -->|"build.rs -> wasm32"| WASM[".wasm
module (embedded)"]
+ WASM -->|"cloudpickled into a PythonUDF"| SRV["Spark executors"]
+ SRV -->|"wasmtime runs it per row"| RES["UDF result column"]
+```
+
+See [Rust UDFs via WebAssembly](udfs.md) for the full story.
+
+## Design principles
+
+1. **Don't fork PySpark's semantics** - mirror the public API and match the
+ reference client's protobuf output, verified by golden-proto tests.
+2. **One core, thin front-ends** - the Python package adds no client logic the
+ Rust API doesn't already have.
+3. **Correctness is gated in CI** - golden-proto tests guard plan building; the
+ official Apache Spark Connect test suite guards transport and Arrow paths
(see
+ the CI notes under `dev/design/`).
+4. **Version = Spark version** - the crate/wheel version tracks the Spark
release
+ it targets, so the number tells you which Spark it speaks.
+
+For the deeper design write-ups (design notes, CI, the official test suite, and
+acceptance criteria), see
[`dev/design/`](https://github.com/apache/spark-connect-rust/tree/master/dev/design)
+in the repository.
diff --git a/docs/catalog.md b/docs/catalog.md
new file mode 100644
index 0000000..bc67e59
--- /dev/null
+++ b/docs/catalog.md
@@ -0,0 +1,117 @@
+# Catalog
+
+The Spark catalog provides metadata and management operations for databases,
tables, columns, and functions.
+
+## Accessing the Catalog
+
+```rust
+let catalog = spark.catalog();
+```
+
+## Databases
+
+List, switch, and inspect databases:
+
+```rust
+// List all databases (returns a DataFrame)
+let dbs = spark.catalog().list_databases()?;
+dbs.show(20)?;
+
+// Current database
+let current = spark.catalog().current_database()?;
+
+// Set current database
+spark.catalog().set_current_database("my_db")?;
+
+// Check existence
+let exists = spark.catalog().database_exists("my_db")?;
+```
+
+## Tables
+
+List, inspect, and check tables:
+
+```rust
+// List tables in current database (returns a DataFrame)
+let tables = spark.catalog().list_tables()?;
+tables.show(20)?;
+
+// List tables in specific database
+let tables = spark.catalog().list_tables_in_database("my_db")?;
+
+// Check existence
+let exists = spark.catalog().table_exists("my_table")?;
+let exists = spark.catalog().table_exists_with_database("my_table",
Some("my_db"))?;
+```
+
+## Columns
+
+List columns in a table:
+
+```rust
+// List columns in current database (returns a DataFrame)
+let cols = spark.catalog().list_columns("my_table")?;
+cols.show(20)?;
+
+// List columns in specific database
+let cols = spark.catalog().list_columns_with_database("my_table",
Some("my_db"))?;
+```
+
+## Functions
+
+List and check functions:
+
+```rust
+// List all functions (returns a DataFrame)
+let funcs = spark.catalog().list_functions()?;
+funcs.show(20)?;
+
+// List functions in specific database
+let funcs = spark.catalog().list_functions_in_database("my_db")?;
+
+// Check existence
+let exists = spark.catalog().function_exists("my_func")?;
+let exists = spark.catalog().function_exists_with_database("my_func",
Some("my_db"))?;
+```
+
+## Temporary Views
+
+Register and drop temporary views (local to session):
+
+```rust
+// Register temp view
+df.create_temp_view("my_view")?;
+
+// Register and replace
+df.create_or_replace_temp_view("my_view")?;
+
+// Register global temp view (cross-session)
+df.create_global_temp_view("my_global_view")?;
+
+// Query temp view
+let result = spark.sql("SELECT * FROM my_view")?;
+
+// Drop temp view
+spark.catalog().drop_temp_view("my_view")?;
+
+// Drop global temp view
+spark.catalog().drop_global_temp_view("my_global_view")?;
+```
+
+## Cache Management
+
+Cache and uncache tables for performance:
+
+```rust
+// Cache table
+spark.catalog().cache_table("my_table")?;
+
+// Uncache table
+spark.catalog().uncache_table("my_table")?;
+
+// Clear all caches
+spark.catalog().clear_cache()?;
+```
+
+!!! note
+ Temporary views are session-local and persist until the session ends.
Global temporary views are prefixed with `global_temp.` by default.
diff --git a/docs/columns-and-functions.md b/docs/columns-and-functions.md
new file mode 100644
index 0000000..340290a
--- /dev/null
+++ b/docs/columns-and-functions.md
@@ -0,0 +1,161 @@
+# Columns and Functions
+
+Column expressions form the building blocks of transformations. Reference
columns, build expressions with operators, and apply built-in functions from
the `spark_connect::functions` module.
+
+## Column References and Literals
+
+Access DataFrame columns and create constant values.
+
+```rust
+use spark_connect::{functions as f, lit, lit_string};
+
+// Column reference
+let col_expr = f::col("column_name");
+
+// Literal value
+let lit_expr = lit(42);
+let lit_expr = lit_string("hello");
+
+// Use in expression
+let df = df.filter(f::col("age").gt(lit(18)));
+```
+
+## Column Operations
+
+### Arithmetic & Comparison
+
+```rust
+use spark_connect::{functions as f, lit, lit_string};
+
+// Arithmetic
+let expr = f::col("x") + f::col("y");
+let expr = f::col("x") - f::col("y");
+let expr = f::col("x") * lit(2);
+let expr = f::col("x") / lit(10);
+
+// Comparison
+let expr = f::col("age").gt(lit(18));
+let expr = f::col("age").ge(lit(18));
+let expr = f::col("salary").eq(lit(50000));
+let expr = f::col("status").ne(lit_string("inactive"));
+```
+
+### Boolean Logic
+
+```rust
+use spark_connect::{functions as f, lit, lit_string, lit_boolean};
+
+// AND
+let expr = (f::col("age").gt(lit(18)))
+ .and(f::col("city").eq(lit_string("NYC")));
+
+// OR
+let expr = (f::col("status").eq(lit_string("active")))
+ .or(f::col("vip").eq(lit_boolean(true)));
+
+// NOT
+let expr = f::col("archived").eq(lit_boolean(true)).not();
+```
+
+### Aliasing, Casting, Null Handling
+
+```rust
+use spark_connect::{functions as f, lit_string};
+use spark_connect::column::when; // CASE/WHEN builder (supports `.otherwise`)
+
+// Alias
+let expr = f::col("amount").alias("total");
+
+// Cast to type
+let expr = f::col("id").cast_str("string");
+let expr = f::col("price").cast_str("decimal(10, 2)");
+
+// Null checks
+let expr = f::col("email").is_null();
+let expr = f::col("email").is_not_null();
+
+// Null fallback (coalesce-style) via when/otherwise
+let expr = when(f::col("email").is_not_null(), f::col("email"))
+ .otherwise(f::col("backup_email"));
+```
+
+## Built-in Functions
+
+The `functions` module provides hundreds of operations across string, math,
date, aggregate, and conditional categories.
+
+### String Functions
+
+| Function | Usage | Purpose |
+|----------|-------|---------|
+| `upper` | `f::upper(f::col("col"))` | Convert to uppercase |
+| `lower` | `f::lower(f::col("col"))` | Convert to lowercase |
+| `concat` | `f::concat(vec![...])` | Concatenate strings |
+| `substring` | `f::substring(f::col("col"), 1, 3)` | Extract substring |
+| `length` | `f::length(f::col("col"))` | String length |
+| `trim` | `f::trim(f::col("col"))` | Remove leading/trailing spaces |
+| `reverse` | `f::reverse(f::col("col"))` | Reverse string |
+
+### Math Functions
+
+| Function | Usage | Purpose |
+|----------|-------|---------|
+| `abs` | `f::abs(f::col("x"))` | Absolute value |
+| `sqrt` | `f::sqrt(f::col("x"))` | Square root |
+| `round` | `f::round(f::col("x"), 2)` | Round to d decimals |
+| `ceil` | `f::ceil(f::col("x"))` | Ceiling |
+| `floor` | `f::floor(f::col("x"))` | Floor |
+| `sin/cos/tan` | `f::sin(...)` | Trigonometric |
+| `log/log10/exp` | `f::log(...)` | Logarithmic/exponential |
+
+### Date and Time Functions
+
+| Function | Usage | Purpose |
+|----------|-------|---------|
+| `current_date` | `f::current_date()` | Current date |
+| `current_timestamp` | `f::current_timestamp()` | Current timestamp |
+| `to_date` | `f::to_date(...)` | Parse date string |
+| `date_add` | `f::date_add(...)` | Add days |
+| `date_sub` | `f::date_sub(...)` | Subtract days |
+| `datediff` | `f::datediff(...)` | Days between dates |
+| `year/month/day` | `f::year(f::col("d"))` | Extract date part |
+
+### Aggregate Functions
+
+| Function | Usage | Purpose |
+|----------|-------|---------|
+| `count` | `f::count(f::col("id"))` | Count non-null rows |
+| `sum` | `f::sum(f::col("amt"))` | Sum values |
+| `avg` | `f::avg(f::col("price"))` | Average |
+| `min` | `f::min(f::col("val"))` | Minimum |
+| `max` | `f::max(f::col("val"))` | Maximum |
+| `stddev` | `f::stddev(f::col("x"))` | Standard deviation |
+| `collect_list` | `f::collect_list(...)` | Collect into array |
+
+### Conditional Functions
+
+```rust
+use spark_connect::{functions as f, lit, lit_string};
+use spark_connect::column::when; // CASE/WHEN builder (supports `.otherwise`)
+
+// CASE / WHEN
+let expr = when(f::col("age").lt(lit(18)), lit_string("minor"))
+ .when(f::col("age").lt(lit(65)), lit_string("adult"))
+ .otherwise(lit_string("senior"));
+
+// IF NULL fallback
+let expr = when(f::col("phone").is_not_null(), f::col("phone"))
+ .otherwise(lit_string("N/A"));
+```
+
+### Array and Collection Functions
+
+| Function | Usage | Purpose |
+|----------|-------|---------|
+| `array` | `f::array(vec![...])` | Create array |
+| `explode` | `f::explode(f::col("arr"))` | Expand array to rows |
+| `size` | `f::size(f::col("arr"))` | Array/map size |
+| `element_at` | `f::element_at(...)` | Get array element |
+| `array_contains` | `f::array_contains(...)` | Check membership |
+
+!!! tip
+ See [DataFrames](dataframes.md) for transformation examples and
[SQL](sql.md) for SQL-based expressions.
diff --git a/docs/compatibility.md b/docs/compatibility.md
new file mode 100644
index 0000000..c4d741e
--- /dev/null
+++ b/docs/compatibility.md
@@ -0,0 +1,59 @@
+# Compatibility
+
+This client aims for **full API parity** with the reference PySpark Spark
Connect
+client (`pyspark.sql.connect.*`) and the shared modules it depends on. The
goal is
+that existing Spark Connect code runs unchanged and gets byte-for-byte
identical
+results.
+
+## Supported Spark
+
+Apache Spark **4.2.0 and later**. The crate and wheel version tracks the Spark
+release it targets (starting at `4.2.0`), so the version number tells you which
+Spark it speaks.
+
+## How parity is verified
+
+Two independent gates in CI cover the two halves of a Spark Connect client -
plan
+building and the transport/result path:
+
+- **Golden-proto tests** assert that the `spark.connect` protobuf plans this
+ client builds (plans, expressions, and every SQL function) match the
reference
+ client **byte-for-byte**. This guards plan-building correctness.
+- **The official Apache Spark Connect test suite** runs the standard PySpark
tests
+ against this client, exercising the **transport and Arrow result paths**
+ end-to-end against a real server. See the official-test-suite notes
+ and the CI notes under `dev/design/`.
+
+Together the two gates cover what the client sends and what it does with what
+comes back.
+
+## The parity ledger
+
+Coverage is tracked mechanically. `scripts/gen_parity_ledger.py` AST-parses the
+reference PySpark source and emits one row per public **class**, **function**,
and
+**method** into a ledger (`dev/parity/inventory.csv`), so nothing is silently
+dropped. The current inventory covers roughly:
+
+| Kind | Count |
+|-----------|-------|
+| Methods | ~1340 |
+| Functions | ~620 |
+| Classes | ~300 |
+
+Each row carries a status:
+
+- **done** / **verified** - mirrored by the client (and, where applicable,
+ confirmed against the reference).
+- **n/a** - *architecture-satisfied*: the equivalent wire behavior is provided
by
+ the Rust core (`spark-connect` / `-core` / `-proto`), so a reference-private
or
+ generated symbol is not mirrored one-to-one.
+
+The ledger is a development tracking tool (kept under `dev/parity/`); the
official
+test suite is the authoritative gate.
+
+## What "drop-in" means
+
+From Python, `pyspark-client-rust` replaces the `pyspark-client` package: same
+`import pyspark`, same public API, same server - see
[Installation](installation.md).
+Use it exactly like
[PySpark](https://spark.apache.org/docs/latest/api/python/).
+The native [Rust API](dataframes.md) mirrors the same surface.
diff --git a/docs/configuration.md b/docs/configuration.md
new file mode 100644
index 0000000..39e4cea
--- /dev/null
+++ b/docs/configuration.md
@@ -0,0 +1,78 @@
+# Configuration and Connection
+
+Configure connections to local or remote Spark Connect servers, set session
parameters, and manage authentication.
+
+## Connection String Format
+
+Connection strings follow the format `sc://host:port/;param=value`. Parameters
are optional:
+
+| Parameter | Purpose | Example |
+| --- | --- | --- |
+| `token` | Authentication token | `sc://host:15002/;token=abc123` |
+| `user_id` | User identity | `sc://host:15002/;user_id=alice` |
+| `session_id` | Session identifier | `sc://host:15002/;session_id=sess_xyz` |
+| `use_ssl` | Enable TLS | `sc://host:15002/;use_ssl=true` |
+
+!!! note
+ TLS and authentication parameters are passed in the connection string.
Consult your infrastructure team for any required credentials or certificates.
+
+## Building a Session
+
+```rust
+use spark_connect::SparkSessionBuilder;
+
+let spark = SparkSessionBuilder::default()
+ .remote("sc://localhost:15002")
+ .get_or_create()?;
+
+// Session configuration is applied at runtime via `spark.conf()`:
+spark.conf().set("spark.sql.shuffle.partitions", "20")?;
+```
+
+## Runtime Configuration
+
+Access and modify session configuration at runtime:
+
+```rust
+// Get a config value
+let partitions = spark.conf().get("spark.sql.shuffle.partitions")?;
+
+// Set a config value
+spark.conf().set("spark.sql.adaptive.enabled", "true")?;
+```
+
+## Starting a Local Spark Connect Server
+
+To run a Spark Connect server locally for development:
+
+```bash
+# Set SPARK_HOME to your Spark installation
+export SPARK_HOME=/path/to/spark
+
+# Start the server on sc://localhost:15002
+$SPARK_HOME/sbin/start-connect-server.sh \
+ --packages "org.apache.spark:spark-connect_2.13:4.2.0"
+```
+
+The server listens on port `15002` by default. To use a different port, add
`--conf spark.connect.grpc.binding.port=<port>`.
+
+!!! tip
+ The Spark Connect server requires a JVM and Apache Spark 4.2.0 or later.
Stop it with `$SPARK_HOME/sbin/stop-connect-server.sh`.
+
+## Remote Connections
+
+To connect to a remote Spark Connect server:
+
+```rust
+let spark = SparkSessionBuilder::default()
+ .remote("sc://spark.example.com:15002/;token=YOUR_TOKEN")
+ .get_or_create()?;
+```
+
+Replace `spark.example.com`, the port, and authentication parameters as
needed. If TLS is required:
+
+```rust
+let spark = SparkSessionBuilder::default()
+ .remote("sc://spark.example.com:15002/;use_ssl=true;token=YOUR_TOKEN")
+ .get_or_create()?;
+```
diff --git a/docs/contributing.md b/docs/contributing.md
new file mode 100644
index 0000000..9b5851d
--- /dev/null
+++ b/docs/contributing.md
@@ -0,0 +1,51 @@
+# Contributing
+
+Contributions are welcome. This is an Apache Software Foundation project, so a
few
+conventions apply.
+
+## Issues and discussion
+
+Bugs and feature requests are tracked in **ASF JIRA** under the
+[SPARK](https://issues.apache.org/jira/browse/SPARK) project - GitHub Issues
are
+disabled. Reference the JIRA key in your pull request title, e.g.
+`[SPARK-XXXXX] Short summary`.
+
+## Building and testing
+
+Requires a Rust toolchain and `protobuf-compiler`.
+
+```bash
+cargo build # build the library crates
+cargo test # unit + golden-proto tests
+cargo build -p examples # the Rust-native examples
+```
+
+Builder correctness is covered by **golden-proto tests**: plans, expressions,
and
+all SQL functions are asserted byte-for-byte against captured reference protos.
+Transport and Arrow paths are exercised by the official Apache Spark Connect
test
+suite in CI - see the CI notes under `dev/design/`.
+
+The `wasm-udf` example crates need the `wasm32` target and are built by
manifest
+path (they're excluded from the workspace):
+
+```bash
+rustup target add wasm32-unknown-unknown
+cargo build --manifest-path examples/wasm-udf-inline/Cargo.toml
+```
+
+## Style
+
+```bash
+cargo fmt --all # format
+cargo clippy --workspace # lint
+```
+
+Match the surrounding code, keep changes focused, and add or update golden
tests
+when you change plan or expression building.
+
+## Pull requests
+
+Open PRs against
+[`apache/spark-connect-rust`](https://github.com/apache/spark-connect-rust).
+Keep the PR description tied to its JIRA, and make sure `cargo test`, `rustfmt
+--check`, and `clippy` pass.
diff --git a/docs/data-sources.md b/docs/data-sources.md
new file mode 100644
index 0000000..450bcc6
--- /dev/null
+++ b/docs/data-sources.md
@@ -0,0 +1,106 @@
+# Reading and Writing
+
+The `spark_connect` crate lets you read data from and write data to multiple
file formats and data sources, with the server resolving all paths.
+
+## Reading Data
+
+### Basic Read
+
+Use `spark.read()` to access the DataFrameReader, specifying a format and path.
+
+```rust
+// Generic format API
+let df = spark.read()
+ .format("parquet")
+ .load(Some("/path/to/file.parquet"));
+
+// Shortcuts
+let df = spark.read().parquet("/path/to/file.parquet");
+let df = spark.read().csv("/path/to/file.csv");
+let df = spark.read().json("/path/to/file.json");
+```
+
+### Common Read Options
+
+Set read behavior with `.option(key, value)`:
+
+```rust
+let df = spark.read()
+ .format("csv")
+ .option("header", "true")
+ .option("delimiter", ",")
+ .option("inferSchema", "true")
+ .load(Some("/path/to/data.csv"));
+```
+
+| Option | Format | Description |
+|--------|--------|-------------|
+| `header` | CSV | First row contains column names |
+| `delimiter` | CSV | Field separator (default: comma) |
+| `inferSchema` | CSV, JSON | Auto-detect column types |
+| `recursiveFileLookup` | Parquet, CSV, JSON | Recursively search directories |
+| `mode` | All | `PERMISSIVE`, `DROPMALFORMED`, `FAILFAST` for error handling |
+
+## Supported Formats
+
+- **Parquet**: Columnar format; default for DataFrames.
+- **CSV**: Delimiter-separated values; requires `header` option.
+- **JSON**: Line-delimited JSON objects.
+- **ORC**: Optimized columnar format.
+- **Delta**: Unified format with ACID transactions (Delta Lake).
+
+## Writing Data
+
+### Basic Write
+
+Use `df.write()` to access the DataFrameWriter, specifying a format and save
location:
+
+```rust
+df.write()
+ .format("parquet")
+ .mode("overwrite")
+ .save(Some("/path/to/output"))?;
+
+// Shortcuts
+df.write().parquet("/path/to/output.parquet")?;
+df.write().csv("/path/to/output.csv")?;
+df.write().json("/path/to/output.json")?;
+```
+
+### Save Modes
+
+Control behavior when the target path already exists:
+
+| Mode | Behavior |
+|------|----------|
+| `overwrite` | Replace existing data |
+| `append` | Add data to existing directory |
+| `ignore` | Do nothing if path exists |
+| `error` | Raise an error (default) |
+
+### Partitioning
+
+Organize output by one or more columns:
+
+```rust
+df.write()
+ .format("parquet")
+ .partition_by(vec!["year".to_string(), "month".to_string()])
+ .mode("overwrite")
+ .save(Some("/path/to/output"))?;
+```
+
+### Catalog Tables
+
+Save as a managed or external table:
+
+```rust
+df.write().mode("overwrite").save_as_table("my_table")?;
+df.write()
+ .mode("overwrite")
+ .option("path", "/custom/path")
+ .save_as_table("my_table")?;
+```
+
+!!! note
+ All file paths are resolved by the Spark Connect server, not the client.
Ensure the server has access to the target location.
diff --git a/docs/dataframes.md b/docs/dataframes.md
new file mode 100644
index 0000000..710d75b
--- /dev/null
+++ b/docs/dataframes.md
@@ -0,0 +1,154 @@
+# DataFrames
+
+DataFrames are the core abstraction in Spark Connect-immutable, distributed
tables. Create them from ranges, SQL, external data, or in-memory collections,
then transform and aggregate them using a fluent API.
+
+## Creating DataFrames
+
+```rust
+use spark_connect::SparkSession;
+
+let spark = SparkSession::builder()
+ .remote("sc://localhost:15002")
+ .get_or_create()?;
+
+// From a range
+let df = spark.range(100)?;
+
+// From SQL
+let df = spark.sql("SELECT 1 as id")?;
+
+// From external source (planned)
+// let df = spark.read().parquet("path/to/file.parquet")?;
+
+// From in-memory data (planned)
+// let df = spark.create_data_frame(...)?;
+```
+
+!!! note
+ Remote session requires a Spark Connect server running at
`sc://localhost:15002`.
+
+## Transformations
+
+Transform DataFrames with chainable operations like `select`, `filter`, and
`with_column`.
+
+```rust
+use spark_connect::{functions as f, lit};
+
+// Select columns
+let df = df.select(vec![f::col("id"), f::col("name")]);
+
+// Filter rows
+let df = df.filter(f::col("id").gt(lit(10)));
+
+// Add/modify column
+let df = df.with_column("double_id", f::col("id") * lit(2));
+
+// Rename column
+let df = df.with_column_renamed("id", "row_id");
+
+// Drop columns
+let df = df.drop(vec!["temp_col"]);
+
+// Distinct rows
+let df = df.distinct();
+
+// Sort
+let df = df.order_by(vec![f::col("id").expression().clone()]);
+
+// Limit
+let df = df.limit(5);
+```
+
+## Grouping and Aggregation
+
+Group rows and compute aggregates like count, sum, and average.
+
+```rust
+use spark_connect::{functions as f, lit};
+
+// Group by one column
+let agg_df = df.group_by(vec![f::col("category")])
+ .agg(vec![f::count(lit(1)).expression().clone()]);
+
+// Group by multiple columns with multiple aggregates
+let agg_df = df.group_by(vec![f::col("category"), f::col("year")])
+ .agg(vec![
+ f::sum(f::col("amount")).expression().clone(),
+ f::avg(f::col("value")).expression().clone(),
+ f::min(f::col("price")).alias("min_price").expression().clone(),
+ ]);
+```
+
+## Joins
+
+Combine DataFrames on shared keys.
+
+```rust
+use spark_connect::{functions as f, plan::JoinType};
+
+// Inner join (default)
+let joined = df1.join(
+ &df2,
+ Some(f::col("df1.id").eq(f::col("df2.id"))),
+ JoinType::Inner,
+);
+
+// Left outer join
+let joined = df1.join(&df2, Some(f::col("id").eq(f::col("id"))),
JoinType::LeftOuter);
+
+// Cross join
+let crossed = df1.join(&df2, None, JoinType::Cross);
+```
+
+## Set Operations
+
+Combine or deduplicate across DataFrames.
+
+```rust
+// Union (stacks rows, allows duplicates)
+let combined = df1.union(&df2);
+
+// Union by name (aligns columns)
+let combined = df1.union_by_name(&df2);
+
+// Except (rows in df1 not in df2)
+let diff = df1.except_all(&df2);
+```
+
+## Actions
+
+Execute and retrieve results.
+
+```rust
+// Display first n rows
+df.show(10)?;
+
+// Collect all rows to driver
+let rows = df.collect()?;
+
+// Count rows
+let count = df.count()?;
+
+// Get first n rows
+let first_rows = df.take(5)?;
+
+// Get first row
+let first = df.first()?;
+```
+
+## Example Pipeline
+
+```rust
+use spark_connect::{functions as f, lit};
+
+let result = spark.range(101)?
+ .with_column("squared", f::col("id") * f::col("id"))
+ .filter(f::col("squared").gt(lit(100)))
+ .select(vec![f::col("id"), f::col("squared")])
+ .order_by(vec![f::col("id").expression().clone()])
+ .limit(10);
+result.show(10)?;
+```
+
+!!! tip
+ See [Columns and Functions](columns-and-functions.md) for expression
building, [SQL](sql.md) for SQL queries, and [Reading and
Writing](data-sources.md) for I/O.
diff --git a/docs/examples.md b/docs/examples.md
new file mode 100644
index 0000000..9d53d7b
--- /dev/null
+++ b/docs/examples.md
@@ -0,0 +1,93 @@
+# Examples
+
+Curated recipes for common Spark Connect tasks. See the repository's
`examples/` directory for more runnable Rust examples.
+
+## Read and Aggregate CSV
+
+Read a CSV file and compute summary statistics:
+
+```rust
+use spark_connect::functions as f;
+
+let df = spark.read()
+ .format("csv")
+ .option("header", "true")
+ .option("inferSchema", "true")
+ .load(Some("data.csv"));
+
+df
+ .group_by(vec![f::col("category")])
+ .agg(vec![
+ f::sum(f::col("amount")).expression().clone(),
+ f::avg(f::col("price")).expression().clone(),
+ ])
+ .show(20)?;
+```
+
+## Word Count
+
+Classic MapReduce-style word count over a text column:
+
+```rust
+use spark_connect::{functions as f, lit_string};
+
+let df = spark.sql(
+ "SELECT * FROM VALUES ('hello world'), ('hello spark') AS t(line)"
+)?;
+
+df
+ .select(vec![f::explode(
+ f::split(f::col("line"), lit_string(" "))
+ ).alias("word")])
+ .group_by(vec![f::col("word")])
+ .agg(vec![f::count(f::col("word")).expression().clone()])
+ .show(20)?;
+```
+
+## Join Two DataFrames
+
+Join two DataFrames on a common key:
+
+```rust
+use spark_connect::{functions as f, lit, plan::JoinType};
+
+let users = spark.range(2)?
+ .with_column("name", f::col("id"));
+let orders = spark.range(3)?
+ .with_column("user_id", f::col("id"))
+ .with_column("amount", lit(100));
+
+users
+ .join(&orders, Some(f::col("id").eq(f::col("user_id"))), JoinType::Inner)
+ .select(vec![f::col("name"), f::col("amount")])
+ .show(20)?;
+```
+
+## Write Parquet
+
+Save a DataFrame to Parquet format:
+
+```rust
+let df = spark.range(1000)?;
+df.write()
+ .mode("overwrite")
+ .format("parquet")
+ .save(Some("/tmp/output"))?;
+```
+
+## SQL Query
+
+Run a SQL query directly:
+
+```rust
+let df = spark.range(100)?;
+df.create_or_replace_temp_view("numbers")?;
+
+spark
+ .sql("SELECT * FROM numbers WHERE id > 50")?
+ .show(20)?;
+```
+
+## More Examples
+
+Explore the repository's `examples/` directory for additional runnable
examples, including streaming, complex transformations, and integration with
external data sources.
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 0000000..2199dce
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,111 @@
+# Spark Connect Rust Client
+
+<div class="sc-hero" markdown>
+
+A fast, native **Rust** client for **Apache Spark Connect** - and a drop-in
+`pyspark` replacement.
+
+<p class="sc-tagline" markdown>
+It builds <code>spark.connect</code> protobuf plans, manages the gRPC channel,
and
+decodes Arrow results in Rust - speaking the same protocol and returning the
same
+results as the reference client.
+</p>
+
+<div class="sc-badges" markdown>
+
+
+
+</div>
+
+[Get started](installation.md){ .md-button .md-button--primary }
+[Quickstart](quickstart.md){ .md-button }
+[View on GitHub](https://github.com/apache/spark-connect-rust){ .md-button }
+
+</div>
+
+!!! tip "Coming from Python?"
+ `pyspark-client-rust` is a **faster, drop-in replacement for the
+ [`pyspark-client`](https://pypi.org/project/pyspark-client/) package** on
PyPI.
+ Uninstall any existing `pyspark` / `pyspark-client`, `pip install
+ pyspark-client-rust`, and your Spark Connect code runs **unchanged** - same
+ imports, same API, same server (see [Installation](installation.md)). Use
it
+ exactly like [PySpark](https://spark.apache.org/docs/latest/api/python/);
the
+ rest of these docs cover the **native Rust API**.
+
+## Why
+
+The reference Spark Connect client is pure Python - it builds protobuf plans,
+manages the gRPC channel, and decodes Arrow results in Python. This project
moves
+that work into **Rust**: a synchronous, PySpark-shaped DataFrame API you can
use
+directly from Rust, and - through PyO3 - a byte-for-byte compatible `pyspark`
+package for the Python world.
+
+<div class="sc-grid" markdown>
+
+<div class="sc-card" markdown>
+### Native Rust API
+A synchronous `spark_connect` crate that mirrors PySpark's DataFrame, Column,
+functions, SQL, streaming, and catalog surface.
+</div>
+
+<div class="sc-card" markdown>
+### Same protocol & results
+Speaks the same `spark.connect` gRPC/protobuf protocol against the same server,
+validated against the official Apache Spark test suite.
+</div>
+
+<div class="sc-card" markdown>
+### Arrow-native results
+Results decode through Apache Arrow; optionally convert to
+[DataFusion](https://datafusion.apache.org/) or [Polars](https://pola.rs/).
+</div>
+
+<div class="sc-card" markdown>
+### Rust UDFs via WebAssembly
+Compile a Rust function to WASM and run it as a Spark UDF on the executors - no
+server-side plugin. See [Rust UDFs](udfs.md).
+</div>
+
+<div class="sc-card" markdown>
+### Drop-in `pyspark`
+A faster replacement for the `pyspark-client` PyPI package - existing Python
code
+runs unchanged.
+</div>
+
+<div class="sc-card" markdown>
+### Spark 4.2.0+
+The crate and wheel version tracks the Spark release it targets, so the version
+number tells you which Spark it speaks.
+</div>
+
+</div>
+
+## Hello, Spark Connect (in Rust)
+
+```rust
+use spark_connect::{SparkSession, functions as f, lit};
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+ let spark = SparkSession::builder()
+ .remote("sc://localhost:15002")
+ .get_or_create()?;
+
+ let df = spark
+ .range(1_000_000)?
+ .select(vec![(f::col("id") * lit(2)).alias("x")])
+ .filter((f::col("x") % lit(3)).eq(lit(0)));
+
+ println!("count = {}", df.count()?);
+ df.show(20)?;
+ Ok(())
+}
+```
+
+## Where to next
+
+- **[Installation](installation.md)** - add the Rust crate, or `pip install`
the drop-in.
+- **[Quickstart](quickstart.md)** - connect to a server and run your first
query.
+- **[DataFrames](dataframes.md)** ยท **[Columns and
Functions](columns-and-functions.md)** ยท **[SQL](sql.md)** - the core API.
+- **[Reading and Writing](data-sources.md)** ยท **[Structured
Streaming](streaming.md)** ยท **[Catalog](catalog.md)**.
+- **[Rust UDFs via WebAssembly](udfs.md)** - run Rust functions on the
executors.
+- **[Architecture](architecture.md)** - how the client is put together.
diff --git a/docs/installation.md b/docs/installation.md
new file mode 100644
index 0000000..a6f16af
--- /dev/null
+++ b/docs/installation.md
@@ -0,0 +1,91 @@
+# Installation
+
+Add the Spark Connect Rust client to your project. The native Rust crate is
the primary library; Python support is a drop-in replacement for PySpark.
+
+## Rust (Native Crate)
+
+Add the crate to your `Cargo.toml`:
+
+```toml
+[dependencies]
+apache-spark-connect = "4.2"
+```
+
+### Optional Features
+
+Enable optional features for additional functionality:
+
+```toml
+[dependencies]
+apache-spark-connect = { version = "4.2", features = ["datafusion", "polars"] }
+```
+
+**Requirements:**
+- Rust 1.70 or later
+- `protobuf-compiler` (e.g., `brew install protobuf` on macOS, `apt-get
install protobuf-compiler` on Linux)
+- A running Spark Connect server (see [Configuration and
Connection](configuration.md))
+
+### Build from Source
+
+To build the native library locally:
+
+```bash
+git clone https://github.com/apache/spark-connect-rust
+cd spark-connect-rust
+cargo build --release
+```
+
+**Build requirements:**
+- Rust toolchain (`rustup` / `cargo`)
+- `protobuf-compiler`
+
+## Python (Drop-in Replacement)
+
+`pyspark-client-rust` is a faster, drop-in replacement for the official
`pyspark` Spark Connect client on PyPI. It uses the same `pyspark` import path
and public API, but executes plan building, transport, and Arrow decoding in
Rust for better performance.
+
+### Installation
+
+If you have an existing `pyspark` or `pyspark-client` installation, uninstall
it first to avoid import clashes:
+
+```bash
+pip uninstall pyspark pyspark-client -y
+pip install pyspark-client-rust
+```
+
+**Requirements:**
+- Python 3.9 or later
+- A running Spark Connect server
+
+### Usage and API Reference
+
+Use it exactly like PySpark. For the complete Python API reference, see the
[official PySpark
documentation](https://spark.apache.org/docs/latest/api/python/).
+
+```python
+from pyspark.sql import SparkSession
+
+spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
+```
+
+### Build from Source (Python)
+
+To build and install the Python wheel:
+
+```bash
+pip install maturin
+maturin build --release --out dist
+pip install dist/pyspark_client_rust-*.whl
+```
+
+**Build requirements:**
+- Python 3.9 or later
+- Rust toolchain (`rustup` / `cargo`)
+- `protobuf-compiler`
+
+## Version and Compatibility
+
+The crate and package versions track Apache Spark: version `4.2.x` supports
Spark 4.2.0 and later. The wire protocol is identical to the reference client,
so existing Spark Connect code works unchanged.
+
+## Next Steps
+
+- [Quickstart](quickstart.md) - write your first Rust query
+- [Configuration and Connection](configuration.md) - connect to a remote or
local Spark Connect server
diff --git a/docs/quickstart.md b/docs/quickstart.md
new file mode 100644
index 0000000..ab057b4
--- /dev/null
+++ b/docs/quickstart.md
@@ -0,0 +1,72 @@
+# Quickstart
+
+Get up and running with Spark Connect in minutes. This guide walks you through
your first Rust query.
+
+## Prerequisites
+
+Before you start, ensure you have a running Spark Connect server. See
[Configuration and Connection](configuration.md) for how to start one locally:
+
+```bash
+$SPARK_HOME/sbin/start-connect-server.sh --packages
"org.apache.spark:spark-connect_2.13:4.2.0"
+```
+
+The server listens on `sc://localhost:15002` by default.
+
+## Your First Query
+
+Connect to the server and run a simple query:
+
+```rust
+use spark_connect::SparkSession;
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+ let spark = SparkSession::builder()
+ .remote("sc://localhost:15002")
+ .get_or_create()?;
+
+ let df = spark.range(10)?;
+ df.show(20)?;
+ println!("Count: {}", df.count()?);
+
+ Ok(())
+}
+```
+
+## Filtering and Selection
+
+Add a filter and select specific columns:
+
+```rust
+use spark_connect::functions as f;
+
+let df = spark.range(100)?;
+df
+ .filter(f::col("id").gt(lit(50)))
+ .select(vec![f::col("id")])
+ .show(20)?;
+```
+
+## Aggregation and Grouping
+
+Compute aggregations over grouped data:
+
+```rust
+use spark_connect::{functions as f, lit};
+
+let df = spark.range(20)?;
+df
+ .with_column("category", f::col("id") % lit(3))
+ .group_by(vec![f::col("category")])
+ .agg(vec![
+ f::sum(f::col("id")).alias("total").expression().clone(),
+ f::avg(f::col("id")).alias("average").expression().clone(),
+ ])
+ .show(20)?;
+```
+
+## Next Steps
+
+- [DataFrames](dataframes.md) - column operations, joins, window functions
+- [SQL](sql.md) - run SQL queries directly
+- [Reading and Writing](data-sources.md) - work with CSV, Parquet, Delta, and
other formats
+- [Configuration and Connection](configuration.md) - connect to remote
servers, set session config
diff --git a/docs/sql.md b/docs/sql.md
new file mode 100644
index 0000000..e7ac787
--- /dev/null
+++ b/docs/sql.md
@@ -0,0 +1,158 @@
+# SQL
+
+Execute SQL directly against DataFrames and data sources. Mix SQL queries with
the DataFrame API for maximum flexibility.
+
+## Running SQL Queries
+
+Use `spark.sql()` to execute SQL and retrieve results as a DataFrame.
+
+```rust
+use spark_connect::SparkSession;
+
+let spark = SparkSession::builder()
+ .remote("sc://localhost:15002")
+ .get_or_create()?;
+
+// Simple query
+let df = spark.sql("SELECT 1 as id, 'hello' as msg")?;
+df.show(10)?;
+
+// Aggregate query
+let df = spark.sql(
+ r#"SELECT category, COUNT(*) as cnt, AVG(price) as avg_price
+ FROM products
+ GROUP BY category
+ ORDER BY cnt DESC"#
+)?;
+df.show(10)?;
+```
+
+## Registering Temporary Views
+
+Make DataFrames queryable via SQL by creating temporary views.
+
+```rust
+use spark_connect::SparkSession;
+
+let spark = SparkSession::builder()
+ .remote("sc://localhost:15002")
+ .get_or_create()?;
+
+// Create from range
+let df = spark.range(4)?;
+
+// Register as temp view
+df.create_or_replace_temp_view("users")?;
+
+// Query it
+let result = spark.sql("SELECT * FROM users WHERE id > 1")?;
+result.show(10)?;
+
+// Replace view
+let df_updated = spark.sql("SELECT id FROM users")?;
+df_updated.create_or_replace_temp_view("users")?;
+```
+
+!!! note
+ Temporary views are scoped to the session and are dropped when the session
ends.
+
+## Dynamic values
+
+`SparkSession::sql` takes a single query string; there is no server-side
+parameter-binding API. For **caller-controlled values, prefer the typed
+DataFrame API** rather than splicing them into SQL text -- values are passed as
+literals, so untrusted input cannot break the query or inject SQL:
+
+```rust
+use spark_connect::{SparkSession, functions as f, lit, lit_string};
+
+let spark = SparkSession::builder()
+ .remote("sc://localhost:15002")
+ .get_or_create()?;
+
+let threshold = 50;
+let category = "electronics";
+
+// Injection-safe: the values are literals, not part of the SQL string.
+let df = spark.sql("SELECT * FROM products")?
+ .filter(f::col("price").gt(lit(threshold)))
+ .filter(f::col("category").eq(lit_string(category)));
+```
+
+!!! warning
+ Building a query by interpolation (`format!`) is **not** injection-safe and
+ must only be used with trusted, validated input:
+
+ ```rust
+ // Trusted input only - NOT safe for user-controlled values.
+ let threshold = 50;
+ let df = spark.sql(&format!("SELECT * FROM products WHERE price > {}",
threshold))?;
+ ```
+
+## Mixing SQL and DataFrames
+
+Alternate between SQL queries and DataFrame API transformations.
+
+```rust
+use spark_connect::{SparkSession, functions as f, lit};
+
+// Start with SQL
+let raw = spark.sql("SELECT * FROM raw_data")?;
+
+// Transform with DataFrame API
+let cleaned = raw
+ .filter(f::col("value").gt(lit(0)))
+ .select(vec![f::col("id"), f::col("value")])
+ .with_column("scaled", f::col("value") * lit(2));
+
+// Register result for SQL
+cleaned.create_or_replace_temp_view("cleaned_data")?;
+
+// Query it with SQL
+let aggregated = spark.sql(
+ r#"SELECT id, COUNT(*) as cnt, AVG(scaled) as avg_scaled
+ FROM cleaned_data
+ GROUP BY id"#
+)?;
+aggregated.show(10)?;
+```
+
+## SQL and DataFrame Interop Example
+
+```rust
+use spark_connect::{SparkSession, functions as f, lit, lit_string};
+use spark_connect::column::when; // CASE/WHEN builder (supports `.otherwise`)
+
+// Create test data (from range with derived columns)
+let df = spark.range(5)?
+ .with_column(
+ "category",
+ when(f::col("id").le(lit(2)), lit_string("Electronics"))
+ .otherwise(lit_string("Books"))
+ )
+ .with_column(
+ "price",
+ when(f::col("id").eq(lit(1)), lit(299))
+ .when(f::col("id").eq(lit(2)), lit(150))
+ .when(f::col("id").eq(lit(3)), lit(25))
+ .otherwise(lit(30))
+ );
+df.create_or_replace_temp_view("products")?;
+
+// SQL: aggregate by category
+let by_cat = spark.sql(
+ r#"SELECT category, COUNT(*) as count, AVG(price) as avg_price
+ FROM products
+ GROUP BY category"#
+)?;
+by_cat.create_or_replace_temp_view("category_stats")?;
+
+// DataFrame API: filter and order
+let result = spark.sql("SELECT * FROM category_stats")?
+ .filter(f::col("avg_price").gt(lit(50)))
+ .order_by(vec![f::col("count").desc().expression().clone()]);
+result.show(10)?;
+```
+
+!!! tip
+ Temporary views make it easy to break complex transformations into
readable steps. See [DataFrames](dataframes.md) for more on the API, and
[Catalog](catalog.md) for managing tables and schemas.
diff --git a/docs/streaming.md b/docs/streaming.md
new file mode 100644
index 0000000..38f0dba
--- /dev/null
+++ b/docs/streaming.md
@@ -0,0 +1,115 @@
+# Structured Streaming
+
+The `spark_connect` crate supports Structured Streaming for continuous data
processing from streaming sources and sinks.
+
+## Reading Streams
+
+Use `spark.read_stream()` to create a streaming DataFrame:
+
+```rust
+// Rate source (generates rows at specified rate; useful for testing)
+let df = spark.read_stream()
+ .format("rate")
+ .option("rowsPerSecond", "10")
+ .load(None);
+
+// File source (monitors directory for new files)
+let df = spark.read_stream()
+ .format("csv")
+ .option("header", "true")
+ .schema("value string")
+ .load(Some("/path/to/stream/input"));
+```
+
+## Stream Transformations
+
+Apply SQL operations on streaming DataFrames as you would on static ones:
+
+```rust
+use spark_connect::{col, lit};
+
+let result = df
+ .filter(col("value").gt(lit(10)))
+ .select(vec![col("timestamp"), col("value")])
+ .write_stream()
+ .format("console")
+ .start("")?;
+```
+
+## Writing Streams
+
+Use `df.write_stream()` to write a streaming DataFrame to a sink:
+
+```rust
+use spark_connect::streaming::Trigger;
+
+// Console sink
+let query = df.write_stream()
+ .format("console")
+ .output_mode("append")
+ .trigger(Trigger::ProcessingTime("5 seconds".to_string()))
+ .option("checkpointLocation", "/tmp/checkpoint")
+ .start("")?;
+
+// Memory sink (debug only; stores in table)
+let query = df.write_stream()
+ .format("memory")
+ .query_name("my_stream")
+ .output_mode("append")
+ .start("")?;
+
+// File sink (Parquet)
+let query = df.write_stream()
+ .format("parquet")
+ .output_mode("append")
+ .option("checkpointLocation", "/tmp/checkpoint")
+ .start("/path/to/output")?;
+```
+
+### Output Modes
+
+| Mode | Behavior |
+|------|----------|
+| `append` | Add new rows to sink only |
+| `complete` | Rewrite entire result set (aggregations only) |
+| `update` | Update changed rows only (aggregations only) |
+
+### Triggers
+
+Control how often results are written:
+
+- `trigger("processingTime=5 seconds")` - Write every 5 seconds.
+- `trigger("once=true")` - Process one micro-batch then stop.
+- `trigger("continuous=1 second")` - Continuous mode (lower latency).
+
+## Managing Queries
+
+### Awaiting Termination
+
+Block until a query stops (either by error or `stop()`):
+
+```rust
+let query = df.write_stream().format("console").start("")?;
+query.await_termination(None)?;
+```
+
+### Stopping a Query
+
+Gracefully stop a streaming query:
+
+```rust
+query.stop()?;
+```
+
+### Active Queries
+
+List all active streaming queries in the session:
+
+```rust
+for q in spark.streams().active()? {
+ println!("Query {}: {:?}", q.id(), q.is_active()?);
+}
+```
+
+!!! warning
+ Checkpoints are mandatory for fault tolerance. Always set
`checkpointLocation` for production queries.
diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css
new file mode 100644
index 0000000..5807235
--- /dev/null
+++ b/docs/stylesheets/extra.css
@@ -0,0 +1,161 @@
+/* Slate & Rust-ember theme for the Spark Connect Rust Client docs.
+ A near-black slate chrome with a warm rust-ember accent - a nod to Rust's
+ identity, kept understated. Tuned for both the light (default) and the slate
+ (dark) schemes. Colors live here as CSS custom properties; mkdocs.yml sets
+ `primary: custom` / `accent: custom` so Material picks these up. */
+
+:root {
+ /* Slate chrome + rust-ember accent (light scheme). */
+ --md-primary-fg-color: #1f2733;
+ --md-primary-fg-color--light: #33404f;
+ --md-primary-fg-color--dark: #141a23;
+ --md-primary-bg-color: #ffffff;
+ --md-primary-bg-color--light: #ffffffcc;
+
+ --md-accent-fg-color: #c2410c;
+ --md-accent-fg-color--transparent: rgba(194, 65, 12, 0.10);
+
+ /* Rust brand ember for the logo/wordmark flourishes. */
+ --sc-ember: #c2410c;
+ --sc-ember-bright: #ea580c;
+}
+
+/* Light scheme: links track the ember accent for a cohesive feel. */
+[data-md-color-scheme="default"] {
+ --md-typeset-a-color: #b23a0b;
+}
+
+/* Dark (slate) scheme: deepen the chrome and lift the ember to a warm amber
+ so it reads on the dark background. */
+[data-md-color-scheme="slate"] {
+ --md-primary-fg-color: #161c26;
+ --md-primary-fg-color--light: #2a3543;
+ --md-primary-fg-color--dark: #0c1017;
+
+ --md-accent-fg-color: #fb923c;
+ --md-accent-fg-color--transparent: rgba(251, 146, 60, 0.14);
+ --md-typeset-a-color: #fb923c;
+
+ --md-default-bg-color: #0d1117;
+ --md-code-bg-color: #131a24;
+
+ --sc-ember: #fb923c;
+ --sc-ember-bright: #fdba74;
+}
+
+/* Header: a subtle slate gradient gives the navbar some depth, with a thin
+ ember hairline beneath it. */
+.md-header {
+ background: linear-gradient(
+ 100deg,
+ var(--md-primary-fg-color--dark),
+ var(--md-primary-fg-color--light)
+ );
+ box-shadow:
+ 0 0 0.2rem rgba(0, 0, 0, 0.25),
+ inset 0 -2px 0 0 var(--sc-ember);
+}
+.md-tabs {
+ background: transparent;
+}
+
+/* Typography: tighter, more confident headings. */
+.md-typeset h1 {
+ font-weight: 700;
+ letter-spacing: -0.02em;
+}
+.md-typeset h2 {
+ font-weight: 650;
+ letter-spacing: -0.01em;
+ margin-top: 2.2rem;
+}
+.md-typeset h3 {
+ font-weight: 600;
+}
+
+/* Softer corners on admonitions/details, tables, and code. */
+.md-typeset .admonition,
+.md-typeset details {
+ border-radius: 0.4rem;
+ border-left-width: 0.2rem;
+}
+.md-typeset code {
+ border-radius: 0.25rem;
+}
+.md-typeset pre > code,
+.md-typeset .highlight {
+ border-radius: 0.5rem;
+}
+
+/* Reference tables: tidy, with an ember-tinted header row. */
+.md-typeset table:not([class]) {
+ border-radius: 0.4rem;
+ overflow: hidden;
+}
+.md-typeset table:not([class]) th {
+ font-weight: 600;
+ background: var(--md-accent-fg-color--transparent);
+}
+
+/* Primary buttons (used on the home page hero). */
+.md-typeset .md-button--primary {
+ background-color: var(--sc-ember);
+ border-color: var(--sc-ember);
+ color: #fff;
+}
+.md-typeset .md-button--primary:hover {
+ background-color: var(--sc-ember-bright);
+ border-color: var(--sc-ember-bright);
+}
+
+/* ---- Home page hero -------------------------------------------------- */
+
+.sc-hero {
+ text-align: center;
+ padding: 2.6rem 1rem 1.6rem;
+}
+.sc-hero h1 {
+ font-size: 2.5rem;
+ line-height: 1.1;
+ margin: 0 0 0.4rem;
+}
+.sc-hero .sc-tagline {
+ font-size: 1.15rem;
+ color: var(--md-default-fg-color--light);
+ max-width: 40rem;
+ margin: 0 auto 1.4rem;
+}
+.sc-badges {
+ display: flex;
+ gap: 0.4rem;
+ justify-content: center;
+ flex-wrap: wrap;
+ margin-bottom: 1.4rem;
+}
+
+/* Feature cards grid on the home page. */
+.sc-grid {
+ display: grid;
+ grid-template-columns: repeat(auto-fit, minmax(15rem, 1fr));
+ gap: 1rem;
+ margin: 1.6rem 0;
+}
+.sc-card {
+ border: 1px solid var(--md-default-fg-color--lightest);
+ border-radius: 0.6rem;
+ padding: 1rem 1.1rem;
+ transition: border-color 0.2s ease, transform 0.2s ease;
+}
+.sc-card:hover {
+ border-color: var(--sc-ember);
+ transform: translateY(-2px);
+}
+.sc-card h3 {
+ margin: 0 0 0.35rem;
+ font-size: 1rem;
+}
+.sc-card p {
+ margin: 0;
+ font-size: 0.82rem;
+ color: var(--md-default-fg-color--light);
+}
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
new file mode 100644
index 0000000..f15c70f
--- /dev/null
+++ b/docs/troubleshooting.md
@@ -0,0 +1,66 @@
+# Troubleshooting
+
+Common issues and solutions when using Spark Connect Rust.
+
+### Cannot Connect to Server / Connection Refused
+
+**Symptom:** `Error: Failed to connect to sc://localhost:15002` or connection
timeout.
+
+**Solution:**
+1. Verify the server is running: `$SPARK_HOME/sbin/start-connect-server.sh
--packages "org.apache.spark:spark-connect_2.13:4.2.0"`
+2. Check the port is correct. By default it's `15002`. To verify, look for
`spark.connect.grpc.binding.port` in the server logs.
+3. If connecting to a remote server, verify network connectivity: `ping
spark.example.com` and check firewalls.
+4. Ensure Apache Spark 4.2.0 or later is installed:
`$SPARK_HOME/bin/spark-shell --version`
+
+!!! tip
+ Run the server in the foreground to see startup messages:
`$SPARK_HOME/sbin/start-connect-server.sh --packages
"org.apache.spark:spark-connect_2.13:4.2.0" 2>&1`
+
+### TLS / Authentication Error
+
+**Symptom:** `Error: TLS handshake failed` or `UNAUTHENTICATED` error when
connecting.
+
+**Solution:**
+1. Verify you are using the correct connection string format with TLS enabled:
+ ```rust
+let spark = SparkSessionBuilder::default()
+ .remote("sc://host:15002/;use_ssl=true;token=YOUR_TOKEN")
+ .get_or_create()?;
+```
+2. If the server uses self-signed certificates, consult your infrastructure
team.
+3. Check that your token or authentication credentials are valid.
+4. Verify the server is configured to accept your authentication method (check
server config or logs).
+
+!!! note
+ Authentication parameters are passed in the connection string, not as
config options.
+
+### Slow First Query / Long Session Startup
+
+**Symptom:** The first query takes 10โ30 seconds, or SparkSession creation is
slow.
+
+**Solution:**
+This is normal. The JVM and Spark runtime warm up on first use. Subsequent
queries are fast.
+
+- If this is unacceptable for your use case, consider keeping a long-lived
session and reusing it.
+- Profile your workload: most of the overhead is in JVM startup, not the Rust
client.
+- On a remote server, network latency may add to startup time.
+
+### Protobuf Compilation Error
+
+**Symptom:** Error during `cargo build`: `protobuf compiler not found` or
similar.
+
+**Solution:**
+1. Install `protobuf-compiler`:
+ - macOS: `brew install protobuf`
+ - Linux (Debian/Ubuntu): `apt-get install protobuf-compiler`
+ - Other systems: See [Protocol Buffers
documentation](https://developers.google.com/protocol-buffers/docs/downloads)
+2. Verify installation: `protoc --version`
+3. Retry: `cargo build --release`
+
+### Dataset Version Mismatch
+
+**Symptom:** Errors about incompatible schema or unexpected null values.
+
+**Solution:**
+1. Verify the Spark Connect server and client are compatible versions: both
should be Spark 4.2.0 or later.
+2. If you recently upgraded Apache Spark, restart the Spark Connect server to
apply the new version.
+3. Check that data on disk matches the expected schema (e.g., after schema
changes, re-write the data).
diff --git a/docs/types.md b/docs/types.md
new file mode 100644
index 0000000..0d4a572
--- /dev/null
+++ b/docs/types.md
@@ -0,0 +1,127 @@
+# Types and Schemas
+
+Spark SQL uses a rich type system. Learn how to work with schemas in the
`spark_connect` crate and cast columns between types.
+
+## Spark SQL Types
+
+Map Spark SQL types to their Rust `spark_connect::types::DataType` equivalents:
+
+| Spark Type | Rust DataType |
+|-----------|---------|
+| StringType | `DataType::String { collation: "UTF8_BINARY".to_string() }` |
+| IntegerType | `DataType::Integer` |
+| LongType | `DataType::Long` |
+| FloatType | `DataType::Float` |
+| DoubleType | `DataType::Double` |
+| BooleanType | `DataType::Boolean` |
+| BinaryType | `DataType::Binary` |
+| DateType | `DataType::Date` |
+| TimestampType | `DataType::Timestamp` |
+| DecimalType | `DataType::Decimal { precision: 10, scale: 2 }` |
+| ArrayType | `DataType::Array { element_type: Box::new(DataType::String { ...
}), contains_null: true }` |
+| MapType | `DataType::Map { key_type: Box::new(...), value_type:
Box::new(...), value_contains_null: true }` |
+| StructType | `DataType::Struct { fields: Vec<StructField> }` |
+
+## Inspecting Schemas
+
+Access schema information from a DataFrame:
+
+```rust
+// Get full schema
+let schema = df.schema()?;
+
+// Print schema
+df.print_schema()?;
+
+// Get list of (name, type) tuples
+let dtypes = df.dtypes()?;
+for (name, dtype) in dtypes {
+ println!("{}: {}", name, dtype);
+}
+```
+
+## Building a Schema
+
+Define a schema explicitly when reading or creating DataFrames:
+
+```rust
+use spark_connect::types::DataType;
+
+// Define schema as DDL string
+let schema_ddl = "name string, age int, salary long";
+
+let df = spark.read()
+ .schema(schema_ddl.to_string())
+ .csv("/path/to/data.csv");
+```
+
+## Casting Columns
+
+Convert a column to a different type with `.cast()`:
+
+```rust
+use spark_connect::types::DataType;
+
+// Cast to LongType
+let df = df.with_column("id", col("id").cast(DataType::Long));
+
+// Cast to DoubleType
+let df = df.with_column("price", col("price").cast(DataType::Double));
+
+// Cast with a type name string
+let df = df.with_column("created", col("created").cast_str("timestamp"));
+```
+
+## Nested Types
+
+Work with complex nested structures:
+
+```rust
+use spark_connect::types::{DataType, StructField};
+use std::collections::BTreeMap;
+
+// Array of strings
+let array_of_strings = DataType::Array {
+ element_type: Box::new(DataType::String {
+ collation: "UTF8_BINARY".to_string(),
+ }),
+ contains_null: true,
+};
+
+// Map with string keys and integer values
+let map_type = DataType::Map {
+ key_type: Box::new(DataType::String {
+ collation: "UTF8_BINARY".to_string(),
+ }),
+ value_type: Box::new(DataType::Integer),
+ value_contains_null: true,
+};
+
+// Nested struct
+let nested = DataType::Struct {
+ fields: vec![
+ StructField {
+ name: "name".to_string(),
+ data_type: DataType::String {
+ collation: "UTF8_BINARY".to_string(),
+ },
+ nullable: true,
+ metadata: BTreeMap::new(),
+ },
+ StructField {
+ name: "tags".to_string(),
+ data_type: DataType::Array {
+ element_type: Box::new(DataType::String {
+ collation: "UTF8_BINARY".to_string(),
+ }),
+ contains_null: true,
+ },
+ nullable: true,
+ metadata: BTreeMap::new(),
+ },
+ ],
+};
+```
+
+!!! note
+ Nullable fields allow `NULL` values. Set to `False` when a field must
always have a value.
diff --git a/docs/udfs.md b/docs/udfs.md
new file mode 100644
index 0000000..358b4e0
--- /dev/null
+++ b/docs/udfs.md
@@ -0,0 +1,160 @@
+# Rust UDFs via WebAssembly
+
+Run **Rust** user-defined functions on Spark, distributed on the executors,
with
+no server-side plugin. You write plain Rust functions; the toolchain compiles
+them to WebAssembly, embeds the module, and ships it inside a standard Spark
+`PythonUDF`. The executors need only the `wasmtime` Python package - no Rust
+toolchain server-side.
+
+!!! note "Opt-in"
+ This is behind the `wasm-udf` feature. Users who don't run Rust UDFs pull
+ none of the extra dependencies and need none of the preconditions below.
+
+## Write plain Rust functions
+
+Annotate a module of functions with `#[spark_wasm_udf]`. For each function the
+macro infers the Spark signature from the Rust types, exports it to WASM, and
+generates a constructor under `udf::` - a direct call `udf::<name>(col0, col1,
+...)` (one column per argument, **arity checked at compile time**) that
returns the
+result `Column`. A one-line `build.rs` compiles the module.
+
+```rust
+// src/main.rs
+use spark_connect_macros::spark_wasm_udf;
+
+#[spark_wasm_udf]
+mod udfs {
+ pub fn add_one(x: i64) -> i64 { x + 1 } // (Long)
-> Long
+ pub fn shout(s: String) -> String { format!("{}!", s.to_uppercase()) }
+ pub fn sum(xs: Vec<i64>) -> i64 { xs.iter().sum() } //
ArrayType arg
+ pub fn double_or_null(x: Option<i64>) -> Option<i64> { x.map(|v| v * 2) }
+}
+
+#[cfg(not(target_arch = "wasm32"))]
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+ use spark_connect::functions::col;
+ use spark_connect::SparkSessionBuilder;
+
+ let spark = SparkSessionBuilder::default()
+ .remote("sc://localhost:15002")
+ .get_or_create()?;
+
+ // Pass columns straight in - one per argument. Signatures are inferred and
+ // the compiled module is embedded; there is no `.wasm` file to load.
+ spark
+ .range(5)?
+ .select(vec![
+ col("id"),
+ udf::add_one(col("id"))?.alias("plus_one"),
+ ])
+ .show(20)?;
+ Ok(())
+}
+```
+
+```rust
+// build.rs
+fn main() {
+ spark_connect_build::embed_wasm_udf("src/main.rs");
+}
+```
+
+```toml
+# Cargo.toml
+[dependencies]
+spark-connect-macros = { package = "apache-spark-connect-macros", version =
"4.2" }
+
+# host-only: the client is not pulled into the wasm build
+[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
+spark-connect = { package = "apache-spark-connect", version = "4.2", features
= ["wasm-udf"] }
+
+[build-dependencies]
+spark-connect-build = { package = "apache-spark-connect-build", version =
"4.2" }
+spark-connect-macros = { package = "apache-spark-connect-macros", version =
"4.2" }
+```
+
+Run it:
+
+```bash
+rustup target add wasm32-unknown-unknown # once
+export SPARK_CONNECT_WASM_PACKER_PATH=$PWD/python # so the client finds the
packer
+cargo run
+```
+
+!!! tip "Runnable examples"
+ See `examples/wasm-udf-inline/` (the UDFs and client in one file) and
+ `examples/wasm-udfs/` + `examples/src/wasm_udf_macro.rs` (UDFs as a
reusable
+ crate, called as `wasm_udfs::udf::add_one(col("id"))?`).
+
+## Supported types
+
+Arguments and results cross the WASM boundary with a length-prefixed binary ABI
+(`spark_connect::wasm_udf::AbiType`), inferred from the Rust signature:
+
+| Rust | Spark SQL type |
+|-------------|----------------------|
+| `i32` | `IntegerType` |
+| `i64` | `LongType` |
+| `f32` | `FloatType` |
+| `f64` | `DoubleType` |
+| `bool` | `BooleanType` |
+| `String` | `StringType` |
+| `Vec<u8>` | `BinaryType` |
+| `Vec<T>` | `ArrayType` (of `T`) |
+| `Option<T>` | nullable `T` |
+
+These nest arbitrarily - e.g. `Vec<Option<String>>` -> `ArrayType(StringType,
+nullable)`.
+
+## How it works
+
+```text
+build time build.rs -> embed_wasm_udf() recompiles the source for wasm32
+ (host-only code is cfg'd out) and embeds the .wasm module.
+
+client udf::add_one(col) builds a standard PythonUDF: a tiny Python
runner
+ plus the module are cloudpickled by value, so executors need only
+ `wasmtime` - nothing to pre-deploy.
+
+executors the Python worker instantiates the module with wasmtime and
invokes
+ the exported entrypoint once per input row over the binary ABI.
+```
+
+The user never writes WASM, never touches the ABI, and never loads a `.wasm`
+file - they write `add_one(x: i64) -> i64` and call `udf::add_one(col("id"))`.
+
+## Preconditions (only when using Rust UDFs)
+
+Nothing here is needed unless the `wasm-udf` feature is enabled:
+
+- **Build machine** - the `wasm32-unknown-unknown` target (`rustup target add
+ wasm32-unknown-unknown`), plus `apache-spark-connect-macros` and
+ `apache-spark-connect-build` as (build-)dependencies.
+- **Client** (building the UDF command) - a Python interpreter with
`cloudpickle`
+ and `pyspark`, and the repo's `python/` directory importable as
+ `pyspark_wasm_udf` (point `SPARK_CONNECT_WASM_PACKER_PATH` at it, or set
+ `SPARK_CONNECT_PYTHON`).
+- **Executors** - the `wasmtime` Python package.
+- **Spark** - 4.2.0+.
+
+## Advanced
+
+For non-deterministic UDFs or custom packer configuration, the macro also
+generates a builder `udf::<name>_udf()`:
+
+```rust
+udf::add_one_udf()
+ .as_nondeterministic()
+ .call(vec![col("id")])?;
+```
+
+If you'd rather load a prebuilt module and spell out the types yourself, the
+lower-level factory mirrors `pyspark.sql.functions.udf`:
+
+```rust
+use spark_connect::wasm_udf::{udf, AbiType};
+
+let wasm = std::fs::read("shout.wasm")?;
+let shout = udf("shout", wasm, "shout", vec![AbiType::Str], AbiType::Str);
+let c = shout.call(vec![col("name")])?;
+```
diff --git a/mkdocs.yml b/mkdocs.yml
new file mode 100644
index 0000000..940af9c
--- /dev/null
+++ b/mkdocs.yml
@@ -0,0 +1,105 @@
+site_name: Spark Connect Rust Client
+site_description: >-
+ A Rust rewrite of the PySpark Spark Connect client - a drop-in `pyspark`
+ package backed by a native Rust core.
+site_url: https://apache.github.io/spark-connect-rust/
+repo_url: https://github.com/apache/spark-connect-rust
+repo_name: apache/spark-connect-rust
+edit_uri: edit/master/docs/
+copyright: >-
+ Copyright ยฉ The Apache Software Foundation. Apache Spark, Spark, and the
Spark
+ logo are trademarks of the Apache Software Foundation.
+
+theme:
+ name: material
+ icon:
+ logo: material/language-rust
+ repo: fontawesome/brands/github
+ features:
+ - navigation.tabs
+ - navigation.tabs.sticky
+ - navigation.sections
+ - navigation.top
+ - navigation.instant
+ - navigation.instant.progress
+ - navigation.footer
+ - toc.follow
+ - content.code.copy
+ - content.code.annotate
+ - content.tabs.link
+ - search.highlight
+ - search.suggest
+ font:
+ text: Inter
+ code: JetBrains Mono
+ # Slate & rust-ember. Exact brand colors are CSS custom properties in
+ # stylesheets/extra.css (hence `primary: custom` / `accent: custom`).
+ palette:
+ - media: "(prefers-color-scheme: light)"
+ scheme: default
+ primary: custom
+ accent: custom
+ toggle:
+ icon: material/weather-night
+ name: Switch to dark mode
+ - media: "(prefers-color-scheme: dark)"
+ scheme: slate
+ primary: custom
+ accent: custom
+ toggle:
+ icon: material/weather-sunny
+ name: Switch to light mode
+
+extra_css:
+ - stylesheets/extra.css
+
+extra:
+ social:
+ - icon: fontawesome/brands/github
+ link: https://github.com/apache/spark-connect-rust
+ name: Source on GitHub
+ - icon: fontawesome/brands/python
+ link: https://pypi.org/project/pyspark-client-rust/
+ name: pyspark-client-rust on PyPI
+
+markdown_extensions:
+ - admonition
+ - attr_list
+ - md_in_html
+ - tables
+ - toc:
+ permalink: true
+ - pymdownx.highlight:
+ anchor_linenums: true
+ use_pygments: true
+ - pymdownx.superfences:
+ custom_fences:
+ - name: mermaid
+ class: mermaid
+ format: !!python/name:pymdownx.superfences.fence_code_format
+ - pymdownx.inlinehilite
+ - pymdownx.snippets
+ - pymdownx.tabbed:
+ alternate_style: true
+
+nav:
+ - Home: index.md
+ - Get started:
+ - Installation: installation.md
+ - Quickstart: quickstart.md
+ - Configuration and Connection: configuration.md
+ - Guides:
+ - DataFrames: dataframes.md
+ - Columns and Functions: columns-and-functions.md
+ - SQL: sql.md
+ - Reading and Writing: data-sources.md
+ - Structured Streaming: streaming.md
+ - Catalog: catalog.md
+ - Types and Schemas: types.md
+ - Rust UDFs via WebAssembly: udfs.md
+ - Architecture: architecture.md
+ - Reference:
+ - Examples: examples.md
+ - Compatibility: compatibility.md
+ - Troubleshooting: troubleshooting.md
+ - Contributing: contributing.md
diff --git a/scripts/gen_parity_ledger.py b/scripts/gen_parity_ledger.py
index b8f521e..9acb840 100644
--- a/scripts/gen_parity_ledger.py
+++ b/scripts/gen_parity_ledger.py
@@ -10,11 +10,12 @@ Rust rewrite so nothing is silently dropped.
Usage:
python scripts/gen_parity_ledger.py \
--src ~/workspace/origin/spark/python/pyspark \
- --out docs/parity
+ --out dev/parity
-Status of each item is tracked in ``docs/parity/status.csv`` (hand/tool
edited);
-this script only (re)generates the *inventory* (``docs/parity/inventory.csv``)
-and never clobbers status.
+Each item's ``status`` and ``notes`` live in the columns of the ledger itself
+(``dev/parity/inventory.csv``, hand/tool edited). When regenerating, this
script
+reads back any existing ``status``/``notes`` for a symbol and preserves them,
so
+it never clobbers curated status.
"""
from __future__ import annotations
@@ -108,7 +109,7 @@ def walk_file(path: Path, rel: str, rows: list[dict]) ->
None:
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--src", required=True, help="pyspark package root")
- ap.add_argument("--out", default="docs/parity")
+ ap.add_argument("--out", default="dev/parity")
args = ap.parse_args()
src = Path(os.path.expanduser(args.src))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
