HyukjinKwon opened a new pull request, #94:
URL: https://github.com/apache/spark-connect-rust/pull/94
## What changes were proposed in this pull request?
Brings the `pyspark-client-rust` public API to keyword-argument and
signature parity with **PySpark 4.2.0** (`pyspark.sql.connect.*`). Verified by
an independent signature harness that diffs `inspect.signature` for every
public class/method/function against the reference `v4.2.0` tag:
| | Before | After |
|---|---|---|
| Function keyword-arg mismatches (`*args`) | **513** | **0** |
| Class-method param mismatches | **70** | **2** (intentional) |
**GAP 1 — functions.** `scripts/gen_fn_dispatch.py` now emits explicit
`def`s carrying each function's real 4.2.0 parameter names. Optional params
default to an `_UNSET` sentinel and are forwarded only when supplied, so the
historical positional dispatch is preserved byte-for-byte (golden tests
unchanged) while keyword calls now work: `F.first(col, ignorenulls=True)`,
`F.round(col, scale=2)`, `F.split(str, pattern=..., limit=...)`,
`F.from_json(col, schema=...)`, etc. The hand-written
`col`/`lit`/`expr`/`column`/`sha2`/`window` are aligned too. The generated Rust
dispatch is now guarded behind `--write-dispatch` so the hand-tuned
`dispatch_generated.rs` (which carries multi-arg arms like
`round_scale`/`approx_count_distinct_rsd`) is never clobbered on regeneration.
**GAP 2 — methods.** camelCase param renames and re-added/wired dropped
params across `dataframe.rs`, `column.rs`, `group.rs`, `catalog.rs`,
`readwriter.rs`, `streaming.rs`, `session.rs`, `stat.rs`:
`sample(withReplacement)`, `sort`/`orderBy(ascending)`,
`drop`/`toDF`/`selectExpr`, `melt`/`unpivot(variableColumnName,
valueColumnName)`, `persist(storageLevel)`, all `Catalog` methods,
`tableExists`/`functionExists(dbName)`, the reader/writer/streaming option
lists, `save`/`saveAsTable(format, mode, partitionBy, **options)`,
`insertInto(overwrite)`, `trigger(realTime)`, `createDataFrame(samplingRatio,
verifySchema)`, `sql(**kwargs)`, `corr(method)`,
`mapInPandas`/`mapInArrow(barrier, profile)`, `printSchema(level)`,
`localCheckpoint(storageLevel)`, `groupingSets(*cols)`,
`awaitTermination(timeout)`. A new core
`Catalog::cache_table_with_storage_level` wires `cacheTable(storageLevel)` onto
the `CacheTable` proto field.
The **2 remaining** mismatches are intentional: `Column.alias`/`Column.name`
expose an explicit `metadata=None` where the reference uses `**kwargs`; our
form accepts every reference call (`alias("x")`, `alias("x", metadata=m)`) and
is stricter/clearer, so it is deliberately kept.
A handful of params are accepted for signature parity but not yet
behaviorally wired because the connect **core** lacks the field (clearly
commented in-code, and listed in `PARITY_GAP_ANALYSIS.md` as follow-ups):
`mapInPandas/mapInArrow(profile)`, `localCheckpoint(storageLevel)`,
`printSchema(level)`, `createDataFrame(verifySchema)`, `trigger(realTime)`
(raises `NotImplementedError`), and multi-path `read.parquet(*paths)`.
## Why are the changes needed?
The project claims "100% public-API parity with PySpark 4.2.0 / runs
unchanged." In practice, keyword-argument calls raised `TypeError` on ~507
functions and dozens of methods, and several parameters were missing — so
common PySpark code did not run unchanged. `PARITY_GAP_ANALYSIS.md` documents
the original audit and the before/after.
## Does this PR introduce any user-facing change?
Yes — many more calls now work as they do in reference PySpark (keyword
arguments across functions/methods, and previously-missing parameters). No
behavior change for calls that already worked (positional dispatch is
preserved).
## How was this patch tested?
- Signature harness against the `v4.2.0` tag: **0** function mismatches,
**2** intentional class mismatches.
- All Rust golden/core tests pass (`cargo test -p apache-spark-connect`):
**0 failed**.
- `scripts/e2e_parity_check.py` runs **16/16 green** against a real Spark
Connect **4.2.0** server, exercising keyword function calls,
`sample(withReplacement=)`, `sort(ascending=)`,
`createDataFrame(verifySchema=)`, `sql(**kwargs)`, and a `write.save(format=,
mode=)` / `read.load(format=)` round-trip.
> Note: replace `SPARK-XXXXX` with the assigned ASF JIRA id before merging.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
