Hyukjin Kwon created SPARK-59140:
------------------------------------
Summary: Close PySpark 4.2.0 public-API parity gaps in the Rust
drop-in client
Key: SPARK-59140
URL: https://issues.apache.org/jira/browse/SPARK-59140
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Rust drop-in client (pyspark-client-rust) advertises "100% public-API
parity with PySpark 4.2.0 / runs unchanged", but keyword-argument calls raised
TypeError on ~507 SQL functions and dozens of
DataFrame/Column/Catalog/Reader/Writer/Streaming/Session methods, and several
parameters were missing entirely. So common PySpark code did not run unchanged.
This issue tracks bringing the public API of pyspark.sql.connect.* to
keyword-argument and signature parity with PySpark 4.2.0.
Verified with an independent signature harness against the v4.2.0 tag:
- function keyword-arg mismatches: 513 -> 0
- class-method param mismatches: 70 -> 2 (both intentional: Column.alias/name
expose an explicit metadata= where the reference uses **kwargs)
Also verified: all Rust golden/core tests pass, and a focused end-to-end suite
runs 16/16 green against a real Spark Connect 4.2.0 server.
A few params are accepted for signature parity but not yet behaviorally wired
pending connect-core/proto support (documented in-code and in
PARITY_GAP_ANALYSIS.md): mapInPandas/mapInArrow(profile),
localCheckpoint(storageLevel), printSchema(level),
createDataFrame(verifySchema), trigger(realTime), multi-path
read.parquet(*paths).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
