Hyukjin Kwon created SPARK-59143:
------------------------------------
Summary: Close remaining PySpark 4.2.0 Spark Connect public-API
parity gaps in the Rust drop-in (connect submodule import paths and minor
signatures)
Key: SPARK-59143
URL: https://issues.apache.org/jira/browse/SPARK-59143
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
A close, skeptical introspection diff of the Rust drop-in against the reference
`pyspark==4.2.0` Spark Connect client (comparing every public
class/method/function
signature) surfaced a small set of remaining public-API parity gaps. This issue
tracks closing them.
Structural (import-path) gaps:
- The `pyspark.sql.connect.*` package ships only `dataframe`, `column`,
`session`,
`plan`, and `proto`. Reference code and the official Connect test suite also
import from `pyspark.sql.connect.{catalog, conf, group, window, readwriter,
tvf,
udf, udtf, observation, merge}` and
`pyspark.sql.connect.streaming.{readwriter,
query}`, which currently raise ImportError. Add thin re-export shims mirroring
the existing `connect/dataframe.py` pattern.
- The public `pyspark.sql.conf` and `pyspark.sql.tvf` modules are absent
(`from pyspark.sql.conf import RuntimeConfig` / `from pyspark.sql.tvf import
TableValuedFunction` fail).
- The runtime-config class is exposed from the Rust extension as `RuntimeConf`
instead of the reference name `RuntimeConfig`.
Minor signature divergences vs reference (reference-valid code can break):
- `StructType.treeString` is missing the `maxDepth` parameter.
- `StructType.fromJson` / `StructField.fromJson` use param name `data`;
reference
uses `json` (keyword calls break).
- `TableValuedFunction.json_tuple` / `stack` take a single `fields` list;
reference
uses varargs `*fields`.
- `VariantType.toInternal` uses param name `obj`; reference uses `variant`.
The function surface (`pyspark.sql.functions`) was already at full parity (all
529
reference functions present, no signature mismatches).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
