[
https://issues.apache.org/jira/browse/SPARK-59143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59143:
-----------------------------------
Labels: pull-request-available (was: )
> Close remaining PySpark 4.2.0 Spark Connect public-API parity gaps in the
> Rust drop-in (connect submodule import paths and minor signatures)
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59143
> URL: https://issues.apache.org/jira/browse/SPARK-59143
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> A close, skeptical introspection diff of the Rust drop-in against the
> reference
> `pyspark==4.2.0` Spark Connect client (comparing every public
> class/method/function
> signature) surfaced a small set of remaining public-API parity gaps. This
> issue
> tracks closing them.
> Structural (import-path) gaps:
> - The `pyspark.sql.connect.*` package ships only `dataframe`, `column`,
> `session`,
> `plan`, and `proto`. Reference code and the official Connect test suite also
> import from `pyspark.sql.connect.{catalog, conf, group, window, readwriter,
> tvf,
> udf, udtf, observation, merge}` and
> `pyspark.sql.connect.streaming.{readwriter,
> query}`, which currently raise ImportError. Add thin re-export shims
> mirroring
> the existing `connect/dataframe.py` pattern.
> - The public `pyspark.sql.conf` and `pyspark.sql.tvf` modules are absent
> (`from pyspark.sql.conf import RuntimeConfig` / `from pyspark.sql.tvf import
> TableValuedFunction` fail).
> - The runtime-config class is exposed from the Rust extension as `RuntimeConf`
> instead of the reference name `RuntimeConfig`.
> Minor signature divergences vs reference (reference-valid code can break):
> - `StructType.treeString` is missing the `maxDepth` parameter.
> - `StructType.fromJson` / `StructField.fromJson` use param name `data`;
> reference
> uses `json` (keyword calls break).
> - `TableValuedFunction.json_tuple` / `stack` take a single `fields` list;
> reference
> uses varargs `*fields`.
> - `VariantType.toInternal` uses param name `obj`; reference uses `variant`.
> The function surface (`pyspark.sql.functions`) was already at full parity
> (all 529
> reference functions present, no signature mismatches).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
