This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 5f8edba [SPARK-59143] Close remaining PySpark 4.2.0 Spark Connect
public-API parity gaps
5f8edba is described below
commit 5f8edba717a45d4f0e853818aaafe506cc9d6d5a
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Tue Sep 1 15:35:52 2026 +0900
[SPARK-59143] Close remaining PySpark 4.2.0 Spark Connect public-API parity
gaps
## What changes were proposed in this pull request?
Closes the remaining PySpark 4.2.0 Spark Connect public-API parity gaps
found by a
close `inspect.signature` diff of the drop-in against the reference
`pyspark==4.2.0`
Connect client.
**Import-path gaps** — thin re-export shims, same pattern as the existing
`connect/dataframe.py`:
- New `pyspark.sql.connect.*` submodules: `catalog`, `conf`, `group`,
`window`,
`readwriter`, `tvf`, `udf`, `udtf`, `observation`, `merge`,
`streaming.readwriter`, `streaming.query`. The reference client and the
official
Connect test suite import from these paths, which previously raised
`ImportError`.
- New public `pyspark.sql.conf` (exposing `RuntimeConfig`, aliasing the
Connect
`RuntimeConf`) and `pyspark.sql.tvf` modules.
**Signature gaps** vs the reference Connect client:
- `StructType.treeString` gains the `maxDepth` parameter (default
`2147483647`),
wired through a new `DataType::tree_string_with_depth` in the core;
`tree_string()` is preserved for existing callers.
- `StructType.fromJson` / `StructField.fromJson` parameter renamed `data` →
`json`.
- `TableValuedFunction.json_tuple` / `stack` now take varargs `*fields`.
- `VariantType.toInternal` parameter renamed `obj` → `variant`.
The `pyspark.sql.functions` surface was already at full parity (all 529
reference
functions, no signature mismatches) and is unchanged.
## Why are the changes needed?
The package advertises drop-in parity with the PySpark 4.2.0 Spark Connect
client.
These were the residual deviations where reference-valid imports and calls
could
break.
## Does this PR introduce any user-facing change?
Yes, additive: previously-failing imports now resolve, and the four methods
above
accept the reference parameter names/forms.
## How was this patch tested?
New offline tests in `python/tests/test_dropin_offline.py`:
- `test_connect_submodule_import_paths` (14 paths) — passes against the
built extension.
- Behavior tests for `treeString(maxDepth)`, `fromJson(json=...)`, and
`VariantType.toInternal(variant=...)` — validated against reference
`pyspark==4.2.0` to confirm they encode the correct behavior; they
exercise the
rebuilt extension under CI.
The Rust changes were authored but not compiled on the dev box (offline);
the
build + official Connect suite in CI is the build/behavior gate.
This pull request and its description were written by Isaac.
Closes #95 from HyukjinKwon/SPARK-59143-close-connect-parity-gaps.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 254dd506ad732ee83f7b1d720fcca0678be0dd33)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/tvf.rs | 3 +-
crates/pyspark-rs/src/types.rs | 21 +++----
crates/spark-connect/src/types.rs | 37 +++++++++---
crates/spark-connect/tests/types_behavior.rs | 28 ++++++++++
python/pyspark/sql/conf.py | 23 ++++++++
python/pyspark/sql/connect/catalog.py | 28 ++++++++++
python/pyspark/sql/connect/conf.py | 20 +++++++
python/pyspark/sql/connect/group.py | 21 +++++++
python/pyspark/sql/connect/merge.py | 20 +++++++
python/pyspark/sql/connect/observation.py | 20 +++++++
python/pyspark/sql/connect/readwriter.py | 24 ++++++++
python/pyspark/sql/connect/streaming/query.py | 20 +++++++
python/pyspark/sql/connect/streaming/readwriter.py | 20 +++++++
python/pyspark/sql/connect/tvf.py | 20 +++++++
python/pyspark/sql/connect/udf.py | 20 +++++++
python/pyspark/sql/connect/udtf.py | 25 +++++++++
python/pyspark/sql/connect/window.py | 20 +++++++
python/pyspark/sql/tvf.py | 20 +++++++
python/tests/test_dropin_offline.py | 65 ++++++++++++++++++++++
19 files changed, 435 insertions(+), 20 deletions(-)
diff --git a/crates/pyspark-rs/src/tvf.rs b/crates/pyspark-rs/src/tvf.rs
index 22cf760..523e4bd 100644
--- a/crates/pyspark-rs/src/tvf.rs
+++ b/crates/pyspark-rs/src/tvf.rs
@@ -75,7 +75,7 @@ impl PyTableValuedFunction {
))
}
- #[pyo3(name = "json_tuple")]
+ #[pyo3(name = "json_tuple", signature = (input, *fields))]
fn json_tuple(
&self,
py: Python<'_>,
@@ -108,6 +108,7 @@ impl PyTableValuedFunction {
))
}
+ #[pyo3(signature = (n, *fields))]
fn stack(
&self,
py: Python<'_>,
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index ad89a8b..b58db7c 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -1513,12 +1513,12 @@ impl PyStructField {
fn __sf_from_json(
_cls: &Bound<'_, pyo3::types::PyType>,
py: Python<'_>,
- data: &Bound<'_, PyAny>,
+ json: &Bound<'_, PyAny>,
) -> PyResult<PyStructField> {
let s: String = py
.import("json")?
.getattr("dumps")?
- .call1((data,))?
+ .call1((json,))?
.extract()?;
let field = StructField::from_json_str(&s)
.map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError,
_>(e.to_string()))?;
@@ -1808,13 +1808,14 @@ impl PyStructType {
.map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError,
_>(e.to_string()))
}
- /// Tree string. Mirrors `StructType.treeString()`.
- #[pyo3(name = "treeString")]
- fn __st_tree_string(&self) -> PyResult<String> {
+ /// Tree string. Mirrors `StructType.treeString(maxDepth)`.
+ #[allow(non_snake_case)]
+ #[pyo3(name = "treeString", signature = (maxDepth = 2147483647))]
+ fn __st_tree_string(&self, maxDepth: i32) -> PyResult<String> {
DataType::Struct {
fields: self.fields.clone(),
}
- .tree_string()
+ .tree_string_with_depth(maxDepth)
.map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError,
_>(e.to_string()))
}
@@ -1838,12 +1839,12 @@ impl PyStructType {
fn __st_from_json(
_cls: &Bound<'_, pyo3::types::PyType>,
py: Python<'_>,
- data: &Bound<'_, PyAny>,
+ json: &Bound<'_, PyAny>,
) -> PyResult<Py<PyStructType>> {
let s: String = py
.import("json")?
.getattr("dumps")?
- .call1((data,))?
+ .call1((json,))?
.extract()?;
match DataType::from_json_str(&s)
.map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError,
_>(e.to_string()))?
@@ -2319,8 +2320,8 @@ impl PyVariantType {
obj
}
#[pyo3(name = "toInternal")]
- fn __obj_to_internal<'py>(&self, obj: Bound<'py, PyAny>) -> Bound<'py,
PyAny> {
- obj
+ fn __obj_to_internal<'py>(&self, variant: Bound<'py, PyAny>) -> Bound<'py,
PyAny> {
+ variant
}
#[classmethod]
#[pyo3(name = "fromDDL")]
diff --git a/crates/spark-connect/src/types.rs
b/crates/spark-connect/src/types.rs
index aa8a42d..1f06c73 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -855,10 +855,17 @@ impl DataType {
/// Tree-string for a StructType, mirroring `StructType.treeString()`.
pub fn tree_string(&self) -> Result<String> {
+ self.tree_string_with_depth(i32::MAX)
+ }
+
+ /// Like [`DataType::tree_string`], but stops recursing into nested structs
+ /// once `max_depth` nesting levels have been printed (top-level fields are
+ /// depth 1). Mirrors `StructType.treeString(maxDepth)`.
+ pub fn tree_string_with_depth(&self, max_depth: i32) -> Result<String> {
match self {
DataType::Struct { .. } => {
let mut out = String::from("root\n");
- self.append_tree(&mut out, " |");
+ self.append_tree(&mut out, " |", 1, max_depth);
Ok(out)
}
_ => Err(SparkError::value(
@@ -868,19 +875,31 @@ impl DataType {
}
}
- fn append_tree(&self, out: &mut String, prefix: &str) {
+ fn append_tree(&self, out: &mut String, prefix: &str, depth: i32,
max_depth: i32) {
if let DataType::Struct { fields } = self {
for f in fields {
+ // A nested struct is rendered as the bare type name
("struct") and its
+ // fields are shown by recursing below, so `max_depth` can
truncate them.
+ // Using `simple_string()` here would inline the children (e.g.
+ // "struct<inner:int>"), leaking them past `max_depth`.
Non-struct types
+ // have no depth-controlled children, so their simple string
is used.
+ let is_struct = matches!(f.data_type, DataType::Struct { .. });
+ let type_repr = if is_struct {
+ f.data_type.type_name()
+ } else {
+ f.data_type.simple_string()
+ };
out.push_str(&format!(
"{}-- {}: {} (nullable = {})\n",
- prefix,
- f.name,
- f.data_type.simple_string(),
- f.nullable
+ prefix, f.name, type_repr, f.nullable
));
- if matches!(f.data_type, DataType::Struct { .. }) {
- f.data_type
- .append_tree(out, &format!("{} |",
prefix.trim_end_matches('|')));
+ if is_struct && depth < max_depth {
+ f.data_type.append_tree(
+ out,
+ &format!("{} |", prefix.trim_end_matches('|')),
+ depth + 1,
+ max_depth,
+ );
}
}
}
diff --git a/crates/spark-connect/tests/types_behavior.rs
b/crates/spark-connect/tests/types_behavior.rs
index 95cfa1d..8f32b79 100644
--- a/crates/spark-connect/tests/types_behavior.rs
+++ b/crates/spark-connect/tests/types_behavior.rs
@@ -293,6 +293,34 @@ fn struct_to_ddl_tree_and_nullable() {
assert!(DataType::Integer.tree_string().is_err());
}
+#[test]
+fn tree_string_with_depth_truncates_nested_structs() {
+ use spark_connect::types::StructField;
+ let nested = DataType::Struct {
+ fields: vec![StructField {
+ name: "outer".to_string(),
+ data_type: DataType::Struct {
+ fields: vec![StructField {
+ name: "inner".to_string(),
+ data_type: DataType::Integer,
+ nullable: true,
+ metadata: Default::default(),
+ }],
+ },
+ nullable: true,
+ metadata: Default::default(),
+ }],
+ };
+ // Full depth shows the nested field, rendering the struct as the bare
type name.
+ let full = nested.tree_string().unwrap();
+ assert!(full.contains("outer: struct (nullable = true)"));
+ assert!(full.contains("inner: int (nullable = true)"));
+ // max_depth = 1 prints only the top level; the nested field is omitted.
+ let shallow = nested.tree_string_with_depth(1).unwrap();
+ assert!(shallow.contains("outer: struct (nullable = true)"));
+ assert!(!shallow.contains("inner"));
+}
+
#[test]
fn from_json_str_roundtrips() {
use spark_connect::types::StructField;
diff --git a/python/pyspark/sql/conf.py b/python/pyspark/sql/conf.py
new file mode 100644
index 0000000..c873ce6
--- /dev/null
+++ b/python/pyspark/sql/conf.py
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Runtime configuration. The connect runtime-config object is the Rust-backed
RuntimeConf; RuntimeConfig is the public reference name and aliases it."""
+from pyspark._pyspark import RuntimeConf
+
+# Public reference name; the Spark Connect runtime config is RuntimeConf.
+RuntimeConfig = RuntimeConf
+
+__all__ = ["RuntimeConfig", "RuntimeConf"]
diff --git a/python/pyspark/sql/connect/catalog.py
b/python/pyspark/sql/connect/catalog.py
new file mode 100644
index 0000000..2646122
--- /dev/null
+++ b/python/pyspark/sql/connect/catalog.py
@@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Catalog under the connect import path (mirrors
pyspark.sql.connect.catalog)."""
+from pyspark.sql.catalog import (
+ Catalog,
+ CatalogMetadata,
+ Column,
+ Database,
+ Function,
+ Table,
+ TablePartition,
+)
+
+__all__ = ["Catalog", "CatalogMetadata", "Column", "Database", "Function",
"Table", "TablePartition"]
diff --git a/python/pyspark/sql/connect/conf.py
b/python/pyspark/sql/connect/conf.py
new file mode 100644
index 0000000..0ba8a10
--- /dev/null
+++ b/python/pyspark/sql/connect/conf.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""RuntimeConf under the connect import path (mirrors
pyspark.sql.connect.conf; the connect runtime-config class is named RuntimeConf
upstream)."""
+from pyspark._pyspark import RuntimeConf
+
+__all__ = ["RuntimeConf"]
diff --git a/python/pyspark/sql/connect/group.py
b/python/pyspark/sql/connect/group.py
new file mode 100644
index 0000000..f866e1f
--- /dev/null
+++ b/python/pyspark/sql/connect/group.py
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""GroupedData / PandasCogroupedOps under the connect import path."""
+from pyspark.sql.group import GroupedData
+from pyspark.sql.pandas.group_ops import PandasCogroupedOps
+
+__all__ = ["GroupedData", "PandasCogroupedOps"]
diff --git a/python/pyspark/sql/connect/merge.py
b/python/pyspark/sql/connect/merge.py
new file mode 100644
index 0000000..e509995
--- /dev/null
+++ b/python/pyspark/sql/connect/merge.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""MergeIntoWriter under the connect import path."""
+from pyspark.sql.merge import MergeIntoWriter
+
+__all__ = ["MergeIntoWriter"]
diff --git a/python/pyspark/sql/connect/observation.py
b/python/pyspark/sql/connect/observation.py
new file mode 100644
index 0000000..066b195
--- /dev/null
+++ b/python/pyspark/sql/connect/observation.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Observation under the connect import path."""
+from pyspark.sql.observation import Observation
+
+__all__ = ["Observation"]
diff --git a/python/pyspark/sql/connect/readwriter.py
b/python/pyspark/sql/connect/readwriter.py
new file mode 100644
index 0000000..35ab99d
--- /dev/null
+++ b/python/pyspark/sql/connect/readwriter.py
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""DataFrameReader / DataFrameWriter(V2) under the connect import path."""
+from pyspark.sql.readwriter import (
+ DataFrameReader,
+ DataFrameWriter,
+ DataFrameWriterV2,
+)
+
+__all__ = ["DataFrameReader", "DataFrameWriter", "DataFrameWriterV2"]
diff --git a/python/pyspark/sql/connect/streaming/query.py
b/python/pyspark/sql/connect/streaming/query.py
new file mode 100644
index 0000000..0bda959
--- /dev/null
+++ b/python/pyspark/sql/connect/streaming/query.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""StreamingQuery / StreamingQueryManager under the connect import path."""
+from pyspark.sql.streaming.query import StreamingQuery, StreamingQueryManager
+
+__all__ = ["StreamingQuery", "StreamingQueryManager"]
diff --git a/python/pyspark/sql/connect/streaming/readwriter.py
b/python/pyspark/sql/connect/streaming/readwriter.py
new file mode 100644
index 0000000..61cc912
--- /dev/null
+++ b/python/pyspark/sql/connect/streaming/readwriter.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""DataStreamReader / DataStreamWriter under the connect import path."""
+from pyspark.sql.streaming.readwriter import DataStreamReader, DataStreamWriter
+
+__all__ = ["DataStreamReader", "DataStreamWriter"]
diff --git a/python/pyspark/sql/connect/tvf.py
b/python/pyspark/sql/connect/tvf.py
new file mode 100644
index 0000000..24d81d9
--- /dev/null
+++ b/python/pyspark/sql/connect/tvf.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""TableValuedFunction under the connect import path (mirrors
pyspark.sql.connect.tvf)."""
+from pyspark._pyspark import TableValuedFunction
+
+__all__ = ["TableValuedFunction"]
diff --git a/python/pyspark/sql/connect/udf.py
b/python/pyspark/sql/connect/udf.py
new file mode 100644
index 0000000..3b7daaa
--- /dev/null
+++ b/python/pyspark/sql/connect/udf.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""UserDefinedFunction / UDFRegistration under the connect import path."""
+from pyspark.sql.udf import UDFRegistration, UserDefinedFunction
+
+__all__ = ["UDFRegistration", "UserDefinedFunction"]
diff --git a/python/pyspark/sql/connect/udtf.py
b/python/pyspark/sql/connect/udtf.py
new file mode 100644
index 0000000..a7034ca
--- /dev/null
+++ b/python/pyspark/sql/connect/udtf.py
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""User-defined table function classes under the connect import path."""
+from pyspark.sql.udtf import (
+ AnalyzeArgument,
+ AnalyzeResult,
+ UDTFRegistration,
+ UserDefinedTableFunction,
+)
+
+__all__ = ["AnalyzeArgument", "AnalyzeResult", "UDTFRegistration",
"UserDefinedTableFunction"]
diff --git a/python/pyspark/sql/connect/window.py
b/python/pyspark/sql/connect/window.py
new file mode 100644
index 0000000..5bfb3aa
--- /dev/null
+++ b/python/pyspark/sql/connect/window.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Window / WindowSpec under the connect import path."""
+from pyspark.sql.window import Window, WindowSpec
+
+__all__ = ["Window", "WindowSpec"]
diff --git a/python/pyspark/sql/tvf.py b/python/pyspark/sql/tvf.py
new file mode 100644
index 0000000..e7c84d3
--- /dev/null
+++ b/python/pyspark/sql/tvf.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Table-valued functions (spark.tvf.*)."""
+from pyspark._pyspark import TableValuedFunction
+
+__all__ = ["TableValuedFunction"]
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index dc01a95..6c74762 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2664,3 +2664,68 @@ def test_no_pyclass_reports_builtins_module():
if isinstance(obj, type) and obj.__module__ == "builtins"
)
assert offenders == [], f"missing module=: {offenders}"
+
+
+# ------------------------------------------------- connect.* import-path
parity
+
+# The reference Spark Connect client exposes these classes under the
+# pyspark.sql.connect.* package (and pyspark.sql.conf / pyspark.sql.tvf);
+# existing code and the official test suite import from these paths.
[email protected](
+ "module, names",
+ [
+ ("pyspark.sql.connect.catalog",
+ ["Catalog", "Database", "Table", "Function", "CatalogMetadata",
"TablePartition"]),
+ ("pyspark.sql.connect.conf", ["RuntimeConf"]),
+ ("pyspark.sql.connect.group", ["GroupedData", "PandasCogroupedOps"]),
+ ("pyspark.sql.connect.window", ["Window", "WindowSpec"]),
+ ("pyspark.sql.connect.readwriter",
+ ["DataFrameReader", "DataFrameWriter", "DataFrameWriterV2"]),
+ ("pyspark.sql.connect.tvf", ["TableValuedFunction"]),
+ ("pyspark.sql.connect.udf", ["UserDefinedFunction",
"UDFRegistration"]),
+ ("pyspark.sql.connect.udtf",
+ ["UserDefinedTableFunction", "UDTFRegistration", "AnalyzeArgument",
"AnalyzeResult"]),
+ ("pyspark.sql.connect.observation", ["Observation"]),
+ ("pyspark.sql.connect.merge", ["MergeIntoWriter"]),
+ ("pyspark.sql.connect.streaming.readwriter", ["DataStreamReader",
"DataStreamWriter"]),
+ ("pyspark.sql.connect.streaming.query", ["StreamingQuery",
"StreamingQueryManager"]),
+ ("pyspark.sql.conf", ["RuntimeConfig", "RuntimeConf"]),
+ ("pyspark.sql.tvf", ["TableValuedFunction"]),
+ ],
+)
+def test_connect_submodule_import_paths(module, names):
+ import importlib
+
+ mod = importlib.import_module(module)
+ missing = [n for n in names if not hasattr(mod, n)]
+ assert missing == [], f"{module} is missing {missing}"
+
+
+# --------------------------------------------- v4.2.0 signature-parity fixes
+
+def test_structtype_from_json_accepts_json_kwarg():
+ # Reference names the fromJson parameter `json`, not `data`.
+ import json as _json
+
+ st = T.StructType([T.StructField("a", T.IntegerType()), T.StructField("b",
T.StringType())])
+ payload = _json.loads(st.json())
+ assert T.StructType.fromJson(json=payload).simpleString() ==
st.simpleString()
+
+ sf = T.StructField("a", T.IntegerType())
+ assert T.StructField.fromJson(json=_json.loads(sf.json())).name == "a"
+
+
+def test_structtype_tree_string_max_depth():
+ nested = T.StructType([
+ T.StructField("outer", T.StructType([T.StructField("inner",
T.IntegerType())])),
+ ])
+ full = nested.treeString()
+ assert "inner" in full
+ # maxDepth=1 prints only the top level; the nested field is omitted.
+ assert "inner" not in nested.treeString(1)
+ assert "outer" in nested.treeString(1)
+
+
+def test_variant_type_to_internal_accepts_variant_kwarg():
+ # Reference names the VariantType.toInternal parameter `variant`, not
`obj`.
+ assert T.VariantType().toInternal(variant=None) is None
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
