This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 2dff0bf  [SPARK-60957][CONNECT][PYTHON] Import pyspark.sql.conversion, 
and PandasUDFType from pyspark.sql.functions, as in PySpark
2dff0bf is described below

commit 2dff0bfee476578fed9b7822735b60646aac4dfa
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 12:07:56 2026 +0900

    [SPARK-60957][CONNECT][PYTHON] Import pyspark.sql.conversion, and 
PandasUDFType from pyspark.sql.functions, as in PySpark
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60957](https://issues.apache.org/jira/browse/SPARK-60957): these 
imports now work as in PySpark 4.3.0:
    
    - **`import pyspark.sql.conversion`:** `pyspark.sql.types` now has 
`_has_type`, which the vendored `pyspark.sql.conversion` imports. The function 
is PySpark's: it looks into the fields of a struct, the element of an array, 
the key and value of a map, and the `sqlType` of a user-defined type.
    - **`from pyspark.sql.functions import PandasUDFType`, after `import 
pyspark.sql.pandas.functions`:**
      - `pyspark.sql.pandas.functions` now defines `PandasUDFType` and 
`ArrowUDFType` before it imports `pyspark.sql.functions`. The re-export at the 
end of `pyspark.sql.functions` then finds them, whichever module is imported 
first.
      - The re-export no longer swallows an error.
    - **`pyspark.sql.pandas.functions` attributes:** it also has `arrow_udf` 
and `PythonEvalType`, as in PySpark. The vendored 
`pyspark.sql.pandas.typehints` reads `PythonEvalType` from there.
    
    ### Why are the changes needed?
    
    - `import pyspark.sql.conversion` raised `ImportError: cannot import name 
'_has_type' from 'pyspark.sql.types'`.
    - When `pyspark.sql.pandas.functions` was imported first:
      1. it imported `pyspark.sql.functions` before it defined the enums;
      2. the re-export then met the half-imported module, and its `ImportError` 
was swallowed;
      3. a later `from pyspark.sql.functions import PandasUDFType` raised 
`ImportError`.
    - `from pyspark.sql.pandas.functions import arrow_udf` raised `ImportError`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, these imports work as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - With each of the 96 modules under `pyspark.sql` imported first, `from 
pyspark.sql.functions import PandasUDFType, ArrowUDFType` and `import 
pyspark.sql.conversion` work. Eleven modules do not import on their own, before 
or after this change:
      - the server-side `pyspark.sql.worker` modules;
      - the internal `functions_generated`;
      - the classic `pyspark.sql.pandas.conversion`.
    - A probe of `pyspark.sql.conversion` on the client's types gives the same 
output as official PySpark 4.3.0: Arrow schema, values, the round trip of rows, 
`_has_type` and the pandas conversion.
    - New offline tests:
      - the re-exports, in a new interpreter that imports 
`pyspark.sql.pandas.functions` first;
      - `_has_type`;
      - the conversion of local rows to Arrow and back.
    
      They fail without this change and pass with official PySpark 4.3.0.
    - The lib tests and every integration test, e2e included, against a local 
Spark 4.3.0 server (643 passed), the build of all tests with `wasm-udf`, the 
offline Python suites (616 passed, 8 live cases skipped), and the live drop-in 
and Arrow suites (57 passed) and live cases (10 passed).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #179 from HyukjinKwon/SPARK-60957-imports.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/functions.py        |  5 +---
 python/pyspark/sql/pandas/functions.py |  8 ++++--
 python/pyspark/sql/types.py            | 16 +++++++++++
 python/tests/test_dropin_offline.py    | 49 ++++++++++++++++++++++++++++++++++
 4 files changed, 72 insertions(+), 6 deletions(-)

diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index 0e75123..e8783c2 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -742,10 +742,7 @@ try:
     )
 except Exception:  # pragma: no cover
     pass
-try:
-    from pyspark.sql.pandas.functions import PandasUDFType, ArrowUDFType  # 
noqa: F401
-except Exception:  # pragma: no cover
-    pass
+from pyspark.sql.pandas.functions import PandasUDFType, ArrowUDFType  # noqa: 
E402,F401
 
 # Build __all__ with all function names
 __all__ = [
diff --git a/python/pyspark/sql/pandas/functions.py 
b/python/pyspark/sql/pandas/functions.py
index 289f0b1..d7d67de 100644
--- a/python/pyspark/sql/pandas/functions.py
+++ b/python/pyspark/sql/pandas/functions.py
@@ -19,7 +19,7 @@
 Canonical home of the ``PandasUDFType`` / ``ArrowUDFType`` enums (mirroring 
reference
 pyspark); ``pyspark.sql.functions`` re-exports them from here.
 """
-from pyspark.sql.functions import pandas_udf  # noqa: F401
+from pyspark.util import PythonEvalType  # noqa: F401  
(pyspark.sql.pandas.typehints reads it here)
 
 
 class PandasUDFType:
@@ -43,4 +43,8 @@ class ArrowUDFType:
     GROUPED_AGG_ITER = 254
 
 
-__all__ = ["pandas_udf", "PandasUDFType", "ArrowUDFType"]
+# After the enums: importing this module first imports pyspark.sql.functions, 
whose
+# re-export of the enums then finds them already defined here.
+from pyspark.sql.functions import arrow_udf, pandas_udf  # noqa: E402,F401
+
+__all__ = ["arrow_udf", "pandas_udf", "PandasUDFType", "ArrowUDFType"]
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index c615d23..83a584d 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -347,6 +347,22 @@ def _drop_metadata(d):
         return MapType(_drop_metadata(d.keyType), _drop_metadata(d.valueType), 
d.valueContainsNull)
     return d
 
+def _has_type(dt, dts):
+    """Return whether there are specified types (mirrors 
pyspark.sql.types._has_type)."""
+    if isinstance(dt, dts):
+        return True
+    elif isinstance(dt, StructType):
+        return any(_has_type(f.dataType, dts) for f in dt.fields)
+    elif isinstance(dt, ArrayType):
+        return _has_type(dt.elementType, dts)
+    elif isinstance(dt, MapType):
+        return _has_type(dt.keyType, dts) or _has_type(dt.valueType, dts)
+    elif isinstance(dt, UserDefinedType):
+        return _has_type(dt.sqlType(), dts)
+    else:
+        return False
+
+
 def _create_row(fields, values):
     """A Row of ``values`` named by ``fields``, as in PySpark, whose pickled 
rows call it."""
     row = Row(*values)
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 7a3f10f..deabc98 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -3297,6 +3297,55 @@ def test_arrow_udf_type_enum_values():
     assert PandasUDFType.GROUPED_AGG_ITER == 217
 
 
+def test_udf_types_are_re_exported_whichever_module_is_imported_first():
+    # A new interpreter, as this one has imported pyspark.sql.functions 
already.
+    import subprocess
+    import sys
+
+    subprocess.run([sys.executable, "-c", """
+import pyspark.sql.pandas.functions as PF
+from pyspark.sql.functions import ArrowUDFType, PandasUDFType, arrow_udf, 
pandas_udf
+assert (PandasUDFType, ArrowUDFType) == (PF.PandasUDFType, PF.ArrowUDFType)
+assert (arrow_udf, pandas_udf) == (PF.arrow_udf, PF.pandas_udf)
+from pyspark.util import PythonEvalType
+assert PF.PythonEvalType is PythonEvalType
+"""], check=True)
+
+
+def test_has_type_looks_into_nested_and_user_defined_types():
+    from pyspark.sql.types import _has_type
+
+    nested = T.StructType([T.StructField("m", T.MapType(T.StringType(), 
T.ArrayType(T.NullType())))])
+    assert _has_type(nested, T.NullType)
+    assert _has_type(nested, (T.DecimalType, T.ArrayType))
+    assert not _has_type(nested, T.DecimalType)
+    # A user-defined type is looked into by its sqlType: an array of doubles.
+    assert _has_type(T.StructType([T.StructField("p", _PointUDT())]), 
T.DoubleType)
+
+
+def test_conversion_converts_local_rows_to_arrow_and_back():
+    import decimal
+
+    from pyspark.sql import Row
+    from pyspark.sql.conversion import ArrowTableToRowsConversion, 
LocalDataToArrowConversion
+
+    schema = T.StructType([
+        T.StructField("id", T.LongType()),
+        T.StructField("a", T.ArrayType(T.DoubleType())),
+        T.StructField("m", T.MapType(T.StringType(), T.LongType())),
+        T.StructField("st", T.StructType([T.StructField("x", T.LongType())])),
+        T.StructField("d", T.DecimalType(10, 2)),
+    ])
+    rows = [(1, [1.0, None], {"k": 1}, (5,), decimal.Decimal("1.25")), (2, 
None, None, None, None)]
+    table = LocalDataToArrowConversion.convert(rows, schema, 
use_large_var_types=False)
+    assert table.to_pylist()[0] == {
+        "id": 1, "a": [1.0, None], "m": [("k", 1)], "st": {"x": 5}, "d": 
decimal.Decimal("1.25")}
+    assert ArrowTableToRowsConversion.convert(table, schema, 
return_as_tuples=True) == [
+        (1, [1.0, None], {"k": 1}, Row(x=5), decimal.Decimal("1.25")),
+        (2, None, None, None, None),
+    ]
+
+
 def test_datatype_class_constants():
     assert (T.DayTimeIntervalType.DAY, T.DayTimeIntervalType.HOUR) == (0, 1)
     assert (T.DayTimeIntervalType.MINUTE, T.DayTimeIntervalType.SECOND) == (2, 
3)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to