This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new f2a52ca  [SPARK-61095][CONNECT][PYTHON] Add the modules geo_utils, 
sql_formatter, table_arg and tvf_argument of pyspark.sql in the Python client 
as the PySpark client has them
f2a52ca is described below

commit f2a52cafefbf4398d15a5606817bc82f8b6af3ee
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 16:57:00 2026 +0900

    [SPARK-61095][CONNECT][PYTHON] Add the modules geo_utils, sql_formatter, 
table_arg and tvf_argument of pyspark.sql in the Python client as the PySpark 
client has them
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61095](https://issues.apache.org/jira/browse/SPARK-61095): 
`pyspark.sql` now has the modules `geo_utils`, `sql_formatter`, `table_arg` and 
`tvf_argument`, as the PySpark client's `pyspark.sql` package does.
    
    - `geo_utils`, the spatial reference systems of the spatial types, and 
`sql_formatter`, `SQLStringFormatter`, are vendored from Apache Spark v4.3.0. 
The drift check covers them.
    - `tvf_argument` has `TableValuedFunctionArgument`. A `Column` and a 
`TableArg` are instances of it: the drop-in's are Rust-backed classes, so they 
are registered as its virtual subclasses.
    - `table_arg` has `TableArg`, the Rust-backed class of 
`DataFrame.asTable()`.
    - As PySpark's, `import pyspark.sql` imports its submodules `conf`, 
`geo_utils`, `profiler`, `sql_formatter`, `streaming`, `table_arg`, 
`tvf_argument` and `variant_utils`, so they are attributes of `pyspark.sql`.
    
    ### Why are the changes needed?
    
    The PySpark client ships these four modules in its `pyspark.sql` package 
(`connect_packages`), but the drop-in lacked them: `from pyspark.sql.table_arg 
import TableArg` raised a `ModuleNotFoundError`. Also, `pyspark.sql.streaming` 
and the other submodules were not attributes of `pyspark.sql` until they were 
imported themselves.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These modules, and the attributes of `pyspark.sql`, are those of the 
PySpark 4.3.0 client.
    
    ### How was this patch tested?
    
    - A new live test checks that `DataFrame.asTable()` and its `partitionBy` 
are `TableArg`s of `pyspark.sql.table_arg`, and `TableValuedFunctionArgument`s. 
It passes with official PySpark 4.3.0 and fails without this change.
    - A new offline test checks the four modules, the attributes of 
`pyspark.sql`, and that a `Column` and a `TableArg` are 
`TableValuedFunctionArgument`s.
    - A comparison of the names of 23 modules with official PySpark 4.3.0 found 
these modules missing. The other names that differ come from the imports and 
type aliases inside PySpark's modules, or are classic APIs.
    - The lib tests of both crates (595 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2047 passed, 12 skipped), and the 
live drop-in and Arrow suites (151 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #300 from HyukjinKwon/SPARK-61095-sql-modules.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/__init__.py      |  13 +++++
 python/pyspark/sql/geo_utils.py     | 103 ++++++++++++++++++++++++++++++++++++
 python/pyspark/sql/sql_formatter.py |  86 ++++++++++++++++++++++++++++++
 python/pyspark/sql/table_arg.py     |  23 ++++++++
 python/pyspark/sql/tvf_argument.py  |  40 ++++++++++++++
 python/tests/test_dropin_e2e.py     |  14 +++++
 python/tests/test_dropin_offline.py |  25 +++++++++
 scripts/check_vendored_upstream.sh  |   5 ++
 8 files changed, 309 insertions(+)

diff --git a/python/pyspark/sql/__init__.py b/python/pyspark/sql/__init__.py
index a97ef93..9c11a7a 100644
--- a/python/pyspark/sql/__init__.py
+++ b/python/pyspark/sql/__init__.py
@@ -91,3 +91,16 @@ if Geography is not None:
     __all__.append("Geography")
 if Geometry is not None:
     __all__.append("Geometry")
+
+# As PySpark's, whose modules of pyspark.sql import these submodules, so that 
they are
+# attributes of pyspark.sql.
+from pyspark.sql import (  # noqa: E402,F401
+    conf,
+    geo_utils,
+    profiler,
+    sql_formatter,
+    streaming,
+    table_arg,
+    tvf_argument,
+    variant_utils,
+)
diff --git a/python/pyspark/sql/geo_utils.py b/python/pyspark/sql/geo_utils.py
new file mode 100644
index 0000000..904f82a
--- /dev/null
+++ b/python/pyspark/sql/geo_utils.py
@@ -0,0 +1,103 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Dict, Optional
+
+
+# Class for maintaining information about a spatial reference system (SRS).
+@dataclass(frozen=True)
+class SpatialReferenceSystemInformation:
+    # Field storing the spatial reference identifier (SRID) value of this SRS.
+    srid: int
+    # Field storing the string ID of the corresponding coordinate reference 
system (CRS).
+    string_id: str
+    # Field indicating whether the spatial reference system (SRS) is 
geographic or not.
+    is_geographic: bool
+
+
+# Class for maintaining the mappings between supported SRID/CRS values and the 
corresponding SRS.
+class SpatialReferenceSystemCache:
+    # We use a singleton pattern, which is aligned with the JVM side / Scala 
implementation.
+    _instance: Optional["SpatialReferenceSystemCache"] = None
+
+    @classmethod
+    def instance(cls) -> "SpatialReferenceSystemCache":
+        if cls._instance is None:
+            cls._instance = SpatialReferenceSystemCache()
+        return cls._instance
+
+    def __init__(self) -> None:
+        # Hash map for defining the mappings from the integer SRID to the full 
SRS information.
+        self._srid_to_srs: Dict[int, SpatialReferenceSystemInformation] = {}
+        # Hash map for defining the mappings from the string ID to the full 
SRS information.
+        self._string_id_to_srs: Dict[str, SpatialReferenceSystemInformation] = 
{}
+        self._populate_srs_information_mapping()
+
+    # Helper method for building the SRID-to-SRS and stringID-to-SRS mappings.
+    def _populate_srs_information_mapping(self) -> None:
+        # Currently, we only support a limited set of SRID / CRS values, even 
on Scala side. The
+        # SRS list below will be updated soon, and the maps will be populated 
with more SRS data.
+        srs_list = [
+            SpatialReferenceSystemInformation(0, "SRID:0", False),
+            SpatialReferenceSystemInformation(3857, "EPSG:3857", False),
+            SpatialReferenceSystemInformation(4326, "OGC:CRS84", True),
+        ]
+        # Populate the mappings using the same SRS information objects, 
avoiding any duplication.
+        for srs in srs_list:
+            self._srid_to_srs[srs.srid] = srs
+            self._string_id_to_srs[srs.string_id] = srs
+
+    # Returns the SRS corresponding to the input SRID. If not supported, 
returns `None`.
+    def get_srs_by_srid(self, srid: int) -> 
Optional[SpatialReferenceSystemInformation]:
+        return self._srid_to_srs.get(srid)
+
+    # Returns the SRS corresponding to the input string ID. If not supported, 
returns `None`.
+    def get_srs_by_string_id(self, string_id: str) -> 
Optional[SpatialReferenceSystemInformation]:
+        return self._string_id_to_srs.get(string_id)
+
+
+# Class for providing SRS mappings for geographic spatial reference systems.
+class GeographicSpatialReferenceSystemMapper:
+    # Returns the string ID corresponding to the input SRID. If not supported, 
returns `None`.
+    @staticmethod
+    def get_string_id(srid: int) -> Optional[str]:
+        srs = SpatialReferenceSystemCache.instance().get_srs_by_srid(srid)
+        return srs.string_id if srs is not None and srs.is_geographic else None
+
+    # Returns the SRID corresponding to the input string ID. If not supported, 
returns `None`.
+    @staticmethod
+    def get_srid(string_id: str) -> Optional[int]:
+        srs = 
SpatialReferenceSystemCache.instance().get_srs_by_string_id(string_id)
+        return srs.srid if srs is not None and srs.is_geographic else None
+
+
+# Class for providing SRS mappings for cartesian spatial reference systems.
+class CartesianSpatialReferenceSystemMapper:
+    # Returns the string ID corresponding to the input SRID. If not supported, 
returns `None`.
+    @staticmethod
+    def get_string_id(srid: int) -> Optional[str]:
+        srs = SpatialReferenceSystemCache.instance().get_srs_by_srid(srid)
+        return srs.string_id if srs is not None else None
+
+    # Returns the SRID corresponding to the input string ID. If not supported, 
returns `None`.
+    @staticmethod
+    def get_srid(string_id: str) -> Optional[int]:
+        srs = 
SpatialReferenceSystemCache.instance().get_srs_by_string_id(string_id)
+        return srs.srid if srs is not None else None
diff --git a/python/pyspark/sql/sql_formatter.py 
b/python/pyspark/sql/sql_formatter.py
new file mode 100644
index 0000000..1366ef2
--- /dev/null
+++ b/python/pyspark/sql/sql_formatter.py
@@ -0,0 +1,86 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import string
+import typing
+from typing import Any, Optional, List, Tuple, Sequence, Mapping
+import uuid
+
+if typing.TYPE_CHECKING:
+    from pyspark.sql import SparkSession, DataFrame
+from pyspark.sql.utils import get_lit_sql_str
+from pyspark.errors import PySparkValueError
+
+
+class SQLStringFormatter(string.Formatter):
+    """
+    A standard ``string.Formatter`` in Python that can understand PySpark 
instances
+    with basic Python objects. This object has to be clear after the use for 
single SQL
+    query; cannot be reused across multiple SQL queries without cleaning.
+    """
+
+    def __init__(self, session: "SparkSession") -> None:
+        self._session: "SparkSession" = session
+        self._temp_views: List[Tuple[DataFrame, str]] = []
+
+    def get_field(self, field_name: str, args: Sequence[Any], kwargs: 
Mapping[str, Any]) -> Any:
+        obj, first = super().get_field(field_name, args, kwargs)
+        return self._convert_value(obj, field_name), first
+
+    def _convert_value(self, val: Any, field_name: str) -> Optional[str]:
+        """
+        Converts the given value into a SQL string.
+        """
+        from py4j.java_gateway import is_instance_of
+
+        from pyspark import SparkContext
+        from pyspark.sql import Column, DataFrame, SparkSession
+
+        if isinstance(val, Column):
+            jsession = SparkSession.active()._jsparkSession
+            jexpr = jsession.expression(val._jc)
+
+            assert SparkContext._gateway is not None
+            gw = SparkContext._gateway
+            if is_instance_of(
+                gw, jexpr, 
"org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute"
+            ) or is_instance_of(
+                gw, jexpr, 
"org.apache.spark.sql.catalyst.expressions.AttributeReference"
+            ):
+                return jexpr.sql()
+            else:
+                raise PySparkValueError(
+                    errorClass="VALUE_NOT_PLAIN_COLUMN_REFERENCE",
+                    messageParameters={"val": str(val), "field_name": 
field_name},
+                )
+        elif isinstance(val, DataFrame):
+            for df, n in self._temp_views:
+                if df is val:
+                    return n
+            df_name = "_pyspark_%s" % str(uuid.uuid4()).replace("-", "")
+            self._temp_views.append((val, df_name))
+            val.createOrReplaceTempView(df_name)
+            return df_name
+        elif isinstance(val, str):
+            return get_lit_sql_str(val)
+        else:
+            return val
+
+    def clear(self) -> None:
+        for _, n in self._temp_views:
+            self._session.catalog.dropTempView(n)
+        self._temp_views = []
diff --git a/python/pyspark/sql/table_arg.py b/python/pyspark/sql/table_arg.py
new file mode 100644
index 0000000..8f4419b
--- /dev/null
+++ b/python/pyspark/sql/table_arg.py
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""``TableArg`` of PySpark, a DataFrame as a table argument of a table-valued 
function, of
+``DataFrame.asTable()``: the drop-in's is Rust-backed, and a 
``TableValuedFunctionArgument``."""
+
+from pyspark._pyspark import TableArg
+from pyspark.sql.tvf_argument import TableValuedFunctionArgument  # noqa: F401
+
+__all__ = ["TableArg"]
diff --git a/python/pyspark/sql/tvf_argument.py 
b/python/pyspark/sql/tvf_argument.py
new file mode 100644
index 0000000..d437ce0
--- /dev/null
+++ b/python/pyspark/sql/tvf_argument.py
@@ -0,0 +1,40 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""``TableValuedFunctionArgument`` of PySpark, the base of the arguments of 
table-valued
+functions, of which a ``Column`` and a ``TableArg`` are instances. The 
drop-in's are Rust-backed
+classes, so they are its virtual subclasses."""
+
+import abc
+
+
+class TableValuedFunctionArgument(abc.ABC):
+    """
+    Base class for arguments passed to Table-Valued Functions.
+
+    This class is intentionally left empty and serves as a marker to achieve
+    parity with the Scala `TableValuedFunctionArgument` trait.
+    """
+
+    pass
+
+
+from pyspark._pyspark import Column as _Column, TableArg as _TableArg  # noqa: 
E402
+
+TableValuedFunctionArgument.register(_Column)
+TableValuedFunctionArgument.register(_TableArg)
+
+__all__ = ["TableValuedFunctionArgument"]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index c01428f..3ec0338 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1161,6 +1161,20 @@ def 
test_reader_writer_and_catalog_return_the_results_of_pyspark(session, tmp_pa
         session.sql("DROP TABLE IF EXISTS e2e_external_eagerly")
 
 
+def test_table_arguments_are_those_of_pyspark_sql_table_arg(session):
+    # As the PySpark client: DataFrame.asTable() is a TableArg of 
pyspark.sql.table_arg, and a
+    # TableValuedFunctionArgument of pyspark.sql.tvf_argument.
+    from pyspark.sql.table_arg import TableArg
+    from pyspark.sql.tvf_argument import TableValuedFunctionArgument
+
+    table = session.range(3).asTable()
+    assert isinstance(table, TableArg)
+    assert isinstance(table, TableValuedFunctionArgument)
+    assert isinstance(table.partitionBy("id"), TableArg)
+
+
+
+
 def test_subqueries_and_table_arguments_return_the_rows_of_pyspark(session):
     # As PySpark's DataFrame.scalar(), DataFrame.exists(), Column.isin(df) and 
asTable():
     # scalar, correlated and IN subqueries, and the rows of a DataFrame as a 
UDTF argument.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 4cb9ed5..abe705c 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -821,6 +821,31 @@ def test_functions_udtf_and_arrow_decorators():
 
 # --------------------------------------------------------------------------- 
functions / columns
 
+def test_sql_modules_are_those_of_the_pyspark_client():
+    # As the PySpark client: pyspark.sql has the modules geo_utils, 
sql_formatter, table_arg and
+    # tvf_argument, of which a Column and a TableArg are 
TableValuedFunctionArguments, and
+    # `import pyspark.sql` imports its submodules of conf, profiler, streaming 
and variant_utils.
+    import pyspark.sql
+    from pyspark.sql import functions as F
+    from pyspark.sql.geo_utils import (
+        CartesianSpatialReferenceSystemMapper,
+        GeographicSpatialReferenceSystemMapper,
+    )
+    from pyspark.sql.sql_formatter import SQLStringFormatter
+    from pyspark.sql.table_arg import TableArg
+    from pyspark.sql.tvf_argument import TableValuedFunctionArgument
+
+    for name in ("conf", "geo_utils", "profiler", "sql_formatter", 
"streaming", "table_arg",
+                 "tvf_argument", "variant_utils"):
+        assert hasattr(pyspark.sql, name), name
+    assert isinstance(F.col("a"), TableValuedFunctionArgument)
+    assert issubclass(TableArg, TableValuedFunctionArgument)
+    assert CartesianSpatialReferenceSystemMapper and 
GeographicSpatialReferenceSystemMapper
+    assert SQLStringFormatter.__module__ == "pyspark.sql.sql_formatter"
+
+
+
+
 def test_functions_exports_fresh_process():
     import subprocess
     import sys
diff --git a/scripts/check_vendored_upstream.sh 
b/scripts/check_vendored_upstream.sh
index 84cff3c..4c7410c 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -89,6 +89,11 @@ VENDORED_FILES=(
   # ExecutePlanResponse is that of the message class of 
pyspark/sql/connect/proto.py.
   "sql/connect/shell/progress.py"
 
+  # The spatial reference systems of the spatial types, and the 
SQLStringFormatter, of
+  # pyspark.sql.
+  "sql/geo_utils.py"
+  "sql/sql_formatter.py"
+
   # Top-level modules
   "accumulators.py"
   "memory_profiler_ext.py"


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to