This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new eb79c3d  [SPARK-61116][CONNECT][PYTHON] Port the Read, Range and SQL 
plans, Column.to_plan, and the type and conversion helpers of the connect 
import path, in the Python client as PySpark does
eb79c3d is described below

commit eb79c3d4670ec2dc522ab7c34ec4773b772bb1e6
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 23:20:00 2026 +0900

    [SPARK-61116][CONNECT][PYTHON] Port the Read, Range and SQL plans, 
Column.to_plan, and the type and conversion helpers of the connect import path, 
in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61116](https://issues.apache.org/jira/browse/SPARK-61116): The rest 
of the connect import path that builds or reads messages is PySpark's. It is 
ported from Apache Spark v4.3.0 over the generated `pyspark.sql.connect.proto` 
(SPARK-61111):
    
    - `pyspark.sql.connect.plan`:
      - `Read`, `Range` and `SQL` replace their placeholders, whose 
constructors took any arguments and built no relation.
      - `SubqueryAlias`, of the views of `SQL`.
      - `CachedRemoteRelation`, which removes its relation from the server once 
it is collected, through the client.
      - `RemoveRemoteCachedRelation`.
    
      `pyspark.testing.connectutils.PlanOnlyTestFixture` builds DataFrames of 
these plans with `DataFrame(plan, session)`.
    - `Column.to_plan(session)`: the `spark.connect.Expression` of a column. It 
was the last justified non-gap of the introspection diff.
    - `pyspark.sql.connect.types`: `parse_attr_name`, `verify_col_name` and 
`verify_numeric_col_name`.
    - `pyspark.sql.connect.conversion`, vendored byte for byte: 
`storage_level_to_proto`, `proto_to_storage_level` and 
`proto_to_remote_cached_dataframe`.
    - `SparkConnectClient.is_closed`: whether the session is stopped.
    
    `AGENTS.md` and `dev/parity/4.3.0-introspection-diff.md` now record no 
justified non-gaps.
    
    ### Why are the changes needed?
    
    - `DataFrame(Range(0, 3, 1), spark)` failed: `Range` built no relation.
    - `Column.to_plan` was missing.
    - `pyspark.sql.connect.conversion` and the helpers of 
`pyspark.sql.connect.types` could not be imported.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These plans, functions and `Column.to_plan` exist as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - New offline tests check:
      - the relations of the plans, which are those of official PySpark 4.3.0 
apart from plan IDs, and their plan IDs
      - the command and the checks of `SQL`, and `Read.print`
      - `Column.to_plan`
      - the type helpers
      - the storage level conversions, and the DataFrame of a cached relation, 
which removes the relation from a mock Connect server once it is collected
    - A new live test runs DataFrames of `Range`, `SQL` with positional and 
named arguments, `SQL` with views, and `Read` on a server. It passes with 
official PySpark 4.3.0.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2174 passed, 8 skipped), and the 
live drop-in and Arrow suites (173 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #314 from HyukjinKwon/SPARK-61116-connect-plans.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 AGENTS.md                                     |   5 +-
 crates/pyspark-rs/src/column.rs               |   6 +
 dev/parity/4.3.0-introspection-diff.md        |  11 +-
 python/.coveragerc                            |   1 +
 python/pyspark/sql/connect/client/__init__.py |   6 +
 python/pyspark/sql/connect/conversion.py      |  59 ++++++++++
 python/pyspark/sql/connect/plan.py            | 163 +++++++++++++++++++++++---
 python/pyspark/sql/connect/types.py           | 106 ++++++++++++++++-
 python/tests/test_ml_e2e.py                   |  19 +++
 python/tests/test_ml_offline.py               | 126 ++++++++++++++++++++
 scripts/check_vendored_upstream.sh            |   2 +
 11 files changed, 478 insertions(+), 26 deletions(-)

diff --git a/AGENTS.md b/AGENTS.md
index 0842fb7..07d0be7 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -129,9 +129,8 @@ Cover these classes: `DataFrame`, `Column`, `SparkSession`, 
`SparkSession.Builde
 `Catalog`, `Window`, `WindowSpec`, `Observation` (add any new connect classes 
the
 new version introduces).
 
-**Justified non-gaps** (absent on purpose because we use a native Rust 
transport,
-not the Python gRPC client): `Column.to_plan(session)`. Anything else in the 
diff is a
-real gap to close. Re-run the diff until it shows only these.
+There are no justified non-gaps: anything in the diff is a real gap to close. 
Re-run the
+diff until it shows none.
 
 `SparkSession.client` is not the gRPC `SparkConnectClient`: it is the client 
that the
 vendored `pyspark.ml` (and any module that builds its own `spark.connect` 
messages) uses
diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index 3ea4699..01fb5df 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -673,6 +673,12 @@ impl PyColumn {
         }
     }
 
+    /// The `spark.connect.Expression` of this column, as PySpark Connect's 
`Column.to_plan`: that
+    /// of its expression, of the generated `pyspark.sql.connect.proto`.
+    fn to_plan(&self, py: Python<'_>, session: &Bound<'_, PyAny>) -> 
PyResult<Py<PyAny>> {
+        self._expr().to_plan(py, session)
+    }
+
     fn __str__(&self) -> String {
         let _origin = crate::origin::with_origin("__str__");
         self.__repr__()
diff --git a/dev/parity/4.3.0-introspection-diff.md 
b/dev/parity/4.3.0-introspection-diff.md
index f8f0fa3..1512008 100644
--- a/dev/parity/4.3.0-introspection-diff.md
+++ b/dev/parity/4.3.0-introspection-diff.md
@@ -14,7 +14,7 @@ Re-run against the 4.3.0 release: the results are the same as 
against RC1 and RC
 | Class | ref | ours | missing |
 |---|---|---|---|
 | DataFrame | 118 | 119 | 0 |
-| Column | 37 | 36 | 0 (justified: `to_plan`) |
+| Column | 37 | 37 | 0 |
 | SparkSession | 40 | 42 | 0 |
 | SparkSession.Builder | 8 | 8 | 0 |
 | DataFrameReader | 14 | 14 | 0 |
@@ -26,10 +26,11 @@ Re-run against the 4.3.0 release: the results are the same 
as against RC1 and RC
 | WindowSpec | 4 | 4 | 0 |
 | Observation | 1 | 2 | 0 |
 
-Justified non-gaps (native Rust transport, not the Python gRPC client): 
`Column.to_plan`.
-`SparkSession.client` is the client that `pyspark.ml` uses over the Rust 
transport
-(`pyspark.sql.connect.client.SparkConnectClient`: `execute_command` and the ML 
cache), with
-the generated `pyspark.sql.connect.proto` (SPARK-61111).
+There are no justified non-gaps. `SparkSession.client` is the client that 
`pyspark.ml` uses
+over the Rust transport (`pyspark.sql.connect.client.SparkConnectClient`: 
`execute_command`
+and the ML cache), with the generated `pyspark.sql.connect.proto` 
(SPARK-61111), and
+`Column.to_plan` is the `spark.connect.Expression` of a column, of those 
messages
+(SPARK-61116).
 
 ## `SparkSession.Hook` (new in 4.3.0) — implemented
 
diff --git a/python/.coveragerc b/python/.coveragerc
index ffd1a5a..284e3c6 100644
--- a/python/.coveragerc
+++ b/python/.coveragerc
@@ -61,6 +61,7 @@ omit =
     # The logger and the types of the connect import path, vendored.
     */pyspark/sql/connect/logging.py
     */pyspark/sql/connect/_typing.py
+    */pyspark/sql/connect/conversion.py
     */pyspark/_pyspark*
     */tests/*
 
diff --git a/python/pyspark/sql/connect/client/__init__.py 
b/python/pyspark/sql/connect/client/__init__.py
index 5dca376..3701fcb 100644
--- a/python/pyspark/sql/connect/client/__init__.py
+++ b/python/pyspark/sql/connect/client/__init__.py
@@ -139,6 +139,12 @@ class SparkConnectClient:
     def _server_session_id(self) -> str:
         return self._session.session_id
 
+    @property
+    def is_closed(self) -> bool:
+        """Whether the session of the client is stopped, as 
SparkConnectClient's ``is_closed``,
+        whether its channel is closed."""
+        return self._session.is_stopped
+
     def execute_command(self, command: Any, observations: Any = None) -> Any:
         """Execute ``command``, a ``spark.connect.Command``, as 
SparkConnectClient's
         ``execute_command``: no data, the properties of the results of the 
responses, of their
diff --git a/python/pyspark/sql/connect/conversion.py 
b/python/pyspark/sql/connect/conversion.py
new file mode 100644
index 0000000..137b045
--- /dev/null
+++ b/python/pyspark/sql/connect/conversion.py
@@ -0,0 +1,59 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from typing import TYPE_CHECKING
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.storagelevel import StorageLevel
+
+if TYPE_CHECKING:
+    from pyspark.sql.connect.dataframe import DataFrame
+
+
+def storage_level_to_proto(storage_level: StorageLevel) -> pb2.StorageLevel:
+    assert storage_level is not None and isinstance(storage_level, 
StorageLevel)
+    return pb2.StorageLevel(
+        use_disk=storage_level.useDisk,
+        use_memory=storage_level.useMemory,
+        use_off_heap=storage_level.useOffHeap,
+        deserialized=storage_level.deserialized,
+        replication=storage_level.replication,
+    )
+
+
+def proto_to_storage_level(storage_level: pb2.StorageLevel) -> StorageLevel:
+    assert storage_level is not None and isinstance(storage_level, 
pb2.StorageLevel)
+    return StorageLevel(
+        useDisk=storage_level.use_disk,
+        useMemory=storage_level.use_memory,
+        useOffHeap=storage_level.use_off_heap,
+        deserialized=storage_level.deserialized,
+        replication=storage_level.replication,
+    )
+
+
+def proto_to_remote_cached_dataframe(relation: pb2.CachedRemoteRelation) -> 
"DataFrame":
+    assert relation is not None and isinstance(relation, 
pb2.CachedRemoteRelation)
+
+    from pyspark.sql.connect.dataframe import DataFrame
+    from pyspark.sql.connect.session import SparkSession
+    import pyspark.sql.connect.plan as plan
+
+    session = SparkSession.active()
+    return DataFrame(
+        plan=plan.CachedRemoteRelation(relation.relation_id, session),
+        session=session,
+    )
diff --git a/python/pyspark/sql/connect/plan.py 
b/python/pyspark/sql/connect/plan.py
index 990125e..9e2ccc1 100644
--- a/python/pyspark/sql/connect/plan.py
+++ b/python/pyspark/sql/connect/plan.py
@@ -22,15 +22,19 @@ those of ``pyspark.ml.connect.proto``: a plan of a child, 
of a ``plan(session)``
 ``spark.connect.Relation``, which ``DataFrame(plan, session)`` sends as it is. 
It has no
 ``to_proto``, ``_collect_references``, ``print`` or ``_repr_html_``, which are 
of the plans and the
 Columns of PySpark Connect's DataFrame, which the core builds. 
``_DataFramePlan`` is the plan of a
-DataFrame of the core, its ``DataFrame._plan``. ``Read``, ``Range`` and 
``SQL`` are not plans of
-this client. The messages are those of the generated 
``pyspark.sql.connect.proto``, which needs
-the ``protobuf`` package.
+DataFrame of the core, its ``DataFrame._plan``. ``Read``, ``Range``, ``SQL``, 
``SubqueryAlias``,
+``CachedRemoteRelation`` and ``RemoveRemoteCachedRelation`` are PySpark 
Connect's, which
+``pyspark.testing.connectutils`` and ``pyspark.sql.connect.conversion`` build 
DataFrames of. The
+messages are those of the generated ``pyspark.sql.connect.proto``, which needs 
the ``protobuf``
+package.
 """
 
 from threading import Lock
-from typing import Any, Dict, Optional, Sequence
+from typing import Any, Dict, List, Optional, Sequence
 
 import pyspark.sql.connect.proto as proto
+from pyspark.sql.column import Column
+from pyspark.sql.connect.logging import logger
 
 
 class LogicalPlan:
@@ -135,24 +139,153 @@ class _DataFramePlan(LogicalPlan):
 
 
 class Read(LogicalPlan):
-    """Not a plan of this client: the Rust core reads tables, as 
``spark.read.table``."""
+    def __init__(
+        self,
+        table_name: str,
+        options: Optional[Dict[str, str]] = None,
+        is_streaming: Optional[bool] = None,
+    ) -> None:
+        super().__init__(None)
+        self.table_name = table_name
+        self.options = options or {}
+        self._is_streaming = is_streaming
+
+    def plan(self, session: Any) -> "proto.Relation":
+        plan = self._create_proto_relation()
+        plan.read.named_table.unparsed_identifier = self.table_name
+        if self._is_streaming is not None:
+            plan.read.is_streaming = self._is_streaming
+        for k, v in self.options.items():
+            plan.read.named_table.options[k] = v
+        return plan
 
-    def __init__(self, *args: Any, **kwargs: Any) -> None:
-        pass
+    def print(self, indent: int = 0) -> str:
+        return f"{' ' * indent}<Read table_name={self.table_name}>\n"
 
 
-class Range(LogicalPlan):
-    """Not a plan of this client: the Rust core builds ranges, as 
``spark.range``."""
+class CachedRemoteRelation(LogicalPlan):
+    """Logical plan object for a DataFrame reference which represents a 
DataFrame that's been
+    cached on the server with a given id."""
 
-    def __init__(self, *args: Any, **kwargs: Any) -> None:
-        pass
+    def __init__(self, relation_id: str, spark_session: Any):
+        super().__init__(None)
+        self._relation_id = relation_id
+        # Needs to hold the session to make a request itself.
+        self._spark_session = spark_session
+
+    def plan(self, session: Any) -> "proto.Relation":
+        plan = self._create_proto_relation()
+        plan.cached_remote_relation.relation_id = self._relation_id
+        return plan
+
+    def __del__(self) -> None:
+        session = self._spark_session
+        # If session is already closed, all cached DataFrame should be 
released.
+        if session is not None and not session.client.is_closed and 
self._relation_id is not None:
+            try:
+                # The command of the Rust transport, as PySpark Connect's 
sends it.
+                command = 
RemoveRemoteCachedRelation(self).command(session=session.client)
+                session.client.execute_command(command)
+            except Exception as e:
+                logger.warning(f"RemoveRemoteCachedRelation failed with 
exception: {e}.")
+
+
+class SubqueryAlias(LogicalPlan):
+    """Alias for a relation."""
+
+    def __init__(self, child: Optional["LogicalPlan"], alias: str) -> None:
+        super().__init__(child)
+        self._alias = alias
+
+    def plan(self, session: Any) -> "proto.Relation":
+        plan = self._create_proto_relation()
+        if self._child is not None:
+            plan.subquery_alias.input.CopyFrom(self._child.plan(session))
+        plan.subquery_alias.alias = self._alias
+        return plan
 
 
 class SQL(LogicalPlan):
-    """Not a plan of this client: the Rust core builds queries, as 
``spark.sql``."""
+    def __init__(
+        self,
+        query: str,
+        args: Optional[List[Column]] = None,
+        named_args: Optional[Dict[str, Column]] = None,
+        views: Optional[Sequence[SubqueryAlias]] = None,
+    ) -> None:
+        if args is not None:
+            assert isinstance(args, List)
+            assert all(isinstance(arg, Column) for arg in args)
+
+        if named_args is not None:
+            assert isinstance(named_args, Dict)
+            for k, arg in named_args.items():
+                assert isinstance(k, str)
+                assert isinstance(arg, Column)
+
+        if views is not None:
+            assert isinstance(views, List)
+            assert all(isinstance(v, SubqueryAlias) for v in views)
+
+        super().__init__(None, views)
+        self._query = query
+        self._args = args
+        self._named_args = named_args
+        self._views = views
+
+    def plan(self, session: Any) -> "proto.Relation":
+        plan = self._create_proto_relation()
+        plan.sql.query = self._query
+
+        if self._args is not None and len(self._args) > 0:
+            plan.sql.pos_arguments.extend([arg.to_plan(session) for arg in 
self._args])
+        if self._named_args is not None and len(self._named_args) > 0:
+            for k, arg in self._named_args.items():
+                plan.sql.named_arguments[k].CopyFrom(arg.to_plan(session))
+
+        return self._with_relations(plan, session)
+
+    def command(self, session: Any) -> "proto.Command":
+        cmd = proto.Command()
+        cmd.sql_command.input.CopyFrom(self.plan(session))
+        return cmd
+
+
+class Range(LogicalPlan):
+    def __init__(
+        self,
+        start: int,
+        end: int,
+        step: int,
+        num_partitions: Optional[int] = None,
+    ) -> None:
+        super().__init__(None)
+        self._start = start
+        self._end = end
+        self._step = step
+        self._num_partitions = num_partitions
+
+    def plan(self, session: Any) -> "proto.Relation":
+        plan = self._create_proto_relation()
+        plan.range.start = self._start
+        plan.range.end = self._end
+        plan.range.step = self._step
+        if self._num_partitions is not None:
+            plan.range.num_partitions = self._num_partitions
+        return plan
+
+
+class RemoveRemoteCachedRelation(LogicalPlan):
+    def __init__(self, relation: CachedRemoteRelation) -> None:
+        super().__init__(None)
+        self._relation = relation
 
-    def __init__(self, *args: Any, **kwargs: Any) -> None:
-        pass
+    def command(self, session: Any) -> "proto.Command":
+        plan = self._create_proto_relation()
+        plan.cached_remote_relation.relation_id = self._relation._relation_id
+        cmd = proto.Command()
+        
cmd.remove_cached_remote_relation_command.relation.CopyFrom(plan.cached_remote_relation)
+        return cmd
 
 
-__all__ = ["LogicalPlan", "Read", "Range", "SQL"]
+__all__ = ["LogicalPlan", "Read", "Range", "SQL", "SubqueryAlias", 
"CachedRemoteRelation"]
diff --git a/python/pyspark/sql/connect/types.py 
b/python/pyspark/sql/connect/types.py
index 2ee9ac4..3878fe0 100644
--- a/python/pyspark/sql/connect/types.py
+++ b/python/pyspark/sql/connect/types.py
@@ -19,12 +19,13 @@ PySpark Connect's: ``UnparsedDataType``, a DDL string of a 
type that the server
 conversions between the types and their ``spark.connect.DataType`` messages,
 ``pyspark_types_to_proto_types`` and ``proto_schema_to_pyspark_data_type``, 
are those of the Rust
 transport, of the messages of the generated ``pyspark.sql.connect.proto``, 
which needs the
-``protobuf`` package."""
+``protobuf`` package. ``parse_attr_name``, ``verify_col_name`` and 
``verify_numeric_col_name`` are
+PySpark Connect's, from Apache Spark v4.3.0."""
 
-from typing import TYPE_CHECKING, Any, Dict
+from typing import TYPE_CHECKING, Any, Dict, List, Optional
 
 from pyspark.errors import PySparkAssertionError
-from pyspark.sql.types import DataType
+from pyspark.sql.types import ArrayType, DataType, NumericType, StructType
 
 if TYPE_CHECKING:
     import pyspark.sql.connect.proto as pb2
@@ -119,4 +120,103 @@ def proto_schema_to_pyspark_data_type(schema: 
"pb2.DataType") -> DataType:
     return _data_type_of_proto(schema.SerializeToString())
 
 
+# The python version of org.apache.spark.sql.catalyst.util.AttributeNameParser
+def parse_attr_name(name: str) -> Optional[List[str]]:
+    name_parts: List[str] = []
+    tmp: str = ""
+
+    in_backtick = False
+    i = 0
+    n = len(name)
+    while i < n:
+        char = name[i]
+        if in_backtick:
+            if char == "`":
+                if i + 1 < n and name[i + 1] == "`":
+                    tmp += "`"
+                    i += 1
+                else:
+                    in_backtick = False
+                    if i + 1 < n and name[i + 1] != ".":
+                        return None
+            else:
+                tmp += char
+        else:
+            if char == "`":
+                if len(tmp) > 0:
+                    return None
+                in_backtick = True
+            elif char == ".":
+                if name[i - 1] == "." or i == n - 1:
+                    return None
+                name_parts.append(tmp)
+                tmp = ""
+            else:
+                tmp += char
+        i += 1
+
+    if in_backtick:
+        return None
+
+    name_parts.append(tmp)
+    return name_parts
+
+
+# Verify whether the input column name can be resolved with the given schema.
+# Note that this method can not 100% match the analyzer behavior, it is 
designed to
+# try the best to eliminate unnecessary validation RPCs.
+def verify_col_name(name: str, schema: StructType) -> bool:
+    parts = parse_attr_name(name)
+    if parts is None or len(parts) == 0:
+        return False
+
+    def _quick_verify(parts: List[str], dt: DataType) -> bool:
+        if len(parts) == 0:
+            return True
+
+        _schema: Optional[StructType] = None
+        if isinstance(dt, StructType):
+            _schema = dt
+        elif isinstance(dt, ArrayType) and isinstance(dt.elementType, 
StructType):
+            _schema = dt.elementType
+        else:
+            return False
+
+        part = parts[0]
+        for field in _schema:
+            if field.name == part:
+                return _quick_verify(parts[1:], field.dataType)
+
+        return False
+
+    return _quick_verify(parts, schema)
+
+
+def verify_numeric_col_name(name: str, schema: StructType) -> bool:
+    parts = parse_attr_name(name)
+    if parts is None or len(parts) == 0:
+        return False
+
+    def _quick_verify(parts: List[str], dt: DataType) -> bool:
+        if len(parts) == 0 and isinstance(dt, NumericType):
+            return True
+
+        _schema: Optional[StructType] = None
+        if isinstance(dt, StructType):
+            _schema = dt
+        elif isinstance(dt, ArrayType) and isinstance(dt.elementType, 
StructType):
+            _schema = dt.elementType
+        else:
+            return False
+
+        part = parts[0]
+        for field in _schema:
+            if field.name == part:
+                return _quick_verify(parts[1:], field.dataType)
+
+        return False
+
+    return _quick_verify(parts, schema)
+
+
 __all__ = ["UnparsedDataType"]
diff --git a/python/tests/test_ml_e2e.py b/python/tests/test_ml_e2e.py
index c11b286..e3f91c8 100644
--- a/python/tests/test_ml_e2e.py
+++ b/python/tests/test_ml_e2e.py
@@ -161,6 +161,25 @@ def test_ml_evaluators_and_tuning_as_pyspark(session):
     assert cv_model.bestModel.transform(df).count() == 8
 
 
+def test_connect_plans_run_on_the_server_as_pyspark(session):
+    # As official PySpark 4.3.0: DataFrames of the Read, Range and SQL plans of
+    # pyspark.sql.connect.plan, as pyspark.testing.connectutils builds them.
+    from pyspark.sql import functions as F
+    from pyspark.sql.connect.dataframe import DataFrame
+    from pyspark.sql.connect.plan import SQL, Range, Read, SubqueryAlias
+
+    assert [r.id for r in DataFrame(Range(0, 6, 2, 2), session).collect()] == 
[0, 2, 4]
+    assert DataFrame(SQL("SELECT ? AS a", args=[F.lit(5)]), session).first().a 
== 5
+    assert DataFrame(SQL("SELECT :b AS b", named_args={"b": F.lit("x")}), 
session).first().b == "x"
+    views = [SubqueryAlias(Range(0, 3, 1), "v")]
+    assert DataFrame(SQL("SELECT sum(id) AS s FROM v", views=views), 
session).first().s == 3
+    session.range(4).createOrReplaceTempView("ml_e2e_plans")
+    try:
+        assert DataFrame(Read("ml_e2e_plans"), session).count() == 4
+    finally:
+        session.catalog.dropTempView("ml_e2e_plans")
+
+
 def test_ml_transformed_dataframes_keep_their_models_as_pyspark(session):
     # As official PySpark 4.3.0: a DataFrame of a transform keeps the model 
alive, and the
     # model is deleted from the cache of the server once nothing keeps it.
diff --git a/python/tests/test_ml_offline.py b/python/tests/test_ml_offline.py
index 1aa5009..addb857 100644
--- a/python/tests/test_ml_offline.py
+++ b/python/tests/test_ml_offline.py
@@ -48,6 +48,7 @@ from google.protobuf import text_format  # noqa: E402
 
 from pyspark.errors import PySparkTypeError, PySparkValueError  # noqa: E402
 from pyspark.sql import SparkSession  # noqa: E402
+from pyspark.sql import functions as F  # noqa: E402
 from pyspark.sql import types as T  # noqa: E402
 
 
@@ -350,6 +351,131 @@ def test_ml_data_types_are_those_of_pyspark(data_type, 
expected):
     assert proto_schema_to_pyspark_data_type(message) == data_type
 
 
[email protected](
+    "build, expected",
+    [
+        (
+            lambda P: P.Read("db.t", {"k": "v"}, True),
+            'read { named_table { unparsed_identifier: "db.t" options { key: 
"k" value: "v" } } '
+            "is_streaming: true }",
+        ),
+        (lambda P: P.Range(0, 3, 1, 2), "range { start: 0 end: 3 step: 1 
num_partitions: 2 }"),
+        (lambda P: P.Range(5, 9, 2), "range { start: 5 end: 9 step: 2 }"),
+        (
+            lambda P: P.SQL("SELECT ?", args=[F.lit(1)]),
+            'sql { query: "SELECT ?" pos_arguments { literal { integer: 1 } } 
}',
+        ),
+        (
+            lambda P: P.SQL("SELECT :a", named_args={"a": F.lit("x")}),
+            'sql { query: "SELECT :a" named_arguments { key: "a" value { 
literal { string: "x" } } } }',
+        ),
+        (
+            lambda P: P.SQL("SELECT * FROM v", 
views=[P.SubqueryAlias(P.Range(0, 1, 1), "v")]),
+            'with_relations { root { sql { query: "SELECT * FROM v" } } 
references { subquery_alias '
+            '{ input { range { start: 0 end: 1 step: 1 } } alias: "v" } } }',
+        ),
+        (
+            lambda P: P.SubqueryAlias(P.Read("t"), "x"),
+            'subquery_alias { input { read { named_table { 
unparsed_identifier: "t" } } } alias: "x" }',
+        ),
+    ],
+)
+def test_connect_plans_build_the_relations_of_pyspark(build, expected):
+    # As official PySpark 4.3.0's plans of pyspark.sql.connect.plan, of which
+    # pyspark.testing.connectutils builds DataFrames: their relations, of 
their own plan IDs.
+    import re
+
+    import pyspark.sql.connect.plan as P
+
+    plan = build(P)
+    relation = plan.plan(None)
+    assert relation.common.plan_id == plan._plan_id
+    assert re.sub(r"common \{ plan_id: \d+ \} ?", "", _text(relation)) == 
expected
+
+
+def test_connect_sql_plans_as_pyspark():
+    # As official PySpark 4.3.0: the command of a SQL plan, its print, and its 
arguments, which
+    # are Columns.
+    import pyspark.sql.connect.plan as P
+
+    command = P.SQL("SELECT 1").command(None)
+    assert command.sql_command.input.sql.query == "SELECT 1"
+    assert P.Read("t").print(2) == "  <Read table_name=t>\n"
+    with pytest.raises(AssertionError):
+        P.SQL("SELECT ?", args=[1])
+    with pytest.raises(AssertionError):
+        P.SQL("SELECT 1", views=[P.Range(0, 1, 1)])
+
+
+def test_column_to_plan_as_pyspark():
+    # As PySpark Connect's Column.to_plan: the spark.connect.Expression of the 
column.
+    from pyspark.sql import functions as F
+
+    assert _text(F.col("a").to_plan(None)) == (
+        'unresolved_attribute { unparsed_identifier: "a" is_metadata_column: 
false }'
+    )
+    column = F.lit(1) + F.col("b")
+    assert column.to_plan(None) == column._expr.to_plan(None)
+    assert column.to_plan(None).unresolved_function.function_name == "+"
+
+
+def test_connect_type_helpers_as_pyspark():
+    # As official PySpark 4.3.0's helpers of pyspark.sql.connect.types.
+    from pyspark.sql.connect.types import (
+        parse_attr_name,
+        verify_col_name,
+        verify_numeric_col_name,
+    )
+
+    assert parse_attr_name("a.`b.c`.d") == ["a", "b.c", "d"]
+    assert parse_attr_name("`a``b`") == ["a`b"]
+    assert parse_attr_name("a..b") is None and parse_attr_name("`a`b") is None
+    assert parse_attr_name("a.") is None and parse_attr_name("`a") is None
+    schema = T.StructType(
+        [
+            T.StructField("n", T.IntegerType()),
+            T.StructField("s", T.StringType()),
+            T.StructField(
+                "st", T.StructType([T.StructField("x", T.DoubleType()), 
T.StructField("y", T.StringType())])
+            ),
+            T.StructField("arr", T.ArrayType(T.StructType([T.StructField("z", 
T.LongType())]))),
+        ]
+    )
+    assert verify_col_name("st.x", schema) and verify_col_name("arr.z", schema)
+    assert not verify_col_name("st.w", schema) and not verify_col_name("a..b", 
schema)
+    assert verify_numeric_col_name("n", schema) and 
verify_numeric_col_name("st.x", schema)
+    assert verify_numeric_col_name("arr.z", schema)
+    assert not verify_numeric_col_name("s", schema) and not 
verify_numeric_col_name("st.y", schema)
+
+
+def test_connect_conversions_as_pyspark(ml_mock):
+    # As official PySpark 4.3.0's pyspark.sql.connect.conversion: the storage 
levels of their
+    # messages, and the DataFrame of a relation cached on the server, which 
removes the relation
+    # from the server once it is collected.
+    from pyspark.sql.connect.conversion import (
+        proto_to_remote_cached_dataframe,
+        proto_to_storage_level,
+        storage_level_to_proto,
+    )
+    from pyspark.storagelevel import StorageLevel
+
+    spark, service = ml_mock
+    level = StorageLevel(True, True, False, False, 2)
+    message = storage_level_to_proto(level)
+    assert message == pb2.StorageLevel(
+        use_disk=True, use_memory=True, use_off_heap=False, 
deserialized=False, replication=2
+    )
+    assert proto_to_storage_level(message) == level
+
+    df = 
proto_to_remote_cached_dataframe(pb2.CachedRemoteRelation(relation_id="r1"))
+    assert df._plan.plan(spark.client).cached_remote_relation.relation_id == 
"r1"
+    assert service.execute_requests == []
+    del df
+    gc.collect()
+    (command,) = _commands(service)
+    assert command.remove_cached_remote_relation_command.relation.relation_id 
== "r1"
+
+
 def test_ml_vector_udt_is_sent_as_pyspark():
     from pyspark.ml.linalg import VectorUDT
     from pyspark.sql.connect.types import proto_schema_to_pyspark_data_type
diff --git a/scripts/check_vendored_upstream.sh 
b/scripts/check_vendored_upstream.sh
index 2e0c0b8..5f7d65e 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -107,6 +107,8 @@ VENDORED_FILES=(
   # The logger of Spark Connect, and the types of the connect import path.
   "sql/connect/logging.py"
   "sql/connect/_typing.py"
+  # The conversions of the storage levels and the cached relations of their 
messages.
+  "sql/connect/conversion.py"
 
   # The spatial reference systems of the spatial types, and the 
SQLStringFormatter, of
   # pyspark.sql.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to