This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new eb79c3d [SPARK-61116][CONNECT][PYTHON] Port the Read, Range and SQL
plans, Column.to_plan, and the type and conversion helpers of the connect
import path, in the Python client as PySpark does
eb79c3d is described below
commit eb79c3d4670ec2dc522ab7c34ec4773b772bb1e6
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 23:20:00 2026 +0900
[SPARK-61116][CONNECT][PYTHON] Port the Read, Range and SQL plans,
Column.to_plan, and the type and conversion helpers of the connect import path,
in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61116](https://issues.apache.org/jira/browse/SPARK-61116): The rest
of the connect import path that builds or reads messages is PySpark's. It is
ported from Apache Spark v4.3.0 over the generated `pyspark.sql.connect.proto`
(SPARK-61111):
- `pyspark.sql.connect.plan`:
- `Read`, `Range` and `SQL` replace their placeholders, whose
constructors took any arguments and built no relation.
- `SubqueryAlias`, of the views of `SQL`.
- `CachedRemoteRelation`, which removes its relation from the server once
it is collected, through the client.
- `RemoveRemoteCachedRelation`.
`pyspark.testing.connectutils.PlanOnlyTestFixture` builds DataFrames of
these plans with `DataFrame(plan, session)`.
- `Column.to_plan(session)`: the `spark.connect.Expression` of a column. It
was the last justified non-gap of the introspection diff.
- `pyspark.sql.connect.types`: `parse_attr_name`, `verify_col_name` and
`verify_numeric_col_name`.
- `pyspark.sql.connect.conversion`, vendored byte for byte:
`storage_level_to_proto`, `proto_to_storage_level` and
`proto_to_remote_cached_dataframe`.
- `SparkConnectClient.is_closed`: whether the session is stopped.
`AGENTS.md` and `dev/parity/4.3.0-introspection-diff.md` now record no
justified non-gaps.
### Why are the changes needed?
- `DataFrame(Range(0, 3, 1), spark)` failed: `Range` built no relation.
- `Column.to_plan` was missing.
- `pyspark.sql.connect.conversion` and the helpers of
`pyspark.sql.connect.types` could not be imported.
### Does this PR introduce _any_ user-facing change?
Yes. These plans, functions and `Column.to_plan` exist as in PySpark 4.3.0.
### How was this patch tested?
- New offline tests check:
- the relations of the plans, which are those of official PySpark 4.3.0
apart from plan IDs, and their plan IDs
- the command and the checks of `SQL`, and `Read.print`
- `Column.to_plan`
- the type helpers
- the storage level conversions, and the DataFrame of a cached relation,
which removes the relation from a mock Connect server once it is collected
- A new live test runs DataFrames of `Range`, `SQL` with positional and
named arguments, `SQL` with views, and `Read` on a server. It passes with
official PySpark 4.3.0.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2174 passed, 8 skipped), and the
live drop-in and Arrow suites (173 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #314 from HyukjinKwon/SPARK-61116-connect-plans.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
AGENTS.md | 5 +-
crates/pyspark-rs/src/column.rs | 6 +
dev/parity/4.3.0-introspection-diff.md | 11 +-
python/.coveragerc | 1 +
python/pyspark/sql/connect/client/__init__.py | 6 +
python/pyspark/sql/connect/conversion.py | 59 ++++++++++
python/pyspark/sql/connect/plan.py | 163 +++++++++++++++++++++++---
python/pyspark/sql/connect/types.py | 106 ++++++++++++++++-
python/tests/test_ml_e2e.py | 19 +++
python/tests/test_ml_offline.py | 126 ++++++++++++++++++++
scripts/check_vendored_upstream.sh | 2 +
11 files changed, 478 insertions(+), 26 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index 0842fb7..07d0be7 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -129,9 +129,8 @@ Cover these classes: `DataFrame`, `Column`, `SparkSession`,
`SparkSession.Builde
`Catalog`, `Window`, `WindowSpec`, `Observation` (add any new connect classes
the
new version introduces).
-**Justified non-gaps** (absent on purpose because we use a native Rust
transport,
-not the Python gRPC client): `Column.to_plan(session)`. Anything else in the
diff is a
-real gap to close. Re-run the diff until it shows only these.
+There are no justified non-gaps: anything in the diff is a real gap to close.
Re-run the
+diff until it shows none.
`SparkSession.client` is not the gRPC `SparkConnectClient`: it is the client
that the
vendored `pyspark.ml` (and any module that builds its own `spark.connect`
messages) uses
diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index 3ea4699..01fb5df 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -673,6 +673,12 @@ impl PyColumn {
}
}
+ /// The `spark.connect.Expression` of this column, as PySpark Connect's
`Column.to_plan`: that
+ /// of its expression, of the generated `pyspark.sql.connect.proto`.
+ fn to_plan(&self, py: Python<'_>, session: &Bound<'_, PyAny>) ->
PyResult<Py<PyAny>> {
+ self._expr().to_plan(py, session)
+ }
+
fn __str__(&self) -> String {
let _origin = crate::origin::with_origin("__str__");
self.__repr__()
diff --git a/dev/parity/4.3.0-introspection-diff.md
b/dev/parity/4.3.0-introspection-diff.md
index f8f0fa3..1512008 100644
--- a/dev/parity/4.3.0-introspection-diff.md
+++ b/dev/parity/4.3.0-introspection-diff.md
@@ -14,7 +14,7 @@ Re-run against the 4.3.0 release: the results are the same as
against RC1 and RC
| Class | ref | ours | missing |
|---|---|---|---|
| DataFrame | 118 | 119 | 0 |
-| Column | 37 | 36 | 0 (justified: `to_plan`) |
+| Column | 37 | 37 | 0 |
| SparkSession | 40 | 42 | 0 |
| SparkSession.Builder | 8 | 8 | 0 |
| DataFrameReader | 14 | 14 | 0 |
@@ -26,10 +26,11 @@ Re-run against the 4.3.0 release: the results are the same
as against RC1 and RC
| WindowSpec | 4 | 4 | 0 |
| Observation | 1 | 2 | 0 |
-Justified non-gaps (native Rust transport, not the Python gRPC client):
`Column.to_plan`.
-`SparkSession.client` is the client that `pyspark.ml` uses over the Rust
transport
-(`pyspark.sql.connect.client.SparkConnectClient`: `execute_command` and the ML
cache), with
-the generated `pyspark.sql.connect.proto` (SPARK-61111).
+There are no justified non-gaps. `SparkSession.client` is the client that
`pyspark.ml` uses
+over the Rust transport (`pyspark.sql.connect.client.SparkConnectClient`:
`execute_command`
+and the ML cache), with the generated `pyspark.sql.connect.proto`
(SPARK-61111), and
+`Column.to_plan` is the `spark.connect.Expression` of a column, of those
messages
+(SPARK-61116).
## `SparkSession.Hook` (new in 4.3.0) — implemented
diff --git a/python/.coveragerc b/python/.coveragerc
index ffd1a5a..284e3c6 100644
--- a/python/.coveragerc
+++ b/python/.coveragerc
@@ -61,6 +61,7 @@ omit =
# The logger and the types of the connect import path, vendored.
*/pyspark/sql/connect/logging.py
*/pyspark/sql/connect/_typing.py
+ */pyspark/sql/connect/conversion.py
*/pyspark/_pyspark*
*/tests/*
diff --git a/python/pyspark/sql/connect/client/__init__.py
b/python/pyspark/sql/connect/client/__init__.py
index 5dca376..3701fcb 100644
--- a/python/pyspark/sql/connect/client/__init__.py
+++ b/python/pyspark/sql/connect/client/__init__.py
@@ -139,6 +139,12 @@ class SparkConnectClient:
def _server_session_id(self) -> str:
return self._session.session_id
+ @property
+ def is_closed(self) -> bool:
+ """Whether the session of the client is stopped, as
SparkConnectClient's ``is_closed``,
+ whether its channel is closed."""
+ return self._session.is_stopped
+
def execute_command(self, command: Any, observations: Any = None) -> Any:
"""Execute ``command``, a ``spark.connect.Command``, as
SparkConnectClient's
``execute_command``: no data, the properties of the results of the
responses, of their
diff --git a/python/pyspark/sql/connect/conversion.py
b/python/pyspark/sql/connect/conversion.py
new file mode 100644
index 0000000..137b045
--- /dev/null
+++ b/python/pyspark/sql/connect/conversion.py
@@ -0,0 +1,59 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from typing import TYPE_CHECKING
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.storagelevel import StorageLevel
+
+if TYPE_CHECKING:
+ from pyspark.sql.connect.dataframe import DataFrame
+
+
+def storage_level_to_proto(storage_level: StorageLevel) -> pb2.StorageLevel:
+ assert storage_level is not None and isinstance(storage_level,
StorageLevel)
+ return pb2.StorageLevel(
+ use_disk=storage_level.useDisk,
+ use_memory=storage_level.useMemory,
+ use_off_heap=storage_level.useOffHeap,
+ deserialized=storage_level.deserialized,
+ replication=storage_level.replication,
+ )
+
+
+def proto_to_storage_level(storage_level: pb2.StorageLevel) -> StorageLevel:
+ assert storage_level is not None and isinstance(storage_level,
pb2.StorageLevel)
+ return StorageLevel(
+ useDisk=storage_level.use_disk,
+ useMemory=storage_level.use_memory,
+ useOffHeap=storage_level.use_off_heap,
+ deserialized=storage_level.deserialized,
+ replication=storage_level.replication,
+ )
+
+
+def proto_to_remote_cached_dataframe(relation: pb2.CachedRemoteRelation) ->
"DataFrame":
+ assert relation is not None and isinstance(relation,
pb2.CachedRemoteRelation)
+
+ from pyspark.sql.connect.dataframe import DataFrame
+ from pyspark.sql.connect.session import SparkSession
+ import pyspark.sql.connect.plan as plan
+
+ session = SparkSession.active()
+ return DataFrame(
+ plan=plan.CachedRemoteRelation(relation.relation_id, session),
+ session=session,
+ )
diff --git a/python/pyspark/sql/connect/plan.py
b/python/pyspark/sql/connect/plan.py
index 990125e..9e2ccc1 100644
--- a/python/pyspark/sql/connect/plan.py
+++ b/python/pyspark/sql/connect/plan.py
@@ -22,15 +22,19 @@ those of ``pyspark.ml.connect.proto``: a plan of a child,
of a ``plan(session)``
``spark.connect.Relation``, which ``DataFrame(plan, session)`` sends as it is.
It has no
``to_proto``, ``_collect_references``, ``print`` or ``_repr_html_``, which are
of the plans and the
Columns of PySpark Connect's DataFrame, which the core builds.
``_DataFramePlan`` is the plan of a
-DataFrame of the core, its ``DataFrame._plan``. ``Read``, ``Range`` and
``SQL`` are not plans of
-this client. The messages are those of the generated
``pyspark.sql.connect.proto``, which needs
-the ``protobuf`` package.
+DataFrame of the core, its ``DataFrame._plan``. ``Read``, ``Range``, ``SQL``,
``SubqueryAlias``,
+``CachedRemoteRelation`` and ``RemoveRemoteCachedRelation`` are PySpark
Connect's, which
+``pyspark.testing.connectutils`` and ``pyspark.sql.connect.conversion`` build
DataFrames of. The
+messages are those of the generated ``pyspark.sql.connect.proto``, which needs
the ``protobuf``
+package.
"""
from threading import Lock
-from typing import Any, Dict, Optional, Sequence
+from typing import Any, Dict, List, Optional, Sequence
import pyspark.sql.connect.proto as proto
+from pyspark.sql.column import Column
+from pyspark.sql.connect.logging import logger
class LogicalPlan:
@@ -135,24 +139,153 @@ class _DataFramePlan(LogicalPlan):
class Read(LogicalPlan):
- """Not a plan of this client: the Rust core reads tables, as
``spark.read.table``."""
+ def __init__(
+ self,
+ table_name: str,
+ options: Optional[Dict[str, str]] = None,
+ is_streaming: Optional[bool] = None,
+ ) -> None:
+ super().__init__(None)
+ self.table_name = table_name
+ self.options = options or {}
+ self._is_streaming = is_streaming
+
+ def plan(self, session: Any) -> "proto.Relation":
+ plan = self._create_proto_relation()
+ plan.read.named_table.unparsed_identifier = self.table_name
+ if self._is_streaming is not None:
+ plan.read.is_streaming = self._is_streaming
+ for k, v in self.options.items():
+ plan.read.named_table.options[k] = v
+ return plan
- def __init__(self, *args: Any, **kwargs: Any) -> None:
- pass
+ def print(self, indent: int = 0) -> str:
+ return f"{' ' * indent}<Read table_name={self.table_name}>\n"
-class Range(LogicalPlan):
- """Not a plan of this client: the Rust core builds ranges, as
``spark.range``."""
+class CachedRemoteRelation(LogicalPlan):
+ """Logical plan object for a DataFrame reference which represents a
DataFrame that's been
+ cached on the server with a given id."""
- def __init__(self, *args: Any, **kwargs: Any) -> None:
- pass
+ def __init__(self, relation_id: str, spark_session: Any):
+ super().__init__(None)
+ self._relation_id = relation_id
+ # Needs to hold the session to make a request itself.
+ self._spark_session = spark_session
+
+ def plan(self, session: Any) -> "proto.Relation":
+ plan = self._create_proto_relation()
+ plan.cached_remote_relation.relation_id = self._relation_id
+ return plan
+
+ def __del__(self) -> None:
+ session = self._spark_session
+ # If session is already closed, all cached DataFrame should be
released.
+ if session is not None and not session.client.is_closed and
self._relation_id is not None:
+ try:
+ # The command of the Rust transport, as PySpark Connect's
sends it.
+ command =
RemoveRemoteCachedRelation(self).command(session=session.client)
+ session.client.execute_command(command)
+ except Exception as e:
+ logger.warning(f"RemoveRemoteCachedRelation failed with
exception: {e}.")
+
+
+class SubqueryAlias(LogicalPlan):
+ """Alias for a relation."""
+
+ def __init__(self, child: Optional["LogicalPlan"], alias: str) -> None:
+ super().__init__(child)
+ self._alias = alias
+
+ def plan(self, session: Any) -> "proto.Relation":
+ plan = self._create_proto_relation()
+ if self._child is not None:
+ plan.subquery_alias.input.CopyFrom(self._child.plan(session))
+ plan.subquery_alias.alias = self._alias
+ return plan
class SQL(LogicalPlan):
- """Not a plan of this client: the Rust core builds queries, as
``spark.sql``."""
+ def __init__(
+ self,
+ query: str,
+ args: Optional[List[Column]] = None,
+ named_args: Optional[Dict[str, Column]] = None,
+ views: Optional[Sequence[SubqueryAlias]] = None,
+ ) -> None:
+ if args is not None:
+ assert isinstance(args, List)
+ assert all(isinstance(arg, Column) for arg in args)
+
+ if named_args is not None:
+ assert isinstance(named_args, Dict)
+ for k, arg in named_args.items():
+ assert isinstance(k, str)
+ assert isinstance(arg, Column)
+
+ if views is not None:
+ assert isinstance(views, List)
+ assert all(isinstance(v, SubqueryAlias) for v in views)
+
+ super().__init__(None, views)
+ self._query = query
+ self._args = args
+ self._named_args = named_args
+ self._views = views
+
+ def plan(self, session: Any) -> "proto.Relation":
+ plan = self._create_proto_relation()
+ plan.sql.query = self._query
+
+ if self._args is not None and len(self._args) > 0:
+ plan.sql.pos_arguments.extend([arg.to_plan(session) for arg in
self._args])
+ if self._named_args is not None and len(self._named_args) > 0:
+ for k, arg in self._named_args.items():
+ plan.sql.named_arguments[k].CopyFrom(arg.to_plan(session))
+
+ return self._with_relations(plan, session)
+
+ def command(self, session: Any) -> "proto.Command":
+ cmd = proto.Command()
+ cmd.sql_command.input.CopyFrom(self.plan(session))
+ return cmd
+
+
+class Range(LogicalPlan):
+ def __init__(
+ self,
+ start: int,
+ end: int,
+ step: int,
+ num_partitions: Optional[int] = None,
+ ) -> None:
+ super().__init__(None)
+ self._start = start
+ self._end = end
+ self._step = step
+ self._num_partitions = num_partitions
+
+ def plan(self, session: Any) -> "proto.Relation":
+ plan = self._create_proto_relation()
+ plan.range.start = self._start
+ plan.range.end = self._end
+ plan.range.step = self._step
+ if self._num_partitions is not None:
+ plan.range.num_partitions = self._num_partitions
+ return plan
+
+
+class RemoveRemoteCachedRelation(LogicalPlan):
+ def __init__(self, relation: CachedRemoteRelation) -> None:
+ super().__init__(None)
+ self._relation = relation
- def __init__(self, *args: Any, **kwargs: Any) -> None:
- pass
+ def command(self, session: Any) -> "proto.Command":
+ plan = self._create_proto_relation()
+ plan.cached_remote_relation.relation_id = self._relation._relation_id
+ cmd = proto.Command()
+
cmd.remove_cached_remote_relation_command.relation.CopyFrom(plan.cached_remote_relation)
+ return cmd
-__all__ = ["LogicalPlan", "Read", "Range", "SQL"]
+__all__ = ["LogicalPlan", "Read", "Range", "SQL", "SubqueryAlias",
"CachedRemoteRelation"]
diff --git a/python/pyspark/sql/connect/types.py
b/python/pyspark/sql/connect/types.py
index 2ee9ac4..3878fe0 100644
--- a/python/pyspark/sql/connect/types.py
+++ b/python/pyspark/sql/connect/types.py
@@ -19,12 +19,13 @@ PySpark Connect's: ``UnparsedDataType``, a DDL string of a
type that the server
conversions between the types and their ``spark.connect.DataType`` messages,
``pyspark_types_to_proto_types`` and ``proto_schema_to_pyspark_data_type``,
are those of the Rust
transport, of the messages of the generated ``pyspark.sql.connect.proto``,
which needs the
-``protobuf`` package."""
+``protobuf`` package. ``parse_attr_name``, ``verify_col_name`` and
``verify_numeric_col_name`` are
+PySpark Connect's, from Apache Spark v4.3.0."""
-from typing import TYPE_CHECKING, Any, Dict
+from typing import TYPE_CHECKING, Any, Dict, List, Optional
from pyspark.errors import PySparkAssertionError
-from pyspark.sql.types import DataType
+from pyspark.sql.types import ArrayType, DataType, NumericType, StructType
if TYPE_CHECKING:
import pyspark.sql.connect.proto as pb2
@@ -119,4 +120,103 @@ def proto_schema_to_pyspark_data_type(schema:
"pb2.DataType") -> DataType:
return _data_type_of_proto(schema.SerializeToString())
+# The python version of org.apache.spark.sql.catalyst.util.AttributeNameParser
+def parse_attr_name(name: str) -> Optional[List[str]]:
+ name_parts: List[str] = []
+ tmp: str = ""
+
+ in_backtick = False
+ i = 0
+ n = len(name)
+ while i < n:
+ char = name[i]
+ if in_backtick:
+ if char == "`":
+ if i + 1 < n and name[i + 1] == "`":
+ tmp += "`"
+ i += 1
+ else:
+ in_backtick = False
+ if i + 1 < n and name[i + 1] != ".":
+ return None
+ else:
+ tmp += char
+ else:
+ if char == "`":
+ if len(tmp) > 0:
+ return None
+ in_backtick = True
+ elif char == ".":
+ if name[i - 1] == "." or i == n - 1:
+ return None
+ name_parts.append(tmp)
+ tmp = ""
+ else:
+ tmp += char
+ i += 1
+
+ if in_backtick:
+ return None
+
+ name_parts.append(tmp)
+ return name_parts
+
+
+# Verify whether the input column name can be resolved with the given schema.
+# Note that this method can not 100% match the analyzer behavior, it is
designed to
+# try the best to eliminate unnecessary validation RPCs.
+def verify_col_name(name: str, schema: StructType) -> bool:
+ parts = parse_attr_name(name)
+ if parts is None or len(parts) == 0:
+ return False
+
+ def _quick_verify(parts: List[str], dt: DataType) -> bool:
+ if len(parts) == 0:
+ return True
+
+ _schema: Optional[StructType] = None
+ if isinstance(dt, StructType):
+ _schema = dt
+ elif isinstance(dt, ArrayType) and isinstance(dt.elementType,
StructType):
+ _schema = dt.elementType
+ else:
+ return False
+
+ part = parts[0]
+ for field in _schema:
+ if field.name == part:
+ return _quick_verify(parts[1:], field.dataType)
+
+ return False
+
+ return _quick_verify(parts, schema)
+
+
+def verify_numeric_col_name(name: str, schema: StructType) -> bool:
+ parts = parse_attr_name(name)
+ if parts is None or len(parts) == 0:
+ return False
+
+ def _quick_verify(parts: List[str], dt: DataType) -> bool:
+ if len(parts) == 0 and isinstance(dt, NumericType):
+ return True
+
+ _schema: Optional[StructType] = None
+ if isinstance(dt, StructType):
+ _schema = dt
+ elif isinstance(dt, ArrayType) and isinstance(dt.elementType,
StructType):
+ _schema = dt.elementType
+ else:
+ return False
+
+ part = parts[0]
+ for field in _schema:
+ if field.name == part:
+ return _quick_verify(parts[1:], field.dataType)
+
+ return False
+
+ return _quick_verify(parts, schema)
+
+
__all__ = ["UnparsedDataType"]
diff --git a/python/tests/test_ml_e2e.py b/python/tests/test_ml_e2e.py
index c11b286..e3f91c8 100644
--- a/python/tests/test_ml_e2e.py
+++ b/python/tests/test_ml_e2e.py
@@ -161,6 +161,25 @@ def test_ml_evaluators_and_tuning_as_pyspark(session):
assert cv_model.bestModel.transform(df).count() == 8
+def test_connect_plans_run_on_the_server_as_pyspark(session):
+ # As official PySpark 4.3.0: DataFrames of the Read, Range and SQL plans of
+ # pyspark.sql.connect.plan, as pyspark.testing.connectutils builds them.
+ from pyspark.sql import functions as F
+ from pyspark.sql.connect.dataframe import DataFrame
+ from pyspark.sql.connect.plan import SQL, Range, Read, SubqueryAlias
+
+ assert [r.id for r in DataFrame(Range(0, 6, 2, 2), session).collect()] ==
[0, 2, 4]
+ assert DataFrame(SQL("SELECT ? AS a", args=[F.lit(5)]), session).first().a
== 5
+ assert DataFrame(SQL("SELECT :b AS b", named_args={"b": F.lit("x")}),
session).first().b == "x"
+ views = [SubqueryAlias(Range(0, 3, 1), "v")]
+ assert DataFrame(SQL("SELECT sum(id) AS s FROM v", views=views),
session).first().s == 3
+ session.range(4).createOrReplaceTempView("ml_e2e_plans")
+ try:
+ assert DataFrame(Read("ml_e2e_plans"), session).count() == 4
+ finally:
+ session.catalog.dropTempView("ml_e2e_plans")
+
+
def test_ml_transformed_dataframes_keep_their_models_as_pyspark(session):
# As official PySpark 4.3.0: a DataFrame of a transform keeps the model
alive, and the
# model is deleted from the cache of the server once nothing keeps it.
diff --git a/python/tests/test_ml_offline.py b/python/tests/test_ml_offline.py
index 1aa5009..addb857 100644
--- a/python/tests/test_ml_offline.py
+++ b/python/tests/test_ml_offline.py
@@ -48,6 +48,7 @@ from google.protobuf import text_format # noqa: E402
from pyspark.errors import PySparkTypeError, PySparkValueError # noqa: E402
from pyspark.sql import SparkSession # noqa: E402
+from pyspark.sql import functions as F # noqa: E402
from pyspark.sql import types as T # noqa: E402
@@ -350,6 +351,131 @@ def test_ml_data_types_are_those_of_pyspark(data_type,
expected):
assert proto_schema_to_pyspark_data_type(message) == data_type
[email protected](
+ "build, expected",
+ [
+ (
+ lambda P: P.Read("db.t", {"k": "v"}, True),
+ 'read { named_table { unparsed_identifier: "db.t" options { key:
"k" value: "v" } } '
+ "is_streaming: true }",
+ ),
+ (lambda P: P.Range(0, 3, 1, 2), "range { start: 0 end: 3 step: 1
num_partitions: 2 }"),
+ (lambda P: P.Range(5, 9, 2), "range { start: 5 end: 9 step: 2 }"),
+ (
+ lambda P: P.SQL("SELECT ?", args=[F.lit(1)]),
+ 'sql { query: "SELECT ?" pos_arguments { literal { integer: 1 } }
}',
+ ),
+ (
+ lambda P: P.SQL("SELECT :a", named_args={"a": F.lit("x")}),
+ 'sql { query: "SELECT :a" named_arguments { key: "a" value {
literal { string: "x" } } } }',
+ ),
+ (
+ lambda P: P.SQL("SELECT * FROM v",
views=[P.SubqueryAlias(P.Range(0, 1, 1), "v")]),
+ 'with_relations { root { sql { query: "SELECT * FROM v" } }
references { subquery_alias '
+ '{ input { range { start: 0 end: 1 step: 1 } } alias: "v" } } }',
+ ),
+ (
+ lambda P: P.SubqueryAlias(P.Read("t"), "x"),
+ 'subquery_alias { input { read { named_table {
unparsed_identifier: "t" } } } alias: "x" }',
+ ),
+ ],
+)
+def test_connect_plans_build_the_relations_of_pyspark(build, expected):
+ # As official PySpark 4.3.0's plans of pyspark.sql.connect.plan, of which
+ # pyspark.testing.connectutils builds DataFrames: their relations, of
their own plan IDs.
+ import re
+
+ import pyspark.sql.connect.plan as P
+
+ plan = build(P)
+ relation = plan.plan(None)
+ assert relation.common.plan_id == plan._plan_id
+ assert re.sub(r"common \{ plan_id: \d+ \} ?", "", _text(relation)) ==
expected
+
+
+def test_connect_sql_plans_as_pyspark():
+ # As official PySpark 4.3.0: the command of a SQL plan, its print, and its
arguments, which
+ # are Columns.
+ import pyspark.sql.connect.plan as P
+
+ command = P.SQL("SELECT 1").command(None)
+ assert command.sql_command.input.sql.query == "SELECT 1"
+ assert P.Read("t").print(2) == " <Read table_name=t>\n"
+ with pytest.raises(AssertionError):
+ P.SQL("SELECT ?", args=[1])
+ with pytest.raises(AssertionError):
+ P.SQL("SELECT 1", views=[P.Range(0, 1, 1)])
+
+
+def test_column_to_plan_as_pyspark():
+ # As PySpark Connect's Column.to_plan: the spark.connect.Expression of the
column.
+ from pyspark.sql import functions as F
+
+ assert _text(F.col("a").to_plan(None)) == (
+ 'unresolved_attribute { unparsed_identifier: "a" is_metadata_column:
false }'
+ )
+ column = F.lit(1) + F.col("b")
+ assert column.to_plan(None) == column._expr.to_plan(None)
+ assert column.to_plan(None).unresolved_function.function_name == "+"
+
+
+def test_connect_type_helpers_as_pyspark():
+ # As official PySpark 4.3.0's helpers of pyspark.sql.connect.types.
+ from pyspark.sql.connect.types import (
+ parse_attr_name,
+ verify_col_name,
+ verify_numeric_col_name,
+ )
+
+ assert parse_attr_name("a.`b.c`.d") == ["a", "b.c", "d"]
+ assert parse_attr_name("`a``b`") == ["a`b"]
+ assert parse_attr_name("a..b") is None and parse_attr_name("`a`b") is None
+ assert parse_attr_name("a.") is None and parse_attr_name("`a") is None
+ schema = T.StructType(
+ [
+ T.StructField("n", T.IntegerType()),
+ T.StructField("s", T.StringType()),
+ T.StructField(
+ "st", T.StructType([T.StructField("x", T.DoubleType()),
T.StructField("y", T.StringType())])
+ ),
+ T.StructField("arr", T.ArrayType(T.StructType([T.StructField("z",
T.LongType())]))),
+ ]
+ )
+ assert verify_col_name("st.x", schema) and verify_col_name("arr.z", schema)
+ assert not verify_col_name("st.w", schema) and not verify_col_name("a..b",
schema)
+ assert verify_numeric_col_name("n", schema) and
verify_numeric_col_name("st.x", schema)
+ assert verify_numeric_col_name("arr.z", schema)
+ assert not verify_numeric_col_name("s", schema) and not
verify_numeric_col_name("st.y", schema)
+
+
+def test_connect_conversions_as_pyspark(ml_mock):
+ # As official PySpark 4.3.0's pyspark.sql.connect.conversion: the storage
levels of their
+ # messages, and the DataFrame of a relation cached on the server, which
removes the relation
+ # from the server once it is collected.
+ from pyspark.sql.connect.conversion import (
+ proto_to_remote_cached_dataframe,
+ proto_to_storage_level,
+ storage_level_to_proto,
+ )
+ from pyspark.storagelevel import StorageLevel
+
+ spark, service = ml_mock
+ level = StorageLevel(True, True, False, False, 2)
+ message = storage_level_to_proto(level)
+ assert message == pb2.StorageLevel(
+ use_disk=True, use_memory=True, use_off_heap=False,
deserialized=False, replication=2
+ )
+ assert proto_to_storage_level(message) == level
+
+ df =
proto_to_remote_cached_dataframe(pb2.CachedRemoteRelation(relation_id="r1"))
+ assert df._plan.plan(spark.client).cached_remote_relation.relation_id ==
"r1"
+ assert service.execute_requests == []
+ del df
+ gc.collect()
+ (command,) = _commands(service)
+ assert command.remove_cached_remote_relation_command.relation.relation_id
== "r1"
+
+
def test_ml_vector_udt_is_sent_as_pyspark():
from pyspark.ml.linalg import VectorUDT
from pyspark.sql.connect.types import proto_schema_to_pyspark_data_type
diff --git a/scripts/check_vendored_upstream.sh
b/scripts/check_vendored_upstream.sh
index 2e0c0b8..5f7d65e 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -107,6 +107,8 @@ VENDORED_FILES=(
# The logger of Spark Connect, and the types of the connect import path.
"sql/connect/logging.py"
"sql/connect/_typing.py"
+ # The conversions of the storage levels and the cached relations of their
messages.
+ "sql/connect/conversion.py"
# The spatial reference systems of the spatial types, and the
SQLStringFormatter, of
# pyspark.sql.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]