This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 33d7b67 [SPARK-61120][CONNECT][PYTHON] Add DataFrame._session,
SparkSession._client, and the methods and results of the client that vendored
modules use, in the Python client as PySpark does
33d7b67 is described below
commit 33d7b675d5bf829dc779fd7f588637903a8309b8
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 06:40:59 2026 +0900
[SPARK-61120][CONNECT][PYTHON] Add DataFrame._session,
SparkSession._client, and the methods and results of the client that vendored
modules use, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61120](https://issues.apache.org/jira/browse/SPARK-61120): Modules
vendored from Apache Spark v4.3.0 read private attributes of PySpark Connect's
DataFrame, session and client. This PR adds them:
- `DataFrame._session`, the session of a DataFrame. These read it:
- `pyspark.ml.util._cache_spark_dataset`, which `CrossValidator` and
`TrainValidationSplit` use with `SPARKML_TEMP_DFS_PATH`
- `pyspark.sql.plot`, for the maximum rows of a plot
- `select_dtypes` of `pyspark.pandas`
- `SparkSession._client`, the client of `SparkSession.client`, and
`SparkConnectClient._resources()`, the resources of the cluster of a
`GetResourcesCommand`. `TorchDistributor` reads them on Spark Connect.
- `SparkConnectClient.execute_command` now converts the results as
PySpark's does:
- a `GetResourcesCommand` gives a dict of `ResourceInformation`
- a resource profile gives its profile ID
- a checkpoint gives a DataFrame of its cached relation
It also returns the `pipeline_event_result` of a response.
A `CachedRemoteRelation`, once collected, removes its relation from the
server. It now sends that command and ignores the responses, as PySpark's does.
Before, it went through `execute_command`, whose new conversion of a checkpoint
result would make another cached relation.
- `SparkConnectClient.release_session()`, which releases the session on the
server, and `close()`, which closes the client. A closed client `is_closed`,
and its session is stopped. Apache Spark's tests call them to clean up a new
session.
`dev/parity/known-differences.md` records why two of Apache Spark's ML
parity tests still fail. `MockDataset` of `pyspark.testing.mlutils` creates
itself with `object.__new__`, which a subclass of a class of the Rust extension
refuses.
### Why are the changes needed?
- On the client, `df._session` was a Column of that name. So with
`SPARKML_TEMP_DFS_PATH` set, `CrossValidator(...).fit(df)` raised `TypeError:
'Column' object is not callable`.
- `TorchDistributor` failed on `session._client`.
### Does this PR introduce _any_ user-facing change?
Yes:
- `CrossValidator` and `TrainValidationSplit` work with
`SPARKML_TEMP_DFS_PATH`.
- `TorchDistributor` reads the resources of the cluster.
- `execute_command` returns its results as PySpark's does.
### How was this patch tested?
- New offline tests, against a mock Connect server, check:
- `DataFrame._session` and `SparkSession._client`
- the conversions of the results of the resources, a resource profile, a
checkpoint and a pipeline event
- that the relation of a checkpoint is removed from the server once its
DataFrame is collected, by one command whose responses are ignored
- `_resources()`
- New live tests:
- one fits a `CrossValidator` with `SPARKML_TEMP_DFS_PATH` set, and reads
the resources of the server
- one releases and closes the client of a new session
They pass with official PySpark 4.3.0 and fail without this change.
- Apache Spark's `test_parity_sql_context.py` passes on this client against
a local server, all 21 tests. Before this change,
`test_newSession_returns_distinct_instance` failed.
- Apache Spark's `pyspark/ml/tests/connect/test_parity_*.py` run on this
client against a local server. Apart from the two tests of `MockDataset` and
one that reads Spark's sample data from the server's working directory, they
pass. That includes `test_cross_validator` and `test_train_validation_split`,
which failed without this change.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2180 passed, 8 skipped), and the
live drop-in and Arrow suites (176 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #319 from HyukjinKwon/SPARK-61120-private-attrs.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 7 ++++
crates/pyspark-rs/src/session.rs | 19 +++++++++
crates/spark-connect/src/session.rs | 12 ++++++
dev/parity/known-differences.md | 11 +++++
python/pyspark/sql/connect/client/__init__.py | 44 ++++++++++++++++++--
python/pyspark/sql/connect/plan.py | 5 ++-
python/tests/test_ml_e2e.py | 34 +++++++++++++++
python/tests/test_ml_offline.py | 60 +++++++++++++++++++++++++++
8 files changed, 187 insertions(+), 5 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index ff29ab6..144d6fe 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1808,6 +1808,13 @@ impl PyDataFrame {
crate::session::PySparkSession::new(self.dataframe.spark_session())
}
+ /// The SparkSession of this DataFrame, as PySpark Connect's
`DataFrame._session`, which the
+ /// modules of PySpark read, such as `pyspark.ml`, `pyspark.sql.plot` and
`pyspark.pandas`.
+ #[getter]
+ fn _session(&self) -> crate::session::PySparkSession {
+ self.spark_session()
+ }
+
/// Not supported on Spark Connect, as PySpark Connect's:
`NOT_IMPLEMENTED` of `rdd`, which
/// needs the RDD API of a classic session.
#[getter]
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 5e59672..00e1a91 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -2005,6 +2005,25 @@ impl PySparkSession {
.unbind())
}
+ /// Release the session on the server, without stopping it, for
`client.release_session`.
+ fn _release_session(&self, py: Python<'_>) -> PyResult<()> {
+ let session = self.session.clone();
+ py.detach(|| session.release_session().map_err(Box::new))
+ .to_pyerr()
+ }
+
+ /// Close the session without releasing it on the server, for
`client.close`.
+ fn _close(&self) {
+ self.session.close();
+ }
+
+ /// The client of the session, as PySpark Connect's
`SparkSession._client`, which the
+ /// modules of PySpark read, such as the `TorchDistributor` of
`pyspark.ml`: `client`.
+ #[getter]
+ fn _client(slf: &Bound<'_, Self>) -> PyResult<Py<PyAny>> {
+ Self::client(slf)
+ }
+
/// The session id (property form, mirrors `SparkSession.session_id`).
#[getter]
fn session_id(&self) -> String {
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 0f279aa..9343f73 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -1029,6 +1029,18 @@ impl SparkSession {
block_on(self.client.release_session())
}
+ /// Release the session on the server, as
`SparkConnectClient.release_session`: its state
+ /// there, without stopping this session.
+ pub fn release_session(&self) -> Result<()> {
+ block_on(self.client.release_session())
+ }
+
+ /// Close this session without releasing it on the server, as
`SparkConnectClient.close`
+ /// closes its channel: it is stopped, and `stop()` does nothing more.
+ pub fn close(&self) {
+ self.stopped.store(true, Ordering::SeqCst);
+ }
+
/// Whether `stop()` has been called on this session. Mirrors
/// `pyspark.sql.connect.session.SparkSession.is_stopped`.
pub fn is_stopped(&self) -> bool {
diff --git a/dev/parity/known-differences.md b/dev/parity/known-differences.md
index 4f34958..bfe9fed 100644
--- a/dev/parity/known-differences.md
+++ b/dev/parity/known-differences.md
@@ -12,3 +12,14 @@ difference is intentional or tracked for a fix.
raises PyArrow's `ArrowInvalid`. This client raises `AXIS_LENGTH_MISMATCH`
for both
rather than drop data.
- **NumPy ndarray input** (tracked by SPARK-60772). Not accepted yet.
+
+## `DataFrame`
+
+- **A subclass of `DataFrame` created with `object.__new__`** (intentional, a
PyO3 limit).
+ `DataFrame` is a class of the Rust extension, so `object.__new__(cls)` of a
subclass raises
+ `TypeError: object.__new__(cls) is not safe`. `MockDataset` of
`pyspark.testing.mlutils`
+ creates itself so, and two of Apache Spark's ML parity tests that use it
+ (`test_parity_pipeline.py`'s `test_pipeline` and `test_identity_pipeline`)
fail for that
+ reason only. The rest of `pyspark/ml/tests/connect/test_parity_*.py` pass
with this client,
+ except for a test that reads `data/mllib` of the Spark source tree from the
working directory
+ of the server.
diff --git a/python/pyspark/sql/connect/client/__init__.py
b/python/pyspark/sql/connect/client/__init__.py
index 3701fcb..6b9152c 100644
--- a/python/pyspark/sql/connect/client/__init__.py
+++ b/python/pyspark/sql/connect/client/__init__.py
@@ -148,7 +148,9 @@ class SparkConnectClient:
def execute_command(self, command: Any, observations: Any = None) -> Any:
"""Execute ``command``, a ``spark.connect.Command``, as
SparkConnectClient's
``execute_command``: no data, the properties of the results of the
responses, of their
- field names, and no execution information."""
+ field names, as SparkConnectClient converts them (the resources of a
+ GetResourcesCommand, the profile ID of a resource profile, and the
DataFrame of a
+ checkpoint), and no execution information."""
import pyspark.sql.connect.proto as pb2
properties: Dict[str, Any] = {}
@@ -163,12 +165,29 @@ class SparkConnectClient:
"streaming_query_command_result",
"streaming_query_manager_command_result",
"streaming_query_listener_events_result",
- "get_resources_command_result",
- "ml_command_result",
"pipeline_command_result",
+ "pipeline_event_result",
+ "ml_command_result",
):
if response.HasField(field):
properties[field] = getattr(response, field)
+ if response.HasField("get_resources_command_result"):
+ from pyspark.resource.information import ResourceInformation
+
+ properties["get_resources_command_result"] = {
+ key: ResourceInformation(resource.name,
list(resource.addresses))
+ for key, resource in
response.get_resources_command_result.resources.items()
+ }
+ if response.HasField("create_resource_profile_command_result"):
+ properties["create_resource_profile_command_result"] = (
+ response.create_resource_profile_command_result.profile_id
+ )
+ if response.HasField("checkpoint_command_result"):
+ from pyspark.sql.connect.conversion import
proto_to_remote_cached_dataframe
+
+ properties["checkpoint_command_result"] =
proto_to_remote_cached_dataframe(
+ response.checkpoint_command_result.relation
+ )
return None, properties, None
def add_threadlocal_user_context_extension(self, extension: Any) -> str:
@@ -190,6 +209,25 @@ class SparkConnectClient:
SparkConnectClient's."""
self._session._clear_user_context_extensions()
+ def release_session(self) -> None:
+ """Release the session on the server, as SparkConnectClient's: its
state there."""
+ self._session._release_session()
+
+ def close(self) -> None:
+ """Close the client, as SparkConnectClient's: of a session that
``is_closed``, and is
+ not released on the server."""
+ self._session._close()
+
+ def _resources(self) -> Dict[str, Any]:
+ """The resources of the cluster, as SparkConnectClient's: of a
GetResourcesCommand, by
+ their names."""
+ import pyspark.sql.connect.proto as pb2
+
+ command = pb2.Command()
+ command.get_resources_command.SetInParent()
+ _, properties, _ = self.execute_command(command)
+ return properties["get_resources_command_result"]
+
def _delete_ml_cache(self, cache_ids: List[str], evict_only: bool = False)
-> List[str]:
"""Delete the cached objects of ``cache_ids`` on the server, as
SparkConnectClient's: the
IDs deleted, or none if it fails."""
diff --git a/python/pyspark/sql/connect/plan.py
b/python/pyspark/sql/connect/plan.py
index 9e2ccc1..e65d150 100644
--- a/python/pyspark/sql/connect/plan.py
+++ b/python/pyspark/sql/connect/plan.py
@@ -183,9 +183,10 @@ class CachedRemoteRelation(LogicalPlan):
# If session is already closed, all cached DataFrame should be
released.
if session is not None and not session.client.is_closed and
self._relation_id is not None:
try:
- # The command of the Rust transport, as PySpark Connect's
sends it.
+ # The command, as PySpark Connect's sends it, whose responses
are ignored: they
+ # are not converted, as that of a checkpoint is into another
cached relation.
command =
RemoveRemoteCachedRelation(self).command(session=session.client)
- session.client.execute_command(command)
+ session._execute_command(command.SerializeToString())
except Exception as e:
logger.warning(f"RemoveRemoteCachedRelation failed with
exception: {e}.")
diff --git a/python/tests/test_ml_e2e.py b/python/tests/test_ml_e2e.py
index e3f91c8..11e5187 100644
--- a/python/tests/test_ml_e2e.py
+++ b/python/tests/test_ml_e2e.py
@@ -112,6 +112,40 @@ def test_ml_pipelines_save_and_load_as_pyspark(session,
tmp_path):
assert len(Pipeline.load(estimator_path).getStages()) == 3
+def test_ml_tuning_caches_its_dataset_of_a_dfs_path_as_pyspark(session,
tmp_path, monkeypatch):
+ # As official PySpark 4.3.0: of SPARKML_TEMP_DFS_PATH, CrossValidator
caches its dataset in
+ # a directory there, of the session of the dataset, DataFrame._session.
And the resources of
+ # the cluster are those of the client of the session, SparkSession._client.
+ from pyspark.ml.classification import LogisticRegression
+ from pyspark.ml.evaluation import BinaryClassificationEvaluator
+ from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
+
+ monkeypatch.setenv("SPARKML_TEMP_DFS_PATH", str(tmp_path))
+ df = _labeled(session)
+ lr = LogisticRegression(maxIter=5)
+ cv = CrossValidator(
+ estimator=lr,
+ estimatorParamMaps=ParamGridBuilder().addGrid(lr.regParam, [0.1,
0.01]).build(),
+ evaluator=BinaryClassificationEvaluator(),
+ numFolds=2,
+ seed=1,
+ )
+ assert len(cv.fit(df).avgMetrics) == 2
+ assert isinstance(session._client._resources(), dict)
+
+
+def test_session_clients_release_and_close_as_pyspark(session):
+ # As PySpark Connect's SparkConnectClient.release_session and close: the
session is released
+ # on the server, and the client is closed, of a session that is then
stopped.
+ other = SparkSession.builder.remote(os.environ["SPARK_REMOTE"]).create()
+ client = other.client
+ assert not client.is_closed
+ assert other.range(1).count() == 1
+ client.release_session()
+ client.close()
+ assert client.is_closed and other.is_stopped
+
+
def test_ml_feature_transformers_as_pyspark(session):
from pyspark.ml.feature import IDF, HashingTF, MinMaxScaler, Tokenizer
diff --git a/python/tests/test_ml_offline.py b/python/tests/test_ml_offline.py
index addb857..bb02c63 100644
--- a/python/tests/test_ml_offline.py
+++ b/python/tests/test_ml_offline.py
@@ -576,6 +576,66 @@ def
test_ml_session_client_is_one_object_of_the_session(ml_mock):
other.stop()
+def test_dataframe_session_and_session_client_as_pyspark(ml_mock):
+ # As PySpark Connect's DataFrame._session and SparkSession._client, which
the modules of
+ # PySpark read, such as pyspark.ml, pyspark.sql.plot and pyspark.pandas.
+ spark, _ = ml_mock
+ session = spark.range(1)._session
+ assert isinstance(session, SparkSession)
+ assert session.session_id == spark.session_id
+ assert spark._client is spark.client
+
+
+def test_execute_command_converts_the_results_of_pyspark(ml_mock):
+ # As PySpark Connect's SparkConnectClient.execute_command: the resources
of a
+ # GetResourcesCommand, as _resources returns them, the profile ID of a
resource profile, the
+ # DataFrame of the relation of a checkpoint, and the event of a pipeline.
+ from pyspark.resource.information import ResourceInformation
+
+ spark, service = ml_mock
+ client = spark.client
+ resources = pb2.GetResourcesCommandResult()
+ resources.resources["gpu"].name = "gpu"
+ resources.resources["gpu"].addresses.extend(["0", "1"])
+ service.execute_payload = pb2.ExecutePlanResponse(
+ get_resources_command_result=resources
+ ).SerializeToString()
+ gpu = client._resources()["gpu"]
+ assert isinstance(gpu, ResourceInformation)
+ assert (gpu.name, gpu.addresses) == ("gpu", ["0", "1"])
+ (command,) = _commands(service)
+ assert command.HasField("get_resources_command")
+
+ service.execute_payload = pb2.ExecutePlanResponse(
+
create_resource_profile_command_result=pb2.CreateResourceProfileCommandResult(profile_id=7)
+ ).SerializeToString()
+ assert client.execute_command(command)[1] ==
{"create_resource_profile_command_result": 7}
+
+ service.execute_payload = pb2.ExecutePlanResponse(
+
pipeline_event_result=pb2.PipelineEventResult(event=pb2.PipelineEvent(message="m"))
+ ).SerializeToString()
+ assert
client.execute_command(command)[1]["pipeline_event_result"].event.message == "m"
+
+ service.execute_payload = pb2.ExecutePlanResponse(
+ checkpoint_command_result=pb2.CheckpointCommandResult(
+ relation=pb2.CachedRemoteRelation(relation_id="r9")
+ )
+ ).SerializeToString()
+ df = client.execute_command(command)[1]["checkpoint_command_result"]
+ assert df._plan.plan(client).cached_remote_relation.relation_id == "r9"
+ # The relation is removed from the server once the DataFrame is collected,
of a command
+ # whose responses are ignored, as PySpark's: so the checkpoint that the
mock server answers
+ # with makes no other relation to remove.
+ del df
+ gc.collect()
+ removed = [
+ command.remove_cached_remote_relation_command.relation.relation_id
+ for command in _commands(service)
+ if command.HasField("remove_cached_remote_relation_command")
+ ]
+ assert removed == ["r9"]
+
+
def test_ml_execute_command_returns_the_results_of_pyspark(ml_mock):
# As PySpark Connect's SparkSession.client.execute_command: no data, the
results of the
# responses by their fields, and no execution information.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]