This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 33d7b67  [SPARK-61120][CONNECT][PYTHON] Add DataFrame._session, 
SparkSession._client, and the methods and results of the client that vendored 
modules use, in the Python client as PySpark does
33d7b67 is described below

commit 33d7b675d5bf829dc779fd7f588637903a8309b8
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 06:40:59 2026 +0900

    [SPARK-61120][CONNECT][PYTHON] Add DataFrame._session, 
SparkSession._client, and the methods and results of the client that vendored 
modules use, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61120](https://issues.apache.org/jira/browse/SPARK-61120): Modules 
vendored from Apache Spark v4.3.0 read private attributes of PySpark Connect's 
DataFrame, session and client. This PR adds them:
    - `DataFrame._session`, the session of a DataFrame. These read it:
      - `pyspark.ml.util._cache_spark_dataset`, which `CrossValidator` and 
`TrainValidationSplit` use with `SPARKML_TEMP_DFS_PATH`
      - `pyspark.sql.plot`, for the maximum rows of a plot
      - `select_dtypes` of `pyspark.pandas`
    - `SparkSession._client`, the client of `SparkSession.client`, and 
`SparkConnectClient._resources()`, the resources of the cluster of a 
`GetResourcesCommand`. `TorchDistributor` reads them on Spark Connect.
    - `SparkConnectClient.execute_command` now converts the results as 
PySpark's does:
      - a `GetResourcesCommand` gives a dict of `ResourceInformation`
      - a resource profile gives its profile ID
      - a checkpoint gives a DataFrame of its cached relation
    
      It also returns the `pipeline_event_result` of a response.
    
      A `CachedRemoteRelation`, once collected, removes its relation from the 
server. It now sends that command and ignores the responses, as PySpark's does. 
Before, it went through `execute_command`, whose new conversion of a checkpoint 
result would make another cached relation.
    - `SparkConnectClient.release_session()`, which releases the session on the 
server, and `close()`, which closes the client. A closed client `is_closed`, 
and its session is stopped. Apache Spark's tests call them to clean up a new 
session.
    
    `dev/parity/known-differences.md` records why two of Apache Spark's ML 
parity tests still fail. `MockDataset` of `pyspark.testing.mlutils` creates 
itself with `object.__new__`, which a subclass of a class of the Rust extension 
refuses.
    
    ### Why are the changes needed?
    
    - On the client, `df._session` was a Column of that name. So with 
`SPARKML_TEMP_DFS_PATH` set, `CrossValidator(...).fit(df)` raised `TypeError: 
'Column' object is not callable`.
    - `TorchDistributor` failed on `session._client`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes:
    - `CrossValidator` and `TrainValidationSplit` work with 
`SPARKML_TEMP_DFS_PATH`.
    - `TorchDistributor` reads the resources of the cluster.
    - `execute_command` returns its results as PySpark's does.
    
    ### How was this patch tested?
    
    - New offline tests, against a mock Connect server, check:
      - `DataFrame._session` and `SparkSession._client`
      - the conversions of the results of the resources, a resource profile, a 
checkpoint and a pipeline event
      - that the relation of a checkpoint is removed from the server once its 
DataFrame is collected, by one command whose responses are ignored
      - `_resources()`
    - New live tests:
      - one fits a `CrossValidator` with `SPARKML_TEMP_DFS_PATH` set, and reads 
the resources of the server
      - one releases and closes the client of a new session
    
      They pass with official PySpark 4.3.0 and fail without this change.
    - Apache Spark's `test_parity_sql_context.py` passes on this client against 
a local server, all 21 tests. Before this change, 
`test_newSession_returns_distinct_instance` failed.
    - Apache Spark's `pyspark/ml/tests/connect/test_parity_*.py` run on this 
client against a local server. Apart from the two tests of `MockDataset` and 
one that reads Spark's sample data from the server's working directory, they 
pass. That includes `test_cross_validator` and `test_train_validation_split`, 
which failed without this change.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2180 passed, 8 skipped), and the 
live drop-in and Arrow suites (176 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #319 from HyukjinKwon/SPARK-61120-private-attrs.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs            |  7 ++++
 crates/pyspark-rs/src/session.rs              | 19 +++++++++
 crates/spark-connect/src/session.rs           | 12 ++++++
 dev/parity/known-differences.md               | 11 +++++
 python/pyspark/sql/connect/client/__init__.py | 44 ++++++++++++++++++--
 python/pyspark/sql/connect/plan.py            |  5 ++-
 python/tests/test_ml_e2e.py                   | 34 +++++++++++++++
 python/tests/test_ml_offline.py               | 60 +++++++++++++++++++++++++++
 8 files changed, 187 insertions(+), 5 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index ff29ab6..144d6fe 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1808,6 +1808,13 @@ impl PyDataFrame {
         crate::session::PySparkSession::new(self.dataframe.spark_session())
     }
 
+    /// The SparkSession of this DataFrame, as PySpark Connect's 
`DataFrame._session`, which the
+    /// modules of PySpark read, such as `pyspark.ml`, `pyspark.sql.plot` and 
`pyspark.pandas`.
+    #[getter]
+    fn _session(&self) -> crate::session::PySparkSession {
+        self.spark_session()
+    }
+
     /// Not supported on Spark Connect, as PySpark Connect's: 
`NOT_IMPLEMENTED` of `rdd`, which
     /// needs the RDD API of a classic session.
     #[getter]
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 5e59672..00e1a91 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -2005,6 +2005,25 @@ impl PySparkSession {
             .unbind())
     }
 
+    /// Release the session on the server, without stopping it, for 
`client.release_session`.
+    fn _release_session(&self, py: Python<'_>) -> PyResult<()> {
+        let session = self.session.clone();
+        py.detach(|| session.release_session().map_err(Box::new))
+            .to_pyerr()
+    }
+
+    /// Close the session without releasing it on the server, for 
`client.close`.
+    fn _close(&self) {
+        self.session.close();
+    }
+
+    /// The client of the session, as PySpark Connect's 
`SparkSession._client`, which the
+    /// modules of PySpark read, such as the `TorchDistributor` of 
`pyspark.ml`: `client`.
+    #[getter]
+    fn _client(slf: &Bound<'_, Self>) -> PyResult<Py<PyAny>> {
+        Self::client(slf)
+    }
+
     /// The session id (property form, mirrors `SparkSession.session_id`).
     #[getter]
     fn session_id(&self) -> String {
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 0f279aa..9343f73 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -1029,6 +1029,18 @@ impl SparkSession {
         block_on(self.client.release_session())
     }
 
+    /// Release the session on the server, as 
`SparkConnectClient.release_session`: its state
+    /// there, without stopping this session.
+    pub fn release_session(&self) -> Result<()> {
+        block_on(self.client.release_session())
+    }
+
+    /// Close this session without releasing it on the server, as 
`SparkConnectClient.close`
+    /// closes its channel: it is stopped, and `stop()` does nothing more.
+    pub fn close(&self) {
+        self.stopped.store(true, Ordering::SeqCst);
+    }
+
     /// Whether `stop()` has been called on this session. Mirrors
     /// `pyspark.sql.connect.session.SparkSession.is_stopped`.
     pub fn is_stopped(&self) -> bool {
diff --git a/dev/parity/known-differences.md b/dev/parity/known-differences.md
index 4f34958..bfe9fed 100644
--- a/dev/parity/known-differences.md
+++ b/dev/parity/known-differences.md
@@ -12,3 +12,14 @@ difference is intentional or tracked for a fix.
   raises PyArrow's `ArrowInvalid`. This client raises `AXIS_LENGTH_MISMATCH` 
for both
   rather than drop data.
 - **NumPy ndarray input** (tracked by SPARK-60772). Not accepted yet.
+
+## `DataFrame`
+
+- **A subclass of `DataFrame` created with `object.__new__`** (intentional, a 
PyO3 limit).
+  `DataFrame` is a class of the Rust extension, so `object.__new__(cls)` of a 
subclass raises
+  `TypeError: object.__new__(cls) is not safe`. `MockDataset` of 
`pyspark.testing.mlutils`
+  creates itself so, and two of Apache Spark's ML parity tests that use it
+  (`test_parity_pipeline.py`'s `test_pipeline` and `test_identity_pipeline`) 
fail for that
+  reason only. The rest of `pyspark/ml/tests/connect/test_parity_*.py` pass 
with this client,
+  except for a test that reads `data/mllib` of the Spark source tree from the 
working directory
+  of the server.
diff --git a/python/pyspark/sql/connect/client/__init__.py 
b/python/pyspark/sql/connect/client/__init__.py
index 3701fcb..6b9152c 100644
--- a/python/pyspark/sql/connect/client/__init__.py
+++ b/python/pyspark/sql/connect/client/__init__.py
@@ -148,7 +148,9 @@ class SparkConnectClient:
     def execute_command(self, command: Any, observations: Any = None) -> Any:
         """Execute ``command``, a ``spark.connect.Command``, as 
SparkConnectClient's
         ``execute_command``: no data, the properties of the results of the 
responses, of their
-        field names, and no execution information."""
+        field names, as SparkConnectClient converts them (the resources of a
+        GetResourcesCommand, the profile ID of a resource profile, and the 
DataFrame of a
+        checkpoint), and no execution information."""
         import pyspark.sql.connect.proto as pb2
 
         properties: Dict[str, Any] = {}
@@ -163,12 +165,29 @@ class SparkConnectClient:
                 "streaming_query_command_result",
                 "streaming_query_manager_command_result",
                 "streaming_query_listener_events_result",
-                "get_resources_command_result",
-                "ml_command_result",
                 "pipeline_command_result",
+                "pipeline_event_result",
+                "ml_command_result",
             ):
                 if response.HasField(field):
                     properties[field] = getattr(response, field)
+            if response.HasField("get_resources_command_result"):
+                from pyspark.resource.information import ResourceInformation
+
+                properties["get_resources_command_result"] = {
+                    key: ResourceInformation(resource.name, 
list(resource.addresses))
+                    for key, resource in 
response.get_resources_command_result.resources.items()
+                }
+            if response.HasField("create_resource_profile_command_result"):
+                properties["create_resource_profile_command_result"] = (
+                    response.create_resource_profile_command_result.profile_id
+                )
+            if response.HasField("checkpoint_command_result"):
+                from pyspark.sql.connect.conversion import 
proto_to_remote_cached_dataframe
+
+                properties["checkpoint_command_result"] = 
proto_to_remote_cached_dataframe(
+                    response.checkpoint_command_result.relation
+                )
         return None, properties, None
 
     def add_threadlocal_user_context_extension(self, extension: Any) -> str:
@@ -190,6 +209,25 @@ class SparkConnectClient:
         SparkConnectClient's."""
         self._session._clear_user_context_extensions()
 
+    def release_session(self) -> None:
+        """Release the session on the server, as SparkConnectClient's: its 
state there."""
+        self._session._release_session()
+
+    def close(self) -> None:
+        """Close the client, as SparkConnectClient's: of a session that 
``is_closed``, and is
+        not released on the server."""
+        self._session._close()
+
+    def _resources(self) -> Dict[str, Any]:
+        """The resources of the cluster, as SparkConnectClient's: of a 
GetResourcesCommand, by
+        their names."""
+        import pyspark.sql.connect.proto as pb2
+
+        command = pb2.Command()
+        command.get_resources_command.SetInParent()
+        _, properties, _ = self.execute_command(command)
+        return properties["get_resources_command_result"]
+
     def _delete_ml_cache(self, cache_ids: List[str], evict_only: bool = False) 
-> List[str]:
         """Delete the cached objects of ``cache_ids`` on the server, as 
SparkConnectClient's: the
         IDs deleted, or none if it fails."""
diff --git a/python/pyspark/sql/connect/plan.py 
b/python/pyspark/sql/connect/plan.py
index 9e2ccc1..e65d150 100644
--- a/python/pyspark/sql/connect/plan.py
+++ b/python/pyspark/sql/connect/plan.py
@@ -183,9 +183,10 @@ class CachedRemoteRelation(LogicalPlan):
         # If session is already closed, all cached DataFrame should be 
released.
         if session is not None and not session.client.is_closed and 
self._relation_id is not None:
             try:
-                # The command of the Rust transport, as PySpark Connect's 
sends it.
+                # The command, as PySpark Connect's sends it, whose responses 
are ignored: they
+                # are not converted, as that of a checkpoint is into another 
cached relation.
                 command = 
RemoveRemoteCachedRelation(self).command(session=session.client)
-                session.client.execute_command(command)
+                session._execute_command(command.SerializeToString())
             except Exception as e:
                 logger.warning(f"RemoveRemoteCachedRelation failed with 
exception: {e}.")
 
diff --git a/python/tests/test_ml_e2e.py b/python/tests/test_ml_e2e.py
index e3f91c8..11e5187 100644
--- a/python/tests/test_ml_e2e.py
+++ b/python/tests/test_ml_e2e.py
@@ -112,6 +112,40 @@ def test_ml_pipelines_save_and_load_as_pyspark(session, 
tmp_path):
     assert len(Pipeline.load(estimator_path).getStages()) == 3
 
 
+def test_ml_tuning_caches_its_dataset_of_a_dfs_path_as_pyspark(session, 
tmp_path, monkeypatch):
+    # As official PySpark 4.3.0: of SPARKML_TEMP_DFS_PATH, CrossValidator 
caches its dataset in
+    # a directory there, of the session of the dataset, DataFrame._session. 
And the resources of
+    # the cluster are those of the client of the session, SparkSession._client.
+    from pyspark.ml.classification import LogisticRegression
+    from pyspark.ml.evaluation import BinaryClassificationEvaluator
+    from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
+
+    monkeypatch.setenv("SPARKML_TEMP_DFS_PATH", str(tmp_path))
+    df = _labeled(session)
+    lr = LogisticRegression(maxIter=5)
+    cv = CrossValidator(
+        estimator=lr,
+        estimatorParamMaps=ParamGridBuilder().addGrid(lr.regParam, [0.1, 
0.01]).build(),
+        evaluator=BinaryClassificationEvaluator(),
+        numFolds=2,
+        seed=1,
+    )
+    assert len(cv.fit(df).avgMetrics) == 2
+    assert isinstance(session._client._resources(), dict)
+
+
+def test_session_clients_release_and_close_as_pyspark(session):
+    # As PySpark Connect's SparkConnectClient.release_session and close: the 
session is released
+    # on the server, and the client is closed, of a session that is then 
stopped.
+    other = SparkSession.builder.remote(os.environ["SPARK_REMOTE"]).create()
+    client = other.client
+    assert not client.is_closed
+    assert other.range(1).count() == 1
+    client.release_session()
+    client.close()
+    assert client.is_closed and other.is_stopped
+
+
 def test_ml_feature_transformers_as_pyspark(session):
     from pyspark.ml.feature import IDF, HashingTF, MinMaxScaler, Tokenizer
 
diff --git a/python/tests/test_ml_offline.py b/python/tests/test_ml_offline.py
index addb857..bb02c63 100644
--- a/python/tests/test_ml_offline.py
+++ b/python/tests/test_ml_offline.py
@@ -576,6 +576,66 @@ def 
test_ml_session_client_is_one_object_of_the_session(ml_mock):
         other.stop()
 
 
+def test_dataframe_session_and_session_client_as_pyspark(ml_mock):
+    # As PySpark Connect's DataFrame._session and SparkSession._client, which 
the modules of
+    # PySpark read, such as pyspark.ml, pyspark.sql.plot and pyspark.pandas.
+    spark, _ = ml_mock
+    session = spark.range(1)._session
+    assert isinstance(session, SparkSession)
+    assert session.session_id == spark.session_id
+    assert spark._client is spark.client
+
+
+def test_execute_command_converts_the_results_of_pyspark(ml_mock):
+    # As PySpark Connect's SparkConnectClient.execute_command: the resources 
of a
+    # GetResourcesCommand, as _resources returns them, the profile ID of a 
resource profile, the
+    # DataFrame of the relation of a checkpoint, and the event of a pipeline.
+    from pyspark.resource.information import ResourceInformation
+
+    spark, service = ml_mock
+    client = spark.client
+    resources = pb2.GetResourcesCommandResult()
+    resources.resources["gpu"].name = "gpu"
+    resources.resources["gpu"].addresses.extend(["0", "1"])
+    service.execute_payload = pb2.ExecutePlanResponse(
+        get_resources_command_result=resources
+    ).SerializeToString()
+    gpu = client._resources()["gpu"]
+    assert isinstance(gpu, ResourceInformation)
+    assert (gpu.name, gpu.addresses) == ("gpu", ["0", "1"])
+    (command,) = _commands(service)
+    assert command.HasField("get_resources_command")
+
+    service.execute_payload = pb2.ExecutePlanResponse(
+        
create_resource_profile_command_result=pb2.CreateResourceProfileCommandResult(profile_id=7)
+    ).SerializeToString()
+    assert client.execute_command(command)[1] == 
{"create_resource_profile_command_result": 7}
+
+    service.execute_payload = pb2.ExecutePlanResponse(
+        
pipeline_event_result=pb2.PipelineEventResult(event=pb2.PipelineEvent(message="m"))
+    ).SerializeToString()
+    assert 
client.execute_command(command)[1]["pipeline_event_result"].event.message == "m"
+
+    service.execute_payload = pb2.ExecutePlanResponse(
+        checkpoint_command_result=pb2.CheckpointCommandResult(
+            relation=pb2.CachedRemoteRelation(relation_id="r9")
+        )
+    ).SerializeToString()
+    df = client.execute_command(command)[1]["checkpoint_command_result"]
+    assert df._plan.plan(client).cached_remote_relation.relation_id == "r9"
+    # The relation is removed from the server once the DataFrame is collected, 
of a command
+    # whose responses are ignored, as PySpark's: so the checkpoint that the 
mock server answers
+    # with makes no other relation to remove.
+    del df
+    gc.collect()
+    removed = [
+        command.remove_cached_remote_relation_command.relation.relation_id
+        for command in _commands(service)
+        if command.HasField("remove_cached_remote_relation_command")
+    ]
+    assert removed == ["r9"]
+
+
 def test_ml_execute_command_returns_the_results_of_pyspark(ml_mock):
     # As PySpark Connect's SparkSession.client.execute_command: no data, the 
results of the
     # responses by their fields, and no execution information.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to