This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 915c741e [SPARK-61202][CONNECT][PYTHON] Send the relation of a 
DataFrame into the plans of pyspark.ml without parsing it, so deep DataFrames 
fit and transform, in the Python client as PySpark does
915c741e is described below

commit 915c741ef420a9b939c2ba2553c0cc96aa304c2e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:07:23 2026 +0900

    [SPARK-61202][CONNECT][PYTHON] Send the relation of a DataFrame into the 
plans of pyspark.ml without parsing it, so deep DataFrames fit and transform, 
in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61202](https://issues.apache.org/jira/browse/SPARK-61202): Send the 
relation of a DataFrame into the plans of `pyspark.ml` without parsing it, so 
that a deep DataFrame fits and transforms, as PySpark does.
    
    The plans of `pyspark.ml` took the relation of a DataFrame by parsing it 
twice. `_DataFramePlan.plan` decoded it with the Python protobuf decoder, and 
`DataFrame(plan, session)` and the ML commands decoded the built plan with 
prost. Both decoders refuse more than 100 levels of nesting, so a DataFrame 
deeper than that could not be fitted or transformed. PySpark builds the nested 
message without parsing it, so it works at these depths.
    
    The change:
    
    - `_DataFramePlan.plan` returns a placeholder of the relation of its 
DataFrame: a shallow `Relation` whose `extension` is a marker of the DataFrame. 
The vendored plans of `pyspark.ml` build over it as they do over a relation, so 
the plan of a transform over a deep DataFrame is shallow too.
    - The core keeps a registry of the placeholders: each marker names the plan 
node of its DataFrame and the plan IDs of its session, both held weakly, and an 
entry is pruned with its DataFrame, as `PlanIds` prunes its nodes. The same 
DataFrame gives the same marker.
    - The core replaces each placeholder with the relation of its DataFrame, 
built as `DataFrame.to_relation` builds it, so it is the relation that is sent 
today. The replacement is moved in, never decoded. It happens where a 
Python-built relation or command reaches the core:
      - `DataFrame::from_relation` (`DataFrame(plan, session)`), in the 
positions that the ML plans build over their inputs: 
`ml_relation.transform.input`, `ml_relation.model_summary_dataset`, the 
arguments of `fetch`, and the `root` and `references` of `with_relations`;
      - `execute_command_proto` (`session.client.execute_command`), in the 
datasets of the `fit`, `evaluate` and `create_summary` commands and in the 
arguments of `fetch`;
      - the session hooks, for the root or the command of the request they 
return.
    - The owner of a DataFrame of a plan is still the Python plan, so the model 
keep-alive of SPARK-61158 is unchanged.
    
    What server receives is the same relation as before this change: the same 
plan IDs and the same nesting, with no placeholder left. Official PySpark 4.3.0 
sends the same nested relation, modulo plan IDs, because it also builds it 
without parsing it.
    
    Why not raise the limits: prost's `no-recursion-limit` feature removes the 
check on input that is not trusted, and a stack overflow is then a crash of the 
process. upb (the Python decoder) has no public setting for its limit. A larger 
constant only moves the failure: a OneVsRest transform adds layers per class, 
so any fixed limit fails at some class count. The placeholder does not decode 
the nesting at all, so its depth is bounded only by the stack of the encoder, 
as it is for `count` [...]
    
    Known differences:
    
    - A placeholder is valid while its DataFrame is alive, as its DataFrame is 
kept by the plan that uses it. `df._plan.plan(client)` is therefore no longer a 
self-contained relation. It is internal, the plans of `pyspark.ml` are the 
users. The hook test of `test_ml_e2e.py` keeps its source DataFrame for this 
reason.
    - A placeholder that a user puts under a relation that is not one of the 
positions above reaches the server unresolved, and the server reports an error 
for it. Such a relation is not built by PySpark or by the vendored code.
    - Not changed here: `DataFrame._of_relation`, used when a DataFrame is 
unpickled, and the session hooks, which encode the request into Python and 
decode it back. Both still parse a deep relation: a hook on a DataFrame of 120 
`withColumn` calls raises `DecodeError` with or without this change (checked on 
this branch). They are not on the path of `pyspark.ml`, and they are a 
follow-up.
    
    ### Why are the changes needed?
    
    A DataFrame of about 50 chained `withColumn` calls cannot be fitted: 
`LogisticRegression(maxIter=2).fit(df)` raises `DecodeError: Error parsing 
message: too many levels of nesting`. A `transform` of a `LogisticRegression` 
model over the same DataFrame raises the same error, and a 
`OneVsRestModel.transform` with 10 or more classes too, since each class adds 
layers. A pipeline of 60 `SQLTransformer` stages fails with `ValueError: ... 
recursion limit reached`. `count` and `collect` work  [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `pyspark.ml` fits and transforms DataFrames of any depth that PySpark 
accepts, and `DataFrame._plan.plan(client)` returns a placeholder (see the 
known differences). The sent requests are the same as before for the DataFrames 
that worked.
    
    ### How was this patch tested?
    
    - Rust unit tests in `crates/spark-connect/src/plan.rs` 
(`placeholder_tests`, 7 tests): a placeholder is shallow and resolves to the 
relation of its node; the same node and session give the same placeholder; the 
placeholders of the ML positions (`model_summary_dataset`, `transform.input`, 
the arguments of `fetch`, `with_relations`) and of the ML commands (`fit`, 
`evaluate`, `create_summary`, `fetch`) and of a hook's root are replaced; 
relations without placeholders are unchanged; a pl [...]
    - Rust integration tests in `crates/spark-connect/tests/raw_relation.rs` (3 
new): a placeholder of a 120-level DataFrame is replaced by its relation in 
`from_relation`, with the same plan IDs; a placeholder of a dropped DataFrame 
is an error; a live test runs a 120-level placeholder on the server.
    - The tests of deep plans (the 200-level unit test and the two 120-level 
`raw_relation.rs` tests) run on a thread with a 64 MiB stack. The plan is 
converted recursively, and in a debug build, as in the WASM UDF and coverage 
jobs, a plan of that depth takes more than the 2 MiB stack of a test thread. In 
debug builds, the lib tests (485 passed) and `raw_relation.rs` against a local 
Spark 4.3.0 server (10 passed) pass.
    - Offline Python test 
`test_ml_plans_send_a_deep_dataframe_without_parsing_it_as_pyspark` in 
`python/tests/test_dropin_offline.py`: two transformer plans over a 120-level 
DataFrame; the request that the mock server receives carries the relation of 
the DataFrame byte for byte, and a placeholder of a dropped DataFrame is an 
error.
    - Live Python test `test_ml_deep_dataframes_fit_and_transform_as_pyspark` 
in `python/tests/test_ml_e2e.py`: `LogisticRegression` fits and transforms a 
120-level DataFrame with the predictions of the shallow DataFrame, and 
`OneVsRest` with 12 classes transforms a 120-level DataFrame with the 
predictions of the shallow one. The DataFrames are over a repartition of their 
local data: over a local relation, the optimizer folds one projection per 
iteration, and a server in testing mode (`SP [...]
    - Existing tests that read `_plan.plan(...)` as a relation now read the 
relation that the server receives (`_relation` in 
`python/tests/test_ml_offline.py`, 9 places), and the session hook test keeps 
its source DataFrame.
    - The tests of both crates, e2e included, against a local Spark 4.3.0 
server (926 passed), the build of all tests, also with `wasm-udf`, the offline 
Python suites (2457 passed, 8 skipped), and the live drop-in, Arrow and ML 
suites (218 passed) and live cases (11 passed). rustfmt passes, and clippy 
reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #387 from HyukjinKwon/SPARK-61202-ml-plans-deep-dataframes.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs         |  17 +-
 crates/pyspark-rs/src/session.rs           |   8 +-
 crates/spark-connect/src/dataframe.rs      |  26 +-
 crates/spark-connect/src/plan.rs           | 452 ++++++++++++++++++++++++++++-
 crates/spark-connect/src/session.rs        |  10 +-
 crates/spark-connect/tests/raw_relation.rs | 108 ++++++-
 python/pyspark/sql/connect/plan.py         |   9 +-
 python/tests/test_dropin_offline.py        |  59 ++++
 python/tests/test_ml_e2e.py                |  37 ++-
 python/tests/test_ml_offline.py            |  24 +-
 10 files changed, 722 insertions(+), 28 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index b98ffa34..39d1789c 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -648,11 +648,10 @@ impl PyDataFrame {
         // objects on the server the relation refers to 
(`__source_instance__`).
         let owner: std::sync::Arc<dyn std::any::Any + Send + Sync> =
             std::sync::Arc::new(plan.clone().unbind());
-        Ok(PyDataFrame::new(DataFrame::from_relation(
-            session.borrow().session.clone(),
-            relation,
-            Some(owner),
-        )))
+        Ok(PyDataFrame::new(
+            DataFrame::from_relation(session.borrow().session.clone(), 
relation, Some(owner))
+                .to_pyerr()?,
+        ))
     }
 
     /// The bytes of the relation of this DataFrame as it is sent, for the 
plan of `_plan`.
@@ -661,6 +660,14 @@ impl PyDataFrame {
         pyo3::types::PyBytes::new(py, 
&self.dataframe.to_relation().encode_to_vec())
     }
 
+    /// The bytes of the placeholder of the relation of this DataFrame, which 
the plan of `_plan`
+    /// sends in its place: shallow, whatever the depth of the DataFrame, and 
replaced by the
+    /// core with that relation when the plan reaches it.
+    fn _relation_placeholder<'py>(&self, py: Python<'py>) -> Bound<'py, 
pyo3::types::PyBytes> {
+        use prost::Message;
+        pyo3::types::PyBytes::new(py, 
&self.dataframe.placeholder_relation().encode_to_vec())
+    }
+
     /// A DataFrame pickles, as PySpark Connect's, of its session, which gives 
back the active
     /// session of its ID, and of its plan, here the bytes of its relation. 
The session unpickles
     /// first: so on a Python worker of the server, which has no active 
session, a pickled
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 128558b4..d1c963f5 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -425,10 +425,14 @@ fn call_execute_plan_hooks(
         }
     }
     let bytes = message.call_method0("SerializeToString")?;
-    spark_connect_proto::ExecutePlanRequest::decode(
+    let mut request = spark_connect_proto::ExecutePlanRequest::decode(
         bytes.cast::<pyo3::types::PyBytes>()?.as_bytes(),
     )
-    .map_err(|error| 
pyo3::exceptions::PyValueError::new_err(error.to_string()))
+    .map_err(|error| 
pyo3::exceptions::PyValueError::new_err(error.to_string()))?;
+    // A hook may put in the plan of a DataFrame, which the Python plan sends 
as a placeholder.
+    spark_connect::plan::resolve_request(&mut request)
+        .map_err(|error| 
pyo3::exceptions::PyValueError::new_err(error.to_string()))?;
+    Ok(request)
 }
 
 /// Held while a session is got or created, as PySpark's `SparkSession._lock`, 
so that threads
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index b484216d..6807a12c 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1064,9 +1064,21 @@ impl DataFrame {
 
     /// A DataFrame of `relation`, a relation built outside the core, such as 
the plans of
     /// `pyspark.ml`, which the core sends as it is, keeping `owner`, if any, 
alive while any
-    /// DataFrame of it is. Not a stable API.
+    /// DataFrame of it is. The placeholders of `relation`, which stand for 
the DataFrames that
+    /// the plans of `pyspark.ml` take as input (see `placeholder_relation`), 
are replaced by
+    /// the relations of those DataFrames. Not a stable API.
     #[doc(hidden)]
     pub fn from_relation(
+        session: SparkSession,
+        mut relation: proto::Relation,
+        owner: Option<Arc<dyn std::any::Any + Send + Sync>>,
+    ) -> Result<DataFrame> {
+        crate::plan::resolve_relation(&mut relation)?;
+        Ok(DataFrame::from_raw_relation(session, relation, owner))
+    }
+
+    /// The DataFrame of `relation`, as `from_relation`, of a relation without 
placeholders.
+    fn from_raw_relation(
         session: SparkSession,
         relation: proto::Relation,
         owner: Option<Arc<dyn std::any::Any + Send + Sync>>,
@@ -1087,7 +1099,7 @@ impl DataFrame {
     #[doc(hidden)]
     pub fn from_sent_relation(session: SparkSession, relation: 
proto::Relation) -> DataFrame {
         let plan_id = relation.common.as_ref().and_then(|common| 
common.plan_id);
-        let dataframe = DataFrame::from_relation(session, relation, None);
+        let dataframe = DataFrame::from_raw_relation(session, relation, None);
         if let Some(plan_id) = plan_id {
             dataframe
                 .session
@@ -1104,6 +1116,16 @@ impl DataFrame {
         build_input_relation(&self.plan, &self.session)
     }
 
+    /// A relation that stands for the relation of this DataFrame, 
`to_relation`, in a relation
+    /// built outside the core, such as a plan of `pyspark.ml`, which the core 
replaces by that
+    /// relation when the relation reaches it (`from_relation`, 
`execute_command_proto`). It
+    /// is shallow, so it is cheap to build and to decode whatever the depth 
of this DataFrame.
+    /// Not a stable API.
+    #[doc(hidden)]
+    pub fn placeholder_relation(&self) -> proto::Relation {
+        crate::plan::placeholder_relation(&self.plan, 
self.session.plan_ids_shared())
+    }
+
     /// Get the schema of this DataFrame.
     ///
     /// The first successful lookup is cached and shared with exact clones.
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 5d398205..d675a4ca 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -6,7 +6,9 @@
 use spark_connect_proto as proto;
 use std::collections::HashMap;
 use std::sync::atomic::{AtomicI64, Ordering};
-use std::sync::{Arc, Mutex, Weak};
+use std::sync::{Arc, LazyLock, Mutex, Weak};
+
+use spark_connect_core::error::{Result, SparkError};
 
 use crate::column::Column;
 use crate::expression::{Expression, ExpressionNode, LiteralExpression};
@@ -598,6 +600,175 @@ impl PlanIds {
     }
 }
 
+/// The `Any` type of a placeholder: the relation that stands for the relation 
of a DataFrame, in a
+/// relation that is built outside the core, such as a plan of `pyspark.ml`, 
which the Python
+/// client builds over the placeholder of each DataFrame it takes as input. 
The value of a
+/// placeholder is its marker. The core replaces it by the relation of its 
DataFrame when the
+/// relation or the command that holds it reaches the core, so the nesting of 
a deep DataFrame is
+/// never decoded: a protobuf decoder limits the depth of what it decodes.
+const PLACEHOLDER_TYPE_URL: &str = 
"type.googleapis.com/spark.connect.rust.DataFramePlaceholder";
+
+/// The placeholders of all the sessions, by the addresses of their plan node 
and of their
+/// session's plan IDs, each with its marker. Both are held weakly, as 
`PlanIds` holds its nodes,
+/// so an entry goes with its DataFrame, and its address is not reused while 
the entry exists.
+#[derive(Default)]
+struct Placeholders {
+    entries: HashMap<(usize, usize), Placeholder>,
+    prune_at: usize,
+}
+
+struct Placeholder {
+    marker: i64,
+    node: Weak<LogicalPlan>,
+    ids: Weak<PlanIds>,
+}
+
+static PLACEHOLDERS: LazyLock<Mutex<Placeholders>> =
+    LazyLock::new(|| Mutex::new(Placeholders::default()));
+
+/// The placeholder of the relation of `node`, a DataFrame of the session of 
`ids`: the same
+/// marker each time the DataFrame gives it.
+fn placeholder_marker(node: &Arc<LogicalPlan>, ids: &Arc<PlanIds>) -> i64 {
+    const MIN_PRUNE_AT: usize = 1024;
+    let mut placeholders = PLACEHOLDERS.lock().unwrap_or_else(|e| 
e.into_inner());
+    // The entry's `Weak`s keep both allocations from being freed, so the node 
and the session
+    // of an entry at these addresses are those of this call.
+    let key = (Arc::as_ptr(node) as usize, Arc::as_ptr(ids) as usize);
+    if let Some(entry) = placeholders.entries.get(&key) {
+        return entry.marker;
+    }
+    if placeholders.entries.len() >= placeholders.prune_at.max(MIN_PRUNE_AT) {
+        placeholders
+            .entries
+            .retain(|_, entry| entry.node.strong_count() > 0 && 
entry.ids.strong_count() > 0);
+        placeholders.prune_at = placeholders.entries.len() * 2;
+    }
+    let marker = NEXT_PLAN_ID.fetch_add(1, Ordering::Relaxed);
+    placeholders.entries.insert(
+        key,
+        Placeholder {
+            marker,
+            node: Arc::downgrade(node),
+            ids: Arc::downgrade(ids),
+        },
+    );
+    marker
+}
+
+/// The node and the plan IDs of the placeholder `marker`, if its DataFrame is 
still alive.
+fn placeholder_target(marker: i64) -> Option<(Arc<LogicalPlan>, Arc<PlanIds>)> 
{
+    let placeholders = PLACEHOLDERS.lock().unwrap_or_else(|e| e.into_inner());
+    placeholders
+        .entries
+        .values()
+        .find(|entry| entry.marker == marker)
+        .and_then(|entry| Some((entry.node.upgrade()?, entry.ids.upgrade()?)))
+}
+
+/// The placeholder of the relation of `node`, a DataFrame of the session of 
`ids`, as the
+/// relation that the Python client sends in its place.
+pub(crate) fn placeholder_relation(node: &Arc<LogicalPlan>, ids: 
&Arc<PlanIds>) -> proto::Relation {
+    let marker = placeholder_marker(node, ids);
+    proto::Relation {
+        common: None,
+        rel_type: Some(proto::relation::RelType::Extension(prost_types::Any {
+            type_url: PLACEHOLDER_TYPE_URL.to_string(),
+            value: marker.to_le_bytes().to_vec(),
+        })),
+    }
+}
+
+/// Replace the placeholders of `relation` by the relations of their 
DataFrames, and of the
+/// relations that the plans of `pyspark.ml` take as input (their 
`ml_relation`, `with_relations`
+/// and fetched arguments). Each replacement is the relation of a DataFrame of 
the core, which has
+/// no placeholder.
+pub(crate) fn resolve_relation(relation: &mut proto::Relation) -> Result<()> {
+    if let Some(proto::relation::RelType::Extension(any)) = &relation.rel_type 
{
+        if any.type_url == PLACEHOLDER_TYPE_URL {
+            let marker = i64::from_le_bytes(
+                any.value
+                    .as_slice()
+                    .try_into()
+                    .map_err(|_| SparkError::value_msg("a placeholder of no 
marker"))?,
+            );
+            let (node, ids) = placeholder_target(marker).ok_or_else(|| {
+                SparkError::value_msg("a placeholder of a DataFrame that no 
longer exists")
+            })?;
+            *relation = relation_with_ids(&node, &ids);
+            return Ok(());
+        }
+    }
+    match relation.rel_type.as_mut() {
+        Some(proto::relation::RelType::MlRelation(ml)) => {
+            if let Some(summary) = ml.model_summary_dataset.as_mut() {
+                resolve_relation(summary)?;
+            }
+            match ml.ml_type.as_mut() {
+                Some(proto::ml_relation::MlType::Transform(transform)) => {
+                    if let Some(input) = transform.input.as_mut() {
+                        resolve_relation(input)?;
+                    }
+                }
+                Some(proto::ml_relation::MlType::Fetch(fetch)) => 
resolve_fetch(fetch)?,
+                None => {}
+            }
+        }
+        Some(proto::relation::RelType::WithRelations(with)) => {
+            if let Some(root) = with.root.as_mut() {
+                resolve_relation(root)?;
+            }
+            for reference in with.references.iter_mut() {
+                resolve_relation(reference)?;
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Replace the placeholders of the relations of the ML command `command`, as 
`resolve_relation`.
+pub(crate) fn resolve_command(command: &mut proto::Command) -> Result<()> {
+    let Some(proto::command::CommandType::MlCommand(ml)) = 
command.command_type.as_mut() else {
+        return Ok(());
+    };
+    let dataset = match ml.command.as_mut() {
+        Some(proto::ml_command::Command::Fit(fit)) => fit.dataset.as_mut(),
+        Some(proto::ml_command::Command::Evaluate(evaluate)) => 
evaluate.dataset.as_mut(),
+        Some(proto::ml_command::Command::CreateSummary(summary)) => 
summary.dataset.as_mut(),
+        Some(proto::ml_command::Command::Fetch(fetch)) => return 
resolve_fetch(fetch),
+        _ => None,
+    };
+    match dataset {
+        Some(dataset) => resolve_relation(dataset),
+        None => Ok(()),
+    }
+}
+
+/// Replace the placeholders of the relations that a fetch of `fetch` takes as 
arguments.
+fn resolve_fetch(fetch: &mut proto::Fetch) -> Result<()> {
+    for method in fetch.methods.iter_mut() {
+        for args in method.args.iter_mut() {
+            if let Some(proto::fetch::method::args::ArgsType::Input(input)) =
+                args.args_type.as_mut()
+            {
+                resolve_relation(input)?;
+            }
+        }
+    }
+    Ok(())
+}
+
+/// Replace the placeholders of the relation or of the command of `request`, a 
request that a
+/// session hook built in Python, as `DataFrame::from_relation` does. Not a 
stable API.
+#[doc(hidden)]
+pub fn resolve_request(request: &mut proto::ExecutePlanRequest) -> Result<()> {
+    match request.plan.as_mut().and_then(|plan| plan.op_type.as_mut()) {
+        Some(proto::plan::OpType::Root(relation)) => 
resolve_relation(relation),
+        Some(proto::plan::OpType::Command(command)) => 
resolve_command(command),
+        _ => Ok(()),
+    }
+}
+
 /// Serialize a shared plan node for a request: the node and every node below 
it carry
 /// their stable plan IDs.
 pub(crate) fn relation_with_ids(node: &Arc<LogicalPlan>, ids: &PlanIds) -> 
proto::Relation {
@@ -2519,3 +2690,282 @@ mod inline_udtf_tests {
         assert_eq!(udtf.return_type, None);
     }
 }
+
+#[cfg(test)]
+mod placeholder_tests {
+    use super::*;
+    use prost::Message;
+    use spark_connect_proto as proto;
+
+    /// The placeholder of `node`, of the session of `ids`, as the relation 
that the plans of
+    /// `pyspark.ml` send in its place.
+    fn placeholder(node: &Arc<LogicalPlan>, ids: &Arc<PlanIds>) -> 
proto::Relation {
+        placeholder_relation(node, ids)
+    }
+
+    /// The `ml_relation` of a transform of `input`, as 
`pyspark.ml.connect.proto` builds it.
+    fn transform_of(input: proto::Relation) -> proto::Relation {
+        proto::Relation {
+            common: None,
+            rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+                proto::MlRelation {
+                    model_summary_dataset: None,
+                    ml_type: 
Some(proto::ml_relation::MlType::Transform(Box::new(
+                        proto::ml_relation::Transform {
+                            input: Some(Box::new(input)),
+                            params: None,
+                            operator: None,
+                        },
+                    ))),
+                },
+            ))),
+        }
+    }
+
+    #[test]
+    fn a_placeholder_is_shallow_and_stands_for_the_relation_of_its_node() {
+        // The plan is converted recursively, so a plan of 200 levels takes 
more than the 2 MiB
+        // stack of a test thread in a debug build, whose frames are much 
larger than those of a
+        // release build.
+        let run = || {
+            let ids = Arc::new(PlanIds::new());
+            let mut node = Arc::new(range(0, 3, 1));
+            for _ in 0..200 {
+                node = Arc::new(LogicalPlan::Limit {
+                    input: node,
+                    limit: 3,
+                });
+            }
+            let relation = relation_with_ids(&node, &ids);
+            let shallow = placeholder(&node, &ids);
+            assert!(shallow.encoded_len() < 128, "{}", shallow.encoded_len());
+            let mut resolved = transform_of(shallow.clone());
+            resolve_relation(&mut resolved).unwrap();
+            assert_eq!(resolved, transform_of(relation));
+        };
+        std::thread::Builder::new()
+            .stack_size(64 * 1024 * 1024)
+            .spawn(run)
+            .unwrap()
+            .join()
+            .unwrap_or_else(|panic| std::panic::resume_unwind(panic));
+    }
+
+    #[test]
+    fn a_placeholder_is_the_same_for_the_same_node_and_session() {
+        let ids = Arc::new(PlanIds::new());
+        let node = Arc::new(range(0, 3, 1));
+        assert_eq!(placeholder(&node, &ids), placeholder(&node, &ids));
+        let other = Arc::new(PlanIds::new());
+        assert_ne!(placeholder(&node, &ids), placeholder(&node, &other));
+        let different = Arc::new(range(0, 4, 1));
+        assert_ne!(placeholder(&node, &ids), placeholder(&different, &ids));
+    }
+
+    #[test]
+    fn 
placeholders_resolve_in_every_position_that_the_ml_plans_build_over_their_inputs()
 {
+        let ids = Arc::new(PlanIds::new());
+        let node = Arc::new(range(0, 3, 1));
+        let relation = relation_with_ids(&node, &ids);
+        let shallow = placeholder(&node, &ids);
+        let fetch = || proto::Fetch {
+            obj_ref: None,
+            methods: vec![proto::fetch::Method {
+                method: "transform".to_string(),
+                args: vec![proto::fetch::method::Args {
+                    args_type: 
Some(proto::fetch::method::args::ArgsType::Input(shallow.clone())),
+                }],
+            }],
+        };
+        let mut relations = vec![
+            // An ML summary of a model, over its dataset.
+            proto::Relation {
+                common: None,
+                rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+                    proto::MlRelation {
+                        model_summary_dataset: Some(Box::new(shallow.clone())),
+                        ml_type: None,
+                    },
+                ))),
+            },
+            // A fetch of a summary, whose argument is a dataset.
+            proto::Relation {
+                common: None,
+                rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+                    proto::MlRelation {
+                        model_summary_dataset: None,
+                        ml_type: 
Some(proto::ml_relation::MlType::Fetch(fetch())),
+                    },
+                ))),
+            },
+            // Two transforms, one over the other, over the input of the 
second.
+            transform_of(transform_of(shallow.clone())),
+            // The references of a relation with its subqueries.
+            proto::Relation {
+                common: None,
+                rel_type: 
Some(proto::relation::RelType::WithRelations(Box::new(
+                    proto::WithRelations {
+                        root: Some(Box::new(shallow.clone())),
+                        references: vec![shallow.clone()],
+                    },
+                ))),
+            },
+        ];
+        for relation_with_placeholder in relations.iter_mut() {
+            resolve_relation(relation_with_placeholder).unwrap();
+        }
+        // The placeholders are replaced by the relation of the node, 
everywhere they were.
+        let expected = [
+            proto::Relation {
+                common: None,
+                rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+                    proto::MlRelation {
+                        model_summary_dataset: 
Some(Box::new(relation.clone())),
+                        ml_type: None,
+                    },
+                ))),
+            },
+            proto::Relation {
+                common: None,
+                rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+                    proto::MlRelation {
+                        model_summary_dataset: None,
+                        ml_type: 
Some(proto::ml_relation::MlType::Fetch(proto::Fetch {
+                            obj_ref: None,
+                            methods: vec![proto::fetch::Method {
+                                method: "transform".to_string(),
+                                args: vec![proto::fetch::method::Args {
+                                    args_type: 
Some(proto::fetch::method::args::ArgsType::Input(
+                                        relation.clone(),
+                                    )),
+                                }],
+                            }],
+                        })),
+                    },
+                ))),
+            },
+            transform_of(transform_of(relation.clone())),
+            proto::Relation {
+                common: None,
+                rel_type: 
Some(proto::relation::RelType::WithRelations(Box::new(
+                    proto::WithRelations {
+                        root: Some(Box::new(relation.clone())),
+                        references: vec![relation.clone()],
+                    },
+                ))),
+            },
+        ];
+        assert_eq!(relations, expected);
+    }
+
+    #[test]
+    fn the_ml_commands_resolve_their_datasets_and_fetched_arguments() {
+        let ids = Arc::new(PlanIds::new());
+        let node = Arc::new(range(0, 3, 1));
+        let relation = relation_with_ids(&node, &ids);
+        let shallow = placeholder(&node, &ids);
+        let command = |ml_command: proto::ml_command::Command| proto::Command {
+            command_type: 
Some(proto::command::CommandType::MlCommand(proto::MlCommand {
+                command: Some(ml_command),
+            })),
+        };
+        let fit = |dataset: proto::Relation| {
+            command(proto::ml_command::Command::Fit(proto::ml_command::Fit {
+                estimator: None,
+                params: None,
+                dataset: Some(dataset),
+            }))
+        };
+        let mut fitted = fit(shallow.clone());
+        resolve_command(&mut fitted).unwrap();
+        assert_eq!(fitted, fit(relation.clone()));
+
+        let mut evaluated = command(proto::ml_command::Command::Evaluate(
+            proto::ml_command::Evaluate {
+                evaluator: None,
+                params: None,
+                dataset: Some(shallow.clone()),
+            },
+        ));
+        resolve_command(&mut evaluated).unwrap();
+        assert_eq!(
+            evaluated,
+            command(proto::ml_command::Command::Evaluate(
+                proto::ml_command::Evaluate {
+                    evaluator: None,
+                    params: None,
+                    dataset: Some(relation.clone()),
+                },
+            ))
+        );
+
+        let mut summary = command(proto::ml_command::Command::CreateSummary(
+            proto::ml_command::CreateSummary {
+                model_ref: None,
+                dataset: Some(shallow.clone()),
+            },
+        ));
+        resolve_command(&mut summary).unwrap();
+        assert_eq!(
+            summary,
+            command(proto::ml_command::Command::CreateSummary(
+                proto::ml_command::CreateSummary {
+                    model_ref: None,
+                    dataset: Some(relation.clone()),
+                },
+            ))
+        );
+
+        // A request of a session hook: the root of its plan, or its command.
+        let mut root = proto::ExecutePlanRequest {
+            plan: Some(proto::Plan {
+                op_type: 
Some(proto::plan::OpType::Root(transform_of(shallow.clone()))),
+            }),
+            ..Default::default()
+        };
+        resolve_request(&mut root).unwrap();
+        assert_eq!(
+            root.plan.unwrap().op_type,
+            Some(proto::plan::OpType::Root(transform_of(relation)))
+        );
+    }
+
+    #[test]
+    fn relations_without_placeholders_are_left_as_they_are() {
+        let ids = Arc::new(PlanIds::new());
+        let node = Arc::new(range(0, 3, 1));
+        let mut relation = relation_with_ids(&node, &ids);
+        let before = relation.clone();
+        resolve_relation(&mut relation).unwrap();
+        assert_eq!(relation, before);
+        let mut command = proto::Command::default();
+        resolve_command(&mut command).unwrap();
+        assert_eq!(command, proto::Command::default());
+    }
+
+    #[test]
+    fn a_placeholder_of_a_dropped_node_is_an_error_and_keeps_nothing_alive() {
+        let ids = Arc::new(PlanIds::new());
+        let node = Arc::new(range(0, 3, 1));
+        let alive = Arc::downgrade(&node);
+        let shallow = placeholder(&node, &ids);
+        std::mem::drop(node);
+        assert!(alive.upgrade().is_none());
+        let mut relation = transform_of(shallow);
+        let error = resolve_relation(&mut relation).unwrap_err();
+        assert!(error.to_string().contains("no longer exists"), "{error}");
+    }
+
+    #[test]
+    fn a_malformed_placeholder_is_an_error() {
+        let mut relation = proto::Relation {
+            common: None,
+            rel_type: 
Some(proto::relation::RelType::Extension(prost_types::Any {
+                type_url: PLACEHOLDER_TYPE_URL.to_string(),
+                value: vec![1, 2, 3],
+            })),
+        };
+        let error = resolve_relation(&mut relation).unwrap_err();
+        assert!(error.to_string().contains("no marker"), "{error}");
+    }
+}
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 048e3e37..4ae698e0 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -372,6 +372,10 @@ impl SparkSession {
         &self.plan_ids
     }
 
+    pub(crate) fn plan_ids_shared(&self) -> &Arc<PlanIds> {
+        &self.plan_ids
+    }
+
     /// Get the underlying client.
     pub(crate) fn client(&self) -> &Arc<SparkConnectClient> {
         &self.client
@@ -788,12 +792,14 @@ impl SparkSession {
 
     /// Execute `command`, a command built outside the core, such as the ML 
commands of
     /// `pyspark.ml`, and return the responses of the server, as the Python 
client's
-    /// `execute_command` reads them. Not a stable API.
+    /// `execute_command` reads them. The placeholders of its relations are 
replaced, as
+    /// `DataFrame::from_relation` does. Not a stable API.
     #[doc(hidden)]
     pub fn execute_command_proto(
         &self,
-        command: spark_connect_proto::Command,
+        mut command: spark_connect_proto::Command,
     ) -> Result<Vec<spark_connect_proto::ExecutePlanResponse>> {
+        crate::plan::resolve_command(&mut command)?;
         let Some(command_type) = command.command_type else {
             return Err(SparkError::value_msg("a command of no command type"));
         };
diff --git a/crates/spark-connect/tests/raw_relation.rs 
b/crates/spark-connect/tests/raw_relation.rs
index 90dfe2df..c472dff8 100644
--- a/crates/spark-connect/tests/raw_relation.rs
+++ b/crates/spark-connect/tests/raw_relation.rs
@@ -44,6 +44,18 @@ fn live() -> Option<SparkSession> {
     )
 }
 
+/// Runs `f` on a thread with a stack of 64 MiB. A plan is converted 
recursively, so a deep plan
+/// takes more than the 2 MiB stack of a test thread in a debug build, whose 
frames are much
+/// larger than those of a release build.
+fn on_a_large_stack(f: impl FnOnce() + Send + 'static) {
+    std::thread::Builder::new()
+        .stack_size(64 * 1024 * 1024)
+        .spawn(f)
+        .unwrap()
+        .join()
+        .unwrap_or_else(|panic| std::panic::resume_unwind(panic));
+}
+
 fn sql_relation(query: &str) -> proto::Relation {
     proto::Relation {
         common: Some(proto::RelationCommon {
@@ -59,7 +71,7 @@ fn sql_relation(query: &str) -> proto::Relation {
 
 #[test]
 fn raw_relations_are_sent_as_they_are_with_a_plan_id_of_the_session() {
-    let df = DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS 
a"), None);
+    let df = DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS 
a"), None).unwrap();
     let relation = df.to_relation();
     let Some(proto::relation::RelType::Sql(sql)) = &relation.rel_type else {
         panic!("expected the SQL relation, got {relation:?}");
@@ -74,7 +86,7 @@ fn 
raw_relations_are_sent_as_they_are_with_a_plan_id_of_the_session() {
 
 #[test]
 fn transformations_of_raw_relations_take_them_as_input() {
-    let df = DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS 
a"), None);
+    let df = DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS 
a"), None).unwrap();
     let relation = df.to_relation();
     let filtered = df.filter(col("a").gt(lit(0)));
     let Some(proto::relation::RelType::Filter(filter)) = 
filtered.to_relation().rel_type else {
@@ -87,7 +99,8 @@ fn transformations_of_raw_relations_take_them_as_input() {
 fn raw_relations_keep_their_owners_alive_while_a_dataframe_uses_them() {
     let owner: Arc<dyn Any + Send + Sync> = Arc::new(String::from("plan"));
     let alive = Arc::downgrade(&owner);
-    let df = DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS 
a"), Some(owner));
+    let df =
+        DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS a"), 
Some(owner)).unwrap();
     let derived = df.select([col("a")]);
     drop(df);
     assert!(alive.upgrade().is_some());
@@ -119,6 +132,92 @@ fn 
sent_relations_keep_the_plan_ids_of_the_dataframes_they_were_of() {
     assert_ne!(other.common.and_then(|common| common.plan_id), plan_id);
 }
 
+#[test]
+fn placeholders_of_a_dataframe_are_replaced_by_its_relation() {
+    on_a_large_stack(|| {
+        // As the plans of pyspark.ml over the DataFrames that they take as 
input: a relation over
+        // the placeholder of a DataFrame, which is replaced by that 
DataFrame's relation.
+        let session = offline();
+        let mut df = session.range(3).unwrap();
+        for i in 0..120 {
+            df = df.with_column(&format!("c{i}"), col("id").add(lit(i)));
+        }
+        let placeholder = df.placeholder_relation();
+        // Shallow: the placeholder of a DataFrame of 120 levels is a few 
bytes.
+        assert!(prost::Message::encoded_len(&placeholder) < 128);
+        let input = DataFrame::from_relation(
+            session.clone(),
+            proto::Relation {
+                common: None,
+                rel_type: 
Some(proto::relation::RelType::WithRelations(Box::new(
+                    proto::WithRelations {
+                        root: Some(Box::new(placeholder.clone())),
+                        references: vec![placeholder.clone()],
+                    },
+                ))),
+            },
+            None,
+        )
+        .unwrap();
+        let Some(proto::relation::RelType::WithRelations(with)) = 
input.to_relation().rel_type
+        else {
+            panic!("expected WithRelations");
+        };
+        assert_eq!(with.root.as_deref(), Some(&df.to_relation()));
+        assert_eq!(with.references, vec![df.to_relation()]);
+        // The same placeholder gives the same relation.
+        assert_eq!(df.placeholder_relation(), placeholder);
+        // The DataFrame of the placeholder is a node of its own, whose 
relation is that of `df`.
+        let single = DataFrame::from_relation(session.clone(), 
placeholder.clone(), None).unwrap();
+        assert_eq!(single.to_relation().rel_type, df.to_relation().rel_type);
+        // A placeholder of a DataFrame of another session is the relation of 
that DataFrame too.
+        let other = offline();
+        let copy = DataFrame::from_relation(other, placeholder, None).unwrap();
+        assert_eq!(copy.to_relation().rel_type, df.to_relation().rel_type);
+    });
+}
+
+#[test]
+fn placeholders_of_dropped_dataframes_are_errors() {
+    let session = offline();
+    let df = session.range(3).unwrap();
+    let placeholder = df.placeholder_relation();
+    drop(df);
+    let Err(error) = DataFrame::from_relation(session, placeholder, None) else 
{
+        panic!("a placeholder of a dropped DataFrame resolves");
+    };
+    assert!(error.to_string().contains("no longer exists"), "{error}");
+}
+
+#[test]
+fn a_deep_placeholder_runs_on_the_server() {
+    on_a_large_stack(|| {
+        let Some(session) = live() else {
+            return;
+        };
+        let mut df = session.range(3).unwrap();
+        for i in 0..120 {
+            df = df.with_column(&format!("c{i}"), col("id").add(lit(i)));
+        }
+        let wrapped = DataFrame::from_relation(
+            session.clone(),
+            proto::Relation {
+                common: None,
+                rel_type: 
Some(proto::relation::RelType::WithRelations(Box::new(
+                    proto::WithRelations {
+                        root: Some(Box::new(df.placeholder_relation())),
+                        references: vec![],
+                    },
+                ))),
+            },
+            None,
+        )
+        .unwrap();
+        assert_eq!(wrapped.count().unwrap(), 3);
+        assert_eq!(wrapped.columns().unwrap().len(), 121);
+    });
+}
+
 #[test]
 fn commands_of_no_command_type_are_errors() {
     let error = offline()
@@ -136,7 +235,8 @@ fn raw_relations_run_on_the_server() {
         session.clone(),
         sql_relation("SELECT id AS a FROM range(3)"),
         None,
-    );
+    )
+    .unwrap();
     assert_eq!(df.columns().unwrap(), vec!["a".to_string()]);
     assert_eq!(df.clone().filter(col("a").gt(lit(0))).count().unwrap(), 2);
     let other = session.range(2).unwrap();
diff --git a/python/pyspark/sql/connect/plan.py 
b/python/pyspark/sql/connect/plan.py
index e65d150e..a261a56f 100644
--- a/python/pyspark/sql/connect/plan.py
+++ b/python/pyspark/sql/connect/plan.py
@@ -128,14 +128,19 @@ class LogicalPlan:
 
 class _DataFramePlan(LogicalPlan):
     """The plan of a DataFrame of the Rust core, as its ``DataFrame._plan``: 
its relation, as the
-    core sends it, of the plan IDs of its session."""
+    core sends it, of the plan IDs of its session.
+
+    Its ``plan`` is a placeholder of that relation, which the core replaces by 
the relation when
+    the plan built over it reaches the core, as ``DataFrame(plan, session)`` 
or a command does.
+    So a plan of ``pyspark.ml`` over a deep DataFrame is never decoded in 
Python or in Rust, as
+    protobuf decoders limit the depth of what they decode."""
 
     def __init__(self, df: Any) -> None:
         super().__init__(None)
         self._df = df
 
     def plan(self, session: Any) -> "proto.Relation":
-        return proto.Relation.FromString(self._df._relation_bytes())
+        return proto.Relation.FromString(self._df._relation_placeholder())
 
 
 class Read(LogicalPlan):
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 52eaa5ac..699bdcb2 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -10372,6 +10372,65 @@ def 
test_arrow_input_explicit_schema_matching_width(schema, values):
     assert names is None
 
 
+def 
test_ml_plans_send_a_deep_dataframe_without_parsing_it_as_pyspark(mock_connect):
+    # As PySpark's pyspark.ml, which builds its plans over the relation of a 
DataFrame without
+    # parsing it: a DataFrame of 120 chained withColumn calls is nested deeper 
than a protobuf
+    # decoder parses. A plan over it holds a shallow placeholder of that 
relation, which the core
+    # replaces with the relation when the plan reaches it, so the server 
receives the relation.
+    import gc
+
+    import pyspark.sql.connect.proto as pb2
+    from pyspark.sql import functions as F
+    from pyspark.sql.connect.dataframe import DataFrame
+    from pyspark.sql.connect.plan import LogicalPlan
+
+    class Transform(LogicalPlan):
+        # The plan of a transformer of pyspark.ml.connect.proto, over its 
input.
+        def __init__(self, child, name):
+            super().__init__(child)
+            self._name = name
+
+        def plan(self, session):
+            plan = self._create_proto_relation()
+            
plan.ml_relation.transform.input.CopyFrom(self._child.plan(session))
+            
plan.ml_relation.transform.obj_ref.CopyFrom(pb2.ObjectRef(id=self._name))
+            return plan
+
+    spark, service = mock_connect
+    deep = spark.range(2)
+    for i in range(120):
+        deep = deep.withColumn(f"c{i}", F.lit(i))
+    relation = deep._relation_bytes()
+    # The placeholder of the relation is shallow, whatever the depth of the 
DataFrame.
+    assert len(deep._plan.plan(spark.client).SerializeToString()) < 128
+    # Two plans of transformers, one over the other, over the DataFrame. The 
request is read as
+    # bytes, as the deep relation is not parsed by protobuf: 
ExecutePlanRequest.plan=3,
+    # Plan.root=1, Relation.ml_relation=300, MlRelation.transform=1, 
Transform.input=3,
+    # Transform.obj_ref=1 and ObjectRef.id=1.
+    nested = DataFrame(Transform(Transform(deep._plan, "m1"), "m2"), spark)
+    nested.collect()
+    root = 
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+    outer = parse_message(parse_message(root)[300][0])[1][0]
+    assert parse_message(parse_message(outer)[1][0]) == {1: [b"m2"]}
+    inner = parse_message(parse_message(outer)[3][0])
+    inner_transform = parse_message(parse_message(inner[300][0])[1][0])
+    assert parse_message(inner_transform[1][0]) == {1: [b"m1"]}
+    assert inner_transform[3][0] == relation
+
+    # A placeholder of a DataFrame that is gone is an error, and keeps nothing 
alive.
+    gone = spark.range(3)
+    placeholder = gone._plan.plan(spark.client)
+    del gone
+    gc.collect()
+
+    class Gone(LogicalPlan):
+        def plan(self, session):
+            return placeholder
+
+    with pytest.raises(Exception, match="no longer exists"):
+        DataFrame(Gone(None), spark)
+
+
 @pytest.mark.parametrize("schema", [
     T.StructType([]),
     T.StructType([T.StructField("x", T.LongType()), T.StructField("y", 
T.LongType())]),
diff --git a/python/tests/test_ml_e2e.py b/python/tests/test_ml_e2e.py
index 11e5187c..12864f11 100644
--- a/python/tests/test_ml_e2e.py
+++ b/python/tests/test_ml_e2e.py
@@ -240,6 +240,39 @@ def 
test_ml_transformed_dataframes_keep_their_models_as_pyspark(session):
     assert error.value.getCondition() == "CONNECT_ML.CACHE_INVALID"
 
 
+def test_ml_deep_dataframes_fit_and_transform_as_pyspark(session):
+    # As PySpark: a DataFrame of 120 chained withColumn calls, nested deeper 
than a protobuf
+    # decoder parses, is fitted and transformed, and so is OneVsRest over 12 
classes, whose
+    # transform adds a layer per class. Their predictions are those of the 
shallow DataFrames.
+    # The chains are over a repartition of the local data: over a local 
relation, the optimizer
+    # folds one projection per iteration, which a server in testing mode 
(SPARK_TESTING, as in
+    # CI) fails past its 100 iterations, for PySpark too.
+    from pyspark.ml.classification import LogisticRegression, OneVsRest
+    from pyspark.ml.linalg import Vectors
+    from pyspark.sql import functions as F
+
+    rows = [(float(i % 2), Vectors.dense([float(i % 7), float(i % 5)])) for i 
in range(40)]
+    base = session.createDataFrame(rows, ["label", "features"]).repartition(1)
+    deep = base
+    for i in range(120):
+        deep = deep.withColumn(f"c{i}", F.lit(i))
+    model = LogisticRegression(maxIter=2).fit(deep)
+    assert model.coefficients == 
LogisticRegression(maxIter=2).fit(base).coefficients
+    predictions = sorted(r.prediction for r in model.transform(deep).collect())
+    assert len(predictions) == 40
+    assert predictions == sorted(r.prediction for r in 
model.transform(base).collect())
+
+    rows12 = [(float(i % 12), Vectors.dense([float(i % 7), float(i % 5)])) for 
i in range(60)]
+    base12 = session.createDataFrame(rows12, ["label", 
"features"]).repartition(1)
+    deep12 = base12
+    for i in range(120):
+        deep12 = deep12.withColumn(f"c{i}", F.lit(i))
+    ovr = OneVsRest(classifier=LogisticRegression(maxIter=2)).fit(base12)
+    shallow = sorted(r.prediction for r in ovr.transform(base12).collect())
+    assert sorted(r.prediction for r in ovr.transform(deep12).collect()) == 
shallow
+    assert len(shallow) == 60
+
+
 def test_ml_dataframes_of_plans_as_pyspark(session):
     # As PySpark Connect's DataFrame(plan, session): a DataFrame of the 
relation of a plan,
     # which further transformations take as input.
@@ -269,7 +302,9 @@ def 
test_session_hooks_rewrite_the_requests_sent_to_the_server_as_pyspark(sessio
     import pyspark.sql.connect.proto as pb2
     from pyspark.sql.connect.session import SparkSession as ConnectSparkSession
 
-    replacement = session.range(5)._plan.plan(session.client)
+    # The placeholder of a plan is valid while its DataFrame lives, so the 
DataFrame is kept.
+    source = session.range(5)
+    replacement = source._plan.plan(session.client)
     calls = []
 
     class Replace(ConnectSparkSession.Hook):
diff --git a/python/tests/test_ml_offline.py b/python/tests/test_ml_offline.py
index bb02c634..a9d97e9e 100644
--- a/python/tests/test_ml_offline.py
+++ b/python/tests/test_ml_offline.py
@@ -109,6 +109,12 @@ def _commands(service):
     return [pb2.ExecutePlanRequest.FromString(r).plan.command for r in 
service.execute_requests]
 
 
+def _relation(df):
+    # The relation of a DataFrame, as the core sends it: the relation that the 
plans of pyspark.ml
+    # send in place of its placeholder, which the core replaces with it.
+    return pb2.Relation.FromString(df._relation_bytes())
+
+
 class _Color(enum.Enum):
     RED = 1
 
@@ -468,7 +474,7 @@ def test_connect_conversions_as_pyspark(ml_mock):
     assert proto_to_storage_level(message) == level
 
     df = 
proto_to_remote_cached_dataframe(pb2.CachedRemoteRelation(relation_id="r1"))
-    assert df._plan.plan(spark.client).cached_remote_relation.relation_id == 
"r1"
+    assert _relation(df).cached_remote_relation.relation_id == "r1"
     assert service.execute_requests == []
     del df
     gc.collect()
@@ -622,7 +628,7 @@ def 
test_execute_command_converts_the_results_of_pyspark(ml_mock):
         )
     ).SerializeToString()
     df = client.execute_command(command)[1]["checkpoint_command_result"]
-    assert df._plan.plan(client).cached_remote_relation.relation_id == "r9"
+    assert _relation(df).cached_remote_relation.relation_id == "r9"
     # The relation is removed from the server once the DataFrame is collected, 
of a command
     # whose responses are ignored, as PySpark's: so the checkpoint that the 
mock server answers
     # with makes no other relation to remove.
@@ -702,20 +708,20 @@ def 
test_ml_fit_and_transform_send_the_messages_of_pyspark(ml_mock):
         "inputCol": pb2.Expression.Literal(string="words"),
         "outputCol": pb2.Expression.Literal(string="counts"),
     }
-    assert fit.dataset == df._plan.plan(spark.client)
+    assert fit.dataset == _relation(df)
     assert fit.dataset.HasField("project")
 
     assert str(model._java_obj) == "model-1"
     assert model.uid == estimator.uid and model.getVocabSize() == 5
 
     predictions = model.transform(df)
-    relation = predictions._plan.plan(spark.client)
+    relation = _relation(predictions)
     transform = relation.ml_relation.transform
     assert transform.obj_ref.id == "model-1"
     assert transform.input == fit.dataset
     assert transform.params.params["vocabSize"] == 
pb2.Expression.Literal(integer=5)
 
-    selected = predictions.select("counts")._plan.plan(spark.client)
+    selected = _relation(predictions.select("counts"))
     assert selected.project.input == relation
     # A column of the transformed DataFrame refers to the plan ID of its 
relation.
     reference = predictions["counts"]._expr.to_plan(None).unresolved_attribute
@@ -731,12 +737,12 @@ def 
test_ml_transformers_send_the_relation_of_pyspark(ml_mock):
     spark, service = ml_mock
     df = spark.range(1).selectExpr("'a b' AS text")
     tokenizer = Tokenizer(inputCol="text", outputCol="words")
-    transform = 
tokenizer.transform(df)._plan.plan(spark.client).ml_relation.transform
+    transform = _relation(tokenizer.transform(df)).ml_relation.transform
     assert transform.transformer.name == 
"org.apache.spark.ml.feature.Tokenizer"
     assert transform.transformer.uid == tokenizer.uid
     assert transform.transformer.type == 
pb2.MlOperator.OPERATOR_TYPE_TRANSFORMER
     assert transform.params.params["inputCol"] == 
pb2.Expression.Literal(string="text")
-    assert transform.input == df._plan.plan(spark.client)
+    assert transform.input == _relation(df)
     assert service.execute_requests == []
 
 
@@ -761,14 +767,14 @@ def test_ml_dataframes_of_plans_keep_their_plans(ml_mock):
     plan = SQLPlan()
     alive = weakref.ref(plan)
     df = DataFrame(plan=plan, session=spark)
-    relation = df._plan.plan(spark.client)
+    relation = _relation(df)
     assert relation.sql.query == "SELECT 1 AS a"
     assert df["a"]._expr.to_plan(None).unresolved_attribute.plan_id == 
relation.common.plan_id
     derived = df.filter("a > 0")
     del plan, df
     gc.collect()
     assert alive() is not None
-    assert derived._plan.plan(spark.client).filter.input == relation
+    assert _relation(derived).filter.input == relation
     del derived
     gc.collect()
     assert alive() is None


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to