This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 915c741e [SPARK-61202][CONNECT][PYTHON] Send the relation of a
DataFrame into the plans of pyspark.ml without parsing it, so deep DataFrames
fit and transform, in the Python client as PySpark does
915c741e is described below
commit 915c741ef420a9b939c2ba2553c0cc96aa304c2e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:07:23 2026 +0900
[SPARK-61202][CONNECT][PYTHON] Send the relation of a DataFrame into the
plans of pyspark.ml without parsing it, so deep DataFrames fit and transform,
in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61202](https://issues.apache.org/jira/browse/SPARK-61202): Send the
relation of a DataFrame into the plans of `pyspark.ml` without parsing it, so
that a deep DataFrame fits and transforms, as PySpark does.
The plans of `pyspark.ml` took the relation of a DataFrame by parsing it
twice. `_DataFramePlan.plan` decoded it with the Python protobuf decoder, and
`DataFrame(plan, session)` and the ML commands decoded the built plan with
prost. Both decoders refuse more than 100 levels of nesting, so a DataFrame
deeper than that could not be fitted or transformed. PySpark builds the nested
message without parsing it, so it works at these depths.
The change:
- `_DataFramePlan.plan` returns a placeholder of the relation of its
DataFrame: a shallow `Relation` whose `extension` is a marker of the DataFrame.
The vendored plans of `pyspark.ml` build over it as they do over a relation, so
the plan of a transform over a deep DataFrame is shallow too.
- The core keeps a registry of the placeholders: each marker names the plan
node of its DataFrame and the plan IDs of its session, both held weakly, and an
entry is pruned with its DataFrame, as `PlanIds` prunes its nodes. The same
DataFrame gives the same marker.
- The core replaces each placeholder with the relation of its DataFrame,
built as `DataFrame.to_relation` builds it, so it is the relation that is sent
today. The replacement is moved in, never decoded. It happens where a
Python-built relation or command reaches the core:
- `DataFrame::from_relation` (`DataFrame(plan, session)`), in the
positions that the ML plans build over their inputs:
`ml_relation.transform.input`, `ml_relation.model_summary_dataset`, the
arguments of `fetch`, and the `root` and `references` of `with_relations`;
- `execute_command_proto` (`session.client.execute_command`), in the
datasets of the `fit`, `evaluate` and `create_summary` commands and in the
arguments of `fetch`;
- the session hooks, for the root or the command of the request they
return.
- The owner of a DataFrame of a plan is still the Python plan, so the model
keep-alive of SPARK-61158 is unchanged.
What server receives is the same relation as before this change: the same
plan IDs and the same nesting, with no placeholder left. Official PySpark 4.3.0
sends the same nested relation, modulo plan IDs, because it also builds it
without parsing it.
Why not raise the limits: prost's `no-recursion-limit` feature removes the
check on input that is not trusted, and a stack overflow is then a crash of the
process. upb (the Python decoder) has no public setting for its limit. A larger
constant only moves the failure: a OneVsRest transform adds layers per class,
so any fixed limit fails at some class count. The placeholder does not decode
the nesting at all, so its depth is bounded only by the stack of the encoder,
as it is for `count` [...]
Known differences:
- A placeholder is valid while its DataFrame is alive, as its DataFrame is
kept by the plan that uses it. `df._plan.plan(client)` is therefore no longer a
self-contained relation. It is internal, the plans of `pyspark.ml` are the
users. The hook test of `test_ml_e2e.py` keeps its source DataFrame for this
reason.
- A placeholder that a user puts under a relation that is not one of the
positions above reaches the server unresolved, and the server reports an error
for it. Such a relation is not built by PySpark or by the vendored code.
- Not changed here: `DataFrame._of_relation`, used when a DataFrame is
unpickled, and the session hooks, which encode the request into Python and
decode it back. Both still parse a deep relation: a hook on a DataFrame of 120
`withColumn` calls raises `DecodeError` with or without this change (checked on
this branch). They are not on the path of `pyspark.ml`, and they are a
follow-up.
### Why are the changes needed?
A DataFrame of about 50 chained `withColumn` calls cannot be fitted:
`LogisticRegression(maxIter=2).fit(df)` raises `DecodeError: Error parsing
message: too many levels of nesting`. A `transform` of a `LogisticRegression`
model over the same DataFrame raises the same error, and a
`OneVsRestModel.transform` with 10 or more classes too, since each class adds
layers. A pipeline of 60 `SQLTransformer` stages fails with `ValueError: ...
recursion limit reached`. `count` and `collect` work [...]
### Does this PR introduce _any_ user-facing change?
Yes. `pyspark.ml` fits and transforms DataFrames of any depth that PySpark
accepts, and `DataFrame._plan.plan(client)` returns a placeholder (see the
known differences). The sent requests are the same as before for the DataFrames
that worked.
### How was this patch tested?
- Rust unit tests in `crates/spark-connect/src/plan.rs`
(`placeholder_tests`, 7 tests): a placeholder is shallow and resolves to the
relation of its node; the same node and session give the same placeholder; the
placeholders of the ML positions (`model_summary_dataset`, `transform.input`,
the arguments of `fetch`, `with_relations`) and of the ML commands (`fit`,
`evaluate`, `create_summary`, `fetch`) and of a hook's root are replaced;
relations without placeholders are unchanged; a pl [...]
- Rust integration tests in `crates/spark-connect/tests/raw_relation.rs` (3
new): a placeholder of a 120-level DataFrame is replaced by its relation in
`from_relation`, with the same plan IDs; a placeholder of a dropped DataFrame
is an error; a live test runs a 120-level placeholder on the server.
- The tests of deep plans (the 200-level unit test and the two 120-level
`raw_relation.rs` tests) run on a thread with a 64 MiB stack. The plan is
converted recursively, and in a debug build, as in the WASM UDF and coverage
jobs, a plan of that depth takes more than the 2 MiB stack of a test thread. In
debug builds, the lib tests (485 passed) and `raw_relation.rs` against a local
Spark 4.3.0 server (10 passed) pass.
- Offline Python test
`test_ml_plans_send_a_deep_dataframe_without_parsing_it_as_pyspark` in
`python/tests/test_dropin_offline.py`: two transformer plans over a 120-level
DataFrame; the request that the mock server receives carries the relation of
the DataFrame byte for byte, and a placeholder of a dropped DataFrame is an
error.
- Live Python test `test_ml_deep_dataframes_fit_and_transform_as_pyspark`
in `python/tests/test_ml_e2e.py`: `LogisticRegression` fits and transforms a
120-level DataFrame with the predictions of the shallow DataFrame, and
`OneVsRest` with 12 classes transforms a 120-level DataFrame with the
predictions of the shallow one. The DataFrames are over a repartition of their
local data: over a local relation, the optimizer folds one projection per
iteration, and a server in testing mode (`SP [...]
- Existing tests that read `_plan.plan(...)` as a relation now read the
relation that the server receives (`_relation` in
`python/tests/test_ml_offline.py`, 9 places), and the session hook test keeps
its source DataFrame.
- The tests of both crates, e2e included, against a local Spark 4.3.0
server (926 passed), the build of all tests, also with `wasm-udf`, the offline
Python suites (2457 passed, 8 skipped), and the live drop-in, Arrow and ML
suites (218 passed) and live cases (11 passed). rustfmt passes, and clippy
reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #387 from HyukjinKwon/SPARK-61202-ml-plans-deep-dataframes.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 17 +-
crates/pyspark-rs/src/session.rs | 8 +-
crates/spark-connect/src/dataframe.rs | 26 +-
crates/spark-connect/src/plan.rs | 452 ++++++++++++++++++++++++++++-
crates/spark-connect/src/session.rs | 10 +-
crates/spark-connect/tests/raw_relation.rs | 108 ++++++-
python/pyspark/sql/connect/plan.py | 9 +-
python/tests/test_dropin_offline.py | 59 ++++
python/tests/test_ml_e2e.py | 37 ++-
python/tests/test_ml_offline.py | 24 +-
10 files changed, 722 insertions(+), 28 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index b98ffa34..39d1789c 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -648,11 +648,10 @@ impl PyDataFrame {
// objects on the server the relation refers to
(`__source_instance__`).
let owner: std::sync::Arc<dyn std::any::Any + Send + Sync> =
std::sync::Arc::new(plan.clone().unbind());
- Ok(PyDataFrame::new(DataFrame::from_relation(
- session.borrow().session.clone(),
- relation,
- Some(owner),
- )))
+ Ok(PyDataFrame::new(
+ DataFrame::from_relation(session.borrow().session.clone(),
relation, Some(owner))
+ .to_pyerr()?,
+ ))
}
/// The bytes of the relation of this DataFrame as it is sent, for the
plan of `_plan`.
@@ -661,6 +660,14 @@ impl PyDataFrame {
pyo3::types::PyBytes::new(py,
&self.dataframe.to_relation().encode_to_vec())
}
+ /// The bytes of the placeholder of the relation of this DataFrame, which
the plan of `_plan`
+ /// sends in its place: shallow, whatever the depth of the DataFrame, and
replaced by the
+ /// core with that relation when the plan reaches it.
+ fn _relation_placeholder<'py>(&self, py: Python<'py>) -> Bound<'py,
pyo3::types::PyBytes> {
+ use prost::Message;
+ pyo3::types::PyBytes::new(py,
&self.dataframe.placeholder_relation().encode_to_vec())
+ }
+
/// A DataFrame pickles, as PySpark Connect's, of its session, which gives
back the active
/// session of its ID, and of its plan, here the bytes of its relation.
The session unpickles
/// first: so on a Python worker of the server, which has no active
session, a pickled
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 128558b4..d1c963f5 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -425,10 +425,14 @@ fn call_execute_plan_hooks(
}
}
let bytes = message.call_method0("SerializeToString")?;
- spark_connect_proto::ExecutePlanRequest::decode(
+ let mut request = spark_connect_proto::ExecutePlanRequest::decode(
bytes.cast::<pyo3::types::PyBytes>()?.as_bytes(),
)
- .map_err(|error|
pyo3::exceptions::PyValueError::new_err(error.to_string()))
+ .map_err(|error|
pyo3::exceptions::PyValueError::new_err(error.to_string()))?;
+ // A hook may put in the plan of a DataFrame, which the Python plan sends
as a placeholder.
+ spark_connect::plan::resolve_request(&mut request)
+ .map_err(|error|
pyo3::exceptions::PyValueError::new_err(error.to_string()))?;
+ Ok(request)
}
/// Held while a session is got or created, as PySpark's `SparkSession._lock`,
so that threads
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index b484216d..6807a12c 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1064,9 +1064,21 @@ impl DataFrame {
/// A DataFrame of `relation`, a relation built outside the core, such as
the plans of
/// `pyspark.ml`, which the core sends as it is, keeping `owner`, if any,
alive while any
- /// DataFrame of it is. Not a stable API.
+ /// DataFrame of it is. The placeholders of `relation`, which stand for
the DataFrames that
+ /// the plans of `pyspark.ml` take as input (see `placeholder_relation`),
are replaced by
+ /// the relations of those DataFrames. Not a stable API.
#[doc(hidden)]
pub fn from_relation(
+ session: SparkSession,
+ mut relation: proto::Relation,
+ owner: Option<Arc<dyn std::any::Any + Send + Sync>>,
+ ) -> Result<DataFrame> {
+ crate::plan::resolve_relation(&mut relation)?;
+ Ok(DataFrame::from_raw_relation(session, relation, owner))
+ }
+
+ /// The DataFrame of `relation`, as `from_relation`, of a relation without
placeholders.
+ fn from_raw_relation(
session: SparkSession,
relation: proto::Relation,
owner: Option<Arc<dyn std::any::Any + Send + Sync>>,
@@ -1087,7 +1099,7 @@ impl DataFrame {
#[doc(hidden)]
pub fn from_sent_relation(session: SparkSession, relation:
proto::Relation) -> DataFrame {
let plan_id = relation.common.as_ref().and_then(|common|
common.plan_id);
- let dataframe = DataFrame::from_relation(session, relation, None);
+ let dataframe = DataFrame::from_raw_relation(session, relation, None);
if let Some(plan_id) = plan_id {
dataframe
.session
@@ -1104,6 +1116,16 @@ impl DataFrame {
build_input_relation(&self.plan, &self.session)
}
+ /// A relation that stands for the relation of this DataFrame,
`to_relation`, in a relation
+ /// built outside the core, such as a plan of `pyspark.ml`, which the core
replaces by that
+ /// relation when the relation reaches it (`from_relation`,
`execute_command_proto`). It
+ /// is shallow, so it is cheap to build and to decode whatever the depth
of this DataFrame.
+ /// Not a stable API.
+ #[doc(hidden)]
+ pub fn placeholder_relation(&self) -> proto::Relation {
+ crate::plan::placeholder_relation(&self.plan,
self.session.plan_ids_shared())
+ }
+
/// Get the schema of this DataFrame.
///
/// The first successful lookup is cached and shared with exact clones.
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 5d398205..d675a4ca 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -6,7 +6,9 @@
use spark_connect_proto as proto;
use std::collections::HashMap;
use std::sync::atomic::{AtomicI64, Ordering};
-use std::sync::{Arc, Mutex, Weak};
+use std::sync::{Arc, LazyLock, Mutex, Weak};
+
+use spark_connect_core::error::{Result, SparkError};
use crate::column::Column;
use crate::expression::{Expression, ExpressionNode, LiteralExpression};
@@ -598,6 +600,175 @@ impl PlanIds {
}
}
+/// The `Any` type of a placeholder: the relation that stands for the relation
of a DataFrame, in a
+/// relation that is built outside the core, such as a plan of `pyspark.ml`,
which the Python
+/// client builds over the placeholder of each DataFrame it takes as input.
The value of a
+/// placeholder is its marker. The core replaces it by the relation of its
DataFrame when the
+/// relation or the command that holds it reaches the core, so the nesting of
a deep DataFrame is
+/// never decoded: a protobuf decoder limits the depth of what it decodes.
+const PLACEHOLDER_TYPE_URL: &str =
"type.googleapis.com/spark.connect.rust.DataFramePlaceholder";
+
+/// The placeholders of all the sessions, by the addresses of their plan node
and of their
+/// session's plan IDs, each with its marker. Both are held weakly, as
`PlanIds` holds its nodes,
+/// so an entry goes with its DataFrame, and its address is not reused while
the entry exists.
+#[derive(Default)]
+struct Placeholders {
+ entries: HashMap<(usize, usize), Placeholder>,
+ prune_at: usize,
+}
+
+struct Placeholder {
+ marker: i64,
+ node: Weak<LogicalPlan>,
+ ids: Weak<PlanIds>,
+}
+
+static PLACEHOLDERS: LazyLock<Mutex<Placeholders>> =
+ LazyLock::new(|| Mutex::new(Placeholders::default()));
+
+/// The placeholder of the relation of `node`, a DataFrame of the session of
`ids`: the same
+/// marker each time the DataFrame gives it.
+fn placeholder_marker(node: &Arc<LogicalPlan>, ids: &Arc<PlanIds>) -> i64 {
+ const MIN_PRUNE_AT: usize = 1024;
+ let mut placeholders = PLACEHOLDERS.lock().unwrap_or_else(|e|
e.into_inner());
+ // The entry's `Weak`s keep both allocations from being freed, so the node
and the session
+ // of an entry at these addresses are those of this call.
+ let key = (Arc::as_ptr(node) as usize, Arc::as_ptr(ids) as usize);
+ if let Some(entry) = placeholders.entries.get(&key) {
+ return entry.marker;
+ }
+ if placeholders.entries.len() >= placeholders.prune_at.max(MIN_PRUNE_AT) {
+ placeholders
+ .entries
+ .retain(|_, entry| entry.node.strong_count() > 0 &&
entry.ids.strong_count() > 0);
+ placeholders.prune_at = placeholders.entries.len() * 2;
+ }
+ let marker = NEXT_PLAN_ID.fetch_add(1, Ordering::Relaxed);
+ placeholders.entries.insert(
+ key,
+ Placeholder {
+ marker,
+ node: Arc::downgrade(node),
+ ids: Arc::downgrade(ids),
+ },
+ );
+ marker
+}
+
+/// The node and the plan IDs of the placeholder `marker`, if its DataFrame is
still alive.
+fn placeholder_target(marker: i64) -> Option<(Arc<LogicalPlan>, Arc<PlanIds>)>
{
+ let placeholders = PLACEHOLDERS.lock().unwrap_or_else(|e| e.into_inner());
+ placeholders
+ .entries
+ .values()
+ .find(|entry| entry.marker == marker)
+ .and_then(|entry| Some((entry.node.upgrade()?, entry.ids.upgrade()?)))
+}
+
+/// The placeholder of the relation of `node`, a DataFrame of the session of
`ids`, as the
+/// relation that the Python client sends in its place.
+pub(crate) fn placeholder_relation(node: &Arc<LogicalPlan>, ids:
&Arc<PlanIds>) -> proto::Relation {
+ let marker = placeholder_marker(node, ids);
+ proto::Relation {
+ common: None,
+ rel_type: Some(proto::relation::RelType::Extension(prost_types::Any {
+ type_url: PLACEHOLDER_TYPE_URL.to_string(),
+ value: marker.to_le_bytes().to_vec(),
+ })),
+ }
+}
+
+/// Replace the placeholders of `relation` by the relations of their
DataFrames, and of the
+/// relations that the plans of `pyspark.ml` take as input (their
`ml_relation`, `with_relations`
+/// and fetched arguments). Each replacement is the relation of a DataFrame of
the core, which has
+/// no placeholder.
+pub(crate) fn resolve_relation(relation: &mut proto::Relation) -> Result<()> {
+ if let Some(proto::relation::RelType::Extension(any)) = &relation.rel_type
{
+ if any.type_url == PLACEHOLDER_TYPE_URL {
+ let marker = i64::from_le_bytes(
+ any.value
+ .as_slice()
+ .try_into()
+ .map_err(|_| SparkError::value_msg("a placeholder of no
marker"))?,
+ );
+ let (node, ids) = placeholder_target(marker).ok_or_else(|| {
+ SparkError::value_msg("a placeholder of a DataFrame that no
longer exists")
+ })?;
+ *relation = relation_with_ids(&node, &ids);
+ return Ok(());
+ }
+ }
+ match relation.rel_type.as_mut() {
+ Some(proto::relation::RelType::MlRelation(ml)) => {
+ if let Some(summary) = ml.model_summary_dataset.as_mut() {
+ resolve_relation(summary)?;
+ }
+ match ml.ml_type.as_mut() {
+ Some(proto::ml_relation::MlType::Transform(transform)) => {
+ if let Some(input) = transform.input.as_mut() {
+ resolve_relation(input)?;
+ }
+ }
+ Some(proto::ml_relation::MlType::Fetch(fetch)) =>
resolve_fetch(fetch)?,
+ None => {}
+ }
+ }
+ Some(proto::relation::RelType::WithRelations(with)) => {
+ if let Some(root) = with.root.as_mut() {
+ resolve_relation(root)?;
+ }
+ for reference in with.references.iter_mut() {
+ resolve_relation(reference)?;
+ }
+ }
+ _ => {}
+ }
+ Ok(())
+}
+
+/// Replace the placeholders of the relations of the ML command `command`, as
`resolve_relation`.
+pub(crate) fn resolve_command(command: &mut proto::Command) -> Result<()> {
+ let Some(proto::command::CommandType::MlCommand(ml)) =
command.command_type.as_mut() else {
+ return Ok(());
+ };
+ let dataset = match ml.command.as_mut() {
+ Some(proto::ml_command::Command::Fit(fit)) => fit.dataset.as_mut(),
+ Some(proto::ml_command::Command::Evaluate(evaluate)) =>
evaluate.dataset.as_mut(),
+ Some(proto::ml_command::Command::CreateSummary(summary)) =>
summary.dataset.as_mut(),
+ Some(proto::ml_command::Command::Fetch(fetch)) => return
resolve_fetch(fetch),
+ _ => None,
+ };
+ match dataset {
+ Some(dataset) => resolve_relation(dataset),
+ None => Ok(()),
+ }
+}
+
+/// Replace the placeholders of the relations that a fetch of `fetch` takes as
arguments.
+fn resolve_fetch(fetch: &mut proto::Fetch) -> Result<()> {
+ for method in fetch.methods.iter_mut() {
+ for args in method.args.iter_mut() {
+ if let Some(proto::fetch::method::args::ArgsType::Input(input)) =
+ args.args_type.as_mut()
+ {
+ resolve_relation(input)?;
+ }
+ }
+ }
+ Ok(())
+}
+
+/// Replace the placeholders of the relation or of the command of `request`, a
request that a
+/// session hook built in Python, as `DataFrame::from_relation` does. Not a
stable API.
+#[doc(hidden)]
+pub fn resolve_request(request: &mut proto::ExecutePlanRequest) -> Result<()> {
+ match request.plan.as_mut().and_then(|plan| plan.op_type.as_mut()) {
+ Some(proto::plan::OpType::Root(relation)) =>
resolve_relation(relation),
+ Some(proto::plan::OpType::Command(command)) =>
resolve_command(command),
+ _ => Ok(()),
+ }
+}
+
/// Serialize a shared plan node for a request: the node and every node below
it carry
/// their stable plan IDs.
pub(crate) fn relation_with_ids(node: &Arc<LogicalPlan>, ids: &PlanIds) ->
proto::Relation {
@@ -2519,3 +2690,282 @@ mod inline_udtf_tests {
assert_eq!(udtf.return_type, None);
}
}
+
+#[cfg(test)]
+mod placeholder_tests {
+ use super::*;
+ use prost::Message;
+ use spark_connect_proto as proto;
+
+ /// The placeholder of `node`, of the session of `ids`, as the relation
that the plans of
+ /// `pyspark.ml` send in its place.
+ fn placeholder(node: &Arc<LogicalPlan>, ids: &Arc<PlanIds>) ->
proto::Relation {
+ placeholder_relation(node, ids)
+ }
+
+ /// The `ml_relation` of a transform of `input`, as
`pyspark.ml.connect.proto` builds it.
+ fn transform_of(input: proto::Relation) -> proto::Relation {
+ proto::Relation {
+ common: None,
+ rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+ proto::MlRelation {
+ model_summary_dataset: None,
+ ml_type:
Some(proto::ml_relation::MlType::Transform(Box::new(
+ proto::ml_relation::Transform {
+ input: Some(Box::new(input)),
+ params: None,
+ operator: None,
+ },
+ ))),
+ },
+ ))),
+ }
+ }
+
+ #[test]
+ fn a_placeholder_is_shallow_and_stands_for_the_relation_of_its_node() {
+ // The plan is converted recursively, so a plan of 200 levels takes
more than the 2 MiB
+ // stack of a test thread in a debug build, whose frames are much
larger than those of a
+ // release build.
+ let run = || {
+ let ids = Arc::new(PlanIds::new());
+ let mut node = Arc::new(range(0, 3, 1));
+ for _ in 0..200 {
+ node = Arc::new(LogicalPlan::Limit {
+ input: node,
+ limit: 3,
+ });
+ }
+ let relation = relation_with_ids(&node, &ids);
+ let shallow = placeholder(&node, &ids);
+ assert!(shallow.encoded_len() < 128, "{}", shallow.encoded_len());
+ let mut resolved = transform_of(shallow.clone());
+ resolve_relation(&mut resolved).unwrap();
+ assert_eq!(resolved, transform_of(relation));
+ };
+ std::thread::Builder::new()
+ .stack_size(64 * 1024 * 1024)
+ .spawn(run)
+ .unwrap()
+ .join()
+ .unwrap_or_else(|panic| std::panic::resume_unwind(panic));
+ }
+
+ #[test]
+ fn a_placeholder_is_the_same_for_the_same_node_and_session() {
+ let ids = Arc::new(PlanIds::new());
+ let node = Arc::new(range(0, 3, 1));
+ assert_eq!(placeholder(&node, &ids), placeholder(&node, &ids));
+ let other = Arc::new(PlanIds::new());
+ assert_ne!(placeholder(&node, &ids), placeholder(&node, &other));
+ let different = Arc::new(range(0, 4, 1));
+ assert_ne!(placeholder(&node, &ids), placeholder(&different, &ids));
+ }
+
+ #[test]
+ fn
placeholders_resolve_in_every_position_that_the_ml_plans_build_over_their_inputs()
{
+ let ids = Arc::new(PlanIds::new());
+ let node = Arc::new(range(0, 3, 1));
+ let relation = relation_with_ids(&node, &ids);
+ let shallow = placeholder(&node, &ids);
+ let fetch = || proto::Fetch {
+ obj_ref: None,
+ methods: vec![proto::fetch::Method {
+ method: "transform".to_string(),
+ args: vec![proto::fetch::method::Args {
+ args_type:
Some(proto::fetch::method::args::ArgsType::Input(shallow.clone())),
+ }],
+ }],
+ };
+ let mut relations = vec![
+ // An ML summary of a model, over its dataset.
+ proto::Relation {
+ common: None,
+ rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+ proto::MlRelation {
+ model_summary_dataset: Some(Box::new(shallow.clone())),
+ ml_type: None,
+ },
+ ))),
+ },
+ // A fetch of a summary, whose argument is a dataset.
+ proto::Relation {
+ common: None,
+ rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+ proto::MlRelation {
+ model_summary_dataset: None,
+ ml_type:
Some(proto::ml_relation::MlType::Fetch(fetch())),
+ },
+ ))),
+ },
+ // Two transforms, one over the other, over the input of the
second.
+ transform_of(transform_of(shallow.clone())),
+ // The references of a relation with its subqueries.
+ proto::Relation {
+ common: None,
+ rel_type:
Some(proto::relation::RelType::WithRelations(Box::new(
+ proto::WithRelations {
+ root: Some(Box::new(shallow.clone())),
+ references: vec![shallow.clone()],
+ },
+ ))),
+ },
+ ];
+ for relation_with_placeholder in relations.iter_mut() {
+ resolve_relation(relation_with_placeholder).unwrap();
+ }
+ // The placeholders are replaced by the relation of the node,
everywhere they were.
+ let expected = [
+ proto::Relation {
+ common: None,
+ rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+ proto::MlRelation {
+ model_summary_dataset:
Some(Box::new(relation.clone())),
+ ml_type: None,
+ },
+ ))),
+ },
+ proto::Relation {
+ common: None,
+ rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+ proto::MlRelation {
+ model_summary_dataset: None,
+ ml_type:
Some(proto::ml_relation::MlType::Fetch(proto::Fetch {
+ obj_ref: None,
+ methods: vec![proto::fetch::Method {
+ method: "transform".to_string(),
+ args: vec![proto::fetch::method::Args {
+ args_type:
Some(proto::fetch::method::args::ArgsType::Input(
+ relation.clone(),
+ )),
+ }],
+ }],
+ })),
+ },
+ ))),
+ },
+ transform_of(transform_of(relation.clone())),
+ proto::Relation {
+ common: None,
+ rel_type:
Some(proto::relation::RelType::WithRelations(Box::new(
+ proto::WithRelations {
+ root: Some(Box::new(relation.clone())),
+ references: vec![relation.clone()],
+ },
+ ))),
+ },
+ ];
+ assert_eq!(relations, expected);
+ }
+
+ #[test]
+ fn the_ml_commands_resolve_their_datasets_and_fetched_arguments() {
+ let ids = Arc::new(PlanIds::new());
+ let node = Arc::new(range(0, 3, 1));
+ let relation = relation_with_ids(&node, &ids);
+ let shallow = placeholder(&node, &ids);
+ let command = |ml_command: proto::ml_command::Command| proto::Command {
+ command_type:
Some(proto::command::CommandType::MlCommand(proto::MlCommand {
+ command: Some(ml_command),
+ })),
+ };
+ let fit = |dataset: proto::Relation| {
+ command(proto::ml_command::Command::Fit(proto::ml_command::Fit {
+ estimator: None,
+ params: None,
+ dataset: Some(dataset),
+ }))
+ };
+ let mut fitted = fit(shallow.clone());
+ resolve_command(&mut fitted).unwrap();
+ assert_eq!(fitted, fit(relation.clone()));
+
+ let mut evaluated = command(proto::ml_command::Command::Evaluate(
+ proto::ml_command::Evaluate {
+ evaluator: None,
+ params: None,
+ dataset: Some(shallow.clone()),
+ },
+ ));
+ resolve_command(&mut evaluated).unwrap();
+ assert_eq!(
+ evaluated,
+ command(proto::ml_command::Command::Evaluate(
+ proto::ml_command::Evaluate {
+ evaluator: None,
+ params: None,
+ dataset: Some(relation.clone()),
+ },
+ ))
+ );
+
+ let mut summary = command(proto::ml_command::Command::CreateSummary(
+ proto::ml_command::CreateSummary {
+ model_ref: None,
+ dataset: Some(shallow.clone()),
+ },
+ ));
+ resolve_command(&mut summary).unwrap();
+ assert_eq!(
+ summary,
+ command(proto::ml_command::Command::CreateSummary(
+ proto::ml_command::CreateSummary {
+ model_ref: None,
+ dataset: Some(relation.clone()),
+ },
+ ))
+ );
+
+ // A request of a session hook: the root of its plan, or its command.
+ let mut root = proto::ExecutePlanRequest {
+ plan: Some(proto::Plan {
+ op_type:
Some(proto::plan::OpType::Root(transform_of(shallow.clone()))),
+ }),
+ ..Default::default()
+ };
+ resolve_request(&mut root).unwrap();
+ assert_eq!(
+ root.plan.unwrap().op_type,
+ Some(proto::plan::OpType::Root(transform_of(relation)))
+ );
+ }
+
+ #[test]
+ fn relations_without_placeholders_are_left_as_they_are() {
+ let ids = Arc::new(PlanIds::new());
+ let node = Arc::new(range(0, 3, 1));
+ let mut relation = relation_with_ids(&node, &ids);
+ let before = relation.clone();
+ resolve_relation(&mut relation).unwrap();
+ assert_eq!(relation, before);
+ let mut command = proto::Command::default();
+ resolve_command(&mut command).unwrap();
+ assert_eq!(command, proto::Command::default());
+ }
+
+ #[test]
+ fn a_placeholder_of_a_dropped_node_is_an_error_and_keeps_nothing_alive() {
+ let ids = Arc::new(PlanIds::new());
+ let node = Arc::new(range(0, 3, 1));
+ let alive = Arc::downgrade(&node);
+ let shallow = placeholder(&node, &ids);
+ std::mem::drop(node);
+ assert!(alive.upgrade().is_none());
+ let mut relation = transform_of(shallow);
+ let error = resolve_relation(&mut relation).unwrap_err();
+ assert!(error.to_string().contains("no longer exists"), "{error}");
+ }
+
+ #[test]
+ fn a_malformed_placeholder_is_an_error() {
+ let mut relation = proto::Relation {
+ common: None,
+ rel_type:
Some(proto::relation::RelType::Extension(prost_types::Any {
+ type_url: PLACEHOLDER_TYPE_URL.to_string(),
+ value: vec![1, 2, 3],
+ })),
+ };
+ let error = resolve_relation(&mut relation).unwrap_err();
+ assert!(error.to_string().contains("no marker"), "{error}");
+ }
+}
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 048e3e37..4ae698e0 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -372,6 +372,10 @@ impl SparkSession {
&self.plan_ids
}
+ pub(crate) fn plan_ids_shared(&self) -> &Arc<PlanIds> {
+ &self.plan_ids
+ }
+
/// Get the underlying client.
pub(crate) fn client(&self) -> &Arc<SparkConnectClient> {
&self.client
@@ -788,12 +792,14 @@ impl SparkSession {
/// Execute `command`, a command built outside the core, such as the ML
commands of
/// `pyspark.ml`, and return the responses of the server, as the Python
client's
- /// `execute_command` reads them. Not a stable API.
+ /// `execute_command` reads them. The placeholders of its relations are
replaced, as
+ /// `DataFrame::from_relation` does. Not a stable API.
#[doc(hidden)]
pub fn execute_command_proto(
&self,
- command: spark_connect_proto::Command,
+ mut command: spark_connect_proto::Command,
) -> Result<Vec<spark_connect_proto::ExecutePlanResponse>> {
+ crate::plan::resolve_command(&mut command)?;
let Some(command_type) = command.command_type else {
return Err(SparkError::value_msg("a command of no command type"));
};
diff --git a/crates/spark-connect/tests/raw_relation.rs
b/crates/spark-connect/tests/raw_relation.rs
index 90dfe2df..c472dff8 100644
--- a/crates/spark-connect/tests/raw_relation.rs
+++ b/crates/spark-connect/tests/raw_relation.rs
@@ -44,6 +44,18 @@ fn live() -> Option<SparkSession> {
)
}
+/// Runs `f` on a thread with a stack of 64 MiB. A plan is converted
recursively, so a deep plan
+/// takes more than the 2 MiB stack of a test thread in a debug build, whose
frames are much
+/// larger than those of a release build.
+fn on_a_large_stack(f: impl FnOnce() + Send + 'static) {
+ std::thread::Builder::new()
+ .stack_size(64 * 1024 * 1024)
+ .spawn(f)
+ .unwrap()
+ .join()
+ .unwrap_or_else(|panic| std::panic::resume_unwind(panic));
+}
+
fn sql_relation(query: &str) -> proto::Relation {
proto::Relation {
common: Some(proto::RelationCommon {
@@ -59,7 +71,7 @@ fn sql_relation(query: &str) -> proto::Relation {
#[test]
fn raw_relations_are_sent_as_they_are_with_a_plan_id_of_the_session() {
- let df = DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS
a"), None);
+ let df = DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS
a"), None).unwrap();
let relation = df.to_relation();
let Some(proto::relation::RelType::Sql(sql)) = &relation.rel_type else {
panic!("expected the SQL relation, got {relation:?}");
@@ -74,7 +86,7 @@ fn
raw_relations_are_sent_as_they_are_with_a_plan_id_of_the_session() {
#[test]
fn transformations_of_raw_relations_take_them_as_input() {
- let df = DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS
a"), None);
+ let df = DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS
a"), None).unwrap();
let relation = df.to_relation();
let filtered = df.filter(col("a").gt(lit(0)));
let Some(proto::relation::RelType::Filter(filter)) =
filtered.to_relation().rel_type else {
@@ -87,7 +99,8 @@ fn transformations_of_raw_relations_take_them_as_input() {
fn raw_relations_keep_their_owners_alive_while_a_dataframe_uses_them() {
let owner: Arc<dyn Any + Send + Sync> = Arc::new(String::from("plan"));
let alive = Arc::downgrade(&owner);
- let df = DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS
a"), Some(owner));
+ let df =
+ DataFrame::from_relation(offline(), sql_relation("SELECT 1 AS a"),
Some(owner)).unwrap();
let derived = df.select([col("a")]);
drop(df);
assert!(alive.upgrade().is_some());
@@ -119,6 +132,92 @@ fn
sent_relations_keep_the_plan_ids_of_the_dataframes_they_were_of() {
assert_ne!(other.common.and_then(|common| common.plan_id), plan_id);
}
+#[test]
+fn placeholders_of_a_dataframe_are_replaced_by_its_relation() {
+ on_a_large_stack(|| {
+ // As the plans of pyspark.ml over the DataFrames that they take as
input: a relation over
+ // the placeholder of a DataFrame, which is replaced by that
DataFrame's relation.
+ let session = offline();
+ let mut df = session.range(3).unwrap();
+ for i in 0..120 {
+ df = df.with_column(&format!("c{i}"), col("id").add(lit(i)));
+ }
+ let placeholder = df.placeholder_relation();
+ // Shallow: the placeholder of a DataFrame of 120 levels is a few
bytes.
+ assert!(prost::Message::encoded_len(&placeholder) < 128);
+ let input = DataFrame::from_relation(
+ session.clone(),
+ proto::Relation {
+ common: None,
+ rel_type:
Some(proto::relation::RelType::WithRelations(Box::new(
+ proto::WithRelations {
+ root: Some(Box::new(placeholder.clone())),
+ references: vec![placeholder.clone()],
+ },
+ ))),
+ },
+ None,
+ )
+ .unwrap();
+ let Some(proto::relation::RelType::WithRelations(with)) =
input.to_relation().rel_type
+ else {
+ panic!("expected WithRelations");
+ };
+ assert_eq!(with.root.as_deref(), Some(&df.to_relation()));
+ assert_eq!(with.references, vec![df.to_relation()]);
+ // The same placeholder gives the same relation.
+ assert_eq!(df.placeholder_relation(), placeholder);
+ // The DataFrame of the placeholder is a node of its own, whose
relation is that of `df`.
+ let single = DataFrame::from_relation(session.clone(),
placeholder.clone(), None).unwrap();
+ assert_eq!(single.to_relation().rel_type, df.to_relation().rel_type);
+ // A placeholder of a DataFrame of another session is the relation of
that DataFrame too.
+ let other = offline();
+ let copy = DataFrame::from_relation(other, placeholder, None).unwrap();
+ assert_eq!(copy.to_relation().rel_type, df.to_relation().rel_type);
+ });
+}
+
+#[test]
+fn placeholders_of_dropped_dataframes_are_errors() {
+ let session = offline();
+ let df = session.range(3).unwrap();
+ let placeholder = df.placeholder_relation();
+ drop(df);
+ let Err(error) = DataFrame::from_relation(session, placeholder, None) else
{
+ panic!("a placeholder of a dropped DataFrame resolves");
+ };
+ assert!(error.to_string().contains("no longer exists"), "{error}");
+}
+
+#[test]
+fn a_deep_placeholder_runs_on_the_server() {
+ on_a_large_stack(|| {
+ let Some(session) = live() else {
+ return;
+ };
+ let mut df = session.range(3).unwrap();
+ for i in 0..120 {
+ df = df.with_column(&format!("c{i}"), col("id").add(lit(i)));
+ }
+ let wrapped = DataFrame::from_relation(
+ session.clone(),
+ proto::Relation {
+ common: None,
+ rel_type:
Some(proto::relation::RelType::WithRelations(Box::new(
+ proto::WithRelations {
+ root: Some(Box::new(df.placeholder_relation())),
+ references: vec![],
+ },
+ ))),
+ },
+ None,
+ )
+ .unwrap();
+ assert_eq!(wrapped.count().unwrap(), 3);
+ assert_eq!(wrapped.columns().unwrap().len(), 121);
+ });
+}
+
#[test]
fn commands_of_no_command_type_are_errors() {
let error = offline()
@@ -136,7 +235,8 @@ fn raw_relations_run_on_the_server() {
session.clone(),
sql_relation("SELECT id AS a FROM range(3)"),
None,
- );
+ )
+ .unwrap();
assert_eq!(df.columns().unwrap(), vec!["a".to_string()]);
assert_eq!(df.clone().filter(col("a").gt(lit(0))).count().unwrap(), 2);
let other = session.range(2).unwrap();
diff --git a/python/pyspark/sql/connect/plan.py
b/python/pyspark/sql/connect/plan.py
index e65d150e..a261a56f 100644
--- a/python/pyspark/sql/connect/plan.py
+++ b/python/pyspark/sql/connect/plan.py
@@ -128,14 +128,19 @@ class LogicalPlan:
class _DataFramePlan(LogicalPlan):
"""The plan of a DataFrame of the Rust core, as its ``DataFrame._plan``:
its relation, as the
- core sends it, of the plan IDs of its session."""
+ core sends it, of the plan IDs of its session.
+
+ Its ``plan`` is a placeholder of that relation, which the core replaces by
the relation when
+ the plan built over it reaches the core, as ``DataFrame(plan, session)``
or a command does.
+ So a plan of ``pyspark.ml`` over a deep DataFrame is never decoded in
Python or in Rust, as
+ protobuf decoders limit the depth of what they decode."""
def __init__(self, df: Any) -> None:
super().__init__(None)
self._df = df
def plan(self, session: Any) -> "proto.Relation":
- return proto.Relation.FromString(self._df._relation_bytes())
+ return proto.Relation.FromString(self._df._relation_placeholder())
class Read(LogicalPlan):
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 52eaa5ac..699bdcb2 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -10372,6 +10372,65 @@ def
test_arrow_input_explicit_schema_matching_width(schema, values):
assert names is None
+def
test_ml_plans_send_a_deep_dataframe_without_parsing_it_as_pyspark(mock_connect):
+ # As PySpark's pyspark.ml, which builds its plans over the relation of a
DataFrame without
+ # parsing it: a DataFrame of 120 chained withColumn calls is nested deeper
than a protobuf
+ # decoder parses. A plan over it holds a shallow placeholder of that
relation, which the core
+ # replaces with the relation when the plan reaches it, so the server
receives the relation.
+ import gc
+
+ import pyspark.sql.connect.proto as pb2
+ from pyspark.sql import functions as F
+ from pyspark.sql.connect.dataframe import DataFrame
+ from pyspark.sql.connect.plan import LogicalPlan
+
+ class Transform(LogicalPlan):
+ # The plan of a transformer of pyspark.ml.connect.proto, over its
input.
+ def __init__(self, child, name):
+ super().__init__(child)
+ self._name = name
+
+ def plan(self, session):
+ plan = self._create_proto_relation()
+
plan.ml_relation.transform.input.CopyFrom(self._child.plan(session))
+
plan.ml_relation.transform.obj_ref.CopyFrom(pb2.ObjectRef(id=self._name))
+ return plan
+
+ spark, service = mock_connect
+ deep = spark.range(2)
+ for i in range(120):
+ deep = deep.withColumn(f"c{i}", F.lit(i))
+ relation = deep._relation_bytes()
+ # The placeholder of the relation is shallow, whatever the depth of the
DataFrame.
+ assert len(deep._plan.plan(spark.client).SerializeToString()) < 128
+ # Two plans of transformers, one over the other, over the DataFrame. The
request is read as
+ # bytes, as the deep relation is not parsed by protobuf:
ExecutePlanRequest.plan=3,
+ # Plan.root=1, Relation.ml_relation=300, MlRelation.transform=1,
Transform.input=3,
+ # Transform.obj_ref=1 and ObjectRef.id=1.
+ nested = DataFrame(Transform(Transform(deep._plan, "m1"), "m2"), spark)
+ nested.collect()
+ root =
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+ outer = parse_message(parse_message(root)[300][0])[1][0]
+ assert parse_message(parse_message(outer)[1][0]) == {1: [b"m2"]}
+ inner = parse_message(parse_message(outer)[3][0])
+ inner_transform = parse_message(parse_message(inner[300][0])[1][0])
+ assert parse_message(inner_transform[1][0]) == {1: [b"m1"]}
+ assert inner_transform[3][0] == relation
+
+ # A placeholder of a DataFrame that is gone is an error, and keeps nothing
alive.
+ gone = spark.range(3)
+ placeholder = gone._plan.plan(spark.client)
+ del gone
+ gc.collect()
+
+ class Gone(LogicalPlan):
+ def plan(self, session):
+ return placeholder
+
+ with pytest.raises(Exception, match="no longer exists"):
+ DataFrame(Gone(None), spark)
+
+
@pytest.mark.parametrize("schema", [
T.StructType([]),
T.StructType([T.StructField("x", T.LongType()), T.StructField("y",
T.LongType())]),
diff --git a/python/tests/test_ml_e2e.py b/python/tests/test_ml_e2e.py
index 11e5187c..12864f11 100644
--- a/python/tests/test_ml_e2e.py
+++ b/python/tests/test_ml_e2e.py
@@ -240,6 +240,39 @@ def
test_ml_transformed_dataframes_keep_their_models_as_pyspark(session):
assert error.value.getCondition() == "CONNECT_ML.CACHE_INVALID"
+def test_ml_deep_dataframes_fit_and_transform_as_pyspark(session):
+ # As PySpark: a DataFrame of 120 chained withColumn calls, nested deeper
than a protobuf
+ # decoder parses, is fitted and transformed, and so is OneVsRest over 12
classes, whose
+ # transform adds a layer per class. Their predictions are those of the
shallow DataFrames.
+ # The chains are over a repartition of the local data: over a local
relation, the optimizer
+ # folds one projection per iteration, which a server in testing mode
(SPARK_TESTING, as in
+ # CI) fails past its 100 iterations, for PySpark too.
+ from pyspark.ml.classification import LogisticRegression, OneVsRest
+ from pyspark.ml.linalg import Vectors
+ from pyspark.sql import functions as F
+
+ rows = [(float(i % 2), Vectors.dense([float(i % 7), float(i % 5)])) for i
in range(40)]
+ base = session.createDataFrame(rows, ["label", "features"]).repartition(1)
+ deep = base
+ for i in range(120):
+ deep = deep.withColumn(f"c{i}", F.lit(i))
+ model = LogisticRegression(maxIter=2).fit(deep)
+ assert model.coefficients ==
LogisticRegression(maxIter=2).fit(base).coefficients
+ predictions = sorted(r.prediction for r in model.transform(deep).collect())
+ assert len(predictions) == 40
+ assert predictions == sorted(r.prediction for r in
model.transform(base).collect())
+
+ rows12 = [(float(i % 12), Vectors.dense([float(i % 7), float(i % 5)])) for
i in range(60)]
+ base12 = session.createDataFrame(rows12, ["label",
"features"]).repartition(1)
+ deep12 = base12
+ for i in range(120):
+ deep12 = deep12.withColumn(f"c{i}", F.lit(i))
+ ovr = OneVsRest(classifier=LogisticRegression(maxIter=2)).fit(base12)
+ shallow = sorted(r.prediction for r in ovr.transform(base12).collect())
+ assert sorted(r.prediction for r in ovr.transform(deep12).collect()) ==
shallow
+ assert len(shallow) == 60
+
+
def test_ml_dataframes_of_plans_as_pyspark(session):
# As PySpark Connect's DataFrame(plan, session): a DataFrame of the
relation of a plan,
# which further transformations take as input.
@@ -269,7 +302,9 @@ def
test_session_hooks_rewrite_the_requests_sent_to_the_server_as_pyspark(sessio
import pyspark.sql.connect.proto as pb2
from pyspark.sql.connect.session import SparkSession as ConnectSparkSession
- replacement = session.range(5)._plan.plan(session.client)
+ # The placeholder of a plan is valid while its DataFrame lives, so the
DataFrame is kept.
+ source = session.range(5)
+ replacement = source._plan.plan(session.client)
calls = []
class Replace(ConnectSparkSession.Hook):
diff --git a/python/tests/test_ml_offline.py b/python/tests/test_ml_offline.py
index bb02c634..a9d97e9e 100644
--- a/python/tests/test_ml_offline.py
+++ b/python/tests/test_ml_offline.py
@@ -109,6 +109,12 @@ def _commands(service):
return [pb2.ExecutePlanRequest.FromString(r).plan.command for r in
service.execute_requests]
+def _relation(df):
+ # The relation of a DataFrame, as the core sends it: the relation that the
plans of pyspark.ml
+ # send in place of its placeholder, which the core replaces with it.
+ return pb2.Relation.FromString(df._relation_bytes())
+
+
class _Color(enum.Enum):
RED = 1
@@ -468,7 +474,7 @@ def test_connect_conversions_as_pyspark(ml_mock):
assert proto_to_storage_level(message) == level
df =
proto_to_remote_cached_dataframe(pb2.CachedRemoteRelation(relation_id="r1"))
- assert df._plan.plan(spark.client).cached_remote_relation.relation_id ==
"r1"
+ assert _relation(df).cached_remote_relation.relation_id == "r1"
assert service.execute_requests == []
del df
gc.collect()
@@ -622,7 +628,7 @@ def
test_execute_command_converts_the_results_of_pyspark(ml_mock):
)
).SerializeToString()
df = client.execute_command(command)[1]["checkpoint_command_result"]
- assert df._plan.plan(client).cached_remote_relation.relation_id == "r9"
+ assert _relation(df).cached_remote_relation.relation_id == "r9"
# The relation is removed from the server once the DataFrame is collected,
of a command
# whose responses are ignored, as PySpark's: so the checkpoint that the
mock server answers
# with makes no other relation to remove.
@@ -702,20 +708,20 @@ def
test_ml_fit_and_transform_send_the_messages_of_pyspark(ml_mock):
"inputCol": pb2.Expression.Literal(string="words"),
"outputCol": pb2.Expression.Literal(string="counts"),
}
- assert fit.dataset == df._plan.plan(spark.client)
+ assert fit.dataset == _relation(df)
assert fit.dataset.HasField("project")
assert str(model._java_obj) == "model-1"
assert model.uid == estimator.uid and model.getVocabSize() == 5
predictions = model.transform(df)
- relation = predictions._plan.plan(spark.client)
+ relation = _relation(predictions)
transform = relation.ml_relation.transform
assert transform.obj_ref.id == "model-1"
assert transform.input == fit.dataset
assert transform.params.params["vocabSize"] ==
pb2.Expression.Literal(integer=5)
- selected = predictions.select("counts")._plan.plan(spark.client)
+ selected = _relation(predictions.select("counts"))
assert selected.project.input == relation
# A column of the transformed DataFrame refers to the plan ID of its
relation.
reference = predictions["counts"]._expr.to_plan(None).unresolved_attribute
@@ -731,12 +737,12 @@ def
test_ml_transformers_send_the_relation_of_pyspark(ml_mock):
spark, service = ml_mock
df = spark.range(1).selectExpr("'a b' AS text")
tokenizer = Tokenizer(inputCol="text", outputCol="words")
- transform =
tokenizer.transform(df)._plan.plan(spark.client).ml_relation.transform
+ transform = _relation(tokenizer.transform(df)).ml_relation.transform
assert transform.transformer.name ==
"org.apache.spark.ml.feature.Tokenizer"
assert transform.transformer.uid == tokenizer.uid
assert transform.transformer.type ==
pb2.MlOperator.OPERATOR_TYPE_TRANSFORMER
assert transform.params.params["inputCol"] ==
pb2.Expression.Literal(string="text")
- assert transform.input == df._plan.plan(spark.client)
+ assert transform.input == _relation(df)
assert service.execute_requests == []
@@ -761,14 +767,14 @@ def test_ml_dataframes_of_plans_keep_their_plans(ml_mock):
plan = SQLPlan()
alive = weakref.ref(plan)
df = DataFrame(plan=plan, session=spark)
- relation = df._plan.plan(spark.client)
+ relation = _relation(df)
assert relation.sql.query == "SELECT 1 AS a"
assert df["a"]._expr.to_plan(None).unresolved_attribute.plan_id ==
relation.common.plan_id
derived = df.filter("a > 0")
del plan, df
gc.collect()
assert alive() is not None
- assert derived._plan.plan(spark.client).filter.input == relation
+ assert _relation(derived).filter.input == relation
del derived
gc.collect()
assert alive() is None
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]