This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 024f11ab [SPARK-61158][CONNECT][PYTHON] Delete the cached server model
of a native LinearSVC or LogisticRegression fit when its last owner is dropped,
as pyspark.ml does
024f11ab is described below
commit 024f11ab519a0b6f8183e1d282f8560cd9690398
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:19:19 2026 +0900
[SPARK-61158][CONNECT][PYTHON] Delete the cached server model of a native
LinearSVC or LogisticRegression fit when its last owner is dropped, as
pyspark.ml does
### What changes were proposed in this pull request?
[SPARK-61158](https://issues.apache.org/jira/browse/SPARK-61158): Delete
the cached server model of a native LinearSVC or LogisticRegression fit when
its last owner is dropped, as pyspark.ml does.
- The cached model (`CachedModel` in `crates/spark-connect/src/ml.rs`) is
now a shared reference. A fitted model, its clones and every DataFrame it
transforms hold it.
- `LogicalPlan::MlTransform` keeps the input plan and the owner of the
cached model alive, because the input relation is sent inline. Every ML
transform that takes an input keeps it, so a DataFrame derived from a model
transform keeps the model alive too.
- When the last owner drops, `CachedModelRef::drop` sends `MlCommand.Delete
{ obj_refs }` on the fitting session through the new
`SparkSession::release_ml_objects`. The request runs on the shared runtime, and
its result and errors are ignored, so the drop never blocks and never panics at
interpreter exit or after the session is stopped.
- Official `pyspark.ml` does the same: its `RemoteModelRef` calls
`del_remote_cache` when the last reference goes, and a transformed DataFrame
keeps its source transformer alive.
- `docs/compatibility.md` describes the new lifetime.
### Why are the changes needed?
A native model stayed in the server's ML cache for the whole session. Each
fit added an entry to `MlCommand.GetCacheInfo` that nothing removed. Repro: fit
`LinearSVC(maxIter=2)` ten times in one session, `del` each model,
`gc.collect()`, then count `GetCacheInfo` entries. Ours gives 10. Official
`pyspark.ml.classification.LinearSVC` gives 0.
### Does this PR introduce _any_ user-facing change?
Yes. A native LinearSVC or LogisticRegression model's server cache entry is
deleted when the last model, clone or transformed DataFrame that refers to it
is dropped. Before, it lived until the session closed. Clones and lazy
DataFrames still work after the model is dropped.
### How was this patch tested?
-
`linear_svc_model_is_deleted_from_the_server_when_its_last_owner_is_dropped`
(`crates/spark-connect/tests/e2e_linear_svc.rs`) and
`logistic_regression_model_is_deleted_from_the_server_when_its_last_owner_is_dropped`
(`crates/spark-connect/tests/e2e_coverage.rs`) count `GetCacheInfo` after fits
and drops, check that a clone, a lazy DataFrame, and a DataFrame of another
model over the lazy one keep the model cached, and that the cache entry goes
once the last of them is dropped.
- `test_linear_svc_releases_its_cached_model_once_the_last_owner_is_gone`
(`python/tests/test_dropin_e2e.py`) fits ten models, deletes them and collects
the garbage, then checks the cache is back to its baseline. It also checks that
a lazy transformed DataFrame collects after `del model`.
- The tests of both crates, e2e included, against a local Spark 4.3.0
server (877 passed), the build of all tests, also with `wasm-udf`, the offline
Python suites (2201 passed, 8 skipped), and the live drop-in, Arrow and ML
suites (185 passed) and live cases (10 passed). rustfmt passes, and clippy
reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #347 from HyukjinKwon/SPARK-61158-native-ml-release-models.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect/src/ml.rs | 98 +++++++++++++++++--------
crates/spark-connect/src/plan.rs | 12 ++-
crates/spark-connect/src/session.rs | 32 ++++++++
crates/spark-connect/tests/e2e_coverage.rs | 101 +++++++++++++++++++++++++
crates/spark-connect/tests/e2e_linear_svc.rs | 106 +++++++++++++++++++++++++++
docs/compatibility.md | 11 ++-
python/tests/test_dropin_e2e.py | 63 ++++++++++++++++
7 files changed, 385 insertions(+), 38 deletions(-)
diff --git a/crates/spark-connect/src/ml.rs b/crates/spark-connect/src/ml.rs
index ef88561b..ea40cc3b 100644
--- a/crates/spark-connect/src/ml.rs
+++ b/crates/spark-connect/src/ml.rs
@@ -8,10 +8,12 @@
use spark_connect_proto as proto;
use std::collections::HashMap;
+use std::sync::Arc;
use uuid::Uuid;
use crate::dataframe::DataFrame;
-use crate::plan::LogicalPlan;
+use crate::plan::{LogicalPlan, RelationOwner};
+use crate::session::SparkSession;
/// ML parameter handling: stores name -> literal value mappings.
///
@@ -458,6 +460,8 @@ impl Transformer for StandardScalerModel {
// Wrap the built MlRelation as the plan; its `to_proto` emits the
relation as-is.
let plan = LogicalPlan::MlTransform {
ml_relation: relation,
+ input: df.plan.clone(),
+ owner: None,
};
Ok(DataFrame::new(df.session.clone(), plan))
@@ -545,6 +549,8 @@ impl Transformer for VectorAssembler {
relation.rel_type =
Some(proto::relation::RelType::MlRelation(Box::new(ml_relation)));
let plan = LogicalPlan::MlTransform {
ml_relation: relation,
+ input: df.plan.clone(),
+ owner: None,
};
Ok(DataFrame::new(df.session.clone(), plan))
}
@@ -664,6 +670,8 @@ impl Transformer for StringIndexerModel {
relation.rel_type =
Some(proto::relation::RelType::MlRelation(Box::new(ml_relation)));
let plan = LogicalPlan::MlTransform {
ml_relation: relation,
+ input: df.plan.clone(),
+ owner: None,
};
Ok(DataFrame::new(df.session.clone(), plan))
}
@@ -789,6 +797,8 @@ impl Transformer for MaxAbsScalerModel {
relation.rel_type =
Some(proto::relation::RelType::MlRelation(Box::new(ml_relation)));
let plan = LogicalPlan::MlTransform {
ml_relation: relation,
+ input: df.plan.clone(),
+ owner: None,
};
Ok(DataFrame::new(df.session.clone(), plan))
}
@@ -834,28 +844,48 @@ fn fit_via_command(
)))
}
-/// A session-owned model reference. Keep it cached until session closure so
that
-/// model clones and lazy transformed DataFrames remain usable after a model
is dropped.
+/// The server's cached model, shared by a model, its clones and every
DataFrame that it
+/// transforms (through the [`RelationOwner`] of their plans). The last owner
to drop deletes
+/// the cached object on the server, as PySpark's `RemoteModelRef` does.
#[derive(Debug, Clone)]
-struct CachedModel {
+struct CachedModel(Arc<CachedModelRef>);
+
+struct CachedModelRef {
model_ref: proto::ObjectRef,
- session_id: String,
+ session: SparkSession,
+}
+
+impl std::fmt::Debug for CachedModelRef {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("CachedModelRef")
+ .field("model_ref", &self.model_ref)
+ .finish_non_exhaustive()
+ }
+}
+
+impl Drop for CachedModelRef {
+ fn drop(&mut self) {
+ // Sent without waiting, and its errors (a stopped session, a closed
channel) are
+ // ignored, as PySpark ignores them when it releases a model.
+ self.session
+ .release_ml_objects(vec![self.model_ref.clone()]);
+ }
}
impl CachedModel {
fn from_info(
info: &proto::ml_command_result::MlOperatorInfo,
- session_id: &str,
+ session: &SparkSession,
estimator: &str,
) -> spark_connect_core::error::Result<Self> {
match &info.r#type {
Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(reference))
if !reference.id.is_empty() =>
{
- Ok(Self {
+ Ok(Self(Arc::new(CachedModelRef {
model_ref: reference.clone(),
- session_id: session_id.to_string(),
- })
+ session: session.clone(),
+ })))
}
_ =>
Err(spark_connect_core::error::SparkError::connect_msg(format!(
"{estimator}.fit: server returned no model object reference"
@@ -868,7 +898,7 @@ impl CachedModel {
ml_type: Some(proto::ml_relation::MlType::Transform(Box::new(
proto::ml_relation::Transform {
operator:
Some(proto::ml_relation::transform::Operator::ObjRef(
- self.model_ref.clone(),
+ self.0.model_ref.clone(),
)),
input: Some(Box::new(df.relation())),
params: Some(params.to_proto()),
@@ -883,7 +913,7 @@ impl CachedModel {
df: &DataFrame,
params: &Params,
) -> spark_connect_core::error::Result<DataFrame> {
- if df.session.session_id() != self.session_id {
+ if df.session.session_id() != self.0.session.session_id() {
return Err(spark_connect_core::error::SparkError::value_msg(
"A fitted ML model must transform data from its fitting
session",
));
@@ -894,10 +924,13 @@ impl CachedModel {
))),
..Default::default()
};
+ // The DataFrame keeps this model, and the plan of its input, alive.
Ok(DataFrame::new(
df.session.clone(),
LogicalPlan::MlTransform {
ml_relation: relation,
+ input: df.plan.clone(),
+ owner: Some(RelationOwner(self.0.clone())),
},
))
}
@@ -972,9 +1005,9 @@ impl LogisticRegression {
fn model_from_info(
&self,
info: proto::ml_command_result::MlOperatorInfo,
- session_id: &str,
+ session: &SparkSession,
) -> spark_connect_core::error::Result<LogisticRegressionModel> {
- let cached = CachedModel::from_info(&info, session_id,
"LogisticRegression")?;
+ let cached = CachedModel::from_info(&info, session,
"LogisticRegression")?;
Ok(LogisticRegressionModel {
operator: MlOperator::with_uid(
"org.apache.spark.ml.classification.LogisticRegressionModel",
@@ -1015,9 +1048,7 @@ impl Estimator for LogisticRegression {
fn fit_impl(&mut self, df: &DataFrame) ->
spark_connect_core::error::Result<Box<dyn Model>> {
let info = fit_via_command(&self.operator, &self.params, df)?;
- Ok(Box::new(
- self.model_from_info(info, df.session.session_id())?,
- ))
+ Ok(Box::new(self.model_from_info(info, &df.session)?))
}
}
@@ -1025,7 +1056,8 @@ impl Estimator for LogisticRegression {
///
/// Obtain this model through [`Estimator::fit`]. Its cached object reference
is
/// distinct from its operator UID and belongs to the session used for fitting.
-/// The server keeps the cached model for that session's lifetime.
+/// The server keeps the cached model while a model, a clone of it, or a
DataFrame
+/// it transformed is alive, then deletes it.
#[derive(Debug, Clone)]
pub struct LogisticRegressionModel {
pub operator: MlOperator,
@@ -1171,9 +1203,9 @@ impl LinearSVC {
fn model_from_info(
&self,
info: proto::ml_command_result::MlOperatorInfo,
- session_id: &str,
+ session: &SparkSession,
) -> spark_connect_core::error::Result<LinearSVCModel> {
- let cached = CachedModel::from_info(&info, session_id, "LinearSVC")?;
+ let cached = CachedModel::from_info(&info, session, "LinearSVC")?;
Ok(LinearSVCModel {
operator: MlOperator::with_uid(
"org.apache.spark.ml.classification.LinearSVCModel",
@@ -1208,14 +1240,13 @@ impl Estimator for LinearSVC {
fn fit_impl(&mut self, df: &DataFrame) ->
spark_connect_core::error::Result<Box<dyn Model>> {
let info = fit_via_command(&self.operator, &self.params, df)?;
- Ok(Box::new(
- self.model_from_info(info, df.session.session_id())?,
- ))
+ Ok(Box::new(self.model_from_info(info, &df.session)?))
}
}
/// A LinearSVC model fitted by Spark Connect and cached for its fitting
session.
-/// Obtain it through [`Estimator::fit`]; prediction uses the cached model
object.
+/// Obtain it through [`Estimator::fit`]; prediction uses the cached model
object,
+/// which the server deletes once no model, clone or transformed DataFrame
refers to it.
#[derive(Debug, Clone)]
pub struct LinearSVCModel {
operator: MlOperator,
@@ -1501,6 +1532,8 @@ impl Transformer for PipelineModel {
relation.rel_type =
Some(proto::relation::RelType::MlRelation(Box::new(ml_relation)));
let plan = LogicalPlan::MlTransform {
ml_relation: relation,
+ input: df.plan.clone(),
+ owner: None,
};
Ok(DataFrame::new(df.session.clone(), plan))
}
@@ -1690,6 +1723,8 @@ impl Transformer for CrossValidatorModel {
relation.rel_type =
Some(proto::relation::RelType::MlRelation(Box::new(ml_relation)));
let plan = LogicalPlan::MlTransform {
ml_relation: relation,
+ input: df.plan.clone(),
+ owner: None,
};
Ok(DataFrame::new(df.session.clone(), plan))
}
@@ -2108,7 +2143,7 @@ mod tests {
let _ = va.transform(&df).unwrap();
}
- fn fitted_logistic_regression(session_id: &str) -> LogisticRegressionModel
{
+ fn fitted_logistic_regression(session: &SparkSession) ->
LogisticRegressionModel {
let estimator = LogisticRegression::new().set_prediction_col("pred");
estimator
.model_from_info(
@@ -2121,7 +2156,7 @@ mod tests {
uid: Some("operator-uid".to_string()),
..Default::default()
},
- session_id,
+ session,
)
.unwrap()
}
@@ -2129,7 +2164,7 @@ mod tests {
#[test]
fn logistic_regression_transform_uses_the_fitted_object_reference() {
let session = offline_session();
- let mut model = fitted_logistic_regression(session.session_id());
+ let mut model = fitted_logistic_regression(&session);
assert_eq!(model.operator.uid, "operator-uid");
assert_eq!(model.operator.op_type, OperatorType::Model);
let data = session.range(3).unwrap();
@@ -2161,6 +2196,7 @@ mod tests {
#[test]
fn logistic_regression_fit_requires_a_nonempty_model_reference() {
let estimator = LogisticRegression::new();
+ let session = offline_session();
for kind in [
None,
Some(proto::ml_command_result::ml_operator_info::Type::Name(
@@ -2174,14 +2210,14 @@ mod tests {
r#type: kind,
..Default::default()
};
- assert!(estimator.model_from_info(info, "session").is_err());
+ assert!(estimator.model_from_info(info, &session).is_err());
}
}
#[test]
fn logistic_regression_rejects_a_different_session() {
let session = offline_session();
- let mut model = fitted_logistic_regression(session.session_id());
+ let mut model = fitted_logistic_regression(&session);
let other = session.new_session();
assert!(model.transform(&other.range(3).unwrap()).is_err());
}
@@ -2239,9 +2275,7 @@ mod tests {
)),
..Default::default()
};
- let mut model = estimator
- .model_from_info(info, session.session_id())
- .unwrap();
+ let mut model = estimator.model_from_info(info, &session).unwrap();
assert_eq!(model.operator().uid, estimator.operator().uid);
assert_eq!(
model.operator().name,
@@ -2268,7 +2302,7 @@ mod tests {
.is_err());
assert!(model.clone_box().transform(&df).is_ok());
assert!(estimator
- .model_from_info(Default::default(), session.session_id())
+ .model_from_info(Default::default(), &session)
.is_err());
}
}
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 72ec7527..82cb00a8 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -381,8 +381,14 @@ pub enum LogicalPlan {
left: Arc<LogicalPlan>,
right: Arc<LogicalPlan>,
},
- /// ML Transformation: wraps an MlRelation proto directly.
- MlTransform { ml_relation: proto::Relation },
+ /// ML Transformation: wraps an MlRelation proto directly. The proto
carries the input
+ /// inline, so `input` keeps the input plan, and the owners in it, alive;
`owner` keeps
+ /// the cached model that the relation refers to alive, if any.
+ MlTransform {
+ ml_relation: proto::Relation,
+ input: Arc<LogicalPlan>,
+ owner: Option<RelationOwner>,
+ },
/// A catalog operation exposed as a relation (e.g. `listTables`,
`listColumns`),
/// evaluated lazily so `.collect()` returns the server's rows - including
zero
/// rows for an empty database, which must not be an error.
@@ -1511,7 +1517,7 @@ impl LogicalPlan {
relation.rel_type =
Some(proto::relation::RelType::Zip(Box::new(zip)));
}
- LogicalPlan::MlTransform { ml_relation } => {
+ LogicalPlan::MlTransform { ml_relation, .. } => {
return ml_relation.clone();
}
LogicalPlan::Catalog { catalog } => {
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index c50906e8..cbbbf63f 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -728,6 +728,38 @@ impl SparkSession {
Ok(responses)
}
+ /// Delete the ML cached objects `refs` on the server, as PySpark's
`RemoteModelRef` does
+ /// when the last reference to a model goes. It does not wait for the
server: the request
+ /// runs on the shared runtime, and its errors (a stopped session, a
closed channel) are
+ /// ignored. Safe to call from `Drop`.
+ pub(crate) fn release_ml_objects(&self, refs: Vec<proto::ObjectRef>) {
+ let command = proto::Command {
+ command_type:
Some(proto::command::CommandType::MlCommand(proto::MlCommand {
+ command: Some(proto::ml_command::Command::Delete(
+ proto::ml_command::Delete {
+ obj_refs: refs,
+ evict_only: None,
+ },
+ )),
+ })),
+ };
+ let mut plan = proto::Plan::default();
+ plan.op_type = Some(proto::plan::OpType::Command(command));
+ let request = proto::ExecutePlanRequest {
+ session_id: self.client.session_id().to_string(),
+ user_context: Some(proto::UserContext::default()),
+ operation_id: Some(uuid::Uuid::new_v4().to_string()),
+ plan: Some(plan),
+ ..Default::default()
+ };
+ let client = Arc::clone(&self.client);
+ spark_connect_core::runtime::get_runtime().spawn(async move {
+ if let Ok(mut stream) =
client.execute_plan_reattachable(request).await {
+ while let Ok(Some(_)) = stream.message().await {}
+ }
+ });
+ }
+
/// Get the runtime configuration for this session.
///
/// Mirrors `pyspark.sql.SparkSession.conf`.
diff --git a/crates/spark-connect/tests/e2e_coverage.rs
b/crates/spark-connect/tests/e2e_coverage.rs
index bd342a10..2b531d4e 100644
--- a/crates/spark-connect/tests/e2e_coverage.rs
+++ b/crates/spark-connect/tests/e2e_coverage.rs
@@ -1581,6 +1581,107 @@ fn
logistic_regression_fits_and_predicts_with_cached_models() {
s.stop().unwrap();
}
+/// The number of models that the server caches for the session, from
`MlCommand.GetCacheInfo`.
+fn cached_models(s: &SparkSession) -> usize {
+ let command = spark_connect_proto::Command {
+ command_type:
Some(spark_connect_proto::command::CommandType::MlCommand(
+ spark_connect_proto::MlCommand {
+ command:
Some(spark_connect_proto::ml_command::Command::GetCacheInfo(
+ spark_connect_proto::ml_command::GetCacheInfo {},
+ )),
+ },
+ )),
+ };
+ s.execute_command_proto(command)
+ .unwrap()
+ .into_iter()
+ .find_map(|response| match response.response_type {
+
Some(spark_connect_proto::execute_plan_response::ResponseType::MlCommandResult(
+ result,
+ )) => match result.result_type {
+
Some(spark_connect_proto::ml_command_result::ResultType::Param(literal)) => {
+ match literal.literal_type {
+
Some(spark_connect_proto::expression::literal::LiteralType::Array(
+ array,
+ )) => Some(array.elements.len()),
+ _ => None,
+ }
+ }
+ _ => None,
+ },
+ _ => None,
+ })
+ .expect("GetCacheInfo returns the cached objects as an array")
+}
+
+/// Wait until the server holds `expected` cached models. The release of a
model is sent
+/// without waiting for the server, so its deletion can land after the drop.
+fn wait_for_cached_models(s: &SparkSession, expected: usize) -> usize {
+ let deadline = std::time::Instant::now() +
std::time::Duration::from_secs(30);
+ loop {
+ let count = cached_models(s);
+ if count == expected || std::time::Instant::now() >= deadline {
+ return count;
+ }
+ std::thread::sleep(std::time::Duration::from_millis(50));
+ }
+}
+
+#[test]
+fn
logistic_regression_model_is_deleted_from_the_server_when_its_last_owner_is_dropped()
{
+ if !should_run() {
+ return;
+ }
+ use spark_connect::expression::Expression;
+ use spark_connect::ml::{Estimator, LogisticRegression};
+ use spark_connect::{Column, DataFrame};
+
+ let s = session();
+ let as_features = |df: DataFrame| {
+ df.with_column(
+ "input",
+ Column::new(Expression::function(
+ "array_to_vector",
+ vec![col("xs").expression().clone()],
+ )),
+ )
+ };
+ let training = as_features(s.sql(
+ "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(CASE WHEN x > 0 THEN 1
ELSE 0 END AS DOUBLE) AS target FROM VALUES (-4), (-3), (-2), (-1), (1), (2),
(3), (4) AS t(x)",
+ ).unwrap());
+ let held_out = as_features(s.sql(
+ "SELECT id, array(CAST(x AS DOUBLE)) AS xs FROM VALUES (0, -3.5), (1,
-1.5), (2, 1.5), (3, 3.5) AS t(id, x)",
+ ).unwrap());
+ let mut estimator = LogisticRegression::new()
+ .set_feature_col("input")
+ .set_label_col("target")
+ .set_prediction_col("pred")
+ .set_max_iter(30);
+ let before = cached_models(&s);
+ // Models that are dropped right after fitting leave nothing on the server.
+ for _ in 0..3 {
+ let model = estimator.fit(&training).unwrap();
+ drop(model);
+ }
+ assert_eq!(wait_for_cached_models(&s, before), before);
+
+ // A clone and a lazy DataFrame that the model transformed keep it cached,
and so does a
+ // DataFrame that selects from the lazy one.
+ let mut model = estimator.fit(&training).unwrap();
+ assert_eq!(cached_models(&s), before + 1);
+ let clone = model.clone_box();
+ let lazy = model.transform(&held_out).unwrap();
+ let selected = lazy.select([col("id"), col("input")]);
+ drop(model);
+ drop(clone);
+ drop(lazy);
+ assert_eq!(cached_models(&s), before + 1);
+ assert_eq!(selected.clone().collect().unwrap().len(), 4);
+ drop(selected);
+ assert_eq!(wait_for_cached_models(&s, before), before);
+ s.stop().unwrap();
+}
+
#[test]
fn logistic_regression_fit_propagates_server_errors() {
if !should_run() {
diff --git a/crates/spark-connect/tests/e2e_linear_svc.rs
b/crates/spark-connect/tests/e2e_linear_svc.rs
index 16de17db..1a1cbc76 100644
--- a/crates/spark-connect/tests/e2e_linear_svc.rs
+++ b/crates/spark-connect/tests/e2e_linear_svc.rs
@@ -160,3 +160,109 @@ fn
linear_svc_fit_rejects_invalid_training_data_and_parameters() {
.is_err());
s.stop().unwrap();
}
+
+/// The number of models that the server caches for the session, from
`MlCommand.GetCacheInfo`.
+fn cached_models(s: &SparkSession) -> usize {
+ let command = spark_connect_proto::Command {
+ command_type:
Some(spark_connect_proto::command::CommandType::MlCommand(
+ spark_connect_proto::MlCommand {
+ command:
Some(spark_connect_proto::ml_command::Command::GetCacheInfo(
+ spark_connect_proto::ml_command::GetCacheInfo {},
+ )),
+ },
+ )),
+ };
+ let responses = s.execute_command_proto(command).unwrap();
+ responses
+ .into_iter()
+ .find_map(|response| match response.response_type {
+
Some(spark_connect_proto::execute_plan_response::ResponseType::MlCommandResult(
+ result,
+ )) => match result.result_type {
+
Some(spark_connect_proto::ml_command_result::ResultType::Param(literal)) => {
+ match literal.literal_type {
+
Some(spark_connect_proto::expression::literal::LiteralType::Array(
+ array,
+ )) => Some(array.elements.len()),
+ _ => None,
+ }
+ }
+ _ => None,
+ },
+ _ => None,
+ })
+ .expect("GetCacheInfo returns the cached objects as an array")
+}
+
+/// Wait until the server holds `expected` cached models. The release of a
model is sent
+/// without waiting for the server, so its deletion can land after the drop.
+fn wait_for_cached_models(s: &SparkSession, expected: usize) -> usize {
+ let deadline = std::time::Instant::now() +
std::time::Duration::from_secs(30);
+ loop {
+ let count = cached_models(s);
+ if count == expected || std::time::Instant::now() >= deadline {
+ return count;
+ }
+ std::thread::sleep(std::time::Duration::from_millis(50));
+ }
+}
+
+#[test]
+fn
linear_svc_model_is_deleted_from_the_server_when_its_last_owner_is_dropped() {
+ let Some(s) = session() else {
+ return;
+ };
+ let training = vectorize(
+ s.sql(
+ "SELECT array(CAST(x AS DOUBLE)) AS xs, \
+ CAST(CASE WHEN x > 0 THEN 1 ELSE 0 END AS DOUBLE) AS target \
+ FROM VALUES (-4), (-3), (-2), (-1), (1), (2), (3), (4) AS t(x)",
+ )
+ .unwrap(),
+ "input",
+ );
+ let held_out = vectorize(
+ s.sql(
+ "SELECT id, array(CAST(x AS DOUBLE)) AS xs \
+ FROM VALUES (0, -3.5), (1, -1.5), (2, 1.5), (3, 3.5) AS t(id, x)",
+ )
+ .unwrap(),
+ "input",
+ );
+ let mut estimator = LinearSVC::new()
+ .set_feature_col("input")
+ .set_label_col("target")
+ .set_max_iter(10);
+ let before = cached_models(&s);
+ // Models that are dropped right after fitting leave nothing on the server.
+ for _ in 0..3 {
+ let model = estimator.fit(&training).unwrap();
+ drop(model);
+ }
+ assert_eq!(wait_for_cached_models(&s, before), before);
+
+ // A clone and a lazy DataFrame that the model transformed keep it cached.
+ let mut model = estimator.fit(&training).unwrap();
+ let mut other = estimator
+ .clone()
+ .set_prediction_col("second")
+ .set_raw_prediction_col("second_margin")
+ .fit(&training)
+ .unwrap();
+ assert_eq!(cached_models(&s), before + 2);
+ let expected = predictions(model.transform(&held_out).unwrap(),
"prediction");
+ let clone = model.clone_box();
+ let lazy = model.transform(&held_out).unwrap();
+ // A DataFrame of another model over `lazy` keeps the first model alive
through the input
+ // plan, since the relation of the input is sent inline.
+ let chained = other.transform(&lazy).unwrap();
+ drop(model);
+ drop(clone);
+ drop(other);
+ drop(lazy);
+ assert_eq!(cached_models(&s), before + 2);
+ assert_eq!(predictions(chained.clone(), "second"), expected);
+ drop(chained);
+ assert_eq!(wait_for_cached_models(&s, before), before);
+ s.stop().unwrap();
+}
diff --git a/docs/compatibility.md b/docs/compatibility.md
index 003420a4..e1a0b22d 100644
--- a/docs/compatibility.md
+++ b/docs/compatibility.md
@@ -70,8 +70,11 @@ and PyTorch training.
Fitting returns a server-cached model. Prediction uses that model's object
reference and must use the same Spark session. Models are obtained through
`Estimator::fit`; direct struct construction of `LogisticRegressionModel` is no
-longer supported. The cached model remains available until the session is
closed,
-including when a model is cloned or a transformed DataFrame is executed later.
+longer supported. The cached model stays available while the model, a clone of
+it, or a DataFrame that it transformed is alive, including when such a
DataFrame is
+executed later. When the last of them is dropped, the cached model is deleted
on the
+server, as `pyspark.ml` does, without waiting for the server; a fit that is
dropped
+at once leaves nothing cached.
## Native LinearSVC
@@ -84,7 +87,9 @@ and `pyspark.ml.connect.LinearSVC`, with the upstream
keyword-only constructor
parameters and mutating `setFeaturesCol`, `setRegParam`, etc. methods.
`fit` returns a server-cached model; `transform` must use the fitting session.
-Clones and lazy prediction DataFrames remain usable until that session closes.
+Clones and lazy prediction DataFrames keep the cached model usable while they
are
+alive. When the last of the model, its clones and the DataFrames it
transformed is
+dropped, the server deletes the cached model, as `pyspark.ml` does.
The Python model uses the existing `MLModel.transform` interface. Model
inspection
(coefficients, intercept and summaries), single-vector prediction and model
persistence are not exposed by this addition.
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 481e48a3..7d637d39 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -3279,6 +3279,69 @@ def
test_logistic_regression_fits_the_data_and_predicts_on_new_rows(session):
estimator.fit(session.range(3))
+def
test_linear_svc_releases_its_cached_model_once_the_last_owner_is_gone(session):
+ # As pyspark.ml does, the server deletes a fitted model's cached object
once no model,
+ # copy or DataFrame that the model transformed is left in Python.
+ import gc
+
+ import pyspark.sql.connect.proto as pb2
+
+ try:
+ from pyspark._pyspark import LinearSVC
+ except ImportError:
+ from pyspark.ml.classification import LinearSVC
+ from pyspark.ml.connect.functions import array_to_vector
+
+ # A session of its own: the ML cache is per session, so the models of the
other tests,
+ # whose releases may still be on their way, do not change the counts below.
+ spark = session.newSession()
+
+ def cached_models():
+ cmd = pb2.Command()
+ cmd.ml_command.get_cache_info.SetInParent()
+ _, props, _ = spark.client.execute_command(cmd)
+ return len(props.get("ml_command_result").param.array.elements)
+
+ def wait_for_cached_models(expected):
+ # The release is sent without waiting for the server, so poll for it
to land.
+ deadline = time.time() + 30
+ while cached_models() != expected and time.time() < deadline:
+ time.sleep(0.05)
+ return cached_models()
+
+ try:
+ training = spark.sql(
+ "SELECT array(CAST(x AS DOUBLE)) AS xs, "
+ "CAST(CASE WHEN x > 0 THEN 1 ELSE 0 END AS DOUBLE) AS target "
+ "FROM VALUES (-4), (-3), (-2), (-1), (1), (2), (3), (4) AS t(x)"
+ ).withColumn("input", array_to_vector("xs"))
+ held_out = spark.sql(
+ "SELECT id, array(CAST(x AS DOUBLE)) AS xs "
+ "FROM VALUES (0, -3.5), (1, -1.5), (2, 1.5), (3, 3.5) AS t(id, x)"
+ ).withColumn("input", array_to_vector("xs"))
+ assert cached_models() == 0
+ for _ in range(10):
+ model = LinearSVC(featuresCol="input", labelCol="target",
maxIter=2).fit(training)
+ del model
+ gc.collect()
+ assert wait_for_cached_models(0) == 0
+
+ model = LinearSVC(featuresCol="input", labelCol="target",
predictionCol="pred",
+ maxIter=50).fit(training)
+ lazy = model.transform(held_out)
+ del model
+ gc.collect()
+ assert cached_models() == 1
+ rows = lazy.orderBy("id").collect()
+ assert [r.id for r in rows] == [0, 1, 2, 3]
+ assert all(r.pred in (0.0, 1.0) for r in rows)
+ del lazy
+ gc.collect()
+ assert wait_for_cached_models(0) == 0
+ finally:
+ spark.stop()
+
+
@pytest.mark.parametrize("use_setters", [False, True], ids=["constructor",
"setters"])
def test_linear_svc_fits_the_labels_and_predicts_on_new_rows(session,
use_setters):
try:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]