This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 8ae55fe [SPARK-61010][CONNECT] Fit LogisticRegression and predict
with its cached model
8ae55fe is described below
commit 8ae55fef764357dd61266716bdd1ef7c1c7f1310
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Sat Nov 7 07:57:43 2026 +0900
[SPARK-61010][CONNECT] Fit LogisticRegression and predict with its cached
model
### What changes were proposed in this pull request?
Make the native Rust `LogisticRegression` train with Spark Connect's
`MlCommand.Fit`, passing the actual dataset and estimator parameters. Preserve
the returned cached model reference and use it in `MlRelation.Transform` for
predictions, rather than reconstructing an unfitted operator from its UID.
Validate fitted-model responses, propagate server training errors, reject
transforms from a different session, and document the native estimator's JVM
backend and vector feature requirement.
### Why are the changes needed?
`fit` previously returned a model wrapper without submitting the training
data to Spark. Its transform plan identified an operator instead of the fitted
model object, so it could not produce predictions from learned coefficients.
[SPARK-61010](https://issues.apache.org/jira/browse/SPARK-61010)
### Does this PR introduce _any_ user-facing change?
Yes. Native Rust LogisticRegression and its Python binding now fit binary
and multiclass classifiers and predict on new rows using the trained model.
Invalid training datasets fail during `fit`.
Models belong to the session that fitted them. Cached models remain on the
server until session closure so cloned models and lazy transformed DataFrames
remain valid after the original model is dropped. The new private fitted state
means Rust callers must obtain `LogisticRegressionModel` through `fit`, rather
than constructing it with a struct literal.
This wrapper targets
`org.apache.spark.ml.classification.LogisticRegression`; it accepts Spark
vectors. The deprecated Torch-based
`pyspark.ml.connect.classification.LogisticRegression` is a different
implementation.
### How was this patch tested?
- Added offline tests for cached-reference serialization, malformed fit
responses, model metadata and session isolation.
- Added three strict live Rust tests covering binary and multiclass
predictions, custom/default column names, separate fits on reversed labels,
cloning, delayed execution, cross-session rejection and invalid training data.
All passed against Spark 4.3.0.
- Added a Python binding integration test for training-dependent
predictions and invalid training data. Passed with both the native binding and
official JVM PySpark as reference.
- Rust test run: 669 passed, 26 ignored; server-gated tests also skip their
bodies when no server is configured. Library tests, all integration-test
compilation, `wasm-udf` integration-test compilation and formatting passed.
- Python offline suite: 887 passed, 8 skipped. Fresh method-name comparison
across 12 core wrapper classes found no gaps under the existing transport
exclusions.
- Library Clippy passed with warnings. Workspace/all-target Clippy still
fails on existing test and Python-binding errors; comparison with the exact
base commit found no new diagnostics.
- Official SQL Connect suite through the Rust transport: all 87 files
passed across batches (2,638 tests passed, 589 skipped), using the existing CI
exclusions and known-failure manifest. The functions file passed after matching
upstream's `local[4]` setup; the initial partition-sensitive failure also
reproduced with official PySpark.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6)
This pull request and its description were written by Isaac.
Closes #217 from zhengruifeng/fix-logistic-regression-fit.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect/src/ml.rs | 212 ++++++++++++++++++++----
crates/spark-connect/tests/coverage_ml_extra.rs | 20 +--
crates/spark-connect/tests/e2e_coverage.rs | 161 ++++++++++++++++--
crates/spark-connect/tests/ml_extra_golden.rs | 23 +--
docs/compatibility.md | 15 ++
python/tests/test_dropin_e2e.py | 34 ++++
6 files changed, 376 insertions(+), 89 deletions(-)
diff --git a/crates/spark-connect/src/ml.rs b/crates/spark-connect/src/ml.rs
index 031ca3b..6af6605 100644
--- a/crates/spark-connect/src/ml.rs
+++ b/crates/spark-connect/src/ml.rs
@@ -865,6 +865,38 @@ impl LogisticRegression {
pub fn max_iter(&self) -> i64 {
self.max_iter
}
+
+ fn model_from_info(
+ &self,
+ info: proto::ml_command_result::MlOperatorInfo,
+ session_id: &str,
+ ) -> spark_connect_core::error::Result<LogisticRegressionModel> {
+ let model_ref = match info.r#type {
+
Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(reference))
+ if !reference.id.is_empty() =>
+ {
+ reference
+ }
+ _ => {
+ return Err(spark_connect_core::error::SparkError::connect_msg(
+ "LogisticRegression.fit: server returned no model object
reference",
+ ))
+ }
+ };
+ Ok(LogisticRegressionModel {
+ operator: MlOperator::with_uid(
+ "org.apache.spark.ml.classification.LogisticRegressionModel",
+ info.uid.as_deref().unwrap_or(&self.operator.uid),
+ OperatorType::Model,
+ ),
+ params: self.params.clone(),
+ feature_col: self.feature_col.clone(),
+ label_col: self.label_col.clone(),
+ prediction_col: self.prediction_col.clone(),
+ model_ref,
+ session_id: session_id.to_string(),
+ })
+ }
}
impl Default for LogisticRegression {
@@ -890,23 +922,42 @@ impl Estimator for LogisticRegression {
&mut self.params
}
- fn fit_impl(&mut self, _df: &DataFrame) ->
spark_connect_core::error::Result<Box<dyn Model>> {
- let model = LogisticRegressionModel {
- operator: MlOperator::with_uid(
- &self.operator.name,
- &self.operator.uid,
- OperatorType::Model,
- ),
- params: self.params.clone(),
- feature_col: self.feature_col.clone(),
- label_col: self.label_col.clone(),
- prediction_col: self.prediction_col.clone(),
+ fn fit_impl(&mut self, df: &DataFrame) ->
spark_connect_core::error::Result<Box<dyn Model>> {
+ let command = proto::MlCommand {
+ command:
Some(proto::ml_command::Command::Fit(proto::ml_command::Fit {
+ estimator: Some(self.operator.to_proto()),
+ params: Some(self.params.to_proto()),
+ dataset: Some(df.relation()),
+ })),
};
- Ok(Box::new(model))
+ let responses = crate::dataframe::execute_command_collect(
+ &df.session,
+ proto::command::CommandType::MlCommand(command),
+ )?;
+ for response in responses {
+ if let
Some(proto::execute_plan_response::ResponseType::MlCommandResult(result)) =
+ response.response_type
+ {
+ if let
Some(proto::ml_command_result::ResultType::OperatorInfo(info)) =
+ result.result_type
+ {
+ return Ok(Box::new(
+ self.model_from_info(info, df.session.session_id())?,
+ ));
+ }
+ }
+ }
+ Err(spark_connect_core::error::SparkError::connect_msg(
+ "LogisticRegression.fit: server returned no fitted model",
+ ))
}
}
-/// LogisticRegressionModel: fitted model from LogisticRegression.
+/// A LogisticRegression model fitted and cached by the Spark Connect server.
+///
+/// Obtain this model through [`Estimator::fit`]. Its cached object reference
is
+/// distinct from its operator UID and belongs to the session used for fitting.
+/// The server keeps the cached model for that session's lifetime.
#[derive(Debug, Clone)]
pub struct LogisticRegressionModel {
pub operator: MlOperator,
@@ -914,6 +965,8 @@ pub struct LogisticRegressionModel {
pub feature_col: String,
pub label_col: String,
pub prediction_col: String,
+ model_ref: proto::ObjectRef,
+ session_id: String,
}
impl Transformer for LogisticRegressionModel {
@@ -933,15 +986,39 @@ impl Transformer for LogisticRegressionModel {
&mut self.params
}
+ fn build_ml_relation(&self, input: &DataFrame) -> proto::MlRelation {
+ proto::MlRelation {
+ ml_type: Some(proto::ml_relation::MlType::Transform(Box::new(
+ proto::ml_relation::Transform {
+ operator:
Some(proto::ml_relation::transform::Operator::ObjRef(
+ self.model_ref.clone(),
+ )),
+ input: Some(Box::new(input.relation())),
+ params: Some(self.params.to_proto()),
+ },
+ ))),
+ ..Default::default()
+ }
+ }
+
fn transform_impl(&mut self, df: &DataFrame) ->
spark_connect_core::error::Result<DataFrame> {
- let ml_relation = self.build_ml_relation(df);
- let mut relation = proto::Relation::default();
- relation.common = Some(proto::RelationCommon::default());
- relation.rel_type =
Some(proto::relation::RelType::MlRelation(Box::new(ml_relation)));
- let plan = LogicalPlan::MlTransform {
- ml_relation: relation,
+ if df.session.session_id() != self.session_id {
+ return Err(spark_connect_core::error::SparkError::value_msg(
+ "A fitted LogisticRegression model must transform data from
its fitting session",
+ ));
+ }
+ let relation = proto::Relation {
+ rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+ self.build_ml_relation(df),
+ ))),
+ ..Default::default()
};
- Ok(DataFrame::new(df.session.clone(), plan))
+ Ok(DataFrame::new(
+ df.session.clone(),
+ LogicalPlan::MlTransform {
+ ml_relation: relation,
+ },
+ ))
}
}
@@ -1773,11 +1850,11 @@ mod tests {
}
#[test]
- fn all_estimators_fit_and_models_transform() {
+ fn offline_estimators_fit_and_models_transform() {
let s = offline_session();
let df = s.range(3).unwrap();
// Each estimator fits locally and its model transform builds a plan;
this
- // exercises fit_impl + transform_impl + clone_box for every model
type.
+ // exercises the offline model wrappers. LogisticRegression is tested
against a server.
let mut mas =
MaxAbsScaler::new().set_input_col("f").set_output_col("o");
assert_eq!(mas.input_col(), "f");
let mut m = mas.fit(&df).unwrap();
@@ -1793,19 +1870,6 @@ mod tests {
let _ = m.transform(&df).unwrap();
let _ = m.clone_box();
- let mut lr = LogisticRegression::new()
- .set_feature_col("features")
- .set_label_col("label")
- .set_prediction_col("pred")
- .set_max_iter(7);
- assert_eq!(lr.feature_col(), "features");
- assert_eq!(lr.label_col(), "label");
- assert_eq!(lr.prediction_col(), "pred");
- assert_eq!(lr.max_iter(), 7);
- let mut m = lr.fit(&df).unwrap();
- let _ = m.transform(&df).unwrap();
- let _ = m.clone_box();
-
let mut pipe = Pipeline::new().set_stages(vec!["a", "b"]);
assert_eq!(pipe.stages().len(), 2);
let mut m = pipe.fit(&df).unwrap();
@@ -1819,4 +1883,82 @@ mod tests {
assert_eq!(va.input_cols().len(), 2);
let _ = va.transform(&df).unwrap();
}
+
+ fn fitted_logistic_regression(session_id: &str) -> LogisticRegressionModel
{
+ let estimator = LogisticRegression::new().set_prediction_col("pred");
+ estimator
+ .model_from_info(
+ proto::ml_command_result::MlOperatorInfo {
+ r#type:
Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(
+ proto::ObjectRef {
+ id: "cached-model-id".to_string(),
+ },
+ )),
+ uid: Some("operator-uid".to_string()),
+ ..Default::default()
+ },
+ session_id,
+ )
+ .unwrap()
+ }
+
+ #[test]
+ fn logistic_regression_transform_uses_the_fitted_object_reference() {
+ let session = offline_session();
+ let mut model = fitted_logistic_regression(session.session_id());
+ assert_eq!(model.operator.uid, "operator-uid");
+ assert_eq!(model.operator.op_type, OperatorType::Model);
+ let data = session.range(3).unwrap();
+ let transformed = model.transform(&data).unwrap();
+ let relation = transformed.relation();
+ let Some(proto::relation::RelType::MlRelation(ml)) = relation.rel_type
else {
+ panic!("expected an ML relation");
+ };
+ let Some(proto::ml_relation::MlType::Transform(transform)) =
ml.ml_type else {
+ panic!("expected a model transform");
+ };
+ assert_eq!(
+ transform.operator,
+ Some(proto::ml_relation::transform::Operator::ObjRef(
+ proto::ObjectRef {
+ id: "cached-model-id".to_string()
+ },
+ ))
+ );
+ assert!(transform
+ .params
+ .unwrap()
+ .params
+ .contains_key("predictionCol"));
+ assert_eq!(*transform.input.unwrap(), data.relation());
+ let _ = model.clone_box();
+ }
+
+ #[test]
+ fn logistic_regression_fit_requires_a_nonempty_model_reference() {
+ let estimator = LogisticRegression::new();
+ for kind in [
+ None,
+ Some(proto::ml_command_result::ml_operator_info::Type::Name(
+ "operator".to_string(),
+ )),
+ Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(
+ proto::ObjectRef::default(),
+ )),
+ ] {
+ let info = proto::ml_command_result::MlOperatorInfo {
+ r#type: kind,
+ ..Default::default()
+ };
+ assert!(estimator.model_from_info(info, "session").is_err());
+ }
+ }
+
+ #[test]
+ fn logistic_regression_rejects_a_different_session() {
+ let session = offline_session();
+ let mut model = fitted_logistic_regression(session.session_id());
+ let other = session.new_session();
+ assert!(model.transform(&other.range(3).unwrap()).is_err());
+ }
}
diff --git a/crates/spark-connect/tests/coverage_ml_extra.rs
b/crates/spark-connect/tests/coverage_ml_extra.rs
index 7efabf9..f785ec6 100644
--- a/crates/spark-connect/tests/coverage_ml_extra.rs
+++ b/crates/spark-connect/tests/coverage_ml_extra.rs
@@ -9,10 +9,9 @@
use spark_connect::ml::{
BinaryClassificationEvaluator, CrossValidator, Estimator, Evaluator,
LogisticRegression,
- LogisticRegressionModel, MaxAbsScaler, MaxAbsScalerModel, MlOperator,
Model,
- MulticlassClassificationEvaluator, OperatorType, Params, Pipeline,
PipelineModel,
- RegressionEvaluator, StandardScaler, StringIndexer, StringIndexerModel,
Transformer,
- VectorAssembler,
+ MaxAbsScaler, MaxAbsScalerModel, MlOperator, Model,
MulticlassClassificationEvaluator,
+ OperatorType, Params, Pipeline, PipelineModel, RegressionEvaluator,
StandardScaler,
+ StringIndexer, StringIndexerModel, Transformer, VectorAssembler,
};
#[test]
@@ -159,19 +158,6 @@ fn transformer_and_model_accessors() {
let _ = m.params_mut();
let _ = m.clone_box();
- let mut m = LogisticRegressionModel {
- operator: MlOperator::new("lr", OperatorType::Model),
- params: Params::new(),
- feature_col: "f".to_string(),
- label_col: "l".to_string(),
- prediction_col: "p".to_string(),
- };
- let _ = m.operator();
- let _ = m.operator_mut();
- let _ = m.params();
- let _ = m.params_mut();
- let _ = m.clone_box();
-
let mut m = PipelineModel {
operator: MlOperator::new("pl", OperatorType::Model),
params: Params::new(),
diff --git a/crates/spark-connect/tests/e2e_coverage.rs
b/crates/spark-connect/tests/e2e_coverage.rs
index 5dd59af..3fb9f00 100644
--- a/crates/spark-connect/tests/e2e_coverage.rs
+++ b/crates/spark-connect/tests/e2e_coverage.rs
@@ -1087,8 +1087,7 @@ fn ml_estimators_and_transformers() {
return;
}
use spark_connect::ml::{
- Estimator, LogisticRegression, MaxAbsScaler, StandardScaler,
StringIndexer, Transformer,
- VectorAssembler,
+ Estimator, MaxAbsScaler, StandardScaler, StringIndexer, Transformer,
VectorAssembler,
};
let s = session();
@@ -1104,11 +1103,9 @@ fn ml_estimators_and_transformers() {
)
.expect("ml sample");
- // Each operator below exercises the CLIENT-side ML code in ml.rs -
building the
- // MlOperator/MlRelation params and the fit/transform requests - which
runs before
- // the RPC. The actual ML computation is server-side (connect-ML support
varies by
- // build), so we tolerate a server rejection rather than asserting the
result: the
- // coverage target is our request-construction code, not Spark's ML engine.
+ // This smoke test covers request construction and tolerates server
rejection.
+ // LogisticRegression has separate tests below that assert successful
training
+ // and learned predictions.
// Transformer.transform builds a fetch of the transform relation.
let mut va = VectorAssembler::new()
@@ -1116,7 +1113,7 @@ fn ml_estimators_and_transformers() {
.set_output_col("assembled");
let _ = va.transform(&df).and_then(|d| d.count());
- // Estimator.fit builds+sends the fit request; Model.transform builds a
transform.
+ // These estimator wrappers construct models locally; transform builds a
relation.
let mut ss = StandardScaler::new()
.set_input_col("features")
.set_output_col("scaled");
@@ -1135,14 +1132,6 @@ fn ml_estimators_and_transformers() {
if let Ok(mut m) = si.fit(&df) {
let _ = m.transform(&df).and_then(|d| d.count());
}
- let mut lr = LogisticRegression::new()
- .set_feature_col("features")
- .set_label_col("label")
- .set_prediction_col("prediction")
- .set_max_iter(5);
- if let Ok(mut m) = lr.fit(&df) {
- let _ = m.transform(&df).and_then(|d| d.count());
- }
// RegressionEvaluator.evaluate now sends a real MlCommand::Evaluate and
reads
// the metric back (previously a stub returning 0.0). With prediction ==
label
@@ -1167,7 +1156,7 @@ fn ml_estimators_and_transformers() {
.set_metric_name("areaUnderROC");
let _ = bin_eval.evaluate(&eval_df);
- // Pipeline estimator builds+fits (stages carried as params); tolerate
server ML.
+ // Pipeline carries stage names as params; this smoke test tolerates
rejection.
let mut pipeline = Pipeline::new().set_stages(vec!["scaler", "lr"]);
if let Ok(mut m) = pipeline.fit(&df) {
let _ = m.transform(&df).and_then(|d| d.count());
@@ -1432,3 +1421,141 @@ fn udf_builders_client_side() {
let _ = df.map_in_pandas(udf(), false);
let _ = df.map_in_arrow(udf(), true);
}
+
+/// Learning must depend on the labels, and each prediction must use its own
fitted model.
+#[test]
+fn logistic_regression_fits_and_predicts_with_cached_models() {
+ if !should_run() {
+ return;
+ }
+ use spark_connect::expression::Expression;
+ use spark_connect::ml::{Estimator, LogisticRegression};
+ use spark_connect::{Column, DataFrame};
+
+ let s = session();
+ let as_features = |df: DataFrame| {
+ df.with_column(
+ "input",
+ Column::new(Expression::function(
+ "array_to_vector",
+ vec![col("xs").expression().clone()],
+ )),
+ )
+ };
+ let training = as_features(s.sql(
+ "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(CASE WHEN x > 0
THEN 1 ELSE 0 END AS DOUBLE) AS target FROM VALUES (-4), (-3), (-2),
(-1), (1), (2), (3), (4) AS t(x)",
+ ).unwrap());
+ let held_out = as_features(s.sql(
+ "SELECT id, array(CAST(x AS DOUBLE)) AS xs FROM VALUES (0,
-3.5), (1, -1.5), (2, 1.5), (3, 3.5) AS t(id, x)",
+ ).unwrap());
+ let mut estimator = LogisticRegression::new()
+ .set_feature_col("input")
+ .set_label_col("target")
+ .set_prediction_col("pred")
+ .set_max_iter(30);
+ let mut first = estimator.fit(&training).expect("train first model");
+ let flipped = training.with_column("target", lit(1) - col("target"));
+ let mut second = estimator.fit(&flipped).expect("train opposite labels");
+ let first_operator = first.build_ml_relation(&held_out).ml_type.unwrap();
+ let second_operator = second.build_ml_relation(&held_out).ml_type.unwrap();
+ let model_ref = |kind: spark_connect_proto::ml_relation::MlType| {
+ let spark_connect_proto::ml_relation::MlType::Transform(t) = kind else
{
+ panic!("expected transform");
+ };
+ let
Some(spark_connect_proto::ml_relation::transform::Operator::ObjRef(r)) =
t.operator
+ else {
+ panic!("prediction must use the fitted model object reference");
+ };
+ r.id
+ };
+ assert_ne!(model_ref(first_operator), model_ref(second_operator));
+ let predictions = |df: DataFrame| {
+ df.select([col("id"), col("pred")])
+ .order_by(vec![col("id").asc().expression().clone()])
+ .collect()
+ .unwrap()
+ .into_iter()
+ .map(|row| row.get(1).unwrap().as_f64().unwrap())
+ .collect::<Vec<_>>()
+ };
+ let mut cloned = first.clone_box();
+ let pending = first.transform(&held_out).unwrap();
+ drop(first);
+ assert_eq!(predictions(pending), vec![0.0, 0.0, 1.0, 1.0]);
+ assert_eq!(
+ predictions(cloned.transform(&held_out).unwrap()),
+ vec![0.0, 0.0, 1.0, 1.0]
+ );
+ assert_eq!(
+ predictions(second.transform(&held_out).unwrap()),
+ vec![1.0, 1.0, 0.0, 0.0]
+ );
+ let other = s.new_session();
+ assert!(second.transform(&other.range(1).unwrap()).is_err());
+ s.stop().unwrap();
+}
+
+#[test]
+fn logistic_regression_fit_propagates_server_errors() {
+ if !should_run() {
+ return;
+ }
+ use spark_connect::ml::{Estimator, LogisticRegression};
+ let s = session();
+ // Missing training columns must fail during fitting, before prediction is
attempted.
+ let error = LogisticRegression::new().fit(&s.range(3).unwrap());
+ assert!(
+ error.is_err(),
+ "fit must validate and train the supplied dataset"
+ );
+ s.stop().unwrap();
+}
+
+#[test]
+fn logistic_regression_default_columns_support_multiclass_training() {
+ if !should_run() {
+ return;
+ }
+ use spark_connect::expression::Expression;
+ use spark_connect::ml::{Estimator, LogisticRegression};
+ use spark_connect::{Column, DataFrame};
+
+ let s = session();
+ let vectorize = |df: DataFrame| {
+ df.with_column(
+ "features",
+ Column::new(Expression::function(
+ "array_to_vector",
+ vec![col("xs").expression().clone()],
+ )),
+ )
+ };
+ let training = vectorize(
+ s.sql(
+ "SELECT array(CAST(x AS DOUBLE)) AS xs, \
+ CAST(CASE WHEN x < -2 THEN 0 WHEN x > 2 THEN 2 ELSE 1 END AS DOUBLE)
AS label \
+ FROM VALUES (-5), (-4), (-3), (-1), (0), (1), (3), (4), (5) AS t(x)",
+ )
+ .unwrap(),
+ );
+ let held_out = vectorize(
+ s.sql(
+ "SELECT x, array(CAST(x AS DOUBLE)) AS xs \
+ FROM VALUES (-4.5), (0.25), (4.5) AS t(x)",
+ )
+ .unwrap(),
+ );
+ let mut model = LogisticRegression::new().fit(&training).unwrap();
+ let predictions = model
+ .transform(&held_out)
+ .unwrap()
+ .order_by(vec![col("x").asc().expression().clone()])
+ .select([col("prediction")])
+ .collect()
+ .unwrap()
+ .into_iter()
+ .map(|row| row.get(0).unwrap().as_f64().unwrap())
+ .collect::<Vec<_>>();
+ assert_eq!(predictions, vec![0.0, 1.0, 2.0]);
+ s.stop().unwrap();
+}
diff --git a/crates/spark-connect/tests/ml_extra_golden.rs
b/crates/spark-connect/tests/ml_extra_golden.rs
index fa7fa6d..f7f6d86 100644
--- a/crates/spark-connect/tests/ml_extra_golden.rs
+++ b/crates/spark-connect/tests/ml_extra_golden.rs
@@ -13,10 +13,9 @@ use base64::{engine::general_purpose::STANDARD, Engine};
use prost::Message;
use spark_connect::ml::{
- BinaryClassificationEvaluator, Estimator, Evaluator, LogisticRegression,
- LogisticRegressionModel, MaxAbsScaler, MaxAbsScalerModel, MlOperator,
OperatorType, Params,
- Pipeline, PipelineModel, RegressionEvaluator, StringIndexer,
StringIndexerModel, Transformer,
- VectorAssembler,
+ BinaryClassificationEvaluator, Estimator, Evaluator, LogisticRegression,
MaxAbsScaler,
+ MaxAbsScalerModel, MlOperator, OperatorType, Params, Pipeline,
PipelineModel,
+ RegressionEvaluator, StringIndexer, StringIndexerModel, Transformer,
VectorAssembler,
};
use spark_connect_proto as proto;
@@ -161,22 +160,6 @@ fn test_logistic_regression_operator_name() {
assert_eq!(op.op_type, OperatorType::Estimator);
}
-#[test]
-fn test_logistic_regression_model_operator_type() {
- let model = LogisticRegressionModel {
- operator: MlOperator::new(
- "org.apache.spark.ml.classification.LogisticRegression",
- OperatorType::Model,
- ),
- params: Params::new(),
- feature_col: "features".to_string(),
- label_col: "label".to_string(),
- prediction_col: "prediction".to_string(),
- };
- let op = model.operator();
- assert_eq!(op.op_type, OperatorType::Model);
-}
-
#[test]
fn test_regression_evaluator_operator_name() {
let eval = RegressionEvaluator::new();
diff --git a/docs/compatibility.md b/docs/compatibility.md
index 83504d8..39a9a1c 100644
--- a/docs/compatibility.md
+++ b/docs/compatibility.md
@@ -58,6 +58,21 @@ helpers accept `impl Into<Arc<LogicalPlan>>`, so they accept
either an owned
`LogicalPlan` or an existing `Arc<LogicalPlan>`. Serialization still produces
the
same protobuf tree; sharing does not deduplicate repeated relations on the
wire.
+## Native LogisticRegression
+
+The native `spark_connect::ml::LogisticRegression` fits Spark's JVM
+`org.apache.spark.ml.classification.LogisticRegression` through the Connect ML
+protocol. Training features must be ML vectors; convert an array column with
+`array_to_vector` before fitting. This differs from the deprecated Python
+`pyspark.ml.connect.classification.LogisticRegression`, which uses array
features
+and PyTorch training.
+
+Fitting returns a server-cached model. Prediction uses that model's object
+reference and must use the same Spark session. Models are obtained through
+`Estimator::fit`; direct struct construction of `LogisticRegressionModel` is no
+longer supported. The cached model remains available until the session is
closed,
+including when a model is cloned or a transformed DataFrame is executed later.
+
## How parity is verified
Two independent gates in CI cover the two halves of a Spark Connect client -
plan
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 71052cc..15df8bf 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1653,3 +1653,37 @@ def
test_functions_apply_their_optional_arguments_as_pyspark(session):
assert row.set == '{"a":1,"b":2}'
assert row.l1 == 7.0
assert row.normalized == pytest.approx([3 / 7, 4 / 7])
+
+
+def test_logistic_regression_fits_the_data_and_predicts_on_new_rows(session):
+ # The native estimator targets Spark's JVM LogisticRegression. Compare the
same
+ # behavior with that estimator in official PySpark, rather than its
deprecated
+ # torch-based pyspark.ml.connect.classification implementation.
+ try:
+ from pyspark._pyspark import LogisticRegression
+ except ImportError:
+ from pyspark.ml.classification import LogisticRegression
+ from pyspark.ml.connect.functions import array_to_vector
+
+ training = session.sql(
+ "SELECT array(CAST(x AS DOUBLE)) AS xs, "
+ "CAST(CASE WHEN x > 0 THEN 1 ELSE 0 END AS DOUBLE) AS target "
+ "FROM VALUES (-4), (-3), (-2), (-1), (1), (2), (3), (4) AS t(x)"
+ ).withColumn("input", array_to_vector("xs"))
+ held_out = session.sql(
+ "SELECT id, array(CAST(x AS DOUBLE)) AS xs "
+ "FROM VALUES (0, -3.5), (1, -1.5), (2, 1.5), (3, 3.5) AS t(id, x)"
+ ).withColumn("input", array_to_vector("xs"))
+ estimator = LogisticRegression(
+ featuresCol="input", labelCol="target", predictionCol="pred",
maxIter=30
+ )
+ first = estimator.fit(training)
+ second = estimator.fit(training.withColumn("target", F.lit(1.0) -
F.col("target")))
+ assert [r.pred for r in first.transform(held_out).orderBy("id").collect()]
== [
+ 0.0, 0.0, 1.0, 1.0
+ ]
+ assert [r.pred for r in
second.transform(held_out).orderBy("id").collect()] == [
+ 1.0, 1.0, 0.0, 0.0
+ ]
+ with pytest.raises(Exception):
+ estimator.fit(session.range(3))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]