This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 8ae55fe  [SPARK-61010][CONNECT] Fit LogisticRegression and predict 
with its cached model
8ae55fe is described below

commit 8ae55fef764357dd61266716bdd1ef7c1c7f1310
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Sat Nov 7 07:57:43 2026 +0900

    [SPARK-61010][CONNECT] Fit LogisticRegression and predict with its cached 
model
    
    ### What changes were proposed in this pull request?
    
    Make the native Rust `LogisticRegression` train with Spark Connect's 
`MlCommand.Fit`, passing the actual dataset and estimator parameters. Preserve 
the returned cached model reference and use it in `MlRelation.Transform` for 
predictions, rather than reconstructing an unfitted operator from its UID.
    
    Validate fitted-model responses, propagate server training errors, reject 
transforms from a different session, and document the native estimator's JVM 
backend and vector feature requirement.
    
    ### Why are the changes needed?
    
    `fit` previously returned a model wrapper without submitting the training 
data to Spark. Its transform plan identified an operator instead of the fitted 
model object, so it could not produce predictions from learned coefficients.
    
    [SPARK-61010](https://issues.apache.org/jira/browse/SPARK-61010)
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Native Rust LogisticRegression and its Python binding now fit binary 
and multiclass classifiers and predict on new rows using the trained model. 
Invalid training datasets fail during `fit`.
    
    Models belong to the session that fitted them. Cached models remain on the 
server until session closure so cloned models and lazy transformed DataFrames 
remain valid after the original model is dropped. The new private fitted state 
means Rust callers must obtain `LogisticRegressionModel` through `fit`, rather 
than constructing it with a struct literal.
    
    This wrapper targets 
`org.apache.spark.ml.classification.LogisticRegression`; it accepts Spark 
vectors. The deprecated Torch-based 
`pyspark.ml.connect.classification.LogisticRegression` is a different 
implementation.
    
    ### How was this patch tested?
    
    - Added offline tests for cached-reference serialization, malformed fit 
responses, model metadata and session isolation.
    - Added three strict live Rust tests covering binary and multiclass 
predictions, custom/default column names, separate fits on reversed labels, 
cloning, delayed execution, cross-session rejection and invalid training data. 
All passed against Spark 4.3.0.
    - Added a Python binding integration test for training-dependent 
predictions and invalid training data. Passed with both the native binding and 
official JVM PySpark as reference.
    - Rust test run: 669 passed, 26 ignored; server-gated tests also skip their 
bodies when no server is configured. Library tests, all integration-test 
compilation, `wasm-udf` integration-test compilation and formatting passed.
    - Python offline suite: 887 passed, 8 skipped. Fresh method-name comparison 
across 12 core wrapper classes found no gaps under the existing transport 
exclusions.
    - Library Clippy passed with warnings. Workspace/all-target Clippy still 
fails on existing test and Python-binding errors; comparison with the exact 
base commit found no new diagnostics.
    
    - Official SQL Connect suite through the Rust transport: all 87 files 
passed across batches (2,638 tests passed, 589 skipped), using the existing CI 
exclusions and known-failure manifest. The functions file passed after matching 
upstream's `local[4]` setup; the initial partition-sensitive failure also 
reproduced with official PySpark.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Codex (GPT-6)
    
    This pull request and its description were written by Isaac.
    
    Closes #217 from zhengruifeng/fix-logistic-regression-fit.
    
    Authored-by: Ruifeng Zheng <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect/src/ml.rs                  | 212 ++++++++++++++++++++----
 crates/spark-connect/tests/coverage_ml_extra.rs |  20 +--
 crates/spark-connect/tests/e2e_coverage.rs      | 161 ++++++++++++++++--
 crates/spark-connect/tests/ml_extra_golden.rs   |  23 +--
 docs/compatibility.md                           |  15 ++
 python/tests/test_dropin_e2e.py                 |  34 ++++
 6 files changed, 376 insertions(+), 89 deletions(-)

diff --git a/crates/spark-connect/src/ml.rs b/crates/spark-connect/src/ml.rs
index 031ca3b..6af6605 100644
--- a/crates/spark-connect/src/ml.rs
+++ b/crates/spark-connect/src/ml.rs
@@ -865,6 +865,38 @@ impl LogisticRegression {
     pub fn max_iter(&self) -> i64 {
         self.max_iter
     }
+
+    fn model_from_info(
+        &self,
+        info: proto::ml_command_result::MlOperatorInfo,
+        session_id: &str,
+    ) -> spark_connect_core::error::Result<LogisticRegressionModel> {
+        let model_ref = match info.r#type {
+            
Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(reference))
+                if !reference.id.is_empty() =>
+            {
+                reference
+            }
+            _ => {
+                return Err(spark_connect_core::error::SparkError::connect_msg(
+                    "LogisticRegression.fit: server returned no model object 
reference",
+                ))
+            }
+        };
+        Ok(LogisticRegressionModel {
+            operator: MlOperator::with_uid(
+                "org.apache.spark.ml.classification.LogisticRegressionModel",
+                info.uid.as_deref().unwrap_or(&self.operator.uid),
+                OperatorType::Model,
+            ),
+            params: self.params.clone(),
+            feature_col: self.feature_col.clone(),
+            label_col: self.label_col.clone(),
+            prediction_col: self.prediction_col.clone(),
+            model_ref,
+            session_id: session_id.to_string(),
+        })
+    }
 }
 
 impl Default for LogisticRegression {
@@ -890,23 +922,42 @@ impl Estimator for LogisticRegression {
         &mut self.params
     }
 
-    fn fit_impl(&mut self, _df: &DataFrame) -> 
spark_connect_core::error::Result<Box<dyn Model>> {
-        let model = LogisticRegressionModel {
-            operator: MlOperator::with_uid(
-                &self.operator.name,
-                &self.operator.uid,
-                OperatorType::Model,
-            ),
-            params: self.params.clone(),
-            feature_col: self.feature_col.clone(),
-            label_col: self.label_col.clone(),
-            prediction_col: self.prediction_col.clone(),
+    fn fit_impl(&mut self, df: &DataFrame) -> 
spark_connect_core::error::Result<Box<dyn Model>> {
+        let command = proto::MlCommand {
+            command: 
Some(proto::ml_command::Command::Fit(proto::ml_command::Fit {
+                estimator: Some(self.operator.to_proto()),
+                params: Some(self.params.to_proto()),
+                dataset: Some(df.relation()),
+            })),
         };
-        Ok(Box::new(model))
+        let responses = crate::dataframe::execute_command_collect(
+            &df.session,
+            proto::command::CommandType::MlCommand(command),
+        )?;
+        for response in responses {
+            if let 
Some(proto::execute_plan_response::ResponseType::MlCommandResult(result)) =
+                response.response_type
+            {
+                if let 
Some(proto::ml_command_result::ResultType::OperatorInfo(info)) =
+                    result.result_type
+                {
+                    return Ok(Box::new(
+                        self.model_from_info(info, df.session.session_id())?,
+                    ));
+                }
+            }
+        }
+        Err(spark_connect_core::error::SparkError::connect_msg(
+            "LogisticRegression.fit: server returned no fitted model",
+        ))
     }
 }
 
-/// LogisticRegressionModel: fitted model from LogisticRegression.
+/// A LogisticRegression model fitted and cached by the Spark Connect server.
+///
+/// Obtain this model through [`Estimator::fit`]. Its cached object reference 
is
+/// distinct from its operator UID and belongs to the session used for fitting.
+/// The server keeps the cached model for that session's lifetime.
 #[derive(Debug, Clone)]
 pub struct LogisticRegressionModel {
     pub operator: MlOperator,
@@ -914,6 +965,8 @@ pub struct LogisticRegressionModel {
     pub feature_col: String,
     pub label_col: String,
     pub prediction_col: String,
+    model_ref: proto::ObjectRef,
+    session_id: String,
 }
 
 impl Transformer for LogisticRegressionModel {
@@ -933,15 +986,39 @@ impl Transformer for LogisticRegressionModel {
         &mut self.params
     }
 
+    fn build_ml_relation(&self, input: &DataFrame) -> proto::MlRelation {
+        proto::MlRelation {
+            ml_type: Some(proto::ml_relation::MlType::Transform(Box::new(
+                proto::ml_relation::Transform {
+                    operator: 
Some(proto::ml_relation::transform::Operator::ObjRef(
+                        self.model_ref.clone(),
+                    )),
+                    input: Some(Box::new(input.relation())),
+                    params: Some(self.params.to_proto()),
+                },
+            ))),
+            ..Default::default()
+        }
+    }
+
     fn transform_impl(&mut self, df: &DataFrame) -> 
spark_connect_core::error::Result<DataFrame> {
-        let ml_relation = self.build_ml_relation(df);
-        let mut relation = proto::Relation::default();
-        relation.common = Some(proto::RelationCommon::default());
-        relation.rel_type = 
Some(proto::relation::RelType::MlRelation(Box::new(ml_relation)));
-        let plan = LogicalPlan::MlTransform {
-            ml_relation: relation,
+        if df.session.session_id() != self.session_id {
+            return Err(spark_connect_core::error::SparkError::value_msg(
+                "A fitted LogisticRegression model must transform data from 
its fitting session",
+            ));
+        }
+        let relation = proto::Relation {
+            rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+                self.build_ml_relation(df),
+            ))),
+            ..Default::default()
         };
-        Ok(DataFrame::new(df.session.clone(), plan))
+        Ok(DataFrame::new(
+            df.session.clone(),
+            LogicalPlan::MlTransform {
+                ml_relation: relation,
+            },
+        ))
     }
 }
 
@@ -1773,11 +1850,11 @@ mod tests {
     }
 
     #[test]
-    fn all_estimators_fit_and_models_transform() {
+    fn offline_estimators_fit_and_models_transform() {
         let s = offline_session();
         let df = s.range(3).unwrap();
         // Each estimator fits locally and its model transform builds a plan; 
this
-        // exercises fit_impl + transform_impl + clone_box for every model 
type.
+        // exercises the offline model wrappers. LogisticRegression is tested 
against a server.
         let mut mas = 
MaxAbsScaler::new().set_input_col("f").set_output_col("o");
         assert_eq!(mas.input_col(), "f");
         let mut m = mas.fit(&df).unwrap();
@@ -1793,19 +1870,6 @@ mod tests {
         let _ = m.transform(&df).unwrap();
         let _ = m.clone_box();
 
-        let mut lr = LogisticRegression::new()
-            .set_feature_col("features")
-            .set_label_col("label")
-            .set_prediction_col("pred")
-            .set_max_iter(7);
-        assert_eq!(lr.feature_col(), "features");
-        assert_eq!(lr.label_col(), "label");
-        assert_eq!(lr.prediction_col(), "pred");
-        assert_eq!(lr.max_iter(), 7);
-        let mut m = lr.fit(&df).unwrap();
-        let _ = m.transform(&df).unwrap();
-        let _ = m.clone_box();
-
         let mut pipe = Pipeline::new().set_stages(vec!["a", "b"]);
         assert_eq!(pipe.stages().len(), 2);
         let mut m = pipe.fit(&df).unwrap();
@@ -1819,4 +1883,82 @@ mod tests {
         assert_eq!(va.input_cols().len(), 2);
         let _ = va.transform(&df).unwrap();
     }
+
+    fn fitted_logistic_regression(session_id: &str) -> LogisticRegressionModel 
{
+        let estimator = LogisticRegression::new().set_prediction_col("pred");
+        estimator
+            .model_from_info(
+                proto::ml_command_result::MlOperatorInfo {
+                    r#type: 
Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(
+                        proto::ObjectRef {
+                            id: "cached-model-id".to_string(),
+                        },
+                    )),
+                    uid: Some("operator-uid".to_string()),
+                    ..Default::default()
+                },
+                session_id,
+            )
+            .unwrap()
+    }
+
+    #[test]
+    fn logistic_regression_transform_uses_the_fitted_object_reference() {
+        let session = offline_session();
+        let mut model = fitted_logistic_regression(session.session_id());
+        assert_eq!(model.operator.uid, "operator-uid");
+        assert_eq!(model.operator.op_type, OperatorType::Model);
+        let data = session.range(3).unwrap();
+        let transformed = model.transform(&data).unwrap();
+        let relation = transformed.relation();
+        let Some(proto::relation::RelType::MlRelation(ml)) = relation.rel_type 
else {
+            panic!("expected an ML relation");
+        };
+        let Some(proto::ml_relation::MlType::Transform(transform)) = 
ml.ml_type else {
+            panic!("expected a model transform");
+        };
+        assert_eq!(
+            transform.operator,
+            Some(proto::ml_relation::transform::Operator::ObjRef(
+                proto::ObjectRef {
+                    id: "cached-model-id".to_string()
+                },
+            ))
+        );
+        assert!(transform
+            .params
+            .unwrap()
+            .params
+            .contains_key("predictionCol"));
+        assert_eq!(*transform.input.unwrap(), data.relation());
+        let _ = model.clone_box();
+    }
+
+    #[test]
+    fn logistic_regression_fit_requires_a_nonempty_model_reference() {
+        let estimator = LogisticRegression::new();
+        for kind in [
+            None,
+            Some(proto::ml_command_result::ml_operator_info::Type::Name(
+                "operator".to_string(),
+            )),
+            Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(
+                proto::ObjectRef::default(),
+            )),
+        ] {
+            let info = proto::ml_command_result::MlOperatorInfo {
+                r#type: kind,
+                ..Default::default()
+            };
+            assert!(estimator.model_from_info(info, "session").is_err());
+        }
+    }
+
+    #[test]
+    fn logistic_regression_rejects_a_different_session() {
+        let session = offline_session();
+        let mut model = fitted_logistic_regression(session.session_id());
+        let other = session.new_session();
+        assert!(model.transform(&other.range(3).unwrap()).is_err());
+    }
 }
diff --git a/crates/spark-connect/tests/coverage_ml_extra.rs 
b/crates/spark-connect/tests/coverage_ml_extra.rs
index 7efabf9..f785ec6 100644
--- a/crates/spark-connect/tests/coverage_ml_extra.rs
+++ b/crates/spark-connect/tests/coverage_ml_extra.rs
@@ -9,10 +9,9 @@
 
 use spark_connect::ml::{
     BinaryClassificationEvaluator, CrossValidator, Estimator, Evaluator, 
LogisticRegression,
-    LogisticRegressionModel, MaxAbsScaler, MaxAbsScalerModel, MlOperator, 
Model,
-    MulticlassClassificationEvaluator, OperatorType, Params, Pipeline, 
PipelineModel,
-    RegressionEvaluator, StandardScaler, StringIndexer, StringIndexerModel, 
Transformer,
-    VectorAssembler,
+    MaxAbsScaler, MaxAbsScalerModel, MlOperator, Model, 
MulticlassClassificationEvaluator,
+    OperatorType, Params, Pipeline, PipelineModel, RegressionEvaluator, 
StandardScaler,
+    StringIndexer, StringIndexerModel, Transformer, VectorAssembler,
 };
 
 #[test]
@@ -159,19 +158,6 @@ fn transformer_and_model_accessors() {
     let _ = m.params_mut();
     let _ = m.clone_box();
 
-    let mut m = LogisticRegressionModel {
-        operator: MlOperator::new("lr", OperatorType::Model),
-        params: Params::new(),
-        feature_col: "f".to_string(),
-        label_col: "l".to_string(),
-        prediction_col: "p".to_string(),
-    };
-    let _ = m.operator();
-    let _ = m.operator_mut();
-    let _ = m.params();
-    let _ = m.params_mut();
-    let _ = m.clone_box();
-
     let mut m = PipelineModel {
         operator: MlOperator::new("pl", OperatorType::Model),
         params: Params::new(),
diff --git a/crates/spark-connect/tests/e2e_coverage.rs 
b/crates/spark-connect/tests/e2e_coverage.rs
index 5dd59af..3fb9f00 100644
--- a/crates/spark-connect/tests/e2e_coverage.rs
+++ b/crates/spark-connect/tests/e2e_coverage.rs
@@ -1087,8 +1087,7 @@ fn ml_estimators_and_transformers() {
         return;
     }
     use spark_connect::ml::{
-        Estimator, LogisticRegression, MaxAbsScaler, StandardScaler, 
StringIndexer, Transformer,
-        VectorAssembler,
+        Estimator, MaxAbsScaler, StandardScaler, StringIndexer, Transformer, 
VectorAssembler,
     };
 
     let s = session();
@@ -1104,11 +1103,9 @@ fn ml_estimators_and_transformers() {
         )
         .expect("ml sample");
 
-    // Each operator below exercises the CLIENT-side ML code in ml.rs - 
building the
-    // MlOperator/MlRelation params and the fit/transform requests - which 
runs before
-    // the RPC. The actual ML computation is server-side (connect-ML support 
varies by
-    // build), so we tolerate a server rejection rather than asserting the 
result: the
-    // coverage target is our request-construction code, not Spark's ML engine.
+    // This smoke test covers request construction and tolerates server 
rejection.
+    // LogisticRegression has separate tests below that assert successful 
training
+    // and learned predictions.
 
     // Transformer.transform builds a fetch of the transform relation.
     let mut va = VectorAssembler::new()
@@ -1116,7 +1113,7 @@ fn ml_estimators_and_transformers() {
         .set_output_col("assembled");
     let _ = va.transform(&df).and_then(|d| d.count());
 
-    // Estimator.fit builds+sends the fit request; Model.transform builds a 
transform.
+    // These estimator wrappers construct models locally; transform builds a 
relation.
     let mut ss = StandardScaler::new()
         .set_input_col("features")
         .set_output_col("scaled");
@@ -1135,14 +1132,6 @@ fn ml_estimators_and_transformers() {
     if let Ok(mut m) = si.fit(&df) {
         let _ = m.transform(&df).and_then(|d| d.count());
     }
-    let mut lr = LogisticRegression::new()
-        .set_feature_col("features")
-        .set_label_col("label")
-        .set_prediction_col("prediction")
-        .set_max_iter(5);
-    if let Ok(mut m) = lr.fit(&df) {
-        let _ = m.transform(&df).and_then(|d| d.count());
-    }
 
     // RegressionEvaluator.evaluate now sends a real MlCommand::Evaluate and 
reads
     // the metric back (previously a stub returning 0.0). With prediction == 
label
@@ -1167,7 +1156,7 @@ fn ml_estimators_and_transformers() {
         .set_metric_name("areaUnderROC");
     let _ = bin_eval.evaluate(&eval_df);
 
-    // Pipeline estimator builds+fits (stages carried as params); tolerate 
server ML.
+    // Pipeline carries stage names as params; this smoke test tolerates 
rejection.
     let mut pipeline = Pipeline::new().set_stages(vec!["scaler", "lr"]);
     if let Ok(mut m) = pipeline.fit(&df) {
         let _ = m.transform(&df).and_then(|d| d.count());
@@ -1432,3 +1421,141 @@ fn udf_builders_client_side() {
     let _ = df.map_in_pandas(udf(), false);
     let _ = df.map_in_arrow(udf(), true);
 }
+
+/// Learning must depend on the labels, and each prediction must use its own 
fitted model.
+#[test]
+fn logistic_regression_fits_and_predicts_with_cached_models() {
+    if !should_run() {
+        return;
+    }
+    use spark_connect::expression::Expression;
+    use spark_connect::ml::{Estimator, LogisticRegression};
+    use spark_connect::{Column, DataFrame};
+
+    let s = session();
+    let as_features = |df: DataFrame| {
+        df.with_column(
+            "input",
+            Column::new(Expression::function(
+                "array_to_vector",
+                vec![col("xs").expression().clone()],
+            )),
+        )
+    };
+    let training = as_features(s.sql(
+        "SELECT array(CAST(x AS DOUBLE)) AS xs,          CAST(CASE WHEN x > 0 
THEN 1 ELSE 0 END AS DOUBLE) AS target          FROM VALUES (-4), (-3), (-2), 
(-1), (1), (2), (3), (4) AS t(x)",
+    ).unwrap());
+    let held_out = as_features(s.sql(
+        "SELECT id, array(CAST(x AS DOUBLE)) AS xs          FROM VALUES (0, 
-3.5), (1, -1.5), (2, 1.5), (3, 3.5) AS t(id, x)",
+    ).unwrap());
+    let mut estimator = LogisticRegression::new()
+        .set_feature_col("input")
+        .set_label_col("target")
+        .set_prediction_col("pred")
+        .set_max_iter(30);
+    let mut first = estimator.fit(&training).expect("train first model");
+    let flipped = training.with_column("target", lit(1) - col("target"));
+    let mut second = estimator.fit(&flipped).expect("train opposite labels");
+    let first_operator = first.build_ml_relation(&held_out).ml_type.unwrap();
+    let second_operator = second.build_ml_relation(&held_out).ml_type.unwrap();
+    let model_ref = |kind: spark_connect_proto::ml_relation::MlType| {
+        let spark_connect_proto::ml_relation::MlType::Transform(t) = kind else 
{
+            panic!("expected transform");
+        };
+        let 
Some(spark_connect_proto::ml_relation::transform::Operator::ObjRef(r)) = 
t.operator
+        else {
+            panic!("prediction must use the fitted model object reference");
+        };
+        r.id
+    };
+    assert_ne!(model_ref(first_operator), model_ref(second_operator));
+    let predictions = |df: DataFrame| {
+        df.select([col("id"), col("pred")])
+            .order_by(vec![col("id").asc().expression().clone()])
+            .collect()
+            .unwrap()
+            .into_iter()
+            .map(|row| row.get(1).unwrap().as_f64().unwrap())
+            .collect::<Vec<_>>()
+    };
+    let mut cloned = first.clone_box();
+    let pending = first.transform(&held_out).unwrap();
+    drop(first);
+    assert_eq!(predictions(pending), vec![0.0, 0.0, 1.0, 1.0]);
+    assert_eq!(
+        predictions(cloned.transform(&held_out).unwrap()),
+        vec![0.0, 0.0, 1.0, 1.0]
+    );
+    assert_eq!(
+        predictions(second.transform(&held_out).unwrap()),
+        vec![1.0, 1.0, 0.0, 0.0]
+    );
+    let other = s.new_session();
+    assert!(second.transform(&other.range(1).unwrap()).is_err());
+    s.stop().unwrap();
+}
+
+#[test]
+fn logistic_regression_fit_propagates_server_errors() {
+    if !should_run() {
+        return;
+    }
+    use spark_connect::ml::{Estimator, LogisticRegression};
+    let s = session();
+    // Missing training columns must fail during fitting, before prediction is 
attempted.
+    let error = LogisticRegression::new().fit(&s.range(3).unwrap());
+    assert!(
+        error.is_err(),
+        "fit must validate and train the supplied dataset"
+    );
+    s.stop().unwrap();
+}
+
+#[test]
+fn logistic_regression_default_columns_support_multiclass_training() {
+    if !should_run() {
+        return;
+    }
+    use spark_connect::expression::Expression;
+    use spark_connect::ml::{Estimator, LogisticRegression};
+    use spark_connect::{Column, DataFrame};
+
+    let s = session();
+    let vectorize = |df: DataFrame| {
+        df.with_column(
+            "features",
+            Column::new(Expression::function(
+                "array_to_vector",
+                vec![col("xs").expression().clone()],
+            )),
+        )
+    };
+    let training = vectorize(
+        s.sql(
+            "SELECT array(CAST(x AS DOUBLE)) AS xs, \
+         CAST(CASE WHEN x < -2 THEN 0 WHEN x > 2 THEN 2 ELSE 1 END AS DOUBLE) 
AS label \
+         FROM VALUES (-5), (-4), (-3), (-1), (0), (1), (3), (4), (5) AS t(x)",
+        )
+        .unwrap(),
+    );
+    let held_out = vectorize(
+        s.sql(
+            "SELECT x, array(CAST(x AS DOUBLE)) AS xs \
+         FROM VALUES (-4.5), (0.25), (4.5) AS t(x)",
+        )
+        .unwrap(),
+    );
+    let mut model = LogisticRegression::new().fit(&training).unwrap();
+    let predictions = model
+        .transform(&held_out)
+        .unwrap()
+        .order_by(vec![col("x").asc().expression().clone()])
+        .select([col("prediction")])
+        .collect()
+        .unwrap()
+        .into_iter()
+        .map(|row| row.get(0).unwrap().as_f64().unwrap())
+        .collect::<Vec<_>>();
+    assert_eq!(predictions, vec![0.0, 1.0, 2.0]);
+    s.stop().unwrap();
+}
diff --git a/crates/spark-connect/tests/ml_extra_golden.rs 
b/crates/spark-connect/tests/ml_extra_golden.rs
index fa7fa6d..f7f6d86 100644
--- a/crates/spark-connect/tests/ml_extra_golden.rs
+++ b/crates/spark-connect/tests/ml_extra_golden.rs
@@ -13,10 +13,9 @@ use base64::{engine::general_purpose::STANDARD, Engine};
 use prost::Message;
 
 use spark_connect::ml::{
-    BinaryClassificationEvaluator, Estimator, Evaluator, LogisticRegression,
-    LogisticRegressionModel, MaxAbsScaler, MaxAbsScalerModel, MlOperator, 
OperatorType, Params,
-    Pipeline, PipelineModel, RegressionEvaluator, StringIndexer, 
StringIndexerModel, Transformer,
-    VectorAssembler,
+    BinaryClassificationEvaluator, Estimator, Evaluator, LogisticRegression, 
MaxAbsScaler,
+    MaxAbsScalerModel, MlOperator, OperatorType, Params, Pipeline, 
PipelineModel,
+    RegressionEvaluator, StringIndexer, StringIndexerModel, Transformer, 
VectorAssembler,
 };
 use spark_connect_proto as proto;
 
@@ -161,22 +160,6 @@ fn test_logistic_regression_operator_name() {
     assert_eq!(op.op_type, OperatorType::Estimator);
 }
 
-#[test]
-fn test_logistic_regression_model_operator_type() {
-    let model = LogisticRegressionModel {
-        operator: MlOperator::new(
-            "org.apache.spark.ml.classification.LogisticRegression",
-            OperatorType::Model,
-        ),
-        params: Params::new(),
-        feature_col: "features".to_string(),
-        label_col: "label".to_string(),
-        prediction_col: "prediction".to_string(),
-    };
-    let op = model.operator();
-    assert_eq!(op.op_type, OperatorType::Model);
-}
-
 #[test]
 fn test_regression_evaluator_operator_name() {
     let eval = RegressionEvaluator::new();
diff --git a/docs/compatibility.md b/docs/compatibility.md
index 83504d8..39a9a1c 100644
--- a/docs/compatibility.md
+++ b/docs/compatibility.md
@@ -58,6 +58,21 @@ helpers accept `impl Into<Arc<LogicalPlan>>`, so they accept 
either an owned
 `LogicalPlan` or an existing `Arc<LogicalPlan>`. Serialization still produces 
the
 same protobuf tree; sharing does not deduplicate repeated relations on the 
wire.
 
+## Native LogisticRegression
+
+The native `spark_connect::ml::LogisticRegression` fits Spark's JVM
+`org.apache.spark.ml.classification.LogisticRegression` through the Connect ML
+protocol. Training features must be ML vectors; convert an array column with
+`array_to_vector` before fitting. This differs from the deprecated Python
+`pyspark.ml.connect.classification.LogisticRegression`, which uses array 
features
+and PyTorch training.
+
+Fitting returns a server-cached model. Prediction uses that model's object
+reference and must use the same Spark session. Models are obtained through
+`Estimator::fit`; direct struct construction of `LogisticRegressionModel` is no
+longer supported. The cached model remains available until the session is 
closed,
+including when a model is cloned or a transformed DataFrame is executed later.
+
 ## How parity is verified
 
 Two independent gates in CI cover the two halves of a Spark Connect client - 
plan
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 71052cc..15df8bf 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1653,3 +1653,37 @@ def 
test_functions_apply_their_optional_arguments_as_pyspark(session):
     assert row.set == '{"a":1,"b":2}'
     assert row.l1 == 7.0
     assert row.normalized == pytest.approx([3 / 7, 4 / 7])
+
+
+def test_logistic_regression_fits_the_data_and_predicts_on_new_rows(session):
+    # The native estimator targets Spark's JVM LogisticRegression. Compare the 
same
+    # behavior with that estimator in official PySpark, rather than its 
deprecated
+    # torch-based pyspark.ml.connect.classification implementation.
+    try:
+        from pyspark._pyspark import LogisticRegression
+    except ImportError:
+        from pyspark.ml.classification import LogisticRegression
+    from pyspark.ml.connect.functions import array_to_vector
+
+    training = session.sql(
+        "SELECT array(CAST(x AS DOUBLE)) AS xs, "
+        "CAST(CASE WHEN x > 0 THEN 1 ELSE 0 END AS DOUBLE) AS target "
+        "FROM VALUES (-4), (-3), (-2), (-1), (1), (2), (3), (4) AS t(x)"
+    ).withColumn("input", array_to_vector("xs"))
+    held_out = session.sql(
+        "SELECT id, array(CAST(x AS DOUBLE)) AS xs "
+        "FROM VALUES (0, -3.5), (1, -1.5), (2, 1.5), (3, 3.5) AS t(id, x)"
+    ).withColumn("input", array_to_vector("xs"))
+    estimator = LogisticRegression(
+        featuresCol="input", labelCol="target", predictionCol="pred", 
maxIter=30
+    )
+    first = estimator.fit(training)
+    second = estimator.fit(training.withColumn("target", F.lit(1.0) - 
F.col("target")))
+    assert [r.pred for r in first.transform(held_out).orderBy("id").collect()] 
== [
+        0.0, 0.0, 1.0, 1.0
+    ]
+    assert [r.pred for r in 
second.transform(held_out).orderBy("id").collect()] == [
+        1.0, 1.0, 0.0, 0.0
+    ]
+    with pytest.raises(Exception):
+        estimator.fit(session.range(3))


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to