This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 1f30e72a [SPARK-61193][CONNECT] Support native LinearRegression 
training and prediction
1f30e72a is described below

commit 1f30e72ae38c04054a3513b680204a67f4b48702
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Nov 9 09:09:22 2026 +0900

    [SPARK-61193][CONNECT] Support native LinearRegression training and 
prediction
    
    ### What changes were proposed in this pull request?
    
    Add native `LinearRegression` and its fitted model so Rust callers can 
train Spark's JVM regressor and predict on new DataFrames through Spark 
Connect. Fit sends the dataset and parameters through `MlCommand.Fit`; 
transform uses the returned cached model reference and the existing model 
ownership/session handling.
    
    Expose typed Rust setters for all 15 constructor parameters, including 
elastic-net regularization, instance weights, solver selection and 
squared-error/Huber loss. Add a Python binding under 
`pyspark.ml.connect.regression.LinearRegression` and 
`pyspark.ml.connect.LinearRegression`, with the reference keyword-only 
constructor and mutating parameter setters. Encode the two JVM integer 
parameters as 32-bit literals to match PySpark.
    
    ### Why are the changes needed?
    
    The native Rust API currently has no linear regression estimator. This adds 
regression alongside LogisticRegression and LinearSVC, using the existing ML 
transport and RegressionEvaluator.
    
    [SPARK-61193](https://issues.apache.org/jira/browse/SPARK-61193)
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Rust callers and users of the native Python binding can fit a 
regressor with vector features and numeric labels, then transform DataFrames in 
its fitting session. Models, clones and lazy transformed DataFrames retain the 
fitted server object until their last owner is dropped. Invalid training data 
or parameters fail during fit.
    
    The native API added here covers fitting and DataFrame transformation. 
Coefficients/intercept access, summaries, single-vector prediction and 
persistence are not exposed by this addition; these limits are documented. The 
existing vendored `pyspark.ml.regression` module is unchanged.
    
    ### How was this patch tested?
    
    - Added a golden comparison of all 15 parameter encodings against official 
PySpark 4.3.0, with a capture script for regenerating the fixture. Added 
offline validation of fitted model references and metadata.
    - Four live Rust tests passed against Spark 4.3.0: held-out predictions for 
known lines and RMSE < 1e-6, independent fitted datasets, cloning/lazy 
execution, session isolation, default/custom columns, zero-weight outliers, 
iterative elastic-net regularization, Huber loss and invalid data/parameters.
    - Three Python cases passed with both the native binding and official JVM 
PySpark as reference. Seven selected native Python integration tests passed 
across LinearRegression, LogisticRegression and LinearSVC. For the 
reference-only setter case, loss is set through setParams: upstream v4.3.0 
setLoss incorrectly targets the nonexistent lossType parameter. The native 
setLoss targets loss.
    - Rust full test run without SPARK_REMOTE: 741 passed, 26 ignored; 
server-gated test bodies return early without a server. All integration targets 
compile with and without wasm-udf; formatting, library Clippy and Clippy on the 
two new integration/golden test targets passed.
    - Python offline suite: 1,230 passed, 8 skipped. Verified the new 
constructor's 15 names, defaults and keyword-only arguments against PySpark; 
the existing 12 core wrapper classes have no method-name gaps under the 
documented transport exclusions.
    - Workspace/all-target Clippy reports existing errors. Comparing 
diagnostics with the exact base commit found no new errors.
    - Official SQL Connect suite through the Rust transport: all 87 files 
passed (2,638 tests passed, 589 skipped), using the existing CI exclusions and 
known-failure manifest.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Codex (GPT-6)
    
    This pull request and its description were written by Isaac.
    
    Closes #380 from zhengruifeng/add-linear-regression.
    
    Lead-authored-by: Ruifeng Zheng <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 .github/workflows/build_python_connect.yml         |   2 +-
 crates/pyspark-rs/src/lib.rs                       |   1 +
 crates/pyspark-rs/src/ml.rs                        | 154 +++++++++++-
 crates/spark-connect/src/ml.rs                     | 279 +++++++++++++++++++++
 .../spark-connect/tests/e2e_linear_regression.rs   | 243 ++++++++++++++++++
 .../tests/linear_regression_golden.rs              |  36 +++
 docs/compatibility.md                              |  17 ++
 python/pyspark/ml/connect/__init__.py              |   2 +
 python/pyspark/ml/connect/regression.py            |   4 +
 python/tests/test_dropin_e2e.py                    |  67 +++++
 scripts/capture_linear_regression_golden.py        |  60 +++++
 tests/golden/linear_regression.json                |   4 +
 12 files changed, 865 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/build_python_connect.yml 
b/.github/workflows/build_python_connect.yml
index 8f471c3e..8c875361 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -391,7 +391,7 @@ jobs:
         run: |
           cargo test -p apache-spark-connect \
             --test e2e_functions --test e2e_columns --test e2e_dataframe \
-            --test e2e_coverage --test e2e_linear_svc --test e2e_integration \
+            --test e2e_coverage --test e2e_linear_svc --test 
e2e_linear_regression --test e2e_integration \
             -- --test-threads=1
 
       - name: Rust WASM UDF e2e test (against the live server)
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 840843af..084a4797 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -239,6 +239,7 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) -> 
PyResult<()> {
     m.add_class::<ml::PyVectorAssembler>()?;
     m.add_class::<ml::PyLogisticRegression>()?;
     m.add_class::<ml::PyLinearSVC>()?;
+    m.add_class::<ml::PyLinearRegression>()?;
     m.add_class::<ml::PyRegressionEvaluator>()?;
     m.add_class::<ml::PyBinaryClassificationEvaluator>()?;
     m.add_class::<ml::PyPipeline>()?;
diff --git a/crates/pyspark-rs/src/ml.rs b/crates/pyspark-rs/src/ml.rs
index f3f68f46..41a9dbff 100644
--- a/crates/pyspark-rs/src/ml.rs
+++ b/crates/pyspark-rs/src/ml.rs
@@ -7,9 +7,9 @@
 
 use pyo3::prelude::*;
 use spark_connect::ml::{
-    BinaryClassificationEvaluator, CrossValidator, Estimator, Evaluator, 
LinearSVC,
-    LogisticRegression, MaxAbsScaler, Model, 
MulticlassClassificationEvaluator, Pipeline,
-    RegressionEvaluator, StandardScaler, StringIndexer, Transformer, 
VectorAssembler,
+    BinaryClassificationEvaluator, CrossValidator, Estimator, Evaluator, 
LinearRegression,
+    LinearSVC, LogisticRegression, MaxAbsScaler, Model, 
MulticlassClassificationEvaluator,
+    Pipeline, RegressionEvaluator, StandardScaler, StringIndexer, Transformer, 
VectorAssembler,
 };
 
 use crate::dataframe::PyDataFrame;
@@ -394,6 +394,154 @@ impl PyLinearSVC {
     }
 }
 
+/// Linear regression backed by Spark Connect's JVM ML API.
+#[pyclass(name = "LinearRegression", module = "pyspark.ml.connect.regression")]
+pub struct PyLinearRegression {
+    inner: LinearRegression,
+}
+
+#[pymethods]
+impl PyLinearRegression {
+    #[new]
+    #[pyo3(signature = (*, featuresCol="features", labelCol="label", 
predictionCol="prediction", maxIter=100, regParam=0.0, elasticNetParam=0.0, 
tol=1e-6, fitIntercept=true, standardization=true, solver="auto", 
weightCol=None, aggregationDepth=2, loss="squaredError", epsilon=1.35, 
maxBlockSizeInMB=0.0))]
+    #[allow(non_snake_case, clippy::too_many_arguments)]
+    fn new(
+        featuresCol: &str,
+        labelCol: &str,
+        predictionCol: &str,
+        maxIter: i32,
+        regParam: f64,
+        elasticNetParam: f64,
+        tol: f64,
+        fitIntercept: bool,
+        standardization: bool,
+        solver: &str,
+        weightCol: Option<&str>,
+        aggregationDepth: i32,
+        loss: &str,
+        epsilon: f64,
+        maxBlockSizeInMB: f64,
+    ) -> Self {
+        let mut inner = LinearRegression::new()
+            .set_feature_col(featuresCol)
+            .set_label_col(labelCol)
+            .set_prediction_col(predictionCol)
+            .set_max_iter(maxIter)
+            .set_reg_param(regParam)
+            .set_elastic_net_param(elasticNetParam)
+            .set_tol(tol)
+            .set_fit_intercept(fitIntercept)
+            .set_standardization(standardization)
+            .set_solver(solver)
+            .set_aggregation_depth(aggregationDepth)
+            .set_loss(loss)
+            .set_epsilon(epsilon)
+            .set_max_block_size_in_mb(maxBlockSizeInMB);
+        if let Some(col) = weightCol {
+            inner = inner.set_weight_col(col);
+        }
+        Self { inner }
+    }
+
+    #[pyo3(name = "setFeaturesCol")]
+    fn set_feature_col<'py>(mut slf: PyRefMut<'py, Self>, value: &str) -> 
PyRefMut<'py, Self> {
+        slf.inner = slf.inner.clone().set_feature_col(value);
+        slf
+    }
+
+    #[pyo3(name = "setLabelCol")]
+    fn set_label_col<'py>(mut slf: PyRefMut<'py, Self>, value: &str) -> 
PyRefMut<'py, Self> {
+        slf.inner = slf.inner.clone().set_label_col(value);
+        slf
+    }
+
+    #[pyo3(name = "setPredictionCol")]
+    fn set_prediction_col<'py>(mut slf: PyRefMut<'py, Self>, value: &str) -> 
PyRefMut<'py, Self> {
+        slf.inner = slf.inner.clone().set_prediction_col(value);
+        slf
+    }
+
+    #[pyo3(name = "setMaxIter")]
+    fn set_max_iter(mut slf: PyRefMut<'_, Self>, value: i32) -> PyRefMut<'_, 
Self> {
+        slf.inner = slf.inner.clone().set_max_iter(value);
+        slf
+    }
+
+    #[pyo3(name = "setRegParam")]
+    fn set_reg_param(mut slf: PyRefMut<'_, Self>, value: f64) -> PyRefMut<'_, 
Self> {
+        slf.inner = slf.inner.clone().set_reg_param(value);
+        slf
+    }
+
+    #[pyo3(name = "setElasticNetParam")]
+    fn set_elastic_net_param(mut slf: PyRefMut<'_, Self>, value: f64) -> 
PyRefMut<'_, Self> {
+        slf.inner = slf.inner.clone().set_elastic_net_param(value);
+        slf
+    }
+
+    #[pyo3(name = "setTol")]
+    fn set_tol(mut slf: PyRefMut<'_, Self>, value: f64) -> PyRefMut<'_, Self> {
+        slf.inner = slf.inner.clone().set_tol(value);
+        slf
+    }
+
+    #[pyo3(name = "setFitIntercept")]
+    fn set_fit_intercept(mut slf: PyRefMut<'_, Self>, value: bool) -> 
PyRefMut<'_, Self> {
+        slf.inner = slf.inner.clone().set_fit_intercept(value);
+        slf
+    }
+
+    #[pyo3(name = "setStandardization")]
+    fn set_standardization(mut slf: PyRefMut<'_, Self>, value: bool) -> 
PyRefMut<'_, Self> {
+        slf.inner = slf.inner.clone().set_standardization(value);
+        slf
+    }
+
+    #[pyo3(name = "setSolver")]
+    fn set_solver<'py>(mut slf: PyRefMut<'py, Self>, value: &str) -> 
PyRefMut<'py, Self> {
+        slf.inner = slf.inner.clone().set_solver(value);
+        slf
+    }
+
+    #[pyo3(name = "setWeightCol")]
+    fn set_weight_col<'py>(mut slf: PyRefMut<'py, Self>, value: &str) -> 
PyRefMut<'py, Self> {
+        slf.inner = slf.inner.clone().set_weight_col(value);
+        slf
+    }
+
+    #[pyo3(name = "setAggregationDepth")]
+    fn set_aggregation_depth(mut slf: PyRefMut<'_, Self>, value: i32) -> 
PyRefMut<'_, Self> {
+        slf.inner = slf.inner.clone().set_aggregation_depth(value);
+        slf
+    }
+
+    #[pyo3(name = "setLoss")]
+    fn set_loss<'py>(mut slf: PyRefMut<'py, Self>, value: &str) -> 
PyRefMut<'py, Self> {
+        slf.inner = slf.inner.clone().set_loss(value);
+        slf
+    }
+
+    #[pyo3(name = "setEpsilon")]
+    fn set_epsilon(mut slf: PyRefMut<'_, Self>, value: f64) -> PyRefMut<'_, 
Self> {
+        slf.inner = slf.inner.clone().set_epsilon(value);
+        slf
+    }
+
+    #[pyo3(name = "setMaxBlockSizeInMB")]
+    fn set_max_block_size_in_mb(mut slf: PyRefMut<'_, Self>, value: f64) -> 
PyRefMut<'_, Self> {
+        slf.inner = slf.inner.clone().set_max_block_size_in_mb(value);
+        slf
+    }
+
+    fn fit(&self, py: Python<'_>, df: &PyDataFrame) -> PyResult<PyMLModel> {
+        let mut estimator = self.inner.clone();
+        let model = py
+            .detach(|| estimator.fit(&df.dataframe).map_err(Box::new))
+            .to_pyerr()?;
+        Ok(PyMLModel { model })
+    }
+}
+
 /// RegressionEvaluator.
 #[pyclass(name = "RegressionEvaluator", module = 
"pyspark.ml.connect.evaluation")]
 pub struct PyRegressionEvaluator {
diff --git a/crates/spark-connect/src/ml.rs b/crates/spark-connect/src/ml.rs
index ea40cc3b..0d2da7f5 100644
--- a/crates/spark-connect/src/ml.rs
+++ b/crates/spark-connect/src/ml.rs
@@ -42,6 +42,22 @@ impl Params {
         self
     }
 
+    /// Encode an ML parameter whose JVM type is a 32-bit integer.
+    fn set_param_int32(mut self, name: &str, value: i32) -> Self {
+        let literal = proto::expression::Literal {
+            literal_type: 
Some(proto::expression::literal::LiteralType::Integer(value)),
+            ..Default::default()
+        };
+        self.params.insert(
+            name.to_string(),
+            proto::Expression {
+                expr_type: Some(proto::expression::ExprType::Literal(literal)),
+                ..Default::default()
+            },
+        );
+        self
+    }
+
     /// Set a parameter to a double value.
     pub fn set_param_double(mut self, name: &str, value: f64) -> Self {
         let mut literal = proto::Expression::default();
@@ -1283,6 +1299,200 @@ impl Model for LinearSVCModel {
     }
 }
 
+/// Linear regression using Spark's JVM implementation and ML vector features.
+///
+/// Supports squared-error and Huber loss. Unset parameters use Spark's 
defaults.
+#[derive(Debug, Clone)]
+pub struct LinearRegression {
+    operator: MlOperator,
+    params: Params,
+}
+
+impl LinearRegression {
+    pub fn new() -> Self {
+        Self {
+            operator: MlOperator::new(
+                "org.apache.spark.ml.regression.LinearRegression",
+                OperatorType::Estimator,
+            ),
+            params: Params::new(),
+        }
+    }
+
+    /// Feature vector column.
+    pub fn set_feature_col(mut self, value: &str) -> Self {
+        self.params = self.params.set_param_string("featuresCol", value);
+        self
+    }
+
+    /// Numeric label column.
+    pub fn set_label_col(mut self, value: &str) -> Self {
+        self.params = self.params.set_param_string("labelCol", value);
+        self
+    }
+
+    /// Prediction output column.
+    pub fn set_prediction_col(mut self, value: &str) -> Self {
+        self.params = self.params.set_param_string("predictionCol", value);
+        self
+    }
+
+    /// Maximum number of optimizer iterations.
+    pub fn set_max_iter(mut self, value: i32) -> Self {
+        self.params = self.params.set_param_int32("maxIter", value);
+        self
+    }
+
+    /// Regularization strength.
+    pub fn set_reg_param(mut self, value: f64) -> Self {
+        self.params = self.params.set_param_double("regParam", value);
+        self
+    }
+
+    /// Elastic-net mixing: 0 for L2, 1 for L1.
+    pub fn set_elastic_net_param(mut self, value: f64) -> Self {
+        self.params = self.params.set_param_double("elasticNetParam", value);
+        self
+    }
+
+    /// Convergence tolerance.
+    pub fn set_tol(mut self, value: f64) -> Self {
+        self.params = self.params.set_param_double("tol", value);
+        self
+    }
+
+    /// Whether to fit an intercept.
+    pub fn set_fit_intercept(mut self, value: bool) -> Self {
+        self.params = self.params.set_param_bool("fitIntercept", value);
+        self
+    }
+
+    /// Whether to standardize training features.
+    pub fn set_standardization(mut self, value: bool) -> Self {
+        self.params = self.params.set_param_bool("standardization", value);
+        self
+    }
+
+    /// Solver: auto, normal or l-bfgs.
+    pub fn set_solver(mut self, value: &str) -> Self {
+        self.params = self.params.set_param_string("solver", value);
+        self
+    }
+
+    /// Optional instance weight column.
+    pub fn set_weight_col(mut self, value: &str) -> Self {
+        self.params = self.params.set_param_string("weightCol", value);
+        self
+    }
+
+    /// Tree aggregation depth.
+    pub fn set_aggregation_depth(mut self, value: i32) -> Self {
+        self.params = self.params.set_param_int32("aggregationDepth", value);
+        self
+    }
+
+    /// Loss function: squaredError or huber.
+    pub fn set_loss(mut self, value: &str) -> Self {
+        self.params = self.params.set_param_string("loss", value);
+        self
+    }
+
+    /// Huber loss shape parameter; must be greater than 1.
+    pub fn set_epsilon(mut self, value: f64) -> Self {
+        self.params = self.params.set_param_double("epsilon", value);
+        self
+    }
+
+    /// Maximum size in MB of stacked input blocks.
+    pub fn set_max_block_size_in_mb(mut self, value: f64) -> Self {
+        self.params = self.params.set_param_double("maxBlockSizeInMB", value);
+        self
+    }
+
+    fn model_from_info(
+        &self,
+        info: proto::ml_command_result::MlOperatorInfo,
+        session: &SparkSession,
+    ) -> spark_connect_core::error::Result<LinearRegressionModel> {
+        let cached = CachedModel::from_info(&info, session, 
"LinearRegression")?;
+        Ok(LinearRegressionModel {
+            operator: MlOperator::with_uid(
+                "org.apache.spark.ml.regression.LinearRegressionModel",
+                info.uid.as_deref().unwrap_or(&self.operator.uid),
+                OperatorType::Model,
+            ),
+            params: self.params.clone(),
+            cached,
+        })
+    }
+}
+
+impl Default for LinearRegression {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl Estimator for LinearRegression {
+    fn operator(&self) -> &MlOperator {
+        &self.operator
+    }
+    fn operator_mut(&mut self) -> &mut MlOperator {
+        &mut self.operator
+    }
+    fn params(&self) -> &Params {
+        &self.params
+    }
+    fn params_mut(&mut self) -> &mut Params {
+        &mut self.params
+    }
+
+    fn fit_impl(&mut self, df: &DataFrame) -> 
spark_connect_core::error::Result<Box<dyn Model>> {
+        let info = fit_via_command(&self.operator, &self.params, df)?;
+        Ok(Box::new(self.model_from_info(info, &df.session)?))
+    }
+}
+
+/// A linear regression model fitted by Spark Connect.
+///
+/// Obtain it through [`Estimator::fit`]. The model, its clones and transformed
+/// DataFrames retain the server-cached object until the last owner is dropped.
+#[derive(Debug, Clone)]
+pub struct LinearRegressionModel {
+    operator: MlOperator,
+    params: Params,
+    cached: CachedModel,
+}
+
+impl Transformer for LinearRegressionModel {
+    fn operator(&self) -> &MlOperator {
+        &self.operator
+    }
+    fn operator_mut(&mut self) -> &mut MlOperator {
+        &mut self.operator
+    }
+    fn params(&self) -> &Params {
+        &self.params
+    }
+    fn params_mut(&mut self) -> &mut Params {
+        &mut self.params
+    }
+
+    fn build_ml_relation(&self, input: &DataFrame) -> proto::MlRelation {
+        self.cached.build_ml_relation(input, &self.params)
+    }
+
+    fn transform_impl(&mut self, df: &DataFrame) -> 
spark_connect_core::error::Result<DataFrame> {
+        self.cached.transform(df, &self.params)
+    }
+}
+
+impl Model for LinearRegressionModel {
+    fn clone_box(&self) -> Box<dyn Model> {
+        Box::new(self.clone())
+    }
+}
+
 /// RegressionEvaluator: evaluates regression models.
 #[derive(Debug, Clone)]
 pub struct RegressionEvaluator {
@@ -2305,4 +2515,73 @@ mod tests {
             .model_from_info(Default::default(), &session)
             .is_err());
     }
+
+    #[test]
+    fn 
linear_regression_model_validates_the_reference_and_retains_its_metadata() {
+        let session = offline_session();
+        let estimator = LinearRegression::default().set_prediction_col("pred");
+        assert_eq!(
+            estimator.operator().name,
+            "org.apache.spark.ml.regression.LinearRegression"
+        );
+        for kind in [
+            None,
+            Some(proto::ml_command_result::ml_operator_info::Type::Name(
+                "model".into(),
+            )),
+            Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(
+                proto::ObjectRef::default(),
+            )),
+        ] {
+            assert!(estimator
+                .model_from_info(
+                    proto::ml_command_result::MlOperatorInfo {
+                        r#type: kind,
+                        ..Default::default()
+                    },
+                    &session
+                )
+                .is_err());
+        }
+        let mut model = estimator
+            .model_from_info(
+                proto::ml_command_result::MlOperatorInfo {
+                    r#type: 
Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(
+                        proto::ObjectRef {
+                            id: "trained-model".into(),
+                        },
+                    )),
+                    uid: Some("returned-uid".into()),
+                    ..Default::default()
+                },
+                &session,
+            )
+            .unwrap();
+        assert_eq!(model.operator().uid, "returned-uid");
+        assert_eq!(
+            model.operator().name,
+            "org.apache.spark.ml.regression.LinearRegressionModel"
+        );
+        assert_eq!(model.operator().op_type, OperatorType::Model);
+        let df = session.range(2).unwrap();
+        let proto::ml_relation::MlType::Transform(t) =
+            model.build_ml_relation(&df).ml_type.unwrap()
+        else {
+            panic!("transform")
+        };
+        assert_eq!(
+            t.operator,
+            Some(proto::ml_relation::transform::Operator::ObjRef(
+                proto::ObjectRef {
+                    id: "trained-model".into()
+                }
+            ))
+        );
+        assert_eq!(*t.input.unwrap(), df.relation());
+        assert!(t.params.unwrap().params.contains_key("predictionCol"));
+        assert!(model
+            .transform(&session.new_session().range(2).unwrap())
+            .is_err());
+        assert!(model.clone_box().transform(&df).is_ok());
+    }
 }
diff --git a/crates/spark-connect/tests/e2e_linear_regression.rs 
b/crates/spark-connect/tests/e2e_linear_regression.rs
new file mode 100644
index 00000000..175e1c0e
--- /dev/null
+++ b/crates/spark-connect/tests/e2e_linear_regression.rs
@@ -0,0 +1,243 @@
+//! Numerical tests of native LinearRegression. Set SPARK_REMOTE to use a live 
server.
+use spark_connect::column::{col, lit};
+use spark_connect::expression::Expression;
+use spark_connect::ml::{Estimator, Evaluator, LinearRegression, 
RegressionEvaluator};
+use spark_connect::{Column, DataFrame, SparkSession};
+
+fn session() -> Option<SparkSession> {
+    let remote = std::env::var("SPARK_REMOTE").ok()?;
+    Some(
+        SparkSession::builder()
+            .remote(&remote)
+            .get_or_create()
+            .unwrap(),
+    )
+}
+
+fn vectorize(df: DataFrame, name: &str) -> DataFrame {
+    df.with_column(
+        name,
+        Column::new(Expression::function(
+            "array_to_vector",
+            vec![col("xs").expression().clone()],
+        )),
+    )
+}
+
+fn predictions(df: DataFrame, name: &str) -> Vec<f64> {
+    df.order_by(vec![col("id").asc().expression().clone()])
+        .select([col(name)])
+        .collect()
+        .unwrap()
+        .into_iter()
+        .map(|r| r.get(0).unwrap().as_f64().unwrap())
+        .collect()
+}
+
+fn assert_close(actual: Vec<f64>, expected: &[f64], tolerance: f64) {
+    assert_eq!(actual.len(), expected.len());
+    for (a, e) in actual.into_iter().zip(expected) {
+        assert!((a - e).abs() < tolerance, "expected {e}, got {a}");
+    }
+}
+
+#[test]
+fn linear_regression_fits_different_lines_and_keeps_lazy_predictions_alive() {
+    let Some(s) = session() else {
+        return;
+    };
+    let training = vectorize(
+        s.sql(
+            "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(2*x+3 AS DOUBLE) AS 
target, 1.0D AS weight
+         FROM VALUES (-4), (-3), (-2), (-1), (1), (2), (3), (4) AS t(x)",
+        )
+        .unwrap(),
+        "input",
+    );
+    let held_out = vectorize(
+        s.sql(
+            "SELECT id, array(CAST(x AS DOUBLE)) AS xs, CAST(2*x+3 AS DOUBLE) 
AS target
+         FROM VALUES (0, -1.5), (1, 0.5), (2, 4.5) AS t(id, x)",
+        )
+        .unwrap(),
+        "input",
+    );
+    let mut estimator = LinearRegression::new()
+        .set_feature_col("input")
+        .set_label_col("target")
+        .set_prediction_col("pred")
+        .set_max_iter(100)
+        .set_reg_param(0.0)
+        .set_elastic_net_param(0.0)
+        .set_tol(1e-8)
+        .set_fit_intercept(true)
+        .set_standardization(false)
+        .set_solver("normal")
+        .set_weight_col("weight")
+        .set_aggregation_depth(2)
+        .set_loss("squaredError")
+        .set_epsilon(1.35)
+        .set_max_block_size_in_mb(1.0);
+    let mut first = estimator.fit(&training).unwrap();
+    let mut second = estimator
+        .fit(&training.with_column("target", lit(10) - col("target")))
+        .unwrap();
+    let object_id = |model: &dyn spark_connect::ml::Model| {
+        let spark_connect_proto::ml_relation::MlType::Transform(t) =
+            model.build_ml_relation(&held_out).ml_type.unwrap()
+        else {
+            panic!("transform")
+        };
+        let 
Some(spark_connect_proto::ml_relation::transform::Operator::ObjRef(r)) = 
t.operator
+        else {
+            panic!("cached model")
+        };
+        r.id
+    };
+    assert_ne!(object_id(first.as_ref()), object_id(second.as_ref()));
+    let mut cloned = first.clone_box();
+    let delayed =
+        first
+            .transform(&held_out)
+            .unwrap()
+            .select([col("id"), col("target"), col("pred")]);
+    drop(first);
+    assert_close(
+        predictions(cloned.transform(&held_out).unwrap(), "pred"),
+        &[0.0, 4.0, 12.0],
+        1e-6,
+    );
+    drop(cloned);
+    assert_close(
+        predictions(delayed.clone(), "pred"),
+        &[0.0, 4.0, 12.0],
+        1e-6,
+    );
+    let rmse = RegressionEvaluator::new()
+        .set_label_col("target")
+        .set_prediction_col("pred")
+        .set_metric_name("rmse")
+        .evaluate(&delayed)
+        .unwrap();
+    assert!(rmse < 1e-6, "RMSE {rmse}");
+    assert_close(
+        predictions(second.transform(&held_out).unwrap(), "pred"),
+        &[10.0, 6.0, -2.0],
+        1e-6,
+    );
+    assert!(second
+        .transform(&s.new_session().range(1).unwrap())
+        .is_err());
+    s.stop().unwrap();
+}
+
+#[test]
+fn linear_regression_default_columns_and_zero_weight_outlier() {
+    let Some(s) = session() else {
+        return;
+    };
+    let training = vectorize(s.sql(
+        "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(y AS DOUBLE) AS label, 
CAST(w AS DOUBLE) AS weight
+         FROM VALUES (-3, -3, 1), (-2, -1, 1), (2, 7, 1), (3, 9, 1), (100, 
-1000, 0) AS t(x,y,w)"
+    ).unwrap(), "features");
+    let held_out = vectorize(
+        s.sql(
+            "SELECT id, array(CAST(x AS DOUBLE)) AS xs FROM VALUES (0, -1.5), 
(1, 4.5) AS t(id,x)",
+        )
+        .unwrap(),
+        "features",
+    );
+    let mut default_model = LinearRegression::default()
+        .fit(&training.filter(col("weight").gt(lit(0))))
+        .unwrap();
+    let mut weighted_model = LinearRegression::new()
+        .set_weight_col("weight")
+        .fit(&training)
+        .unwrap();
+    assert_close(
+        predictions(default_model.transform(&held_out).unwrap(), "prediction"),
+        &[0.0, 12.0],
+        1e-6,
+    );
+    assert_close(
+        predictions(weighted_model.transform(&held_out).unwrap(), 
"prediction"),
+        &[0.0, 12.0],
+        1e-6,
+    );
+    s.stop().unwrap();
+}
+
+#[test]
+fn linear_regression_iterative_elastic_net_and_huber_loss() {
+    let Some(s) = session() else {
+        return;
+    };
+    let training = vectorize(
+        s.sql(
+            "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(2*x AS DOUBLE) AS 
label
+         FROM VALUES (-3), (-2), (-1), (1), (2), (3) AS t(x)",
+        )
+        .unwrap(),
+        "features",
+    );
+    let held_out = vectorize(
+        s.sql("SELECT 0 AS id, array(2.0D) AS xs").unwrap(),
+        "features",
+    );
+    let mut regularized = LinearRegression::new()
+        .set_solver("l-bfgs")
+        .set_fit_intercept(false)
+        .set_reg_param(1.0)
+        .set_elastic_net_param(0.5)
+        .fit(&training)
+        .unwrap();
+    let value = predictions(regularized.transform(&held_out).unwrap(), 
"prediction")[0];
+    assert!(
+        value > 0.0 && value < 3.9,
+        "regularization should shrink slope: {value}"
+    );
+    let robust_training = vectorize(s.sql(
+        "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(y AS DOUBLE) AS label
+         FROM VALUES (-3, -3.01), (-2, -0.98), (-1, 1.02), (1, 4.98), (2, 
7.01), (3, 8.99), (0, 1000) AS t(x,y)"
+    ).unwrap(), "features");
+    let mut huber = LinearRegression::new()
+        .set_loss("huber")
+        .set_solver("l-bfgs")
+        .set_max_iter(200)
+        .set_tol(1e-8)
+        .fit(&robust_training)
+        .unwrap();
+    assert_close(
+        predictions(huber.transform(&held_out).unwrap(), "prediction"),
+        &[7.0],
+        0.2,
+    );
+    s.stop().unwrap();
+}
+
+#[test]
+fn linear_regression_fit_propagates_invalid_data_and_parameter_errors() {
+    let Some(s) = session() else {
+        return;
+    };
+    assert!(LinearRegression::new().fit(&s.range(3).unwrap()).is_err());
+    let training = vectorize(
+        s.sql("SELECT array(CAST(id AS DOUBLE)) AS xs, CAST(id AS DOUBLE) AS 
label FROM range(5)")
+            .unwrap(),
+        "features",
+    );
+    for mut estimator in [
+        LinearRegression::new().set_reg_param(-1.0),
+        LinearRegression::new().set_elastic_net_param(1.5),
+        LinearRegression::new().set_loss("huber").set_epsilon(1.0),
+        LinearRegression::new()
+            .set_solver("normal")
+            .set_loss("huber"),
+    ] {
+        assert!(estimator.fit(&training).is_err());
+    }
+    assert!(LinearRegression::new()
+        .fit(&training.with_column("label", 
spark_connect::column::lit_string("invalid")))
+        .is_err());
+    s.stop().unwrap();
+}
diff --git a/crates/spark-connect/tests/linear_regression_golden.rs 
b/crates/spark-connect/tests/linear_regression_golden.rs
new file mode 100644
index 00000000..cf64c105
--- /dev/null
+++ b/crates/spark-connect/tests/linear_regression_golden.rs
@@ -0,0 +1,36 @@
+//! Match all native LinearRegression parameter encodings to official PySpark 
4.3.0.
+use base64::{engine::general_purpose::STANDARD, Engine};
+use prost::Message;
+use spark_connect::ml::{Estimator, LinearRegression};
+
+#[test]
+fn linear_regression_parameters_match_official_pyspark() {
+    let fixture: serde_json::Value =
+        
serde_json::from_str(include_str!("../../../tests/golden/linear_regression.json")).unwrap();
+    let bytes = STANDARD
+        .decode(fixture["params_b64"].as_str().unwrap())
+        .unwrap();
+    let expected = 
spark_connect_proto::MlParams::decode(bytes.as_slice()).unwrap();
+    let estimator = LinearRegression::new()
+        .set_feature_col("input")
+        .set_label_col("target")
+        .set_prediction_col("pred")
+        .set_max_iter(50)
+        .set_reg_param(0.25)
+        .set_elastic_net_param(0.5)
+        .set_tol(1e-5)
+        .set_fit_intercept(false)
+        .set_standardization(false)
+        .set_solver("l-bfgs")
+        .set_weight_col("weight")
+        .set_aggregation_depth(3)
+        .set_loss("squaredError")
+        .set_epsilon(1.5)
+        .set_max_block_size_in_mb(2.0);
+    assert_eq!(estimator.params().to_proto(), expected);
+    assert!(LinearRegression::default()
+        .params()
+        .to_proto()
+        .params
+        .is_empty());
+}
diff --git a/docs/compatibility.md b/docs/compatibility.md
index e1a0b22d..e8db4120 100644
--- a/docs/compatibility.md
+++ b/docs/compatibility.md
@@ -76,6 +76,23 @@ executed later. When the last of them is dropped, the cached 
model is deleted on
 server, as `pyspark.ml` does, without waiting for the server; a fit that is 
dropped
 at once leaves nothing cached.
 
+## Native LinearRegression
+
+`spark_connect::ml::LinearRegression` fits Spark's JVM linear regression 
estimator
+through the Connect ML protocol, with ML vector features and numeric labels.
+Rust builder setters cover all 15 constructor parameters, including 
regularization,
+instance weights, solver selection and squared-error or Huber loss. Unset 
parameters
+use Spark's defaults.
+
+The native Python binding is available as
+`pyspark.ml.connect.regression.LinearRegression` and 
`pyspark.ml.connect.LinearRegression`.
+Its keyword-only constructor matches the JVM PySpark estimator's parameter 
names
+and defaults, with mutating `setFeaturesCol`, `setRegParam`, etc. methods.
+Fitted models expose `transform` on DataFrames from the fitting session. The 
model,
+its clones and transformed DataFrames keep the cached server model alive until
+the last owner is dropped. Model inspection (coefficients, intercept and 
summaries),
+single-vector prediction and persistence are not exposed by this native 
addition.
+
 ## Native LinearSVC
 
 `spark_connect::ml::LinearSVC` trains Spark's JVM binary linear support vector
diff --git a/python/pyspark/ml/connect/__init__.py 
b/python/pyspark/ml/connect/__init__.py
index 3d2e3179..e253db52 100644
--- a/python/pyspark/ml/connect/__init__.py
+++ b/python/pyspark/ml/connect/__init__.py
@@ -12,6 +12,7 @@ from pyspark.ml.connect.feature import (
     VectorAssembler,
 )
 from pyspark.ml.connect.classification import LinearSVC, LogisticRegression
+from pyspark.ml.connect.regression import LinearRegression
 from pyspark.ml.connect.evaluation import (
     RegressionEvaluator,
     BinaryClassificationEvaluator,
@@ -44,6 +45,7 @@ __all__ = [
     "VectorAssembler",
     "LogisticRegression",
     "LinearSVC",
+    "LinearRegression",
     "RegressionEvaluator",
     "BinaryClassificationEvaluator",
     "MulticlassClassificationEvaluator",
diff --git a/python/pyspark/ml/connect/regression.py 
b/python/pyspark/ml/connect/regression.py
new file mode 100644
index 00000000..b150f791
--- /dev/null
+++ b/python/pyspark/ml/connect/regression.py
@@ -0,0 +1,4 @@
+"""Native regression estimators backed by Spark Connect's JVM ML API."""
+from pyspark._pyspark import LinearRegression
+
+__all__ = ["LinearRegression"]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 7bc7a6b7..18ab1f3f 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -3921,3 +3921,70 @@ def 
test_linear_svc_fits_the_labels_and_predicts_on_new_rows(session, use_setter
         estimator.fit(training.withColumn("target", F.lit(2.0)))
     with pytest.raises(Exception):
         estimator.setRegParam(-1.0).fit(training)
+
+
+def _linear_regression():
+    try:
+        from pyspark._pyspark import LinearRegression
+    except ImportError:
+        from pyspark.ml.regression import LinearRegression
+    return LinearRegression
+
+
[email protected]("use_setters", [False, True], ids=["constructor", 
"setters"])
+def test_linear_regression_learns_the_data_and_predicts_unseen_rows(session, 
use_setters):
+    from pyspark.ml.connect.functions import array_to_vector
+    LinearRegression = _linear_regression()
+    training = session.sql(
+        "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(2*x+3 AS DOUBLE) AS 
target, 1.0D AS weight "
+        "FROM VALUES (-4), (-3), (-2), (-1), (1), (2), (3), (4) AS t(x)"
+    ).withColumn("input", array_to_vector("xs"))
+    held_out = session.sql(
+        "SELECT id, array(CAST(x AS DOUBLE)) AS xs, CAST(2*x+3 AS DOUBLE) AS 
target "
+        "FROM VALUES (0, -1.5), (1, 0.5), (2, 4.5) AS t(id,x)"
+    ).withColumn("input", array_to_vector("xs"))
+    kwargs = dict(featuresCol="input", labelCol="target", predictionCol="pred",
+                  maxIter=100, regParam=0.0, elasticNetParam=0.0, tol=1e-8,
+                  fitIntercept=True, standardization=False, solver="normal",
+                  weightCol="weight", aggregationDepth=2, loss="squaredError",
+                  epsilon=1.35, maxBlockSizeInMB=1.0)
+    estimator = LinearRegression() if use_setters else 
LinearRegression(**kwargs)
+    if use_setters:
+        for name, value in kwargs.items():
+            if name == "loss" and type(estimator).__module__ == 
"pyspark.ml.regression":
+                # Upstream 4.3.0 setLoss uses the nonexistent lossType 
parameter.
+                # Compare the intended loss setting through its working 
setParams API.
+                assert estimator.setParams(loss=value) is estimator
+            else:
+                assert getattr(estimator, "set" + name[0].upper() + 
name[1:])(value) is estimator
+    first = estimator.fit(training)
+    second = estimator.fit(training.withColumn("target", F.lit(10.0) - 
F.col("target")))
+    assert [r.pred for r in first.transform(held_out).orderBy("id").collect()] 
== pytest.approx([0, 4, 12], abs=1e-6)
+    assert [r.pred for r in 
second.transform(held_out).orderBy("id").collect()] == pytest.approx([10, 6, 
-2], abs=1e-6)
+    with pytest.raises(Exception):
+        estimator.fit(session.range(3))
+    with pytest.raises(Exception):
+        estimator.setRegParam(-1.0).fit(training)
+
+
+def test_linear_regression_weights_and_iterative_losses(session):
+    from pyspark.ml.connect.functions import array_to_vector
+    LinearRegression = _linear_regression()
+    training = session.sql(
+        "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(y AS DOUBLE) AS label, 
CAST(w AS DOUBLE) AS weight "
+        "FROM VALUES (-3,-3,1), (-2,-1,1), (2,7,1), (3,9,1), (100,-1000,0) AS 
t(x,y,w)"
+    ).withColumn("features", array_to_vector("xs"))
+    held_out = session.sql("SELECT array(2.0D) AS xs").withColumn("features", 
array_to_vector("xs"))
+    default = LinearRegression().fit(training.filter(F.col("weight") > 0))
+    weighted = LinearRegression(weightCol="weight").fit(training)
+    assert default.transform(held_out).first().prediction == pytest.approx(7, 
abs=1e-6)
+    assert weighted.transform(held_out).first().prediction == pytest.approx(7, 
abs=1e-6)
+    regularized = LinearRegression(solver="l-bfgs", regParam=1.0, 
elasticNetParam=0.5).fit(
+        training.filter(F.col("weight") > 0))
+    assert 3 < regularized.transform(held_out).first().prediction < 6.9
+    robust_training = session.sql(
+        "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(y AS DOUBLE) AS label "
+        "FROM VALUES (-3,-3.01), (-2,-0.98), (-1,1.02), (1,4.98), (2,7.01), 
(3,8.99), (0,1000) AS t(x,y)"
+    ).withColumn("features", array_to_vector("xs"))
+    robust = LinearRegression(loss="huber", solver="l-bfgs", maxIter=200, 
tol=1e-8).fit(robust_training)
+    assert robust.transform(held_out).first().prediction == pytest.approx(7, 
abs=0.2)
diff --git a/scripts/capture_linear_regression_golden.py 
b/scripts/capture_linear_regression_golden.py
new file mode 100644
index 00000000..87e322c5
--- /dev/null
+++ b/scripts/capture_linear_regression_golden.py
@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+"""Capture native LinearRegression parameter wire goldens from official 
PySpark.
+
+Run with official Spark 4.3.0 on PYTHONPATH and a Connect server available:
+    python scripts/capture_linear_regression_golden.py --remote 
sc://localhost:15002
+"""
+
+import argparse
+import base64
+import json
+from pathlib import Path
+
+from pyspark.ml.connect.serialize import serialize_ml_params
+from pyspark.ml.regression import LinearRegression
+from pyspark.sql import SparkSession
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--remote", default="sc://localhost:15002")
+    parser.add_argument("--out", type=Path, 
default=Path("tests/golden/linear_regression.json"))
+    args = parser.parse_args()
+    spark = SparkSession.builder.remote(args.remote).getOrCreate()
+    try:
+        estimator = LinearRegression(
+            featuresCol="input",
+            labelCol="target",
+            predictionCol="pred",
+            maxIter=50,
+            regParam=0.25,
+            elasticNetParam=0.5,
+            tol=1e-5,
+            fitIntercept=False,
+            standardization=False,
+            solver="l-bfgs",
+            weightCol="weight",
+            aggregationDepth=3,
+            loss="squaredError",
+            epsilon=1.5,
+            maxBlockSizeInMB=2.0,
+        )
+        params = serialize_ml_params(estimator, spark.client)
+        args.out.write_text(
+            json.dumps(
+                {
+                    "source": "apache/spark v4.3.0 
pyspark.ml.regression.LinearRegression",
+                    "params_b64": base64.b64encode(
+                        params.SerializeToString(deterministic=True)
+                    ).decode(),
+                },
+                indent=2,
+            )
+            + "\n"
+        )
+    finally:
+        spark.stop()
+
+
+if __name__ == "__main__":
+    main()
diff --git a/tests/golden/linear_regression.json 
b/tests/golden/linear_regression.json
new file mode 100644
index 00000000..5943d2f7
--- /dev/null
+++ b/tests/golden/linear_regression.json
@@ -0,0 +1,4 @@
+{
+  "source": "apache/spark v4.3.0 pyspark.ml.regression.LinearRegression",
+  "params_b64": 
"ChYKEGFnZ3JlZ2F0aW9uRGVwdGgSAjADChwKD2VsYXN0aWNOZXRQYXJhbRIJWQAAAAAAAOA/ChQKB2Vwc2lsb24SCVkAAAAAAAD4PwoWCgtmZWF0dXJlc0NvbBIHagVpbnB1dAoSCgxmaXRJbnRlcmNlcHQSAhgAChQKCGxhYmVsQ29sEghqBnRhcmdldAoWCgRsb3NzEg5qDHNxdWFyZWRFcnJvcgodChBtYXhCbG9ja1NpemVJbk1CEglZAAAAAAAAAEAKDQoHbWF4SXRlchICMDIKFwoNcHJlZGljdGlvbkNvbBIGagRwcmVkChUKCHJlZ1BhcmFtEglZAAAAAAAA0D8KEgoGc29sdmVyEghqBmwtYmZncwoVCg9zdGFuZGFyZGl6YXRpb24SAhgAChAKA3RvbBIJWfFo44i1+OQ+ChUKCXdlaWdodENvbBIIagZ3ZWlnaHQ="
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to