This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 7e0bc5f [SPARK-61077][CONNECT] Support native LinearSVC training and
prediction
7e0bc5f is described below
commit 7e0bc5f273049e83259a5c2dce5e144e7ea0b7b0
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Sat Nov 7 13:32:32 2026 +0900
[SPARK-61077][CONNECT] Support native LinearSVC training and prediction
### What changes were proposed in this pull request?
Add native `LinearSVC` and its fitted model so the Rust client can train a
binary linear support vector classifier and predict on new rows. Training sends
the dataset and parameters through `MlCommand.Fit`; prediction uses the
returned cached model object reference.
Expose typed Rust setters for all 13 constructor parameters and a Python
binding under `pyspark.ml.connect.classification.LinearSVC` and
`pyspark.ml.connect.LinearSVC`. Its keyword-only constructor follows the JVM
PySpark estimator's parameter names and defaults; Python setters mutate and
return the estimator.
Share the existing LogisticRegression fit and cached-model handling,
including malformed-response validation and session isolation. Add the native
live tests to CI's explicit test list.
### Why are the changes needed?
The Rust client currently has no LinearSVC estimator. This enables binary
support vector classification through Spark Connect with regularization,
instance weights and configurable decision thresholds.
[SPARK-61077](https://issues.apache.org/jira/browse/SPARK-61077)
### Does this PR introduce _any_ user-facing change?
Yes. Rust and Python callers can fit Spark's JVM LinearSVC with vector
features and binary labels 0/1, then transform another DataFrame from the same
session. Invalid training data and parameters fail during fitting. Cached
models remain available until session closure so model clones and lazy
prediction DataFrames remain valid.
This addition exposes fitting and DataFrame transformation.
Coefficient/intercept access, summaries, single-vector prediction and model
persistence are outside this API addition; the compatibility documentation
records these limits.
### How was this patch tested?
- Added three live Rust tests covering learned binary predictions, distinct
fits on reversed labels, all 13 parameter setters, default columns, zero-weight
examples, threshold changes, model cloning/lazy execution, session isolation
and invalid training data. All passed against Spark 4.3.0; all three
LogisticRegression live regression tests passed too.
- Added Python integration cases for constructor and mutating-setter usage.
Both pass with the native binding and official JVM PySpark as reference; the
native LogisticRegression regression case also passes. Verified all 13
constructor names, defaults and keyword-only arguments against the reference.
- Rust: 466 library tests passed; the complete run without a server passed
702 tests, with 26 ignored (server-gated tests return early without
SPARK_REMOTE). All integration targets compile with and without wasm-udf;
formatting and library Clippy passed.
- Python offline suite: 1,023 passed, 8 skipped. Method-name comparison
across the 12 existing core wrapper classes found no gaps under the documented
transport exclusions.
- Workspace/all-target Clippy still fails on existing tests and Python
bindings. Comparing diagnostics with the exact base commit found no new errors.
- Official SQL Connect suite through the Rust transport: all 87 files
passed (2,638 tests passed, 589 skipped), using the existing CI exclusions and
known-failure manifest.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6)
This pull request and its description were written by Isaac.
Closes #272 from zhengruifeng/add-linear-svc.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/build_python_connect.yml | 2 +-
crates/pyspark-rs/src/lib.rs | 1 +
crates/pyspark-rs/src/ml.rs | 141 ++++++++-
crates/spark-connect/src/ml.rs | 450 ++++++++++++++++++++++-----
crates/spark-connect/tests/e2e_linear_svc.rs | 162 ++++++++++
docs/compatibility.md | 16 +
python/pyspark/ml/connect/__init__.py | 3 +-
python/pyspark/ml/connect/classification.py | 4 +-
python/tests/test_dropin_e2e.py | 46 +++
9 files changed, 748 insertions(+), 77 deletions(-)
diff --git a/.github/workflows/build_python_connect.yml
b/.github/workflows/build_python_connect.yml
index 14833b2..c208e41 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -390,7 +390,7 @@ jobs:
run: |
cargo test -p apache-spark-connect \
--test e2e_functions --test e2e_columns --test e2e_dataframe \
- --test e2e_coverage --test e2e_integration \
+ --test e2e_coverage --test e2e_linear_svc --test e2e_integration \
-- --test-threads=1
# ---------------------------------------------------------------------------
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 43d462c..95e95d9 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -248,6 +248,7 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) ->
PyResult<()> {
m.add_class::<ml::PyStringIndexer>()?;
m.add_class::<ml::PyVectorAssembler>()?;
m.add_class::<ml::PyLogisticRegression>()?;
+ m.add_class::<ml::PyLinearSVC>()?;
m.add_class::<ml::PyRegressionEvaluator>()?;
m.add_class::<ml::PyBinaryClassificationEvaluator>()?;
m.add_class::<ml::PyPipeline>()?;
diff --git a/crates/pyspark-rs/src/ml.rs b/crates/pyspark-rs/src/ml.rs
index 252b38c..f3f68f4 100644
--- a/crates/pyspark-rs/src/ml.rs
+++ b/crates/pyspark-rs/src/ml.rs
@@ -7,9 +7,9 @@
use pyo3::prelude::*;
use spark_connect::ml::{
- BinaryClassificationEvaluator, CrossValidator, Estimator, Evaluator,
LogisticRegression,
- MaxAbsScaler, Model, MulticlassClassificationEvaluator, Pipeline,
RegressionEvaluator,
- StandardScaler, StringIndexer, Transformer, VectorAssembler,
+ BinaryClassificationEvaluator, CrossValidator, Estimator, Evaluator,
LinearSVC,
+ LogisticRegression, MaxAbsScaler, Model,
MulticlassClassificationEvaluator, Pipeline,
+ RegressionEvaluator, StandardScaler, StringIndexer, Transformer,
VectorAssembler,
};
use crate::dataframe::PyDataFrame;
@@ -259,6 +259,141 @@ impl PyLogisticRegression {
}
}
+/// Linear support vector classifier backed by Spark Connect's JVM ML API.
+#[pyclass(name = "LinearSVC", module = "pyspark.ml.connect.classification")]
+pub struct PyLinearSVC {
+ inner: LinearSVC,
+}
+
+#[pymethods]
+impl PyLinearSVC {
+ #[new]
+ #[pyo3(signature = (*, featuresCol="features", labelCol="label",
predictionCol="prediction", maxIter=100, regParam=0.0, tol=1e-6,
rawPredictionCol="rawPrediction", fitIntercept=true, standardization=true,
threshold=0.0, weightCol=None, aggregationDepth=2, maxBlockSizeInMB=0.0))]
+ #[allow(non_snake_case, clippy::too_many_arguments)]
+ fn new(
+ featuresCol: &str,
+ labelCol: &str,
+ predictionCol: &str,
+ maxIter: i64,
+ regParam: f64,
+ tol: f64,
+ rawPredictionCol: &str,
+ fitIntercept: bool,
+ standardization: bool,
+ threshold: f64,
+ weightCol: Option<&str>,
+ aggregationDepth: i64,
+ maxBlockSizeInMB: f64,
+ ) -> Self {
+ let mut inner = LinearSVC::new()
+ .set_feature_col(featuresCol)
+ .set_label_col(labelCol)
+ .set_prediction_col(predictionCol)
+ .set_max_iter(maxIter)
+ .set_reg_param(regParam)
+ .set_tol(tol)
+ .set_raw_prediction_col(rawPredictionCol)
+ .set_fit_intercept(fitIntercept)
+ .set_standardization(standardization)
+ .set_threshold(threshold)
+ .set_aggregation_depth(aggregationDepth)
+ .set_max_block_size_in_mb(maxBlockSizeInMB);
+ if let Some(col) = weightCol {
+ inner = inner.set_weight_col(col);
+ }
+ Self { inner }
+ }
+
+ #[pyo3(name = "setFeaturesCol")]
+ fn set_feature_col<'py>(mut slf: PyRefMut<'py, Self>, value: &str) ->
PyRefMut<'py, Self> {
+ slf.inner = slf.inner.clone().set_feature_col(value);
+ slf
+ }
+
+ #[pyo3(name = "setLabelCol")]
+ fn set_label_col<'py>(mut slf: PyRefMut<'py, Self>, value: &str) ->
PyRefMut<'py, Self> {
+ slf.inner = slf.inner.clone().set_label_col(value);
+ slf
+ }
+
+ #[pyo3(name = "setPredictionCol")]
+ fn set_prediction_col<'py>(mut slf: PyRefMut<'py, Self>, value: &str) ->
PyRefMut<'py, Self> {
+ slf.inner = slf.inner.clone().set_prediction_col(value);
+ slf
+ }
+
+ #[pyo3(name = "setMaxIter")]
+ fn set_max_iter(mut slf: PyRefMut<'_, Self>, value: i64) -> PyRefMut<'_,
Self> {
+ slf.inner = slf.inner.clone().set_max_iter(value);
+ slf
+ }
+
+ #[pyo3(name = "setRegParam")]
+ fn set_reg_param(mut slf: PyRefMut<'_, Self>, value: f64) -> PyRefMut<'_,
Self> {
+ slf.inner = slf.inner.clone().set_reg_param(value);
+ slf
+ }
+
+ #[pyo3(name = "setTol")]
+ fn set_tol(mut slf: PyRefMut<'_, Self>, value: f64) -> PyRefMut<'_, Self> {
+ slf.inner = slf.inner.clone().set_tol(value);
+ slf
+ }
+
+ #[pyo3(name = "setRawPredictionCol")]
+ fn set_raw_prediction_col<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ value: &str,
+ ) -> PyRefMut<'py, Self> {
+ slf.inner = slf.inner.clone().set_raw_prediction_col(value);
+ slf
+ }
+
+ #[pyo3(name = "setFitIntercept")]
+ fn set_fit_intercept(mut slf: PyRefMut<'_, Self>, value: bool) ->
PyRefMut<'_, Self> {
+ slf.inner = slf.inner.clone().set_fit_intercept(value);
+ slf
+ }
+
+ #[pyo3(name = "setStandardization")]
+ fn set_standardization(mut slf: PyRefMut<'_, Self>, value: bool) ->
PyRefMut<'_, Self> {
+ slf.inner = slf.inner.clone().set_standardization(value);
+ slf
+ }
+
+ #[pyo3(name = "setThreshold")]
+ fn set_threshold(mut slf: PyRefMut<'_, Self>, value: f64) -> PyRefMut<'_,
Self> {
+ slf.inner = slf.inner.clone().set_threshold(value);
+ slf
+ }
+
+ #[pyo3(name = "setWeightCol")]
+ fn set_weight_col<'py>(mut slf: PyRefMut<'py, Self>, value: &str) ->
PyRefMut<'py, Self> {
+ slf.inner = slf.inner.clone().set_weight_col(value);
+ slf
+ }
+
+ #[pyo3(name = "setAggregationDepth")]
+ fn set_aggregation_depth(mut slf: PyRefMut<'_, Self>, value: i64) ->
PyRefMut<'_, Self> {
+ slf.inner = slf.inner.clone().set_aggregation_depth(value);
+ slf
+ }
+
+ #[pyo3(name = "setMaxBlockSizeInMB")]
+ fn set_max_block_size_in_mb(mut slf: PyRefMut<'_, Self>, value: f64) ->
PyRefMut<'_, Self> {
+ slf.inner = slf.inner.clone().set_max_block_size_in_mb(value);
+ slf
+ }
+
+ fn fit(&self, py: Python<'_>, df: &PyDataFrame) -> PyResult<PyMLModel> {
+ let mut estimator = self.inner.clone();
+ let model = py
+ .detach(|| estimator.fit(&df.dataframe).map_err(Box::new))
+ .to_pyerr()?;
+ Ok(PyMLModel { model })
+ }
+}
+
/// RegressionEvaluator.
#[pyclass(name = "RegressionEvaluator", module =
"pyspark.ml.connect.evaluation")]
pub struct PyRegressionEvaluator {
diff --git a/crates/spark-connect/src/ml.rs b/crates/spark-connect/src/ml.rs
index 6af6605..ef88561 100644
--- a/crates/spark-connect/src/ml.rs
+++ b/crates/spark-connect/src/ml.rs
@@ -800,6 +800,109 @@ impl Model for MaxAbsScalerModel {
}
}
+/// Submit training data and parameters and read the fitted operator metadata.
+fn fit_via_command(
+ operator: &MlOperator,
+ params: &Params,
+ df: &DataFrame,
+) ->
spark_connect_core::error::Result<proto::ml_command_result::MlOperatorInfo> {
+ let command = proto::MlCommand {
+ command: Some(proto::ml_command::Command::Fit(proto::ml_command::Fit {
+ estimator: Some(operator.to_proto()),
+ params: Some(params.to_proto()),
+ dataset: Some(df.relation()),
+ })),
+ };
+ let responses = crate::dataframe::execute_command_collect(
+ &df.session,
+ proto::command::CommandType::MlCommand(command),
+ )?;
+ for response in responses {
+ if let
Some(proto::execute_plan_response::ResponseType::MlCommandResult(result)) =
+ response.response_type
+ {
+ if let
Some(proto::ml_command_result::ResultType::OperatorInfo(info)) =
+ result.result_type
+ {
+ return Ok(info);
+ }
+ }
+ }
+ Err(spark_connect_core::error::SparkError::connect_msg(format!(
+ "{}.fit: server returned no fitted model",
+ operator.name
+ )))
+}
+
+/// A session-owned model reference. Keep it cached until session closure so
that
+/// model clones and lazy transformed DataFrames remain usable after a model
is dropped.
+#[derive(Debug, Clone)]
+struct CachedModel {
+ model_ref: proto::ObjectRef,
+ session_id: String,
+}
+
+impl CachedModel {
+ fn from_info(
+ info: &proto::ml_command_result::MlOperatorInfo,
+ session_id: &str,
+ estimator: &str,
+ ) -> spark_connect_core::error::Result<Self> {
+ match &info.r#type {
+
Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(reference))
+ if !reference.id.is_empty() =>
+ {
+ Ok(Self {
+ model_ref: reference.clone(),
+ session_id: session_id.to_string(),
+ })
+ }
+ _ =>
Err(spark_connect_core::error::SparkError::connect_msg(format!(
+ "{estimator}.fit: server returned no model object reference"
+ ))),
+ }
+ }
+
+ fn build_ml_relation(&self, df: &DataFrame, params: &Params) ->
proto::MlRelation {
+ proto::MlRelation {
+ ml_type: Some(proto::ml_relation::MlType::Transform(Box::new(
+ proto::ml_relation::Transform {
+ operator:
Some(proto::ml_relation::transform::Operator::ObjRef(
+ self.model_ref.clone(),
+ )),
+ input: Some(Box::new(df.relation())),
+ params: Some(params.to_proto()),
+ },
+ ))),
+ ..Default::default()
+ }
+ }
+
+ fn transform(
+ &self,
+ df: &DataFrame,
+ params: &Params,
+ ) -> spark_connect_core::error::Result<DataFrame> {
+ if df.session.session_id() != self.session_id {
+ return Err(spark_connect_core::error::SparkError::value_msg(
+ "A fitted ML model must transform data from its fitting
session",
+ ));
+ }
+ let relation = proto::Relation {
+ rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
+ self.build_ml_relation(df, params),
+ ))),
+ ..Default::default()
+ };
+ Ok(DataFrame::new(
+ df.session.clone(),
+ LogicalPlan::MlTransform {
+ ml_relation: relation,
+ },
+ ))
+ }
+}
+
/// LogisticRegression Estimator: binary/multiclass classification.
#[derive(Debug, Clone)]
pub struct LogisticRegression {
@@ -871,18 +974,7 @@ impl LogisticRegression {
info: proto::ml_command_result::MlOperatorInfo,
session_id: &str,
) -> spark_connect_core::error::Result<LogisticRegressionModel> {
- let model_ref = match info.r#type {
-
Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(reference))
- if !reference.id.is_empty() =>
- {
- reference
- }
- _ => {
- return Err(spark_connect_core::error::SparkError::connect_msg(
- "LogisticRegression.fit: server returned no model object
reference",
- ))
- }
- };
+ let cached = CachedModel::from_info(&info, session_id,
"LogisticRegression")?;
Ok(LogisticRegressionModel {
operator: MlOperator::with_uid(
"org.apache.spark.ml.classification.LogisticRegressionModel",
@@ -893,8 +985,7 @@ impl LogisticRegression {
feature_col: self.feature_col.clone(),
label_col: self.label_col.clone(),
prediction_col: self.prediction_col.clone(),
- model_ref,
- session_id: session_id.to_string(),
+ cached,
})
}
}
@@ -923,32 +1014,9 @@ impl Estimator for LogisticRegression {
}
fn fit_impl(&mut self, df: &DataFrame) ->
spark_connect_core::error::Result<Box<dyn Model>> {
- let command = proto::MlCommand {
- command:
Some(proto::ml_command::Command::Fit(proto::ml_command::Fit {
- estimator: Some(self.operator.to_proto()),
- params: Some(self.params.to_proto()),
- dataset: Some(df.relation()),
- })),
- };
- let responses = crate::dataframe::execute_command_collect(
- &df.session,
- proto::command::CommandType::MlCommand(command),
- )?;
- for response in responses {
- if let
Some(proto::execute_plan_response::ResponseType::MlCommandResult(result)) =
- response.response_type
- {
- if let
Some(proto::ml_command_result::ResultType::OperatorInfo(info)) =
- result.result_type
- {
- return Ok(Box::new(
- self.model_from_info(info, df.session.session_id())?,
- ));
- }
- }
- }
- Err(spark_connect_core::error::SparkError::connect_msg(
- "LogisticRegression.fit: server returned no fitted model",
+ let info = fit_via_command(&self.operator, &self.params, df)?;
+ Ok(Box::new(
+ self.model_from_info(info, df.session.session_id())?,
))
}
}
@@ -965,8 +1033,7 @@ pub struct LogisticRegressionModel {
pub feature_col: String,
pub label_col: String,
pub prediction_col: String,
- model_ref: proto::ObjectRef,
- session_id: String,
+ cached: CachedModel,
}
impl Transformer for LogisticRegressionModel {
@@ -987,42 +1054,199 @@ impl Transformer for LogisticRegressionModel {
}
fn build_ml_relation(&self, input: &DataFrame) -> proto::MlRelation {
- proto::MlRelation {
- ml_type: Some(proto::ml_relation::MlType::Transform(Box::new(
- proto::ml_relation::Transform {
- operator:
Some(proto::ml_relation::transform::Operator::ObjRef(
- self.model_ref.clone(),
- )),
- input: Some(Box::new(input.relation())),
- params: Some(self.params.to_proto()),
- },
- ))),
- ..Default::default()
- }
+ self.cached.build_ml_relation(input, &self.params)
}
fn transform_impl(&mut self, df: &DataFrame) ->
spark_connect_core::error::Result<DataFrame> {
- if df.session.session_id() != self.session_id {
- return Err(spark_connect_core::error::SparkError::value_msg(
- "A fitted LogisticRegression model must transform data from
its fitting session",
- ));
+ self.cached.transform(df, &self.params)
+ }
+}
+
+impl Model for LogisticRegressionModel {
+ fn clone_box(&self) -> Box<dyn Model> {
+ Box::new(self.clone())
+ }
+}
+
+/// Linear support vector classifier using Spark's JVM implementation.
+///
+/// Supports binary labels (0 and 1), ML vector features and L2 regularization.
+/// Unset parameters use Spark's defaults: 100 iterations, no regularization,
+/// tolerance 1e-6, an intercept, standardization and threshold 0.
+#[derive(Debug, Clone)]
+pub struct LinearSVC {
+ operator: MlOperator,
+ params: Params,
+}
+
+impl LinearSVC {
+ pub fn new() -> Self {
+ Self {
+ operator: MlOperator::new(
+ "org.apache.spark.ml.classification.LinearSVC",
+ OperatorType::Estimator,
+ ),
+ params: Params::new(),
}
- let relation = proto::Relation {
- rel_type: Some(proto::relation::RelType::MlRelation(Box::new(
- self.build_ml_relation(df),
- ))),
- ..Default::default()
- };
- Ok(DataFrame::new(
- df.session.clone(),
- LogicalPlan::MlTransform {
- ml_relation: relation,
- },
+ }
+
+ /// Feature vector column.
+ pub fn set_feature_col(mut self, value: &str) -> Self {
+ self.params = self.params.set_param_string("featuresCol", value);
+ self
+ }
+
+ /// Binary label column (0 or 1).
+ pub fn set_label_col(mut self, value: &str) -> Self {
+ self.params = self.params.set_param_string("labelCol", value);
+ self
+ }
+
+ /// Prediction output column.
+ pub fn set_prediction_col(mut self, value: &str) -> Self {
+ self.params = self.params.set_param_string("predictionCol", value);
+ self
+ }
+
+ /// Raw margin output column.
+ pub fn set_raw_prediction_col(mut self, value: &str) -> Self {
+ self.params = self.params.set_param_string("rawPredictionCol", value);
+ self
+ }
+
+ /// Maximum number of optimizer iterations.
+ pub fn set_max_iter(mut self, value: i64) -> Self {
+ self.params = self.params.set_param_int("maxIter", value);
+ self
+ }
+
+ /// L2 regularization strength.
+ pub fn set_reg_param(mut self, value: f64) -> Self {
+ self.params = self.params.set_param_double("regParam", value);
+ self
+ }
+
+ /// Convergence tolerance.
+ pub fn set_tol(mut self, value: f64) -> Self {
+ self.params = self.params.set_param_double("tol", value);
+ self
+ }
+
+ /// Whether to fit an intercept.
+ pub fn set_fit_intercept(mut self, value: bool) -> Self {
+ self.params = self.params.set_param_bool("fitIntercept", value);
+ self
+ }
+
+ /// Whether to standardize training features.
+ pub fn set_standardization(mut self, value: bool) -> Self {
+ self.params = self.params.set_param_bool("standardization", value);
+ self
+ }
+
+ /// Decision threshold applied to the model margin.
+ pub fn set_threshold(mut self, value: f64) -> Self {
+ self.params = self.params.set_param_double("threshold", value);
+ self
+ }
+
+ /// Optional instance weight column.
+ pub fn set_weight_col(mut self, value: &str) -> Self {
+ self.params = self.params.set_param_string("weightCol", value);
+ self
+ }
+
+ /// Tree aggregation depth.
+ pub fn set_aggregation_depth(mut self, value: i64) -> Self {
+ self.params = self.params.set_param_int("aggregationDepth", value);
+ self
+ }
+
+ /// Maximum size in MB of stacked input blocks.
+ pub fn set_max_block_size_in_mb(mut self, value: f64) -> Self {
+ self.params = self.params.set_param_double("maxBlockSizeInMB", value);
+ self
+ }
+
+ fn model_from_info(
+ &self,
+ info: proto::ml_command_result::MlOperatorInfo,
+ session_id: &str,
+ ) -> spark_connect_core::error::Result<LinearSVCModel> {
+ let cached = CachedModel::from_info(&info, session_id, "LinearSVC")?;
+ Ok(LinearSVCModel {
+ operator: MlOperator::with_uid(
+ "org.apache.spark.ml.classification.LinearSVCModel",
+ info.uid.as_deref().unwrap_or(&self.operator.uid),
+ OperatorType::Model,
+ ),
+ params: self.params.clone(),
+ cached,
+ })
+ }
+}
+
+impl Default for LinearSVC {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl Estimator for LinearSVC {
+ fn operator(&self) -> &MlOperator {
+ &self.operator
+ }
+ fn operator_mut(&mut self) -> &mut MlOperator {
+ &mut self.operator
+ }
+ fn params(&self) -> &Params {
+ &self.params
+ }
+ fn params_mut(&mut self) -> &mut Params {
+ &mut self.params
+ }
+
+ fn fit_impl(&mut self, df: &DataFrame) ->
spark_connect_core::error::Result<Box<dyn Model>> {
+ let info = fit_via_command(&self.operator, &self.params, df)?;
+ Ok(Box::new(
+ self.model_from_info(info, df.session.session_id())?,
))
}
}
-impl Model for LogisticRegressionModel {
+/// A LinearSVC model fitted by Spark Connect and cached for its fitting
session.
+/// Obtain it through [`Estimator::fit`]; prediction uses the cached model
object.
+#[derive(Debug, Clone)]
+pub struct LinearSVCModel {
+ operator: MlOperator,
+ params: Params,
+ cached: CachedModel,
+}
+
+impl Transformer for LinearSVCModel {
+ fn operator(&self) -> &MlOperator {
+ &self.operator
+ }
+ fn operator_mut(&mut self) -> &mut MlOperator {
+ &mut self.operator
+ }
+ fn params(&self) -> &Params {
+ &self.params
+ }
+ fn params_mut(&mut self) -> &mut Params {
+ &mut self.params
+ }
+
+ fn build_ml_relation(&self, input: &DataFrame) -> proto::MlRelation {
+ self.cached.build_ml_relation(input, &self.params)
+ }
+
+ fn transform_impl(&mut self, df: &DataFrame) ->
spark_connect_core::error::Result<DataFrame> {
+ self.cached.transform(df, &self.params)
+ }
+}
+
+impl Model for LinearSVCModel {
fn clone_box(&self) -> Box<dyn Model> {
Box::new(self.clone())
}
@@ -1961,4 +2185,90 @@ mod tests {
let other = session.new_session();
assert!(model.transform(&other.range(3).unwrap()).is_err());
}
+
+ #[test]
+ fn linear_svc_parameters_use_the_spark_names_and_literal_types() {
+ use proto::expression::literal::LiteralType as L;
+ let svc = LinearSVC::new()
+ .set_feature_col("input")
+ .set_label_col("target")
+ .set_prediction_col("pred")
+ .set_raw_prediction_col("margin")
+ .set_max_iter(50)
+ .set_reg_param(0.1)
+ .set_tol(1e-5)
+ .set_fit_intercept(false)
+ .set_standardization(false)
+ .set_threshold(0.5)
+ .set_weight_col("weight")
+ .set_aggregation_depth(3)
+ .set_max_block_size_in_mb(1.0);
+ let params = svc.params().to_proto().params;
+ let expected = [
+ ("featuresCol", L::String("input".into())),
+ ("labelCol", L::String("target".into())),
+ ("predictionCol", L::String("pred".into())),
+ ("rawPredictionCol", L::String("margin".into())),
+ ("maxIter", L::Long(50)),
+ ("regParam", L::Double(0.1)),
+ ("tol", L::Double(1e-5)),
+ ("fitIntercept", L::Boolean(false)),
+ ("standardization", L::Boolean(false)),
+ ("threshold", L::Double(0.5)),
+ ("weightCol", L::String("weight".into())),
+ ("aggregationDepth", L::Long(3)),
+ ("maxBlockSizeInMB", L::Double(1.0)),
+ ];
+ assert_eq!(params.len(), expected.len());
+ for (name, literal) in expected {
+ assert_eq!(params[name].literal_type.as_ref(), Some(&literal),
"{name}");
+ }
+ }
+
+ #[test]
+ fn linear_svc_model_uses_its_cached_reference_and_fitting_session() {
+ let session = offline_session();
+ let estimator = LinearSVC::default();
+ assert!(estimator.params().to_proto().params.is_empty());
+ assert_eq!(estimator.operator().op_type, OperatorType::Estimator);
+ let info = proto::ml_command_result::MlOperatorInfo {
+ r#type:
Some(proto::ml_command_result::ml_operator_info::Type::ObjRef(
+ proto::ObjectRef {
+ id: "svc-model-object".into(),
+ },
+ )),
+ ..Default::default()
+ };
+ let mut model = estimator
+ .model_from_info(info, session.session_id())
+ .unwrap();
+ assert_eq!(model.operator().uid, estimator.operator().uid);
+ assert_eq!(
+ model.operator().name,
+ "org.apache.spark.ml.classification.LinearSVCModel"
+ );
+ assert_eq!(model.operator().op_type, OperatorType::Model);
+ let df = session.range(2).unwrap();
+ let proto::ml_relation::MlType::Transform(transform) =
+ model.build_ml_relation(&df).ml_type.unwrap()
+ else {
+ panic!("expected transform");
+ };
+ assert_eq!(
+ transform.operator,
+ Some(proto::ml_relation::transform::Operator::ObjRef(
+ proto::ObjectRef {
+ id: "svc-model-object".into()
+ },
+ ))
+ );
+ assert_eq!(*transform.input.unwrap(), df.relation());
+ assert!(model
+ .transform(&session.new_session().range(2).unwrap())
+ .is_err());
+ assert!(model.clone_box().transform(&df).is_ok());
+ assert!(estimator
+ .model_from_info(Default::default(), session.session_id())
+ .is_err());
+ }
}
diff --git a/crates/spark-connect/tests/e2e_linear_svc.rs
b/crates/spark-connect/tests/e2e_linear_svc.rs
new file mode 100644
index 0000000..16de17d
--- /dev/null
+++ b/crates/spark-connect/tests/e2e_linear_svc.rs
@@ -0,0 +1,162 @@
+//! Native LinearSVC tests. Set SPARK_REMOTE to run against a live Spark
Connect server.
+use spark_connect::column::{col, lit};
+use spark_connect::expression::Expression;
+use spark_connect::ml::{Estimator, LinearSVC};
+use spark_connect::{Column, DataFrame, SparkSession};
+
+fn session() -> Option<SparkSession> {
+ let remote = std::env::var("SPARK_REMOTE").ok()?;
+ Some(
+ SparkSession::builder()
+ .remote(&remote)
+ .get_or_create()
+ .unwrap(),
+ )
+}
+
+fn vectorize(df: DataFrame, name: &str) -> DataFrame {
+ df.with_column(
+ name,
+ Column::new(Expression::function(
+ "array_to_vector",
+ vec![col("xs").expression().clone()],
+ )),
+ )
+}
+
+fn predictions(df: DataFrame, name: &str) -> Vec<f64> {
+ df.order_by(vec![col("id").asc().expression().clone()])
+ .select([col(name)])
+ .collect()
+ .unwrap()
+ .into_iter()
+ .map(|row| row.get(0).unwrap().as_f64().unwrap())
+ .collect()
+}
+
+#[test]
+fn linear_svc_fits_separate_models_and_predicts_on_unseen_data() {
+ let Some(s) = session() else {
+ return;
+ };
+ let training = vectorize(
+ s.sql(
+ "SELECT array(CAST(x AS DOUBLE)) AS xs, \
+ CAST(CASE WHEN x > 0 THEN 1 ELSE 0 END AS DOUBLE) AS target, 1.0D AS
weight \
+ FROM VALUES (-4), (-3), (-2), (-1), (1), (2), (3), (4) AS t(x)",
+ )
+ .unwrap(),
+ "input",
+ );
+ let held_out = vectorize(
+ s.sql(
+ "SELECT id, array(CAST(x AS DOUBLE)) AS xs \
+ FROM VALUES (0, -3.5), (1, -1.5), (2, 1.5), (3, 3.5) AS t(id, x)",
+ )
+ .unwrap(),
+ "input",
+ );
+ let mut estimator = LinearSVC::new()
+ .set_feature_col("input")
+ .set_label_col("target")
+ .set_prediction_col("pred")
+ .set_raw_prediction_col("margin")
+ .set_max_iter(50)
+ .set_reg_param(0.1)
+ .set_tol(1e-6)
+ .set_fit_intercept(false)
+ .set_standardization(false)
+ .set_threshold(0.0)
+ .set_weight_col("weight")
+ .set_aggregation_depth(2)
+ .set_max_block_size_in_mb(1.0);
+ let mut first = estimator.fit(&training).expect("fit binary classifier");
+ let mut second = estimator
+ .fit(&training.with_column("target", lit(1) - col("target")))
+ .unwrap();
+ let object_id = |model: &dyn spark_connect::ml::Model| {
+ let spark_connect_proto::ml_relation::MlType::Transform(t) =
+ model.build_ml_relation(&held_out).ml_type.unwrap()
+ else {
+ panic!("transform")
+ };
+ let
Some(spark_connect_proto::ml_relation::transform::Operator::ObjRef(r)) =
t.operator
+ else {
+ panic!("cached model")
+ };
+ r.id
+ };
+ assert_ne!(object_id(first.as_ref()), object_id(second.as_ref()));
+ let mut cloned = first.clone_box();
+ let delayed = first.transform(&held_out).unwrap();
+ drop(first);
+ let output = cloned.transform(&held_out).unwrap();
+ assert!(output.columns().unwrap().contains(&"margin".to_string()));
+ assert_eq!(predictions(delayed, "pred"), vec![0.0, 0.0, 1.0, 1.0]);
+ assert_eq!(predictions(output, "pred"), vec![0.0, 0.0, 1.0, 1.0]);
+ assert_eq!(
+ predictions(second.transform(&held_out).unwrap(), "pred"),
+ vec![1.0, 1.0, 0.0, 0.0]
+ );
+ let mut params = cloned.params().clone().set_param_double("threshold",
1e6);
+ std::mem::swap(cloned.params_mut(), &mut params);
+ assert_eq!(
+ predictions(cloned.transform(&held_out).unwrap(), "pred"),
+ vec![0.0; 4]
+ );
+ assert!(second
+ .transform(&s.new_session().range(1).unwrap())
+ .is_err());
+ s.stop().unwrap();
+}
+
+#[test]
+fn linear_svc_defaults_and_zero_weight_training_examples() {
+ let Some(s) = session() else {
+ return;
+ };
+ let training = vectorize(s.sql(
+ "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(y AS DOUBLE) AS label,
CAST(w AS DOUBLE) AS weight \
+ FROM VALUES (-3, 0, 1), (-2, 0, 1), (2, 1, 1), (3, 1, 1), (100, 0, 0)
AS t(x, y, w)"
+ ).unwrap(), "features");
+ let held_out = vectorize(
+ s.sql(
+ "SELECT id, array(CAST(x AS DOUBLE)) AS xs FROM VALUES (0, -1.5),
(1, 1.5) AS t(id, x)",
+ )
+ .unwrap(),
+ "features",
+ );
+ let mut default_model = LinearSVC::default()
+ .fit(&training.filter(col("weight").gt(lit(0))))
+ .unwrap();
+ let mut weighted_model = LinearSVC::new()
+ .set_weight_col("weight")
+ .fit(&training)
+ .unwrap();
+ assert_eq!(
+ predictions(default_model.transform(&held_out).unwrap(), "prediction"),
+ vec![0.0, 1.0]
+ );
+ assert_eq!(
+ predictions(weighted_model.transform(&held_out).unwrap(),
"prediction"),
+ vec![0.0, 1.0]
+ );
+ s.stop().unwrap();
+}
+
+#[test]
+fn linear_svc_fit_rejects_invalid_training_data_and_parameters() {
+ let Some(s) = session() else {
+ return;
+ };
+ assert!(LinearSVC::new().fit(&s.range(3).unwrap()).is_err());
+ let multiclass = vectorize(s.sql(
+ "SELECT array(CAST(x AS DOUBLE)) AS xs, CAST(x AS DOUBLE) AS label
FROM VALUES (0), (1), (2) AS t(x)"
+ ).unwrap(), "features");
+ assert!(LinearSVC::new().fit(&multiclass).is_err());
+ assert!(LinearSVC::new()
+ .set_reg_param(-1.0)
+ .fit(&multiclass)
+ .is_err());
+ s.stop().unwrap();
+}
diff --git a/docs/compatibility.md b/docs/compatibility.md
index 39a9a1c..0a4de97 100644
--- a/docs/compatibility.md
+++ b/docs/compatibility.md
@@ -73,6 +73,22 @@ reference and must use the same Spark session. Models are
obtained through
longer supported. The cached model remains available until the session is
closed,
including when a model is cloned or a transformed DataFrame is executed later.
+## Native LinearSVC
+
+`spark_connect::ml::LinearSVC` trains Spark's JVM binary linear support vector
+classifier through the Connect ML protocol. Use vector features and labels 0
or 1.
+Rust builder setters cover all 13 constructor parameters, including
`set_reg_param`,
+`set_tol`, `set_weight_col` and `set_threshold`. Unset parameters use Spark's
defaults.
+The Python binding is available as
`pyspark.ml.connect.classification.LinearSVC`
+and `pyspark.ml.connect.LinearSVC`, with the upstream keyword-only constructor
+parameters and mutating `setFeaturesCol`, `setRegParam`, etc. methods.
+
+`fit` returns a server-cached model; `transform` must use the fitting session.
+Clones and lazy prediction DataFrames remain usable until that session closes.
+The Python model uses the existing `MLModel.transform` interface. Model
inspection
+(coefficients, intercept and summaries), single-vector prediction and model
+persistence are not exposed by this addition.
+
## How parity is verified
Two independent gates in CI cover the two halves of a Spark Connect client -
plan
diff --git a/python/pyspark/ml/connect/__init__.py
b/python/pyspark/ml/connect/__init__.py
index 651fba4..3d2e317 100644
--- a/python/pyspark/ml/connect/__init__.py
+++ b/python/pyspark/ml/connect/__init__.py
@@ -11,7 +11,7 @@ from pyspark.ml.connect.feature import (
StringIndexer,
VectorAssembler,
)
-from pyspark.ml.connect.classification import LogisticRegression
+from pyspark.ml.connect.classification import LinearSVC, LogisticRegression
from pyspark.ml.connect.evaluation import (
RegressionEvaluator,
BinaryClassificationEvaluator,
@@ -43,6 +43,7 @@ __all__ = [
"StringIndexer",
"VectorAssembler",
"LogisticRegression",
+ "LinearSVC",
"RegressionEvaluator",
"BinaryClassificationEvaluator",
"MulticlassClassificationEvaluator",
diff --git a/python/pyspark/ml/connect/classification.py
b/python/pyspark/ml/connect/classification.py
index de105a6..263aa13 100644
--- a/python/pyspark/ml/connect/classification.py
+++ b/python/pyspark/ml/connect/classification.py
@@ -1,4 +1,4 @@
"""Classification estimators (pyspark.ml.connect.classification)."""
-from pyspark._pyspark import LogisticRegression
+from pyspark._pyspark import LinearSVC, LogisticRegression
-__all__ = ["LogisticRegression"]
+__all__ = ["LinearSVC", "LogisticRegression"]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 54dad46..ca18cd7 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2397,3 +2397,49 @@ def
test_logistic_regression_fits_the_data_and_predicts_on_new_rows(session):
]
with pytest.raises(Exception):
estimator.fit(session.range(3))
+
+
[email protected]("use_setters", [False, True], ids=["constructor",
"setters"])
+def test_linear_svc_fits_the_labels_and_predicts_on_new_rows(session,
use_setters):
+ try:
+ from pyspark._pyspark import LinearSVC
+ except ImportError:
+ from pyspark.ml.classification import LinearSVC
+ from pyspark.ml.connect.functions import array_to_vector
+
+ training = session.sql(
+ "SELECT array(CAST(x AS DOUBLE)) AS xs, "
+ "CAST(CASE WHEN x > 0 THEN 1 ELSE 0 END AS DOUBLE) AS target, 1.0D AS
weight "
+ "FROM VALUES (-4), (-3), (-2), (-1), (1), (2), (3), (4) AS t(x)"
+ ).withColumn("input", array_to_vector("xs"))
+ held_out = session.sql(
+ "SELECT id, array(CAST(x AS DOUBLE)) AS xs "
+ "FROM VALUES (0, -3.5), (1, -1.5), (2, 1.5), (3, 3.5) AS t(id, x)"
+ ).withColumn("input", array_to_vector("xs"))
+ kwargs = dict(featuresCol="input", labelCol="target", predictionCol="pred",
+ rawPredictionCol="margin", maxIter=50, regParam=0.1,
tol=1e-6,
+ fitIntercept=False, standardization=False, threshold=0.0,
+ weightCol="weight", aggregationDepth=2, maxBlockSizeInMB=1.0)
+ if use_setters:
+ estimator = LinearSVC()
+ for name, value in kwargs.items():
+ setter = getattr(estimator, "set" + name[0].upper() + name[1:])
+ assert setter(value) is estimator
+ else:
+ estimator = LinearSVC(**kwargs)
+ first = estimator.fit(training)
+ second = estimator.fit(training.withColumn("target", F.lit(1.0) -
F.col("target")))
+ result = first.transform(held_out)
+ assert "margin" in result.columns
+ assert [r.pred for r in result.orderBy("id").collect()] == [0.0, 0.0, 1.0,
1.0]
+ assert [r.pred for r in
second.transform(held_out).orderBy("id").collect()] == [
+ 1.0, 1.0, 0.0, 0.0
+ ]
+ estimator.setThreshold(1e6)
+ assert [r.pred for r in
estimator.fit(training).transform(held_out).orderBy("id").collect()] == [0.0] *
4
+ with pytest.raises(Exception):
+ estimator.fit(session.range(3))
+ with pytest.raises(Exception):
+ estimator.fit(training.withColumn("target", F.lit(2.0)))
+ with pytest.raises(Exception):
+ estimator.setRegParam(-1.0).fit(training)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]