zhengruifeng opened a new pull request, #217: URL: https://github.com/apache/spark-connect-rust/pull/217
### What changes were proposed in this pull request? Make the native Rust `LogisticRegression` train with Spark Connect's `MlCommand.Fit`, passing the actual dataset and estimator parameters. Preserve the returned cached model reference and use it in `MlRelation.Transform` for predictions, rather than reconstructing an unfitted operator from its UID. Validate fitted-model responses, propagate server training errors, reject transforms from a different session, and document the native estimator's JVM backend and vector feature requirement. ### Why are the changes needed? `fit` previously returned a model wrapper without submitting the training data to Spark. Its transform plan identified an operator instead of the fitted model object, so it could not produce predictions from learned coefficients. [SPARK-61010](https://issues.apache.org/jira/browse/SPARK-61010) ### Does this PR introduce _any_ user-facing change? Yes. Native Rust LogisticRegression and its Python binding now fit binary and multiclass classifiers and predict on new rows using the trained model. Invalid training datasets fail during `fit`. Models belong to the session that fitted them. Cached models remain on the server until session closure so cloned models and lazy transformed DataFrames remain valid after the original model is dropped. The new private fitted state means Rust callers must obtain `LogisticRegressionModel` through `fit`, rather than constructing it with a struct literal. This wrapper targets `org.apache.spark.ml.classification.LogisticRegression`; it accepts Spark vectors. The deprecated Torch-based `pyspark.ml.connect.classification.LogisticRegression` is a different implementation. ### How was this patch tested? - Added offline tests for cached-reference serialization, malformed fit responses, model metadata and session isolation. - Added three strict live Rust tests covering binary and multiclass predictions, custom/default column names, separate fits on reversed labels, cloning, delayed execution, cross-session rejection and invalid training data. All passed against Spark 4.3.0. - Added a Python binding integration test for training-dependent predictions and invalid training data. Passed with both the native binding and official JVM PySpark as reference. - Rust test run: 669 passed, 26 ignored; server-gated tests also skip their bodies when no server is configured. Library tests, all integration-test compilation, `wasm-udf` integration-test compilation and formatting passed. - Python offline suite: 887 passed, 8 skipped. Fresh method-name comparison across 12 core wrapper classes found no gaps under the existing transport exclusions. - Library Clippy passed with warnings. Workspace/all-target Clippy still fails on existing test and Python-binding errors; comparison with the exact base commit found no new diagnostics. - Official SQL Connect suite through the Rust transport: all 87 files passed across batches (2,638 tests passed, 589 skipped), using the existing CI exclusions and known-failure manifest. The functions file passed after matching upstream's `local[4]` setup; the initial partition-sensitive failure also reproduced with official PySpark. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-6) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
