Hyukjin Kwon created SPARK-61714:
------------------------------------
Summary: Add PySpark's Pipeline, CrossValidator,
TrainValidationSplit and their models to the Rust API
Key: SPARK-61714
URL: https://issues.apache.org/jira/browse/SPARK-61714
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Part of completing the native ML classes and the Rust API to the full
pyspark.ml 4.3.0 surface, after the validators of the Python classes
(SPARK-61708).
The Rust API gets PySpark's meta-algorithms, of any estimators, transformers,
models and evaluators, as PySpark 4.3.0 runs them over Spark Connect, on the
client:
- stages of cloneable estimators, transformers and models, and param maps of
the uid of a stage and the name of its param, with PySpark's ParamGridBuilder;
- Pipeline and PipelineModel, of Pipeline._fit;
- CrossValidator, TrainValidationSplit and their models: PySpark's splits,
cached, the fits of the param maps in parallel, numpy's metrics and best index,
and the fit of the best map;
- the persistence of PySpark's Python format, which PySpark loads of the Rust
API and the Rust API of PySpark.
They replace the Pipeline of stage names and the CrossValidator of numeric
params of the Rust API, whose fits raised NOT_IMPLEMENTED.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]