[ 
https://issues.apache.org/jira/browse/SPARK-61714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61714:
------------------------------------

    Assignee: Hyukjin Kwon

> Add PySpark's Pipeline, CrossValidator, TrainValidationSplit and their models 
> to the Rust API
> ---------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61714
>                 URL: https://issues.apache.org/jira/browse/SPARK-61714
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Part of completing the native ML classes and the Rust API to the full 
> pyspark.ml 4.3.0 surface, after the validators of the Python classes 
> (SPARK-61708).
> The Rust API gets PySpark's meta-algorithms, of any estimators, transformers, 
> models and evaluators, as PySpark 4.3.0 runs them over Spark Connect, on the 
> client:
> - stages of cloneable estimators, transformers and models, and param maps of 
> the uid of a stage and the name of its param, with PySpark's ParamGridBuilder;
> - Pipeline and PipelineModel, of Pipeline._fit;
> - CrossValidator, TrainValidationSplit and their models: PySpark's splits, 
> cached, the fits of the param maps in parallel, numpy's metrics and best 
> index, and the fit of the best map;
> - the persistence of PySpark's Python format, which PySpark loads of the Rust 
> API and the Rust API of PySpark.
> They replace the Pipeline of stage names and the CrossValidator of numeric 
> params of the Rust API, whose fits raised NOT_IMPLEMENTED.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to