[ 
https://issues.apache.org/jira/browse/SPARK-61630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61630:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make the native LogisticRegression and evaluators follow pyspark.ml, not the 
> deprecated pyspark.ml.connect, and stop converting array features implicitly, 
> in the Python client
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61630
>                 URL: https://issues.apache.org/jira/browse/SPARK-61630
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> The native ML classes of the Python client took the deprecated 
> pyspark.ml.connect API (deprecated since Spark 4.0, PyTorch and pandas based) 
> as their compatibility target in several places. The target is the standard 
> pyspark.ml API over Spark Connect. (1) The native LogisticRegression has the 
> six params of the deprecated PyTorch LogisticRegression, outputs a bigint 
> prediction and array<double> probabilities without a raw prediction, checks 
> the labels on the client as the deprecated _fit does, and raises the 
> deprecated Transformer's ValueError for an existing output column. It should 
> have the params of pyspark.ml.classification.LogisticRegression, with 
> setParams, and the JVM model's outputs and errors. (2) The native 
> RegressionEvaluator, BinaryClassificationEvaluator and 
> MulticlassClassificationEvaluator have the params and defaults of the 
> deprecated evaluators (probabilityCol instead of rawPredictionCol, an 
> accuracy default instead of f1, positional arguments, the scoring of array 
> probabilities). They should have those of pyspark.ml.evaluation, with 
> setParams. (3) The native estimators convert an array features column to a 
> vector implicitly, as the deprecated classes take arrays. pyspark.ml sends 
> the column as it is, and the JVM rejects a column that is not a vector.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to