[ 
https://issues.apache.org/jira/browse/SPARK-61631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61631:
---------------------------------
    Fix Version/s: connect-rust-4.3.0

> Make the native evaluators follow pyspark.ml.evaluation, not the deprecated 
> pyspark.ml.connect, in the Python client
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61631
>                 URL: https://issues.apache.org/jira/browse/SPARK-61631
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> The native RegressionEvaluator, BinaryClassificationEvaluator and 
> MulticlassClassificationEvaluator of the Python client follow the deprecated 
> pyspark.ml.connect evaluators (deprecated since Spark 4.0). 
> BinaryClassificationEvaluator takes probabilityCol instead of 
> rawPredictionCol. MulticlassClassificationEvaluator defaults to accuracy 
> instead of f1 and takes positional arguments. evaluate casts the prediction 
> and the probability, and scores element 1 of an array of probabilities. The 
> evaluators lack the params weightCol, throughOrigin, numBins, metricLabel, 
> beta and eps, and setParams. They should follow the standard 
> pyspark.ml.evaluation API over Spark Connect: its keyword-only signatures, 
> params, defaults, setters, getters, setParams and isLargerBetter, with the 
> dataset sent as pyspark.ml sends it.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to