[
https://issues.apache.org/jira/browse/SPARK-61631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61631:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Make the native evaluators follow pyspark.ml.evaluation, not the deprecated
> pyspark.ml.connect, in the Python client
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61631
> URL: https://issues.apache.org/jira/browse/SPARK-61631
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The native RegressionEvaluator, BinaryClassificationEvaluator and
> MulticlassClassificationEvaluator of the Python client follow the deprecated
> pyspark.ml.connect evaluators (deprecated since Spark 4.0).
> BinaryClassificationEvaluator takes probabilityCol instead of
> rawPredictionCol. MulticlassClassificationEvaluator defaults to accuracy
> instead of f1 and takes positional arguments. evaluate casts the prediction
> and the probability, and scores element 1 of an array of probabilities. The
> evaluators lack the params weightCol, throughOrigin, numBins, metricLabel,
> beta and eps, and setParams. They should follow the standard
> pyspark.ml.evaluation API over Spark Connect: its keyword-only signatures,
> params, defaults, setters, getters, setParams and isLargerBetter, with the
> dataset sent as pyspark.ml sends it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]