[
https://issues.apache.org/jira/browse/SPARK-61631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61631.
----------------------------------
Resolution: Duplicate
Superseded: SPARK-61642 (https://github.com/apache/spark-connect-rust/pull/650)
moved the native evaluators to pyspark.ml.evaluation (signatures, params,
defaults, isLargerBetter, no client-side casts), and SPARK-61624
(https://github.com/apache/spark-connect-rust/pull/620) gave them setParams and
the params argument of evaluate. The pyspark.ml getters of the native classes
are a separate, post-4.3.0 item.
> Make the native evaluators follow pyspark.ml.evaluation, not the deprecated
> pyspark.ml.connect, in the Python client
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61631
> URL: https://issues.apache.org/jira/browse/SPARK-61631
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The native RegressionEvaluator, BinaryClassificationEvaluator and
> MulticlassClassificationEvaluator of the Python client follow the deprecated
> pyspark.ml.connect evaluators (deprecated since Spark 4.0).
> BinaryClassificationEvaluator takes probabilityCol instead of
> rawPredictionCol. MulticlassClassificationEvaluator defaults to accuracy
> instead of f1 and takes positional arguments. evaluate casts the prediction
> and the probability, and scores element 1 of an array of probabilities. The
> evaluators lack the params weightCol, throughOrigin, numBins, metricLabel,
> beta and eps, and setParams. They should follow the standard
> pyspark.ml.evaluation API over Spark Connect: its keyword-only signatures,
> params, defaults, setters, getters, setParams and isLargerBetter, with the
> dataset sent as pyspark.ml sends it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]