[
https://issues.apache.org/jira/browse/SPARK-61531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61531:
-----------------------------------
Labels: pull-request-available (was: )
> Let the native evaluators take the numeric and array columns and the pandas
> DataFrames that PySpark's evaluators take
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61531
> URL: https://issues.apache.org/jira/browse/SPARK-61531
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark's pyspark.ml.connect evaluators compute with torch on the client.
> They read the columns as tensors: int labels and predictions, float32 or
> array probabilities (column 1 of a 2-D array is the positive class), and they
> take a pandas DataFrame too. The native evaluators of this client send the
> DataFrame unchanged to Spark's JVM evaluators, which require a double
> prediction, and a double or vector raw prediction. So PySpark's own docstring
> and test inputs failed:
> - MulticlassClassificationEvaluator on int columns: "Column prediction must
> be of type ... DoubleType but was actually ... LongType".
> - BinaryClassificationEvaluator on an array<double> or float probability:
> "must be of type equal to one of the following types: [double, struct<...>]
> but was actually of type array<double>".
> - A pandas DataFrame raised "argument 'df': 'DataFrame' object is not an
> instance of 'DataFrame'".
> The evaluators now:
> - take `dataset`, PySpark's name of the argument, as a DataFrame or a pandas
> DataFrame, which is sent through createDataFrame of the active session;
> - cast a prediction or a probability of another numeric type to a double;
> - send an array probability as a vector by array_to_vector, which the JVM
> evaluator reads at its element 1.
> docs/compatibility.md now says:
> - `setProbabilityCol` is a setter of this client, as PySpark's evaluators
> have none;
> - a model without a probability column (the native LinearSVC) is scored with
> probabilityCol="rawPrediction";
> - the native scalers take vector features, as the other native estimators do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]