Hyukjin Kwon created SPARK-61531:
------------------------------------

             Summary: Let the native evaluators take the numeric and array 
columns and the pandas DataFrames that PySpark's evaluators take
                 Key: SPARK-61531
                 URL: https://issues.apache.org/jira/browse/SPARK-61531
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


PySpark's pyspark.ml.connect evaluators compute with torch on the client. They 
read the columns as tensors: int labels and predictions, float32 or array 
probabilities (column 1 of a 2-D array is the positive class), and they take a 
pandas DataFrame too. The native evaluators of this client send the DataFrame 
unchanged to Spark's JVM evaluators, which require a double prediction, and a 
double or vector raw prediction. So PySpark's own docstring and test inputs 
failed:
- MulticlassClassificationEvaluator on int columns: "Column prediction must be 
of type ... DoubleType but was actually ... LongType".
- BinaryClassificationEvaluator on an array<double> or float probability: "must 
be of type equal to one of the following types: [double, struct<...>] but was 
actually of type array<double>".
- A pandas DataFrame raised "argument 'df': 'DataFrame' object is not an 
instance of 'DataFrame'".

The evaluators now:
- take `dataset`, PySpark's name of the argument, as a DataFrame or a pandas 
DataFrame, which is sent through createDataFrame of the active session;
- cast a prediction or a probability of another numeric type to a double;
- send an array probability as a vector by array_to_vector, which the JVM 
evaluator reads at its element 1.

docs/compatibility.md now says:
- `setProbabilityCol` is a setter of this client, as PySpark's evaluators have 
none;
- a model without a probability column (the native LinearSVC) is scored with 
probabilityCol="rawPrediction";
- the native scalers take vector features, as the other native estimators do.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to