Hyukjin Kwon created SPARK-61630:
------------------------------------

             Summary: Make the native LogisticRegression and evaluators follow 
pyspark.ml, not the deprecated pyspark.ml.connect, and stop converting array 
features implicitly, in the Python client
                 Key: SPARK-61630
                 URL: https://issues.apache.org/jira/browse/SPARK-61630
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


The native ML classes of the Python client took the deprecated 
pyspark.ml.connect API (deprecated since Spark 4.0, PyTorch and pandas based) 
as their compatibility target in several places. The target is the standard 
pyspark.ml API over Spark Connect. (1) The native LogisticRegression has the 
six params of the deprecated PyTorch LogisticRegression, outputs a bigint 
prediction and array<double> probabilities without a raw prediction, checks the 
labels on the client as the deprecated _fit does, and raises the deprecated 
Transformer's ValueError for an existing output column. It should have the 
params of pyspark.ml.classification.LogisticRegression, with setParams, and the 
JVM model's outputs and errors. (2) The native RegressionEvaluator, 
BinaryClassificationEvaluator and MulticlassClassificationEvaluator have the 
params and defaults of the deprecated evaluators (probabilityCol instead of 
rawPredictionCol, an accuracy default instead of f1, positional arguments, the 
scoring of array probabilities). They should have those of 
pyspark.ml.evaluation, with setParams. (3) The native estimators convert an 
array features column to a vector implicitly, as the deprecated classes take 
arrays. pyspark.ml sends the column as it is, and the JVM rejects a column that 
is not a vector.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to