Hyukjin Kwon created SPARK-61545:
------------------------------------
Summary: Let the native ML estimators and scalers take the array
columns of PySpark's pyspark.ml.connect classes, and scale as PySpark's scalers
do
Key: SPARK-61545
URL: https://issues.apache.org/jira/browse/SPARK-61545
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's pyspark.ml.connect classes compute with numpy and torch on array
columns. MaxAbsScaler and StandardScaler take an array<double> (or another
array of numbers) input column and output an array<double>, and StandardScaler
scales to (x - mean) / std. LogisticRegression takes array features. The native
classes of the Python client run Spark's JVM estimators, which require ML
vectors, so PySpark's own docstring examples failed with "Column features must
be of type ... VectorUDT ... but was actually ... array<double>".
The native estimators and models now take the array columns that PySpark's take:
- A fit, and the transform of its model, send an array of numbers of the
scaler's inputCol, or of another estimator's featuresCol, as the vector that
the JVM reads (array_to_vector; decimal arrays cast to array<double> first).
The transform keeps the array column as it was in its output.
- The models of MaxAbsScaler and StandardScaler output an array<double>
(vector_to_array), as PySpark's do.
- StandardScaler sends the JVM's withMean, so that it scales to (x - mean) /
std as PySpark's does.
docs/compatibility.md describes the remaining differences: the JVM's doubles
for float arrays, and the vector outputs of LogisticRegression.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]