Hyukjin Kwon created SPARK-61238:
------------------------------------
Summary: Fitted pyspark.ml.connect scalers and indexers fail on
transform, and the ML cache queries are missing
Key: SPARK-61238
URL: https://issues.apache.org/jira/browse/SPARK-61238
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
pyspark.ml.connect StandardScaler, MaxAbsScaler and StringIndexer fit locally,
so their transform sends a transformer of type MODEL, which the server rejects:
`StandardScaler(inputCol="v", outputCol="o").fit(df).transform(df)` raises
java.lang.AssertionError in MLHandler._transformMLRelation. The fit now runs on
the server and the transform refers to the cached model, as PySpark does. The
client also lacks _get_ml_cache_info and _query_model_size, which the official
ml/tests/connect cache and model offloading tests call.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
