Hyukjin Kwon created SPARK-61111:
------------------------------------
Summary: Support the ML API of pyspark.ml, of estimators, models,
pipelines and evaluators on the server, in the Python client as PySpark does
Key: SPARK-61111
URL: https://issues.apache.org/jira/browse/SPARK-61111
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The PySpark client package ships {{pyspark.ml}}, the ML API of estimators,
models, pipelines, evaluators and tuning ({{connect_packages}} of
{{python/packaging/client/setup.py}}). In PySpark Connect, these run on the
server through the Connect ML protocol. For example, {{from
pyspark.ml.classification import LogisticRegression;
LogisticRegression().fit(df)}} fits the model on the server, keeps it in the
server's ML cache and reads its attributes and summary from it.
{{pyspark.ml.util}} reaches the server through the Connect client:
* {{session.client.execute_command}} of {{MlCommand}}s (fit, fetch, delete,
write, read)
* {{_delete_ml_cache}}
* DataFrames of the plans of {{pyspark.ml.connect.proto}}
({{TransformerRelation}}, {{AttributeRelation}})
* the plans of DataFrames ({{df._plan.plan(client)}})
* {{pyspark.sql.connect.proto}}
The Python client ships only {{pyspark.ml.connect}}, {{linalg}}, {{param}},
{{torch}} and {{deepspeed}}. {{from pyspark.ml import Pipeline}},
{{pyspark.ml.classification}}, {{feature}}, {{regression}}, {{clustering}},
{{evaluation}}, {{tuning}} and the other modules of {{pyspark.ml}} raise
{{ImportError}}.
The fix is to vendor these modules, and the ML internals of
{{pyspark.ml.connect}} that they use, from Apache Spark v4.3.0. The client then
needs to give them what they use of the Connect client, over its Rust transport:
* the generated {{pyspark.sql.connect.proto}} modules
* a {{client}} of the commands and the ML cache
* DataFrames of official plans, and the plans of DataFrames
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
