Hyukjin Kwon created SPARK-61399:
------------------------------------
Summary: Raise in fit of the native Pipeline and CrossValidator
instead of returning a model that the server cannot use
Key: SPARK-61399
URL: https://issues.apache.org/jira/browse/SPARK-61399
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Found in a post-merge review of SPARK-61324 and SPARK-61334. The fit of the
native pyspark.ml.connect Pipeline and CrossValidator (Pipeline::fit_impl and
CrossValidator::fit_impl in the Rust API) ignores the DataFrame and returns a
local model without a server call, so fit silently succeeds and the model's
transform then fails on the server with java.lang.AssertionError. The native
Pipeline takes stage names, not stages, and the native CrossValidator has no
estimator, evaluator or param grid, so neither can be fitted.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]