Hyukjin Kwon created SPARK-61923:
------------------------------------
Summary: Send and save the ML params as PySpark's
serialize_ml_params does: params of no converter, params of subclasses, and the
class of subclasses of meta-algorithms
Key: SPARK-61923
URL: https://issues.apache.org/jira/browse/SPARK-61923
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
The saves of the native ML classes of the drop-in differ from PySpark 4.3.0's:
- RegexTokenizer.gaps, a param of no converter, is never written by save, so a
saved tokenizer (also inside a Pipeline or PipelineModel) loads with gaps=True
and gives other results; values that PySpark's save refuses (gaps=(1,), 2**63,
1, "yes", [True]) save silently.
- The params of a Python subclass that are set are not sent, at a fit,
transform, evaluate or save, where PySpark's serialize_ml_params sends every
item of _paramMap, which the server refuses.
- Subclasses of OneVsRest, Pipeline, CrossValidator and TrainValidationSplit
save a constant class name, where PySpark writes the instance's own module and
class name.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]