[
https://issues.apache.org/jira/browse/SPARK-61386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61386:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Make the native ML classes consistent and match PySpark's params,
> constructors, uids and errors
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-61386
> URL: https://issues.apache.org/jira/browse/SPARK-61386
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Low-priority findings of the s16 review of the native ML classes
> (spark_connect::ml and pyspark.ml.connect of the drop-in), each checked
> against official PySpark 4.3.0 (pyspark.ml.connect and the JVM-backed
> pyspark.ml classes over Spark Connect):
> - Rust API consistency: JVM Int params were i64 on LogisticRegression and
> LinearSVC but i32 on LinearRegression and KMeans; KMeans, LinearSVC and
> LinearRegression had no getters where LogisticRegression has them;
> LogisticRegressionModel, StringIndexerModel and MaxAbsScalerModel exposed pub
> fields where the other models keep them private; the public Params builder
> could send only Long ints and no arrays; CrossValidator::set_seed sent a Long
> where KMeans narrows the seed as PySpark sends a Python int.
> - Python params: values were not converted with PySpark's TypeConverters
> (KMeans(k=3.0) and seed=1.0 raised TypeError, an int past a long raised
> OverflowError, setWeightCol(None) raised), constructors sent every default
> param where PySpark sends only the explicit ones, and uids were bare UUIDs
> where PySpark's are <Class>_<12 hex>.
> - The older natives (evaluators, LogisticRegression, scalers, StringIndexer,
> VectorAssembler, Pipeline, CrossValidator) took positional arguments, where
> PySpark's @keyword_only raises "Method __init__ forces keyword arguments."
> - A setter called while another thread fitted the same estimator raised
> "RuntimeError: Already borrowed".
> - The transform of a model on a DataFrame of another session raised
> PySparkValueError, where PySpark raises CONNECT_ML.CACHE_INVALID from the
> server.
> - release_ml_objects sent its Delete on a stopped session, where
> release_cached_remote_relation skips stopped sessions.
> - Docs: the removal of BinaryClassificationEvaluator set_score_col/score_col
> and Python scoreCol= (shipped in v4.2.0) was not in the migration notes, and
> the docs said to set a seed for reproducible fits, though without one both
> clients use the JVM's default seed.
> The fixes apply to every native class, RandomForestClassifier included.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]