Hyukjin Kwon created SPARK-61865:
------------------------------------
Summary: Send and convert the ML params as PySpark does: lists of
ints of the type of their first element, ints beyond their type, and the copy
of the validator models
Key: SPARK-61865
URL: https://issues.apache.org/jira/browse/SPARK-61865
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Follow-ups of SPARK-61766 and the ML reviews for the ML params of the drop-in
and the Rust API, against PySpark 4.3.0:
- A list of ints of an ML param (MultilayerPerceptronClassifier.layers,
VectorSlicer.indices, QuantileDiscretizer.numBucketsArray) is sent as an
Integer array when every element fits 32 bits, else as a Long array, so a list
such as [2, 2**32+3, 2] fits a model of layers [2, 3, 2]. PySpark types the
list by its first element, and its LiteralExpression asserts each element's
range, so it raises a bare AssertionError at the fit, transform or save. Of
several unsendable params, PySpark raises the error of the first of its
_paramMap.
- The Rust API's PredictionModel::num_features truncates a long to an i32, and
the grid ints beyond a long saturate where PySpark raises VALUE_NOT_BETWEEN.
- The apply_param_map of CrossValidatorModel and TrainValidationSplitModel
applies the map to the best model only, where PySpark's copy(extra) asserts
subModels, copies them and sets the model's own params.
- RandomForestClassificationModel::summary() of no summary gives another
message than PySpark's "No training summary available for this
RandomForestClassificationModel".
- Docs: the deprecations of StringIndexerModel.labels and ChiSqSelector, and
the arithmetic of the LR thresholds.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]