[ 
https://issues.apache.org/jira/browse/SPARK-61865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61865:
-----------------------------------
    Labels: pull-request-available  (was: )

> Send and convert the ML params as PySpark does: lists of ints of the type of 
> their first element, ints beyond their type, and the copy of the validator 
> models
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61865
>                 URL: https://issues.apache.org/jira/browse/SPARK-61865
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Follow-ups of SPARK-61766 and the ML reviews for the ML params of the drop-in 
> and the Rust API, against PySpark 4.3.0:
> - A list of ints of an ML param (MultilayerPerceptronClassifier.layers, 
> VectorSlicer.indices, QuantileDiscretizer.numBucketsArray) is sent as an 
> Integer array when every element fits 32 bits, else as a Long array, so a 
> list such as [2, 2**32+3, 2] fits a model of layers [2, 3, 2]. PySpark types 
> the list by its first element, and its LiteralExpression asserts each 
> element's range, so it raises a bare AssertionError at the fit, transform or 
> save. Of several unsendable params, PySpark raises the error of the first of 
> its _paramMap.
> - The Rust API's PredictionModel::num_features truncates a long to an i32, 
> and the grid ints beyond a long saturate where PySpark raises 
> VALUE_NOT_BETWEEN.
> - The apply_param_map of CrossValidatorModel and TrainValidationSplitModel 
> applies the map to the best model only, where PySpark's copy(extra) asserts 
> subModels, copies them and sets the model's own params.
> - RandomForestClassificationModel::summary() of no summary gives another 
> message than PySpark's "No training summary available for this 
> RandomForestClassificationModel".
> - Docs: the deprecations of StringIndexerModel.labels and ChiSqSelector, and 
> the arithmetic of the LR thresholds.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to