[
https://issues.apache.org/jira/browse/SPARK-23455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368116#comment-16368116
]
Liang-Chi Hsieh commented on SPARK-23455:
-----------------------------------------
Currently, {{DefaultParamsWriter}} saves the following metadata + params:
* - class
* - timestamp
* - sparkVersion
* - uid
* - paramMap
* - (optionally, extra metadata)
User-supplied params and default params are all saved in {{paramMap}} field in
JSON. We can have a {{defaultParamMap}} for saving default params.
For backward compatibility, when loading metadata, if it is a metadata file
prior to Spark 2.4, we shouldn't raise error if we can't find
{{defaultParamMap}} field in the file.
> Default Params in ML should be saved separately
> -----------------------------------------------
>
> Key: SPARK-23455
> URL: https://issues.apache.org/jira/browse/SPARK-23455
> Project: Spark
> Issue Type: Improvement
> Components: ML
> Affects Versions: 2.4.0
> Reporter: Liang-Chi Hsieh
> Priority: Major
>
> We save ML's user-supplied params and default params as one entity in JSON.
> During loading the saved models, we set all the loaded params into created ML
> model instances as user-supplied params.
> It causes some problems, e.g., if we strictly disallow some params to be set
> at the same time, a default param can fail the param check because it is
> treated as user-supplied param after loading.
> The loaded default params should not be set as user-supplied params. We
> should save ML default params separately in JSON.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
