GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/20633
[SPARK-23455][ML] Default Params in ML should be saved separately in
metadata
## What changes were proposed in this pull request?
We save ML's user-supplied params and default params as one entity in
metadata. During loading the saved models, we set all the loaded params into
created ML model instances as user-supplied params.
It causes some problems, e.g., if we strictly disallow some params to be
set at the same time, a default param can fail the param check because it is
treated as user-supplied param after loading.
The loaded default params should not be set as user-supplied params. We
should save ML default params separately in metadata.
For backward compatibility, when loading metadata, if it is a metadata file
from previous Spark, we shouldn't raise error if we can't find the default
param field.
## How was this patch tested?
Pass existing tests and added tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 save-ml-default-params
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20633.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20633
----
commit 69648d67546b292037d26ef3a282bf26afd4863e
Author: Liang-Chi Hsieh <viirya@...>
Date: 2018-02-17T02:34:11Z
Save default params separately in JSON.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
