GitHub user hhbyyh opened a pull request:
https://github.com/apache/spark/pull/18313
[SPARK-21087] [ML] CrossValidator, TrainValidationSplit should preserve all
models after fitting: Scala
## What changes were proposed in this pull request?
Allow `CrossValidatorModel` and `TrainValidationSplitModel` preserve the
full list of fitted models.
add a new string param `modelPath`, If set, all the models fitted during
the training will be preserved under the specific directory path. By default
the models will not be saved.
Save the models during the training to avoid expensive memory consumption
for caching the models.
Sample for cross validation models:
file name pattern: paramMap-split#-metric
Sample for train validation split:
file name pattern: paramMap-metric
## How was this patch tested?
new unit tests and local test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hhbyyh/spark saveModels
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18313.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18313
----
commit f6dfc6624bee4dd3ab4048510b30517047fb979a
Author: Yuhao Yang <[email protected]>
Date: 2017-06-15T05:53:13Z
save all models
commit 4c51912e0166a7b105533687b04a991e0e35257f
Author: Yuhao Yang <[email protected]>
Date: 2017-06-15T06:06:34Z
precision rounding
commit 0fc43e19c29f67c847749fb7ea0cf21ac47eb69f
Author: Yuhao Yang <[email protected]>
Date: 2017-06-15T06:12:35Z
comment update
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
