GitHub user yinxusen opened a pull request:
https://github.com/apache/spark/pull/11119
[SPARK-10780][ML][WIP] Add initial model to kmeans
https://issues.apache.org/jira/browse/SPARK-10780
I mark it as WIP because there are several issues need to discuss:
1. Codegen for shared params doesn't support general type. But IMHO there
is no need to rewrite the codegen to support general type params, for now. So I
add an `extras` in `SharedParamsCodeGen` to support this kind of exception. We
may modify the codegen later if there are more general type shared parameters.
2. Since we add `Model` in parameters of every `Estimator`, the save/load
of estimators should be modified to support model persistence, refer to
https://github.com/apache/spark/pull/9971. I ignore it in this version of the
code because the save/load works well only if we don't set `initialModel`. I'll
add save/load for the new parameter after the discussion.
3. I don't add model check (e.g. number of centers equals to k, the
dimension of centers equals to data points') here because the inner
`MLlibKMeans` checks it for us. For those estimators whose MLlib companions
don't provide warm-starts, we need to check the initial model in ML package.
4. I add three different setters for `KMeans`. The first one uses for
collaboration with model save/load, the second one is for accepting generalized
model, and the third one is more user-friendly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yinxusen/spark SPARK-10780
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11119.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11119
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
