GitHub user jkbradley opened a pull request:
https://github.com/apache/spark/pull/7294
[SPARK-7131] [ml] Copy Decision Tree, Random Forest impl to spark.ml
This PR copies the RandomForest implementation from spark.mllib to
spark.ml. Note that this includes the DecisionTree implementation, but not the
GradientBoostedTrees one (which will come later).
I essentially copied a minimal amount of code to spark.ml, removed the use
of bins (and only used splits), and modified code only as much as necessary to
get it to compile. The spark.ml implementation still uses some spark.mllib
classes (privately), which can be moved in future PRs.
This refactoring will be helpful in extending the node representation to
include more information, such as class probabilities.
Specifically:
* Copied code from spark.mllib to spark.ml:
* mllib.tree.DecisionTree, mllib.tree.RandomForest copied to
ml.tree.impl.RandomForest (main implementation)
* NodeIdCache (needed to use splits instead of bins)
* TreePoint (use splits instead of bins)
* Added ml.tree.LearningNode used in RandomForest training (needed vars)
* Removed bins from implementation, and only used splits
* Small fix in JavaDecisionTreeRegressorSuite
CC: @mengxr @manishamde @codedeft
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkbradley/spark dt-move-impl
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7294.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7294
----
commit 19143fb389ba550d86c9444deae2203fa5ea818f
Author: Joseph K. Bradley <[email protected]>
Date: 2015-06-19T17:07:41Z
More progress, but not done yet. Rebased with master after 1.4 release.
commit cc0182368218831ff5f342d815875bde4cb8ae9a
Author: Joseph K. Bradley <[email protected]>
Date: 2015-07-01T17:49:07Z
still editing RF to get it to work
commit d5224a93d101fbc48a3b3d64b8978aed093a7255
Author: Joseph K. Bradley <[email protected]>
Date: 2015-07-06T19:06:37Z
modified tree and forest to use moved impl
commit 0df3759fcfea73115417935aff2087f7b44d0bcf
Author: Joseph K. Bradley <[email protected]>
Date: 2015-07-07T04:53:12Z
Need to remove use of Bucketizer
commit bd5e0639d3e9fa0cad68bc0b816f62a269b8a046
Author: Joseph K. Bradley <[email protected]>
Date: 2015-07-07T21:18:42Z
fixed bucketizing issue
commit 836e7d4a2eff628034fdd71cc06a2a3f8bb78253
Author: Joseph K. Bradley <[email protected]>
Date: 2015-07-08T04:32:35Z
Fixed test suite failures
commit 9a4d72140d10b85d2669ab8e5cc360c388e46b52
Author: Joseph K. Bradley <[email protected]>
Date: 2015-07-08T18:39:23Z
cleanups. removed InfoGainStats from ml, using old one for now.
commit 4e6d2a4249f6c65684ef4a5f190f7cafb5d278f3
Author: Joseph K. Bradley <[email protected]>
Date: 2015-07-08T18:48:03Z
removed unnecessary use of copyValues, setParent for trees
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
