Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/2332#discussion_r17321560
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala ---
@@ -764,7 +764,31 @@ object DecisionTree extends Serializable with Logging {
val gain = impurity - leftWeight * leftImpurity - rightWeight *
rightImpurity
- new InformationGainStats(gain, impurity, leftImpurity, rightImpurity,
predict, prob)
+ // if information gain doesn't satisfy minimum information gain,
+ // then this split is invalid, return invalid information gain stats.
+ if (gain < metadata.minInfoGain) {
+ return InformationGainStats.invalidInformationGainStats
+ }
+
+ new InformationGainStats(gain, impurity, leftImpurity, rightImpurity)
+ }
+
+ /**
+ * Calculate predict value for current node, given stats of any split.
+ * Note that this function is called only once for each node.
+ * @param leftImpurityCalculator left node aggregates for a split
+ * @param rightImpurityCalculator right node aggregates for a node
+ * @return predict value for current node
+ */
+ private def calculatePredict(
--- End diff --
See comment for Predict.scala: If prediction info is stored in
InformationGainStats, this calculation can remain in calculateGainForSplit().
It is really not very expensive to compute the prediction, so I think we gain a
lot in simplicity and lose very little in computational cost by keeping
everything in InformationGainStats.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
