Github user manishamde commented on a diff in the pull request:
https://github.com/apache/spark/pull/1950#discussion_r16275227
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala ---
@@ -550,25 +579,35 @@ object DecisionTree extends Serializable with Logging
{
// Apply each filter and check sample validity. Return false when
invalid condition found.
for (filter <- parentFilters) {
- val features = labeledPoint.features
val featureIndex = filter.split.feature
- val threshold = filter.split.threshold
val comparison = filter.comparison
- val categories = filter.split.categories
val isFeatureContinuous = filter.split.featureType == Continuous
- val feature = features(featureIndex)
if (isFeatureContinuous) {
+ val binId = treePoint.features(featureIndex)
+ val bin = bins(featureIndex)(binId)
+ val featureValue = bin.highSplit.threshold
+ val threshold = filter.split.threshold
comparison match {
- case -1 => if (feature > threshold) return false
- case 1 => if (feature <= threshold) return false
+ case -1 => if (featureValue > threshold) return false
+ case 1 => if (featureValue <= threshold) return false
}
} else {
- val containsFeature = categories.contains(feature)
+ val numFeatureCategories =
strategy.categoricalFeaturesInfo(featureIndex)
+ val isSpaceSufficientForAllCategoricalSplits =
+ numBins > math.pow(2, numFeatureCategories.toInt - 1) - 1
+ val isUnorderedFeature =
--- End diff --
I guess we can bundle this calculation as well in the your future commit.
Let's create a map for feature id to features types which can be re-used by all
internal methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
