Github user sethah commented on the pull request:
https://github.com/apache/spark/pull/9008#issuecomment-187920267
Another issue is that the information gain for candidate splits is not
computed correctly with fractional samples. This is because the information
gain calculation
[here](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala#L636)
uses the sample counts which are converted to `Long` type. This produces
incorrect results in general, and `NaN` values when the total count is less
than 1. The `count` function
[here](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Impurity.scala#L149)
should return a `Double` type instead. Can we add a test to ensure that the
trees are invariant under constant multiplication of the weights?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
