Github user NathanHowell commented on the pull request:
https://github.com/apache/spark/pull/8246#issuecomment-139323929
@jkbradley yes, this fixed a real problem. the problem stemmed from
attempting to use categorical features with high arity (250+), which has a side
effect of increasing the number of bins used for continuous features and
massively increasing the sample size. the sampling and bin calculation would
take multiple hours. after the patch it completes in ~2 minutes.
training itself would run okay but also wasn't terribly fast. I've had to
switch to a different RF implementation (https://github.com/stripe/brushfire)
in order to have training complete in a reasonable amount of time. the mllib
implementation would train a single tree in 3-4 days on 2000 executors, whereas
brushfire does the same in about 6 hours with per-split bin calculation and
about 45 minutes with log bucketing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
