[
https://issues.apache.org/jira/browse/MAHOUT-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909813#comment-13909813
]
Sean Owen commented on MAHOUT-1419:
-----------------------------------
Hmm. The data set is about 300MB compressed, so I won't attach it here. I'll
leave it up here for a few days:
https://drive.google.com/file/d/0B_hfrkaWlLi4VkNZd2k0WGJsLVk/edit?usp=sharing
Now that I'm getting into Scala, I can tell you that this snippet will generate
a suitable input file of as many lines as you like:
{code}
val r = new scala.util.Random()
val pw = new java.io.PrintWriter("random.csv")
(1 to 10000000).foreach(e => pw.println(r.nextDouble + "," + r.nextDouble + ","
+ r.nextDouble + "," + r.nextDouble + "," + (if (r.nextBoolean) 1 else 0)))
{code}
> Random decision forest is excessively slow on numeric features
> --------------------------------------------------------------
>
> Key: MAHOUT-1419
> URL: https://issues.apache.org/jira/browse/MAHOUT-1419
> Project: Mahout
> Issue Type: Bug
> Components: Classification
> Affects Versions: 0.7, 0.8, 0.9
> Reporter: Sean Owen
> Attachments: MAHOUT-1419.patch
>
>
> Follow-up to MAHOUT-1417. There's a customer running this and observing it
> take an unreasonably long time on about 2GB of data -- like, >24 hours when
> other RDF M/R implementations take 9 minutes. The difference is big enough to
> probably be considered a defect. MAHOUT-1417 got that down to about 5 hours.
> I am trying to further improve it.
> One key issue seems to be how splits are evaluated over numeric features. A
> split is tried for every distinct numeric value of the feature in the whole
> data set. Since these are floating point values, they could (and in the
> customer's case are) all distinct. 200K rows means 200K splits to evaluate
> every time a node is built on the feature.
> A better approach is to sample percentiles out of the feature and evaluate
> only those as splits. Really doing that efficiently would require a lot of
> rewrite. However, there are some modest changes possible which get some of
> the benefit, and appear to make it run about 3x faster. That is --on a data
> set that exhibits this problem -- meaning one using numeric features which
> are generally distinct. Which is not exotic.
> There are comparable but different problems with handling of categorical
> features, but that's for a different patch.
> I have a patch, but it changes behavior to some extent since it is evaluating
> only a sample of splits instead of every single possible one. In particular
> it makes the output of "OptIgSplit" no longer match the "DefaultIgSplit".
> Although I think the point is that "optimized" may mean giving different
> choices of split here, which could yield differing trees. So that test
> probably has to go.
> (Along the way I found a number of micro-optimizations in this part of the
> code that added up to maybe a 3% speedup. And fixed an NPE too.)
> I will propose a patch shortly with all of this for thoughts.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
