On Sun, Dec 9, 2012 at 2:12 AM, Marty Kube < [email protected]> wrote:
> ... > I've been looking at the mmap suggestion some. When you said: > > 1) use shared memory via mmap to store the forest. This allows multiple > mapper threads to access the same forest. The current Mahout in-memory > structure for this is not suitable for shared memory, however. > > Can you be a little more specific about why the current in-memory > structure is not suitable for shared memory? > Because it uses Java pointers instead of offsets. The mmap'ed structure could be mapped into memory at any address and thus must be position independent. > > I'm finding that Java does not support shared memory so one would need to > run the forest cache through JNI in order to use mmap and shared memory. > Not quite true. See http://docs.oracle.com/javase/1.4.2/docs/api/java/nio/channels/FileChannel.html#map(java.nio.channels.FileChannel.MapMode,%20long,%20long) > The other track I came up with is to use a distributed cache like memcache > or hazelcast. To me those solutions seem target to cross host caches so I > worry about performance. What I really want is a within host shared cache > across JVMs. You should definitely worry about performance on these. There are two good approaches. If your shared objects are pretty small, then distributed cache can get the objects into the local file system for mapping. If they are larger, then you can use MapR's NFS capabilities to present anything in the cluster as a normal file which can then be mapped. > On 12/08/2012 03:43 AM, Ted Dunning wrote: > >> There are several approaches that might help: >> >> 1) use shared memory via mmap to store the forest. This allows multiple >> mapper threads to access the same forest. The current Mahout in-memory >> structure for this is not suitable for shared memory, however. >> >> 2) split the forests across many mappers (as you suggest). You would have >> to tag your outputs cleverly so that they wind up at the right reducer. >> Tags would include input data segment and forest segment. Mahout >> doesn't >> support this, but it should be easily doable. >> >> 3) thin the forests. There isn't a lot of literature on this, but I am >> pretty sure that I have seen some articles where less informative trees in >> the random forest were removed. Another option with a similar effect is >> to >> use the random forest as an oracle so that you can generate a huge amount >> of training data for some other technique that may be prone to >> over-fitting. This alternative model can be trained to fit the output of >> the random forest very precisely. Over-fitting isn't an issue because you >> can generate as much training data as you like. This isn't supported in >> Mahout. >> >> >> On Sat, Dec 8, 2012 at 2:03 AM, Marty Kube < >> martykube@**beavercreekconsulting.com<[email protected]>> >> wrote: >> >> So here is a better description of the decision forest classification >>> implementation I'm working on. This is for large scale classification >>> after training. >>> >>> We have many attributes being classified, each attribute has it's own >>> forest. The forest are big enough when loaded into RAM that you get only >>> one JVM per host. But you really want one thread per processor on the >>> host, so we ended up threading the mappers. We have a lot of feature >>> vectors so we send the features to the mappers. >>> >>> This seems a bit awkward. I've been thinking about spreading the trees >>> out across mappers to reduce the RAM per JVM with the goal of getting >>> closer to one JVM per core. But then we'll need to do a more complex >>> join >>> between forests and feature vectors. Right now we are essentially doing >>> a >>> replicated join with the forest being the replicated set. >>> >>> Has anyone tried this - Is there support for this in Mahout? >>> >>> >>> On 12/06/2012 09:32 PM, Marty Kube wrote: >>> >>> Yes I'm on a project in which we classify a large data set. We do use >>>> mapreduce to do the classification as the data set is much larger than >>>> the >>>> working memory. We have a non-mahout implementation... >>>> >>>> So we put the decision forest in memory via a distributed cache and >>>> partition the data set and run it past the models. The models are >>>> getting >>>> pretty big and keeping them in memory is a challenge. I guess I was >>>> looking >>>> for an implementation that doesn't require keeping the decision forest >>>> in >>>> memory. I'll have a look at the TestForest implementation. >>>> >>>> >>>> On 12/06/2012 12:06 AM, deneche abdelhakim wrote: >>>> >>>> You mean you want to classify a large dataset ? >>>>> The partial implementation is useful when the training dataset is too >>>>> large >>>>> to fit in memory. If it's does fit then you better train the forest >>>>> using >>>>> the in-memory implementation. >>>>> If you want to classify a large amount of rows then you can add the >>>>> parameter -mr to TestForest to classify the data using mapreduce. An >>>>> example of this can be found in the wiki: >>>>> >>>>> https://cwiki.apache.org/****MAHOUT/partial-implementation.****html<https://cwiki.apache.org/**MAHOUT/partial-implementation.**html> >>>>> <https://cwiki.apache.**org/MAHOUT/partial-**implementation.html<https://cwiki.apache.org/MAHOUT/partial-implementation.html> >>>>> > >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Dec 6, 2012 at 2:45 AM, Marty Kube < >>>>> martykube@**beavercreekconsult**ing.com<http://beavercreekconsulting.com> >>>>> <martykube@**beavercreekconsulting.com<[email protected]> >>>>> >> >>>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>>> I'm working improving classification throughput for a decision forest. >>>>>> I >>>>>> was wondering about the use case for Partial Implementation. >>>>>> >>>>>> The quick start guide suggests that Partial Implementation is designed >>>>>> for >>>>>> building forest on large datasets. >>>>>> >>>>>> My problem is classification after training. Is Partial Implementation >>>>>> helpful for this use case? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>> >
