+1 on Tommaso's suggestion. On Thu, Nov 15, 2012 at 8:25 AM, Tommaso Teofili <[email protected]>wrote:
> Hi Christian, > > it's nice to hear back from you on the list :) > The backprop algorithm is something that I'd be very happy to see > implemented here, and also I've spent some time myself on it some months > ago but didn't manage to finalize the implementation so far. > > Your approach sounds reasonable, I don't have read the paper pointed by > Edward (thanks!) yet but it may help us evaluate how to split things. > > Regarding deep learning that's surely something interesting we should keep > an eye on, maybe we can start from Christian's proposal, implement that and > maybe move to the DL if/when we have something ready. > > Thanks and have a nice day, > Tommaso > > > p.s.: > Regarding the matrix library my opinion is that, for starting, we should > try to use something that just works (I don't know Colt so I can't say) in > order to go straight to the algorithm itself but for the mid / long term > I'd also prefer to use an own matrix multiplication / inverse / etc. just > because that would be useful also for other tasks. > > > 2012/11/15 [email protected] <[email protected]> > > > Dear All, > > what do you think about to scale out the learning of Multi Layer > > Perceptrons > > (MLP) with BSPs? > > I heard the talk of Tommaso at the apacheConAt first glance the pramming > > model > > BSP seems to fit better the MapReduce for this purpose. > > The basic idea is to distribute the backprop algorithm is the following: > > > > Distribution of learning can be done by (batch learning): > > 1 Partioning of the data in x chunks > > 2 On each working node: Learning the weight changes (as matrices) in each > > chunk > > 3 Combining the matrixes (weight changes) and simultaneous update of the > > weights > > in each node - back to 2 > > > > Maybe this procedure can be done with random parts of the chunks > > (distributed > > quasi online learning). > > > > I wrote the (basic) backprob algorithm of a multi layer preceptron (see > > mahout > > patch https://issues.apache.org/jira/browse/MAHOUT-976). It uses the > > Mahout > > Matrix Library, which is under the hood the Colt Matrix Library from > Cern. > > > > Probably using the Cern Matrix Library would also suitable for Hama. Then > > it > > could be easy to port the MLP to Hama. > > > > What do you think about it? > > > > Thanks for you response. > > > > Cheers > > Christian >
