For each job do you have only 100k and 300k users/items? In that case I consider the svd job to be small enough to run on a single core/machine. You can use svdlibc to execute the job and then use hadoop to manage the whole process. if you have 100k small jobs then using C might be more advantageous.
In my opinion Mahout SVD is useful when single job is way too big to fit in memory of a single machine. On Fri, Jul 23, 2010 at 4:14 AM, Laszlo Dosa <[email protected]>wrote: > Hi, > > > > I tried to calculate SVD based recommendation by: > > 1) Calculate A'A > > 2) Calculate V_k by using Lanczos to get top k eigenvectors of A'A > > 3) Calculate preferences for each user by Vector usersPreference = > V_k.timesSquared(row of A); > > 4) Calculate 20 recommendation for each user by selecting the top 20 > preference from usersPreference > > > > The number of users and items is between 100 000 and 300 000. > > > > In my experience Step 3) scales linearly by the number of users. > > Each job runs fast, but I have ~100 000 jobs. This slows down the whole > calculation. > > > > Is the approach wrong that I use? > > Or can you suggest something to make this faster? > > > > Regards, > > Laszlo > > > > -- Akshay Uday Bhat. Graduate Student, Computer Science, Cornell University Website: http://www.akshaybhat.com
