Hi Akshay, I started to experiment with a subset of the data. The number of users and items can easily grow up to 10 000 000.
I am looking for a solution where having more users and items can be compensate by more hardware. Thus the time of calculation is constant. Is it possible to do this with svdlibc? If yes can you give me some advice how? Regards, Laszlo -----Original Message----- From: Akshay Bhat [mailto:[email protected]] Sent: vrijdag 23 juli 2010 10:23 To: [email protected] Subject: Re: distributed svd For each job do you have only 100k and 300k users/items? In that case I consider the svd job to be small enough to run on a single core/machine. You can use svdlibc to execute the job and then use hadoop to manage the whole process. if you have 100k small jobs then using C might be more advantageous. In my opinion Mahout SVD is useful when single job is way too big to fit in memory of a single machine. On Fri, Jul 23, 2010 at 4:14 AM, Laszlo Dosa <[email protected]>wrote: > Hi, > > > > I tried to calculate SVD based recommendation by: > > 1) Calculate A'A > > 2) Calculate V_k by using Lanczos to get top k eigenvectors of A'A > > 3) Calculate preferences for each user by Vector usersPreference = > V_k.timesSquared(row of A); > > 4) Calculate 20 recommendation for each user by selecting the top 20 > preference from usersPreference > > > > The number of users and items is between 100 000 and 300 000. > > > > In my experience Step 3) scales linearly by the number of users. > > Each job runs fast, but I have ~100 000 jobs. This slows down the whole > calculation. > > > > Is the approach wrong that I use? > > Or can you suggest something to make this faster? > > > > Regards, > > Laszlo > > > > -- Akshay Uday Bhat. Graduate Student, Computer Science, Cornell University Website: http://www.akshaybhat.com
