Thanks a lot. -----Original Message----- From: Ted Dunning [mailto:[email protected]] Sent: Friday, July 23, 2010 11:53 PM To: [email protected] Subject: Re: distributed svd
One thing to note is that you can stack some of your user vectors into a matrix and do recommendations for several users at a time. This wins a bit on speed because you re-use elements multiple times. Using hadoop for this requires that you split your users and each mapper does a multiplication for a piece of the users with the decomposed components available as side data. The reducer is the identity function. There has been a bit of work done on this lately. Check the mailing list archives. Also, do the cost estimates to see if it wouldn't be better to do the recommends in real-time. This can be advantageous if most of your users don't come in frequently. Batch recommendation can be more efficient, but if you are just updating recommendations for users who won't be stopping by, then real-time recommendation might require considerably less total compute power. On Fri, Jul 23, 2010 at 1:46 AM, Laszlo Dosa <[email protected]>wrote: > Hi Akshay, > > I started to experiment with a subset of the data. > The number of users and items can easily grow up to 10 000 000. > > I am looking for a solution where having more users and items can be > compensate by more hardware. > Thus the time of calculation is constant. > > Is it possible to do this with svdlibc? If yes can you give me some advice > how? > > Regards, > Laszlo > > -----Original Message----- > From: Akshay Bhat [mailto:[email protected]] > Sent: vrijdag 23 juli 2010 10:23 > To: [email protected] > Subject: Re: distributed svd > > For each job do you have only 100k and 300k users/items? > > In that case I consider the svd job to be small enough to run on a single > core/machine. > You can use svdlibc to execute the job and then use hadoop to manage the > whole process. > if you have 100k small jobs then using C might be more advantageous. > > In my opinion Mahout SVD is useful when single job is way too big to fit > in memory of a single machine. > > > On Fri, Jul 23, 2010 at 4:14 AM, Laszlo Dosa > <[email protected]>wrote: > > > Hi, > > > > > > > > I tried to calculate SVD based recommendation by: > > > > 1) Calculate A'A > > > > 2) Calculate V_k by using Lanczos to get top k eigenvectors of A'A > > > > 3) Calculate preferences for each user by Vector usersPreference = > > V_k.timesSquared(row of A); > > > > 4) Calculate 20 recommendation for each user by selecting the top 20 > > preference from usersPreference > > > > > > > > The number of users and items is between 100 000 and 300 000. > > > > > > > > In my experience Step 3) scales linearly by the number of users. > > > > Each job runs fast, but I have ~100 000 jobs. This slows down the whole > > calculation. > > > > > > > > Is the approach wrong that I use? > > > > Or can you suggest something to make this faster? > > > > > > > > Regards, > > > > Laszlo > > > > > > > > > > > -- > Akshay Uday Bhat. > Graduate Student, Computer Science, Cornell University > Website: http://www.akshaybhat.com > >
