Thanks jake. -----Original Message----- From: Jake Mannix [mailto:[email protected]] Sent: Saturday, July 24, 2010 1:10 AM To: [email protected] Subject: Re: distributed svd
Hi Lazlo, On Fri, Jul 23, 2010 at 1:14 AM, Laszlo Dosa <[email protected]> wrote: I tried to calculate SVD based recommendation by: 1) Calculate A'A 2) Calculate V_k by using Lanczos to get top k eigenvectors of A'A The SVD implementation in Mahout does not require you do these two steps separately - they happen at the same time if you pass in the A matrix to the DistributedLanczosSolver directly. 3) Calculate preferences for each user by Vector usersPreference = V_k.timesSquared(row of A); What you want to do here is actually take the A matrix, represented by a DistributedRowMatrix, and the V_k matrix (you actually get basically V_k's transposed from the SVD job), represented by that same class, and then just compute A . V . S^-1, where S is the diagonal matrix of singular values (note that these are the sqrts of the eigenvalues you get out of Lancos). A . V can be computed in one mapreduce job (DistributedRowMatrix.times() ), so that's the way to do that efficiently. -jake 4) Calculate 20 recommendation for each user by selecting the top 20 preference from usersPreference The number of users and items is between 100 000 and 300 000. In my experience Step 3) scales linearly by the number of users. Each job runs fast, but I have ~100 000 jobs. This slows down the whole calculation. Is the approach wrong that I use? Or can you suggest something to make this faster? Regards, Laszlo
