Sounds like you should invert your loops. These sparse matrices are probably very reasonable for solving each one on a single machine in memory. As such, take a look at the LSMR implementation which is a good implementation of a conjugate gradient-like algorithm that plays nice with sparse data. It could probably even be used in a re-weighting scheme where the matrix being solved varies in each iteration (at a serious cost in understanding for the algorithm).
What I would suggest is that you define your mapper so that each mapper solves one of your problem LS problems. You have 50,000 of them and so that will provide plenty of parallelism. LSMR works out of core if you have to do that, but it really sounds like you should be able to handle several in memory at the same time. Can you say more about your memory on each node and how many non-zero elements are in your matrices? On Mon, Jun 20, 2011 at 12:25 PM, Vincent Xue <[email protected]> wrote: > Hello everyone, > > I have been using the Mahout library to implement 'Iterative Recursive > Least Squares' for some analysis on biological microarrays. In my > setup, I have to perform IRLS about 54,000 times on different > probesets, and each job involves multiplying large sparse matrices and > solving linear systems via conjugate gradient. I am using mahout > because my matrices are sparse but large (500,000 x 30,000). > > The problem I have now is that each IRLS Job takes about 48 hours to > complete running on a small cluster of 75 nodes with 16 cores each. I > can run about 200 jobs at the same time, but this is not very helpful > considering I have 54,000 jobs to process. > > Using the mahout library, the matrix multiplication is greatly sped up > ( less than 5 minutes for each product). However solving a linear > system using conjugate gradient is a time consuming process and is the > bulk of the computation. Considering that IRLS calls for several > iterations, this problem is magnified by however many iterations I run > it for. > > Considering the issues, I hope someone can help me find a solution. > Below lists several concerns: > > 1) Is using Mahout for this computation the correct approach. (I have > tried running this in R and the simple step of multiplying matrices > would take days, if it could fit in memory) > 2) When running the IRLS, even with 200 jobs running (or maps), the > cpu usage for each node barely goes above 5%. How can I use more CPU? > > Thank you for your help > Vincent >
