Hi. I was wondering how useful an in memory sparse matrix multiplier would be for mahout. In my current project I needed to multiply many large sparse matrices but submitting hundreds of jobs to the cluster creates too much overhead.
I wrote up an implementation of sparse matrix multiplication using threads which can multiply a 30,000 x 48,0000 matrix by its transpose in about 5 minutes using 16 cores. Granted this matrix is composed mostly of 1s, and -1s, (with about 16 elements per row), is this considered fast? I have seen that my implementation is much faster than iterating though a matrix naively and would like some input to whether or not my 5 minute benchmark is by skewed. Many thanks for the input, Vincent
