Hi Laslo, You are right that given the way that Lanczos spits out eigenvectors, the way to calculate U_k using primitive operations (i.e. those available on DistributedRowMatrix) is to do two transpose() operations and then a times() operation. Then of course, you need to do the S^-1 multiplication, which is another pass over the data.
V_k is by it's nature, returned as eigenvectors - k of them, each with m entries. A is n rows, each with cardinality m. The problem you have is that you really do need to join these two sets of vectors on their column label. You can do that by doing a MR pass over both sets, in parallel, and then doing the MatrixMultiplicationJob after, or by making one MR job which takes multiple inputs, and transposes them simultaneously, doing a reduce-side join and doing the multiplication there, but then you're still stuck with a final summation (ie identity mapper followed by summing reducer) anyways, I think. I think the only real speedup to be had here, given the way the data is presented, is to do the A.transpose() and the V_k.transpose() in parallel, and to fold the multiplication by S^-1 into either one of these tasks, or into the multiplication (since S^-1 fits in memory). -jake
