I have not found matrix inversion algorithms in Spark and I would be surprised to see them. Except for matrices with very special structure (like those nearly the identity), inverting and n*n matrix is slower than O(n^2), which does not scale. Whenever a matrix is inverted, usually a decomposition or a low rank approximation is used, just as Sean pointed out. See further https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations#Matrix_algebra
or if you really want to dig into it Stoer and Bulirsch http://www.springer.com/us/book/9780387954523 On Mon, Sep 26, 2016 at 11:00 PM Sean Owen <so...@cloudera.com> wrote: > I don't recall any code in Spark that computes a matrix inverse. There is > code that solves linear systems Ax = b with a decomposition. For example > from looking at the code recently, I think the regression implementation > actually solves AtAx = Atb using a Cholesky decomposition. But, A = n x k, > where n is large but k is smallish (number of features), so AtA is k x k > and can be solved in-memory with a library. > > On Tue, Sep 27, 2016 at 3:05 AM, Cooper <ahmad.raban...@gmail.com> wrote: > > How is the problem of large-scale matrix inversion approached in Apache > Spark > > ? > > > > This linear algebra operation is obviously the very base of a lot of > other > > algorithms (regression, classification, etc). However, I have not been > able > > to find a Spark API on parallel implementation of matrix inversion. Can > you > > please clarify approaching this operation on the Spark internals ? > > > > Here <http://ieeexplore.ieee.org/abstract/document/7562171/> is a > paper on > > the parallelized matrix inversion in Spark, however I am trying to use an > > existing code instead of implementing one from scratch, if available. > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Large-scale-matrix-inverse-in-Spark-tp27796.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > > >
