Github user fommil commented on the pull request:
https://github.com/apache/incubator-spark/pull/575#issuecomment-35126301
@mikiobraun apologies for misquoting you: I have obviously inferred too
much from your commendation of `netlib-java`'s recent updates, and when we
discussed a potential merger of the projects (you suggested replacing the JBLAS
native layer with `netlib-java`, but cited lack of time). I also know you
originally wrote JBLAS because you thought MTJ was unmaintained. We'll call it
even, since you claimed at the ICML that netlib-java was not maintained :-P
To correct a minor point: `netlib-java` will use **system optimised**
compliant natives if they are available, falling back to fortran (which is
still faster than any Java implementation I've seen), and falling back to a
pure JVM implementation (which is on par with most of the other
implementations).
MTJ, like Breeze, is a linear algebra library on top of `netlib-java`. In
this regard, it is perhaps best to be comparing JBLAS to MTJ or Breeze... but
with MTJ/Breeze both using `netlib-java` so having comparable performance.
Regarding pre-built ATLAS binaries: my benchmarks have shown ATLAS to be
**significantly** slower than Mac's bundled `vecLib`. Also, ATLAS is already
available on most Linux distributions with a single command line call...
whereas `netlib-java` can even use commercial vendor packages if needed (such
as Intel, AMD or GPU implementations).
Having said all this, the binary compatibility on Linux (of both
`netlib-java` and JBLAS) is pretty flakey, as the various distributions tend to
introduce arbitrary binary incompatibilities. I tested on Debian, Ubuntu and
RHEL, but I've heard reports that it breaks in SuSe.
From a purely selfish point of view, I recommend Breeze with `netlib-java`
if the end API is intended to be Scala, to avoid creating endless layers upon
layers and everybody filing bugs with two of my projects :-D That said, I think
MTJ's sparse support is pretty good... have you seen the recent ARPACK changes
I made? (They'd be pretty trivial to re-implement in Breeze since it is
provided by `netlib-java`)
@mengxr I am interested in your benchmarks. Have you seen Peter Abeles'
suite? https://code.google.com/p/java-matrix-benchmark/
