[
https://issues.apache.org/jira/browse/SPARK-6442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629225#comment-14629225
]
Rahul Palamuttam edited comment on SPARK-6442 at 7/16/15 5:46 AM:
------------------------------------------------------------------
I think instead of breeze, nd4j from deeplearning4j is a good library to look
at that tackles all 1,3,4. (2 [is in the works |
https://github.com/deeplearning4j/nd4j/issues/66])
I've been using it as an experimental backend for [SciSpark's sRDD |
http://apache-spark-developers-list.1001551.n3.nabble.com/SciSpark-NASA-AIST14-proposal-td10115.html]
It also has support for n-dimensions like numpy (unlike breeze). It's a very
young project. They have one version published in mvnrepository. I use it by
pulling from the git repo and and installing via maven locally.
To give a general idea of how the [nd4j library |
https://github.com/deeplearning4j/nd4j] works.
a) You can choose which backend you want - jblas, netlib-java, or x86. The x86
one uses netlib-java for BLAS operations and drops down to C level for-loops
(via JNI) for elementwise-operations.
b) They also provide a scala-api backend (with the operators) via DSL in a
separate project called nd4s.
c) They've recently provided the option of ordering elements contiguously in
memory. So element-wise-operations now benefit from cache locality. (The
performance is comparable to numpy and breeze)
I guess the general consensus is the community is spread thin, so it could be
worth it to wait for nd4j to mature a bit more and then tackle the problem.
was (Author: rahul palamuttam):
I think instead of breeze, nd4j from deeplearning4j is a good library to look
at that tackles all 4.
I've been using it as an experimental backend for [SciSpark's sRDD |
http://apache-spark-developers-list.1001551.n3.nabble.com/SciSpark-NASA-AIST14-proposal-td10115.html]
It also has support for n-dimensions like numpy (unlike breeze). It's a very
young project. They have one version published in mvnrepository. I use it by
pulling from the git repo and and installing via maven locally.
To give a general idea of how the [nd4j library |
https://github.com/deeplearning4j/nd4j] works.
a) You can choose which backend you want - jblas, netlib-java, or x86. The x86
one uses netlib-java for BLAS operations and drops down to C level for-loops
(via JNI) for elementwise-operations.
b) They also provide a scala-api backend (with the operators) via DSL in a
separate project called nd4s.
c) They've recently provided the option of ordering elements contiguously in
memory. So element-wise-operations now benefit from cache locality. (The
performance is comparable to numpy and breeze)
I guess the general consensus is the community is spread thin, so it could be
worth it to wait for nd4j to mature a bit more and then tackle the problem.
> MLlib Local Linear Algebra Package
> ----------------------------------
>
> Key: SPARK-6442
> URL: https://issues.apache.org/jira/browse/SPARK-6442
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
> Reporter: Burak Yavuz
> Priority: Critical
>
> MLlib's local linear algebra package doesn't have any support for any type of
> matrix operations. With 1.5, we wish to add support to a complete package of
> optimized linear algebra operations for Scala/Java users.
> The main goal is to support lazy operations so that element-wise can be
> implemented in a single for-loop, and complex operations can be interfaced
> through BLAS.
> The design doc: http://goo.gl/sf5LCE
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
