GitHub user mengxr opened a pull request:
https://github.com/apache/spark/pull/296
[SPARK-1390] [WIP] Refactoring of matrices backed by RDDs
This is a WIP for refactoring interfaces for matrices backed by RDDs. It
would be better if we have a clear separation of local matrices and those
backed by RDDs. Right now, we have
1. `org.apache.spark.mllib.linalg.SparseMatrix`, which is a wrapper over an
RDD of matrix entries, i.e., coordinate list format.
2. `org.apache.spark.mllib.linalg.TallSkinnyDenseMatrix`, which is a
wrapper over RDD[Array[Double]], i.e. row-oriented format.
We will see naming collision when we introduce local `SparseMatrix`, and
the name `TallSkinnyDenseMatrix` is not exact if we switch to `RDD[Vector]`
from `RDD[Array[Double]]`. It would be better to have "RDD" in the class name
to suggest that operations may trigger jobs.
The proposed names are (all under `org.apache.spark.mllib.linalg.rdd`):
1. `RDDMatrix`: trait for matrices backed by one or more RDDs
2. `CoordinateRDDMatrix`: wrapper of `RDD[(Long, Long, Double)]`
3. `RowRDDMatrix`: wrapper of `RDD[Vector]` whose rows do not have special
ordering
4. `IndexedRowRDDMatrix`: wrapper of `RDD[(Long, Vector)]` whose rows are
associated with indices
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mengxr/spark mat
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/296.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #296
----
commit 7836e2fe7b2d55008eb63ec56d207fbe28c0c2ab
Author: Xiangrui Meng <[email protected]>
Date: 2014-04-02T05:31:27Z
initial refactoring of matrices backed by RDDs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
