Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/9916#discussion_r50151020
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
---
@@ -317,40 +317,69 @@ class BlockMatrix @Since("1.3.0") (
}
/**
- * Adds two block matrices together. The matrices must have the same
size and matching
- * `rowsPerBlock` and `colsPerBlock` values. If one of the blocks that
are being added are
- * instances of [[SparseMatrix]], the resulting sub matrix will also be
a [[SparseMatrix]], even
- * if it is being added to a [[DenseMatrix]]. If two dense matrices are
added, the output will
- * also be a [[DenseMatrix]].
+ * For given matrices `this` and `other` of compatible dimensions and
compatible block dimensions,
+ * it applies a binary function on their corresponding blocks.
+ *
+ * @param other The second BlockMatrix argument for the operator
specified by `binMap`
+ * @param binMap A function taking two breeze matrices and returning a
breeze matrix
+ * @return A [[BlockMatrix]] whose blocks are the results of a specified
binary map on blocks
+ * of `this` and `other`.
*/
- @Since("1.3.0")
- def add(other: BlockMatrix): BlockMatrix = {
+ private[mllib] def blockMap(
+ other: BlockMatrix,
+ binMap: (BM[Double], BM[Double]) => BM[Double]): BlockMatrix = {
require(numRows() == other.numRows(), "Both matrices must have the
same number of rows. " +
s"A.numRows: ${numRows()}, B.numRows: ${other.numRows()}")
require(numCols() == other.numCols(), "Both matrices must have the
same number of columns. " +
s"A.numCols: ${numCols()}, B.numCols: ${other.numCols()}")
if (rowsPerBlock == other.rowsPerBlock && colsPerBlock ==
other.colsPerBlock) {
- val addedBlocks = blocks.cogroup(other.blocks, createPartitioner())
+ val newBlocks = blocks.cogroup(other.blocks, createPartitioner())
.map { case ((blockRowIndex, blockColIndex), (a, b)) =>
if (a.size > 1 || b.size > 1) {
throw new SparkException("There are multiple MatrixBlocks with
indices: " +
s"($blockRowIndex, $blockColIndex). Please remove them.")
}
if (a.isEmpty) {
- new MatrixBlock((blockRowIndex, blockColIndex), b.head)
+ val zeroBlock = BM.zeros[Double](b.head.numRows,
b.head.numCols)
--- End diff --
This is making an explicit matrix of all 0s. It would be better to create
a sparse matrix of all zeros to avoid the memory allocation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
