Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4279#discussion_r23827460
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
---
@@ -172,6 +172,64 @@ class BlockMatrix(
assert(cols <= nCols, s"The number of columns $cols is more than
claimed $nCols.")
}
+ def validate: Unit = {
+ logDebug("Validating BlockMatrix...")
+ // check if the matrix is larger than the claimed dimensions
+ try {
+ estimateDim()
+ logDebug("BlockMatrix dimensions are okay...")
+ } catch {
+ case exc: AssertionError => throw new SparkException(s"$exc\nPlease
instantiate a " +
+ s"new BlockMatrix with the correct dimensions.")
+ case e: Exception =>
+ throw new
SparkException(s"${e.getMessage}\n${e.getStackTraceString}")
+ }
+ // Check if there are multiple MatrixBlocks with the same index.
+ val indexCounts = blocks.countByKey().filter(p => p._2 > 1)
+ if (indexCounts.size > 50) {
+ throw new SparkException(s"There are ${indexCounts.size}
MatrixBlocks with duplicate " +
+ s"indices. Please remove blocks with duplicate indices. You may
call reduceByKey on " +
+ s"the underlying RDD and sum the duplicates. You may convert the
matrices to Breeze " +
+ s"before summing them up.")
+ } else if (indexCounts.size > 0) {
+ var errorMsg = s"The following indices have more than one Matrix:\n"
+ indexCounts.foreach(index => errorMsg += s"Index: ${index._1},
count: ${index._2}\n")
+ errorMsg += "Please remove these blocks with duplicate indices. You
may call " +
+ "reduceByKey on the underlying RDD and sum the duplicates. You may
convert the " +
+ "matrices to Breeze before summing them up."
+ throw new SparkException(errorMsg)
+ }
+ logDebug("MatrixBlock indices are okay...")
+ // Check if each MatrixBlock (except edges) has the dimensions
rowsPerBlock x colsPerBlock
+ // The first tuple is the index and the second tuple is the dimensions
of the MatrixBlock
+ val blockDimensionMismatches = blocks.filter { case ((blockRowIndex,
blockColIndex), block) =>
+ if ((blockRowIndex == numRowBlocks - 1) || (blockColIndex ==
numColBlocks - 1)) {
+ false // neglect edge blocks
+ } else {
+ // include it if the dimensions don't match
+ !(block.numRows == rowsPerBlock && block.numCols == colsPerBlock)
+ }
+ }.map { case ((blockRowIndex, blockColIndex), mat) =>
+ ((blockRowIndex, blockColIndex), (mat.numRows, mat.numCols))
+ }
+ val dimensionMismatchCount = blockDimensionMismatches.count()
--- End diff --
Use `foreach` and throw exception when hit the first error.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
