dongjoon-hyun opened a new pull request, #59559: URL: https://github.com/apache/spark/pull/59559
### What changes were proposed in this pull request? This PR aims to make `DenseMatrix.hashCode` value-based and consistent with `equals` in both `org.apache.spark.ml.linalg` and `org.apache.spark.mllib.linalg`. - `ml.linalg.DenseMatrix.hashCode` now computes the same hash code as `SparseMatrix.hashCode`, i.e., Breeze's `QuasiTensor.hashCode`. It starts from `43` and mixes the `##` of each value with `MurmurHash3.mix` in column-major order, skipping zero hash codes such as those of `0.0` and `-0.0`. A transposed matrix is also visited in column-major order. - `mllib.linalg.DenseMatrix.hashCode` delegates to `asML.hashCode`. ### Why are the changes needed? `DenseMatrix.hashCode` hashes `toArray`, which allocates a new array on every call, and an array is hashed by its identity. As a result, the same instance returns a different hash code on every call, and equal matrices have different hash codes. This violates the `hashCode` contract, so a `DenseMatrix` cannot be found in hash-based collections such as `HashSet` and `HashMap`. This has been the case since Apache Spark [v1.4.0](https://github.com/apache/spark/releases/tag/v1.4.0) (2015-06-11) for `mllib` (SPARK-6428) and [v2.0.0](https://github.com/apache/spark/releases/tag/v2.0.0) (2016-07-26) for `ml` (SPARK-14549). For example, RDD operations silently return wrong results when equal matrices are used as keys. `org.apache.spark.mllib.linalg.DenseMatrix` has the same problem. ```scala val m = new org.apache.spark.ml.linalg.DenseMatrix(2, 2, Array(1.0, 2.0, 3.0, 4.0)) val rdd = sc.parallelize(Seq(m, m.copy, m.copy, m.toSparse), 2) rdd.distinct().count() // Before: 4, After: 1 rdd.countByValue().values.sum // Before: 1, After: 4 val left = sc.parallelize(Seq(m -> "left"), 2) val right = sc.parallelize(Seq(m.copy -> "right"), 2) left.join(right).count() // Before: 0, After: 1 ``` Note that `asBreeze.hashCode()` cannot be used for `DenseMatrix` because Breeze's `DenseMatrix` visits its values in row-major order while `CSCMatrix` visits them in column-major order. ### Does this PR introduce _any_ user-facing change? Yes. `DenseMatrix.hashCode` becomes deterministic and consistent with `equals`, including the equality between a dense matrix and a sparse matrix with the same entries. ```scala val m = new org.apache.spark.ml.linalg.DenseMatrix(2, 2, Array(1.0, 2.0, 3.0, 4.0)) m.hashCode == m.hashCode // Before: false, After: true m == m.toSparse // Before: true, After: true m.hashCode == m.toSparse.hashCode // Before: false, After: true scala.collection.mutable.HashSet(m).contains(m) // Before: false, After: true ``` ### How was this patch tested? Pass the CIs with the newly added test cases in `ml.linalg.MatricesSuite` and `mllib.linalg.MatricesSuite`. Without this fix, both new test cases fail at the first assertion, `dm.hashCode === dm.hashCode`. ``` $ build/sbt 'mllib-local/testOnly *MatricesSuite' $ build/sbt 'mllib/testOnly org.apache.spark.mllib.linalg.MatricesSuite' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
