dbtsai opened a new pull request, #59485: URL: https://github.com/apache/spark/pull/59485
### What changes were proposed in this pull request? Replace Breeze delegation in Spark's vector and matrix primitives: - Dispatch vector element access to Spark's dense and sparse implementations. - Convert sparse CSC/CSR layouts with counts, prefix sums, and scatter, preserving sorted indices and removing explicitly stored zeros without dense storage. - Implement matrix equality and sparse hashing in Spark, retaining Breeze's existing floating-point and explicit-zero behavior. - Add a shared matrix formatter that preserves output layout, truncation, transpose handling, and locale-sensitive formatting. - Reuse the ML matrix implementations from legacy MLlib through the existing `asML` adapters. - Add compatibility regressions and Breeze attribution in the NOTICE files. This PR is limited to these primitives. The `asBreeze`/`fromBreeze` adapters and Breeze dependency declarations remain for existing algorithm consumers. ### Why are the changes needed? These basic operations can work directly with Spark's dense and sparse storage. Moving them into Spark is a preparatory step toward reducing its dependence on Breeze and its transitive dependencies. This PR does not yet reduce the shipped dependency set. ### Does this PR introduce _any_ user-facing change? No. Existing equality, sparse hash, and formatting behavior is preserved, including the compatibility edge cases for NaN, signed zeros, explicitly stored zeros, and transposed sparse matrices. There are no public API or dependency changes. ### How was this patch tested? Added regression tests for sparse layout conversion, empty and very large sparse matrices, floating-point edge cases, equality and hashing parity, and formatting/truncation/locale parity. Legacy MLlib tests cover equality, sparse hashing, and formatting parity as well. Local verification: **197 tests passed across 13 suites**, using a direct Scala compiler/ScalaTest fallback with cached dependencies: - `mllib-local`: 107 tests across `MatricesSuite`, `VectorsSuite`, `BreezeMatrixConversionSuite`, `BreezeVectorConversionSuite`, `BLASSuite`, `MultivariateGaussianSuite`, `UtilsSuite`, and `TestingUtilsSuite`. - Legacy `mllib`: 90 tests across `MatricesSuite`, `VectorsSuite`, `BreezeMatrixConversionSuite`, `BreezeVectorConversionSuite`, and `BLASSuite`. The fallback compiled the current production and test sources without stubs or source/build-configuration changes. It used Java 21, Breeze 2.1.0, and cached netlib 3.0.4. The local-module run used Scala 2.13.17; the legacy run used Scala 2.13.18 and cached Spark 4.2.0-SNAPSHOT supporting binaries, including the real `SparkFunSuite`, with the newly compiled local/legacy linear-algebra classes first on the classpath. **Build limitation:** the standard SBT invocation was attempted but could not download the sbt 1.13.0 launcher, so it never reached compilation or tests. These fallback results are not a full build against the current branch's pinned dependencies. Standard build and CI verification remain necessary. Standard commands to reproduce the covered suites in a fully provisioned checkout: ```sh build/sbt 'mllib-local/testOnly *MatricesSuite *VectorsSuite *BreezeMatrixConversionSuite *BreezeVectorConversionSuite *BLASSuite *MultivariateGaussianSuite *UtilsSuite *TestingUtilsSuite' build/sbt 'mllib/testOnly org.apache.spark.mllib.linalg.MatricesSuite org.apache.spark.mllib.linalg.VectorsSuite org.apache.spark.mllib.linalg.BreezeMatrixConversionSuite org.apache.spark.mllib.linalg.BreezeVectorConversionSuite org.apache.spark.mllib.linalg.BLASSuite' ``` `git diff --check` and changed-source ASCII/line-length checks also passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
