This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 24d0b5a4fbef [MINOR][ML] Avoid repeated MLP weight copies across layers
24d0b5a4fbef is described below
commit 24d0b5a4fbef7e36b824b22bff97335a06df0883
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Wed Aug 26 08:38:33 2026 +0800
[MINOR][ML] Avoid repeated MLP weight copies across layers
### What changes were proposed in this pull request?
This PR converts the feed-forward model's weight vector to a dense array
once and shares that
array among the non-overlapping Breeze views used by its layer models.
It also adds a regression test that constructs a model from sparse weights
and verifies that all
affine layers share one dense backing array.
### Why are the changes needed?
MLP weights produced by normal training are typically dense, but `Vector`,
`initialWeights`, and
the persisted model format do not guarantee a dense representation.
`FeedForwardModel` previously
called `weights.toArray` inside the layer loop. Each affine layer retained
its resulting full-size
dense array even though it used only one slice, producing one complete
weight copy per affine
layer. Sparse weights also underwent the same full sparse-to-dense
conversion repeatedly.
Converting once preserves support for both dense and sparse inputs while
avoiding the repeated
allocations and retained copies.
### Does this PR introduce _any_ user-facing change?
No. Model results and APIs are unchanged; the implementation uses less
memory.
### How was this patch tested?
A regression test was added to `ANNSuite` using sparse model weights.
```
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 build/sbt \
'mllib/testOnly org.apache.spark.ml.ann.ANNSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #58276 from zhengruifeng/ml-mlp-share-dense-weights-dev-3.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 80f8b8a79dad2ee9e24528ddeda92a78de831140)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala | 5 ++++-
mllib/src/test/scala/org/apache/spark/ml/ann/ANNSuite.scala | 13 +++++++++++++
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala
b/mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala
index 889f6febbe40..7b7ec5f21e56 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala
@@ -479,10 +479,13 @@ private[ml] class FeedForwardModel private(
val layers = topology.layers
val layerModels = new Array[LayerModel](layers.length)
+ // Trained weights are normally dense, but Vector and persisted models do
not guarantee it.
+ // Convert once so all layer views share one dense backing array.
+ private val denseWeights = weights.toArray
private var offset = 0
for (i <- layers.indices) {
layerModels(i) = layers(i).createModel(
- new BDV[Double](weights.toArray, offset, 1, layers(i).weightSize))
+ new BDV[Double](denseWeights, offset, 1, layers(i).weightSize))
offset += layers(i).weightSize
}
private var outputs: Array[BDM[Double]] = null
diff --git a/mllib/src/test/scala/org/apache/spark/ml/ann/ANNSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/ann/ANNSuite.scala
index e2d00c98f1ca..c81d3763169b 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/ann/ANNSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/ann/ANNSuite.scala
@@ -25,6 +25,19 @@ import org.apache.spark.util.ArrayImplicits._
class ANNSuite extends SparkFunSuite with MLlibTestSparkContext {
+ test("layer models share one dense weights array") {
+ val topology = FeedForwardTopology.multiLayerPerceptron(Array(2, 3, 2))
+ val numWeights = topology.layers.map(_.weightSize).sum
+ val weights = Vectors.sparse(numWeights, Array(0, numWeights - 1),
Array(1.0, 2.0))
+
+ val model = topology.model(weights)
+ val affineLayers = model.layerModels.collect { case layer:
AffineLayerModel => layer }
+
+ assert(affineLayers.length === 2)
+ assert(affineLayers.forall(_.weights.data eq
affineLayers.head.weights.data))
+ assert(affineLayers.head.weights.data.length === numWeights)
+ }
+
// TODO: test for weights comparison with Weka MLP
test("ANN with Sigmoid learns XOR function with LBFGS optimizer") {
val inputs = Array(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
