This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ad37629bce46 [SPARK-58583][ML] Construct squared Euclidean silhouette
stats on executors
ad37629bce46 is described below
commit ad37629bce4664611d164df1a800fdf404327031
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Wed Aug 5 08:44:26 2026 +0000
[SPARK-58583][ML] Construct squared Euclidean silhouette stats on executors
### What changes were proposed in this pull request?
This PR constructs `SquaredEuclideanSilhouette.ClusterStats` in the
executor-side RDD
transformation before collecting the per-cluster map to the driver.
### Why are the changes needed?
The previous code collected tuple values and then constructed a second map
of `ClusterStats`
on the driver. Constructing the values before collection avoids that
driver-side conversion and
its temporary map allocation.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Static checks completed successfully:
- `git diff --check`
- ASCII scan of the changed Scala file
- changed-line length scan
No test suite was run because it was not requested; this is a
behavior-preserving relocation of
a serializable value-object construction.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57782 from zhengruifeng/silhouette-cluster-stats-executor-dev3.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit 428545ec202233ab448eb558e2e90bf39756f975)
Signed-off-by: Uros Bojanic <[email protected]>
---
.../org/apache/spark/ml/evaluation/ClusteringMetrics.scala | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringMetrics.scala
b/mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringMetrics.scala
index 98fbe471f297..12735f835e68 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringMetrics.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringMetrics.scala
@@ -330,13 +330,10 @@ private[evaluation] object SquaredEuclideanSilhouette
extends Silhouette {
}
)
- clustersStatsRDD
- .collectAsMap()
- .toMap
- .transform {
- case (_, (featureSum: DenseVector, squaredNormSum: Double, weightSum:
Double)) =>
- SquaredEuclideanSilhouette.ClusterStats(featureSum, squaredNormSum,
weightSum)
- }
+ clustersStatsRDD.mapValues {
+ case (featureSum: DenseVector, squaredNormSum: Double, weightSum:
Double) =>
+ SquaredEuclideanSilhouette.ClusterStats(featureSum, squaredNormSum,
weightSum)
+ }.collectAsMap().toMap
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
