[
https://issues.apache.org/jira/browse/SPARK-57957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-57957.
----------------------------------
Fix Version/s: 4.1.3
3.5.9
4.0.4
4.2.1
4.3.0
Resolution: Fixed
Issue resolved by pull request 57035
[https://github.com/apache/spark/pull/57035]
> Deflake GaussianMixtureSuite 'GMM support instance weighting' on macOS by
> using well-posed data
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-57957
> URL: https://issues.apache.org/jira/browse/SPARK-57957
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.3, 3.5.9, 4.0.4, 4.2.1, 4.3.0
>
>
> The `GMM support instance weighting` test in `GaussianMixtureSuite` fits k=5
> Gaussians on `KMeansSuite.generateKMeansData(50, 3, 5)`, which produces 5
> clusters of identical points (zero within-cluster variance). With k=5 the
> covariances are singular and the EM fit is ill-posed, so the weighted
> (uniform weight) and unweighted fits converge to different component-collapse
> patterns. On the macOS-26 CI runner (build_maven_java21_macos26) this is
> deterministic and the test fails on the mixture weight comparison (e.g.
> 0.0197 vs 0.1047). Reducing the instance weight (SPARK-37317 style) and
> increasing maxIter both fail to help (verified on macOS-26); increasing
> maxIter makes a component collapse further (7.6e-11 vs 0.116). Fix: run the
> invariant on well-posed data - fit rDataset with k=2 (real-variance data
> already used stably by another test in the suite). Test-only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
