HyukjinKwon opened a new pull request, #57035: URL: https://github.com/apache/spark/pull/57035
### What changes were proposed in this pull request? Change the `GMM support instance weighting` test to fit `rDataset` with `k=2` instead of fitting `k=5` on `KMeansSuite.generateKMeansData(50, 3, 5)`. ### Why are the changes needed? The generated KMeans data is 5 clusters of identical points (zero within-cluster variance). Fitting k=5 Gaussians makes the covariances singular and the EM fit ill-posed, so the uniform-weighted and unweighted fits converge to different component-collapse patterns. On the macOS-26 runner (`build_maven_java21_macos26`) this deterministically fails the mixture-weight comparison (0.0197 vs 0.1047). Reducing the instance weight (as in SPARK-37317, which reduced 100->90) and increasing maxIter were both verified on a macOS-26 runner to NOT help — increasing maxIter actually makes a component collapse further (7.6e-11 vs 0.116). Running the same `unweighted == uniform-weighted` invariant on well-posed data (`rDataset`, real variance, `k=2` — already used stably by another test in this suite) makes both fits converge to the same optimum. ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? Ran `GaussianMixtureSuite` on a macOS-26 GitHub Actions runner: 13 succeeded, 0 failed (previously the instance-weighting test failed deterministically on macOS-26). Also passes on Linux. ### Was this patch authored or co-authored using generative AI tooling? Yes. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
