zhengruifeng opened a new pull request, #58846: URL: https://github.com/apache/spark/pull/58846
### What changes were proposed in this pull request? This PR replaces the final `RDD.collect()` in MLlib Word2Vec training with a `SparkContext.runJob` result handler. The driver copies each completed partition's aggregated vectors into the global model arrays as the partition result arrives. ### Why are the changes needed? `collect()` retains aggregated vectors from every partition on the driver while the global Word2Vec model arrays are also resident. For large vocabularies and vector sizes, this increases peak driver memory and can cause an out-of-memory error. Processing partition results incrementally allows completed results to be reclaimed instead of retaining the complete aggregated RDD result. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Not run yet. This is a draft PR. The change only modifies how already-aggregated partition results are materialized on the driver. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
