zhengruifeng opened a new pull request, #58611: URL: https://github.com/apache/spark/pull/58611
### What changes were proposed in this pull request? This PR reduces the transform closure size of `NaiveBayesModel` for multinomial, complement, Bernoulli, and Gaussian models. The model keeps its existing transient lazy prediction state. The column-expression prediction hooks snapshot `predictRawFunc` and optional thresholds so their UDFs do not retain the complete model and its parameter graph. Companion-object helpers construct the cached Bernoulli and Gaussian state and perform stateless prediction and probability calculations. This retries #58557 with a smaller implementation after its revert in #58610. ### Why are the changes needed? The default probabilistic classifier hooks invoke bound model methods. Their UDF closures therefore retain the complete `NaiveBayesModel`, even though scoring only needs its fitted vectors, matrices, model-specific derived state, and optional thresholds. This adds avoidable driver memory pressure for long-lived Spark Connect servers. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The following checks passed: ``` build/sbt mllib/compile build/sbt 'mllib/testOnly org.apache.spark.ml.classification.NaiveBayesSuite' ``` `NaiveBayesSuite` ran 17 tests covering all four model types. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
