zhengruifeng opened a new pull request, #58559: URL: https://github.com/apache/spark/pull/58559
### What changes were proposed in this pull request? This PR reduces the transform closure size of `FMClassificationModel`. The column-expression prediction hooks snapshot only the intercept, linear coefficients, factor matrix, and optional thresholds needed by each requested output column. Companion-object helpers perform raw prediction and probability conversion without retaining the complete Spark ML model or its parameter graph. ### Why are the changes needed? The default probabilistic classifier hooks invoke bound model methods. Their UDF closures therefore retain the complete `FMClassificationModel` even though scoring only needs its fitted coefficients and a small amount of immutable prediction state. This adds avoidable driver memory pressure for long-lived Spark Connect servers. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The following checks passed: ``` build/sbt mllib/compile build/sbt 'mllib/testOnly org.apache.spark.ml.classification.FMClassifierSuite' ``` `FMClassifierSuite` ran 16 tests, including its explicit threshold tests. No new tests were added because its existing `testPredictMethods` coverage exercises every combination of raw-prediction, probability, and prediction output columns, in addition to single-instance prediction and model persistence. A temporary local probe used a deterministic model with 1,024 features and 8 factors. It extracted each `ScalaUDF.function` from the analyzed transform plan and serialized it with Spark's closure serializer. The former bound-model hooks were recreated in a temporary subclass. "All" serializes the functions for raw-prediction, probability, and prediction together. | Model/output | Before | After | Reduction | |---|---:|---:|---:| | Raw prediction | 81,509 B | 76,466 B | 6.2% | | Probability | 81,516 B | 76,474 B | 6.2% | | Prediction | 81,446 B | 76,440 B | 6.1% | | All | 82,052 B | 76,706 B | 6.5% | ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
