zhengruifeng opened a new pull request, #58336: URL: https://github.com/apache/spark/pull/58336
### What changes were proposed in this pull request? This PR adds protected column-expression hooks to `ClassificationModel` and `ProbabilisticClassificationModel`. The shared transform implementations use these hooks to build raw prediction, probability, and prediction columns. Their default implementations preserve the existing row-based UDF behavior, while individual models can provide native Catalyst expressions or UDFs backed by compact prediction state. `LogisticRegressionModel` uses the new hooks to snapshot only coefficients, intercepts, mode, and thresholds needed by each expression. Binary raw/probability thresholding uses native column expressions, while the remaining UDFs call companion-object helpers without retaining the model. ### Why are the changes needed? The previous classifier transform UDFs invoked bound model methods. As a result, their closures retained the full model, including its parameter graph, even when a prediction stage needed only a small subset of immutable scoring state. The column-expression hooks provide a common design for classifiers to reduce those closures and to use native Spark expressions where possible. This lowers driver memory pressure from retained query plans on long-lived Spark Connect servers. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The following checks were run locally: ``` build/sbt mllib/compile build/sbt mllib/scalastyle build/sbt 'mllib/testOnly org.apache.spark.ml.classification.ClassifierSuite org.apache.spark.ml.classification.LogisticRegressionSuite' ``` The focused test run completed 66 tests successfully, with one existing ignored test. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
