zhengruifeng opened a new pull request, #58341:
URL: https://github.com/apache/spark/pull/58341
### What changes were proposed in this pull request?
This PR adds protected column-expression hooks to `ClassificationModel` for
producing raw
predictions, converting raw predictions to labels, and predicting labels
directly from features.
`ClassificationModel.transform` uses these hooks while retaining the
existing UDF-based defaults.
`LinearSVCModel` overrides the hooks to snapshot only coefficients,
intercept, and threshold in
its UDF closures. Raw-prediction-to-label conversion uses Spark column
expressions, and the
model-capturing `margin` function field is removed.
### Why are the changes needed?
The previous transform implementation built UDFs from bound model methods.
This serialized the
entire model into each closure and performed parameter lookup for every
input row. The hooks let
classification models provide compact closures or native Spark expressions
without changing
prediction behavior.
An in-process comparison used 4,096-feature models for closure serialization
and 3,000,000 cached
rows with 32 features for 20 alternating execution measurements:
| Transform path | Previous closure | New closure | Previous median | New
median |
|---|---:|---:|---:|---:|
| Raw prediction | 38,268 B | 34,627 B (-9.5%) | - | - |
| Prediction only | 38,206 B | 34,580 B (-9.5%) | 426.3 ms | 229.2 ms
(-46.2%) |
| Raw + prediction | 76,552 B (2 UDFs) | 34,627 B (1 UDF) (-54.8%) | 818.8
ms | 586.7 ms (-28.3%) |
The temporary benchmark probe was removed after measurement.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- `build/sbt mllib/scalastyle`
- `build/sbt 'mllib/testOnly
org.apache.spark.ml.classification.LinearSVCSuite'`
(16 tests passed; 1 pre-existing test ignored)
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
