Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19588#discussion_r150771136
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala ---
@@ -311,22 +342,39 @@ class VectorIndexerModel private[ml] (
// TODO: Check more carefully about whether this whole class will be
included in a closure.
/** Per-vector transform function */
- private val transformFunc: Vector => Vector = {
+ private lazy val transformFunc: Vector => Vector = {
--- End diff --
I don't think changing it to a `def` is a good idea.
Because, it will cause the whole `Model` object to be serialized.
Current code, it will create a closure, which only involve the outer
variables which are needed. It can help to avoid unexpected serialization.
```
lazy val transformFunc = {
// the returned closure `f` will only reference the following 4 outer
variables.
val sortedCatFeatureIndices = categoryMaps.keys.toArray.sorted
val localVectorMap = categoryMaps
val localNumFeatures = numFeatures
val localHandleInvalid = getHandleInvalid
val f : Vector => Vector = { (v: Vector) => ...}
f // return closure `f`
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
