This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6d6e2dc77bfc [SPARK-58293][ML] Delay VectorIndexerModel attribute
initialization
6d6e2dc77bfc is described below
commit 6d6e2dc77bfc96638751687e49da2fe1b3936131
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri Jul 24 18:27:13 2026 +0800
[SPARK-58293][ML] Delay VectorIndexerModel attribute initialization
### What changes were proposed in this pull request?
Replaces the retained `partialFeatureAttributes` field in
`VectorIndexerModel` with an on-demand private method. The attributes are now
computed only when output metadata is prepared.
### Why are the changes needed?
The attributes are derived state used only while preparing output metadata.
Retaining them on every fitted or loaded model allocates one attribute per
feature even when the model is never transformed. Computing them at the call
site avoids that unnecessary retained state.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The test suite was not run locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex
Closes #57483 from zhengruifeng/delay-partial-feature-attributes-dev3.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../scala/org/apache/spark/ml/feature/VectorIndexer.scala | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala
index 7fce6d51dcfc..57d28bba2eeb 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala
@@ -326,10 +326,10 @@ class VectorIndexerModel private[ml] (
}
/**
- * Pre-computed feature attributes, with some missing info.
- * In transform(), set attribute name and other info, if available.
+ * Computes feature attributes with some missing information.
+ * In transform(), set attribute name and other information, if available.
*/
- private val partialFeatureAttributes: Array[Attribute] = {
+ private def partialFeatureAttributes: Array[Attribute] = {
val attrs = new Array[Attribute](numFeatures)
var categoricalFeatureCount = 0 // validity check for numFeatures,
categoryMaps
var featureIndex = 0
@@ -358,8 +358,11 @@ class VectorIndexerModel private[ml] (
}
featureIndex += 1
}
- require(categoricalFeatureCount == categoryMaps.size, "VectorIndexerModel
given categoryMaps" +
- s" with keys outside expected range [0,...,numFeatures), where
numFeatures=$numFeatures")
+ require(
+ categoricalFeatureCount == categoryMaps.size,
+ "VectorIndexerModel given categoryMaps" +
+ s" with keys outside expected range [0,...,numFeatures), " +
+ s"where numFeatures=$numFeatures")
attrs
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
