This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 678ee31f0f93 [SPARK-58215][ML][CONNECT] Include parameter metadata in
tree regressor size estimates
678ee31f0f93 is described below
commit 678ee31f0f93f8b700d079f85b224ced36152339
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Tue Jul 21 08:35:25 2026 +0800
[SPARK-58215][ML][CONNECT] Include parameter metadata in tree regressor
size estimates
### What changes were proposed in this pull request?
This patch extends the parameter-metadata accounting from #57322 to
`DecisionTreeRegressionModel`, `RandomForestRegressionModel`, and
`GBTRegressionModel`. Their custom `estimatedSize` implementations now include
the existing parameter/default maps and UID in addition to learned tree state.
The existing Spark Connect early-stop test limits are raised to reflect the
classifier metadata accounting introduced by #57322.
### Why are the changes needed?
Spark Connect uses a model's estimated size for ML cache accounting and
tree-training early stopping. The tree regression models override the general
estimate but previously omitted their parameter metadata, undercounting the
memory charged to cached models. This keeps the regression models consistent
with the classifier counterparts.
### Does this PR introduce _any_ user-facing change?
Yes. Spark Connect reports a larger estimated size for tree regression
models, so ML cache accounting and model-size-based early stopping include
model parameter metadata.
### How was this patch tested?
Updated the existing Connect tree-model early-stop limits to account for
metadata. `git diff --check` passed. The relevant test suite was not run.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57367 from
zhengruifeng/SPARK-58182-tree-regressor-size-metadata-dev2.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala | 2 +-
mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala | 2 +-
.../scala/org/apache/spark/ml/regression/RandomForestRegressor.scala | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala
b/mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala
index 5387e0e282a3..135224d9a8f9 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala
@@ -190,7 +190,7 @@ class DecisionTreeRegressionModel private[ml] (
// For ml connect only
private[ml] def this() = this("", Node.dummyNode, -1)
- private[spark] override def estimatedSize: Long = getEstimatedSize()
+ private[spark] override def estimatedSize: Long = estimateMatadataSize +
getEstimatedSize()
override def predict(features: Vector): Double = {
rootNode.predictImpl(features).prediction
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
b/mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
index 71436036d1ea..b533953b91dc 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
@@ -245,7 +245,7 @@ class GBTRegressionModel private[ml](
// For ml connect only
private[ml] def this() = this("", Array(new DecisionTreeRegressionModel),
Array(Double.NaN), -1)
- private[spark] override def estimatedSize: Long = getEstimatedSize()
+ private[spark] override def estimatedSize: Long = estimateMatadataSize +
getEstimatedSize()
@Since("1.4.0")
override def trees: Array[DecisionTreeRegressionModel] = _trees
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala
b/mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala
index 8d9b4817833b..7826c7843504 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala
@@ -215,7 +215,7 @@ class RandomForestRegressionModel private[ml] (
// For ml connect only
private[ml] def this() = this("", Array(new DecisionTreeRegressionModel), -1)
- private[spark] override def estimatedSize: Long = getEstimatedSize()
+ private[spark] override def estimatedSize: Long = estimateMatadataSize +
getEstimatedSize()
@Since("1.4.0")
override def trees: Array[DecisionTreeRegressionModel] = _trees
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
