Ruifeng Zheng created SPARK-58181:
-------------------------------------
Summary: Improve connect ML model size estimation
Key: SPARK-58181
URL: https://issues.apache.org/jira/browse/SPARK-58181
Project: Spark
Issue Type: Improvement
Components: Connect, ML
Affects Versions: 4.3.0
Reporter: Ruifeng Zheng
### Background
Spark Connect uses Model.estimatedSize to enforce per-model and per-session
limits in its server-side ML model cache.
Model size estimates currently follow two inconsistent paths:
- Most models use SizeEstimator.estimate(self), which can overcount shared
state reachable through Model.parent, including SparkSession state
(SPARK-57521).
- Some models override estimatedSize with a model-specific calculation.
Several of these include parameter metadata through
Params.estimateMatadataSize, while tree-based models historically counted only
learned
tree nodes.
This makes cache accounting inconsistent across ML model implementations.
### Goal
Provide a consistent, model-owned definition of estimated size:
estimated model size =
parameter metadata
+ learned model state
+ explicitly retained distributed state, where applicable
The estimate must exclude shared or incidental runtime state, especially the
parent estimator and any Spark session/context reachable through it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
