zhengruifeng commented on code in PR #56584:
URL: https://github.com/apache/spark/pull/56584#discussion_r3599692696
##########
mllib/src/main/scala/org/apache/spark/ml/Model.scala:
##########
@@ -57,5 +57,19 @@ abstract class Model[M <: Model[M]] extends Transformer {
self =>
* 4, For 3-rd extension, if external languages are used, it is recommended
to override
* this method and return a proper size.
*/
- private[spark] def estimatedSize: Long = SizeEstimator.estimate(self)
+ private[spark] def estimatedSize: Long = {
+ // SPARK-57521: Temporarily clear the parent reference during size
estimation.
+ // After fit(), the parent estimator may retain indirect references to the
SparkSession
+ // (via closures or query plan state from DataFrame operations executed
during fit).
+ // SizeEstimator traverses the entire reachable object graph, causing it
to count
+ // shared SparkSession state as part of every model's size.
+ // The parent is @transient (not persisted) and is not needed for
transform() or save().
+ val savedParent = parent
+ parent = null
Review Comment:
@cloud-fan codex told me that there isn't usage of estimating a spark
session by `SizeEstimator`, can we just set its size 0 via
`KnownSizeEstimation`?
##########
mllib/src/main/scala/org/apache/spark/ml/Model.scala:
##########
@@ -57,5 +57,19 @@ abstract class Model[M <: Model[M]] extends Transformer {
self =>
* 4, For 3-rd extension, if external languages are used, it is recommended
to override
* this method and return a proper size.
*/
- private[spark] def estimatedSize: Long = SizeEstimator.estimate(self)
+ private[spark] def estimatedSize: Long = {
+ // SPARK-57521: Temporarily clear the parent reference during size
estimation.
+ // After fit(), the parent estimator may retain indirect references to the
SparkSession
+ // (via closures or query plan state from DataFrame operations executed
during fit).
+ // SizeEstimator traverses the entire reachable object graph, causing it
to count
+ // shared SparkSession state as part of every model's size.
+ // The parent is @transient (not persisted) and is not needed for
transform() or save().
+ val savedParent = parent
+ parent = null
Review Comment:
@cloud-fan codex told me that there isn't any usage of estimating a spark
session by `SizeEstimator`, can we just set its size 0 via
`KnownSizeEstimation`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
