zhengruifeng commented on PR #56584: URL: https://github.com/apache/spark/pull/56584#issuecomment-5031414259
### DBR serverless RCA I added per-object `SizeEstimator` traversal logging in the runtime debug PR: https://github.com/databricks-eng/runtime/pull/239556 I ran the minimal repro: ```python from pyspark.ml.feature import StringIndexer from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() df = spark.createDataFrame([("a",), ("b",), ("c",)], ["label"]) model = StringIndexer(inputCol="label", outputCol="idx").fit(df) spark.stop() ``` #### Findings * The same `StringIndexerModel` was estimated twice. Its shallow-object entry is only **72 bytes**: ```text StringIndexerModel@25f72e48 with 72 bytes ``` This is not the total model size; each trace line reports only the size added while visiting that one object. * Summing the per-object contributions for one complete traversal gives **33,173,464 bytes (31.64 MiB)** across **383,749 unique objects** for a model with only the labels `a`, `b`, and `c`. * The beginning of the trace shows the problematic graph: ```text StringIndexerModel -> ParamMap / default ParamMap -> parent StringIndexer -> Log4jLogger / Log4j configuration -> ... SparkSession / SparkContext and associated runtime state ``` `StringIndexer.fit` creates the model with `.setParent(this)`. The parent estimator has an initialized logger, and traversal of that logger reaches large global Log4j maps plus the Spark runtime graph. * `SparkSession` itself logs only **136 bytes** because the debug line is shallow. Its referenced state is accounted for by separate descendant entries. The largest individual entries include a **1,463,184-byte** `ConcurrentHashMap$Node[]`, a **1,255,560-byte** `Object[]`, and a **698,080-byte** `byte[]`. This confirms the proposed fix: `Model.estimatedSize` should estimate a parentless copy (or otherwise exclude `parent`) so cache accounting includes model metadata and learned state, but not the shared estimator/logger/session graph. The DBR trace is 31.64 MiB; the overcount will scale with the size of the initialized session/runtime state. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
