zhengruifeng commented on PR #56584: URL: https://github.com/apache/spark/pull/56584#issuecomment-4998699802
I took a closer look at the `StringIndexer` path. The current test demonstrates that traversing `model.parent` materially increases the estimate, but it does not identify the object path responsible for that increase. `StringIndexer.fit` stores only the direct model-to-estimator edge: ```scala copyValues(new StringIndexerModel(uid, labelsArray).setParent(this)) ``` The input `Dataset`, `selectedCols`, and intermediate DataFrames in `sortByFreq` / `sortByAlphabet` are local variables, not fields on `StringIndexer`. I could not find a source-level path from `StringIndexer` to `SparkSession` through retained DataFrame/query state. One concrete retained side effect is logging: `fit` calls `transformSchema(..., logging = true)`, which initializes the inherited transient `Logging.log_` field. `SizeEstimator` still traverses transient fields, so the parent can pull in a logging/runtime graph that is unrelated to learned labels. Whether that graph reaches `SparkSession` needs a runtime object-path trace; it is not established by the source or the current regression test. I suggest describing the confirmed issue as overcounting arbitrary estimator-owned state reachable through `Model.parent`. If the intent is specifically to exclude a session, it would be useful to capture a concrete path such as `model.parent -> ... -> SparkSession` from the reproducer. This distinction also matters for the long-term design: some models intentionally retain DataFrames or sessions, while a `StringIndexerModel` should not be charged for incidental estimator state. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
