sunchao commented on code in PR #57443: URL: https://github.com/apache/spark/pull/57443#discussion_r3693700576
########## sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala: ########## @@ -500,6 +654,13 @@ case class InMemoryRelation( def cachedPlan: SparkPlan = cacheBuilder.cachedPlan + override def statsAvailable: Boolean = + cacheBuilder.storageLevel.useDisk && cacheBuilder.repeatableMaterializedStats.isDefined Review Comment: Thanks @cloud-fan — addressed in 9fafde74efd. `MaterializedLeafMetadata` now captures row count, byte size, durability, and repeatability from one synchronized cache-generation snapshot, and `InjectRuntimeFilter` reads that snapshot once per eligible cache. A cheap precheck skips memory-only and non-repeatable caches without folding their partition statistics. The selected creation-side size is reused by both threshold checks while preserving projected-plan size estimates. I also added coverage for cold/partially materialized caches, memory-only caches, projected thresholds, cache rebuilds, and partition recomputation. All 179 focused tests and the Catalyst/SQL Scala style checks pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
