LuciferYang commented on PR #58419: URL: https://github.com/apache/spark/pull/58419#issuecomment-5834526314
On the cached adaptive-plan re-entry question: it cannot overlap, and the barrier is that a cached plan is executed at most once rather than anything `AdaptiveSparkPlanExec` enforces about the flag itself. `_isFinalPlan` is not write-once. `withFinalPlanUpdate` clears it on entry and sets it after the final plan is built, so it would regress if that method were entered twice on one instance. What stops that on the cached path is the call sites: `InMemoryRelation` drives the cached plan only through `SparkPlan.execute` / `executeColumnar`, both `final` and memoized per instance by `LazyTry`, which also caches a failure, and which of the two runs is fixed by an immutable `supportsColumnarInput`. So `doExecute` runs once, the clear-then-set pair happens once, and a planner reading across that window sees not-final then final, never the reverse. The rebuild paths do not reopen it either. `unpersist` and `clearCache` drop the buffers without touching `cachedPlan` or the flag, and re-materializing calls the memoized `execute` again, so the same RDD comes back without re-entering the plan. `recacheByPlan` builds a new `InMemoryRelation` over a new `QueryExecution`, so the fresh not-final flag belongs to a different object; an already-planned `InMemoryTableScanExec` keeps its original relation, builder and plan. A materialization that threw leaves that instance reporting unknown partitioning for good, which is the conservative direction. So the value a prepared union was decided against only sharpens, and `StampUnionDecisions`'s doc now says that, with the reason: a node planned against a cached plan cannot see the no-final-plan window reopen, and a rebuild replaces the relation instead of re-entering that plan. One thing worth stating since the freeze rests on it: the invariant is "a cached plan's AQE is executed at most once", so anything that later drives a cached plan through `finalPhysicalPlan`, `executeCollect`, `executeTake` or `executeTail` would break it. Nothing does today; the only `finalPhysicalPlan` caller is `FileFormatWriter`, on a write plan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
