zhuqi-lucas commented on PR #25098: URL: https://github.com/apache/datafusion/pull/25098#issuecomment-5611697699
Thanks @asolimando — done in this PR. `EnsureRequirements` now overrides `optimize_with_context` and builds the shared `StatisticsContext` from `context.statistics_registry()` (following your #24716), so the rule both shares the memoization cache across the pass and consults registered providers; `optimize` delegates via `ConfigOnlyContext`. Default behavior is unchanged (an empty registry builds a plain context). Added `ensure_distribution_uses_context_statistics_registry`, which shows a scan whose tiny real stats would not warrant a round-robin repartition gets one once a registry provider reports it as large — i.e. the registry is actually threaded through the distribution pass. On the cache reset: I agree it is a footgun to leave to consumers. Of your two ideas I would lean toward **(b) Arc-cloning nodes into the cache** as the near-term fix — it is bounded by the plan size for the pass, removes the stale-pointer (ABA) hazard, and lets consumers drop the reset entirely; **(a) a unique per-`ExecutionPlan` id** is cleaner long-term but touches every plan node, so probably its own change. Happy to help with either whenever you are back. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
