kosiew opened a new pull request, #26487: URL: https://github.com/apache/datafusion/pull/26487
## Which issue does this PR close? - Part of #23393 ## Rationale for this change Symmetric hash joins use memory reservations to track retained execution state. Previously, `ExprIntervalGraph::size()` estimated memory using live node and edge counts, which could underreport memory retained by the graph after nodes or edges were removed. It also omitted heap allocations owned by interval bounds. This change improves reservation accuracy by accounting for retained graph backing capacity and interval-owned heap allocations, while excluding shared physical-expression allocations. ## What changes are included in this PR? - Replace live node and edge count estimates with capacity-based accounting using the underlying `petgraph::StableGraph` node and edge storage types. - Include heap allocations owned by interval bounds using `DFHeapSize` and a shared `DFHeapSizeCtx` to avoid double-counting shared allocations. - Exclude shared `Arc<dyn PhysicalExpr>` pointees from graph-owned memory accounting. - Add regression coverage verifying that `SymmetricHashJoinStream` reservations reflect the graph's reported size after construction, interval updates, and removal. ## Are these changes tested? Three regression tests are added: - `graph_size_counts_retained_capacity`: Verifies empty and grown graph accounting, including retained capacity after node removal and graph clearing. - `graph_size_counts_interval_heaps_not_expression_pointees`: Verifies UTF-8 interval heap accounting, deduplication of shared timestamp timezone allocations, and exclusion of shared physical-expression pointees. - `stream_reserves_retained_interval_graph`: Verifies that stream reservations and memory metrics account for the retained interval graph and are updated when the graph changes or is removed. The tests use deterministic capacity and size assertions without relying on global allocator measurements. ## Are there any user-facing changes? No changes to query results or public APIs are intended. Memory reservations for symmetric hash joins may now more accurately reflect retained interval graph allocations, potentially affecting memory-limit enforcement. No documentation changes are expected. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
