This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-23349-4ca85d20fead97334d337c48360ceaebaff529e8 in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit b790763529ed709ad1b264f434e06a1b7878c947 Author: Saad Tajwar <[email protected]> AuthorDate: Wed Jul 8 18:54:07 2026 -0700 refactor: centralizing shared-allocation accounting for Arc DFHeapSize impls (#23349) ## Which issue does this PR close? - Closes #22867 ## Rationale for this change DFHeapSize depends on a critical invariant: shared heap allocations must be counted once per traversal context. If Arc implementations drift, memory accounting becomes inconsistent and hard to reason about. Consolidating the one-time-accounting logic improves: Correctness durability (single source of truth for Arc dedup behavior) Maintainability (less repeated logic) Reviewability (future changes touch one helper) ## What changes are included in this PR? Helper functions for `Arc` allocation identity and deduplication, namely for providing pointer extraction & `DFHeapSizeCtx` set membership checks, with the `DFHeapSize` implementations on `Arc` backed types using the new helpers ## Are these changes tested? Yes, the below are passing: ``` cargo test -p datafusion-common heap_size --lib cargo test -p datafusion-common --doc heap_size ``` ## Are there any user-facing changes? No --- datafusion/common/src/heap_size.rs | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/datafusion/common/src/heap_size.rs b/datafusion/common/src/heap_size.rs index 405736dbf9..037f807fce 100644 --- a/datafusion/common/src/heap_size.rs +++ b/datafusion/common/src/heap_size.rs @@ -76,6 +76,12 @@ pub struct DFHeapSizeCtx { seen: HashSet<usize>, } +impl DFHeapSizeCtx { + fn count_allocation_once(&mut self, ptr: usize) -> bool { + self.seen.insert(ptr) + } +} + impl DFHeapSize for Statistics { fn heap_size(&self, ctx: &mut DFHeapSizeCtx) -> usize { self.num_rows.heap_size(ctx) @@ -281,11 +287,21 @@ impl<K: DFHeapSize, V: DFHeapSize> DFHeapSize for HashMap<K, V> { } } +fn arc_ptr<T>(arc: &Arc<T>) -> usize { + Arc::as_ptr(arc) as usize +} + +/// For unsized types, `Arc::as_ptr` returns the data address + metadata - we only need the thin address +/// Casting through `*const i32` gets us the thin pointer +fn arc_unsized_ptr<T: ?Sized>(arc: &Arc<T>) -> usize { + Arc::as_ptr(arc) as *const i32 as usize +} + impl<T: DFHeapSize> DFHeapSize for Arc<T> { fn heap_size(&self, ctx: &mut DFHeapSizeCtx) -> usize { - let ptr = Arc::as_ptr(self) as usize; + let ptr = arc_ptr(self); - if !ctx.seen.insert(ptr) { + if !ctx.count_allocation_once(ptr) { return 0; } @@ -296,9 +312,9 @@ impl<T: DFHeapSize> DFHeapSize for Arc<T> { impl DFHeapSize for Arc<str> { fn heap_size(&self, ctx: &mut DFHeapSizeCtx) -> usize { - let ptr = Arc::as_ptr(self) as *const i32 as usize; + let ptr = arc_unsized_ptr(self); - if !ctx.seen.insert(ptr) { + if !ctx.count_allocation_once(ptr) { return 0; } @@ -309,9 +325,9 @@ impl DFHeapSize for Arc<str> { impl DFHeapSize for Arc<dyn DFHeapSize> { fn heap_size(&self, ctx: &mut DFHeapSizeCtx) -> usize { - let ptr = Arc::as_ptr(self) as *const i32 as usize; + let ptr = arc_unsized_ptr(self); - if !ctx.seen.insert(ptr) { + if !ctx.count_allocation_once(ptr) { return 0; } --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
