This is an automated email from the ASF dual-hosted git repository. blaginin pushed a commit to branch annarose/dict-coercion in repository https://gitbox.apache.org/repos/asf/datafusion-sandbox.git
commit dab903e66889b64e78734bcf0e99890eaf06ba1e Author: Vignesh <[email protected]> AuthorDate: Tue Feb 3 05:05:47 2026 +0530 Improve memory accounting for ArrowBytesViewMap (#20077) ## Which issue does this PR close? - Closes #20074 ## Rationale for this change ArrowBytesViewMap was previously accounting for the logical number of null bits when reporting memory usage. This under-reported memory consumption is because NullBufferBuilder may allocate more memory than is currently used. Memory accounting in DataFusion is expected to reflect allocated memory rather than logical usage to ensure accurate memory tracking. ## What changes are included in this PR? - Update ArrowBytesViewMap::size to use NullBufferBuilder::allocated_size instead of calculating size from the number of used null bits. ## Are these changes tested? - Yes. Existing tests were run: - cargo test -p datafusion-physical-expr-common ## Are there any user-facing changes? - No. This change only affects internal memory accounting and does not alter query behavior or public APIs. --------- Co-authored-by: Copilot <[email protected]> --- datafusion/physical-expr-common/src/binary_view_map.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/datafusion/physical-expr-common/src/binary_view_map.rs b/datafusion/physical-expr-common/src/binary_view_map.rs index ff93aaf1e..ff0b7c71e 100644 --- a/datafusion/physical-expr-common/src/binary_view_map.rs +++ b/datafusion/physical-expr-common/src/binary_view_map.rs @@ -437,7 +437,7 @@ where let views_size = self.views.len() * size_of::<u128>(); let in_progress_size = self.in_progress.capacity(); let completed_size: usize = self.completed.iter().map(|b| b.len()).sum(); - let nulls_size = self.nulls.len() / 8; + let nulls_size = self.nulls.allocated_size(); self.map_size + views_size --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
