pmoust commented on issue #23393: URL: https://github.com/apache/datafusion/issues/23393#issuecomment-6086599381
One more data point for this audit, from `ArrowBytesViewMap::size()` in `datafusion-physical-expr-common` 54.1.0 (`binary_view_map.rs`). We measured it with a heap profile during a string-keyed `GROUP BY` (ClickBench Q35, 20M rows). - **Views:** `size()` (around lines 473-486) counts the views as `views.len() * 16` (around line 474). The vector grows by doubling (`views.push`, around line 451), so its capacity runs 1-2x its length. In our run the views held 89 MB, of which about 22 MB were uncounted on average. - **Hash table:** it is counted through `map_size`. `HashTableAllocExt::insert_accounted` (`datafusion-common/src/utils/proxy.rs`, around lines 158-184) adds `capacity * size_of::<Entry>()` on each growth. hashbrown actually allocates `buckets * (size_of::<Entry>() + 1)`, with buckets at capacity × 8/7, so about 18% more before allocator size-class rounding. In our run the table held 242 MB, about 36 MB of it uncounted. - **Timing:** the reservation is updated only after a batch has been interned (`row_hash.rs`, around lines 1067-1090). A table doubling is therefore allocated before it is charged, while the old table is still live during the rehash. Counting `views.capacity()`, and sizing the table from `allocation_size()` rather than from growth bumps, would close most of this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
