comphead commented on code in PR #26429:
URL: https://github.com/apache/datafusion/pull/26429#discussion_r4590357031
##########
datafusion/common/src/utils/memory.rs:
##########
@@ -119,6 +120,40 @@ pub fn estimate_memory_size<T>(num_elements: usize,
fixed_size: usize) -> Result
})
}
+/// Estimates the size in bytes of the table that a hashbrown [`HashTable`]
+/// allocates to make room for `additional` more entries, or returns 0 if they
+/// fit in its capacity.
+///
+/// A table that grows allocates the larger table before it moves the entries
+/// and frees the current one, so for a moment it holds both. Memory accounting
+/// that measures the table before or after inserting misses the larger table,
+/// which reserving this size before inserting covers.
+///
+/// Returns `usize::MAX` if the estimate overflows.
+///
+/// # Example
+///
+/// ```rust
+/// # use datafusion_common::utils::memory::estimate_hash_table_growth;
+/// # use hashbrown::hash_table::HashTable;
+/// let table: HashTable<u64> = HashTable::with_capacity(100);
+/// let capacity = table.capacity();
+/// // The table holds `capacity` entries without growing
+/// assert_eq!(estimate_hash_table_growth(&table, capacity), 0);
+/// // One more entry makes it allocate a larger table
+/// assert!(estimate_hash_table_growth(&table, capacity + 1) >
table.allocation_size());
+/// ```
+pub fn estimate_hash_table_growth<T>(table: &HashTable<T>, additional: usize)
-> usize {
+ let required = table.len().saturating_add(additional);
+ if required <= table.capacity() {
+ return 0;
+ }
+ // A full table grows to hold at least one more entry than its capacity,
+ // which doubles its buckets
+ let capacity = required.max(table.capacity().saturating_add(1));
+ estimate_memory_size::<T>(capacity, 0).unwrap_or(usize::MAX)
Review Comment:
Still applies after the rebase. `test_estimate_hash_table_growth`
(`memory.rs:879`) compares the estimate with `table.allocation_size()` after
the inserts, which is the final table, so it passes while the reservation
misses the peak. Recording `before + after` whenever `allocation_size()`
changes inside the insert loop, and asserting that the current table plus the
estimate covers the largest such sum, would pin the property the reservation is
for.
##########
datafusion/physical-plan/src/aggregates/hash_stream.rs:
##########
@@ -757,6 +778,28 @@ impl FinalHashAggregateStream {
Ok(())
}
+ /// Sorts and spills one complete in-memory state run: the groups
+ /// aggregated so far.
+ fn spill(
Review Comment:
One more reason for a single helper. The two copies now handle errors
differently. `FinalHashAggregateStream::spill` (`hash_stream.rs:773`) resizes
the reservation before returning a spill error, while `Aggregating::spill`
(`single_stream.rs:322`) returns on the spill error first and leaves the
reservation at the pre-spill size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]