pmoust commented on issue #24704: URL: https://github.com/apache/datafusion/issues/24704#issuecomment-6112826344
Data point from a production-shaped run at NOFire (Ravel, a telemetry database on DataFusion), in case it helps size the fix. ClickBench Q29 (`regexp_replace(Referer) AS k, AVG(length(Referer)), COUNT(*), MIN(Referer) ... GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25`) over the 100M-row hits set, read from a custom `TableProvider` whose scan yields Utf8View columns, with spilling disabled and a 9.9 GB `FairSpillPool`-style query pool: - 54.1.0: completes, pool peak 1.62 to 1.65 GB at 32 target partitions (two boxes, four runs). - 55.2.0: `ResourcesExhausted` at every pool size tried (9.9, 16.1, 26 GB). Process RSS stays at 12 GB throughout. With `target_partitions = 1` it completes at a 1.20 GB peak. Peaks at 2, 4 and 8 partitions are 3,908,874,628 / 4,886,093,285, 5,863,311,942 and 9,780,706,250 bytes: within 0.1 % of integer multiples (4x, 5x, 6x, 10x) of 977,218,657 bytes, which is one partition's materialized aggregate output. Reading the source, this is the `MaterializedOutput::next_batch` slices in `aggregate_hash_table/common.rs` meeting `RepartitionExec::send`'s `batch.get_array_memory_size()` reservation: every buffered slice is charged the whole output's buffer capacity, and the refusal arrives as a spill request from `RepartitionExec`. `SortExec` and `HashJoinExec` do not show it, because they charge through `get_record_batch_memory_size` and the counted-buffer memory counter, which dedupe shared buffers; `NestedLoopJoinExec` and `CrossJoinExec` charge the build side the same way `RepartitionExec` does and are affected. Our workaround is a physical optimizer rule that copies each sliced batch into right-sized buffers between the aggregate and the round-robin repartition (NOFireAI/ravel#2774, `crates/ravel-sql/src/compact_slices.rs`). With it the same query completes at a 1.63 to 2.08 GB peak on 55.2.0. The copy is not free, so a native fix that charges a slice by its referenced bytes in the exchange (or emits owned batches from the aggregate) would be preferable to carrying it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
