dwsmith1983 opened a new issue, #7197: URL: https://github.com/apache/datafusion-comet/issues/7197
Under `fair_unified`, the pool limits each consumer that can spill to `(task_share - unspillable) / num_spill` (#7096). When consumers that cannot spill, such as the build side of a shuffled hash join or a sort-merge join's buffered rows, hold the task's whole share, that limit is zero. A sort in the same task is then refused on its first request. DataFusion 55.2's `ExternalSorter` does not spill in that case. It returns "Not enough memory to continue external sort" when it cannot reserve a batch while `in_mem_batches` is empty, when the retry after a spill fails, and when the `try_resize` in `sort_batch_stream` fails. So the query fails, although Spark would still grant the task memory: Spark lets a task hold more than `off_heap_size / running_tasks` while other tasks have not asked for memory yet. The same paths fail when one batch is larger than the sort's share, for example with wide rows, many cores, little off-heap memory and several consumers that can spill in one task. `a_consumer_that_can_spill_is_refused_once_the_others_hold_the_tasks_share` in `fair_pool.rs` pins the shape. The contributor guide, the tuning guide and the 1.2.0 migration note list the workarounds: a larger `spark.memory.offHeap.size`, a smaller `spark.comet.batchSize`, or `spark.comet.exec.memoryPool=greedy_unified`. Options include letting a consumer that holds nothing take its first batch from Spark, or falling back to Spark's grant when the limit is below one batch. Either changes behavior that the #7096 benchmarks did not measure, so it needs its own measurement. Found in #7096 (https://github.com/apache/datafusion-comet/pull/7096#discussion_r4321652975). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
