ding-young commented on PR #17163: URL: https://github.com/apache/datafusion/pull/17163#issuecomment-3212798833
> BTW, what's the reason for the remaining `sort_tpch` query failures? There are several different failure points, but here are the main types I’ve observed so far: (1) During `insert_batch`, we hit a memory limit and attempt to call `sort_and_spill_in_mem_batches`, which then fails. (2) During sort, when there have already been previous spills in `insert_batch`, we try to call `sort_and_spill_in_mem_batch` again before merging via `StreamingMergeBuilder`, and it fails. (3) In multi-level merge, when we can no longer reduce the number of spill files or buffer size to fit the memory limit. For (1) and (2), the core issue is that in order to spill in-memory batches, we need to sort them via streaming merge using a loser tree. But since all in-memory batches are already buffered at that point, attempting to merge them all as loser tree inputs further increases memory pressure, leading to failure. I’m currently experimenting with limiting the fanout (merge degree) of the loser tree to see if this can help mitigate the problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
