comphead commented on issue #9359: URL: https://github.com/apache/arrow-datafusion/issues/9359#issuecomment-2041551566
I think I managed to find the test query. First observation is SMJ using external sorter, but the spill doesn't work, will dig into it. ``` Resources exhausted: Failed to allocate additional 1048576 bytes for ExternalSorterMerge[0] with 0 bytes already allocated - maximum available is 1043520 ``` This gives me a suggestion ExternalSorter used but spill is not used Playing with sort params I managed to make SMJ merge phase complain on mem exhaustion ``` ./datafusion-cli/target/debug/datafusion-cli -m 1m DataFusion CLI v37.0.0 ❯ set datafusion.optimizer.prefer_hash_join = false; 0 row(s) fetched. Elapsed 0.003 seconds. ❯ set datafusion.execution.sort_in_place_threshold_bytes = 104857; 0 row(s) fetched. Elapsed 0.001 seconds. ❯ set datafusion.execution.sort_spill_reservation_bytes = 104857; 0 row(s) fetched. Elapsed 0.000 seconds. ❯ select * from (select unnest(range(0, 10000)) id) t inner join (select unnest(range(0, 100000)) id) t1 on t.id = t1.id; Resources exhausted: Failed to allocate additional 165208 bytes for SMJStream[5] with 0 bytes already allocated - maximum available is 65456 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
