Dandandan commented on issue #2321: URL: https://github.com/apache/datafusion-ballista/issues/2321#issuecomment-5312698577
I think I found the issue in SPM not propagating the limit. Some longer term directions of improvement: * Currently `SortPreservingMergeExec` can only use ~5.6 GB because it's planned as a single slot (even though it usually will be the only task running). We can make it consume multiple slots (e.g. 8) based on the fan-in / input partitions and therefore it could use the full 44.8 GB * Reducing fan-in of `SortPreservingMergeExec`, although I think in this case there is not much improvement (I don't know how much partitions the bench is uses, fewer partitions will help here) * `ballista.shuffle.sort_based.batch_size` might be a lever to reduce memory size * Some workarounds could be: multi level SPM, using SortExec (which supports spill),... * Distributed sort: I think this might be the most promising for unbounded sort, as otherwise it will always be bound by a single node -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
