Dandandan commented on PR #2308: URL: https://github.com/apache/datafusion-ballista/pull/2308#issuecomment-5308173447
In my experiments today, I was being able to recover most of the performance difference between ballista and datafusion-distributed mostly (after enabling AQE + some other configs, so it can use hash join / broadcast join) by keeping shuffle data in memory (RAM disk). This suggests we should have an option to keep shuffle data in memory, and only spill once it exceeds a budget. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
