Dandandan commented on PR #2308: URL: https://github.com/apache/datafusion-ballista/pull/2308#issuecomment-5303653540
Looks good in general. I think there is quite some room to push the shuffle model further to make it equal / favour the shuffle model in some cases (also for performance reasons). Some smaller/bigger ideas: * In memory shuffle. Currently it writes it to disk always and reads it back. (Streaming avoids this cost but at some other cost). * Reduce the number of files by coalescing partitions, cross partition batch coalescing * Avoid metadata write / read, optimize shuffle format * Shuffle affinity (route tasks to where shuffle data lives) * Better parallelization during write/read * Overlap multiple stages, instead of always waiting on full stage * Bubble execution * Make better use of output statistics etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
