andygrove commented on PR #2038: URL: https://github.com/apache/datafusion-ballista/pull/2038#issuecomment-4973685586
Thanks for the contribution @avantgardnerio. It's going to take me a while to work through this and understand it all. One initial question, though. This PR changes the default execution model from one task per partition to multiple tasks per partition. Rather than making this a default change in Ballista, maybe we could consider a config around this? Something like "max partitions per task"? I am concerned that forcing this change on the [Spark user persona](https://datafusion.apache.org/ballista/contributors-guide/user-personas.html#persona-2-the-spark-user-wanting-the-same-execution-model) will cause confusion. If it is something they can opt into for better performance, that is fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
