villebro commented on issue #2843:
URL: 
https://github.com/apache/datafusion-ballista/issues/2843#issuecomment-6883632937

   @milenkovicm I really like this, and I feel auto would be a great default 
execution mode, as it's probably what fits the expectations and preferences 
typical users have. My quick thoughts:
   - I assume the decision would need to be made from the logical plan as 
opposed to the physical plan, as currently the physical plan depends on the 
mode.
   - Routing Copy and DML to staged execution makes sense.
   - The trait can't only take logical plans, as clients can also submit 
physical plans directly. So when a logical plan is available, use that, 
otherwise pass in the physical plan.
   - If auto becomes the default, it should fall back to staged on pull-staged 
schedulers, and we should document that pipelined jobs skip AQE.
   
   Beyond this I feel it would be a great idea to support multiple strategies 
through some form of registry, so we could add more strategies in the future 
(both in vanilla Ballista, but also on custom crates). That way one wouldn't 
have to settle for a single strategy, but could have many, and set one either 
at startup as the default, or per session or query through config. I'm planning 
on doing this for scheduler HA strategies, and I feel this type of 
extensibility would be valuable in all areas of Ballista.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to