alamb commented on issue #1221:
URL:
https://github.com/apache/arrow-datafusion/issues/1221#issuecomment-965222832
@jon-chuang sounds like a good start.
I think something else that the scheduler should be able to take advantage
of in the future might be "data locality" -- that is if a plan looks like
```
(plan section 1) -- writes intermediate results --> (plan section 2)
```
It is likely advantageous in may cases to run `section 1` and `section 2` on
the same executor, if possible, to avoid having to send ("reshuffle") the
intermediate results around
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
