Github user yingdachen commented on the issue:
https://github.com/apache/tez/pull/33
@jteagles thanks for your input.
Regarding the implication of RM(YARN) supporting gang scheduling, it
largely depends on how application/runtime is implemented, and may not be an
absolute necessity. While it would be best if strong gang-scheduling can be
guaranteed by RM, but even when without, AM can still assist in several ways:
1. AM can schedule all tasks in upstream and downstream vertices
simultaneously, as is implemented in this change
2. AM can gather information about downstream/upstream tasks and inform
dependent tasks to put things into action only when all dependencies are
satisfied. This will require mechanisms such as waiting/timeout/retry to be
implemented in application runtime, but that would be something needed anyways,
even with RM gang-scheduling. Some of the changes along this line would be
covered by TEZ-4000.
---
