Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21366
@foxish @liyinan926 @dvogelbacher ok, I think what we have here is closer
to what we want. I primarily addressed the situations outlined by the most
recent comments and observations made about the previous design.
The PR is getting pretty lengthy, so I wouldn't object to a request to
break down the patch into smaller chunks if anyone requests that for ease of
review and legibility.
In [the latest
patch](https://github.com/apache/spark/pull/21366/commits/8615c067328c0c64d0d048922b221477580acdb4)
we preserve the idea of observables and having multiple modules poll the state
of the cluster, but instead of conceiving of state changes as updates to pods,
the modules now always receive a full snapshot of the current state of the
cluster. More formally, the watch pushes snapshots that are just the previous
state with the state of the updated executor applied. The polling, on the
contrary, pushes fresh snapshots.
The aim is to get as close to a level-based paradigm as possible. I think
this gets us closest, but I've left some fringe unlikely edge cases for future
patches.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
