Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5116#issuecomment-89094589
@WangTaoTheTonic there seems to be two bigger issues with this patch in its
current state:
(1) *Once a Master fails, it is marked as lost forever.* This means the
following race condition is possible: a Master fails and then recovers, and
then all other Masters fail. In this case, there is still one living Master,
but the code will `System.exit(-1)` anyway.
(2) *There is not a type safe way to determine if a response is from a
backup Master*. The code currently searches for strings like "Can only" in the
response messages to conclude that it is from a backup. This is very brittle;
if someone decides to reword those messages slightly then everything will break.
To solve (1), we can have the client ping all Masters periodically or
whenever a disassociation event is fired. This introduces more complexity and I
would be inclined to leave it out. Instead we should add a huge comment to
explain the potential race condition. Note that this condition is somewhat
unlikely because the submission process is meant to be fire-and-forget instead
of long-running.
To solve (2), we may need to add a field in the response messages to convey
the fact that the response is from a backup. I haven't given the exact
parameter a ton of thought yet, but whatever we introduce here we have to keep
for backward compatibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
