Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/328#issuecomment-71133987
I am not sure that the infinite number of tries is actually bad. This sort
of depends on the situation, I guess:
- On YARN, it may make sense, because the node will then go back into the
pool of available resource
- On standalone, it will anyways be there for Flink, so the TaskManager
might as well keep trying to offer itself for work. Think of a network
partitioning event - after the partitions re-joined, the cluster should work as
a whole again.
How about the following: We have a config parameter how long nodes should
attempt to register. YARN could set a timeout (say 2-5 minutes), while by
default, the timeout is infinite.
Concerning the attempt pause: Having attempts with exponential backoff (and
a cap) is the common thing (and I think it was the default before). Start with
a 50ms pause and double it each attempt and cap it at 1 or 2 minutes or so. If
you miss early attempts, the pause will not be long. If you missed an all
attempts within the first second, you are guaranteed to not wait more than
twice as long as you already waited anyways.
For the sake of transparency and making sure that the states are actually
in sync: How about we have three response messages for the registration attempt:
1. Refused (for whatever reason, the message should have a string that
the TM can log)
2. Accepted (with the assigned ID)
3. Already registered (with the assigned ID) - The current logic handles
this correctly as well, but this will allow us to log better at the TaskManager
and debug problems there much better. Since this is a mechanism which may have
weird cornercase behavior, it would be good to know as much about what was
happening as possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
