Tweak timeout is tempting as your solution might work most of the time yet fail in certain cases (which others have pointed out). If the goal is absolute correctness then we should avoid timeout, which does not guarantee correctness as it only makes the problem hard to manifest. Fencing is the right solution here - the zxid and also znode cversion can be used as fencing token if you use ZooKeeper. Fencing will guarantee at any single point in time you will have one active leader in action (it does not guarantee that at a single point of time there are multiple parties *think* they are the leader). An alternative solution, depends on your use case, is to instead of requiring a single active leader in action at any time, make your workload idempotent so multiple active leaders don't do any damage.
On Thu, Dec 6, 2018 at 1:05 PM Jordan Zimmerman <jor...@jordanzimmerman.com> wrote: > > Old service leader will detect network partition max 15 seconds after it > > happened. > > If the old service leader is in a very long GC it will not detect the > partition. In the face of VM pauses, etc. it's not possible to avoid 2 > leaders for a short period of time. > > -JZ
