Hi Tim First of all: your suggestions to improve the failover sound great to me. Thanks a lot for your effort in this field.
I also highly welcome James' answer that the failover implementation must come along with NFS settings recommendations that work good with the implementation. There are so many blogs etc that describe the failover setup without saying one word about NFS settings that probably everybody thinks it does not matter. Regarding the edge case you described, citation: "The only scenario I can think of ... is if network access is winking in and out rapidly for the different processes, so they get writes in and then immediately lose the lock and someone else gets a write in and immediately loses the lock." I don't think this can happen since the NFS v4 server has a lease period and a lease grace period. When the master broker loses the connection to the persistence store (for example due to network issues), the NFS server reserves the lock for him during the lease grace period. So if the grace period is 45 seconds (default on NetApp), no other broker can get the lock for 45 seconds. If the former master broker connects again before the grace period is over, he regains the lock as if it would have never lost it. My failover tests showed that this really works. If I disconnected the master broker and reconnected it after some seconds, no failover happened. In general my tests showed that the failover to the slave broker works fine in all cases. The problem of the current failover implementation is that the former master stays active for a long time or forever. Therefore, your suggestions to improve the master's checks of its own state can probably improve this. Regards Stephan On Tue, Jun 23, 2015 at 5:07 PM, James A. Robinson <[email protected]> wrote: > Tim, > > I thought this was an interesting read: > > http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html > > Jim > > On Sun, Jun 21, 2015 at 09:45 Tim Bain <[email protected]> wrote: > > > Thanks for the feedback and questions. > > > > I hadn't considered any of the buffering/flushing/synchronization aspects > > of the underlying NFS configuration, but you're absolutely right that the > > guidelines for how to configure this solution need to acknowledge the > > interplay between the two sets of settings and provide guidelines for how > > to select settings for each one in order to achieve the goals. I have no > > experience with configuring NFS, so although I understand the theory that > > writes can be deferred, it would be great to have people who understand > > those aspects be involved when that documentation gets written. > > > > Re: the type of lock to be used in the solution I'm using, I have no idea > > (due to lack of experience with NFS other than as a user) and would > > appreciate suggestions from those who are more knowledgeable about NFS. > I > > had assumed we'd use the same type of lock as is currently being used > (with > > the current behavior in the face of network failures), but if another > type > > would be more appropriate, that would be great to know about. > > > > Re: the master re-reading the data file, I wasn't planning on it but it > > could be done. What I proposed would have a separate thread checking the > > content of the file on the same periodicity (but not necessarily > > synchronized so not necessarily occurring at the same time) as the > writing > > thread. If locking is working correctly and network access is reasonably > > stable, if another process has written to the lock file (such that the > > write thread would see a difference just before it wrote), then the write > > would fail if the other process still held the lock, and it would succeed > > (as you'd want it to) if the other process had lost the lock due to > losing > > network access. So under normal conditions, there wouldn't be a need to > > check content before writing. The only scenario I can think of where it > > might gain you something is if network access is winking in and out > rapidly > > for the different processes, so they get writes in and then immediately > > lose the lock and someone else gets a write in and immediately loses the > > lock. If that's happening, and if the reading thread is timed so that it > > reads the file after the write thread writes to the file and before the > > other processes write to the file during this cycle, then multiple > > processes could think each think they've got the lock long enough to > become > > master. That seems like a very unlikely scenario, but easy enough to > guard > > against by doing the read-before-write that you asked about, so I think > > it's worth doing. > > > > Tim > > > > On Fri, Jun 19, 2015 at 10:19 AM, James A. Robinson <[email protected]> > > wrote: > > > > > On Mon, Jun 15, 2015 at 7:08 AM Tim Bain <[email protected]> > wrote: > > > > > > > > It seems pretty clear that the assumption that acquiring a single > file > > > lock > > > > without doing any further checks will provide thread-safety in all > > cases > > > is > > > > not an accurate one. > > > > > > > > As I see it, here are the failures of the existing approach: > > > > > > > > - Failure #1 is that when an NFS failure occurs, the master broker > > > never > > > > recognizes that an NFS failure has occurred and never recognizes > > that > > > the > > > > slave broker has replaced it as the master. The broker needs to > > > identify > > > > those things even when it has no other reason to write to NFS. > > > > > > > > - Failure #2 is that the slave broker believes that it can > > immediately > > > > become the master. This wouldn't be a problem if the master > broker > > > > instantaneously recognized that it has been supplanted and > > immediately > > > > ceded control, but assuming anything happens instantaneously > > > (especially > > > > across multiple machines) is pretty unrealistic. This means there > > > will be > > > > a period of unresponsiveness when a failover occurs. > > > > > > > > - Failure #3 is that once the master recognizes that it no longer > is > > > the > > > > master, it needs to abort all pending writes (in a way that > > guarantees > > > that > > > > the data files will not be corrupted if NFS returns when some have > > > been > > > > cancelled and others have not). > > > > > > I don't know if this has been called out already, but it will > > > important for users to coordinate how their NFS is configured. For > > > example, activating asynchronous writes on the NFS side would > > > probably make a mess of any assumptions we can make from the client > > > side. I also wonder how buffering might affect how a heartbeat > > > would have to work, whether or not mtime won't get propogated until > > > enough data has been written to cause a flush to disk. > > > > > > > I've got a proposed solution that I think will address all of these > > > > failures, but hopefully others will see ways to improve it. Here's > what > > > > I've got in mind: > > > > > > > > 1. It is no longer sufficient to hold an NFS lock on the DB lock > > file. > > > > In order to maintain master status, you must successfully write to > > the > > > DB > > > > lock file within some time period. If you fail to do so within > that > > > time > > > > period, you must close all use of the DB files and relinquish > master > > > status > > > > to another broker. > > > > > > > > 2. When the master shuts down during normal NFS circumstances, it > > will > > > > delete the DB lock file. If at any point a slave broker sees that > > > there is > > > > no DB lock file or that the DB lock file is so stale that the > master > > > must > > > > have shut down (more on that later), it may immediately attempt to > > > create > > > > one and begin writing to it. If that write succeeds, it is the > > > master. > > > > > > Random curiousity, is this a network lock manager (NLM) based lock? > > > > > > > 3. All brokers should determine whether the current master is > still > > > > alive by checking the current content of the DB lock file against > > the > > > > content read the last time you checked, rather than simply locking > > the > > > file > > > > and assuming that tells you who's got ownership. This means that > the > > > > current master needs to update some content in the DB lock file to > > > make it > > > > unique each on each write; I propose that the content of the file > be > > > the > > > > broker's host, the broker's PID, the current local time on the > > broker > > > at > > > > the time it did its write, and a UUID that will guarantee > uniqueness > > > of the > > > > content from write to write even in the face of time issues. Note > > > that > > > > only the UUID is actually required for this algorithm to work, > but I > > > think > > > > that having the other information will make it easier to > > troubleshoot. > > > > > > That sounds good. > > > > > > > 4. Because time can drift between machines, it is not sufficient > to > > > > compare the write date on the DB lock file with your host's > current > > > time > > > > when determining that a file is stale; you must successfully read > > the > > > file > > > > repeatedly over a time period and receive the same value each time > > in > > > order > > > > to decide that the DB lock file is stale. > > > > > > > > 5. The master should use the same approach as the slaves to > > determine > > > if > > > > it's still in control, by checking for changes to the content of > the > > > DB > > > > lock file. This means the master needs to positively confirm that > > each > > > > periodic write to the DB lock file succeeded by reading it back > (in > > a > > > > separate thread, using a timeout on the read operation to identify > > > > situations where NFS doesn't respond), rather than simply assuming > > > that its > > > > call to write() worked successfully. > > > > > > > > 6. When a slave determines that the master has failed to write to > > the > > > DB > > > > lock file for longer than the timeout, it attempts to acquire the > > > write > > > > lock on the DB lock file and write to it to become the master. If > > it > > > > succeeds (because the master has lost NFS connectivity but the > slave > > > has > > > > not), it is the master. If it fails because another slave > acquired > > > the > > > > lock first or because the slave has lost NFS connectivity, it goes > > > back to > > > > monitoring for the master to fail to write to the DB lock file. > > > > > > > > > Would the master re-read the data file again just before the next > > > write as a guard against some other process somehow snagging the > > > lock out from under it? > > > > > > Jim > > > > > >
