Re: [HACKERS] Improving backend startup interlock
Tom Lane writes: $ man flock No manual entry for flock. $ HPUX has generally taken the position of adopting both BSD and SysV features, so if it doesn't exist here, it's not portable to older Unixen ... If only local locking is at issue then finding any one of fcntl() locking, flock(), or lockf() would do. All Unixen will have one or more of these and autoconf machinery exists to find them. The issue Tom raised about NFS support remains: locking over NFS introduces new failure modes. It also only works for NFS clients that support NFS locking, which not all do. Mind you NFS users are currently entirely unprotected from someone starting a postmaster on a different NFS client using the same data directory right now, which file locking would prevent. So there is some win for NFS users as well as local filesystem users. (Anyone using NFS care to put their hand up? Maybe nobody does?) Is the benefit of better local filesystem behaviour plus multiple client protection for NFS users who have file locking enough to outweigh the drawbacks? My two cents says it is, but my two cents are worth approximately USD$0.01, which is to say not very much ... Regards, Giles ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Improving backend startup interlock
Giles Lean [EMAIL PROTECTED] wrote: Tom Lane writes: $ man flock No manual entry for flock. $ HPUX has generally taken the position of adopting both BSD and SysV features, so if it doesn't exist here, it's not portable to older Unixen ... If only local locking is at issue then finding any one of fcntl() locking, flock(), or lockf() would do. All Unixen will have one or more of these and autoconf machinery exists to find them. The issue Tom raised about NFS support remains: locking over NFS introduces new failure modes. It also only works for NFS clients that support NFS locking, which not all do. Mind you NFS users are currently entirely unprotected from someone starting a postmaster on a different NFS client using the same data directory right now, which file locking would prevent. So there is some win for NFS users as well as local filesystem users. (Anyone using NFS care to put their hand up? Maybe nobody does?) Is the benefit of better local filesystem behaviour plus multiple client protection for NFS users who have file locking enough to outweigh the drawbacks? My two cents says it is, but my two cents are worth approximately USD$0.01, which is to say not very much ... Well, I am going to do some tests with postgresql and our netapp filer later in October. If that setup proves to work fast and reliable I would also be interested in such a locking. I don't care about the feature if I find the postgresql/NFS/netapp-filer setup to be unreliable or bad performing. I'll see. Regards, Michael Paesold ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Improving backend startup interlock
Michael Paesold wrote: Giles Lean [EMAIL PROTECTED] wrote: Mind you NFS users are currently entirely unprotected from someone starting a postmaster on a different NFS client using the same data directory right now, which file locking would prevent. So there is some win for NFS users as well as local filesystem users. (Anyone using NFS care to put their hand up? Maybe nobody does?) Is the benefit of better local filesystem behaviour plus multiple client protection for NFS users who have file locking enough to outweigh the drawbacks? My two cents says it is, but my two cents are worth approximately USD$0.01, which is to say not very much ... Well, I am going to do some tests with postgresql and our netapp filer later in October. If that setup proves to work fast and reliable I would also be interested in such a locking. I don't care about the feature if I find the postgresql/NFS/netapp-filer setup to be unreliable or bad performing. We have multiple Oracle databases running over NFS from an HPUX server to a netapp and have been pleased with the performance overall. It does require some tuning to get it right, and it hasn't been entirely without issues, but I don't see us going back to local storage. We also just recently set up a Linux box running Oracle against an NFS mounted netapp. Soon I'll be adding Postgres on the same machine, initially using locally attached storage, but at some point I may need to shift to the netapp due to data volume. If you do try Postgres on the netapp, please post your results/experience and I'll do the same. Anyway, I guess I qualify as interested in an NFS safe locking method. Joe ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Improving backend startup interlock
Have people considered flock (advisory locking) on the postmaster.pid file for backend detection? It has a nonblocking option. Don't most OS's support it? I can't understand why we can't get an easier solution to postmaster detection than shared memory. --- Tom Lane wrote: Giles Lean [EMAIL PROTECTED] writes: I'm certainly no fan of NFS locking, but if someone trusts their NFS client and server implementations enough to put their data on, they might as well trust it to get a single lock file for startup right too. IMHO. Your mileage may vary. Well, my local man page for lockf() sez The advisory record-locking capabilities of lockf() are implemented throughout the network by the ``network lock daemon'' (see lockd(1M)). If the file server crashes and is rebooted, the lock daemon attempts to recover all locks associated with the crashed server. If a lock cannot be reclaimed, the process that held the lock is issued a SIGLOST signal. and the lockd man page mentions that not only lockd but statd have to be running locally *and* at the NFS server. This sure sounds like file locking on NFS introduces additional failure modes above and beyond what we have already. Since the entire point of this locking exercise is to improve PG's robustness, solutions that depend on other daemons not crashing don't sound like a step forward to me. I'm willing to trust the local kernel, but I get antsy if I have to trust more than that. regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Improving backend startup interlock
Bruce Momjian [EMAIL PROTECTED] writes: Have people considered flock (advisory locking) on the postmaster.pid file for backend detection? $ man flock No manual entry for flock. $ HPUX has generally taken the position of adopting both BSD and SysV features, so if it doesn't exist here, it's not portable to older Unixen ... regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] Improving backend startup interlock
I have the beginnings of an idea about improving our interlock logic for postmaster startup. The existing method is pretty good, but we have had multiple reports that it can fail during system boot if the old postmaster wasn't given a chance to shut down cleanly: there's a fair-sized chance that the old postmaster PID will have been assigned to some other process, and that fools the interlock check. I think we can improve matters by combining the existing checks for old-postmaster-PID and old-shared-memory-segment into one cohesive entity. To do this, we must abandon the existing special case for private memory when running a bootstrap or standalone backend. Even a standalone backend will be required to get a shmem segment just like a postmaster would. This ensures that we can use both parts of the safety check, even when the old holder of the data directory interlock was a standalone backend. Here's a sketch of the improved startup procedure: 1. Try to open and read the $PGDATA/postmaster.pid file. If we fail because it's not there, okay to continue, because old postmaster must have shut down cleanly; skip to step 8. If we fail for any other reason (eg, permissions failure), complain and abort startup. (Because we write the postmaster.pid file mode 600, getting past this step guarantees we are either the same UID as the old postmaster or root; else we'd have failed to read the old file. This fact justifies some assumptions below.) 2. Extract old postmaster PID and old shared memory key from file. (Both will now always be there, per above; abort if file contents are not as expected.) We do not bother with trying kill(PID, 0) anymore, because it doesn't prove anything. 3. Try to attach to the old shared memory segment using the old key. There are three possible outcomes: A: fail because it's not there. Then we know the old postmaster (or standalone backend) is gone, and so are all its children. Okay to skip to step 7. B: fail for some other reason, eg permissions violation. Because we know we are the same UID (or root) as before, this must indicate that the old shmem segment actually belongs to someone else; so we have a chance collision with someone else's shmem key. Ignore the shmem segment, skip to step 7. (In short, we can treat all failures alike, which is a Good Thing.) C: attach succeeds. Continue to step 4. 4. Examine header of old shmem segment to see if it contains the right magic number *and* old postmaster PID. If not, it isn't really a Postgres shmem segment, so ignore it; detach and skip to step 7. 5. If old shmem segment still has other processes attached to it, abort: these must be an old postmaster and/or old backends still alive. (We can check nattach 1 in the SysV case, or just assume they are there in the hugepages-segment case that Neil wants to add.) 6. Detach from and delete the old shmem segment. (Deletion isn't strictly necessary, but we should do it to avoid sucking resources.) 7. Delete the old postmaster.pid file. If this fails for any reason, abort. (Either we've got permissions problems or a race condition with someone else trying to start up.) 8. Create a shared memory segment. 9. Create a new postmaster.pid file and record my PID and segment key. If we fail to do this (with O_EXCL create), abort; someone else must be trying to start up at the same time. Be careful to create the lockfile mode 600, per notes above. This is not quite ready for prime time yet, because it's not very bulletproof against the scenario where two would-be postmasters are starting concurrently. The first one might get all the way through the sequence before the second one arrives at step 7 --- in which case the second one will be deleting the first one's lockfile. Oops. A possible answer is to create a second lockfile that only exists for the duration of the startup sequence, and use that to ensure that only one process is trying this sequence at a time. This reintroduces the same problem we're trying to get away from (must rely on kill(PID, 0) to determine validity of the lock file), but at least the window of vulnerability is much smaller than before. Does anyone see a better way? A more general objection is that this approach will hardwire, even more solidly than before, the assumption that we are using a shared-memory API that provides identifiable shmem segments (ie, something we can record a key for and later try to attach to). I think some people wanted to get away from that. But so far I've not seen any proposal for an alternative startup interlock that doesn't require attachable shared memory. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Improving backend startup interlock
Tom Lane wrote: [ discussion of new startup interlock ] This is not quite ready for prime time yet, because it's not very bulletproof against the scenario where two would-be postmasters are starting concurrently. A solution to this is to require would-be postmasters to obtain an exclusive lock on a lock file before touching the pid file. (The lock file perhaps could be the pid file, but it doesn't have to be.) Is there some reason that file locking is not acceptable? Is there any platform or filesystem supported for use with PostgreSQL which doesn't have working exclusive file locking? A possible answer is to create a second lockfile that only exists for the duration of the startup sequence, and use that to ensure that only one process is trying this sequence at a time. ... This reintroduces the same problem we're trying to get away from (must rely on kill(PID, 0) to determine validity of the lock file), but at least the window of vulnerability is much smaller than before. A lock file locked for the whole time the postmaster is running can be responsible for preventing multiple postmasters running without relying on pids. All that is needed is that the OS drop exclusive file locks on process exit and that locks not survive across reboots. The checks of the shared memory segment (number of attachements etc) look after orphaned back end processes, per the proposal. Regards, Giles ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Improving backend startup interlock
Giles Lean [EMAIL PROTECTED] writes: Is there some reason that file locking is not acceptable? Is there any platform or filesystem supported for use with PostgreSQL which doesn't have working exclusive file locking? How would we know? We have never tried to use such a feature. For sure I would not trust it on an NFS filesystem. (Although we disparage running an NFS-mounted database, people do it anyway.) regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Improving backend startup interlock
Tom Lane wrote: Giles Lean [EMAIL PROTECTED] writes: Is there some reason that file locking is not acceptable? Is there any platform or filesystem supported for use with PostgreSQL which doesn't have working exclusive file locking? How would we know? We have never tried to use such a feature. I asked because I've not been following this project long enough to know if it had been tried and rejected previously. Newcomers being prone to making silly suggestions and all that. :-) For sure I would not trust it on an NFS filesystem. (Although we disparage running an NFS-mounted database, people do it anyway.) scratches head I can't work out if that's an objection or not. I'm certainly no fan of NFS locking, but if someone trusts their NFS client and server implementations enough to put their data on, they might as well trust it to get a single lock file for startup right too. IMHO. Your mileage may vary. Regards, Giles ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
