Great job! Thank you On Jan 21, 2016 6:34 AM, "Markus Mayer" <[email protected]> wrote:
> There are several bug reports out there that squid is failing to start up > on OS X with an error similar to this: > > Ipc::Mem::Segment::create failed to ftruncate(/squid-cf__metadata.shm): > (22) Invalid argument > > It may be one of the other shared memory segments that is failing, but the > problem is always the same: upon restarting squid, ftruncate() bails with > EINVAL and squid subsequently aborts. If one is lucky and waits long > enough, restarting squid may eventually succeed. Or one may need to reboot. > > From what I was able to find, the root cause for this has never really > been determined, and there hasn't been a fix in squid that addresses this. > (It was still happening to me intermittently on OS X 10.11.2 and the most > recent squid from GIT.) > > So, I took some time to dig into this, down to first looking at the Darwin > kernel sources ( > http://www.opensource.apple.com/source/xnu/xnu-3248.20.55/bsd/kern/posix_shm.c) > and subsequently downloading XNU, instrumenting posix_shm.c and running my > own custom built Darwin kernel (which was surprisingly straight forward). > Since there are several places where ftruncate() and, specifically, > pshm_truncate() can return EINVAL, instrumenting the kernel seemed to be > the most promising way of figuring out which EINVAL squid is hitting and > why. > > Turns out it's this one: > > if ((pinfo->pshm_flags & (PSHM_DEFINED|PSHM_ALLOCATING|PSHM_ALLOCATED)) > != PSHM_DEFINED) { > PSHM_SUBSYS_UNLOCK(); > return(EINVAL); > } > > It happens because PSHM_ALLOCATED is already set. Luckily, the only place > where PSHM_ALLOCATED gets set is at the end of pshm_truncate() itself. So, > finding the culprit for setting it was easy. > > What this means is that OS X will only support ftruncate() *ONCE* on a shm > segment, and this is done fully on purpose. The first time pshm_truncate() > is called, it'll do all the required mapping and so forth, then set > PSHM_ALLOCATED for that segment. When one tries to call ftruncate() on the > same shm region again, it'll return EINVAL, because PSHM_ALLOCATED is set. > > There are cases where shm regions don't get removed when squid terminates. > So, upon restarting squid, shm_open() will return the already existing shm > segment (or segments) which will have been "allocated" already. squid will > try to call ftruncate() anyway -- and fail. > > Now, one could argue that OS X should really be able to handle ftruncate() > on an existing shm segment (like other POSIX systems do), and it probably > should. Convincing Apple of that is likely not going to be easy. Even if it > were to succeed, it would only fix future releases, not the current OS X > releases. > > So, I see the following solutions squid could implement, which could be > Darwin-only code, so as to not burden other OSes with this workaround. > > - Try to ensure squid can't leave shm segments behind when it terminates. > Not sure how realistic this is. It could always fail in a way that prevents > it from cleaning up after itself. Or somebody could send it a SIGKILL, in > which case it definitely couldn't clean up after itself. > - Simply accepting ftruncate() failing with EINVAL as non fatal condition > on OS X and continue. This would be easy to do, but there are several > different reasons why EINVAL may be returned. It wouldn't be a good idea to > continue in all those cases. So this won't really work reliably. > - Run statSize() on the memory segment after successfully opening it. If > the size returned is greater than 0, we know ftruncate() will fail on OS X, > because the segment isn't a new one. So, squid could call ftruncate() only > if size is 0 and maybe call memset() otherwise to zero it out. > - Just unconditionally call unlink() before shm_open(). If the segment > didn't exist, unlink() will fail, but that won't really matter. This would > likely take the least amount of code. > - Call shm_open with O_EXCL, so that it'll fail if the shm segment already > exists. If the call fails with EEXIST, unlink the shm segment and retry. > This one may not even need to be Darwin-specific. Would be unnecessary on > other platforms, but wouldn't really hurt. > > I did create a sample implementation of the last option. According to my > tests, it does seem to work as intended. > > --- a/src/ipc/mem/Segment.cc > +++ b/src/ipc/mem/Segment.cc > @@ -85,12 +85,31 @@ Ipc::Mem::Segment::Enabled() > void > Ipc::Mem::Segment::create(const off_t aSize) > { > + bool do_retry = false; > + > assert(aSize > 0); > assert(theFD < 0); > > - // OS X does not allow using O_TRUNC here. > - theFD = shm_open(theName.termedBuf(), O_CREAT | O_RDWR, > - S_IRUSR | S_IWUSR); > + do { > + // OS X does not allow using O_TRUNC with shm_open. > + // Also, OS X only permits ftruncate() on new shared memory areas. > + // Therefore, we know that ftruncate() will fail if the shared > memory > + // area already exists. To prevent this, we delete and re-create > the > + // area if it exsisted previously (i.e. from an unclean shutdown). > + theFD = shm_open(theName.termedBuf(), O_CREAT | O_EXCL | O_RDWR, > + S_IRUSR | S_IWUSR); > + if (theFD < 0 && errno == EEXIST) { > + int old_errno = errno; > + unlink(); > + // We want to report the shm_open failure, not the unlink > failure. > + errno = old_errno; > + // Retry once, but only once. > + do_retry = !do_retry; > + } else { > + do_retry = false; > + } > + } while (do_retry); > + > if (theFD < 0) { > debugs(54, 5, HERE << "shm_open " << theName << ": " << > xstrerror()); > fatalf("Ipc::Mem::Segment::create failed to shm_open(%s): %s\n", > > > Regards, > -Markus > > > _______________________________________________ > squid-dev mailing list > [email protected] > http://lists.squid-cache.org/listinfo/squid-dev > >
_______________________________________________ squid-dev mailing list [email protected] http://lists.squid-cache.org/listinfo/squid-dev
