Hello hackers, On AIX, which I think might have been a pioneer in overcommitting memory (?), they have the wonderfully named signal SIGDANGER. It's delivered to every process on the system as a final courtesy some time before the OOM killer starts delivering SIGKILL. The theory is that you might be able to give back memory to avoid the carnage.
Most (all?) popular operating systems overcommit memory, and from time to time someone proposes that they should have SIGDANGER too. I've often wondered if it makes sense or not as a concept: although you might manage to avoid getting killed for a while, is your access pattern sustainable? On the other hand, maybe if you're less afraid of the OOM you could use resources better. I'm not sure, but perhaps it's just too blunt an instrument. Certainly it would be possible for PostgreSQL to respond to SIGDANGER by dropping caches and perhaps even old idle backends via a CHECK_FOR_INTERRUPTS() handler, and combined with log monitoring so that you know you have a serious problem and need to consider making changes so it doesn't happen again, I think over all it's probably a good thing to support, because your system might actually survive the storm. This question was somewhat hypothetical, because Linux doesn't have it, and I don't expect we have too many AIX users. Neither does FreeBSD, though I've been tempted to propose it there. Recently Facebook published oomd[1][2] for Linux. I haven't tried it (for one thing you currently need a patched kernel which is a barrier to casual investigation) but basically it's a better OOM mousetrap that runs in userspace. It could be interesting for PostgreSQL installations, because you can configure it to be smarter about what to kill, and you can teach it to give you feedback and warnings about pressure. It may be that no modification to PostgreSQL is required at all to make good use of this, or it may be that you'd want a new signalling mechanism that we'd have to provide, amounting to a home-made equivalent of SIGDANGER (or something richer). I don't have a specific proposal, but thought this was newsworthy and might give someone some ideas. [1] https://github.com/facebookincubator/oomd [2] https://code.fb.com/production-engineering/open-sourcing-oomd-a-new-approach-to-handling-ooms/ -- Thomas Munro http://www.enterprisedb.com
