On Tue, Sep 6, 2011 at 12:57 PM, Olav Vitters <[email protected]> wrote:
> On Tue, Sep 06, 2011 at 02:07:36PM -0400, Owen Taylor wrote: > > On Fri, 2011-09-02 at 19:17 -0400, Owen Taylor wrote: > > > label.gnome.org ran out of memory this morning, it was looping hard in > > > the OOM killer. I got Red Hat IT to power cycle the machine a few hours > > > ago, but when I just looked the load average was at 24 and the machine > > > looked like it was heading for another OOM death. > > > > > > So, I stopped httpd, so ldap would stay up and we wouldn't have to get > > > the machine rebooted again. I also stopped puppet, since I think that > > > would start httpd when run. > > > > > > If anybody wants to dig in and try to figure out what is going on, that > > > probably would be a good idea. > > > > What was going on is that a user accidentally (I think) uploaded a 97M > > binary file as the content of a fairly frequently accessed page. > > > > When anybody tried to access that page, it would spin forever eating > > enormous amounts of memory, which would eventually take the server down. > > > > I reverted the page change, and removed the offending revisions. > > Damn.. I like your investigation skills. How were you able to figure > this out? I noticed the heavy memory usage and CPU, but couldn't do much > more :-( > > could you also setup 25GB or space on the wiki VM? I wanted to migrate > it but don't have knowledge on VMs. live.gnome.org + other smaller sites > take up 9.7GB, so 25GB should be ok. > > Perhaps we can have the wiki disallow loading files past a certain size so that we don't run into it again? sri
_______________________________________________ gnome-infrastructure mailing list [email protected] http://mail.gnome.org/mailman/listinfo/gnome-infrastructure
