The system is back up now.
The postmortem is that the kernel run out of memory and started killing processes randomly. I noticed that it kept killing apache and puppet, but most likely the memory bloat was coming from JIRA and Confluence.
The system appeared to have been in this state for 8 hours or so. Presumably in the end it ends up killing something very important and the system choked.
I'm going to adjust the init script of these daemons so that they have higher OOM killer scores, so that the lower memory situation, it will cause the Java services to restart.
On 05/01/2013 08:41 AM, Kohsuke Kawaguchi wrote:
I just requested the machine to be rebooted to OSUOSL. Hopefully it'll be back up soon. -- Kohsuke Kawaguchi -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
-- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Jenkins Enterprise, our professional version of Jenkins -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
