Er. Apparently it's a feature, not a bug. "rsyslog was still a running process, it just stopped logging both remotely and to local files. We found a few discussions of this problem from 2012, 2011 and 2009 but they didn’t entirely cover the problem. However, the common thread was a connectivity issue causing problems with the queuing. Although not definitive, that every server in just one of our data centres saw this problem gave weight to a network based issue which may have caused rsyslog to hang for all actions, even though it was a network issue and we still had disk based logging enabled.
After discussing the issue with the Papertrail support guys, in order to combat this we decided to enable reliable f0rwarding which means rsyslog will queue log lines in memory and then to disk if the remote server cannot be reached, posting them when connectivity returns. This is necessary because syslog over TCP is not entirely reliable" (See https://blog.serverdensity.com/reliable-forwarding-with-rsyslog/) ** Changed in: rsyslog (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/1389356 Title: rsyslogd stops responding, doesn't log data, and eventually hangs the system Status in “rsyslog” package in Ubuntu: Invalid Bug description: This is happening on a wide variety of client systems that are all dissimilar--some are virtual machines, some are physical machines, and it spans 11.04, 12.04, 13.10, and 14.04. It started about two months ago. Approximately once per week, we will start getting calls from all our clients that services running on linux boxes are unavailable or extremely slow. Attempts to access the boxes via SSH will either not work (hang and then timeout after ~2 minutes) or succeed (after hanging for ~1 minute). Then the shell prompt takes a while (maybe 30 seconds) to display. After spending several frustrating hours with one particular box, I noticed the following: * Very low disk IO (i.e. the box isn't hammering the disk) * Memory usage was appropriate * Network IO was appropriate and responsive (ping, traceroute, wget, etc...) * Logs were all 'empty'. Last log data was in the log from the previous evening (i.e. /var/log/syslog.1 has a final entry at 8:36 PM PST from the previous night) Running the command 'restart rsyslogd' immediately returns the box to normal operation. After a few more testing sessions, I can see that rsyslogd is running on all these boxes, it just appears to be unresponsive. The issue happens fairly regularly--every 7-10 days, and it happens on multiple disparate systems on different networks at approximately the same time. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1389356/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
