Hi.
I have had a "sort-of-same-thing appear" in embedded system (which
remember could have a completely different set of issues on why the
syslog hanging situation) but I found that syslog itself was not
getting sufficient scheduling time to actually do its work because the
processes currently running where over allocated scheduling time and
it was being overswapped by syslog entries from other processes. I
proved this to myself by "renicing the syslog process" to a higher
priority (about the same as any RT process), and after a while (it had
a huge backlog of things to sort, and act upon - given the syslog
priorities themselves) it was able to redeem itself. Another way I
availed additional buffering was to apply a named pipe (pipes being
4KB in size by default), feeding /dev/log rather than delve into the
syslog source and changing buffers etc which would change the number
of variable under suspicion.
These actions may - I stress may - assist you in understanding the
real syslog issue .
Cheers. Grahame
On 12/02/2009, at 12:00 PM, [email protected] wrote:
From: Jamie Wilkinson <[email protected]>
Date: 11 February 2009 7:07:46 PM
To: [email protected]
Cc: [email protected]
Subject: Re: [SLUG] Apps hanging when writing to /dev/log
Is the syslog daemon logging into /var/log/syslog that it's out of
connections? syslog can run out of file descriptors sometimes; if
it's syslog-ng it'll write "Number of allowed concurrent connections
exceeded".
2009/1/29 Jeremy Visser <[email protected]>:
G'day,
Since around the beginning of the month, I've been encountering a
problem with our server (glenstorm, Ubuntu 8.04), where apps that
attempt to write out logs to /dev/log will hang for 5 or so minutes.
Most notably PAM gets affected, which makes ssh'ing in or using
sudo a
_very_ lengthy process. Also BIND was unresponsive, I guess anything
that writes to the logs would have been.
I only found out a workaround yesterday thanks to Mick Pollard
(lunix),
who suggested I use strace. Apps like sudo would connect() to /dev/
log
(which I presume is a UNIX socket), and use send() to write the
actual
log. The apps were hanging on send(), not connect().
Eventually, I've found that this only occurs if syslogd is running.
While an app is hanging, a `killall syslogd` will instantly un-freeze
it.
`fuser -u /dev/log` reveals that indeed, syslogd listens on /dev/log.
Funny thing is, if I restart syslogd, and do things that write to the
log, it works fine. I can see the log entries coming through in
syslog,
and it's all good. Just after a few hours, it conks out and starts
hanging.
I am not sure if the actual syslogd process is locked up, or what.
Oh, and none of my filesystems are full. That was one of the first
things I checked. :)
Any idea why syslogd is behaving as such?
Jeremy.
--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
From: Jeremy Visser <[email protected]>
Date: 11 February 2009 9:25:22 PM
To: [email protected]
Cc: Jamie Wilkinson <[email protected]>
Subject: Re: [SLUG] Apps hanging when writing to /dev/log
On Wednesday 11 February 2009 19:07:46 Jamie Wilkinson wrote:
Is the syslog daemon logging into /var/log/syslog that it's out of
connections?
Interesting. I presume something like lsof will tell me this.
syslog can run out of file descriptors sometimes; if
it's syslog-ng it'll write "Number of allowed concurrent connections
exceeded".
I know it's not syslog-ng. It's sysklogd, which is stock-standard in
Ubuntu:
http://www.infodrom.org/projects/sysklogd/
--
SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
