RE: Mon Server Goes Foobar, help!
For the mailing list archives, it turns out there was a routing problem preventing our name servers from reaching our ISP's authoritative name servers. This stopped us from getting reverse DNS for our internet address space which caused our syslog server (which was functioning as a syslog collector) to block. Mon would then wait on the block. Thanks for all the help guys. Without it, it would have been a long time before I found the problem. Out. -Original Message- From: Eric Sorenson [mailto:[EMAIL PROTECTED] Sent: Monday, January 12, 2004 12:45 PM To: Gary Richardson Cc: [EMAIL PROTECTED] Subject: RE: Mon Server Goes Foobar, help! On Mon, 12 Jan 2004, Gary Richardson wrote: > I'm doing some more research into this. A ran -d:Profile for perl and found > that 96% of the time is spent in Sys::Syslog::_syslog_send_socket. Is this > normal? Maybe the output logfile is set to fsync-on-write. This is the (unfortunate) default for some syslogs. Try prepending the filename with a '-' to turn it off, like: local1.*-/var/log/mon.log NB not all syslogs support this, check your local man pages for details. -- Eric Sorenson - Systems / Network Administrator, MIS - Transmeta Corporation ___ mon mailing list [EMAIL PROTECTED] http://linux.kernel.org/mailman/listinfo/mon
RE: Mon Server Goes Foobar, help!
On Mon, 12 Jan 2004, Gary Richardson wrote: > I'm doing some more research into this. A ran -d:Profile for perl and found > that 96% of the time is spent in Sys::Syslog::_syslog_send_socket. Is this > normal? Maybe the output logfile is set to fsync-on-write. This is the (unfortunate) default for some syslogs. Try prepending the filename with a '-' to turn it off, like: local1.*-/var/log/mon.log NB not all syslogs support this, check your local man pages for details. -- Eric Sorenson - Systems / Network Administrator, MIS - Transmeta Corporation ___ mon mailing list [EMAIL PROTECTED] http://linux.kernel.org/mailman/listinfo/mon
RE: Mon Server Goes Foobar, help!
--On Monday, January 12, 2004 12:09 PM -0800 Gary Richardson <[EMAIL PROTECTED]> wrote: I'm doing some more research into this. A ran --d:Profile for perl and found that 96% of the time is spent in Sys::Syslog::_syslog_send_socket. Is this normal? Sounds like your syslog server may be having problems. But then I've never run Mon in the Profiler, so I don't know if thats really abnormal. Try disabling syslog's from Mon and see if that helps. The other situation in which I've seen Mon have problems like yours is when an alert script is hanging. But I don't have that problem any more, because I've long since patched my copy of Mon to handle fork alerts cleanly, and clean them up during the normal child processing code. -David Nolan Network Software Developer Computing Services Carnegie Mellon University ___ mon mailing list [EMAIL PROTECTED] http://linux.kernel.org/mailman/listinfo/mon
RE: Mon Server Goes Foobar, help!
Hey All, I’m doing some more research into this. A ran –d:Profile for perl and found that 96% of the time is spent in Sys::Syslog::_syslog_send_socket. Is this normal? Thanks. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Gary Richardson Sent: Friday, January 09, 2004 8:09 PM To: [EMAIL PROTECTED] Subject: Mon Server Goes Foobar, help! Hey, I have a mon server that has been running fine for a few months. All of a sudden it is doing crazy things. We are using mon.cgi for reporting. It is now timing out 9 out of 10 times. When you telnet to the mon port and try to issue commands, sometimes it hangs for a long time and others it hangs for 10 seconds. Running top shows all of the monitors going off at the same time instead of the normal ‘random intervals’. I have a feeling this is related. I have a feeling that a perl module got upgraded in the background and is causing this problem. There haven’t been any configuration changes since before Christmas. Has anyone experienced this or something similar before? Thanks. ___ mon mailing list [EMAIL PROTECTED] http://linux.kernel.org/mailman/listinfo/mon
Re: Mon Server Goes Foobar, help!
On Fri, Jan 09, 2004 at 08:09:24PM -0800, Gary Richardson wrote: > > I have a mon server that has been running fine for a few months. All of a > sudden it is doing crazy things. We are using mon.cgi for reporting. It is > now timing out 9 out of 10 times. When you telnet to the mon port and try to > issue commands, sometimes it hangs for a long time and others it hangs for > 10 seconds. I've had bad monitors do this before. For me, it was a matter of shutting off all monitors, and enabling one at a time. There were no obviously hung monitors or helpful log output in my case. It came back to perl versions and certain badly written scripts (newer perl didn't like something, I forget exactly what a year or two later). -- Nate It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. ___ mon mailing list [EMAIL PROTECTED] http://linux.kernel.org/mailman/listinfo/mon