Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Stephen Harris <[EMAIL PROTECTED]>: > > It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a > > Huh? "SHOULD"? Why? If syslog dies for any reason (bug, DOS, hack, > admin stupidity) then I sure don't want the system freezing up. Should because this is the only audit logging facility presently available to Linux. > ( heh... at work on Solaris I monitor 300+ systems, and it's not unusual > to find 1 box a week with syslog not running for some reason or another. > I can't decide whether it's admin stupidity or bugs in Solaris syslog - of > which there are many :-(( ) On these boxes you should be running the audit log. Which has the property of shutting the system down when it is aborted... > syslog is not meant to be a secure audit system. Messages can be > legitimately dropped. Applications have been coded assuming that they > will not be frozen in syslog(). Linux should not be different in this > respect. Hmm... it might be nice to be this a system tunable parameter > but I'm not sure the best way of doing that (glibc maybe?) The best way would be to have a true audit daemon that has the property of hanging/shutdown of the system. I would prefer a shutdown to single user mode than a hang. That way I would get a chance to examine the log/restart the daemon and examine the log. Even better would be a way to suspend/checkpoint all processing, switch to a "audit emergency" mode with no network activity allowed, and then examine things. It would provide an option to clean the system and reboot, or restart the audit daemon and resume multiuser mode (resuming all suspended/checkpointed processes). Once there is an audit daemon then the security messages/alerts, and only those messages, would be sent to it. That way syslogd is available for non-security related events, and these could be dropped when necessary. - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
- Received message begins Here - Jesse Pollard <[EMAIL PROTECTED]>: > On Sun, 29 Oct 2000, Stephen Harris wrote: > >Horst von Brand wrote: > > > >> > > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, > >> > > kernel 2.2.x), within a few minutes you will find your entire machine > >> > > grinds to a halt. For example, nobody can log in. > >> > >> Great! Yet another way in which root can get the rope to shoot herself in > >> the foot. Anything _really_ new? > > > >OK, let's go a step further - what if syslog dies or breaks in some way > >shape or form so that the syslog() function blocks...? > > > >My worry is the one that was originally raised but ignored: syslog() should > >not BLOCK regardless of whether it's local or remote. syslog is not a > >reliable mechanism and many programs have been written assuming they can > >fire off syslog() calls without worry. > > It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a > few seconds (depending on the log rate...). > > I do believe that restarting syslog should be possible... Perhaps syslog > should be started by inetd at the very beginning. Then it could be restarted > after an exit/abort. > > This can STILL fail if the syslog.conf is completely invalid - but then the > system SHOULD be stopped pending the investigation of why the file has been > corrupted, or syslogd falls back on a default configuration (record everything > in the syslog file). As was pointed out in a separate E-mail: I ment to say "init" instead of "inetd", since inetd can generate syslog messages... - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
- Received message begins Here - Jesse Pollard [EMAIL PROTECTED]: On Sun, 29 Oct 2000, Stephen Harris wrote: Horst von Brand wrote: If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, kernel 2.2.x), within a few minutes you will find your entire machine grinds to a halt. For example, nobody can log in. Great! Yet another way in which root can get the rope to shoot herself in the foot. Anything _really_ new? OK, let's go a step further - what if syslog dies or breaks in some way shape or form so that the syslog() function blocks...? My worry is the one that was originally raised but ignored: syslog() should not BLOCK regardless of whether it's local or remote. syslog is not a reliable mechanism and many programs have been written assuming they can fire off syslog() calls without worry. It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a few seconds (depending on the log rate...). I do believe that restarting syslog should be possible... Perhaps syslog should be started by inetd at the very beginning. Then it could be restarted after an exit/abort. This can STILL fail if the syslog.conf is completely invalid - but then the system SHOULD be stopped pending the investigation of why the file has been corrupted, or syslogd falls back on a default configuration (record everything in the syslog file). As was pointed out in a separate E-mail: I ment to say "init" instead of "inetd", since inetd can generate syslog messages... - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Stephen Harris [EMAIL PROTECTED]: It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a Huh? "SHOULD"? Why? If syslog dies for any reason (bug, DOS, hack, admin stupidity) then I sure don't want the system freezing up. Should because this is the only audit logging facility presently available to Linux. ( heh... at work on Solaris I monitor 300+ systems, and it's not unusual to find 1 box a week with syslog not running for some reason or another. I can't decide whether it's admin stupidity or bugs in Solaris syslog - of which there are many :-(( ) On these boxes you should be running the audit log. Which has the property of shutting the system down when it is aborted... syslog is not meant to be a secure audit system. Messages can be legitimately dropped. Applications have been coded assuming that they will not be frozen in syslog(). Linux should not be different in this respect. Hmm... it might be nice to be this a system tunable parameter but I'm not sure the best way of doing that (glibc maybe?) The best way would be to have a true audit daemon that has the property of hanging/shutdown of the system. I would prefer a shutdown to single user mode than a hang. That way I would get a chance to examine the log/restart the daemon and examine the log. Even better would be a way to suspend/checkpoint all processing, switch to a "audit emergency" mode with no network activity allowed, and then examine things. It would provide an option to clean the system and reboot, or restart the audit daemon and resume multiuser mode (resuming all suspended/checkpointed processes). Once there is an audit daemon then the security messages/alerts, and only those messages, would be sent to it. That way syslogd is available for non-security related events, and these could be dropped when necessary. - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
> > It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a > > Huh? "SHOULD"? Why? If syslog dies for any reason (bug, DOS, hack, > admin stupidity) then I sure don't want the system freezing up. In some cases, I find the syslog messages of more importance then a working system. I like to know what's going on on my machines. > ( heh... at work on Solaris I monitor 300+ systems, and it's not unusual > to find 1 box a week with syslog not running for some reason or another. > I can't decide whether it's admin stupidity or bugs in Solaris syslog - of > which there are many :-(( ) > > syslog is not meant to be a secure audit system. Messages can be > legitimately dropped. I find dropping messages unacceptable. > Applications have been coded assuming that they > will not be frozen in syslog(). Linux should not be different in this > respect. Hmm... it might be nice to be this a system tunable parameter > but I'm not sure the best way of doing that (glibc maybe?) I needs to be in glibc, yes. > > Stephen Harris Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On Sun, 29 Oct 2000, Jesse Pollard wrote: > On Sun, 29 Oct 2000, Stephen Harris wrote: > >Horst von Brand wrote: > > > >> > > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, > >> > > kernel 2.2.x), within a few minutes you will find your entire machine > >> > > grinds to a halt. For example, nobody can log in. > >> > >> Great! Yet another way in which root can get the rope to shoot herself in > >> the foot. Anything _really_ new? > > > >OK, let's go a step further - what if syslog dies or breaks in some way > >shape or form so that the syslog() function blocks...? > > > >My worry is the one that was originally raised but ignored: syslog() should > >not BLOCK regardless of whether it's local or remote. syslog is not a > >reliable mechanism and many programs have been written assuming they can > >fire off syslog() calls without worry. > > It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a > few seconds (depending on the log rate...). No. You should have the OPTION of stopping the machine, if accurate syslog information is more important to you than system functionality. This should also be an OS function, rather than just freezing processes as and when they call syslog()! > I do believe that restarting syslog should be possible... Perhaps syslog > should be started by inetd at the very beginning. Then it could be restarted > after an exit/abort. inetd? I'd have thought init would make a more suitable choice, perhaps with a monitor to lock the machine up or reboot if syslog fails and can't be restored. > This can STILL fail if the syslog.conf is completely invalid - but then the > system SHOULD be stopped pending the investigation of why the file has been > corrupted, or syslogd falls back on a default configuration (record everything > in the syslog file). Again, that might be useful in some cases, but certainly shouldn't be the only, or even default, mode. James. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
> It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a Huh? "SHOULD"? Why? If syslog dies for any reason (bug, DOS, hack, admin stupidity) then I sure don't want the system freezing up. ( heh... at work on Solaris I monitor 300+ systems, and it's not unusual to find 1 box a week with syslog not running for some reason or another. I can't decide whether it's admin stupidity or bugs in Solaris syslog - of which there are many :-(( ) syslog is not meant to be a secure audit system. Messages can be legitimately dropped. Applications have been coded assuming that they will not be frozen in syslog(). Linux should not be different in this respect. Hmm... it might be nice to be this a system tunable parameter but I'm not sure the best way of doing that (glibc maybe?) Stephen Harris [EMAIL PROTECTED] http://www.spuddy.org/ The truth is the truth, and opinion just opinion. But what is what? My employer pays to ignore my opinions; you get to do it for free. * Mow ! Call Spud the Cat on > 01708 442043 < for free Usenet access * - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On Sun, 29 Oct 2000, Stephen Harris wrote: >Horst von Brand wrote: > >> > > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, >> > > kernel 2.2.x), within a few minutes you will find your entire machine >> > > grinds to a halt. For example, nobody can log in. >> >> Great! Yet another way in which root can get the rope to shoot herself in >> the foot. Anything _really_ new? > >OK, let's go a step further - what if syslog dies or breaks in some way >shape or form so that the syslog() function blocks...? > >My worry is the one that was originally raised but ignored: syslog() should >not BLOCK regardless of whether it's local or remote. syslog is not a >reliable mechanism and many programs have been written assuming they can >fire off syslog() calls without worry. It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a few seconds (depending on the log rate...). I do believe that restarting syslog should be possible... Perhaps syslog should be started by inetd at the very beginning. Then it could be restarted after an exit/abort. This can STILL fail if the syslog.conf is completely invalid - but then the system SHOULD be stopped pending the investigation of why the file has been corrupted, or syslogd falls back on a default configuration (record everything in the syslog file). -- - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Horst von Brand wrote: > > > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, > > > kernel 2.2.x), within a few minutes you will find your entire machine > > > grinds to a halt. For example, nobody can log in. > > Great! Yet another way in which root can get the rope to shoot herself in > the foot. Anything _really_ new? OK, let's go a step further - what if syslog dies or breaks in some way shape or form so that the syslog() function blocks...? My worry is the one that was originally raised but ignored: syslog() should not BLOCK regardless of whether it's local or remote. syslog is not a reliable mechanism and many programs have been written assuming they can fire off syslog() calls without worry. Stephen Harris [EMAIL PROTECTED] http://www.spuddy.org/ The truth is the truth, and opinion just opinion. But what is what? My employer pays to ignore my opinions; you get to do it for free. * Mow ! Call Spud the Cat on > 01708 442043 < for free Usenet access * - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Horst von Brand wrote: If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, kernel 2.2.x), within a few minutes you will find your entire machine grinds to a halt. For example, nobody can log in. Great! Yet another way in which root can get the rope to shoot herself in the foot. Anything _really_ new? OK, let's go a step further - what if syslog dies or breaks in some way shape or form so that the syslog() function blocks...? My worry is the one that was originally raised but ignored: syslog() should not BLOCK regardless of whether it's local or remote. syslog is not a reliable mechanism and many programs have been written assuming they can fire off syslog() calls without worry. Stephen Harris [EMAIL PROTECTED] http://www.spuddy.org/ The truth is the truth, and opinion just opinion. But what is what? My employer pays to ignore my opinions; you get to do it for free. * Mow ! Call Spud the Cat on 01708 442043 for free Usenet access * - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On Sun, 29 Oct 2000, Stephen Harris wrote: Horst von Brand wrote: If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, kernel 2.2.x), within a few minutes you will find your entire machine grinds to a halt. For example, nobody can log in. Great! Yet another way in which root can get the rope to shoot herself in the foot. Anything _really_ new? OK, let's go a step further - what if syslog dies or breaks in some way shape or form so that the syslog() function blocks...? My worry is the one that was originally raised but ignored: syslog() should not BLOCK regardless of whether it's local or remote. syslog is not a reliable mechanism and many programs have been written assuming they can fire off syslog() calls without worry. It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a few seconds (depending on the log rate...). I do believe that restarting syslog should be possible... Perhaps syslog should be started by inetd at the very beginning. Then it could be restarted after an exit/abort. This can STILL fail if the syslog.conf is completely invalid - but then the system SHOULD be stopped pending the investigation of why the file has been corrupted, or syslogd falls back on a default configuration (record everything in the syslog file). -- - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On Sun, 29 Oct 2000, Jesse Pollard wrote: On Sun, 29 Oct 2000, Stephen Harris wrote: Horst von Brand wrote: If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, kernel 2.2.x), within a few minutes you will find your entire machine grinds to a halt. For example, nobody can log in. Great! Yet another way in which root can get the rope to shoot herself in the foot. Anything _really_ new? OK, let's go a step further - what if syslog dies or breaks in some way shape or form so that the syslog() function blocks...? My worry is the one that was originally raised but ignored: syslog() should not BLOCK regardless of whether it's local or remote. syslog is not a reliable mechanism and many programs have been written assuming they can fire off syslog() calls without worry. It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a few seconds (depending on the log rate...). No. You should have the OPTION of stopping the machine, if accurate syslog information is more important to you than system functionality. This should also be an OS function, rather than just freezing processes as and when they call syslog()! I do believe that restarting syslog should be possible... Perhaps syslog should be started by inetd at the very beginning. Then it could be restarted after an exit/abort. inetd? I'd have thought init would make a more suitable choice, perhaps with a monitor to lock the machine up or reboot if syslog fails and can't be restored. This can STILL fail if the syslog.conf is completely invalid - but then the system SHOULD be stopped pending the investigation of why the file has been corrupted, or syslogd falls back on a default configuration (record everything in the syslog file). Again, that might be useful in some cases, but certainly shouldn't be the only, or even default, mode. James. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a Huh? "SHOULD"? Why? If syslog dies for any reason (bug, DOS, hack, admin stupidity) then I sure don't want the system freezing up. In some cases, I find the syslog messages of more importance then a working system. I like to know what's going on on my machines. ( heh... at work on Solaris I monitor 300+ systems, and it's not unusual to find 1 box a week with syslog not running for some reason or another. I can't decide whether it's admin stupidity or bugs in Solaris syslog - of which there are many :-(( ) syslog is not meant to be a secure audit system. Messages can be legitimately dropped. I find dropping messages unacceptable. Applications have been coded assuming that they will not be frozen in syslog(). Linux should not be different in this respect. Hmm... it might be nice to be this a system tunable parameter but I'm not sure the best way of doing that (glibc maybe?) I needs to be in glibc, yes. Stephen Harris Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
[EMAIL PROTECTED] (Stephen Harris) said: > A lot of talk here has been about syslog and DNS blocking, but the > original message mentioned: > > > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, > > kernel 2.2.x), within a few minutes you will find your entire machine > > grinds to a halt. For example, nobody can log in. Great! Yet another way in which root can get the rope to shoot herself in the foot. Anything _really_ new? -- Horst von Brand [EMAIL PROTECTED] Casilla 9G, Vin~a del Mar, Chile +56 32 672616 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
A lot of talk here has been about syslog and DNS blocking, but the original message mentioned: > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, > kernel 2.2.x), within a few minutes you will find your entire machine > grinds to a halt. For example, nobody can log in. Has this been addressed (and I missed it) or did the question get diverted? If it has been addressed, just please mail me personally to save traffic on the list. Thanks! -- Stephen Harris [EMAIL PROTECTED] http://www.spuddy.org/ The truth is the truth, and opinion just opinion. But what is what? My employer pays to ignore my opinions; you get to do it for free. * Mow ! Call Spud the Cat on > 01708 442043 < for free Usenet access * - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
A lot of talk here has been about syslog and DNS blocking, but the original message mentioned: If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, kernel 2.2.x), within a few minutes you will find your entire machine grinds to a halt. For example, nobody can log in. Has this been addressed (and I missed it) or did the question get diverted? If it has been addressed, just please mail me personally to save traffic on the list. Thanks! -- Stephen Harris [EMAIL PROTECTED] http://www.spuddy.org/ The truth is the truth, and opinion just opinion. But what is what? My employer pays to ignore my opinions; you get to do it for free. * Mow ! Call Spud the Cat on 01708 442043 for free Usenet access * - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
[EMAIL PROTECTED] (Stephen Harris) said: A lot of talk here has been about syslog and DNS blocking, but the original message mentioned: If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, kernel 2.2.x), within a few minutes you will find your entire machine grinds to a halt. For example, nobody can log in. Great! Yet another way in which root can get the rope to shoot herself in the foot. Anything _really_ new? -- Horst von Brand [EMAIL PROTECTED] Casilla 9G, Vin~a del Mar, Chile +56 32 672616 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
> Sadly, you WILL still lose entries if the system crashes before fs metadata > has been flushed to disk. Unless the inode has the correct size stored, the > crap fsync()ed to disk doesn't make much difference. Yep. I can't really think of a case where you wouldn't lose data in case of for example a hard lock. > (This is amplified by dcache.) I'm not that familiar with kernel internals.. To bring up that point : Anyone had a good advice on a good OS internals book ?? I'm gonna read the 2.2.x kernel sources, so I could use a good background. Regards, Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
> This is not an option for us, unfortunately. Many of our IP addresses > are dynamically assigned, with the DNS tables dynamically updated. Not an option in that case. > Thank you for the patch to syslogd, though! Can you try to get your > "-x" option into the standard distributions of syslogd, or should I > work up a bug report / feature request for Red Hat myself? I have no idea who officially maintains it.. putting a bug report with RH should be a good idea. The patch is untested, so someone needs to verify that remote logging indeed nog is IP only. > - Pat > Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On Thu, 26 Oct 2000, Igmar Palsenberg wrote: >> Per chance are you running the name service caching daemon (nscd)? I'd >> also guess you aren't disabling fsync() for your sysylog files (it's part >> of the syslog.conf format) -- this is a conciderable drain on syslogd. > >Agree. It is there for a reason : I case the system hangs, you at least >get the last messages. But it is indeed a major drain. I've send Patrick a >small path that makes reverse lookups a config option. Sadly, you WILL still lose entries if the system crashes before fs metadata has been flushed to disk. Unless the inode has the correct size stored, the crap fsync()ed to disk doesn't make much difference. (This is amplified by dcache.) --Ricky - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Igmar Palsenberg <[EMAIL PROTECTED]> writes: > On 23 Oct 2000, Patrick J. LoPresti wrote: > > > Not true. The named on our loghost is authoritative for the reverse > > mappings for all of the machines which can log there. > > Put the names of your machines in /etc/hosts on your logmachine. This is not an option for us, unfortunately. Many of our IP addresses are dynamically assigned, with the DNS tables dynamically updated. Thank you for the patch to syslogd, though! Can you try to get your "-x" option into the standard distributions of syslogd, or should I work up a bug report / feature request for Red Hat myself? - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
> Perhaps syslogd is not giving higher priority to local messages; if it > did, maybe it could recover from the deadlock. But this would not be > a reliable solution; the only reliable solution is for syslogd to be > independent of any processes which need to talk to it. In that case, don't do lookups at all. Let a script do resolving in a crontab or so. analog uses a nice solution for this : store hostnames in a file. When it doesn't know a hostname, look it up. That saves tons of querys. > > Per chance are you running the name service caching daemon (nscd)? > > No. > > > I'd also guess you aren't disabling fsync() for your sysylog files > > (it's part of the syslog.conf format) -- this is a conciderable > > drain on syslogd. > > I see no documentation for such an option in the syslog.conf man page. > This is with the current Red Hat 6.2 syslogd (package > sysklogd-1.3.31-17). There is. see man syslogd(8). You need to prefix files that don't need to be synced() with a - > - Pat Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
> No, I didn't say they "should" be dropped but merely that dropping them > would fix your problem. Personally, I'd look closely at your setup to > determine exactly why this has become a problem. named is being blocked > on writing to /dev/log. This should only happen if there is sufficient > _local_ syslog traffic to fill the buffer or syslogd has too much remote > traffic to ever read from /dev/log. > > Per chance are you running the name service caching daemon (nscd)? I'd > also guess you aren't disabling fsync() for your sysylog files (it's part > of the syslog.conf format) -- this is a conciderable drain on syslogd. Agree. It is there for a reason : I case the system hangs, you at least get the last messages. But it is indeed a major drain. I've send Patrick a small path that makes reverse lookups a config option. Please note that syslog does NOT resolve things it get's from the local machine, it uses gethostname() for that name. > --Ricky Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: > Jesse Pollard <[EMAIL PROTECTED]> writes: > > > Don't configure syslogd to do reverse lookups. > > Our syslogd has no option to disable the reverse lookups. Requires a recompile. > > You can NEVER guarantee that the reverse lookup will succeed, and > > can be delayed several minutes for a single reply. > > Not true. The named on our loghost is authoritative for the reverse > mappings for all of the machines which can log there. Put the names of your machines in /etc/hosts on your logmachine. > > > I consider this a configuration error. I don't believe syslogd > > should ever do a reverse lookup, since the name you are trying to > > get may never arrive, or if arrives, it may be spoofed. > > There *is* no configuration for these tools which gives the behavior > you describe, so this is not a "configuration error". I agree that it is ugly that you can't disable reverse with syslog. There are implementations of the syslogd (syslog-ng) that do allow this. If you want to stick with the 'normal' syslog, either recompile with an option not do do reverse lookups, put you hosts in /etc/hosts, of hack the source. > > It's not a bug, but a security feature. NO log to syslogd should be > > lost, since it may be related to an attack. > > Historically, no other Unix system has had reliable syslogging. It > would require very defensive programming for syslogd, and that has > clearly not been performed. > > And if this is what GNU/Linux intends, why does glibc use a SOCK_DGRAM > socket for communication with syslogd? By definition, such sockets > are *unreliable*. If syslog is supposed to be reliable, a different > connection type must be used. Because if is fast. Second, you can assume that remote logging is NOT done over the internet, anyone who does is an idiot. So the change that some UDP packet get's lost is close to zero. I think the changes are higher that the connection oriented solutions would get messages lost. > Your philosophy that "no syslog message should ever be lost" is not > necessarily bad. But it is clearly at odds with historical practice, > the current glibc syslog() implementation, and the current syslogd > itself. > > It is true that glibc falls back to using SOCK_STREAM if the > SOCK_DGRAM connection fails. Does that mean GNU/Linux is expects > syslog to be reliable eventually? If so, then my problem is entirely > a bug in syslogd and I will report it as such. I agree with the bad configuration remark at top. WHY do you want named to do the reverse ?? Putting things in /etc/hosts makes thing hell of a lot faster. With a central loghost that makes sense to me. > - Pat Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
> Ulrich Drepper <[EMAIL PROTECTED]> writes: > > > If anything has to be changed it's (as suggested) the configuration > > or even the implementation of syslogd. Make it robust. > > OK, but my current syslogd only listens to /dev/log as a SOCK_DGRAM. > If I wanted reliable syslogging, it would be listening on it as a > SOCK_STREAM. Maybe I care more about performance and backwards > compatibility than reliable syslogging. But whatever my reasons, my > connection to syslogd is already unreliable and therefore *should not > block*. > > (Could a syslogd listen on /dev/log both as SOCK_DGRAM and as > SOCK_STREAM? If so, then your philosophy implies that glibc should be > trying SOCK_STREAM *first*, falling back to SOCK_DGRAM for historical > compatibility. Either way, when it uses datagrams, it should never > block, period.) The reason for this is that DGRAM is faster the STREAM. And with syslogd being on a fast, secure network I can understand the design filosofie behind this. I'll have a look at syslog, and see if I can hack out the reverse lookups. > - Pat Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
> You obviously don't understand the communication channel being used. > "/dev/log" is a UNIX DOMAIN SOCKET -- AF_UNIX. Datagrams are unreliable > for _IP_ (AF_INET). Traffic on an AF_UNIX socket is always reliable. > > Ok, smarty, go change the syslogd source to open /dev/log as SOCK_STREAM > and watch it fail. (syslog wasn't invented yesturday.) > > I would suggest disabling name resolution for syslog, but that's an ugly > option. There's no way to stop a glibc system from doing a DNS query for > a reverse lookup. HOWEVER, you can set the DNS timeout to 1 second and > set the resolver options to prevent recursion (answer from cache only.) Reverse isn't necessary on a central loghost. Saves at least one system call. Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, > kernel 2.2.x), within a few minutes you will find your entire machine > grinds to a halt. For example, nobody can log in. > > This happens because once the /dev/log buffer fills, the syslog() > glibc function blocks. > > This is a problem for us in Real Life because named and syslogd are > deadlocking while trying to talk to each other. On our loghost, > syslogd needs to do reverse name lookups while named needs to call > syslog(). When traffic gets heavy all around, the deadlock is > triggered, and it is quite unpleasant. We are about to configure > named to use flat files instead of syslog, but that feels more like a > workaround than a fix. > > I am not sure whether this is a glibc bug or a kernel bug. I have > used netstat and the glibc sources to confirm that glibc is using a > SOCK_DGRAM Unix socket to send to /dev/log. I thought DGRAM sockets > were supposed to drop packets on the floor instead of blocking. But > perhaps I am wrong and glibc is supposed to be explicitly marking the > socket as non-blocking. > > Regardless of whose bug it is, I suggest that it needs to be fixed. > To my knowledge, other Unix systems do not behave this way; they > simply drop messages when syslogd is not responding. IMO, that is > correct behavior. This is a long standing. A perfect example that triggers the named part is Netscape : I one window does a lookup, all other windows are dead. I don't know the exact cause of this problem, you'll have to ask the glibc guys for this. The correct behaviour in this case is syslog() to stop doing lookups, dropping messages is unacceptable. Looking up a name afterward is no problem, loosing important messages is a problem. If you're talking remote syslogging here, putting all local hostnames in /etc/hosts would save named some work. Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
You obviously don't understand the communication channel being used. "/dev/log" is a UNIX DOMAIN SOCKET -- AF_UNIX. Datagrams are unreliable for _IP_ (AF_INET). Traffic on an AF_UNIX socket is always reliable. Ok, smarty, go change the syslogd source to open /dev/log as SOCK_STREAM and watch it fail. (syslog wasn't invented yesturday.) I would suggest disabling name resolution for syslog, but that's an ugly option. There's no way to stop a glibc system from doing a DNS query for a reverse lookup. HOWEVER, you can set the DNS timeout to 1 second and set the resolver options to prevent recursion (answer from cache only.) Reverse isn't necessary on a central loghost. Saves at least one system call. Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Ulrich Drepper [EMAIL PROTECTED] writes: If anything has to be changed it's (as suggested) the configuration or even the implementation of syslogd. Make it robust. OK, but my current syslogd only listens to /dev/log as a SOCK_DGRAM. If I wanted reliable syslogging, it would be listening on it as a SOCK_STREAM. Maybe I care more about performance and backwards compatibility than reliable syslogging. But whatever my reasons, my connection to syslogd is already unreliable and therefore *should not block*. (Could a syslogd listen on /dev/log both as SOCK_DGRAM and as SOCK_STREAM? If so, then your philosophy implies that glibc should be trying SOCK_STREAM *first*, falling back to SOCK_DGRAM for historical compatibility. Either way, when it uses datagrams, it should never block, period.) The reason for this is that DGRAM is faster the STREAM. And with syslogd being on a fast, secure network I can understand the design filosofie behind this. I'll have a look at syslog, and see if I can hack out the reverse lookups. - Pat Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: Jesse Pollard [EMAIL PROTECTED] writes: Don't configure syslogd to do reverse lookups. Our syslogd has no option to disable the reverse lookups. Requires a recompile. You can NEVER guarantee that the reverse lookup will succeed, and can be delayed several minutes for a single reply. Not true. The named on our loghost is authoritative for the reverse mappings for all of the machines which can log there. Put the names of your machines in /etc/hosts on your logmachine. I consider this a configuration error. I don't believe syslogd should ever do a reverse lookup, since the name you are trying to get may never arrive, or if arrives, it may be spoofed. There *is* no configuration for these tools which gives the behavior you describe, so this is not a "configuration error". I agree that it is ugly that you can't disable reverse with syslog. There are implementations of the syslogd (syslog-ng) that do allow this. If you want to stick with the 'normal' syslog, either recompile with an option not do do reverse lookups, put you hosts in /etc/hosts, of hack the source. It's not a bug, but a security feature. NO log to syslogd should be lost, since it may be related to an attack. Historically, no other Unix system has had reliable syslogging. It would require very defensive programming for syslogd, and that has clearly not been performed. And if this is what GNU/Linux intends, why does glibc use a SOCK_DGRAM socket for communication with syslogd? By definition, such sockets are *unreliable*. If syslog is supposed to be reliable, a different connection type must be used. Because if is fast. Second, you can assume that remote logging is NOT done over the internet, anyone who does is an idiot. So the change that some UDP packet get's lost is close to zero. I think the changes are higher that the connection oriented solutions would get messages lost. Your philosophy that "no syslog message should ever be lost" is not necessarily bad. But it is clearly at odds with historical practice, the current glibc syslog() implementation, and the current syslogd itself. It is true that glibc falls back to using SOCK_STREAM if the SOCK_DGRAM connection fails. Does that mean GNU/Linux is expects syslog to be reliable eventually? If so, then my problem is entirely a bug in syslogd and I will report it as such. I agree with the bad configuration remark at top. WHY do you want named to do the reverse ?? Putting things in /etc/hosts makes thing hell of a lot faster. With a central loghost that makes sense to me. - Pat Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
No, I didn't say they "should" be dropped but merely that dropping them would fix your problem. Personally, I'd look closely at your setup to determine exactly why this has become a problem. named is being blocked on writing to /dev/log. This should only happen if there is sufficient _local_ syslog traffic to fill the buffer or syslogd has too much remote traffic to ever read from /dev/log. Per chance are you running the name service caching daemon (nscd)? I'd also guess you aren't disabling fsync() for your sysylog files (it's part of the syslog.conf format) -- this is a conciderable drain on syslogd. Agree. It is there for a reason : I case the system hangs, you at least get the last messages. But it is indeed a major drain. I've send Patrick a small path that makes reverse lookups a config option. Please note that syslog does NOT resolve things it get's from the local machine, it uses gethostname() for that name. --Ricky Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Perhaps syslogd is not giving higher priority to local messages; if it did, maybe it could recover from the deadlock. But this would not be a reliable solution; the only reliable solution is for syslogd to be independent of any processes which need to talk to it. In that case, don't do lookups at all. Let a script do resolving in a crontab or so. analog uses a nice solution for this : store hostnames in a file. When it doesn't know a hostname, look it up. That saves tons of querys. Per chance are you running the name service caching daemon (nscd)? No. I'd also guess you aren't disabling fsync() for your sysylog files (it's part of the syslog.conf format) -- this is a conciderable drain on syslogd. I see no documentation for such an option in the syslog.conf man page. This is with the current Red Hat 6.2 syslogd (package sysklogd-1.3.31-17). There is. see man syslogd(8). You need to prefix files that don't need to be synced() with a - - Pat Igmar - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Igmar Palsenberg [EMAIL PROTECTED] writes: On 23 Oct 2000, Patrick J. LoPresti wrote: Not true. The named on our loghost is authoritative for the reverse mappings for all of the machines which can log there. Put the names of your machines in /etc/hosts on your logmachine. This is not an option for us, unfortunately. Many of our IP addresses are dynamically assigned, with the DNS tables dynamically updated. Thank you for the patch to syslogd, though! Can you try to get your "-x" option into the standard distributions of syslogd, or should I work up a bug report / feature request for Red Hat myself? - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On Thu, 26 Oct 2000, Igmar Palsenberg wrote: Per chance are you running the name service caching daemon (nscd)? I'd also guess you aren't disabling fsync() for your sysylog files (it's part of the syslog.conf format) -- this is a conciderable drain on syslogd. Agree. It is there for a reason : I case the system hangs, you at least get the last messages. But it is indeed a major drain. I've send Patrick a small path that makes reverse lookups a config option. Sadly, you WILL still lose entries if the system crashes before fs metadata has been flushed to disk. Unless the inode has the correct size stored, the crap fsync()ed to disk doesn't make much difference. (This is amplified by dcache.) --Ricky - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Followup to: <[EMAIL PROTECTED]>
By author:Kurt Roeckx <[EMAIL PROTECTED]>
In newsgroup: linux.dev.kernel
>
> On Mon, Oct 23, 2000 at 03:58:39PM -0400, Patrick J. LoPresti wrote:
> > "David S. Miller" <[EMAIL PROTECTED]> writes:
> >
> > > SOCK_DGRAM over AF_UNIX is reliable, it's a local transport.
> >
> > This is contrary to the documentation ("man socket"). The behavior
> > you describe is what I would expect for SOCK_SEQPACKET, not
> > SOCK_DGRAM.
>
> SOCK_DGRAM is only as reliable as the underlying protocol. If the
> unerlying one is not reliable, neither is SOCK_DGRAM. It will not
> do anything to make it more reliable, unlike others.
>
Again, SOCK_DGRAM is reliable over AF_UNIX, because AF_UNIX does not
implement any nonreliable transports.
-hpa
--
<[EMAIL PROTECTED]> at work, <[EMAIL PROTECTED]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Followup to: [EMAIL PROTECTED]
By author:Kurt Roeckx [EMAIL PROTECTED]
In newsgroup: linux.dev.kernel
On Mon, Oct 23, 2000 at 03:58:39PM -0400, Patrick J. LoPresti wrote:
"David S. Miller" [EMAIL PROTECTED] writes:
SOCK_DGRAM over AF_UNIX is reliable, it's a local transport.
This is contrary to the documentation ("man socket"). The behavior
you describe is what I would expect for SOCK_SEQPACKET, not
SOCK_DGRAM.
SOCK_DGRAM is only as reliable as the underlying protocol. If the
unerlying one is not reliable, neither is SOCK_DGRAM. It will not
do anything to make it more reliable, unlike others.
Again, SOCK_DGRAM is reliable over AF_UNIX, because AF_UNIX does not
implement any nonreliable transports.
-hpa
--
[EMAIL PROTECTED] at work, [EMAIL PROTECTED] in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
- Received message begins Here - > > Jesse Pollard <[EMAIL PROTECTED]> writes: > > > Don't configure syslogd to do reverse lookups. > > Our syslogd has no option to disable the reverse lookups. > > > You can NEVER guarantee that the reverse lookup will succeed, and > > can be delayed several minutes for a single reply. > > Not true. The named on our loghost is authoritative for the reverse > mappings for all of the machines which can log there. If the server is sufficiently busy, the reply WILL be delayed. It can even be on the same host, it just may not get around to a reply immediately. Authority is not equivalent to reply. The advantage of local files is that the reverse lookup doesn't depend on ANY outside agency. > > I consider this a configuration error. I don't believe syslogd > > should ever do a reverse lookup, since the name you are trying to > > get may never arrive, or if arrives, it may be spoofed. > > There *is* no configuration for these tools which gives the behavior > you describe, so this is not a "configuration error". I think it requires a recompile of the syslogd source to make that change. I grant that there isn't a runtime option for it (I do think there should be). > > It's not a bug, but a security feature. NO log to syslogd should be > > lost, since it may be related to an attack. > > Historically, no other Unix system has had reliable syslogging. It > would require very defensive programming for syslogd, and that has > clearly not been performed. Some do, some don't - Cray systems will shut down if the audit daemon disappears. Syslogd on Linux is providing the corresponding facility, at the current time. Neither syslogd, nor audit daemon are reliable on SGI systems, trusted solaris shuts down - I believe. The normal solaris is unknown since I haven't had time to fully configure either it or the audit daemon yet. > And if this is what GNU/Linux intends, why does glibc use a SOCK_DGRAM > socket for communication with syslogd? By definition, such sockets > are *unreliable*. If syslog is supposed to be reliable, a different > connection type must be used. I think that is a "don't care" option on AF_UNIX connections. Required for semantic compatability, but not used. > > Your philosophy that "no syslog message should ever be lost" is not > necessarily bad. But it is clearly at odds with historical practice, > the current glibc syslog() implementation, and the current syslogd > itself. > > It is true that glibc falls back to using SOCK_STREAM if the > SOCK_DGRAM connection fails. Does that mean GNU/Linux is expects > syslog to be reliable eventually? If so, then my problem is entirely > a bug in syslogd and I will report it as such. I think it is only reliable on the local host. If a remote syslog facility is used, then it may be dropped if the input queue becomes too long. - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On Mon, Oct 23, 2000 at 04:56:26PM -0400, Patrick J. LoPresti wrote: > You are effectively suggesting that named should be rewritten not to > use the glibc syslog functions at all. That strikes me as the worst > suggestion so far; it would be far better for syslogd not to do name > lookups. But my syslogd has no option to avoid name lookups; I will > submit a request to add one. /etc/nsswitch.conf: hosts: files dns /etc/hosts: ip.of.named.host name.of.named.host ip.of.other.host name.of.other.host Give explicite IP/name mappings for those which you don't want to be looked via the resolver. This is, of course, system-wide, but use it sparingly. /Matti Aarnio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: >Once the name resolution times out, you might expect things to become >unstuck. But they don't. Negative. Things have been queued. The deadlock will only go away if the very next message processed is the named local message. And then it would have to process a few more local messages so it wouldn't stall again so soon. >> Per chance are you running the name service caching daemon (nscd)? > >No. Please do. That will reduce the ammount of traffic to the name server. >> I'd also guess you aren't disabling fsync() for your sysylog files >> (it's part of the syslog.conf format) -- this is a conciderable >> drain on syslogd. > >I see no documentation for such an option in the syslog.conf man page. >This is with the current Red Hat 6.2 syslogd (package >sysklogd-1.3.31-17). It's in the syslogd and syslog.conf man page (sysklogd-1.3.31-16): (syslog.conf) Regular File Typically messages are logged to real files. The file has to be specified with full pathname, beginning with a slash ``/''. You may prefix each entry with the minus ``-'' sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some performance, especially if you run programs that use logging in a very verbose manner. --Ricky PS: as a side note, you can/will lose information even if sync is enabled. (fsync() will not flush metadata so the file is truncated on restart.) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Ricky Beam <[EMAIL PROTECTED]> writes: > Personally, I'd look closely at your setup to determine exactly why > this has become a problem. named is being blocked on writing to > /dev/log. This should only happen if there is sufficient _local_ > syslog traffic to fill the buffer or syslogd has too much remote > traffic to ever read from /dev/log. There is a lot of local traffic as well, yes. Lots of local traffic means named eventually finds itself waiting in line to log. Lots of remote traffic means syslogd is trying to talk to named a lot (to do reverse lookups). named waiting in line + syslogd trying to talk to named == deadlock; this is not too hard to see. Once the name resolution times out, you might expect things to become unstuck. But they don't. Perhaps syslogd is not giving higher priority to local messages; if it did, maybe it could recover from the deadlock. But this would not be a reliable solution; the only reliable solution is for syslogd to be independent of any processes which need to talk to it. > Per chance are you running the name service caching daemon (nscd)? No. > I'd also guess you aren't disabling fsync() for your sysylog files > (it's part of the syslog.conf format) -- this is a conciderable > drain on syslogd. I see no documentation for such an option in the syslog.conf man page. This is with the current Red Hat 6.2 syslogd (package sysklogd-1.3.31-17). - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: >So I have the glibc maintainer (and others) saying that syslog >messages should never be dropped, and you saying that named should be >dropping its syslog messages. No, I didn't say they "should" be dropped but merely that dropping them would fix your problem. Personally, I'd look closely at your setup to determine exactly why this has become a problem. named is being blocked on writing to /dev/log. This should only happen if there is sufficient _local_ syslog traffic to fill the buffer or syslogd has too much remote traffic to ever read from /dev/log. Per chance are you running the name service caching daemon (nscd)? I'd also guess you aren't disabling fsync() for your sysylog files (it's part of the syslog.conf format) -- this is a conciderable drain on syslogd. --Ricky - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Ricky Beam <[EMAIL PROTECTED]> writes: > syslogd isn't the blocker. The syslog functions in glibc being > called by named are the problem. Stop named from blocking on syslog > writes and the world will be happy again. So I have the glibc maintainer (and others) saying that syslog messages should never be dropped, and you saying that named should be dropping its syslog messages. One more time, from the top: named is calling syslog(), a glibc function. This function *blocks* waiting for delivery to the local syslogd, even though it is using SOCK_DGRAM sockets. There is no option to openlog() or syslog() to get non-blocking behavior (the LOG_NDELAY option means something else entirely). You are effectively suggesting that named should be rewritten not to use the glibc syslog functions at all. That strikes me as the worst suggestion so far; it would be far better for syslogd not to do name lookups. But my syslogd has no option to avoid name lookups; I will submit a request to add one. > I've gotta ask what kind of "load" can cause this to happen. > And for the record, syslogd shouldn't be doing DNS lookups for > things arriving via /dev/log -- that's always the local machine. This particular syslogd also accepts messages from remote hosts. So when there is a lot of syslog traffic, this syslogd talks to named a lot. named occasionally sends messages to syslog. Since syslog pauses waiting for named to respond to name queries, and named blocks waiting for syslog to consume the message, a deadlock is triggered. True, it is not a full deadlock, because the name query times out eventually. But it is bad enough that the system becomes largely non-responsive. - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: >Turning down the DNS timeout would affect *all* name resolution on the >system, right? That is not acceptable. You should be able to set it on a per-process basis (via an ENV var.) >As I said, I already have a workaround, which is to have named log to >a flat file. I agree that this is a poor workaround, and the "right >fix" is to modify syslogd not to perform blocking operations. My only >quibble is that SOCK_DGRAM is an odd transport to use here, even over >AF_UNIX. syslogd isn't the blocker. The syslog functions in glibc being called by named are the problem. Stop named from blocking on syslog writes and the world will be happy again. I've gotta ask what kind of "load" can cause this to happen. And for the record, syslogd shouldn't be doing DNS lookups for things arriving via /dev/log -- that's always the local machine. --Ricky - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Ricky Beam <[EMAIL PROTECTED]> writes: > I would suggest disabling name resolution for syslog, but that's an > ugly option. There's no way to stop a glibc system from doing a DNS > query for a reverse lookup. HOWEVER, you can set the DNS timeout to > 1 second and set the resolver options to prevent recursion (answer > from cache only.) Recursion has nothing to do with it; as I said, the named on this system is itself authoritative for all of the reverse lookups. Turning down the DNS timeout would affect *all* name resolution on the system, right? That is not acceptable. As I said, I already have a workaround, which is to have named log to a flat file. I agree that this is a poor workaround, and the "right fix" is to modify syslogd not to perform blocking operations. My only quibble is that SOCK_DGRAM is an odd transport to use here, even over AF_UNIX. > PS: Technically, this is not a lockup. syslogd should eventually > timeout waiting for the DNS query and go about it's business. Of > course, that may be upwards of 45 seconds -- very annoying. Yes. We are able to log in to the machine eventually and restart the offending processes. But that is little consolation to our users who notice the hang and the fallout afterward. - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: >Ulrich Drepper <[EMAIL PROTECTED]> writes: >> If anything has to be changed it's (as suggested) the configuration >> or even the implementation of syslogd. Make it robust. > >OK, but my current syslogd only listens to /dev/log as a SOCK_DGRAM. >If I wanted reliable syslogging, it would be listening on it as a >SOCK_STREAM. Maybe I care more about performance and backwards >compatibility than reliable syslogging. But whatever my reasons, my >connection to syslogd is already unreliable and therefore *should not >block*. You obviously don't understand the communication channel being used. "/dev/log" is a UNIX DOMAIN SOCKET -- AF_UNIX. Datagrams are unreliable for _IP_ (AF_INET). Traffic on an AF_UNIX socket is always reliable. Ok, smarty, go change the syslogd source to open /dev/log as SOCK_STREAM and watch it fail. (syslog wasn't invented yesturday.) I would suggest disabling name resolution for syslog, but that's an ugly option. There's no way to stop a glibc system from doing a DNS query for a reverse lookup. HOWEVER, you can set the DNS timeout to 1 second and set the resolver options to prevent recursion (answer from cache only.) --Ricky PS: Technically, this is not a lockup. syslogd should eventually timeout waiting for the DNS query and go about it's business. Of course, that may be upwards of 45 seconds -- very annoying. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
[EMAIL PROTECTED] (Patrick J. LoPresti) writes: > OK, but my current syslogd only listens to /dev/log as a SOCK_DGRAM. > [...] I don't care what the current syslogd does. Changing the libc just to work around the limitations of current implementations is wrong. Write a new syslogd and do it right. -- ---. ,-. 1325 Chesapeake Terrace Ulrich Drepper \,---' \ Sunnyvale, CA 94089 USA Red Hat `--' drepper at redhat.com ` - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Ulrich Drepper <[EMAIL PROTECTED]> writes: > If anything has to be changed it's (as suggested) the configuration > or even the implementation of syslogd. Make it robust. OK, but my current syslogd only listens to /dev/log as a SOCK_DGRAM. If I wanted reliable syslogging, it would be listening on it as a SOCK_STREAM. Maybe I care more about performance and backwards compatibility than reliable syslogging. But whatever my reasons, my connection to syslogd is already unreliable and therefore *should not block*. (Could a syslogd listen on /dev/log both as SOCK_DGRAM and as SOCK_STREAM? If so, then your philosophy implies that glibc should be trying SOCK_STREAM *first*, falling back to SOCK_DGRAM for historical compatibility. Either way, when it uses datagrams, it should never block, period.) - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Jesse Pollard <[EMAIL PROTECTED]> writes: > It's not a bug, but a security feature. NO log to syslogd should be lost, > since it may be related to an attack. That's exactly the argument I'm always using to turn down change requests like this. If the syslog() function could drop an entry and not send it is easy enough for somebody who has something to hide to overflow syslog() and then do the whatever s/he does not want to be logged. If anything has to be changed it's (as suggested) the configuration or even the implementation of syslogd. Make it robust. -- ---. ,-. 1325 Chesapeake Terrace Ulrich Drepper \,---' \ Sunnyvale, CA 94089 USA Red Hat `--' drepper at redhat.com ` - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Jesse Pollard <[EMAIL PROTECTED]> writes: > Don't configure syslogd to do reverse lookups. Our syslogd has no option to disable the reverse lookups. > You can NEVER guarantee that the reverse lookup will succeed, and > can be delayed several minutes for a single reply. Not true. The named on our loghost is authoritative for the reverse mappings for all of the machines which can log there. > I consider this a configuration error. I don't believe syslogd > should ever do a reverse lookup, since the name you are trying to > get may never arrive, or if arrives, it may be spoofed. There *is* no configuration for these tools which gives the behavior you describe, so this is not a "configuration error". > It's not a bug, but a security feature. NO log to syslogd should be > lost, since it may be related to an attack. Historically, no other Unix system has had reliable syslogging. It would require very defensive programming for syslogd, and that has clearly not been performed. And if this is what GNU/Linux intends, why does glibc use a SOCK_DGRAM socket for communication with syslogd? By definition, such sockets are *unreliable*. If syslog is supposed to be reliable, a different connection type must be used. Your philosophy that "no syslog message should ever be lost" is not necessarily bad. But it is clearly at odds with historical practice, the current glibc syslog() implementation, and the current syslogd itself. It is true that glibc falls back to using SOCK_STREAM if the SOCK_DGRAM connection fails. Does that mean GNU/Linux is expects syslog to be reliable eventually? If so, then my problem is entirely a bug in syslogd and I will report it as such. - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
[EMAIL PROTECTED] (Patrick J. LoPresti): > > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, > kernel 2.2.x), within a few minutes you will find your entire machine > grinds to a halt. For example, nobody can log in. > > This happens because once the /dev/log buffer fills, the syslog() > glibc function blocks. > > This is a problem for us in Real Life because named and syslogd are > deadlocking while trying to talk to each other. On our loghost, > syslogd needs to do reverse name lookups while named needs to call > syslog(). When traffic gets heavy all around, the deadlock is > triggered, and it is quite unpleasant. We are about to configure > named to use flat files instead of syslog, but that feels more like a > workaround than a fix. Don't configure syslogd to do reverse lookups. You can NEVER guarantee that the reverse lookup will succeed, and can be delayed several minutes for a single reply. > I am not sure whether this is a glibc bug or a kernel bug. I have > used netstat and the glibc sources to confirm that glibc is using a > SOCK_DGRAM Unix socket to send to /dev/log. I thought DGRAM sockets > were supposed to drop packets on the floor instead of blocking. But > perhaps I am wrong and glibc is supposed to be explicitly marking the > socket as non-blocking. I consider this a configuration error. I don't believe syslogd should ever do a reverse lookup, since the name you are trying to get may never arrive, or if arrives, it may be spoofed. It's not a bug, but a security feature. NO log to syslogd should be lost, since it may be related to an attack. > Regardless of whose bug it is, I suggest that it needs to be fixed. > To my knowledge, other Unix systems do not behave this way; they > simply drop messages when syslogd is not responding. IMO, that is > correct behavior. Not good security practice. If syslog drops messages then you can lose messages pertaining to an attack. It would be better to disable reverse lookups by syslogd. This might require a recompile of syslogd. - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
[EMAIL PROTECTED] (Patrick J. LoPresti): If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3, kernel 2.2.x), within a few minutes you will find your entire machine grinds to a halt. For example, nobody can log in. This happens because once the /dev/log buffer fills, the syslog() glibc function blocks. This is a problem for us in Real Life because named and syslogd are deadlocking while trying to talk to each other. On our loghost, syslogd needs to do reverse name lookups while named needs to call syslog(). When traffic gets heavy all around, the deadlock is triggered, and it is quite unpleasant. We are about to configure named to use flat files instead of syslog, but that feels more like a workaround than a fix. Don't configure syslogd to do reverse lookups. You can NEVER guarantee that the reverse lookup will succeed, and can be delayed several minutes for a single reply. I am not sure whether this is a glibc bug or a kernel bug. I have used netstat and the glibc sources to confirm that glibc is using a SOCK_DGRAM Unix socket to send to /dev/log. I thought DGRAM sockets were supposed to drop packets on the floor instead of blocking. But perhaps I am wrong and glibc is supposed to be explicitly marking the socket as non-blocking. I consider this a configuration error. I don't believe syslogd should ever do a reverse lookup, since the name you are trying to get may never arrive, or if arrives, it may be spoofed. It's not a bug, but a security feature. NO log to syslogd should be lost, since it may be related to an attack. Regardless of whose bug it is, I suggest that it needs to be fixed. To my knowledge, other Unix systems do not behave this way; they simply drop messages when syslogd is not responding. IMO, that is correct behavior. Not good security practice. If syslog drops messages then you can lose messages pertaining to an attack. It would be better to disable reverse lookups by syslogd. This might require a recompile of syslogd. - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Jesse Pollard [EMAIL PROTECTED] writes: Don't configure syslogd to do reverse lookups. Our syslogd has no option to disable the reverse lookups. You can NEVER guarantee that the reverse lookup will succeed, and can be delayed several minutes for a single reply. Not true. The named on our loghost is authoritative for the reverse mappings for all of the machines which can log there. I consider this a configuration error. I don't believe syslogd should ever do a reverse lookup, since the name you are trying to get may never arrive, or if arrives, it may be spoofed. There *is* no configuration for these tools which gives the behavior you describe, so this is not a "configuration error". It's not a bug, but a security feature. NO log to syslogd should be lost, since it may be related to an attack. Historically, no other Unix system has had reliable syslogging. It would require very defensive programming for syslogd, and that has clearly not been performed. And if this is what GNU/Linux intends, why does glibc use a SOCK_DGRAM socket for communication with syslogd? By definition, such sockets are *unreliable*. If syslog is supposed to be reliable, a different connection type must be used. Your philosophy that "no syslog message should ever be lost" is not necessarily bad. But it is clearly at odds with historical practice, the current glibc syslog() implementation, and the current syslogd itself. It is true that glibc falls back to using SOCK_STREAM if the SOCK_DGRAM connection fails. Does that mean GNU/Linux is expects syslog to be reliable eventually? If so, then my problem is entirely a bug in syslogd and I will report it as such. - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Jesse Pollard [EMAIL PROTECTED] writes: It's not a bug, but a security feature. NO log to syslogd should be lost, since it may be related to an attack. That's exactly the argument I'm always using to turn down change requests like this. If the syslog() function could drop an entry and not send it is easy enough for somebody who has something to hide to overflow syslog() and then do the whatever s/he does not want to be logged. If anything has to be changed it's (as suggested) the configuration or even the implementation of syslogd. Make it robust. -- ---. ,-. 1325 Chesapeake Terrace Ulrich Drepper \,---' \ Sunnyvale, CA 94089 USA Red Hat `--' drepper at redhat.com ` - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Ulrich Drepper [EMAIL PROTECTED] writes: If anything has to be changed it's (as suggested) the configuration or even the implementation of syslogd. Make it robust. OK, but my current syslogd only listens to /dev/log as a SOCK_DGRAM. If I wanted reliable syslogging, it would be listening on it as a SOCK_STREAM. Maybe I care more about performance and backwards compatibility than reliable syslogging. But whatever my reasons, my connection to syslogd is already unreliable and therefore *should not block*. (Could a syslogd listen on /dev/log both as SOCK_DGRAM and as SOCK_STREAM? If so, then your philosophy implies that glibc should be trying SOCK_STREAM *first*, falling back to SOCK_DGRAM for historical compatibility. Either way, when it uses datagrams, it should never block, period.) - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: Ulrich Drepper [EMAIL PROTECTED] writes: If anything has to be changed it's (as suggested) the configuration or even the implementation of syslogd. Make it robust. OK, but my current syslogd only listens to /dev/log as a SOCK_DGRAM. If I wanted reliable syslogging, it would be listening on it as a SOCK_STREAM. Maybe I care more about performance and backwards compatibility than reliable syslogging. But whatever my reasons, my connection to syslogd is already unreliable and therefore *should not block*. You obviously don't understand the communication channel being used. "/dev/log" is a UNIX DOMAIN SOCKET -- AF_UNIX. Datagrams are unreliable for _IP_ (AF_INET). Traffic on an AF_UNIX socket is always reliable. Ok, smarty, go change the syslogd source to open /dev/log as SOCK_STREAM and watch it fail. (syslog wasn't invented yesturday.) I would suggest disabling name resolution for syslog, but that's an ugly option. There's no way to stop a glibc system from doing a DNS query for a reverse lookup. HOWEVER, you can set the DNS timeout to 1 second and set the resolver options to prevent recursion (answer from cache only.) --Ricky PS: Technically, this is not a lockup. syslogd should eventually timeout waiting for the DNS query and go about it's business. Of course, that may be upwards of 45 seconds -- very annoying. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Ricky Beam [EMAIL PROTECTED] writes: I would suggest disabling name resolution for syslog, but that's an ugly option. There's no way to stop a glibc system from doing a DNS query for a reverse lookup. HOWEVER, you can set the DNS timeout to 1 second and set the resolver options to prevent recursion (answer from cache only.) Recursion has nothing to do with it; as I said, the named on this system is itself authoritative for all of the reverse lookups. Turning down the DNS timeout would affect *all* name resolution on the system, right? That is not acceptable. As I said, I already have a workaround, which is to have named log to a flat file. I agree that this is a poor workaround, and the "right fix" is to modify syslogd not to perform blocking operations. My only quibble is that SOCK_DGRAM is an odd transport to use here, even over AF_UNIX. PS: Technically, this is not a lockup. syslogd should eventually timeout waiting for the DNS query and go about it's business. Of course, that may be upwards of 45 seconds -- very annoying. Yes. We are able to log in to the machine eventually and restart the offending processes. But that is little consolation to our users who notice the hang and the fallout afterward. - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: Turning down the DNS timeout would affect *all* name resolution on the system, right? That is not acceptable. You should be able to set it on a per-process basis (via an ENV var.) As I said, I already have a workaround, which is to have named log to a flat file. I agree that this is a poor workaround, and the "right fix" is to modify syslogd not to perform blocking operations. My only quibble is that SOCK_DGRAM is an odd transport to use here, even over AF_UNIX. syslogd isn't the blocker. The syslog functions in glibc being called by named are the problem. Stop named from blocking on syslog writes and the world will be happy again. I've gotta ask what kind of "load" can cause this to happen. And for the record, syslogd shouldn't be doing DNS lookups for things arriving via /dev/log -- that's always the local machine. --Ricky - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Ricky Beam [EMAIL PROTECTED] writes: syslogd isn't the blocker. The syslog functions in glibc being called by named are the problem. Stop named from blocking on syslog writes and the world will be happy again. So I have the glibc maintainer (and others) saying that syslog messages should never be dropped, and you saying that named should be dropping its syslog messages. One more time, from the top: named is calling syslog(), a glibc function. This function *blocks* waiting for delivery to the local syslogd, even though it is using SOCK_DGRAM sockets. There is no option to openlog() or syslog() to get non-blocking behavior (the LOG_NDELAY option means something else entirely). You are effectively suggesting that named should be rewritten not to use the glibc syslog functions at all. That strikes me as the worst suggestion so far; it would be far better for syslogd not to do name lookups. But my syslogd has no option to avoid name lookups; I will submit a request to add one. I've gotta ask what kind of "load" can cause this to happen. And for the record, syslogd shouldn't be doing DNS lookups for things arriving via /dev/log -- that's always the local machine. This particular syslogd also accepts messages from remote hosts. So when there is a lot of syslog traffic, this syslogd talks to named a lot. named occasionally sends messages to syslog. Since syslog pauses waiting for named to respond to name queries, and named blocks waiting for syslog to consume the message, a deadlock is triggered. True, it is not a full deadlock, because the name query times out eventually. But it is bad enough that the system becomes largely non-responsive. - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: So I have the glibc maintainer (and others) saying that syslog messages should never be dropped, and you saying that named should be dropping its syslog messages. No, I didn't say they "should" be dropped but merely that dropping them would fix your problem. Personally, I'd look closely at your setup to determine exactly why this has become a problem. named is being blocked on writing to /dev/log. This should only happen if there is sufficient _local_ syslog traffic to fill the buffer or syslogd has too much remote traffic to ever read from /dev/log. Per chance are you running the name service caching daemon (nscd)? I'd also guess you aren't disabling fsync() for your sysylog files (it's part of the syslog.conf format) -- this is a conciderable drain on syslogd. --Ricky - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
Ricky Beam [EMAIL PROTECTED] writes: Personally, I'd look closely at your setup to determine exactly why this has become a problem. named is being blocked on writing to /dev/log. This should only happen if there is sufficient _local_ syslog traffic to fill the buffer or syslogd has too much remote traffic to ever read from /dev/log. There is a lot of local traffic as well, yes. Lots of local traffic means named eventually finds itself waiting in line to log. Lots of remote traffic means syslogd is trying to talk to named a lot (to do reverse lookups). named waiting in line + syslogd trying to talk to named == deadlock; this is not too hard to see. Once the name resolution times out, you might expect things to become unstuck. But they don't. Perhaps syslogd is not giving higher priority to local messages; if it did, maybe it could recover from the deadlock. But this would not be a reliable solution; the only reliable solution is for syslogd to be independent of any processes which need to talk to it. Per chance are you running the name service caching daemon (nscd)? No. I'd also guess you aren't disabling fsync() for your sysylog files (it's part of the syslog.conf format) -- this is a conciderable drain on syslogd. I see no documentation for such an option in the syslog.conf man page. This is with the current Red Hat 6.2 syslogd (package sysklogd-1.3.31-17). - Pat - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On 23 Oct 2000, Patrick J. LoPresti wrote: Once the name resolution times out, you might expect things to become unstuck. But they don't. Negative. Things have been queued. The deadlock will only go away if the very next message processed is the named local message. And then it would have to process a few more local messages so it wouldn't stall again so soon. Per chance are you running the name service caching daemon (nscd)? No. Please do. That will reduce the ammount of traffic to the name server. I'd also guess you aren't disabling fsync() for your sysylog files (it's part of the syslog.conf format) -- this is a conciderable drain on syslogd. I see no documentation for such an option in the syslog.conf man page. This is with the current Red Hat 6.2 syslogd (package sysklogd-1.3.31-17). It's in the syslogd and syslog.conf man page (sysklogd-1.3.31-16): (syslog.conf) Regular File Typically messages are logged to real files. The file has to be specified with full pathname, beginning with a slash ``/''. You may prefix each entry with the minus ``-'' sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some performance, especially if you run programs that use logging in a very verbose manner. --Ricky PS: as a side note, you can/will lose information even if sync is enabled. (fsync() will not flush metadata so the file is truncated on restart.) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x
On Mon, Oct 23, 2000 at 04:56:26PM -0400, Patrick J. LoPresti wrote: You are effectively suggesting that named should be rewritten not to use the glibc syslog functions at all. That strikes me as the worst suggestion so far; it would be far better for syslogd not to do name lookups. But my syslogd has no option to avoid name lookups; I will submit a request to add one. /etc/nsswitch.conf: hosts: files dns /etc/hosts: ip.of.named.host name.of.named.host ip.of.other.host name.of.other.host Give explicite IP/name mappings for those which you don't want to be looked via the resolver. This is, of course, system-wide, but use it sparingly. /Matti Aarnio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
