Bug#399930: logrotation race condition with exim writing to logs
Package: exim4 Version: 4.72-6+squeeze3 Severity: normal Hello, we've got hit by this race condition today. I find it ridiculous that this bug is not fixed for 6 years, while the fix is trivial. So far only lame excuses about hypothetical monitoring software that could break if it cannot find a log file. WTF? Is there such a lame monitoring that prefers to check the log instead of TCP socket? Or did you mean something like logcheck? Whatever, running SMTP server is the top priority. Without that even the monitoring itself is worthless, as it cannot send any warnings. So please fix this, upload it to unstable, and we'll see if something else will break. Regards Vladislav Kurz -- Package-specific info: Exim version 4.72 #1 built 25-Oct-2012 18:35:08 Copyright (c) University of Cambridge, 1995 - 2007 Berkeley DB: Berkeley DB 4.8.30: (April 9, 2010) Support for: crypteq iconv() IPv6 GnuTLS move_frozen_messages DKIM Lookups: lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmnz dnsdb dsearch nis nis0 passwd Authenticators: cram_md5 plaintext Routers: accept dnslookup ipliteral manualroute queryprogram redirect Transports: appendfile/maildir/mailstore autoreply lmtp pipe smtp Fixed never_users: 0 Size of off_t: 8 GnuTLS compile-time version: 2.8.6 GnuTLS runtime version: 2.8.6 Configuration file is /var/lib/exim4/config.autogenerated -- System Information: Debian Release: 6.0.6 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: i386 (i686) Kernel: Linux 2.6.32-5-686-bigmem (SMP w/8 CPU cores) Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) Shell: /bin/sh linked to /bin/dash Versions of packages exim4 depends on: ii debconf [debconf-2.0]1.5.36.1Debian configuration management sy ii exim4-base 4.72-6+squeeze3 support files for all Exim MTA (v4 ii exim4-daemon-light 4.72-6+squeeze3 lightweight Exim MTA (v4) daemon exim4 recommends no packages. exim4 suggests no packages. -- debconf information excluded -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#399930: logrotation race condition with exim writing to logs
On Fri, Nov 24, 2006 at 01:39:55PM +0100, Marc Haber wrote: On Fri, Nov 24, 2006 at 11:38:30PM +1100, CaT wrote: Testing as to wether or not the logfile exists does not test wether exim is functioning. It's the logrotation that creates the logfile and so a test for its presence is a test for logrotation and nothing more. As such you need to have your monitoring system test further in order to report the right thing. There is no such your monitoring system. This package is installed on thousands of systems, and some of them are doing monitoring, which you classify as broken, because they have been relying on the log file existing and have not special cased the case where the log file does not exist. Having just been hit by this at work, I'm wondering if there is any compromise to be made here. Could you provide any additional information about what these log monitoring systems are, so that we can see if there is a way to fix them? Cheers, Dominic. -- Dominic Hargreaves | http://www.larted.org.uk/~dom/ PGP key 5178E2A5 from the.earth.li (keyserver,web,email) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#399930: logrotation race condition with exim writing to logs
block #399930 with #400198 thanks On Fri, Nov 24, 2006 at 01:15:01PM +0100, Marc Haber wrote: This might as well be a logrotate bug which I plan to investigate in due time. I have filed this as a bug in logrotate, #400198. I have additionally opened exim wishlist item #418 (http://www.exim.org/bugzilla/show_bug.cgi?id=418), where I ask for an exim option to write to a log file to be created. This option could be abused to force exim to generate the log file in a postrotate script. Other than this and listing a manual change to logrotate.d/exim4-base as a possible workaround, I do not plan to do anything more about this. Greetings Marc -- - Marc Haber | I don't trust Computers. They | Mailadresse im Header Mannheim, Germany | lose things.Winona Ryder | Fon: *49 621 72739834 Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#399930: logrotation race condition with exim writing to logs
On Fri, Nov 24, 2006 at 05:11:25PM +1100, CaT wrote: So there is absolutely no need for the create option in logrotate Not having the logs created might break monitoring mechanisms. Greetings Marc -- - Marc Haber | I don't trust Computers. They | Mailadresse im Header Mannheim, Germany | lose things.Winona Ryder | Fon: *49 621 72739834 Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#399930: logrotation race condition with exim writing to logs
On Fri, Nov 24, 2006 at 10:23:38AM +0100, Marc Haber wrote: On Fri, Nov 24, 2006 at 05:11:25PM +1100, CaT wrote: So there is absolutely no need for the create option in logrotate Not having the logs created might break monitoring mechanisms. Well it is kind of a choice between maybe breaking poorly written monitoring schemes (the lack of a log should merely be a hint to check deeper rather then be the be-all and end-all of ones decision making) and definately breaking the very system that they are meant to monitor. My personal vote is for keeping the system running. Breaken monitoring is a minor annoyance that causes no harm and should only happen when the monitoring is really broken anyway. A broken mail server can well lead to massive tear-shedding when it goes down suddenly and abruptly. For me this meant manual babysitting of the reinjection of mail back from the backup MXes into the virus filter (which wound up getting seriously stressed) and then into the mail storage server (which just plain broke as it needs replacement). This was further aggrivated as it took me away from the task of building the gruntier, better, faster replacement for the mail storage server. My Wednesday sucked. Our customers Wednesday suck. Our support staffs Wednesday sucked. It just plain all sucked. In comparison to that an inapporpriate page/email/sms/whatever from a monitoring system that doesn't check things well enough to indcate a server is down when it isn't is just a minor joykill at worst. Sorry if I seem a bit heavy about this but, well, it just all plain sucked. :/ -- To the extent that we overreact, we proffer the terrorists the greatest tribute. - High Court Judge Michael Kirby -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#399930: logrotation race condition with exim writing to logs
On Fri, Nov 24, 2006 at 01:15:01PM +0100, Marc Haber wrote: On Fri, Nov 24, 2006 at 10:58:28PM +1100, CaT wrote: On Fri, Nov 24, 2006 at 10:23:38AM +0100, Marc Haber wrote: On Fri, Nov 24, 2006 at 05:11:25PM +1100, CaT wrote: So there is absolutely no need for the create option in logrotate Not having the logs created might break monitoring mechanisms. Well it is kind of a choice between maybe breaking poorly written monitoring schemes (the lack of a log should merely be a hint to check deeper rather then be the be-all and end-all of ones decision making) and definately breaking the very system that they are meant to monitor. I consider this race condition to be minor, since it obviously only happens on very heavily loaded systems, which should have a competent Well not really. It's just more likely to happen on a heavily loaded system due to the increased number of msgs it processes. It can happen with any system. It only takes one message to come in at the right time. system admin around to isolate the issue. You are always free to And an on-call monkey who actually gives you a call. :/ As with all problems that cause tears they're mostlikely to occur when you're asleep. change the configuration yourself, /etc/logrotate.d/exim4-base is a dpkg-conffile. I'm already bulding my own debs for this, which is annoying as I'm not doing this to add features but to fix some brokenness. The thing is, there's no need to precreate the logfile (broken monitoring systems aside) and doing so can break things and has broken things. I'd say the opposite tact should be taken. If you are relying on a monitoring system that relies on the logfile always being there then have it pre-created and deal with the race condition of the mintor slot where there is no logfile between the move of the old and the creation of the new. The priority should be to make sure the things the exim package setup do not cause exim to die. Other things should take care of themselves. -- To the extent that we overreact, we proffer the terrorists the greatest tribute. - High Court Judge Michael Kirby -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#399930: logrotation race condition with exim writing to logs
On Fri, Nov 24, 2006 at 10:58:28PM +1100, CaT wrote: On Fri, Nov 24, 2006 at 10:23:38AM +0100, Marc Haber wrote: On Fri, Nov 24, 2006 at 05:11:25PM +1100, CaT wrote: So there is absolutely no need for the create option in logrotate Not having the logs created might break monitoring mechanisms. Well it is kind of a choice between maybe breaking poorly written monitoring schemes (the lack of a log should merely be a hint to check deeper rather then be the be-all and end-all of ones decision making) and definately breaking the very system that they are meant to monitor. I consider this race condition to be minor, since it obviously only happens on very heavily loaded systems, which should have a competent system admin around to isolate the issue. You are always free to change the configuration yourself, /etc/logrotate.d/exim4-base is a dpkg-conffile. This might as well be a logrotate bug which I plan to investigate in due time. Greetings Marc -- - Marc Haber | I don't trust Computers. They | Mailadresse im Header Mannheim, Germany | lose things.Winona Ryder | Fon: *49 621 72739834 Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#399930: logrotation race condition with exim writing to logs
On Fri, Nov 24, 2006 at 11:26:33PM +1100, CaT wrote: On Fri, Nov 24, 2006 at 01:15:01PM +0100, Marc Haber wrote: On Fri, Nov 24, 2006 at 10:58:28PM +1100, CaT wrote: On Fri, Nov 24, 2006 at 10:23:38AM +0100, Marc Haber wrote: On Fri, Nov 24, 2006 at 05:11:25PM +1100, CaT wrote: So there is absolutely no need for the create option in logrotate Not having the logs created might break monitoring mechanisms. Well it is kind of a choice between maybe breaking poorly written monitoring schemes (the lack of a log should merely be a hint to check deeper rather then be the be-all and end-all of ones decision making) and definately breaking the very system that they are meant to monitor. I consider this race condition to be minor, since it obviously only happens on very heavily loaded systems, which should have a competent Well not really. It's just more likely to happen on a heavily loaded system due to the increased number of msgs it processes. It can happen with any system. It only takes one message to come in at the right time. I know. It happens rarely. This setup has been in place for years, even with exim 3. system admin around to isolate the issue. You are always free to And an on-call monkey who actually gives you a call. :/ As with all problems that cause tears they're mostlikely to occur when you're asleep. That's your personal disfortune. How much are you paying for this MTA software? change the configuration yourself, /etc/logrotate.d/exim4-base is a dpkg-conffile. I'm already bulding my own debs for this, which is annoying as I'm not doing this to add features but to fix some brokenness. You are building your own debs to change a single dpkg-conffile? Excuse me? Are you aware what foo is a dpkg-conffile means? The thing is, there's no need to precreate the logfile (broken monitoring systems aside) and doing so can break things and has broken things. I'd say the opposite tact should be taken. If you are relying on a monitoring system that relies on the logfile always being there then have it pre-created and deal with the race condition of the mintor slot where there is no logfile between the move of the old and the creation of the new. The priority should be to make sure the things the exim package setup do not cause exim to die. Other things should take care of themselves. I do not intend to change logrotate configuration at this time of the release process. If you disagree, please take this to debian-devel or the tech ctte. Greetings Marc -- - Marc Haber | I don't trust Computers. They | Mailadresse im Header Mannheim, Germany | lose things.Winona Ryder | Fon: *49 621 72739834 Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#399930: logrotation race condition with exim writing to logs
On Fri, Nov 24, 2006 at 10:23:38AM +0100, Marc Haber wrote: On Fri, Nov 24, 2006 at 05:11:25PM +1100, CaT wrote: So there is absolutely no need for the create option in logrotate Not having the logs created might break monitoring mechanisms. Sorry, the other thing about this: Testing as to wether or not the logfile exists does not test wether exim is functioning. It's the logrotation that creates the logfile and so a test for its presence is a test for logrotation and nothing more. As such you need to have your monitoring system test further in order to report the right thing. Testing for logfile deltas is not effected by having the logrotation mechanism create (or not create) the logfile as you are testing for changes from one check to the next. Not having a change in the logfile does not indicate a service is down though. It could just indicate things are quiet and so it's just a hint for other things. Testing the content of the logfile for certain clues as to status is not effected by the logfiles lack either. You still have to check to see what that means. Is the service down or did the logrotation just fail in a bizarre way. In short there's nothing really gained by precreating the logfile except a small chance for pain. -- To the extent that we overreact, we proffer the terrorists the greatest tribute. - High Court Judge Michael Kirby -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#399930: logrotation race condition with exim writing to logs
On Fri, Nov 24, 2006 at 11:38:30PM +1100, CaT wrote: Testing as to wether or not the logfile exists does not test wether exim is functioning. It's the logrotation that creates the logfile and so a test for its presence is a test for logrotation and nothing more. As such you need to have your monitoring system test further in order to report the right thing. There is no such your monitoring system. This package is installed on thousands of systems, and some of them are doing monitoring, which you classify as broken, because they have been relying on the log file existing and have not special cased the case where the log file does not exist. I am not going to break these setups without thinking about it. [rest of sermon deleted] In short there's nothing really gained by precreating the logfile except a small chance for pain. I happen to disagree. Greetings Marc -- - Marc Haber | I don't trust Computers. They | Mailadresse im Header Mannheim, Germany | lose things.Winona Ryder | Fon: *49 621 72739834 Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#399930: logrotation race condition with exim writing to logs
Actually, the permissions on the exim created logfile will be correct seeing as #-- # The log files themselves are created as required, with a mode that # defaults # to 0640, but which can be changed here. # LOG_MODE=0640 So there is absolutely no need for the create option in logrotate (but seeing as 'create' is set by default it should be replaced with nocreate). -- To the extent that we overreact, we proffer the terrorists the greatest tribute. - High Court Judge Michael Kirby -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
