Re: [Clamav-users] clamav-milter seems unstable with 0.87
On Fri, 2005-09-23 at 09:52 -0500, R. Steven Rainwater wrote: But I guess the big question now is how can determine for sure if it's a specifically formatted email that's causing the clamav crashes and, if so, how can I capture one of the emails? What platform are you using? -trog signature.asc Description: This is a digitally signed message part ___ http://lurker.clamav.net/list/clamav-users.html
RE: [Clamav-users] clamav-milter seems unstable with 0.87
I would like to add to this I have been having the exact same problems, I have tried all of the latest versions of the milter and all exhibit the same problem, I am running the milter with the following options to try and keep the email flowing. --dont-log-clean --dont-scan-on-error --max-children=32 --dont-wait --external --local I am running the following packages clamav-milter 0.87-0volatile sendmail 8.13.4-3 Thank you, Mark Adrian Coetser [EMAIL PROTECTED] http://www.tux-edo.co.za, http://www.thummb.com cel: +27 76 527 8789 tel: +27 11 805 2076 fax: +27 11 805 2330 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of R. Steven Rainwater Sent: 27 September 2005 01:02 AM To: clamav-users@lists.clamav.net Subject: Re: [Clamav-users] clamav-milter seems unstable with 0.87 On Fri, 2005-09-23 at 22:58, N Fung wrote: --- [EMAIL PROTECTED] wrote: I think that certain emails are crashing clamav-milter or clamd when it reads them, causing the processes to hang. I've noticed that each time we get one of the errors in the log, an additional clamav-milter process gets stuck, so that over time, we collect more and more stuck processes until max children is hit and everything blows up. What errors, if any, are you getting in /path/to/clamd.log? Are you seeing segfaults there? No segfaults or error messages at all in the clamd.log. Clamd.log contains lots of SelfCheck: Database status OK messages but mostly contains entries for viruses that have been found. The errors I'm seeing, which I posted in a previous email, are all in the /var/log/maillog where sendmail and all the milters log their activity. -Steve ___ http://lurker.clamav.net/list/clamav-users.html ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
Mark wanted us to know: ... The result is processes that hang forever on our system. We have a cronjob that does a killall -HUP sendmail every 2 hours. that clears out those hanging processes. That seems a bit crude. :) I wrote a cronjob like that, in Perl; but, based on 'ps' output, it only kills sendmail child processes that have been *idle* for over 10 minutes. Seems nice. I figure if someone can't get a message to me in 2 hours of connect time, I almost certainly don't want said message. But with this blanket kill, you also kill connections that may have just been there for a few seconds, and are still very much alive. Unfortunately it's required to get sendmail to reload the w class. -- Regards... Todd OS X: We've been fighting the It's a mac syndrome with upper management for years now. Lately we've taken to just referring to new mac installations as Unix installations when presenting proposals and updates. For some reason, they have no problem with that. -- /. Linux kernel 2.6.11-12mdksmp 2 users, load average: 1.74, 1.54, 1.53 ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
On Mon, Sep 26, 2005 at 09:47:45AM -0700, Todd Lyons wrote:
But with this blanket kill, you also kill connections that may have just
been there for a few seconds, and are still very much alive.
Unfortunately it's required to get sendmail to reload the w class.
You only need to restart the listening daemon for that. Kids handling
existing connections will exit when the connection terminates, and it's
VERY unlikely that you will receive emails for domains via connections
that were made before you announced said domain (at least, I presume
you add the domains to sendmail before changing the MX records).
It's remotely possible, just very very unlikely.
--
#!perl -wpl # mmfppfmpmmpp mmpffm [EMAIL PROTECTED]
$p=3-2*/[^\W\dmpf_]/i;s.[a-z]{$p}.vec($f=join('',$p-1?chr(sub{$_[0]*9+$_[1]*3+
$_[2]}-(map{/p|f/i+/f/i}split//,$)+97):qw(m p f)[map{((ord$)%32-1)/$_%3}(9,
3,1)]),5,1)='`'lt$;$f.eig;# Jan-Pieter Cornet
___
http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
On Thu, 22 Sep 2005, R. Steven Rainwater wrote: If anyone has any clues or suggestions, I've love to hear them. I'm not convinced that your -T=0 is valid syntax. It should be -T0 or --timeout=0. Using an = with a single-character argument is bad form. You should be running clmilter_watch [1] out of cron, preferably shortly after your freshclam cronjob runs. This can be used to restart clamav-milter only when necessary, rather than restarting it every hour (or whatever) whether it needs it or not. [1] http://www.itg.uiuc.edu/itg_software/clmilter_watch/ One side note. In the past I used smtp-vilter instead of clamav-milter and it had what I think may be a better failure mode. It was designed so that if it failed, sendmail would continue to operated normally, just without virus scanning. I much prefer that mode of failure to forcing sendmail to reject all incoming mail. It would be nice to see such a feature in future version of clamav-milter. That's a function of how you configure your sendmail.mc file. For example, mine is: INPUT_MAIL_FILTER(`clamav', `S=local:/var/run/clamav/clmilter.sock, F=, T=S:4m;R:4m')dnl where the F= tells it that in the event of a milter failure, it should continue processing as if the milter did not exist. There are more details of how to configure clamav-milter intelligently (assuming I'm intelligent ;) on my website: http://www.itg.uiuc.edu/~menscher/clamav_notes_0.87.html Damian Menscher -- -=#| Physics Grad Student SysAdmin @ U Illinois Urbana-Champaign |#=- -=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=- -=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=- -=#| [EMAIL PROTECTED] www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=- -=#| The above opinions are not necessarily those of my employers. |#=- ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
On Mon, 2005-09-26 at 17:46, Damian Menscher wrote: I'm not convinced that your -T=0 is valid syntax. It should be -T0 or --timeout=0. I've switched to using --timeout=0 but it didn't help with the clamav-milter crashes, which are my main problem. You should be running clmilter_watch [1] out of cron [1] http://www.itg.uiuc.edu/itg_software/clmilter_watch/ Thanks, I'm downloading it now. Sounds useful. INPUT_MAIL_FILTER(`clamav', `S=local:/var/run/clamav/clmilter.sock, F=, T=S:4m;R:4m')dnl where the F= tells it that in the event of a milter failure, it should continue processing as if the milter did not exist. Hmmm... My sendmail.mc uses exactly the same thing with the F= but when clamav-milter starts spitting out errors, sendmail begins rejecting all email until the milter is restarted. There are more details of how to configure clamav-milter intelligently (assuming I'm intelligent ;) on my website: http://www.itg.uiuc.edu/~menscher/clamav_notes_0.87.html Thanks, I'll check it out. -Steve ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
On Fri, 2005-09-23 at 22:58, N Fung wrote: --- [EMAIL PROTECTED] wrote: I think that certain emails are crashing clamav-milter or clamd when it reads them, causing the processes to hang. I've noticed that each time we get one of the errors in the log, an additional clamav-milter process gets stuck, so that over time, we collect more and more stuck processes until max children is hit and everything blows up. What errors, if any, are you getting in /path/to/clamd.log? Are you seeing segfaults there? No segfaults or error messages at all in the clamd.log. Clamd.log contains lots of SelfCheck: Database status OK messages but mostly contains entries for viruses that have been found. The errors I'm seeing, which I posted in a previous email, are all in the /var/log/maillog where sendmail and all the milters log their activity. -Steve ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
On Thu, 2005-09-22 at 17:16, Todd Lyons wrote: R. Steven Rainwater wanted us to know: Have you tried running clamd and using --external on clamav-milter? Just tried it. Already had two more crashes in less than 10 minutes! :-( Both were of the write(A) return -1, expected 5: Broken pipe variety, if that means anything. Pick up the max children setting. See if that makes a difference. Watch as the number of processes build up. Thanks Todd, this was the first thing I've tried that helped. Prior to 0.87, we were running max children at 25 and never had problems. I bumped it up to 40 now and that seems to have helped somewhat. We're still getting a couple of the error messages in the log every hour but it seems that overall, clamav-milter is now able to continue longer before sendmail starts rejecting everything. I adjusted my cron job to restart clamav-milter once a day instead of once an hour, so things are relatively stable compared to yesterday. I still find it odd that 0.87 seems so broken. All the previous versions we've run on this machine have been very stable. And I take it this is not a problem anyone else is seeing, making it even more of a mystery. Taking a clue from the max children setting, I started monitoring the processes and I now have a hunch about what's happening. I think that certain emails are crashing clamav-milter or clamd when it reads them, causing the processes to hang. I've noticed that each time we get one of the errors in the log, an additional clamav-milter process gets stuck, so that over time, we collect more and more stuck processes until max children is hit and everything blows up. We occasional get emails that take an hour or so to receive and process. Prior to 0.87, you'd see a sendmail process along with the associated clamav-milter and spamass-milter processes hanging out until it finished. What happens with 0.87 is that the sendmail and spamassasin process go ahead and end at the time of the clamav-milter crash but sometimes the clamav processes seem to stick around forever (until I restart clamav-milter). I've got clamav-milter processes that have been running for over 7 hours even though the associated sendmail process is long gone. But I guess the big question now is how can determine for sure if it's a specifically formatted email that's causing the clamav crashes and, if so, how can I capture one of the emails? Also check dmesg to see if it's reporting weird things such as NMI errors (ie bad memory);. I checked this and nothing unusual is being reported. -Steve ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
On Fri, 23 Sep 2005 09:52:37 -0500 in [EMAIL PROTECTED] R. Steven Rainwater [EMAIL PROTECTED] wrote: I still find it odd that 0.87 seems so broken. All the previous versions we've run on this machine have been very stable. And I take it this is not a problem anyone else is seeing, making it even more of a mystery. Have you checked how many of the responses that say it's OK are for people using RH 7.3? I would not expect it to be a large proportion. -- Brian Morrison bdm at fenrir dot org dot uk GnuPG key ID DE32E5C5 - http://wwwkeys.uk.pgp.net/pgpnet/wwwkeys.html ___ http://lurker.clamav.net/list/clamav-users.html
RE: [Clamav-users] clamav-milter seems unstable with 0.87
R. Steven Rainwater wrote: But I guess the big question now is how can determine for sure if it's a specifically formatted email that's causing the clamav crashes and, if so, how can I capture one of the emails? Hmmm... I know there are some archive milters out there that make a copy of all incoming mail they scan. Maybe that would be useful... you might install one of these archive milters, making sure it appears before clamav-milter in the list of milters... then when a thread goes haywire, check the last few emails in the archive for fishyness. -- Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902 Hispanic Business Inc./HireDiversity.com Software Engineer ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
R. Steven Rainwater wanted us to know: Thanks Todd, this was the first thing I've tried that helped. Prior to 0.87, we were running max children at 25 and never had problems. I bumped it up to 40 now and that seems to have helped somewhat. We're still getting a couple of the error messages in the log every hour but it seems that overall, clamav-milter is now able to continue longer before sendmail starts rejecting everything. I adjusted my cron job to restart clamav-milter once a day instead of once an hour, so things are relatively stable compared to yesterday. We set ours at 200. Your first reaction is No server can run with that kind of load, but the problem is not that there are that many connections coming in at the same time, it's that there are a bunch of cablemodems or dial up links (low upstream bandwidth) that just take forever to send out their messages (they're trying to send out too many for the amount of bandwidth available to it). The result is processes that hang forever on our system. We have a cronjob that does a killall -HUP sendmail every 2 hours. That clears out those hanging processes. I figure if someone can't get a message to me in 2 hours of connect time, I almost certainly don't want said message. -- Regards... Todd There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. --Ed Howdershelt Linux kernel 2.6.11-12mdksmp load average: 1.20, 1.14, 1.15 ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
--- [EMAIL PROTECTED] wrote: From: R. Steven Rainwater [EMAIL PROTECTED] Subject: Re: [Clamav-users] clamav-milter seems unstable with 0.87 To: clamav-users@lists.clamav.net Thanks Todd, this was the first thing I've tried that helped. Prior to 0.87, we were running max children at 25 and never had problems. I bumped it up to 40 now and that seems to have helped somewhat. We're still getting a couple of the error messages in the log every hour but it seems that overall, clamav-milter is now able to continue longer before sendmail starts rejecting everything. I adjusted my cron job to restart clamav-milter once a day instead of once an hour, so things are relatively stable compared to yesterday. I still find it odd that 0.87 seems so broken. All the previous versions we've run on this machine have been very stable. And I take it this is not a problem anyone else is seeing, making it even more of a mystery. Taking a clue from the max children setting, I started monitoring the processes and I now have a hunch about what's happening. I think that certain emails are crashing clamav-milter or clamd when it reads them, causing the processes to hang. I've noticed that each time we get one of the errors in the log, an additional clamav-milter process gets stuck, so that over time, we collect more and more stuck processes until max children is hit and everything blows up. Steve, What errors, if any, are you getting in /path/to/clamd.log? Are you seeing segfaults there? -- N. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
R. Steven Rainwater yazmış: After upgrading one of our servers to clamav 0.87, we are experience intermittent failures of clamav-milter which result sendmail refusing all mail until clamav-milter is restarted. Sometimes it will run for 24 hours or more without crashing but generally it seems to crash every few hours. Typical errors messages we're seeing in the log file at the time of failure look like this: Sep 19 00:53:34 wirrn sendmail[22759]: j8J5nY16022759: Milter (clamav-milter): timeout before data read Sep 19 00:53:34 wirrn sendmail[22759]: j8J5nY16022759: Milter (clamav-milter): to error state Or this: Sep 20 10:26:08 wirrn sendmail[22405]: j8KDA4bF022405: Milter (clamav-milter): write(Q) returned -1, expected 5: Broken pipe Sep 20 10:26:08 wirrn sendmail[22405]: j8KDA4bF022405: Milter (clamav-milter): to error state Or this: Sep 20 10:55:15 wirrn sendmail[27926]: j8KDt1bE027926: Milter (clamav-milter): write(A) returned -1, expected 5: Broken pipe Sep 20 10:55:15 wirrn sendmail[27926]: j8KDt1bE027926: Milter (clamav-milter): to error state Sometimes sendmail starts rejecting all email once we hit just one of these errors. Other times, I'll see a string of dozens of such errors over the course of an hour during which time sendmail seems to be rejecting some email and accepting some email. But eventually, in every case so far, it fails completely and begins rejecting all email. I'm sure it would be helpful to know exactly what I'm running here: OS is Redhat 7.3 fully up to date with all patches from the legacy project. Sendmail-8.12.8-9.1.rh73.dag.i386.rpm clamav-*0.87-1.0.rh7.rf.i386.rpm Contest of /etc/sysconfig/clamav-milter: ### Simple config file for clamav-milter, you should ### read the documentation and tweak it as you wish. ### --quiet CLAMAV_FLAGS= --config-file=/etc/clamd.conf --max-children=20 -T=0 --force-scan --dont-log-clean --noreject --quiet local:/var/clamav/clmilter.socket If other config info would help diagnose it, just let me know what's needed. If anyone has any clues or suggestions, I've love to hear them. One side note. In the past I used smtp-vilter instead of clamav-milter and it had what I think may be a better failure mode. It was designed so that if it failed, sendmail would continue to operated normally, just without virus scanning. I much prefer that mode of failure to forcing sendmail to reject all incoming mail. It would be nice to see such a feature in future version of clamav-milter. -Steve ___ http://lurker.clamav.net/list/clamav-users.html Try with adding --timeout=0 to your CLAMAV_FLAGS variable. Did you upgrade from 0.86.2? -- Cevher Cemal Bozkur +-+-+-+-+-+-+-+-+-+ YÖRE NET Teknoloji Tel:+90 212 234 00 90 ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
On Thu, 2005-09-22 at 11:28, Cevher wrote: R. Steven Rainwater yazmış: After upgrading one of our servers to clamav 0.87, we are experience intermittent failures ... Try with adding --timeout=0 to your CLAMAV_FLAGS variable. According to the man page 0 is the default value for timeout. But my config current has this in it already: -T=0 The man page also indicates that -T=0 should be equivalent to --timeout=0. I'll try changing it from T to timeout and see if that makes any difference. Did you upgrade from 0.86.2? Yes. -Steve ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
Update. I tried changing -T=0 to --timeout=0 as one person suggested but it had no effect. For the moment, I've added a chron job that restarts clamav-milter hourly, which is at least keeping the mail flowing for now. If I can't come up with a solution shortly though, I may need to downgrade back to 0.86. Has anyone else seen anything like this before? I've double checked that there were no other changes to the server or sendmail configuration - just the clamav upgrade. -Steve On Thu, 2005-09-22 at 11:20, R. Steven Rainwater wrote: After upgrading one of our servers to clamav 0.87, we are experience intermittent failures of clamav-milter which result sendmail refusing all mail until clamav-milter is restarted. Sometimes it will run for 24 hours or more without crashing but generally it seems to crash every few hours. Typical errors messages we're seeing in the log file at the time of failure look like this: Sep 19 00:53:34 wirrn sendmail[22759]: j8J5nY16022759: Milter (clamav-milter): timeout before data read Sep 19 00:53:34 wirrn sendmail[22759]: j8J5nY16022759: Milter (clamav-milter): to error state Or this: Sep 20 10:26:08 wirrn sendmail[22405]: j8KDA4bF022405: Milter (clamav-milter): write(Q) returned -1, expected 5: Broken pipe Sep 20 10:26:08 wirrn sendmail[22405]: j8KDA4bF022405: Milter (clamav-milter): to error state Or this: Sep 20 10:55:15 wirrn sendmail[27926]: j8KDt1bE027926: Milter (clamav-milter): write(A) returned -1, expected 5: Broken pipe Sep 20 10:55:15 wirrn sendmail[27926]: j8KDt1bE027926: Milter (clamav-milter): to error state Sometimes sendmail starts rejecting all email once we hit just one of these errors. Other times, I'll see a string of dozens of such errors over the course of an hour during which time sendmail seems to be rejecting some email and accepting some email. But eventually, in every case so far, it fails completely and begins rejecting all email. I'm sure it would be helpful to know exactly what I'm running here: OS is Redhat 7.3 fully up to date with all patches from the legacy project. Sendmail-8.12.8-9.1.rh73.dag.i386.rpm clamav-*0.87-1.0.rh7.rf.i386.rpm Contest of /etc/sysconfig/clamav-milter: ### Simple config file for clamav-milter, you should ### read the documentation and tweak it as you wish. ### --quiet CLAMAV_FLAGS= --config-file=/etc/clamd.conf --max-children=20 -T=0 --force-scan --dont-log-clean --noreject --quiet local:/var/clamav/clmilter.socket If other config info would help diagnose it, just let me know what's needed. If anyone has any clues or suggestions, I've love to hear them. One side note. In the past I used smtp-vilter instead of clamav-milter and it had what I think may be a better failure mode. It was designed so that if it failed, sendmail would continue to operated normally, just without virus scanning. I much prefer that mode of failure to forcing sendmail to reject all incoming mail. It would be nice to see such a feature in future version of clamav-milter. -Steve ___ http://lurker.clamav.net/list/clamav-users.html ___ http://lurker.clamav.net/list/clamav-users.html
RE: [Clamav-users] clamav-milter seems unstable with 0.87
R. Steven Rainwater wrote: Update. I tried changing -T=0 to --timeout=0 as one person suggested but it had no effect. For the moment, I've added a chron job that restarts clamav-milter hourly, which is at least keeping the mail flowing for now. If I can't come up with a solution shortly though, I may need to downgrade back to 0.86. Have you tried running clamd and using --external on clamav-milter? -- Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902 Hispanic Business Inc./HireDiversity.com Software Engineer ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
[EMAIL PROTECTED] wanted us to know: R. Steven Rainwater wrote: Update. I tried changing -T=0 to --timeout=0 as one person suggested but it had no effect. For the moment, I've added a chron job that restarts clamav-milter hourly, which is at least keeping the mail flowing for now. If I can't come up with a solution shortly though, I may need to downgrade back to 0.86. Have you tried running clamd and using --external on clamav-milter? Agreed, that's the way we run it here and don't have any problems with 0.87 yet on a medium volume mail server (approx 50K emails/day). -- Regards... Todd They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. --Benjamin Franklin Linux kernel 2.6.11-12mdksmp 2 users, load average: 2.04, 2.18, 1.98 ___ http://lurker.clamav.net/list/clamav-users.html
RE: [Clamav-users] clamav-milter seems unstable with 0.87
On Thu, 2005-09-22 at 15:22, [EMAIL PROTECTED] wrote: R. Steven Rainwater wrote: Update. I tried changing -T=0 to --timeout=0 as one person suggested but it had no effect. For the moment, I've added a chron job that restarts clamav-milter hourly, which is at least keeping the mail flowing for now. If I can't come up with a solution shortly though, I may need to downgrade back to 0.86. Have you tried running clamd and using --external on clamav-milter? Just tried it. Already had two more crashes in less than 10 minutes! :-( Both were of the write(A) return -1, expected 5: Broken pipe variety, if that means anything. -Steve ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamav-milter seems unstable with 0.87
R. Steven Rainwater wanted us to know: Have you tried running clamd and using --external on clamav-milter? Just tried it. Already had two more crashes in less than 10 minutes! :-( Both were of the write(A) return -1, expected 5: Broken pipe variety, if that means anything. Pick up the max children setting. See if that makes a difference. Watch as the number of processes build up. Also check dmesg to see if it's reporting weird things such as NMI errors (ie bad memory);. -- Regards... Todd Well, it's Karch... --frequently heard after every amazing move he does Linux kernel 2.6.11-12mdksmp load average: 1.34, 1.29, 1.60 ___ http://lurker.clamav.net/list/clamav-users.html
