Re: [Clamav-users] clamscan extremly slow
Paul Kosinski in message 'Re: [Clamav-users] clamscan extremly slow' wrote:
Also, I have noticed that Norton/Symantec, McAfee, CA etc. seem to
include new executable code in their signature updates. Likely they
add special-case code for some new threats, rather than only data.
But I would be very unhappy if clamav added new code on the fly: that
could really open the door to a nastier variety of malware.
in case of commercial scanners this is possible because most of them
run on Windows and on intel platform. Clamav runs on different
architectures so including binary code in daily-signatures is hardly
possible, so don't be afraid ;)
Some time ago I thought about possibility of sending packed source
code of plugins [in signature updates], that could be compiled when
downloaded and used by clamav.
This would allow fight malwares, that detecting them requires some
changes in engine.
But I'm not sure if such a change wouldn't generate too much load on
clamav servers.
cheers, Michał Spadliński
--
main(int a[puts(Michal 'GiM' Spadlinski)]){}
signature.asc
Description: Digital signature
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
When I originally started using clamav, clamscan could handle my low (SOHO) volume of email quite well, but recently, it started taking over 20 secs to scan a short email, and was even showing signs of not keeping up with the spam rate. (My email server is an AMD Sempron 2800+, 1600 MHz, 896 MB RAM, 2.4.x kernel). So I decided to try clamdscan, again. In the past, I had had trouble getting it configured (maybe no listen IP address option back then?), which is why I took the clamscan route, but with 0.90.3, configuration was straightforward. What an incredible improvement! Instead of 20+ secs to scan, it scans normal emails in anywhere from .005 sec to .100 secs. I would guess the average speed up is on the order of 1000 to 1! My only worry now is that either clamd will crash, or stop listening too long when updating. I am using procmail on the tail-end of Postfix's virtual delivery and don't see a way to have procmail get Postfix to try delivery again later (like it would with SMTP delivery), rather than bouncing it back to the sender (not their fault). So in the meantime, I flag the mail as possible virus and write some nasty messages to log files. (In the script my procmailrc calls for scanning, I use netcat to PING clamd to see if it's available.) I think I may set up a cron-driven monitor for clamdscan, to restart it if it dies. I could also set up a delay and retry loop in my scanner script. BTW, I use HAVP with libclamav for Web-page scanning, and it never has had any bad slowness. Paul Kosinski P.S. Clamav may be slower than commercial scanners, however, my observation has been that clamav scans the *entire* file, rather than only part of it, as commercial scanners tend to do. (In some cases, they couldn't even *read* the entire file that fast.) I'm not sure how necessary this is -- in the case of files which are not archives such as zip, tar etc. -- but it *is* more thorough. BTW, when I was using Norton AV some years ago, I had to exclude some zip files from being scanned, as they took far too long. So commercial scanners can be excessively slow too. Also, I have noticed that Norton/Symantec, McAfee, CA etc. seem to include new executable code in their signature updates. Likely they add special-case code for some new threats, rather than only data. But I would be very unhappy if clamav added new code on the fly: that could really open the door to a nastier variety of malware. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Paul Kosinski wrote: My only worry now is that either clamd will crash, or stop listening too long when updating. I am using procmail on the tail-end of Postfix's virtual delivery and don't see a way to have procmail get Postfix to try delivery again later (like it would with SMTP delivery), rather than bouncing it back to the sender (not their fault). Paul, Have a look at monit (http://www.tildeslash.com/monit/). This monitoring tool can trace your services and restart them if necessary. I scan my email directly in Postfix and with monit you can create dependencies: if clamd dies, monit tries to restart both of them. Peter -- http://www.boosten.org ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Mon, 18 Jun 2007, Eric Rostetter wrote: I feel there are good reasons to run clamscan instead of another option, and I feel that one can indeed do so if they have sufficient resources... For perspective, in my environment we'd be talking about a database load time of less than a couple seconds. In a situation where mail volume is low, that's hardly detectable. Another issue is the lack of futzing around with config files, sockets, and many of the other questions that populate this list constantly. I'm not saying that's rocket science, but it's one less thing to worry about, and simplicity has value. Jeffrey Moskot System Administrator [EMAIL PROTECTED] ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Rudd Sent: Tuesday, June 19, 2007 12:10 AM To: ClamAV users ML Subject: Re: [Clamav-users] clamscan extremly slow [...] That, or mail servers that scan their email in bulk batches (like those using mailscanner), where the latency of starting clamscan is MUCH smaller than the latency in going through clamd (I've timed both under mailscanner and mimedefang; under mimedefang, using clamd is a HUGE win, as everyone here expects ... under mailscanner, using clamd is a HUGE loss). Though, the fastest method, for mailscanner, is using the ClamAV perl module for directly processing the messages. This wasn't much of a win under mimedefang though. I assume you are talking about clamd as in clamdscan. The actual clamd (speak directly to the daemon) code is just as fast as clamavmodule, has a slight edge on batches (although I can't test the threading code which might be even faster in batches) and a huge saving on memory. So where is the huge loss in that? Rick -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 John Rudd wrote: [snip] That, or mail servers that scan their email in bulk batches (like those using mailscanner), where the latency of starting clamscan is MUCH smaller than the latency in going through clamd (I've timed both under mailscanner and mimedefang; under mimedefang, using clamd is a HUGE win, as everyone here expects ... under mailscanner, using clamd is a HUGE loss). I don't agree... and many people have reported much improved performance after changing to clamdscan; the new direct connection to clamd is even better. Though, the fastest method, for mailscanner, is using the ClamAV perl module for directly processing the messages. The _fastest_ method for MS is any of the 2: clamavmodule, clamd (on latest beta); then comes using clamdscan, and last, as expected, is clamscan. _Performance_ is more than that, the fast and lower memory use winner is the new direct connection with clamd. Of course there are other properties which are also interesting, like reliability, ease of use, whatever. This wasn't much of a win under mimedefang though. So the real answer here is, as with any non-trivial discussion: it depends. It depends on what you're doing, and how you're doing it. Batching: look toward clamscan or the ClamAV perl module and away from clamd. Interactive/live (such as a milter): look toward clamd. Ultimately, if it _REALLY_ matters to you, don't listen to other people's dogma, actually develop a test suite to figure out which one is truly faster or slower for your situation. I agree with this last phrase, and the tricky part is how to build a good test suite. - -- René Berber -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Cygwin) iD8DBQFGeC6HL3NNweKTRgwRCGrQAKCeWd2IkuwBYaDDQtSU2+t1RYnaNwCg40bi U6dxEYfsMlA3OYH6GvWXw50= =1ZBo -END PGP SIGNATURE- ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Mon, 18 Jun 2007, Peter Boosten wrote: clamdscan solved that issue, although I would have appreciated this effect *before* I upgraded to a newer release. This keeps comming up, perhaps it needs to be addressed in the docs. Could you tell us why you used clamscan instead of clamd/clamdscan in the first place ? I'm just a user, but to me it was obvious. Unfortunatley I can't even recall what documentation I used when I set this up a few years ago. == Chris Candreva -- [EMAIL PROTECTED] -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/ ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Christopher X. Candreva wrote: On Mon, 18 Jun 2007, Peter Boosten wrote: clamdscan solved that issue, although I would have appreciated this effect *before* I upgraded to a newer release. This keeps comming up, perhaps it needs to be addressed in the docs. Could you tell us why you used clamscan instead of clamd/clamdscan in the first place ? I'm just a user, but to me it was obvious. Unfortunatley I can't even recall what documentation I used when I set this up a few years ago. I had some problems running clamd on one of the machines a long time ago, and with mimedefang running clamscan is the second option (which had worked until sometime ago). So I configured mimedefang for clamscan. Now I'm running the daemon and changed mimedefang.pl to run clamdscan in stead of clamscan. Kind regards, Peter -- http://www.boosten.org ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Eric Rostetter wrote: I posted on another list as well, but thought this may gets more attention from the developers: They are well aware of it. Clamscan is extremely slow and CPU hungry. clamscan a pdf file of about 1.2 MB and it takes about 1 minute. Same file with a commercial scanner takes 2 sec. This wasn't always like this. As a result, clamav cannot be used anymore for a mail server. 1) Yes, it is slow. 2) Yes, it wasn't always like this (and hence you could down-grade to an older version if you needed). 3) Newer versions are faster (see below). 4) Yes, it still can be used for a mail server (I know, as I'm still using it). Mandriva 2007.0 on a 800MHZ Duron, 512MB RAM Pretty wimpy mail server by today's standards (due to spam/virus counts). I've not run on a machine like that for years (though that was a nice machine for a mail server just 5-6 years ago). rpm -qa |grep clam clamav-db-0.90.2-0.1mdv2007.0 libclamav2-0.90.2-0.1mdv2007.0 clamav-0.90.2-0.1mdv2007.0 Upgrading to 0.90.3 will probably speed things. When I went from 0.90.2 to 0.90.3 my average mail processing time went from 4 minutes to 2 minutes. Upgrading the the latest RC code will probably make it very fast (seconds instead of minutes) though you have to decide if you can or can not run RC code. And of course, you can do other things like run clamdscan+clamd instead, or put your clamav databases on a RAMDISK to try to speed it up some, etc. But I am always waiting for either the RC code to get released, or for someone reliable to release the RC code as an RPM for my OS versions. It is annoyingly slow, though still usable, as is. I don't have a problem right now. This is my home server and I am using clamd. I was testing the newer clamscan on a test drive where I discovered the slowness. We are running at the office a dual core 3000 AMD and lots of ram. I just wanted to make you aware since there were not many posts on the dist mailing list, although that suddenly has changed. Thomas -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mandriva - http://enigmail.mozdev.org iD8DBQFGdoJqCxGhsefPZLARAmvyAKDDXjwRg9HJJhdXUMXbCmGYHBwX2wCaA2R5 7iPLO+CCz1ZQD272GRYOPLU= =BLH/ -END PGP SIGNATURE- ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Mon, 18 Jun 2007, Peter Boosten wrote: I had some problems running clamd on one of the machines a long time ago, and with mimedefang running clamscan is the second option (which had worked until sometime ago). So I configured mimedefang for clamscan. Maybe it's time to ask the mimedefang people to either remove the clamscam option, or put a big NOT FOR PRODUCTION - FOR TESTING ONLY on it. == Chris Candreva -- [EMAIL PROTECTED] -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/ ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Peter Boosten wrote: Eric Rostetter wrote: 1) Yes, it is slow. 2) Yes, it wasn't always like this (and hence you could down-grade to an older version if you needed). 3) Newer versions are faster (see below). 4) Yes, it still can be used for a mail server (I know, as I'm still using it). The latter point isn't entirely true: we had connections from other MTAs timing out on our mail servers, because of clamscan. clamdscan solved that issue, although I would have appreciated this effect *before* I upgraded to a newer release. Kind regards, Peter Clamscan is a terrible tool to use in real time with email. It has always been a terrible tool. I don't think it has ever been recommended for that role, either. That is why the clamd daemon and the clamav libraries exist, why the clam milter exists, and why clamdscan exists. Clamscan is fine for scanning file systems where long lists of files are scanned with very few processes because of the db loading penalty at each startup, but clamd, which provides the same thing, loads the database files once and can be re-used thousands of times an hour via sockets, streams, and file pointers either directly (direct calls to the socket from your code) or from clamdscan which can be called from scripts. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Quoting Christopher X. Candreva [EMAIL PROTECTED]: On Mon, 18 Jun 2007, Peter Boosten wrote: I had some problems running clamd on one of the machines a long time ago, and with mimedefang running clamscan is the second option (which had worked until sometime ago). So I configured mimedefang for clamscan. Maybe it's time to ask the mimedefang people to either remove the clamscam option, or put a big NOT FOR PRODUCTION - FOR TESTING ONLY on it. Yeah, and the same for clamdscan, since clamd could die and leave you without protection... And the same for the milter too... In fact, why not just label all the code that way. In fact, it isn't even a release verison yet, so why would you use it in production? Anyway, my point is, your millage may vary. Don't try to impose your views on everyone else. -- Eric Rostetter The Department of Physics The University of Texas at Austin Go Longhorns! ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Mon, 18 Jun 2007, Eric Rostetter wrote: Anyway, my point is, your millage may vary. Don't try to impose your views on everyone else. Whoa here. Did you chime and and give a good way to use clamscan on production ? Every time this comes up the answer is don't do it. If that is the answer, then I would think taking steps to avoid this continually comming up would be a good thing. If it ISN'T the answer then lets hear the alternative. Otherwise I don't think I'm imposing MY view on anyway. == Chris Candreva -- [EMAIL PROTECTED] -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/ ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Mon, 18 Jun 2007, Dennis Peterson wrote: Clamscan is a terrible tool to use in real time with email. I would recommend it for low volume servers with cycles to burn, given that the other option is a daemon that can potentially fail. Neither is entirely ideal, but we should take the wide variety of environments into account. Maybe the default recommendation should be clamdscan, but clamscan is not an unreasonable choice in certain circumstances. Jeffrey Moskot System Administrator [EMAIL PROTECTED] ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
jef moskot wrote the following on 6/18/2007 12:19 PM -0800: On Mon, 18 Jun 2007, Dennis Peterson wrote: Clamscan is a terrible tool to use in real time with email. I would recommend it for low volume servers with cycles to burn, given that the other option is a daemon that can potentially fail. Neither is entirely ideal, but we should take the wide variety of environments into account. Maybe the default recommendation should be clamdscan, but clamscan is not an unreasonable choice in certain circumstances. I don't know about other solutions, but amavisd-new allows you to use clamd as your primary scanner and define clamscan as a backup scanner, and it will only call clamscan if the clamd socket fails to respond. Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Jun 18, 2007, at 12:19 PM, jef moskot wrote: On Mon, 18 Jun 2007, Dennis Peterson wrote: Clamscan is a terrible tool to use in real time with email. I would recommend it for low volume servers with cycles to burn, given that the other option is a daemon that can potentially fail. Neither is entirely ideal, but we should take the wide variety of environments into account. Maybe the default recommendation should be clamdscan, but clamscan is not an unreasonable choice in certain circumstances. Note that some of the systems which interface between ClamAV and the MTA, such as Amavisd-new, will use a connection to clamd by preference, but will fall back to invoking clamscan as a secondary scanner if the primary connection to clamd ever fails. In general, the machines which I am running ClamAV on seem to have no problems keeping both clamd and freshclam up and running for months at a time, so if you are experiencing clamd failing often, it's possibly a sign of hardware issues like bad RAM, poor cooling, or a dying/marginal power-supply unit -- -Chuck ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Quoting Christopher X. Candreva [EMAIL PROTECTED]: On Mon, 18 Jun 2007, Eric Rostetter wrote: Anyway, my point is, your millage may vary. Don't try to impose your views on everyone else. Whoa here. Did you chime and and give a good way to use clamscan on production ? Not exactly. But I did say that I am using it in production. Now, if it is a good way or not, that is a subjective matter. Every time this comes up the answer is don't do it. No, every time this comes up the majority answers are don't do it or wait until the next version. I don't remember the authors ever saying it wasn't fit for any use. If that is the answer, Who's answer, to what problem? then I would think taking steps to avoid this continually comming up would be a good thing. Yep, and the authors seem to be doing this as each version since the great slowdown has been faster. If it ISN'T the answer then lets hear the alternative. Otherwise I don't think I'm imposing MY view on anyway. I did propose some alternatives. But in any case, some might be: 1) Live with it. 2) Downgrade. 3) Upgrade. 4) Switch to clamd. 5) Buy a faster machine. 6) Move the DB on RAMDISK, flash, etc. 7) Tune your system for better performance. 8) Switch to clamav-milter. 9) Switch to some other virus scanner. Now, I'm not saying which of the above is the _best_ solution, as that is subjective. Hence, as I said, your milage may vary... == Chris Candreva -- [EMAIL PROTECTED] -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/ ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html -- Eric Rostetter The Department of Physics The University of Texas at Austin Go Longhorns! ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
In message [EMAIL PROTECTED] jef moskot [EMAIL PROTECTED] wrote: On Mon, 18 Jun 2007, Dennis Peterson wrote: Clamscan is a terrible tool to use in real time with email. I would recommend it for low volume servers with cycles to burn, given that the other option is a daemon that can potentially fail. Neither is entirely ideal, but we should take the wide variety of environments into account. You can also detect the daemon's failure and fall back to clamscan in real time, getting the best of both worlds. On my server, if I detect a clamd failure, I fall back to running clamscan in a loop that pauses 10 seconds at a time to let a few messages build up before clamscan runs (in other words, to avoid relaunching clamscan for every message) I haven't seen a clamd failure in many moons though, so I'm not sure the added complexity is worth it. -- Dave Warren, [EMAIL PROTECTED] Office: (403) 775-1700 / (888) 300-3480 ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Monday 18 June 2007 2:35 pm, Dave Warren wrote: In message [EMAIL PROTECTED] jef moskot [EMAIL PROTECTED] wrote: On Mon, 18 Jun 2007, Dennis Peterson wrote: Clamscan is a terrible tool to use in real time with email. I would recommend it for low volume servers with cycles to burn, given that the other option is a daemon that can potentially fail. Neither is entirely ideal, but we should take the wide variety of environments into account. I've been following this thread about clamdscan vs clamscan. I don't run a mailserver, however there are times I need to save an email message and scan it to see if its possibly a new type of virus. Of course running clamscan against this file does take an extremely long time, when trying to use clamdscan however I get the following: [EMAIL PROTECTED] ~]$ clamdscan phish1.txt /home/chris/phish1.txt: Access denied. ERROR --- SCAN SUMMARY --- Infected files: 0 Time: 0.036 sec (0 m 0 s) I can't figure out why I keep getting this Access denied error. Anyone with any ideas? -- Chris KeyID 0xE372A7DA98E6705C pgpTFm0ToFtcG.pgp Description: PGP signature ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Mon, 18 Jun 2007, Chris wrote: [EMAIL PROTECTED] ~]$ clamdscan phish1.txt /home/chris/phish1.txt: Access denied. ERROR I can't figure out why I keep getting this Access denied error. Anyone with any ideas? Because you didn't RTFM. :-) clamdscan passes the file name to clamd, which tries to open it. clamd is normally running as an unprivledged user so unless the file is world readable (or readbale by the clamd process), you get that error Sent the file to STDIN and you solve the problem clamdscan - phish1.txt == Chris Candreva -- [EMAIL PROTECTED] -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/ ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Mon, Jun 18, 2007 at 09:39:23AM -0400, Christopher X. Candreva wrote: On Mon, 18 Jun 2007, Peter Boosten wrote: I had some problems running clamd on one of the machines a long time ago, and with mimedefang running clamscan is the second option (which had worked until sometime ago). So I configured mimedefang for clamscan. Maybe it's time to ask the mimedefang people to either remove the clamscam option, or put a big NOT FOR PRODUCTION - FOR TESTING ONLY on it. clamscan has a purpose. As others have also said - YMMV. A very lightly loaded mailserver (~100 msgs/day) shouldn't have a lot of problems with clamscan. At least not with the 0.88.x version. Besides, mimedefang uses clamscan in case a zip file comes in that clamd is unable to scan, because it is packed with the deflate64 method, which clamd cannot handle. In that case clamscan --unzip is called to scan the file again (at least - clam cannot handle deflate64 up until at least 0.90.3, I haven't checked 0.91rc1 yet). So for anyone upgrading clamav from 0.88.7 to 0.90, the sudden massive drop in performance (about 50% slower scan times, 10-20 times slower startup times for clamd and clamscan) would come as a surprise. The release notes of the 0.90 version of clamav unfortunately fail to mention that performance problem. (To be fair - the scan times have been fixed since 0.90.2 (or 0.90.3 for some platforms), and the startup time appears to be fixed in 0.91rc1. Kudos to the delopers for recognising one of the roots of all evil). So I don't think it's mimedefang that should label the clamscan method as not for production use. -- Jan-Pieter Cornet [EMAIL PROTECTED] !! Disclamer: The addressee of this email is not the intended recipient. !! !! This is only a test of the echelon and data retention systems. Please !! !! archive this message indefinitely to allow verification of the logs. !! ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Monday 18 June 2007 5:04 pm, Christopher X. Candreva wrote: On Mon, 18 Jun 2007, Chris wrote: [EMAIL PROTECTED] ~]$ clamdscan phish1.txt /home/chris/phish1.txt: Access denied. ERROR I can't figure out why I keep getting this Access denied error. Anyone with any ideas? Because you didn't RTFM. :-) clamdscan passes the file name to clamd, which tries to open it. clamd is normally running as an unprivledged user so unless the file is world readable (or readbale by the clamd process), you get that error Sent the file to STDIN and you solve the problem clamdscan - phish1.txt Thanks Chris, guess the Fine Manual wasn't clear enough to me. Chris -- Chris KeyID 0xE372A7DA98E6705C pgpPCxhljQCnJ.pgp Description: PGP signature ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
jef moskot wrote: On Mon, 18 Jun 2007, Dennis Peterson wrote: Clamscan is a terrible tool to use in real time with email. I would recommend it for low volume servers with cycles to burn, given that the other option is a daemon that can potentially fail. Neither is entirely ideal, but we should take the wide variety of environments into account. Maybe the default recommendation should be clamdscan, but clamscan is not an unreasonable choice in certain circumstances. If you can sort out how to restart a heavy weight application for each message there's probably no impediment to figuring out how to restart a daemon should it fail. I did it in just a few lines of shell script run out of cron. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Chris wrote: On Monday 18 June 2007 5:04 pm, Christopher X. Candreva wrote: On Mon, 18 Jun 2007, Chris wrote: [EMAIL PROTECTED] ~]$ clamdscan phish1.txt /home/chris/phish1.txt: Access denied. ERROR I can't figure out why I keep getting this Access denied error. Anyone with any ideas? Because you didn't RTFM. :-) clamdscan passes the file name to clamd, which tries to open it. clamd is normally running as an unprivledged user so unless the file is world readable (or readbale by the clamd process), you get that error Sent the file to STDIN and you solve the problem clamdscan - phish1.txt Thanks Chris, guess the Fine Manual wasn't clear enough to me. There IS another option though. The recommended way to run clamd is to run it as non-root user. Meaning it may have some permission problems if not properly setup. The alternative solutions are : - Use STDIN, which might introduce some overhead (read file - stream - saving stream) - Change clamd's user to match your application (most likely mail server) user - Add clamd's user to app's group, and activating AllowSupplementaryGroups on clamd.conf - Run clamd as root The best choice depends on what you're using it for. Regards, Fajar ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Quoting Jan-Pieter Cornet [EMAIL PROTECTED]: clamscan has a purpose. As others have also said - YMMV. A very lightly loaded mailserver (~100 msgs/day) shouldn't have a lot of problems with clamscan. At least not with the 0.88.x version. We've been using it, and deliver hundreds of thousands of messages a day. So it can be done. That is with 0.88 and with 0.90.2 and 0.90.3. So for anyone upgrading clamav from 0.88.7 to 0.90, the sudden massive drop in performance (about 50% slower scan times, 10-20 times slower startup times for clamd and clamscan) would come as a surprise. The release notes of the 0.90 version of clamav unfortunately fail to mention that performance problem. I'm not sure the authors knew about the drastic performance change until after it was released (though I could be wrong there). (To be fair - the scan times have been fixed since 0.90.2 (or 0.90.3 for some platforms), and the startup time appears to be fixed in 0.91rc1. Kudos to the delopers for recognising one of the roots of all evil). Agreed. So I don't think it's mimedefang that should label the clamscan method as not for production use. It is always up to the user to decide if a pre-1.0 release is ready for production release. The user must except that there will be problems with releases in a pre-1.0 software, whether performance or backwards compatability or other such problems. Having said that, it is some of the best pre-1.0 code I've ever used! -- Eric Rostetter The Department of Physics The University of Texas at Austin Go Longhorns! ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Quoting Dennis Peterson [EMAIL PROTECTED]: Not exactly. But I did say that I am using it in production. Now, if it is a good way or not, that is a subjective matter. Not exactly - it is measurable. And it is really bad. No, it _IS_ subjective, and it depends on your available resources. And in my opinion, with my resources, it is tolerable. Your milage may vary. Do I wish it was faster again? Sure! But can I live with it until it is faster again? Sure! And if not, why not use the newest RC which is fast? Come on folks, this isn't rocket science... dp -- Eric Rostetter The Department of Physics The University of Texas at Austin Go Longhorns! ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Eric Rostetter wrote: Quoting Dennis Peterson [EMAIL PROTECTED]: Not exactly. But I did say that I am using it in production. Now, if it is a good way or not, that is a subjective matter. Not exactly - it is measurable. And it is really bad. No, it _IS_ subjective, and it depends on your available resources. And in my opinion, with my resources, it is tolerable. Your milage may vary. Sorry, no. For any particular machine you can measure the performance of each clamav client and you will get distinctly different performance figures. Clamscan has a startup penalty not found in clamdscan, and while the newest version of clamscan is faster at loading the db files, it is not zero seconds. What is subjective is how one responds to the data. In my case I pass file pointers to clamd from a continuously running milter so there is no startup cost at all. Short of compiling the Clamav libraries straight into the milter as is done with clamav-milter, I don't know of a faster way to scan incoming mail in real time while the connection is still made with the client MTA. If you are waiting until after the MTA has accepted the message but before the handing it LDA to scan then performance is less important. Anyway, you're happy so I'm happy :). dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Quoting Dennis Peterson [EMAIL PROTECTED]: No, it _IS_ subjective, and it depends on your available resources. And in my opinion, with my resources, it is tolerable. Your milage may vary. Sorry, no. For any particular machine you can measure the performance of each clamav client and you will get distinctly different performance figures. Correct. I have a feeling you're trying to make a different point, or reply to a different posting, than I am though. If you want to say that clamscan in 0.90.* is slower than that in 0.88.* then you are correct. If you are trying to say that in general you will get faster performance from clamd+clamdscan than from clamscan on a highly loaded server, than you are correct. My reply is to those who say clamscan is too slow for use, or that clamscan should not be used in production, or that clamscan should never be used on a mail server, or that the developers are not doing their job by making clamscan faster now instead of in the next release. I feel those statements are false. I feel there are good reasons to run clamscan instead of another option, and I feel that one can indeed do so if they have sufficient resources and enough smarts. Clamscan has a startup penalty not found in clamdscan, and Yes. To be fair though, clamscan has the same penality as clamd. But clamd starts less often (one would hope at least). while the newest version of clamscan is faster at loading the db files, it is not zero seconds. What is subjective is how one responds to the data. What is subjective is whether the software is feasible for use on a mail server in the various configurations. In my case I pass file pointers to clamd from a continuously running milter so there is no startup cost at all. Sure there is. You're just moving it from one program to another. Short of compiling the Clamav libraries straight into the milter as is done with clamav-milter, I don't know of a faster way to scan incoming mail in real time while the connection is still made with the client MTA. So? That has nothing to do with the point I was trying to make, which is that you _can_ run a fairly high volume mail server using clamscan (any version) if you have sufficient system resources, and are willing to tolerate slow delivery times (up to 4 minutes on my system, with clamscan on 0.90.3 for example). If you are waiting until after the MTA has accepted the message but before the handing it LDA to scan then performance is less important. Which is a perfectly reasonable thing to do... Anyway, you're happy so I'm happy :). Okay! Happy Happy Happy! :) dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html -- Eric Rostetter The Department of Physics The University of Texas at Austin Go Longhorns! ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Jan-Pieter Cornet wrote: On Mon, Jun 18, 2007 at 09:39:23AM -0400, Christopher X. Candreva wrote: On Mon, 18 Jun 2007, Peter Boosten wrote: I had some problems running clamd on one of the machines a long time ago, and with mimedefang running clamscan is the second option (which had worked until sometime ago). So I configured mimedefang for clamscan. Maybe it's time to ask the mimedefang people to either remove the clamscam option, or put a big NOT FOR PRODUCTION - FOR TESTING ONLY on it. clamscan has a purpose. As others have also said - YMMV. A very lightly loaded mailserver (~100 msgs/day) shouldn't have a lot of problems with clamscan. At least not with the 0.88.x version. That, or mail servers that scan their email in bulk batches (like those using mailscanner), where the latency of starting clamscan is MUCH smaller than the latency in going through clamd (I've timed both under mailscanner and mimedefang; under mimedefang, using clamd is a HUGE win, as everyone here expects ... under mailscanner, using clamd is a HUGE loss). Though, the fastest method, for mailscanner, is using the ClamAV perl module for directly processing the messages. This wasn't much of a win under mimedefang though. So the real answer here is, as with any non-trivial discussion: it depends. It depends on what you're doing, and how you're doing it. Batching: look toward clamscan or the ClamAV perl module and away from clamd. Interactive/live (such as a milter): look toward clamd. Ultimately, if it _REALLY_ matters to you, don't listen to other people's dogma, actually develop a test suite to figure out which one is truly faster or slower for your situation. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Henrik Krohns wrote: On Mon, Jun 18, 2007 at 10:45:30PM -0500, Eric Rostetter wrote: if you have sufficient system resources, and are willing to tolerate slow delivery times (up to 4 minutes on my system, with clamscan on 0.90.3 for example). I'm just amazed by all the nitpicking in this thread. If you worked here and delayed the mail for 4 minutes just because clamscan works fine, I would fire you. :D Nothing personal ofcourse. heh. Nitpicking indeed. If I were working somewhere that was so clueless about how email works that 4 minutes delay was considered unacceptable*, then I'd quit. Nothing personal, I just don't feel it's worth my time to work for people who don't understand how email works. (* questionable? not idea? sure.. unacceptable to the point of firing someone? that's incompetent management) ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Saturday 16 June 2007 19:07, Dennis Peterson wrote: Thomas Spuhler wrote: I posted on another list as well, but thought this may gets more attention from the developers: Clamscan is extremely slow and CPU hungry. clamscan a pdf file of about 1.2 MB and it takes about 1 minute. Same file with a commercial scanner takes 2 sec. This wasn't always like this. As a result, clamav cannot be used anymore for a mail server. Clamav version? OS version? Build options? Server specs? dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html Mandriva 2007.0 on a 800MHZ Duron, 512MB RAM rpm -qa |grep clam clamav-db-0.90.2-0.1mdv2007.0 libclamav2-0.90.2-0.1mdv2007.0 clamav-0.90.2-0.1mdv2007.0 and Mandriva 2007.1 same packages -- Thomas ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On 6/17/07, Thomas Spuhler [EMAIL PROTECTED] wrote: I posted on another list as well, but thought this may gets more attention from the developers: Clamscan is extremely slow and CPU hungry. clamscan a pdf file of about 1.2 MB and it takes about 1 minute. Use clamdscan instead of clamscan. --Edwin ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Sunday 17 June 2007 08:43, Török Edvin wrote: On 6/17/07, Thomas Spuhler [EMAIL PROTECTED] wrote: I posted on another list as well, but thought this may gets more attention from the developers: Clamscan is extremely slow and CPU hungry. clamscan a pdf file of about 1.2 MB and it takes about 1 minute. Use clamdscan instead of clamscan. --Edwin ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html That doesn't improve clamscan. (I can use a free commercial that is really fast) -- Thomas ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
--As of June 17, 2007 11:44:04 AM -0700, Thomas Spuhler is alleged to have said: Use clamdscan instead of clamscan. That doesn't improve clamscan. (I can use a free commercial that is really fast) --As for the rest, it is mine. Your problem is the startup time of clamscan, and has been discussed endlessly on this list. Clamdscan avoids that startup time, by keeping it running. Does your commercial scanner quit when it is finished, or is it calling some background process that stays running? (Like clamdscan does...) More recent (and older...) versions of clamscan also have faster startups, so this is being worked on. Daniel T. Staal --- This email copyright the author. Unless otherwise noted, you are expressly allowed to retransmit, quote, or otherwise use the contents for non-commercial purposes. This copyright will expire 5 years after the author's death, or in 30 years, whichever is longer, unless such a period is in excess of local copyright law. --- ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
On Sunday 17 June 2007 11:52, Daniel Staal wrote: --As of June 17, 2007 11:44:04 AM -0700, Thomas Spuhler is alleged to have said: Use clamdscan instead of clamscan. That doesn't improve clamscan. (I can use a free commercial that is really fast) --As for the rest, it is mine. Your problem is the startup time of clamscan, and has been discussed endlessly on this list. Clamdscan avoids that startup time, by keeping it running. Does your commercial scanner quit when it is finished, or is it calling some background process that stays running? (Like clamdscan does...) More recent (and older...) versions of clamscan also have faster startups, so this is being worked on. Daniel T. Staal --- This email copyright the author. Unless otherwise noted, you are expressly allowed to retransmit, quote, or otherwise use the contents for non-commercial purposes. This copyright will expire 5 years after the author's death, or in 30 years, whichever is longer, unless such a period is in excess of local copyright law. --- ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html Thanks for clarification. I saw a similar thread on the Mandriva cooker mailing list. The commercial antivirus program isn't the demonized. I don't want to list the name on a mailing list. -- Thomas ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Thomas Spuhler Sent: Saturday, June 16, 2007 8:37 PM To: clamav-users@lists.clamav.net Subject: [Clamav-users] clamscan extremly slow I posted on another list as well, but thought this may gets more attention from the developers: Clamscan is extremely slow and CPU hungry. clamscan a pdf file of about 1.2 MB and it takes about 1 minute. Same file with a commercial scanner takes 2 sec. This wasn't always like this. As a result, clamav cannot be used anymore for a mail server. -- If you update to the most recent rc release (I believe 0.91rc1) you will see the improvement you are looking for. Rick -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
I posted on another list as well, but thought this may gets more attention from the developers: They are well aware of it. Clamscan is extremely slow and CPU hungry. clamscan a pdf file of about 1.2 MB and it takes about 1 minute. Same file with a commercial scanner takes 2 sec. This wasn't always like this. As a result, clamav cannot be used anymore for a mail server. 1) Yes, it is slow. 2) Yes, it wasn't always like this (and hence you could down-grade to an older version if you needed). 3) Newer versions are faster (see below). 4) Yes, it still can be used for a mail server (I know, as I'm still using it). Mandriva 2007.0 on a 800MHZ Duron, 512MB RAM Pretty wimpy mail server by today's standards (due to spam/virus counts). I've not run on a machine like that for years (though that was a nice machine for a mail server just 5-6 years ago). rpm -qa |grep clam clamav-db-0.90.2-0.1mdv2007.0 libclamav2-0.90.2-0.1mdv2007.0 clamav-0.90.2-0.1mdv2007.0 Upgrading to 0.90.3 will probably speed things. When I went from 0.90.2 to 0.90.3 my average mail processing time went from 4 minutes to 2 minutes. Upgrading the the latest RC code will probably make it very fast (seconds instead of minutes) though you have to decide if you can or can not run RC code. And of course, you can do other things like run clamdscan+clamd instead, or put your clamav databases on a RAMDISK to try to speed it up some, etc. But I am always waiting for either the RC code to get released, or for someone reliable to release the RC code as an RPM for my OS versions. It is annoyingly slow, though still usable, as is. -- Eric Rostetter The Department of Physics The University of Texas at Austin Go Longhorns! ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Eric Rostetter wrote: 1) Yes, it is slow. 2) Yes, it wasn't always like this (and hence you could down-grade to an older version if you needed). 3) Newer versions are faster (see below). 4) Yes, it still can be used for a mail server (I know, as I'm still using it). The latter point isn't entirely true: we had connections from other MTAs timing out on our mail servers, because of clamscan. clamdscan solved that issue, although I would have appreciated this effect *before* I upgraded to a newer release. Kind regards, Peter -- http://www.boosten.org ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamscan extremly slow
Thomas Spuhler wrote: I posted on another list as well, but thought this may gets more attention from the developers: Clamscan is extremely slow and CPU hungry. clamscan a pdf file of about 1.2 MB and it takes about 1 minute. Same file with a commercial scanner takes 2 sec. This wasn't always like this. As a result, clamav cannot be used anymore for a mail server. Clamav version? OS version? Build options? Server specs? dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
