Re: [clamav-users] Slow reload
On Fri 22/Mar/2019 16:14:28 +0100 Bowie Bailey wrote: > On 3/22/2019 10:54 AM, Bowie Bailey wrote: >> >> The only problem I have found is that >> since my databases take so long to load, avfilter_sig times out when >> reloading the >> virus definitions. Is there a way to increase the timeout for avfilter_sig? > > Nevermind. I updated the hardcoded value in avfilter_sig.c and recompiled. > Maybe > that could be added as an option for the next version? > > $ avfilter_sig -t 300 -l reload virus Committed* Best Ale -- [*] http://www.tana.it/svn/avfilter/trunk/ ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On 3/22/2019 10:54 AM, Bowie Bailey wrote: > > The only problem I have found is that > since my databases take so long to load, avfilter_sig times out when > reloading the > virus definitions. Is there a way to increase the timeout for avfilter_sig? Nevermind. I updated the hardcoded value in avfilter_sig.c and recompiled. Maybe that could be added as an option for the next version? $ avfilter_sig -t 300 -l reload virus Sorry, my C programming skills aren't good enough to offer a patch. -- Bowie ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On 3/22/2019 6:58 AM, Alessandro Vesely via clamav-users wrote: > On Thu 21/Mar/2019 21:21:45 +0100 Bowie Bailey wrote: >> >> And here's the problem. SecuriteInfo has their own Eicar signatures, so >> ClamAV found >> those first and not the one you were expecting. My header looks like this: >> >> ClamAV-Found: SecuriteInfo.com.Eicar-Test-Signature.UNOFFICIAL >> SecuriteInfo.com.Eicar-Test-Signature-4.UNOFFICIAL >> SecuriteInfo.com.Eicar-Test-Signature-2.UNOFFICIAL >> Eicar-Test-Signature.UNOFFICIAL >> SecuriteInfo.com.Eicar-Test-Signature.UNOFFICIAL >> SecuriteInfo.com.Eicar-Test-Signature-4.UNOFFICIAL >> SecuriteInfo.com.Eicar-Test-Signature-2.UNOFFICIAL >> Eicar-Test-Signature.UNOFFICIAL > > Uh, yeah, can be. The small database is made like so: > > test.ndb: > sigtool -f Eicar-Test-Signature > $@ > > where the argument to -f is a regular expression. So the database seems to > contain four matching signatures. Your database (like mine) probably has > much more matching signatures, such as Win.Test.EICAR_NDB-1, whose names > don't match the above regex. Maybe you should limit that regex so it only matches the official ClamAV signature. sigtool -f '^Eicar-Test-Signature' > $@ > >> Not sure why everything is duplicated... >> After testing the sigtool command above, it looks like the duplication comes from matching multiple types of files in the database directory. daily.ndu, daily.ndb, daily.ldu, and daily.ldb all matched an Eicar signature on my system. I don't see a way with sigtool to limit the match to a particular file type without having to specify the directory. Maybe you could just drop all but the first match from the results. (Although I guess it really doesn't matter since it is only used for the test) Something like this: sigtool -f '^Eicar-Test-Signature' | grep -m 1 . > $@ Actually, after testing that, I still get the header duplication with only 1 signature in the database. Oh well... ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL >> No, egrep works fine once the regex is adjusted to match the header. > > Fine. I'll change that command to "egrep -i '^ClamAV-Found: .*Eicar' > eicar.mail". > > Thank you for the fix. If you change how the test database is generated, then you can stick with the original regex. > >> I guess I should have specified that I'm running this on CentOS 7, not that >> it >> matters at this point. >> >> Looks like everything is working now. I'll try integrating it with Courier >> tomorrow. If I just want to reject any email that is flagged by ClamAV, I >> shouldn't >> need to adjust the default config, right? > > Correct, reject is the default. You probably need to set "database" to the > same directory you configured as "DatabaseDirectory" in freshclam.conf. > Also, recall that clamd.conf is not read; please see avfilter.conf(5) if you > need to set clamav options. The database option defaulted to the right location, so I didn't need to adjust that. I did need to go through the other clam options and adjust them to match what I was using with clamd. Oddly, the defaults in clamd aren't quite the same as the defaults in clamscan. Annoyingly, the option names aren't quite the same either... Everything is working now. I have avfilter installed and running on Courier with pythonfilter now just doing the rate limiting. The only problem I have found is that since my databases take so long to load, avfilter_sig times out when reloading the virus definitions. Is there a way to increase the timeout for avfilter_sig? -- Bowie ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On 3/21/2019 3:14 PM, Alessandro Vesely via clamav-users wrote: > On Thu 21/Mar/2019 15:05:59 +0100 Bowie Bailey wrote: >> $ pkg-config --atleast-version=0.101.0 libclamav --print-errors >> Package libclamav was not found in the pkg-config search path. >> Perhaps you should add the directory containing `libclamav.pc' >> to the PKG_CONFIG_PATH environment variable >> No package 'libclamav' found >> >> Once I found that file (in /usr/local/lib64/pkgconfig) and added the >> directory to the >> config path, I was able to complete the configure and make without any >> further errors. > > Hm.. that way pkg-config couldn't find itself? > > >> When I ran "make check", it failed on 11 of 12 tests. I was able to fix >> most of the >> tests by adding '/usr/local/lib64' to the LD_LIBRARY_PATH so it could find >> libclamav.so. I assume I'll need to make a similar change somewhere when I >> add the >> filter to Courier. > > Or you might add it to ld.so.conf? Otherwise, you may try building with > CFLAGS=-Wl,-rpath,/usr/local/lib64 and check ldd. The issue is the ability > to have libraries of different versions simultaneously installed on the same > system. See e.g.: > https://unix.stackexchange.com/questions/356624/why-isnt-usr-local-lib-on-the-library-path-by-default Adding it to ld.so.conf worked once I figured out I had to run ldconfig to load the changes. > > At that point, the top of the header should be plenty of virus_header's (one > for each invocation): > > ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ head eicar.mail > ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL > Old-ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL > Eicar-Test-Signature.UNOFFICIAL > Old-ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL > Eicar-Test-Signature.UNOFFICIAL > Eicar-Test-Signature.UNOFFICIAL > From: aut...@example.com > To: vic...@example.net > Subject: test message > Virus-Header: what does this mean? > MIME-Version: 1.0 > Content-Type: multipart/mixed; boundary="=_1_1553193777_12188" And here's the problem. SecuriteInfo has their own Eicar signatures, so ClamAV found those first and not the one you were expecting. My header looks like this: ClamAV-Found: SecuriteInfo.com.Eicar-Test-Signature.UNOFFICIAL SecuriteInfo.com.Eicar-Test-Signature-4.UNOFFICIAL SecuriteInfo.com.Eicar-Test-Signature-2.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL SecuriteInfo.com.Eicar-Test-Signature.UNOFFICIAL SecuriteInfo.com.Eicar-Test-Signature-4.UNOFFICIAL SecuriteInfo.com.Eicar-Test-Signature-2.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL Not sure why everything is duplicated... > And hence: > > ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ egrep > '^ClamAV-Found: Eicar' eicar.mail | wc -l > 1 > > Instead, at yours wc wrote "0". Why? > >> 9. testsuite.at:540: 9. per-virus behavior (testsuite.at:540): FAILED >> (testsuite.at:612) >> >> Suggestions? >> > I'd guess something must have gone wrong in the testsuite script. In the > testsuite, wc is $WC, after a definition in tests/atlocal, but egrep was not > checked during configure, so maybe it should have been grep -E or similar. > Is that the culprit? No, egrep works fine once the regex is adjusted to match the header. I guess I should have specified that I'm running this on CentOS 7, not that it matters at this point. Looks like everything is working now. I'll try integrating it with Courier tomorrow. If I just want to reject any email that is flagged by ClamAV, I shouldn't need to adjust the default config, right? -- Bowie ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
Hi there, On Thu, 21 Mar 2019, J.R. wrote: > The simplest way to achieve this right now would probably be to use > two servers for scanning ... Or just have the mail server send a 'tempfail' and the remote mail server will retry sending usually within 10 minutes... The OP specifically mentioned Thunderbird, not a remote mail server. You often can't be sure what a mail client will do in this kind of situation. Especially when it's Thunderbird. :( Agreed the tempfail would be fine most of the time, but some people do seem to confuse email with Instant Messaging. Just the other day, in a discussion about email security, a guy grumbled about my greylisting his mail for ten minutes! I decided not to tell him that the default greylist period here is a couple of hours, and just whitelisted him... -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On Wed 20/Mar/2019 21:52:49 +0100 Bowie Bailey wrote: > > I'm trying to build avfilter, but it is complaining that it can't find > libclamav. I > have ClamAV 0.101.1 installed (built from source). I was able to find > libclamav.so > in /usr/local/lib64/. Do I need to specify that directory somewhere, or am I > missing > something else? Do I need to do something special with ClamAV to get the dev > stuff > installed for the build? V.3.5 configure script runs 'pkg-config --atleast-version=0.101.0 libclamav' and complains "libclamav missing or too old" if that fails. pkg-config is also used to determine -l, -L, and -I. Those tests should all succeed with a source built. If you cannot find libclamav at runtime, it may be due to the library path having been moved to /usr/local. In that case, you may have to run ldconfig or adjust LD_LIBRARY_PATH or something to that end. Best Ale -- ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
> The simplest way to achieve this right now would probably be to use > two servers for scanning, and arrange for them to update their DBs > at different times. A simple milter with a knowledge of the update > schedule could choose which scanner to use just by checking the time. > I imagine that it wouldn't be difficult, for example, to modify the > well-known MimeDefang to do that. It wouldn't be much more difficult > to create a milter especially for the purpose. Or just have the mail server send a 'tempfail' and the remote mail server will retry sending usually within 10 minutes... ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On 3/20/2019 3:53 PM, Bowie Bailey wrote: > On 3/20/2019 2:57 PM, Alessandro Vesely via clamav-users wrote: >> On Wed 20/Mar/2019 14:53:28 +0100 Bowie Bailey wrote: >> >>> On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote: On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote: > ClamAV is taking about 2 1/2 minutes to reload its database on my mail > server. This > seems to frequently happen when we are sending an email, so the > Thunderbird will time > out on the send (although the message will frequently go through anyway). The mail server should scan the message with the database at hand. A forked child can do the filtering while the parent reloads. Upon loading, the child exits and new messages will be scanned by the parent with the updated database. >>> That would be ideal, but it doesn't seem to be happening that way. If I >>> look at my >>> logs, I see "SelfCheck", then "Reading databases", and at that point all >>> scanning >>> stops until the "Database correctly reloaded" message 2 1/2 minutes later. >> Are you using clamd directly (rather than libclamav)? How is it connected to >> the mail server? >> >> >>> Is there a setting somewhere to allow scanning to continue with the >>> existing child >>> processes while the reload happens? >> It has to be coded inside the mail filter; avfilter for Courier works that >> way, >> for one. > I am running Courier. I am currently calling clamd via pythonfilter. I'll > have to > take a look at avfilter. I'm trying to build avfilter, but it is complaining that it can't find libclamav. I have ClamAV 0.101.1 installed (built from source). I was able to find libclamav.so in /usr/local/lib64/. Do I need to specify that directory somewhere, or am I missing something else? Do I need to do something special with ClamAV to get the dev stuff installed for the build? -- Bowie ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
Hi there, On Wed, 20 Mar 2019, Micah Snyder wrote: On 3/20/19, 10:04 AM, "clamav-users on behalf of Bowie Bailey" wrote: On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote: On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote: ClamAV is taking about 2 1/2 minutes to reload its database on my mail server. This seems to frequently happen when we are sending an email, so the Thunderbird will time out on the send (although the message will frequently go through anyway). The mail server should scan the message with the database at hand. A forked child can do the filtering while the parent reloads. Upon loading, the child exits and new messages will be scanned by the parent with the updated database. That would be ideal, but it doesn't seem to be happening that way. If I look at my logs, I see "SelfCheck", then "Reading databases", and at that point all scanning stops until the "Database correctly reloaded" message 2 1/2 minutes later. Is there a setting somewhere to allow scanning to continue with the existing child processes while the reload happens? I think Alessandro was suggesting how it could work, not how it does work. Clamd doesn't work that way at present. It has been a feature request for a very long time, one that I hope we can address sometime soon, but I don't know when. The simplest way to achieve this right now would probably be to use two servers for scanning, and arrange for them to update their DBs at different times. A simple milter with a knowledge of the update schedule could choose which scanner to use just by checking the time. I imagine that it wouldn't be difficult, for example, to modify the well-known MimeDefang to do that. It wouldn't be much more difficult to create a milter especially for the purpose. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On 3/20/2019 2:57 PM, Alessandro Vesely via clamav-users wrote: > On Wed 20/Mar/2019 14:53:28 +0100 Bowie Bailey wrote: > >> On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote: >>> On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote: >>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server. This seems to frequently happen when we are sending an email, so the Thunderbird will time out on the send (although the message will frequently go through anyway). >>> The mail server should scan the message with the database at hand. A >>> forked child can do the filtering while the parent reloads. Upon loading, >>> the child exits and new messages will be scanned by the parent with the >>> updated database. >> That would be ideal, but it doesn't seem to be happening that way. If I >> look at my >> logs, I see "SelfCheck", then "Reading databases", and at that point all >> scanning >> stops until the "Database correctly reloaded" message 2 1/2 minutes later. > > Are you using clamd directly (rather than libclamav)? How is it connected to > the mail server? > > >> Is there a setting somewhere to allow scanning to continue with the existing >> child >> processes while the reload happens? > > It has to be coded inside the mail filter; avfilter for Courier works that > way, > for one. I am running Courier. I am currently calling clamd via pythonfilter. I'll have to take a look at avfilter. -- Bowie ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On Wed 20/Mar/2019 14:53:28 +0100 Bowie Bailey wrote: > On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote: >> On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote: >> >>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail >>> server. This >>> seems to frequently happen when we are sending an email, so the Thunderbird >>> will time >>> out on the send (although the message will frequently go through anyway). >> >> The mail server should scan the message with the database at hand. A forked >> child can do the filtering while the parent reloads. Upon loading, the >> child exits and new messages will be scanned by the parent with the updated >> database. > > That would be ideal, but it doesn't seem to be happening that way. If I look > at my > logs, I see "SelfCheck", then "Reading databases", and at that point all > scanning > stops until the "Database correctly reloaded" message 2 1/2 minutes later. Are you using clamd directly (rather than libclamav)? How is it connected to the mail server? > Is there a setting somewhere to allow scanning to continue with the existing > child > processes while the reload happens? It has to be coded inside the mail filter; avfilter for Courier works that way, for one. Best Ale -- ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
I think Alessandro was suggesting how it could work, not how it does work. Clamd doesn't work that way at present. It has been a feature request for a very long time, one that I hope we can address sometime soon, but I don't know when. Micah Micah Snyder ClamAV Development Talos Cisco Systems, Inc. On 3/20/19, 10:04 AM, "clamav-users on behalf of Bowie Bailey" wrote: On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote: > On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote: > >> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server. This >> seems to frequently happen when we are sending an email, so the Thunderbird will time >> out on the send (although the message will frequently go through anyway). > > The mail server should scan the message with the database at hand. A forked child can do the filtering while the parent reloads. Upon loading, the child exits and new messages will be scanned by the parent with the updated database. That would be ideal, but it doesn't seem to be happening that way. If I look at my logs, I see "SelfCheck", then "Reading databases", and at that point all scanning stops until the "Database correctly reloaded" message 2 1/2 minutes later. Is there a setting somewhere to allow scanning to continue with the existing child processes while the reload happens? -- Bowie ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote: > On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote: > >> ClamAV is taking about 2 1/2 minutes to reload its database on my mail >> server. This >> seems to frequently happen when we are sending an email, so the Thunderbird >> will time >> out on the send (although the message will frequently go through anyway). > > The mail server should scan the message with the database at hand. A forked > child can do the filtering while the parent reloads. Upon loading, the child > exits and new messages will be scanned by the parent with the updated > database. That would be ideal, but it doesn't seem to be happening that way. If I look at my logs, I see "SelfCheck", then "Reading databases", and at that point all scanning stops until the "Database correctly reloaded" message 2 1/2 minutes later. Is there a setting somewhere to allow scanning to continue with the existing child processes while the reload happens? -- Bowie ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On 3/20/2019 2:57 AM, Arnaud Jacques wrote: > Hello Bowie, > >> I did a check on the SecuriteInfo signatures. I grepped my clamd logs for >> hits on >> SecuriteInfo signatures and then matched them to the file they came from. >> >> #1 was spam_marketing.ndb with 110 hits >> #2 was javascript.ndb with 10 hits >> >> And that was it. securiteinfo.hdb, securiteinfohtml.hdb, and >> securiteinfoascii.hdb >> did not produce a single hit on my system in the past year. Unfortunately, >> removing >> those signatures does not have much of an effect on the reload times. > Do you have Professional subscription ? > If no : you'll have many more hits if you have Professional subscription. > If yes : you forget some important databases to get more hits. Please login > to you > account. > > Anyway, here are sorted loading time on my system (Intel i7-6700 CPU) using > time > clamscan -d : > > securiteinfo0hour.hdb : 0.021s > securiteinfo.mdb : 0.033s > securiteinfopdf.hdb : 0.047s > securiteinfohtml.hdb : 0.076s > securiteinfoascii.hdb : 0.163s > securiteinfoandroid.hdb : 0.214s > spam_marketing.ndb : 0.332s > securiteinfo.hdb : 0.894s > securiteinfoold.hdb : 6.801s > javascript.ndb : 21.133s > > An antivirus is like any other software : it has minimum requirements. > Every years, it is dozen of thousands malwares in the wild. If we want to > detect > all of them, we have to include them in antivirus databases. > That's why I recommend to get a *fast* CPU and a lot of RAM, because it still > growing past years ! > In my opinion it is a bad idea to run clamAV+our provided databases > signatures on > low performance CPU (VMs, embeeded systems, old hardware, ...) > > Could you please tell us the CPU you use ? No, I don't have a subscription. My system is a bit old. It's running a quad-core Xeon 3Ghz with 4G of RAM. The RAM doesn't seem to be an issue, just the slow loading due to the CPU. It scans a bit slow as well, but that's not an issue since we don't have a high email volume. -- Bowie ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote: > ClamAV is taking about 2 1/2 minutes to reload its database on my mail > server. This > seems to frequently happen when we are sending an email, so the Thunderbird > will time > out on the send (although the message will frequently go through anyway). The mail server should scan the message with the database at hand. A forked child can do the filtering while the parent reloads. Upon loading, the child exits and new messages will be scanned by the parent with the updated database. Best Ale -- https://www.tana.it/sw/avfilter/ ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
All these times, I would imagine, would be based on the amount of CPU and RAM, even disk read speed, available to the machine loading. So these times are relative. Sent from my iPhone > On Mar 20, 2019, at 07:48, Steve Basford > wrote: > >> On 2019-03-19 14:35, Bowie Bailey wrote: >> >> I do have a bunch of third party signatures installed from Sanesecurity and >> SecuriteInfo. Is there a way to get timing information on which >> signature files are >> taking the longest to load? Or is this mainly a function of file size? > > Here's a quick sorted output of scanning each database with 1 file to test db > loading and scan times > > eg. > > clamscan --database=x.xxx > > > Sanesecurity: > > MiscreantPunch099-Low.ldb Execution time 10.534 s > phish.ndb Execution time 4.400 s > scamnailer.ndb Execution time 4.070 s > junk.ndbExecution time 1.840 s > spear.ndb Execution time 1.610 s > blurl.ndb Execution time 1.387 s > scam.ndbExecution time 1.292 s > phishtank.ndb Execution time 1.124 s > jurlbl.ndb Execution time 1.062 s > winnow_extended_malware.hdb Execution time 1.017 s > foxhole_filename.cdbExecution time 1.013 s > MiscreantPunch099-INFO-Low.ldb Execution time 1.007 s > porcupine.ndb Execution time 0.990 s > jurlbla.ndb Execution time 0.974 s > doppelstern.hdb Execution time 0.953 s > doppelstern-phishtank.ndb Execution time 0.945 s > rogue.hdb Execution time 0.945 s > foxhole_all.ndb Execution time 0.937 s > foxhole_js.ndb Execution time 0.932 s > foxhole_generic.cdb Execution time 0.931 s > bofhland_malware_URL.ndbExecution time 0.930 s > winnow.complex.patterns.ldb Execution time 0.928 s > winnow_phish_complete_url.ndb Execution time 0.926 s > foxhole_all.cdb Execution time 0.922 s > lott.ndbExecution time 0.919 s > malware.expert.ndb Execution time 0.919 s > bofhland_cracked_URL.ndbExecution time 0.917 s > foxhole_mail.cdbExecution time 0.917 s > winnow_malware.hdb Execution time 0.917 s > winnow.attachments.hdb Execution time 0.916 s > winnow_bad_cw.hdb Execution time 0.916 s > crdfam.clamav.hdb Execution time 0.915 s > winnow_extended_malware_links.ndb Execution time 0.915 s > spearl.ndb Execution time 0.914 s > badmacro.ndbExecution time 0.913 s > foxhole_js.cdb Execution time 0.913 s > winnow_malware_links.ndbExecution time 0.913 s > winnow_spam_complete.ndbExecution time 0.911 s > bofhland_phishing_URL.ndb Execution time 0.910 s > malware.expert.ldb Execution time 0.910 s > bofhland_malware_attach.hdb Execution time 0.909 s > doppelstern.ndb Execution time 0.909 s > winnow_phish_complete.ndb Execution time 0.908 s > shelter.ldb Execution time 0.907 s > spamattach.hdb Execution time 0.905 s > spamimg.hdb Execution time 0.896 s > malware.expert.hdb Execution time 0.895 s > spam.ldbExecution time 0.889 s > > and the same setup for ClamAV Official DB's: > > daily.ldb Execution time35.893 s > main.mdb Execution time13.474 s > daily.hsb Execution time10.514 s > daily.hdb Execution time3.533 s > main.ndb Execution time2.676 s > main.hsb Execution time2.020 s > daily.mdb Execution time1.610 s > main.hdb Execution time1.054 s > daily.ndb Execution time0.983 s > daily.pdb Execution time0.946 s > daily.ldu Execution time0.942 s > daily.idb Execution time0.940 s > daily.ign2Execution time0.937 s > daily.ign Execution time0.930 s > daily.fp Execution time0.921 s > daily.wdb Execution time0.919 s > daily.ftm Execution time0.918 s > main.sfp Execution time0.914 s > daily.hsu Execution time0.911 s > daily.sfp Execution time0.910 s > daily.cfg Execution time0.909 s > daily.crb Execution time0.907 s > daily.msu Execution time0.906 s > daily.hdu Execution time0.902 s > daily.msb Execution time0.902 s > daily.ndu Execution time0.902 s > daily.cdb Execution time0.901 s > main.msb Execution time0.896 s > main.crb Execution time0.894 s > main.fp
Re: [clamav-users] Slow reload
On 2019-03-19 14:35, Bowie Bailey wrote: I do have a bunch of third party signatures installed from Sanesecurity and SecuriteInfo. Is there a way to get timing information on which signature files are taking the longest to load? Or is this mainly a function of file size? Here's a quick sorted output of scanning each database with 1 file to test db loading and scan times eg. clamscan --database=x.xxx Sanesecurity: MiscreantPunch099-Low.ldb Execution time 10.534 s phish.ndb Execution time 4.400 s scamnailer.ndb Execution time 4.070 s junk.ndbExecution time 1.840 s spear.ndb Execution time 1.610 s blurl.ndb Execution time 1.387 s scam.ndbExecution time 1.292 s phishtank.ndb Execution time 1.124 s jurlbl.ndb Execution time 1.062 s winnow_extended_malware.hdb Execution time 1.017 s foxhole_filename.cdbExecution time 1.013 s MiscreantPunch099-INFO-Low.ldb Execution time 1.007 s porcupine.ndb Execution time 0.990 s jurlbla.ndb Execution time 0.974 s doppelstern.hdb Execution time 0.953 s doppelstern-phishtank.ndb Execution time 0.945 s rogue.hdb Execution time 0.945 s foxhole_all.ndb Execution time 0.937 s foxhole_js.ndb Execution time 0.932 s foxhole_generic.cdb Execution time 0.931 s bofhland_malware_URL.ndbExecution time 0.930 s winnow.complex.patterns.ldb Execution time 0.928 s winnow_phish_complete_url.ndb Execution time 0.926 s foxhole_all.cdb Execution time 0.922 s lott.ndbExecution time 0.919 s malware.expert.ndb Execution time 0.919 s bofhland_cracked_URL.ndbExecution time 0.917 s foxhole_mail.cdbExecution time 0.917 s winnow_malware.hdb Execution time 0.917 s winnow.attachments.hdb Execution time 0.916 s winnow_bad_cw.hdb Execution time 0.916 s crdfam.clamav.hdb Execution time 0.915 s winnow_extended_malware_links.ndb Execution time 0.915 s spearl.ndb Execution time 0.914 s badmacro.ndbExecution time 0.913 s foxhole_js.cdb Execution time 0.913 s winnow_malware_links.ndbExecution time 0.913 s winnow_spam_complete.ndbExecution time 0.911 s bofhland_phishing_URL.ndb Execution time 0.910 s malware.expert.ldb Execution time 0.910 s bofhland_malware_attach.hdb Execution time 0.909 s doppelstern.ndb Execution time 0.909 s winnow_phish_complete.ndb Execution time 0.908 s shelter.ldb Execution time 0.907 s spamattach.hdb Execution time 0.905 s spamimg.hdb Execution time 0.896 s malware.expert.hdb Execution time 0.895 s spam.ldbExecution time 0.889 s and the same setup for ClamAV Official DB's: daily.ldb Execution time35.893 s main.mdb Execution time13.474 s daily.hsb Execution time10.514 s daily.hdb Execution time3.533 s main.ndb Execution time2.676 s main.hsb Execution time2.020 s daily.mdb Execution time1.610 s main.hdb Execution time1.054 s daily.ndb Execution time0.983 s daily.pdb Execution time0.946 s daily.ldu Execution time0.942 s daily.idb Execution time0.940 s daily.ign2Execution time0.937 s daily.ign Execution time0.930 s daily.fp Execution time0.921 s daily.wdb Execution time0.919 s daily.ftm Execution time0.918 s main.sfp Execution time0.914 s daily.hsu Execution time0.911 s daily.sfp Execution time0.910 s daily.cfg Execution time0.909 s daily.crb Execution time0.907 s daily.msu Execution time0.906 s daily.hdu Execution time0.902 s daily.msb Execution time0.902 s daily.ndu Execution time0.902 s daily.cdb Execution time0.901 s main.msb Execution time0.896 s main.crb Execution time0.894 s main.fp Execution time0.891 s daily.mdu Execution time0.889 s Just in case that's useful for anyone ;) -- Cheers, Steve Twitter: @sanesecurity ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
Hello Bowie, I did a check on the SecuriteInfo signatures. I grepped my clamd logs for hits on SecuriteInfo signatures and then matched them to the file they came from. #1 was spam_marketing.ndb with 110 hits #2 was javascript.ndb with 10 hits And that was it. securiteinfo.hdb, securiteinfohtml.hdb, and securiteinfoascii.hdb did not produce a single hit on my system in the past year. Unfortunately, removing those signatures does not have much of an effect on the reload times. Do you have Professional subscription ? If no : you'll have many more hits if you have Professional subscription. If yes : you forget some important databases to get more hits. Please login to you account. Anyway, here are sorted loading time on my system (Intel i7-6700 CPU) using time clamscan -d : securiteinfo0hour.hdb : 0.021s securiteinfo.mdb : 0.033s securiteinfopdf.hdb : 0.047s securiteinfohtml.hdb : 0.076s securiteinfoascii.hdb : 0.163s securiteinfoandroid.hdb : 0.214s spam_marketing.ndb : 0.332s securiteinfo.hdb : 0.894s securiteinfoold.hdb : 6.801s javascript.ndb : 21.133s An antivirus is like any other software : it has minimum requirements. Every years, it is dozen of thousands malwares in the wild. If we want to detect all of them, we have to include them in antivirus databases. That's why I recommend to get a *fast* CPU and a lot of RAM, because it still growing past years ! In my opinion it is a bad idea to run clamAV+our provided databases signatures on low performance CPU (VMs, embeeded systems, old hardware, ...) Could you please tell us the CPU you use ? -- Cordialement / Best regards, Arnaud Jacques Gérant de SecuriteInfo.com Téléphone : +33-(0)3.44.39.76.46 E-mail : a...@securiteinfo.com Site web : https://www.securiteinfo.com Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286 Twitter : @SecuriteInfoCom Securiteinfo.com La Sécurité Informatique - La Sécurité des Informations. 266, rue de Villers 60123 Bonneuil en Valois ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On 19 March 2019 21:01:03 Bowie Bailey wrote: On 3/19/2019 4:27 PM, Bowie Bailey wrote: Is there a way to get the details on how long each file take to load, or do I just have to test them one by one? A very simple per Database scan time test... Sorry not sorted in time order but might help anyway... badmacro.ndb: 984 ms blurl.ndb: 1469 ms bofhland_cracked_URL.ndb: 875 ms bofhland_malware_attach.hdb: 891 ms bofhland_malware_URL.ndb: 891 ms bofhland_phishing_URL.ndb: 890 ms crdfam.clamav.hdb: 891 ms doppelstern-phishtank.ndb: 891 ms doppelstern.hdb: 890 ms doppelstern.ndb: 906 ms foxhole_all.cdb: 891 ms foxhole_all.ndb: 921 ms foxhole_filename.cdb: 1000 ms foxhole_generic.cdb: 907 ms foxhole_js.cdb: 890 ms foxhole_js.ndb: 875 ms foxhole_mail.cdb: 891 ms junk.ndb: 1812 ms jurlbl.ndb: 1047 ms jurlbla.ndb: 937 ms lott.ndb: 890 ms malware.expert.hdb: 875 ms malware.expert.ldb: 875 ms malware.expert.ndb: 906 ms MiscreantPunch099-INFO-Low.ldb: 953 ms MiscreantPunch099-Low.ldb: Possible Performance Issue: 10422 ms phish.ndb: 4406 ms phishtank.ndb: 1156 ms porcupine.ndb: 969 ms rogue.hdb: 922 ms scam.ndb: 1235 ms scamnailer.ndb: 4016 ms shelter.ldb: 906 ms spam.ldb: 922 ms spamattach.hdb: 906 ms spamimg.hdb: 891 ms spear.ndb: 1594 ms spearl.ndb: 906 ms winnow.attachments.hdb: 891 ms winnow.complex.patterns.ldb: 891 ms winnow_bad_cw.hdb: 891 ms winnow_extended_malware.hdb: 984 ms winnow_extended_malware_links.ndb: 891 ms winnow_malware.hdb: 907 ms winnow_malware_links.ndb: 890 ms winnow_phish_complete.ndb: 890 ms winnow_phish_complete_url.ndb: 875 ms winnow_spam_complete.ndb: 875 ms Cheers, Steve Twitter: @sanesecurity ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On 3/19/2019 4:27 PM, Bowie Bailey wrote: > On 3/19/2019 10:44 AM, Arnaud Jacques wrote: >> Hello Bowie, >> >> Le 19/03/2019 à 15:35, Bowie Bailey a écrit : >>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail >>> server. This >>> seems to frequently happen when we are sending an email, so the Thunderbird >>> will time >>> out on the send (although the message will frequently go through anyway). >>> >>> I do have a bunch of third party signatures installed from Sanesecurity and >>> SecuriteInfo. Is there a way to get timing information on which signature >>> files are >>> taking the longest to load? Or is this mainly a function of file size? >> javascript.ndb is taking most of the reload time. >> You can test the time to reload without this file > Without javascript.ndb, the reload time drops to 2 minutes. Better, but > still longer > than I'd like. > > Is there a way to get the details on how long each file take to load, or do I > just > have to test them one by one? I did a check on the SecuriteInfo signatures. I grepped my clamd logs for hits on SecuriteInfo signatures and then matched them to the file they came from. #1 was spam_marketing.ndb with 110 hits #2 was javascript.ndb with 10 hits And that was it. securiteinfo.hdb, securiteinfohtml.hdb, and securiteinfoascii.hdb did not produce a single hit on my system in the past year. Unfortunately, removing those signatures does not have much of an effect on the reload times. -- Bowie ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
On 3/19/2019 10:44 AM, Arnaud Jacques wrote: > Hello Bowie, > > Le 19/03/2019 à 15:35, Bowie Bailey a écrit : >> ClamAV is taking about 2 1/2 minutes to reload its database on my mail >> server. This >> seems to frequently happen when we are sending an email, so the Thunderbird >> will time >> out on the send (although the message will frequently go through anyway). >> >> I do have a bunch of third party signatures installed from Sanesecurity and >> SecuriteInfo. Is there a way to get timing information on which signature >> files are >> taking the longest to load? Or is this mainly a function of file size? > > javascript.ndb is taking most of the reload time. > You can test the time to reload without this file Without javascript.ndb, the reload time drops to 2 minutes. Better, but still longer than I'd like. Is there a way to get the details on how long each file take to load, or do I just have to test them one by one? -- Bowie ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Slow reload
Hello Bowie, Le 19/03/2019 à 15:35, Bowie Bailey a écrit : ClamAV is taking about 2 1/2 minutes to reload its database on my mail server. This seems to frequently happen when we are sending an email, so the Thunderbird will time out on the send (although the message will frequently go through anyway). I do have a bunch of third party signatures installed from Sanesecurity and SecuriteInfo. Is there a way to get timing information on which signature files are taking the longest to load? Or is this mainly a function of file size? javascript.ndb is taking most of the reload time. You can test the time to reload without this file. -- Cordialement / Best regards, Arnaud Jacques Gérant de SecuriteInfo.com Téléphone : +33-(0)3.44.39.76.46 E-mail : a...@securiteinfo.com Site web : https://www.securiteinfo.com Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286 Twitter : @SecuriteInfoCom Securiteinfo.com La Sécurité Informatique - La Sécurité des Informations. 266, rue de Villers 60123 Bonneuil en Valois ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
