Re: [clamav-users] DB update and clamav-milter delay

[Andreas Schulze]

Am 29.09.2015 09:31 schrieb Marco:
> With this installation, after every freshclam update session, clamd is
> forced to read the DB:

I had a similar issue. I call freshclam once a minute and the db reload 
generate a significant load.
I fixed it with ithe attached patch.

-- 
Andreas Schulze
Internetdienste | P252

DATEV eG
90329 Nürnberg | Telefon +49 911 319-0 | Telefax +49 911 319-3196
E-Mail i...@datev.de | Internet www.datev.de
Sitz: 90429 Nürnberg, Paumgartnerstr. 6-14 | Registergericht Nürnberg, GenReg 
Nr.70

Vorstand
Prof. Dieter Kempf (Vorsitzender)
Dr. Robert Mayr (stellv. Vorsitzender)
Eckhard Schwarzer (stellv. Vorsitzender)
Dr. Peter Krug
Jörg Rabe von Pappenheim

Vorsitzender des Aufsichtsrates: Dirk Schmale
Description: freshclam should reload the db only on update
Author: A. Schulze
---
This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
Index: clamav-0.98.7/freshclam/manager.c
===
--- clamav-0.98.7.orig/freshclam/manager.c
+++ clamav-0.98.7/freshclam/manager.c
@@ -2861,7 +2861,7 @@ downloadmanager (const struct optstruct
 
 cli_rmdirs (updtmpdir);
 
-if (checkdbdir () < 0)
+if (updated && checkdbdir () < 0)
 {
 if (newver)
 free (newver);
___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Al Varnell]

You are admitting that on a ClamAV mail list?

From the ClamAV Virus Database documentation 
:

>   • How many times per hour shall I run freshclam?
> You can check for database update as often as 4 times per hour provided that 
> you have the following options in freshclam.conf: DNSDatabaseInfo 
> current.cvd.clamav.netDatabaseMirror db.XY.clamav.net DatabaseMirror 
> database.clamav.net 
> Replace XY with your country code. If you don’t have that option, then you 
> must stick with 1 check per hour.

-Al-

On Tue, Sep 29, 2015 at 01:03 AM, Andreas Schulze wrote:
> 
> Am 29.09.2015 09:31 schrieb Marco:
>> With this installation, after every freshclam update session, clamd is
>> forced to read the DB:
> 
> I had a similar issue. I call freshclam once a minute and the db reload 
> generate a significant load.
> I fixed it with ithe attached patch.



smime.p7s
Description: S/MIME cryptographic signature
___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Marco]

Hello,

 I installed clamd server (0.98.7) with clamav-milter using RPM of EPEL.

With this installation, after every freshclam update session, clamd  
is forced to read the DB:


2015-09-29T09:12:41.244383+02:00 av1 clamd[15201]: Reading databases  
from /var/lib/clamav
2015-09-29T09:13:14.950256+02:00 av2 clamav-milter[11957]: Failed to  
stream to clamd

2015-09-29T09:13:14.950546+02:00 av2 clamav-milter[11957]: Streaming failed
2015-09-29T09:13:20.593439+02:00 av1 clamd[15201]: Database  
correctly reloaded (5342538 signatures)


During this time clamav-milter have a trouble and the SMTP server  
experiences a delay receiving the mail. On a frontend MSA SMTP  
server this delay is a problem.


I would like to know if there is a better way to configure the DB  
update, without interruption on service.
In case I remove the reload, is SelfCheck of Clamd sufficient to  
detect a change in DB?

Do you have any hint to improve performances?


I tried to relax the upgrades, but I see the problem is the time spent  
to reload the db:


2015-09-29T01:03:15.710526+02:00 av2 clamd[15201]: Reading databases  
from /var/lib/clamav
2015-09-29T01:03:53.151179+02:00 av2 clamd[15201]: Database correctly  
reloaded (5342845 signatures)


Is there a way to speed up this phase? Maybe putting the db files into  
a RAM fs?


Thank you very much
Marco

___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Kris Deugau]

Marco wrote:
> Hello,
> 
>  I installed clamd server (0.98.7) with clamav-milter using RPM of EPEL.
> 
> With this installation, after every freshclam update session, clamd is
> forced to read the DB:
> 
> 2015-09-29T09:12:41.244383+02:00 av1 clamd[15201]: Reading databases
> from /var/lib/clamav
> 2015-09-29T09:13:14.950256+02:00 av2 clamav-milter[11957]: Failed to
> stream to clamd
> 2015-09-29T09:13:14.950546+02:00 av2 clamav-milter[11957]: Streaming failed
> 2015-09-29T09:13:20.593439+02:00 av1 clamd[15201]: Database correctly
> reloaded (5342538 signatures)
> 
> During this time clamav-milter have a trouble and the SMTP server
> experiences a delay receiving the mail. On a frontend MSA SMTP server
> this delay is a problem.
> 
> I would like to know if there is a better way to configure the DB
> update, without interruption on service.

Short of a patch to clamd to not discard the existing in-memory
signature data while reloading fresh data (or somewhat more complex;
replace signatures one-by-one during a reload), you are stuck with clamd
not responding for 2-5 seconds with the stock signatures, and possibly
up to 30 seconds even on modern hardware if you're using a lot of
third-party signatures.

This seems to come up every couple of months at least and it's been this
way for quite a long time;  is there any upstream interest in fixing
clamd one way or another so it doesn't suffer this outage while
reloading signatures?

-kgd
___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Marco]

Hi Marco,

Are the signatures official ClamAV ones only or are you using 3rd Party
ones as well (if so, could you list them)

Cheers,


Hello Steve,

 I'm checking the Sanesecurity signature. I have these:

blurl.ndb
bofhland_cracked_URL.ndb
bofhland_malware_attach.hdb
bofhland_malware_URL.ndb
bofhland_phishing_URL.ndb
crdfam.clamav.hdb
foxhole_filename.cdb
foxhole_generic.cdb
hackingteam.hsb
javascript.ndb
junk.ndb
jurlbl.ndb
malwarehash.hsb
malwarepatrol.db
phish.ndb
phishtank.ndb
porcupine.ndb
rogue.hdb
sanesecurity.ftm
scam.ndb
securiteinfoascii.hdb
securiteinfo.hdb
securiteinfohtml.hdb
securiteinfo.ign2
securiteinfopdf.hdb
sigwhitelist.ign2
spamattach.hdb
spamimg.hdb
spearl.ndb
spear.ndb
winnow.attachments.hdb
winnow_bad_cw.hdb
winnow_extended_malware.hdb
winnow_malware.hdb
winnow_malware_links.ndb

I now noticed that reload time decrease if I remove SecuriteInfo.  
Really I don't find securiteinfo match in log of viruses, so I could  
remove it permanently...

Do you have any hint about signature to keep for email traffic?

I'll also try to move fs in tmpfs...

Thank you very much
Best Regards
Marco


___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Andreas Schulze]

Am 29.09.2015 um 12:41 schrieb Joel Esler (jesler):
> Once a minute?  That’s fairly excessive.
> Once an hour is appropriate…  Overdoing it, but more appropriate.  Keep in 
> mind that the mirrors are donated to ClamAV and the bandwidth you are 
> consuming is probably fairly heavy.
I see it relaxed because I *do run* one of these mirrors
and I make sure to not bother other mirrors that way...

-- 
A. Schulze DATEV eG
___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Steve Basford]

On Tue, September 29, 2015 9:15 am, Marco wrote:

> I tried to relax the upgrades, but I see the problem is the time spent
> to reload the db:
>
> 2015-09-29T01:03:15.710526+02:00 av2 clamd[15201]: Reading databases
> from /var/lib/clamav 2015-09-29T01:03:53.151179+02:00 av2 clamd[15201]:
> Database correctly
> reloaded (5342845 signatures)
>
Hi Marco,

Are the signatures official ClamAV ones only or are you using 3rd Party
ones as well (if so, could you list them)

Cheers,

Steve
Web : sanesecurity.com
Blog: sanesecurity.blogspot.com

___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Joel Esler (jesler)]

Al,  Thanks for brining that up.

Once a minute?  That’s fairly excessive.

Once an hour is appropriate…  Overdoing it, but more appropriate.  Keep in mind 
that the mirrors are donated to ClamAV and the bandwidth you are consuming is 
probably fairly heavy.  If everyone did that….


--
Joel Esler
Manager, Talos Group




On Sep 29, 2015, at 4:09 AM, Al Varnell 
> wrote:

You are admitting that on a ClamAV mail list?

From the ClamAV Virus Database documentation 
:

• How many times per hour shall I run freshclam?
You can check for database update as often as 4 times per hour provided that 
you have the following options in freshclam.conf: DNSDatabaseInfo 
current.cvd.clamav.netDatabaseMirror db.XY.clamav.net 
DatabaseMirror database.clamav.net
Replace XY with your country code. If you don’t have that option, then you must 
stick with 1 check per hour.

-Al-

On Tue, Sep 29, 2015 at 01:03 AM, Andreas Schulze wrote:

Am 29.09.2015 09:31 schrieb Marco:
With this installation, after every freshclam update session, clamd is
forced to read the DB:

I had a similar issue. I call freshclam once a minute and the db reload 
generate a significant load.
I fixed it with ithe attached patch.

___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Dennis Peterson]

On 9/29/15 3:41 AM, Joel Esler (jesler) wrote:

Al,  Thanks for brining that up.

Once a minute?  That’s fairly excessive.

Once an hour is appropriate…  Overdoing it, but more appropriate.  Keep in mind 
that the mirrors are donated to ClamAV and the bandwidth you are consuming is 
probably fairly heavy.  If everyone did that….


--
Joel Esler
Manager, Talos Group


Not likely - it is a DNS query each minute, not a file transfer. There are other 
reasons why it is a bad idea not the least of which is it is a form of 
self-denial of service. And ignorance. It simply isn't necessary to poll each 
minute.


The OP can avoid loss of service during a signature refresh by using two 
instances of clamd on two different ports and dynamically manage port forwarding 
in IPTables/IPChains. This allows updating each instance independently and a 
near atomic change of clamd instances. It is a poor-man's Big-IP but one that is 
not without some challenges of its own. There's no magic solution.


dp
___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Benny Pedersen]

On September 29, 2015 1:55:47 PM Andreas Schulze  
wrote:



I see it relaxed because I *do run* one of these mirrors
and I make sure to not bother other mirrors that way...


+1, our point is more do not use freshclam from cronned shell, but when its 
local mirror it will miss dns data update to tell freshclamd to get new 
sigs, chicken and eg problem that only can be solved with freshclam in crontabs

___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Benny Pedersen]

On September 29, 2015 10:16:13 AM Marco  wrote:


2015-09-29T01:03:53.151179+02:00 av2 clamd[15201]: Database correctly
reloaded (5342845 signatures)



as i see you use alot of 3rd party sigs


Is there a way to speed up this phase? Maybe putting the db files into
a RAM fs?


waste of ram

more help give clamconf on pastebin with a link here

is your main and daily uncompressed or compressed ?  cvd vs cld files

to developpers make a option in sigtool to resave cld to cvd and visa 
versa, this vill speed up reloads, eg cvd files makes reload slow, but cld 
fast as possible

___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB update and clamav-milter delay

[Joel Esler (jesler)]

On Sep 29, 2015, at 9:57 AM, Kris Deugau 
> wrote:

Marco wrote:
Hello,

I installed clamd server (0.98.7) with clamav-milter using RPM of EPEL.

With this installation, after every freshclam update session, clamd is
forced to read the DB:

2015-09-29T09:12:41.244383+02:00 av1 clamd[15201]: Reading databases
from /var/lib/clamav
2015-09-29T09:13:14.950256+02:00 av2 clamav-milter[11957]: Failed to
stream to clamd
2015-09-29T09:13:14.950546+02:00 av2 clamav-milter[11957]: Streaming failed
2015-09-29T09:13:20.593439+02:00 av1 clamd[15201]: Database correctly
reloaded (5342538 signatures)

During this time clamav-milter have a trouble and the SMTP server
experiences a delay receiving the mail. On a frontend MSA SMTP server
this delay is a problem.

I would like to know if there is a better way to configure the DB
update, without interruption on service.

Short of a patch to clamd to not discard the existing in-memory
signature data while reloading fresh data (or somewhat more complex;
replace signatures one-by-one during a reload), you are stuck with clamd
not responding for 2-5 seconds with the stock signatures, and possibly
up to 30 seconds even on modern hardware if you're using a lot of
third-party signatures.

This seems to come up every couple of months at least and it's been this
way for quite a long time;  is there any upstream interest in fixing
clamd one way or another so it doesn't suffer this outage while
reloading signatures?



Don’t want to speak for the ClamAV team, but I’m sure they’d be welcome with 
ideas and feedback if you file a bug in the bugzilla system

--
Joel Esler
Manager, Threat Intelligence Team & Open Source
Talos Group
http://www.talosintel.com
___
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml