Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
* René Berber [EMAIL PROTECTED]: | Unfortunately I had to rewrite that script until it worked :( | The download URLs for the mirrors are incorrect and for some reason it | pukes on the output of Debian's clamd --debug Thanks for both replies. I'll take that script for a test. I uploaded my adapted script here: http://www.arschkrebs.de/postfix/UpdateSaneSecurity.sh -- Ralf Hildebrandt (i.A. des IT-Zentrums) [EMAIL PROTECTED] Charite - Universitätsmedizin BerlinTel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-BerlinFax. +49 (0)30-450 570-962 IT-Zentrum Standort CBFsend no mail to [EMAIL PROTECTED] ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
* Rick Cooper [EMAIL PROTECTED]: I assume (again) that you meant clamscan --debug causes an issue? What does it do exactly that causes an issue? The output looks like this: LibClamAV debug: Loading databases from /var/lib/clamav/ LibClamAV debug: Loading databases from /var/lib/clamav//main.inc (it has a trainling slash) -- Ralf Hildebrandt (i.A. des IT-Zentrums) [EMAIL PROTECTED] Charite - Universitätsmedizin BerlinTel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-BerlinFax. +49 (0)30-450 570-962 IT-Zentrum Standort CBFsend no mail to [EMAIL PROTECTED] ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
Ralf, I wrote a small script by myself - very simpel. It seems to work now for months. #!/bin/sh cd /tmp # Unofficial Phising rules for ClamAV wget -nd -m http://ftp.tiscali.nl/sanesecurity/phish.ndb.gz wget -nd -m http://ftp.tiscali.nl/sanesecurity/scam.ndb.gz cp phish.ndb.gz /var/lib/clamav/ cp scam.ndb.gz /var/lib/clamav/ cd /var/lib/clamav gunzip -f phish.ndb.gz gunzip -f scam.ndb.gz chown vscan:vscan phish.ndb chown vscan:vscan scam.ndb rcclamd restart Run by root via crontab. Regards, Alexander ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
At 03:17 AM 5/8/2007, Alexander Grüner wrote: Ralf, I wrote a small script by myself - very simpel. It seems to work now for months. #!/bin/sh cd /tmp # Unofficial Phising rules for ClamAV wget -nd -m http://ftp.tiscali.nl/sanesecurity/phish.ndb.gz wget -nd -m http://ftp.tiscali.nl/sanesecurity/scam.ndb.gz cp phish.ndb.gz /var/lib/clamav/ cp scam.ndb.gz /var/lib/clamav/ cd /var/lib/clamav gunzip -f phish.ndb.gz gunzip -f scam.ndb.gz chown vscan:vscan phish.ndb chown vscan:vscan scam.ndb rcclamd restart Run by root via crontab. The above is not particularly safe. For add-on signatures, it's important to test the signatures with clamscan -d phish.ndb /some/small/file *before* copying them into the live clam signature directory. Freshclam does this for you for the official signatures. If the add-on file is corrupted or just the wrong format for some reason, copying it into the clam signature directory will cause clam to refuse to run. A safe (but not very clever) script would look like: #!/bin/sh cd /tmp wget -nd -m http://ftp.tiscali.nl/sanesecurity/phish.ndb.gz \ gunzip -f phish.ndb.gz \ clamscan -d phish.ndb phish.ndb \ cp phish.ndb /var/lib/clamav/ \ chown vscan:vscan /var/lib/clamav/phish.ndb wget -nd -m http://ftp.tiscali.nl/sanesecurity/scam.ndb.gz \ gunzip -f scam.ndb.gz \ clamscan -d scam.ndb scam.ndb \ cp scam.ndb /var/lib/clamav/ \ chown vscan:vscan /var/lib/clamav/scam.ndb rcclamd restart -- Noel Jones ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
Noel Jones wrote: At 03:17 AM 5/8/2007, Alexander Grüner wrote: Ralf, I wrote a small script by myself - very simpel. It seems to work now for months. #!/bin/sh cd /tmp # Unofficial Phising rules for ClamAV wget -nd -m http://ftp.tiscali.nl/sanesecurity/phish.ndb.gz wget -nd -m http://ftp.tiscali.nl/sanesecurity/scam.ndb.gz cp phish.ndb.gz /var/lib/clamav/ cp scam.ndb.gz /var/lib/clamav/ cd /var/lib/clamav gunzip -f phish.ndb.gz gunzip -f scam.ndb.gz chown vscan:vscan phish.ndb chown vscan:vscan scam.ndb rcclamd restart Run by root via crontab. The above is not particularly safe. For add-on signatures, it's important to test the signatures with clamscan -d phish.ndb /some/small/file *before* copying them into the live clam signature directory. Freshclam does this for you for the official signatures. If the add-on file is corrupted or just the wrong format for some reason, copying it into the clam signature directory will cause clam to refuse to run. A safe (but not very clever) script would look like: #!/bin/sh cd /tmp wget -nd -m http://ftp.tiscali.nl/sanesecurity/phish.ndb.gz \ A kinder, gentler usage of wget would include -N to prevent getting the same file over and over thereby burning off Steve's bandwidth. Just be sure to preserve the name and time stamp of the gz files when you ungzip them. Use: gunzip -c phish.ndb phish.ndb.gz or gunzip phish.ndb.gz phish.ndb This leaves the previously downloaded file in place so wget can reference it when polling Steve's server for a newer version. Using a list of URL's with --input-file=FILE would grab both files in a single connection and also help cut down on resource usage. As mentioned elsewhere, use a randomizer when running these scripts so you don't dogpile on the server at 0,10,20 etc. minutes past the hour: Using bash for the shell allows: if [ -z $1 ]; then sleep $[ RANDOM % 600 ] fi This causes the process to sleep for some random number of seconds up to 600 before completing. This also helps prevent excessive peak loads on Steve's server. The test allows bypassing this delay if needed as in: sanesecurity.sh now This causes an immediate download. When dropping the files into the final working dir you'd like to make that as atomic as possible, so use mv (if everything is on one partition) or cp with mv (cp phish.ndb /path/phish.tmp;mv /path/phish.tmp /path/phish.ndb, or rsync phish.ndb /path. This keeps clamd from ever seeing a file while it is being copied to the working directory. clamd is very picky about that. Each mirror has it's own unique URL list for the files so you can create three files: sane.list: http://www.sanesecurity.co.uk/clamav/phish.ndb.gz http://www.sanesecurity.co.uk/clamav/scam.ndb.gz dotsrc.list: http://mirrors.dotsrc.org/clamav-sanesigs/phish.ndb.gz http://mirrors.dotsrc.org/clamav-sanesigs/scam.ndb.gz tiscali.list: http://ftp.tiscali.nl/sanesecurity/phish.ndb.gz http://ftp.tiscali.nl/sanesecurity/scam.ndb.gz #Sane Security Files SaneFileList=/usr/local/share/clamav/tmp/sane.list DotsrcFileList=/usr/local/share/clamav/tmp/dotsrc.list TiscaliFileList=/usr/local/share/clamav/tmp/tiscali.list And since we don't want to slam only Steve's server we'd like some kind of load sharing, so we randomize which list we use anothe randomizer. Put it all together and you have #!/bin/bash if [ -z $1 ]; then sleep $[ RANDOM % 600 ] fi # randomization for selecting server let RoundRobin = $RANDOM % 3 + 1 case $RoundRobin in 1) WgetList=$SaneFileList ;; 2) WgetList=$DotsrcFileList ;; 3) WgetList=$TiscaliFileList ;; esac cd /var/tmp # /tmp contents don't survive a reboot wget -q -N --input-file=$WgetList \ gunzip phish.ndb.gz phish.ndb \ clamscan -d phish.ndb phish.ndb \ rsync phish.ndb /var/lib/clamav/ \ chown vscan:vscan /var/lib/clamav/phish.ndb gunzip scam.ndb.gz scam.ndb \ clamscan -d scam.ndb scam.ndb \ rsync scam.ndb /var/lib/clamav/ \ chown vscan:vscan /var/lib/clamav/scam.ndb rcclamd restart # hopefully this is a reload, not a restart Now you have a cron-driven script that does not dogpile at frequently used cron intervals, that does not download files unless the source is newer, uses a random source to fetch files from, uses lists of files so only a single wget session is required, and tests the integrity of the files before doing an atomic copy to the working directory. Be advised that some, possibly all versions of wget will return a success code even when it fails to download a file so everything that follows the wget will always take place even if a new file has not been downloaded. I've created an ugly hack to keep this from happening, but in truth it just burns a little CPU time to accept the process as is. There's other ways to do this, of course. dp ___ Help us build a comprehensive ClamAV guide: visit
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
Dennis Peterson wrote: A kinder, gentler usage of wget would include -N to prevent getting the same file over and over thereby burning off Steve's bandwidth. Just be sure to preserve the name and time stamp of the gz files when you ungzip them. Use: gunzip -c phish.ndb phish.ndb.gz Sorry, I managed to munge that. Should be: gunzip -c phish.ndb.gz phish.ndb dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of
Ralf Hildebrandt
Sent: Tuesday, May 08, 2007 3:07 AM
To: clamav-users@lists.clamav.net
Subject: Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
* Rick Cooper [EMAIL PROTECTED]:
I assume (again) that you meant clamscan --debug causes an
issue? What does
it do exactly that causes an issue?
The output looks like this:
LibClamAV debug: Loading databases from /var/lib/clamav/
LibClamAV debug: Loading databases from /var/lib/clamav//main.inc
(it has a trainling slash)
--
So if you add -m 1 to the grep command that would take care of that and,
oddly enough, it short circuits the entire thing down to less than a second
on locating the path
Clam_Db_Dir=`${clamscan} --debug $tmp_dir/test.file 21 | grep -m 1 -i -e
'loading databases from.*clamav$'| sed s/.*loading databases from //i`
Works *very* quickly here.
I made some changes to it again this morning (the above) and modified the
MSRBL sections to use rsync instead of curl. I believe the sanesecurity sig
provider is checking at least one of the mirrors to see if they can add his
stuff to their rsync files, if that happens I will change that as well.
Although I should probably start from scratch since there is little left of
the original script anyway.
Rick
Rick
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Steve Basford wrote: [snip] | So, please could you all change your download scripts to download from | the above mirrors, not only will this help avoid me getting hit with | hosting charges but you benefit as you should be able to increase the | frequency you check for download changes. | | The new download Links have been updated on the download page and so | have the scripts: | | http://sanesecurity.co.uk/clamav/downloads.htm | http://sanesecurity.co.uk/clamav/usage.htm [snip] Any of the scripts check the databases before installing? I changed to the latest scamp.sh and I see it just downloads and installs, no check to see if the file is a valid clamav database. - -- René Berber -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Cygwin) iD8DBQFGP4xEL3NNweKTRgwRCEYHAKCP/ztKwvdmc2Iz4iYdsRflsiLPowCg0Xcn JvH5ySQFuWLKy/WTy/RuCqk= =kq+C -END PGP SIGNATURE- ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
At 03:29 PM 5/7/2007, René Berber wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Steve Basford wrote: [snip] | So, please could you all change your download scripts to download from | the above mirrors, not only will this help avoid me getting hit with | hosting charges but you benefit as you should be able to increase the | frequency you check for download changes. | | The new download Links have been updated on the download page and so | have the scripts: | | http://sanesecurity.co.uk/clamav/downloads.htm | http://sanesecurity.co.uk/clamav/usage.htm [snip] Any of the scripts check the databases before installing? I changed to the latest scamp.sh and I see it just downloads and installs, no check to see if the file is a valid clamav database. The Example 1 UpdateSaneSecurity.sh appears to use clamscan -d to test for a valid database before installing them in the live directory. Didn't check the others... -- Noel Jones ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
* Noel Jones [EMAIL PROTECTED]: The Example 1 UpdateSaneSecurity.sh appears to use clamscan -d to test for a valid database before installing them in the live directory. Didn't check the others... Unfortunately I had to rewrite that script until it worked :( The download URLs for the mirrors are incorrect and for some reason it pukes on the output of Debian's clamd --debug -- Ralf Hildebrandt (i.A. des IT-Zentrums) [EMAIL PROTECTED] Charite - Universitätsmedizin BerlinTel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-BerlinFax. +49 (0)30-450 570-962 IT-Zentrum Standort CBFsend no mail to [EMAIL PROTECTED] ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ralf Hildebrandt wrote: | * Noel Jones: | | The Example 1 UpdateSaneSecurity.sh appears to use clamscan -d to | test for a valid database before installing them in the live | directory. Didn't check the others... | | Unfortunately I had to rewrite that script until it worked :( | The download URLs for the mirrors are incorrect and for some reason it | pukes on the output of Debian's clamd --debug Thanks for both replies. I'll take that script for a test. - -- René Berber -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Cygwin) iD8DBQFGP53vL3NNweKTRgwRCITTAJ42mImoI8jRG6UXpbLtmp5ua6ldCQCg3sdS gHTGfj3b/anxc7E21i1gosg= =QI/R -END PGP SIGNATURE- ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
René Berber wrote the following on 5/7/2007 1:29 PM -0800: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Steve Basford wrote: [snip] | So, please could you all change your download scripts to download from | the above mirrors, not only will this help avoid me getting hit with | hosting charges but you benefit as you should be able to increase the | frequency you check for download changes. | | The new download Links have been updated on the download page and so | have the scripts: | | http://sanesecurity.co.uk/clamav/downloads.htm | http://sanesecurity.co.uk/clamav/usage.htm [snip] Any of the scripts check the databases before installing? I changed to the latest scamp.sh and I see it just downloads and installs, no check to see if the file is a valid clamav database. The Example 2 script checks the sig databases using clamscan -d before installing. ;-) Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ralf Hildebrandt Sent: Monday, May 07, 2007 4:49 PM To: clamav-users@lists.clamav.net Subject: Re: [Clamav-users] OT: Sanesecurity Sigs: Important News * Noel Jones [EMAIL PROTECTED]: The Example 1 UpdateSaneSecurity.sh appears to use clamscan -d to test for a valid database before installing them in the live directory. Didn't check the others... Unfortunately I had to rewrite that script until it worked :( The download URLs for the mirrors are incorrect and for some reason it pukes on the output of Debian's clamd --debug -- My apologies, I made the gross assumption that they actually mirrored the main site (as in /clamav/filename) and apparently only the sanesecurity.co.uk site actually does. I just fixed the urls and tested each site (twice), also changed it to test for the db directory being in /usr/local/share/clamav and it won't ask clamscan unless it doesn't find it there. And yes it does test the signatures before installing them. Wouldn't want to have a 4xx to dump a text file in place of the actual sigs. I assume (again) that you meant clamscan --debug causes an issue? What does it do exactly that causes an issue? Again, my apologies for the error. Rick -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
Steve Basford wrote the following on 5/5/2007 10:05 AM -0800: Due to me nearly running out of bandwidth last month (17gb out of a 20gb host package), some urgent changes were needed to the signature hosting, otherwise I'd start getting charged for the extra bandwidth :( [...] The new download Links have been updated on the download page and so have the scripts: http://sanesecurity.co.uk/clamav/downloads.htm http://sanesecurity.co.uk/clamav/usage.htm Steve, I feed URI to URIBL on a daily basis, and it never ceases to amaze me how quickly you detect new phish e-mails. I rarely have to redirect a phish to you for addition. Anyway, I had posted the attached updated script to the ClamAV Users list back in early March, and several people are now using it. It includes the usage of rsync for updating the MSRBL signature files (and points to mirrors.dotsrc.org for the SaneSecurity downloads). Just thought you might want to update the script on your usage site (up to you...). Thanks, and keep up the great work! Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
Oops, sorry, I meant to send this to Steve off-list... :-\ Bill Landry wrote the following on 5/5/2007 10:43 AM -0800: Steve Basford wrote the following on 5/5/2007 10:05 AM -0800: Due to me nearly running out of bandwidth last month (17gb out of a 20gb host package), some urgent changes were needed to the signature hosting, otherwise I'd start getting charged for the extra bandwidth :( [...] The new download Links have been updated on the download page and so have the scripts: http://sanesecurity.co.uk/clamav/downloads.htm http://sanesecurity.co.uk/clamav/usage.htm Steve, I feed URI to URIBL on a daily basis, and it never ceases to amaze me how quickly you detect new phish e-mails. I rarely have to redirect a phish to you for addition. Anyway, I had posted the attached updated script to the ClamAV Users list back in early March, and several people are now using it. It includes the usage of rsync for updating the MSRBL signature files (and points to mirrors.dotsrc.org for the SaneSecurity downloads). Just thought you might want to update the script on your usage site (up to you...). Thanks, and keep up the great work! Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] OT: Sanesecurity Sigs: Important News
Steve Basford wrote: Due to me nearly running out of bandwidth last month (17gb out of a 20gb host package), some urgent changes were needed to the signature hosting, otherwise I'd start getting charged for the extra bandwidth :( So, to keep this short, here's a to-do list ;) *** One: Mirrors *** Three new mirrors are now available, in preferred order: Mirror 1: A huge thanks to http://dotsrc.org/ (formerly known as SunSITE.dk) as they are now a mirror for my signatures, hourly updating from the main site. Mirror 2: Thanks to http://tiscali.nl, as they seem to be a mirror for my signatures, hourly updating from the main site Mirror 3: Thanks to a special offer deal from Surpass Hosting, I setup a sanesecurity.co.uk domain, to try and ease the load from the main sanesecurity.com site. So, please could you all change your download scripts to download from the above mirrors, not only will this help avoid me getting hit with hosting charges but you benefit as you should be able to increase the frequency you check for download changes. This bash script stub doesn't solve the preferred order, but it does randomize the process. There's more than one way to do this - it's the concept that is important :) : # randomization for selecting server let RoundRobin = $RANDOM % 3 + 1 case $RoundRobin in 1) echo Getting list from SaneSecurity /usr/local/bin/wget -q -N --input-file=$SaneFileList /dev/null 21 ;; 2) echo Getting list from Dotsrc /usr/local/bin/wget -q -N --input-file=$DotsrcFileList /dev/null 2 ;; 3) echo Getting list from Tiscali /usr/local/bin/wget -q -N --input-file=$TiscaliFileList /dev/null 21 ;; esac I use a file list so I need only one connection to the server to grab both files. The transfer occurs only if the source is newer than the local copy. That requires the gzip files be processed such that their time stamp remains unchanged. I use redirection: # gzip phish.ndb.gz phish.ndb And there are other ways to do this. The important thing is to preserve the stamp so wget does not waste bandwidth and download an unchanged version. This capability is also available in curl. This is a fantastic and very valuable service, Steve. It is incredibly effective at what it does. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
