Re: [clamav-users] How to pick / specify the correct mirror
Noel Jones skrev den 28-01-2013 23:10: Regardless whether you're using cron or daemonized, the default behavior is to check DNS for the current version to decide if a download is needed. ups my fault, i got it now, thanks Noel ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
On Mon, 28 Jan 2013 22:45:31 -0800
Dennis Peterson articulated:
Bill Landry's unofficial sigs script does it right by using a random
function as an offset from cron time to actually initial a sig
download. For those of us with a couple dozen or more systems this is
a mirror-friendly way of not getting into lockstep with any
fixed-offset scheme. Here is the code segment from Bill's script and
I recommend something like it.
if [ -n $RANDOM ]
then
sleep_time=$(($RANDOM * $(($max_sleep_time -
$min_sleep_time)) / 32767 + $min_sleep_time))
else
sleep_time=0
while [ $sleep_time -lt $min_sleep_time -o
$sleep_time -gt $max_sleep_time ] ; do
sleep_time=`head -1 /dev/urandom | cksum | awk '{print
$2}'` done
fi
A bit of precision overkill given there are only 3600 seconds in an
hour, but it works. I use sleep $[ RANDOM % 900 ] and I have no idea
nor care when the signature download actually begins, but every
machine's crontab starts at the same time. The important thing to
remember here is there is a finite amount of time required to dl all
the sigs so regardless of the start time there will be overlap with
other clients just because there are more clients than there are
seconds in an hour.
I use a much simpler function:
function get_files ()
{
# This is the snooze test. It will delay the start of a download session
# between 0 and 546 seconds (Roughly 9 minutes). It will only work when the
# script is run via CRON. The variable $REST activates this function.
# See if the variable 'REST' has been set or if forcing it from command line
# And if we are running via CRON
if [ ${REST} -gt 0 -a ! -t 0 ]; then
# Use the BASH RANDOM function to generate a random number between 0 32767
RESTING=$((RANDOM/60))
sleep ${RESTING}
fi
...
}
I never saw the benefit of expanding the time span, although it could
be done easily enough.
--
Jerry ♔
Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.
__
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
Jim Preston skrev den 26-01-2013 03:39: I am using freshclam but NOT running it as a daemon. Here is my cron task: 10 * * * * /usr/local/bin/freshclam /dev/null 21 this will be hardcoded time checks, not a fail if one can live with that So as far as I understand, I am using freshclam. If I am wrong or loose some functionality with the cron task, can you elaborate or point me to the relevant documentation? if you used freshclam as a deamon it will update when dns is showing new versions, not wait one hour or more in cron to get the newest updates ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
On 1/28/2013 2:27 PM, Benny Pedersen wrote:
if you used freshclam as a deamon it will update when dns is showing
new versions, not wait one hour or more in cron to get the newest
updates
Incorrect.
When you run freshclam as a daemon, the freshclam.conf specifies how
many times per day to check for updates. There's not a lot of
practical difference between a daemonized freshclam with Checks 24
and a once-an-hour cron job.
The default is Checks 12 which means {check for an update 12 times
per day}.
Regardless whether you're using cron or daemonized, the default
behavior is to check DNS for the current version to decide if a
download is needed.
-- Noel Jones
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
On 1/28/13 4:10 PM, Noel Jones njo...@megan.vbhcs.org wrote: On 1/28/2013 2:27 PM, Benny Pedersen wrote: if you used freshclam as a deamon it will update when dns is showing new versions, not wait one hour or more in cron to get the newest updates Incorrect. When you run freshclam as a daemon, the freshclam.conf specifies how many times per day to check for updates. There's not a lot of practical difference between a daemonized freshclam with Checks 24 and a once-an-hour cron job. I normally run checks 47 or checks 49. That's not quite (or just slightly more often than) every half hour. Using an odd number like that ensures that I don't get stuck on a popular minute for polls... -- Daniel J McDonald, CCIE # 2495, CISSP # 78281 ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
Bill Landry's unofficial sigs script does it right by using a random
function as an offset from cron time to actually initial a sig download.
For those of us with a couple dozen or more systems this is a
mirror-friendly way of not getting into lockstep with any fixed-offset
scheme. Here is the code segment from Bill's script and I recommend
something like it.
if [ -n $RANDOM ]
then
sleep_time=$(($RANDOM * $(($max_sleep_time - $min_sleep_time))
/ 32767 + $min_sleep_time))
else
sleep_time=0
while [ $sleep_time -lt $min_sleep_time -o $sleep_time
-gt $max_sleep_time ] ; do
sleep_time=`head -1 /dev/urandom | cksum | awk '{print $2}'`
done
fi
A bit of precision overkill given there are only 3600 seconds in an
hour, but it works. I use sleep $[ RANDOM % 900 ] and I have no idea nor
care when the signature download actually begins, but every machine's
crontab starts at the same time. The important thing to remember here is
there is a finite amount of time required to dl all the sigs so
regardless of the start time there will be overlap with other clients
just because there are more clients than there are seconds in an hour.
On my personal servers I do this once each day and most of what I catch
are traced to signatures from Sane Security. Less than 15/day, on average.
dp
On 1/28/13 2:50 PM, McDonald, Dan wrote:
On 1/28/13 4:10 PM, Noel Jones njo...@megan.vbhcs.org wrote:
On 1/28/2013 2:27 PM, Benny Pedersen wrote:
if you used freshclam as a deamon it will update when dns is showing
new versions, not wait one hour or more in cron to get the newest
updates
Incorrect.
When you run freshclam as a daemon, the freshclam.conf specifies how
many times per day to check for updates. There's not a lot of
practical difference between a daemonized freshclam with Checks 24
and a once-an-hour cron job.
I normally run checks 47 or checks 49. That's not quite (or just
slightly more often than) every half hour. Using an odd number like that
ensures that I don't get stuck on a popular minute for polls...
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
Thanks Jim. Is this different than db.local.clamav.net? That is what my conf file was pre-populated with when I installed the package. I changed it to us but I am wondering if that is actually changing anything. Thank for the tip on cron task. That seems like a good idea. :) On Thu, Jan 24, 2013 at 2:53 PM, Jim Preston jimli...@commspeed.net wrote: On 01/24/2013 11:40 AM, Lee Graber wrote: I am just starting to try and get clamd + freshclam running on some Ubuntu servers running on EC2 servers in the US Standard Region (east coast). The documentation talks about specifying a mirror which is close to you but it seems to default to the round-robin endpoint. I have run freshclam a number of times and sometimes it takes 14 seconds and sometimes it takes 6 minutes. I can see that the ip address is different on the calls and am trying to figure out how I can get it to take 14 seconds all the time :). I can't find anything that says there is a mirror in the Amazon DataCenter. How can I make this behavior reliable fast. Thanks Lee __**_ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/**ml http://www.clamav.net/support/ml You should have set up freshclam.conf to use the US mirrors (DatabaseMirror db.us.clamav.net). The update time will vary depending on whether it has to download any updates and the load on the particular mirror it connects to. The round-robin functionality is by design to spread load and help when a mirror goes down for any reason. You may also want to change the update frequency. I am running freshclam as a cron task and set it to not be on the hour, e.g. I update at 5 minutes after the hour to try and hit the mirrors at a low load time. -- Jim Preston __**_ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/**ml http://www.clamav.net/support/ml ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
From my experience, it changes very little, especially in the US where there are thirteen mirror sites. I'm told that really isn't enough to meet the demand here, so there are several off-shore sites added to the mix, which I suppose could account for your more lengthy sessions. Changing it to US won't eliminate those off-shore sites from occasionally being selected in the round-robin scheme of things. The freshclam algorithm attempts to determine a set of sites closest to you by examining your IP address. Not a perfect solution, but it usually works to be close enough. In my work with ClamXav users, I normally try to discourage them from changing the setting unless they are experiencing a high rate of connectivity issues with sites that they probably should not be using. Doing so is not a standard ClamXav preference option and changing the freshclam.conf file is not a trivial task for the average Mac user. There have been occasions in Europe where a mirror site has prevented access from certain IP blocks, which they aren't supposed to do, but there is really no way to prevent it. In that case, there did seem to be improvement by changing the CC. Of course, if you are going to be checking for updates more than once an hour, then you are required to change it for load balancing purposes. Again, this is not something the average Mac user can do (other than manually), even if they wanted to, but enterprise sites must play by different rules. -Al- On 1/25/13 12:54 AM, Lee Graber wrote: Thanks Jim. Is this different than db.local.clamav.net? That is what my conf file was pre-populated with when I installed the package. I changed it to us but I am wondering if that is actually changing anything. Thank for the tip on cron task. That seems like a good idea. :) On Thu, Jan 24, 2013 at 2:53 PM, Jim Preston jimli...@commspeed.net wrote: On 01/24/2013 11:40 AM, Lee Graber wrote: I am just starting to try and get clamd + freshclam running on some Ubuntu servers running on EC2 servers in the US Standard Region (east coast). The documentation talks about specifying a mirror which is close to you but it seems to default to the round-robin endpoint. I have run freshclam a number of times and sometimes it takes 14 seconds and sometimes it takes 6 minutes. I can see that the ip address is different on the calls and am trying to figure out how I can get it to take 14 seconds all the time :). I can't find anything that says there is a mirror in the Amazon DataCenter. How can I make this behavior reliable fast. Thanks Lee You should have set up freshclam.conf to use the US mirrors (DatabaseMirror db.us.clamav.net). The update time will vary depending on whether it has to download any updates and the load on the particular mirror it connects to. The round-robin functionality is by design to spread load and help when a mirror goes down for any reason. You may also want to change the update frequency. I am running freshclam as a cron task and set it to not be on the hour, e.g. I update at 5 minutes after the hour to try and hit the mirrors at a low load time. -- Jim Preston ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
Jim Preston skrev den 24-01-2013 23:53: You may also want to change the update frequency. I am running freshclam as a cron task and set it to not be on the hour, e.g. I update at 5 minutes after the hour to try and hit the mirrors at a low load time. freshclam use dns to check if there is new updates, using cron you loose this functionality so configure freshclam.conf to use 24 updates pr day, this will try to keep updates hourly, but if some mirror is down freshclam will not wait one hour to try another show freshclam --list-mirrors if there is problems with some mirrors ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
Lee Graber skrev den 25-01-2013 09:54: That seems like a good idea. :) nope understanding freshclam is even more worse :) ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
Thanks Al, This is actually for a farm of servers (right now it is a farm of 1) running Ubuntu on EC2. I was kinda surprised that there wasn't already a mirror in AWS since they actually included ClamAV in their extended package but such is life. My latest thoughts have been around having a single server (or 2) in my farm whose job (among other things) is to run freshclam and then push the updated cvd files (if needed) to a central S3 location. Then all of the servers I have would simply pull from there when changes are needed. This would mean less overhead on the mirrors and more reliability on my servers all being able to update in a timely fashion. I had thought about checking out the wire protocol for freshclam (I don't think it is doc'd but I could use wireshark) so that I could maybe even still run freshclam on my virus checking servers but I am a bit worried that it could change and so I could just invent my own (it is basically something along the lines of have any of these files changed. Give them to me if they have. And then making sure clamd gets the changes). Has anyone already built something like this? If I do this, I could put the cvds in a public dataset on Amazon S3 so anyone inside AWS could get them for free and fast but I am not sure about that yet. Again, is this something people have done? Thanks Lee On Fri, Jan 25, 2013 at 2:18 AM, Al Varnell alvarn...@mac.com wrote: shclam algorithm attempts to determine a set of sites closest to you by examining your IP address. Not a perfect solution, but it usually works to be close enough. In my work with ClamXav users, I normally try to discourage them from changing the setting unless they are experiencing a high rate of connectivity issues with sites that they probably should not be using. Doing so is not a standard ClamXav preference option and changing the freshclam.conf file is not a trivial task for the average Mac user. There have been occasions in Europe where a mirror site has prevented access from certain IP blocks, which they aren't supposed to do, but there is really no ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
On 01/25/2013 10:03 AM, Benny Pedersen wrote: Jim Preston skrev den 24-01-2013 23:53: You may also want to change the update frequency. I am running freshclam as a cron task and set it to not be on the hour, e.g. I update at 5 minutes after the hour to try and hit the mirrors at a low load time. freshclam use dns to check if there is new updates, using cron you loose this functionality so configure freshclam.conf to use 24 updates pr day, this will try to keep updates hourly, but if some mirror is down freshclam will not wait one hour to try another show freshclam --list-mirrors if there is problems with some mirrors Hi Benny, I do not understand your comment freshclam use dns to check if there is new updates, using cron you loose this functionality I am using freshclam but NOT running it as a daemon. Here is my cron task: 10 * * * * /usr/local/bin/freshclam /dev/null 21 So as far as I understand, I am using freshclam. If I am wrong or loose some functionality with the cron task, can you elaborate or point me to the relevant documentation? Thanks, Jim -- Jim Preston ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
On 1/25/2013 8:39 PM, Jim Preston wrote: On 01/25/2013 10:03 AM, Benny Pedersen wrote: Jim Preston skrev den 24-01-2013 23:53: You may also want to change the update frequency. I am running freshclam as a cron task and set it to not be on the hour, e.g. I update at 5 minutes after the hour to try and hit the mirrors at a low load time. freshclam use dns to check if there is new updates, using cron you loose this functionality so configure freshclam.conf to use 24 updates pr day, this will try to keep updates hourly, but if some mirror is down freshclam will not wait one hour to try another show freshclam --list-mirrors if there is problems with some mirrors Hi Benny, I do not understand your comment freshclam use dns to check if there is new updates, using cron you loose this functionality I am using freshclam but NOT running it as a daemon. Here is my cron task: 10 * * * * /usr/local/bin/freshclam /dev/null 21 So as far as I understand, I am using freshclam. If I am wrong or loose some functionality with the cron task, can you elaborate or point me to the relevant documentation? Thanks, Jim As long as you don't specify the --no-dns option, freshclam will use DNS to compare the current published version before attempting to download anything, regardless whether it's run by hand, a cronjob, or daemonized. At some point in the distant past, the freshclam daemon had a tendency to misbehave/hang/crash/whatever, prompting some folks to run it as a cron job. This has been long fixed, and the background daemon is the recommended method now, as it somewhat randomizes the checking time to spread load on the download servers. But you don't lose anything by running it under cron. -- Noel Jones ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
On 1/24/13 10:40 AM, Lee Graber wrote: I am just starting to try and get clamd + freshclam running on some Ubuntu servers running on EC2 servers in the US Standard Region (east coast). The documentation talks about specifying a mirror which is close to you but it seems to default to the round-robin endpoint. I have run freshclam a number of times and sometimes it takes 14 seconds and sometimes it takes 6 minutes. I can see that the ip address is different on the calls and am trying to figure out how I can get it to take 14 seconds all the time :). I can't find anything that says there is a mirror in the Amazon DataCenter. How can I make this behavior reliable fast. Thanks Lee ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml Since you are running a Unix system you can use your own BIND installation to create a freshclam zone of your choosing. Create a zone, say freshclam.grabner, populate it with IP addresses from existing reliable servers using my.freshclam.grabner in a round-robin array, and use that (my.freshclam.grabner) in your freshclam.conf file. Obviously it becomes your responsibility to maintain an accurate list, but that can be entirely automated with some simple scripting. This isn't anything I would recommend, but there are gaps in the global coverage it can help fill, and it will certainly work while putting all the joy and heartache of success or failure in your camp. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
[clamav-users] How to pick / specify the correct mirror
I am just starting to try and get clamd + freshclam running on some Ubuntu servers running on EC2 servers in the US Standard Region (east coast). The documentation talks about specifying a mirror which is close to you but it seems to default to the round-robin endpoint. I have run freshclam a number of times and sometimes it takes 14 seconds and sometimes it takes 6 minutes. I can see that the ip address is different on the calls and am trying to figure out how I can get it to take 14 seconds all the time :). I can't find anything that says there is a mirror in the Amazon DataCenter. How can I make this behavior reliable fast. Thanks Lee ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] How to pick / specify the correct mirror
On 01/24/2013 11:40 AM, Lee Graber wrote: I am just starting to try and get clamd + freshclam running on some Ubuntu servers running on EC2 servers in the US Standard Region (east coast). The documentation talks about specifying a mirror which is close to you but it seems to default to the round-robin endpoint. I have run freshclam a number of times and sometimes it takes 14 seconds and sometimes it takes 6 minutes. I can see that the ip address is different on the calls and am trying to figure out how I can get it to take 14 seconds all the time :). I can't find anything that says there is a mirror in the Amazon DataCenter. How can I make this behavior reliable fast. Thanks Lee ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml You should have set up freshclam.conf to use the US mirrors (DatabaseMirror db.us.clamav.net). The update time will vary depending on whether it has to download any updates and the load on the particular mirror it connects to. The round-robin functionality is by design to spread load and help when a mirror goes down for any reason. You may also want to change the update frequency. I am running freshclam as a cron task and set it to not be on the hour, e.g. I update at 5 minutes after the hour to try and hit the mirrors at a low load time. -- Jim Preston ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
