Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Paul Kosinski]

Checking less frequently would be nice, but there seems to be a fair
amount of jitter in the update times. Or that an artifact of caching
(DNS or otherwise)?


On Thu, 20 Dec 2018 15:23:13 +
"Joel Esler (jesler)"  wrote:

> Right.  We only publish at certain times a day.  I think a check once
> an hour is probably fine.   
> 
> Sent from my  iPhone
> 
> > On Dec 20, 2018, at 09:55, Paul Kosinski 
> > wrote:
> > 
> > Only DNS TXT queries are done 3-5 times per hour. Freshclam itself
> > is only run whenever that reports that there is something new
> > available, as determined by the DNS TXT result showing a higher
> > version number than the *local* CLD file shows. In practice, this
> > means that freshclam is only run a few times per day. (Recently,
> > updates only seem to occur at about 03:xx, 11:xx and 19:xx GMT,
> > i.e., three times per day.)
> > 
> > 
> > On Wed, 19 Dec 2018 21:14:58 -0800
> > Al Varnell  wrote:
> > 
> >> Note these restrictions:
> >> 
> >>> How many times per hour shall I run freshclam?
> >>> You can check for database update as often as 4 times per hour
> >>> provided that you have the following options in freshclam.conf:
> >>> 
> >>> DNSDatabaseInfo current.cvd.clamav.net
> >>> 
> >>> DatabaseMirror db.XY.clamav.net
> >>> 
> >>> DatabaseMirror database.clamav.net
> >>> 
> >>> Replace XY with your country code. If you don’t have that option,
> >>> then you must stick with 1 check per hour.
> >> 
> >> -Al-
> >> 
> >>> On Wed, Dec 19, 2018 at 12:26 PM, Paul Kosinski wrote:
> >>> They all do DNS TXT queries 3-5 times per hour...
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Paul Kosinski]

When talking about averages, I agree. But what I am worried about is
the "worst case" malicious payload: for example, a brand new and
particularly effective piece of ransomware. It's like car, life or
medical insurance. The probability of needing it is low, but when you
do, you don't want your account to be in arrears.

-pk


On Thu, 20 Dec 2018 18:37:22 + (GMT)
"G.W. Haywood"  wrote:

> Hi there,
> 
> Attempting to bring some sort of perspective to all this...
> 
> The number of updates per day (or hour or minute), and the currency or
> otherwise of the updated data are not, I think, the things that
> matter.
> 
> Isn't what matters most the probability that some malicious payload
> will get past your scanner?
> 
> So, what's the difference in this probability if one updates daily, or
> hourly, or even every minute?  Not terribly easy to estimate, but I'd
> suggest that for real-world payloads (as opposed to random selections
> from some population of known payloads), and for ClamAV, we're looking
> at a range of a few percent in a probability of no less than several
> tens of percent.
> 
> I'm not saying that this exercise is pointless, but I am wondering if
> there might be better uses for the effort.

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Dennis Peterson]

On 12/20/18 10:56 AM, Dennis Peterson wrote:
This can be calculated by counting the number of ClamAV hits in the clamd log 
using ClamAV signatures and the time period between the first and last hits. 
In my case I have clamd logs back to April (252 days) and 58 hits on ClamAV 
signatures or about 4 per day. Total hits from all signature vendors over the 
same period is 5921 or roughly 100/day.


I'm in not much of a hurry to get the next daily update file within seconds of 
it becoming available. That was not the case most recently when I was 
responsible for production email systems with message rates in excess of 
1M/day. And compared to some I've run over the years that is not a lot.


dp

By way of comparison, since early November I've rejected 1300 messages using 
only tcpwrappers from a total of 4958 rejections from all MTA options employed. 
No way to know how effective firewall blocks are but there are whole regions in 
the world I never hear from directly. Proxy providers complicate that.



dp

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Dennis Peterson]

This can be calculated by counting the number of ClamAV hits in the clamd log 
using ClamAV signatures and the time period between the first and last hits. In 
my case I have clamd logs back to April (252 days) and 58 hits on ClamAV 
signatures or about 4 per day. Total hits from all signature vendors over the 
same period is 5921 or roughly 100/day.


I'm in not much of a hurry to get the next daily update file within seconds of 
it becoming available. That was not the case most recently when I was 
responsible for production email systems with message rates in excess of 1M/day. 
And compared to some I've run over the years that is not a lot.


dp


On 12/20/18 10:37 AM, G.W. Haywood wrote:

Hi there,

Attempting to bring some sort of perspective to all this...

The number of updates per day (or hour or minute), and the currency or
otherwise of the updated data are not, I think, the things that matter.

Isn't what matters most the probability that some malicious payload
will get past your scanner?


___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[G.W. Haywood]

Hi there,

Attempting to bring some sort of perspective to all this...

The number of updates per day (or hour or minute), and the currency or
otherwise of the updated data are not, I think, the things that matter.

Isn't what matters most the probability that some malicious payload
will get past your scanner?

So, what's the difference in this probability if one updates daily, or
hourly, or even every minute?  Not terribly easy to estimate, but I'd
suggest that for real-world payloads (as opposed to random selections
from some population of known payloads), and for ClamAV, we're looking
at a range of a few percent in a probability of no less than several
tens of percent.

I'm not saying that this exercise is pointless, but I am wondering if
there might be better uses for the effort.

--

73,
Ged.
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Joel Esler (jesler)]

Right.  We only publish at certain times a day.  I think a check once an hour 
is probably fine.   

Sent from my  iPhone

> On Dec 20, 2018, at 09:55, Paul Kosinski  wrote:
> 
> Only DNS TXT queries are done 3-5 times per hour. Freshclam itself is
> only run whenever that reports that there is something new available,
> as determined by the DNS TXT result showing a higher version number
> than the *local* CLD file shows. In practice, this means that freshclam
> is only run a few times per day. (Recently, updates only seem to occur
> at about 03:xx, 11:xx and 19:xx GMT, i.e., three times per day.)
> 
> 
> On Wed, 19 Dec 2018 21:14:58 -0800
> Al Varnell  wrote:
> 
>> Note these restrictions:
>> 
>>> How many times per hour shall I run freshclam?
>>> You can check for database update as often as 4 times per hour
>>> provided that you have the following options in freshclam.conf:
>>> 
>>> DNSDatabaseInfo current.cvd.clamav.net
>>> 
>>> DatabaseMirror db.XY.clamav.net
>>> 
>>> DatabaseMirror database.clamav.net
>>> 
>>> Replace XY with your country code. If you don’t have that option,
>>> then you must stick with 1 check per hour.
>> 
>> -Al-
>> 
>>> On Wed, Dec 19, 2018 at 12:26 PM, Paul Kosinski wrote:
>>> They all do DNS TXT queries 3-5 times per hour...
>> 
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml


smime.p7s
Description: S/MIME cryptographic signature
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Paul Kosinski]

Only DNS TXT queries are done 3-5 times per hour. Freshclam itself is
only run whenever that reports that there is something new available,
as determined by the DNS TXT result showing a higher version number
than the *local* CLD file shows. In practice, this means that freshclam
is only run a few times per day. (Recently, updates only seem to occur
at about 03:xx, 11:xx and 19:xx GMT, i.e., three times per day.)


On Wed, 19 Dec 2018 21:14:58 -0800
Al Varnell  wrote:

> Note these restrictions:
> 
> > How many times per hour shall I run freshclam?
> > You can check for database update as often as 4 times per hour
> > provided that you have the following options in freshclam.conf:
> > 
> > DNSDatabaseInfo current.cvd.clamav.net
> > 
> > DatabaseMirror db.XY.clamav.net
> > 
> > DatabaseMirror database.clamav.net
> > 
> > Replace XY with your country code. If you don’t have that option,
> > then you must stick with 1 check per hour.
> 
> -Al-
> 
> On Wed, Dec 19, 2018 at 12:26 PM, Paul Kosinski wrote:
> > They all do DNS TXT queries 3-5 times per hour...
> 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[J.R.]

Al...

> Note these restrictions:

You must either be running an old version of ClamAV or using an old
.conf file... Relevant part from my freshclam.conf below... Doing a
DNS lookup requires very little data transfer since it's just a small
UDP packet (~100 bytes maybe) back & forth (and is probably the most
efficient way to do this utilizing existing services). I'm not
advocating saying it's okay to do to excessive lookups, but even if
you did 10 lookups per hour that is a whopping grand total of 2 KB data
being transferred back & forth.

Me personally, I only check once an hour because I don't receive many
emails. But I can understand on a busier server the need to keep all
the various signatures up to date.

# Use DNS to verify virus database version. Freshclam uses DNS TXT records
# to verify database and software versions. With this directive you can change
# the database verification domain.
# WARNING: Do not touch it unless you're configuring freshclam to use your
# own database verification domain.
# Default: current.cvd.clamav.net
#DNSDatabaseInfo current.cvd.clamav.net

# database.clamav.net is now the primary domain name to be used world-wide.
# Now that CloudFlare is being used as our Content Delivery Network (CDN),
# this one domain name works world-wide to direct freshclam to the closest
# geographic endpoint.
DatabaseMirror db.local.clamav.net
DatabaseMirror db.local.clamav.net
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Joel Esler (jesler)]

Inline

> On Dec 19, 2018, at 4:08 PM, J.R.  wrote:
> 
> Joel - In regards to the comment on pointing everyone to Cloudflare...
> I'm guessing that statement means you are using a mix of the
> Cloudflare CDN and the original volunteer mirrors still?


No.  Cloudflare is currently handling EVERYTHING.  Not even our mirrors are in 
the rotation anymore.

> 
> Also, is there a way to force a selection of a particular mirror
> (either by CF datacenter or previous mirror), or are all the database
> hostnames resolve strictly by GeoIP??

No.  The "mirror" is selected based upon which is the fastest from your 
location.

smime.p7s
Description: S/MIME cryptographic signature
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Al Varnell]

Note these restrictions:

> How many times per hour shall I run freshclam?
> You can check for database update as often as 4 times per hour provided that 
> you have the following options in freshclam.conf:
> 
> DNSDatabaseInfo current.cvd.clamav.net
> 
> DatabaseMirror db.XY.clamav.net
> 
> DatabaseMirror database.clamav.net
> 
> Replace XY with your country code. If you don’t have that option, then you 
> must stick with 1 check per hour.

-Al-

On Wed, Dec 19, 2018 at 12:26 PM, Paul Kosinski wrote:
> They all do DNS TXT queries 3-5 times per hour...

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Paul Kosinski]

Whatever the TTL is, there's no reason to make the notification even
more out of date than it needs to be.

Suggestion: Whenever the ClamAV Team puts out an "important" update,
they should set the DNS TXT TTL low (and then raise it after a while).

-pk


On Wed, 19 Dec 2018 13:22:26 -0800
Dennis Peterson  wrote:

> The TTL of the TXT record is 30 minutes so unless you are directly
> polling one of the clamav.net dns servers you are going to get what
> ever is in your local NSCD cache.
> 
> dp
> 
> On 12/19/18 12:26 PM, Paul Kosinski wrote:
> >
> > snip
> >
> > They all do DNS TXT queries 3-5 times per hour, and *only* if that
> > says there are new CDIFFs do they invoke freshclam. As before, this
> > is all based on cron, and the times are staggered to avoid peaking.
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Paul Kosinski]

Yeah, I know that the CDIFFs will/may be cached, but it shouldn't
matter. The file daily-25221.cdiff has the same contents no matter when
you download it via freshclam or whatever (assuming its contents hasn't
been munged by "HTTP-Transform"). But daily.cvd changes over time, as
it should. Thus caching is harmless for CDIFFs -- it only makes future
downloads faster. But caching van worsen the contents of CVDs: you
might expect version 25221 (as per the DNS TXT value), but get version
25220. This happened to us a lot (from the BOS server).

P.S. I did try at the proxy detection thing a bit, and it showed (I
think) that Comcast was doing it. But I gave up when I realized the
problem was more or less inherent for CVDs, but not for CDIFFs.


On Wed, 19 Dec 2018 15:08:06 -0600
"J.R."  wrote:

> Joel - In regards to the comment on pointing everyone to Cloudflare...
> I'm guessing that statement means you are using a mix of the
> Cloudflare CDN and the original volunteer mirrors still?
> 
> Also, is there a way to force a selection of a particular mirror
> (either by CF datacenter or previous mirror), or are all the database
> hostnames resolve strictly by GeoIP??
> 
> 
> Paul - If something was caching your daily.cvd file before, there is
> no reason think it's doing otherwise for each of the .cdiff files. So
> even though each client may be downloading the cdiff on its own, only
> the first hit is actually downloading from cloudflare.
> 
> Don't know if I mentioned this before, there are websites that claim
> to do transparent proxy testing, don't know if they would show
> anything but worth trying just to see.

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Dennis Peterson]

The TTL of the TXT record is 30 minutes so unless you are directly polling one 
of the clamav.net dns servers you are going to get what ever is in your local 
NSCD cache.


dp

On 12/19/18 12:26 PM, Paul Kosinski wrote:


snip

They all do DNS TXT queries 3-5 times per hour, and *only* if that says
there are new CDIFFs do they invoke freshclam. As before, this is all
based on cron, and the times are staggered to avoid peaking.



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[J.R.]

Joel - In regards to the comment on pointing everyone to Cloudflare...
I'm guessing that statement means you are using a mix of the
Cloudflare CDN and the original volunteer mirrors still?

Also, is there a way to force a selection of a particular mirror
(either by CF datacenter or previous mirror), or are all the database
hostnames resolve strictly by GeoIP??


Paul - If something was caching your daily.cvd file before, there is
no reason think it's doing otherwise for each of the .cdiff files. So
even though each client may be downloading the cdiff on its own, only
the first hit is actually downloading from cloudflare.

Don't know if I mentioned this before, there are websites that claim
to do transparent proxy testing, don't know if they would show
anything but worth trying just to see.
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Paul Kosinski]

In light of The Delays, and the fact that CVDs are so much bigger than
CDIFFs, I have changed our ClamAVs to use Scripted Update (CDIFFs) and
thus fetch directly from database.clamav.net.

We currently have fewer than a half-dozen machines on our LAN, which
share a single Comcast dynamic IP address (and hit Cloudflare's BOS) and
one remote machine -- our virtual (cloud) Web server, which has static
IPs (and hits Cloudflare's IAD).

They all do DNS TXT queries 3-5 times per hour, and *only* if that says
there are new CDIFFs do they invoke freshclam. As before, this is all
based on cron, and the times are staggered to avoid peaking.

So far, we have seen none of the dreaded Delays. And since the CDIFFs
should be "immune" to cache misbehavior, we don't expect to.

Importantly, using Scripted Update (CDIFFs and thus CLDs) seems to work
OK with HAVP: but it sort of has to since HAVP just uses libclamav to
(re)load the database. (HAVP can also work with other AV engines, and
they all load their databases however *they* please.)

This is not as elegant as locally mirroring the updates, and uses a bit
more Cloudflare bandwidth (although still less in steady state than
using CVDs). I had considered using Polipo as a RAM-only HTTP proxy just
for this, but that would put more load on our gateway machine, and take
up more of my time, so I will defer it until later. (Much later.)

-pk

P.S. Thanks to Steve Basford for suggesting Polipo.


On Mon, 17 Dec 2018 19:57:35 +
"Joel Esler (jesler)"  wrote:

> Inline:
> 
> > On Dec 15, 2018, at 6:23 PM, Paul Kosinski 
> > wrote:
> > 
> > I don't know if flushing the daily.cvd cache would be adequate,
> > since there are probably some downstream caches that wouldn't
> > follow suit.
> 
> Actually I had someone correct me after I wrote this email, we
> already have been doing that the whole time.  
> 
> > 
> > Pointing *everyone* directly at Cloudflare might be expensive, if
> > that meant millions (or even thousands) of new clients.
> 
> At least it would let us know how many users we have.  Best I can
> tell on a given day, we have 2.5M users daily that hit us.  Obviously
> the unique user count is much higher (as there are several users
> behind one NAT IP, and local mirrors and the like.). Our monthly
> numbers are north of 11M users, (as some people only run freshclam
> once a week or something like that.). I guess what I am trying to say
> is, it may not be that much more traffic.
> 
> > 
> > How does Cloudflare charge Talos for ClamAV? Is the cost only per
> > byte, or is there also a significant per-connection charge. (And if
> > so, is it per HTTP or per TCP connection)? Unless the per-byte cost
> > is near zero (which is unlikely), multiple cdiffs are almost
> > certainly cheaper than one cvd.
> 
> I can't disclose those details, I'm sorry.
> 
> 
> > 
> > For my experiment, I used tinyproxy on our web server machine to
> > access Cloudflare's IAD servers instead of the BOS servers that
> > Comcast routed to, but tinyproxy doesn't do caching. That being the
> > case, I don't much like the idea of having to run squid just to
> > cache what amounts to one cdiff file for each ClamAV update.
> 
> 
> Paul, how about you just point everything you have at us and see if
> it makes a difference?
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Joel Esler (jesler)]

> On Dec 17, 2018, at 3:01 PM, Dennis Peterson  wrote:
> 
> On 12/17/18 11:57 AM, Joel Esler (jesler) wrote:
>> Inline:
>> 
>>> On Dec 15, 2018, at 6:23 PM, Paul Kosinski >> > wrote:
>>> 
>>> I don't know if flushing the daily.cvd cache would be adequate, since
>>> there are probably some downstream caches that wouldn't follow suit.
>> Actually I had someone correct me after I wrote this email, we already have 
>> been doing that the whole time.
>> 
> Thanks for that clarification - your original statement  fooled at least me :)


Me as well, obviously.



smime.p7s
Description: S/MIME cryptographic signature
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Dennis Peterson]

On 12/17/18 11:57 AM, Joel Esler (jesler) wrote:

Inline:


On Dec 15, 2018, at 6:23 PM, Paul Kosinski  wrote:

I don't know if flushing the daily.cvd cache would be adequate, since
there are probably some downstream caches that wouldn't follow suit.

Actually I had someone correct me after I wrote this email, we already have 
been doing that the whole time.


Thanks for that clarification - your original statement  fooled at least me :)


dp

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Joel Esler (jesler)]

Inline:

> On Dec 15, 2018, at 6:23 PM, Paul Kosinski  wrote:
> 
> I don't know if flushing the daily.cvd cache would be adequate, since
> there are probably some downstream caches that wouldn't follow suit.

Actually I had someone correct me after I wrote this email, we already have 
been doing that the whole time.  

> 
> Pointing *everyone* directly at Cloudflare might be expensive, if that
> meant millions (or even thousands) of new clients.

At least it would let us know how many users we have.  Best I can tell on a 
given day, we have 2.5M users daily that hit us.  Obviously the unique user 
count is much higher (as there are several users behind one NAT IP, and local 
mirrors and the like.). Our monthly numbers are north of 11M users, (as some 
people only run freshclam once a week or something like that.). I guess what I 
am trying to say is, it may not be that much more traffic.

> 
> How does Cloudflare charge Talos for ClamAV? Is the cost only per byte,
> or is there also a significant per-connection charge. (And if so, is
> it per HTTP or per TCP connection)? Unless the per-byte cost is near
> zero (which is unlikely), multiple cdiffs are almost certainly cheaper
> than one cvd.

I can't disclose those details, I'm sorry.


> 
> For my experiment, I used tinyproxy on our web server machine to access
> Cloudflare's IAD servers instead of the BOS servers that Comcast routed
> to, but tinyproxy doesn't do caching. That being the case, I don't much
> like the idea of having to run squid just to cache what amounts to one
> cdiff file for each ClamAV update.


Paul, how about you just point everything you have at us and see if it makes a 
difference?

smime.p7s
Description: S/MIME cryptographic signature
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Paul Kosinski]

I don't know if flushing the daily.cvd cache would be adequate, since
there are probably some downstream caches that wouldn't follow suit.

Pointing *everyone* directly at Cloudflare might be expensive, if that
meant millions (or even thousands) of new clients.

How does Cloudflare charge Talos for ClamAV? Is the cost only per byte,
or is there also a significant per-connection charge. (And if so, is
it per HTTP or per TCP connection)? Unless the per-byte cost is near
zero (which is unlikely), multiple cdiffs are almost certainly cheaper
than one cvd.

For my experiment, I used tinyproxy on our web server machine to access
Cloudflare's IAD servers instead of the BOS servers that Comcast routed
to, but tinyproxy doesn't do caching. That being the case, I don't much
like the idea of having to run squid just to cache what amounts to one
cdiff file for each ClamAV update.

-pk




On Sat, 15 Dec 2018 19:55:55 +
"Joel Esler (jesler)"  wrote:

> When Sourcefire acquired ClamAV "back in the day", we stopped
> accepting donations, as accounting for them on a corporate revenue
> side is more of a hassle than it is worth, so we just support it out
> of pocket.
> 
> That being said, this thread is long and I wanted to reply to is.
> 
> What if I flushed the daily.cvd cache every time we publish?  Hm...
> Pointing everyone at cloudflare is an interesting idea, may be
> expensive for me though (since I pay for cloudflare from my budget). 
> 
> Interesting discussions points here...
> 
> > On Dec 15, 2018, at 2:46 PM, Dennis Peterson 
> > wrote:
> > 
> > Things have changed a lot since Thomasz and Lucia were bearing the
> > brunt of support, but other things change slowly.
> > 
> > https://lists.gt.net/clamav/users/115
> > 
> > dp
> > 
> > On 12/15/18 10:32 AM, Gene Heskett wrote:
> >> On Saturday 15 December 2018 10:58:12 Micah Snyder (micasnyd)
> >> wrote:
> >> 
> >>> I was actually wondering about this part too.  You would need
> >>> quite a few machines downstream of your local mirror to make up
> >>> the difference switching from cdiffs for each machine to CVD's,
> >>> at least given the current size of daily.cvd.  It probably is
> >>> about time for us to fold daily into main, and start fresh with a
> >>> smaller daily.
> >>> 
> >>> I do want to say, since I'm not sure I've said it before,
> >>> thank-you to everyone who is making an effort to reduce bandwidth
> >>> usage.  Despite being a part of a huge corporation - we are an
> >>> open source project that doesn't have a subscription service or
> >>> anything to make money for the company.  As a result, we have
> >>> very limited funds year to year and your efforts do make a
> >>> difference.  Thanks!
> >>> 
> >>> -Micah
> >> NP Micah. I am a firm believer in TANSTAAFL, and have wondered why
> >> you haven't gone to small annual fee to help pay for the
> >> bandwidth, but since A, its working flawlessly here, and B, its
> >> free, I only have my freshclam looking for updates 4x a day. So I
> >> am a very light load compared to some I've read saying they are
> >> updating at 30 minute intervals. Since it appears my ISP is also
> >> blocking stuff, I could go down to a daily check. Clamscan of
> >> incoming mail, my main usage here, has only resulted in a .25
> >> megabyte viri/quarantine file in around 90 days. Thats more than
> >> good enough for the girls I go with.
> >> 
> >> Anyone, corporate or private, that is tapping your servers 48x a
> >> day, is flat out abusing the system IMNSHO. Thank you Micah and
> >> Cisco, for this service, I appreciate it.
> >>> On Dec 15, 2018, at 10:14 AM, J.R.
> >>> mailto:themadbea...@gmail.com>> wrote:
> >>> 
> >>> Third... Have you done a cost-benefit analysis? I know you said
> >>> you wanted to help reduce bandwidth, but when you are downloading
> >>> the entire daily.cvd file each time there is an update, that's
> >>> currently a little over 50MB each update. I downloaded the last
> >>> 10 cdiff files and they look to average about 15k... So by that
> >>> math (I'm still drinking my coffee this morning, so I could be
> >>> wildly wrong)... You would need to have over 3,333 machines to be
> >>> saving any bandwidth...

> 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Dennis Peterson]

Ignoring latency which is probably no where near the problem it was with the 
volunteer network of mirrors.


dp

On 12/15/18 2:43 PM, Alain Zidouemba wrote:

When a new

cdiff is released, is a new daily.cvd also released at the same time?

Yes.

-Alain


On Dec 15, 2018, at 4:26 PM, J.R.  wrote:

When a new
cdiff is released, is a new daily.cvd also released at the same time?

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Alain Zidouemba]

> When a new
cdiff is released, is a new daily.cvd also released at the same time?

Yes.

-Alain

> On Dec 15, 2018, at 4:26 PM, J.R.  wrote:
>
> When a new
> cdiff is released, is a new daily.cvd also released at the same time?
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[J.R.]

First question hopefully someone from ClamAV can answer... When a new
cdiff is released, is a new daily.cvd also released at the same time?
I would assume so, but best to get this question answered clearly than
continue to speculate.

Second, I don't think doing a manual flush of the cached file after
uploading a new daily.cvd would hurt (since the file has been changed
anyhow), but it would be interesting to do some investigating to see
if there is any change on Paul's end if he still ends up seeing the
cached copy (which would then definitely point to an ISP issue) or if
the matter resolves itself.

As for comcast (or any ISP) tech support, simply request to be bumped
up to the next support tier / level. That usually gets you from a
person that is just repeating questions from a screen to an actual
individual with some technical knowledge of how their system works.
Also if you do find someone that knows their stuff, ask for their
direct # so you can contact them in the future if you have any other
issues.

As someone else mentioned, it doesn't really matter how many times
per-day freshclam is running, it's merely doing a small DNS query to
see if there is an update or not. No update = no file transfer. The
DNS query itself is probably less than 100 bytes.
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Dennis Peterson]

This raises another point which is and has been the DNS version does not and has 
not meant there was an update to the daily CVD file - just that the cdiffs exist 
to update the users' local copy of the CLD to the current version using a 
reliable and efficient signed process. This only ever mattered to people with 
private local mirrors but there have always been private local work-arounds to 
what is largely not a problem so much as an inconvenience. Security and 
convenience have an inverse relationship.


dp

On 12/15/18 11:55 AM, Joel Esler (jesler) wrote:

When Sourcefire acquired ClamAV "back in the day", we stopped accepting 
donations, as accounting for them on a corporate revenue side is more of a hassle than it 
is worth, so we just support it out of pocket.

That being said, this thread is long and I wanted to reply to is.

What if I flushed the daily.cvd cache every time we publish?  Hm...
Pointing everyone at cloudflare is an interesting idea, may be expensive for me 
though (since I pay for cloudflare from my budget).

Interesting discussions points here...



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Joel Esler (jesler)]

When Sourcefire acquired ClamAV "back in the day", we stopped accepting 
donations, as accounting for them on a corporate revenue side is more of a 
hassle than it is worth, so we just support it out of pocket.

That being said, this thread is long and I wanted to reply to is.

What if I flushed the daily.cvd cache every time we publish?  Hm...
Pointing everyone at cloudflare is an interesting idea, may be expensive for me 
though (since I pay for cloudflare from my budget). 

Interesting discussions points here...

> On Dec 15, 2018, at 2:46 PM, Dennis Peterson  wrote:
> 
> Things have changed a lot since Thomasz and Lucia were bearing the brunt of 
> support, but other things change slowly.
> 
> https://lists.gt.net/clamav/users/115
> 
> dp
> 
> On 12/15/18 10:32 AM, Gene Heskett wrote:
>> On Saturday 15 December 2018 10:58:12 Micah Snyder (micasnyd) wrote:
>> 
>>> I was actually wondering about this part too.  You would need quite a
>>> few machines downstream of your local mirror to make up the difference
>>> switching from cdiffs for each machine to CVD's, at least given the
>>> current size of daily.cvd.  It probably is about time for us to fold
>>> daily into main, and start fresh with a smaller daily.
>>> 
>>> I do want to say, since I'm not sure I've said it before, thank-you to
>>> everyone who is making an effort to reduce bandwidth usage.  Despite
>>> being a part of a huge corporation - we are an open source project
>>> that doesn't have a subscription service or anything to make money for
>>> the company.  As a result, we have very limited funds year to year and
>>> your efforts do make a difference.  Thanks!
>>> 
>>> -Micah
>> NP Micah. I am a firm believer in TANSTAAFL, and have wondered why you
>> haven't gone to small annual fee to help pay for the bandwidth, but
>> since A, its working flawlessly here, and B, its free, I only have my
>> freshclam looking for updates 4x a day. So I am a very light load
>> compared to some I've read saying they are updating at 30 minute
>> intervals. Since it appears my ISP is also blocking stuff, I could go
>> down to a daily check. Clamscan of incoming mail, my main usage here,
>> has only resulted in a .25 megabyte viri/quarantine file in around 90
>> days. Thats more than good enough for the girls I go with.
>> 
>> Anyone, corporate or private, that is tapping your servers 48x a day, is
>> flat out abusing the system IMNSHO. Thank you Micah and Cisco, for this
>> service, I appreciate it.
>>> On Dec 15, 2018, at 10:14 AM, J.R.
>>> mailto:themadbea...@gmail.com>> wrote:
>>> 
>>> Third... Have you done a cost-benefit analysis? I know you said you
>>> wanted to help reduce bandwidth, but when you are downloading the
>>> entire daily.cvd file each time there is an update, that's currently a
>>> little over 50MB each update. I downloaded the last 10 cdiff files and
>>> they look to average about 15k... So by that math (I'm still drinking
>>> my coffee this morning, so I could be wildly wrong)... You would need
>>> to have over 3,333 machines to be saving any bandwidth...
>> 
>> 
> 
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml



smime.p7s
Description: S/MIME cryptographic signature
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Dennis Peterson]

Things have changed a lot since Thomasz and Lucia were bearing the brunt of 
support, but other things change slowly.


https://lists.gt.net/clamav/users/115

dp

On 12/15/18 10:32 AM, Gene Heskett wrote:

On Saturday 15 December 2018 10:58:12 Micah Snyder (micasnyd) wrote:


I was actually wondering about this part too.  You would need quite a
few machines downstream of your local mirror to make up the difference
switching from cdiffs for each machine to CVD's, at least given the
current size of daily.cvd.  It probably is about time for us to fold
daily into main, and start fresh with a smaller daily.

I do want to say, since I'm not sure I've said it before, thank-you to
everyone who is making an effort to reduce bandwidth usage.  Despite
being a part of a huge corporation - we are an open source project
that doesn't have a subscription service or anything to make money for
the company.  As a result, we have very limited funds year to year and
your efforts do make a difference.  Thanks!

-Micah

NP Micah. I am a firm believer in TANSTAAFL, and have wondered why you
haven't gone to small annual fee to help pay for the bandwidth, but
since A, its working flawlessly here, and B, its free, I only have my
freshclam looking for updates 4x a day. So I am a very light load
compared to some I've read saying they are updating at 30 minute
intervals. Since it appears my ISP is also blocking stuff, I could go
down to a daily check. Clamscan of incoming mail, my main usage here,
has only resulted in a .25 megabyte viri/quarantine file in around 90
days. Thats more than good enough for the girls I go with.

Anyone, corporate or private, that is tapping your servers 48x a day, is
flat out abusing the system IMNSHO. Thank you Micah and Cisco, for this
service, I appreciate it.

On Dec 15, 2018, at 10:14 AM, J.R.
mailto:themadbea...@gmail.com>> wrote:

Third... Have you done a cost-benefit analysis? I know you said you
wanted to help reduce bandwidth, but when you are downloading the
entire daily.cvd file each time there is an update, that's currently a
little over 50MB each update. I downloaded the last 10 cdiff files and
they look to average about 15k... So by that math (I'm still drinking
my coffee this morning, so I could be wildly wrong)... You would need
to have over 3,333 machines to be saving any bandwidth...





___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Paul Kosinski]

Indeed, Scripted Update via cdiffs is far more efficient until one has
*lots* of machines running ClamAV on one's LAN.  This tradeoff should
be (and have been) documented.

Better yet, the current Local Mirror mechanism should be either fixed
to support cld files (if it doesn't already) or removed (since cvds
behave badly when there are caches). Also, it occurs to me that even
the clds could have official ClamAV cryptographic signatures without
too much work: just put the proper signature for the resulting new cld
into the corresponding cdiff, and then have freshclam copy it into the
newly generated cld. This would allow clds to be locally mirrored while
preserving authentication security.

That being said, "tapping" ClamAV servers even 48 times per day via
freshclam *doesn't* use 48 times the bandwidth compared to once per day
unless the central database really is updated that often. Most of the
hits pull only a few bytes saying that there is nothing new to download.

-pk


On Sat, 15 Dec 2018 13:32:17 -0500
Gene Heskett  wrote:

> On Saturday 15 December 2018 10:58:12 Micah Snyder (micasnyd) wrote:
> 
> > I was actually wondering about this part too.  You would need quite
> > a few machines downstream of your local mirror to make up the
> > difference switching from cdiffs for each machine to CVD's, at
> > least given the current size of daily.cvd.  It probably is about
> > time for us to fold daily into main, and start fresh with a smaller
> > daily.
> >
> > I do want to say, since I'm not sure I've said it before, thank-you
> > to everyone who is making an effort to reduce bandwidth usage.
> > Despite being a part of a huge corporation - we are an open source
> > project that doesn't have a subscription service or anything to
> > make money for the company.  As a result, we have very limited
> > funds year to year and your efforts do make a difference.  Thanks!
> >
> > -Micah
> 
> NP Micah. I am a firm believer in TANSTAAFL, and have wondered why
> you haven't gone to small annual fee to help pay for the bandwidth,
> but since A, its working flawlessly here, and B, its free, I only
> have my freshclam looking for updates 4x a day. So I am a very light
> load compared to some I've read saying they are updating at 30 minute 
> intervals. Since it appears my ISP is also blocking stuff, I could go 
> down to a daily check. Clamscan of incoming mail, my main usage here, 
> has only resulted in a .25 megabyte viri/quarantine file in around 90 
> days. Thats more than good enough for the girls I go with.
> 
> Anyone, corporate or private, that is tapping your servers 48x a day,
> is flat out abusing the system IMNSHO. Thank you Micah and Cisco, for
> this service, I appreciate it.
> >
> > On Dec 15, 2018, at 10:14 AM, J.R.
> > mailto:themadbea...@gmail.com>> wrote:
> >
> > Third... Have you done a cost-benefit analysis? I know you said you
> > wanted to help reduce bandwidth, but when you are downloading the
> > entire daily.cvd file each time there is an update, that's
> > currently a little over 50MB each update. I downloaded the last 10
> > cdiff files and they look to average about 15k... So by that math
> > (I'm still drinking my coffee this morning, so I could be wildly
> > wrong)... You would need to have over 3,333 machines to be saving
> > any bandwidth...
> 
> 
> 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Gene Heskett]

On Saturday 15 December 2018 10:58:12 Micah Snyder (micasnyd) wrote:

> I was actually wondering about this part too.  You would need quite a
> few machines downstream of your local mirror to make up the difference
> switching from cdiffs for each machine to CVD's, at least given the
> current size of daily.cvd.  It probably is about time for us to fold
> daily into main, and start fresh with a smaller daily.
>
> I do want to say, since I'm not sure I've said it before, thank-you to
> everyone who is making an effort to reduce bandwidth usage.  Despite
> being a part of a huge corporation - we are an open source project
> that doesn't have a subscription service or anything to make money for
> the company.  As a result, we have very limited funds year to year and
> your efforts do make a difference.  Thanks!
>
> -Micah

NP Micah. I am a firm believer in TANSTAAFL, and have wondered why you 
haven't gone to small annual fee to help pay for the bandwidth, but 
since A, its working flawlessly here, and B, its free, I only have my 
freshclam looking for updates 4x a day. So I am a very light load 
compared to some I've read saying they are updating at 30 minute 
intervals. Since it appears my ISP is also blocking stuff, I could go 
down to a daily check. Clamscan of incoming mail, my main usage here, 
has only resulted in a .25 megabyte viri/quarantine file in around 90 
days. Thats more than good enough for the girls I go with.

Anyone, corporate or private, that is tapping your servers 48x a day, is 
flat out abusing the system IMNSHO. Thank you Micah and Cisco, for this 
service, I appreciate it.
>
> On Dec 15, 2018, at 10:14 AM, J.R.
> mailto:themadbea...@gmail.com>> wrote:
>
> Third... Have you done a cost-benefit analysis? I know you said you
> wanted to help reduce bandwidth, but when you are downloading the
> entire daily.cvd file each time there is an update, that's currently a
> little over 50MB each update. I downloaded the last 10 cdiff files and
> they look to average about 15k... So by that math (I'm still drinking
> my coffee this morning, so I could be wildly wrong)... You would need
> to have over 3,333 machines to be saving any bandwidth...



-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Paul Kosinski]

Automated configuration management sounds interesting, but we have
only a few machines running ClamAV, rather than a couple hundred, so I
doubt the effort would pay off.

We could, I suppose, have a "master" ClamAV do the Scripted Update and
then distribute updated clds to the other ClamAVs (using rsync), but
that would mean I would have to worry about synchronizing running
clamds, and perhaps our HAVP daemons.

I now conclude that with only a few ClamAV machines, the very large size
of current cvds (they used to be *much* smaller), and especially given
the vulnerability of cvds to caching, the most reasonable approach is
simply to turn on Scripted Update everywhere and have each ClamAV
machine obtain the cdiffs directly from Cloudflare. (A local HTTP proxy
would save only a trivial amount of external bandwidth, but would be a
pain to set up and maintain, since it isn't otherwise needed.)

pk

P.S. I figured the cdiffs were designed to be time invariant, but I
never saw any documentation -- hence my dumpcap experiment.



On Fri, 14 Dec 2018 22:42:40 -0800
Dennis Peterson  wrote:

>  From a best practices perspective it is best to use freshclam when
> talking to ClamAV resources. Once you have what you need from them
> you can do anything you like internally. You don't have to be nice to
> them at this point. I had a couple hundred RedHat servers to manage
> and they all required scanning software because of the industry I was
> in and because of HIPPA, credit card, social security, phone numbers
> and other personal information rules we were bound to. I created a
> lot of locally generated signatures to look for this information.
> This was before smart file systems that would do this for us.
> 
> When I built the local private mirror I used the cdiff files
> (scripted downloads were permitted) to create local patched .cld
> files. These had to be distributed to the hundreds of other machines
> and for that I initially used rsync because it is just bullet proof,
> and later I moved it all to CFengine (predecessor to puppet, chef).
> 
> The CFengine master server received the cld files from a snapshot
> file system (freshclam triggered the snapshot before and after an
> update) so new updates would not corrupt existing signature files,
> and it then immediately informed all the clients they had work to do
> to become conformal (in CFengine  terms). CFengine is smart enough to
> know to transmit differences between local and remote files on the
> fly (rsync) so net traffic is minimized as the daily and bytecode
> files don't change much. And because of the way the process works
> (creating hidden files until the differences are resolved), the
> hidden files are renamed to the original names which is very close to
> an atomic operation, so problems working with files in transport were
> prevented. The CFengine client would notify the local clamd instance
> when the files were ready. Clamd has to be told not to reload when it
> detects signature change. All very clean, fast, and secure owing to
> using secure processes at each step and hands-free on my part. It
> also passed federal government security audits which was the best
> part.
> 
> Short answer - don't use freshclam to get the signature files from
> your mirror to your clients and it won't matter if they are cld, cvd,
> cud, etc., and it doesn't burden the ClamAV servers by pulling full
> copies of CVD files.
> 
> As for the cdiff files not changing, that is by design because each
> cdiff file brings the local cld file to the cdiff version, and
> because it can't be known how many cdiffs have been created between
> user updates, they are retained for a period of time and freshclam
> applies them in order until the final cdiff matches the current DNS
> TXT record.
> 
> dp
> 
> 
> On 12/14/18 6:58 PM, Paul Kosinski wrote:
> > The Good Deed
> >
> > When we started using ClamAV, we wanted to distribute the database
> > to the several machines on our LAN in order to reduce the load on
> > the volunteer servers and minimize the load on our old DSL (now
> > gone). The best way to do this, it seemed, was to set up a trivial
> > HTTP server to mirror and deliver the new files. And, of course,
> > they had to be cvd files which, according to the FAQ, precluded
> > "Scripted Updates" and the much smaller cdiff files.
> >
> >
> > The Punishment
> >
> > This all worked quite well until ClamAV switched to distributing the
> > updates via Cloudflare: then The Delays started. The Delays
> > initially exhibited themselves when freshclam itself(!) found that
> > the DNS TXT record said that a new daily.cvd was available but upon
> > trying to retrieve it freshclam failed, complaining about network
> > problems. This eventually would cause all the mirrors to be
> > disabled.
> >
> > After much investigation (documented at length in previous posts) I
> > noticed that the daily.cvd from the BOS Cloudflare server was often
> > far behind that from the IAD Cloudflare serve

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Paul Kosinski]

Our Comcast account in in MA and is not a business account (which I
presume would cost more). My view is that Comcast tech support is on
the level of "try restarting your modem" or "try restarting Windows",
so I doubt asking about transparent caching would get very far.

I don't think it's possible to point any ClamAV machine(s) at another
Cloudflare server, as the 5 IP addresses are Anycast addresses, so they
are routed below the IP layer. (Does anyone know why there are 5 IPs?
Wouldn't one do?)

When I originally set up ClamAV, I didn't do any detailed cost/benefit
analysis: the cvds were *much* smaller, and there were no details about
what Scripted Update actually did (e.g., no size specs).

I think a local HTTP proxy wouldn't be worth the effort (nothing else
needs one), and internal replication via rsync would also require extra
effort (on my part). So for now, at least, I'll just switch to having
each ClamAV machine update directly from Cloudflare.

-pk



On Sat, 15 Dec 2018 09:14:22 -0600
"J.R."  wrote:

> I seem to recall you said you had comcast, and I'm assuming it is a
> business account. Have you tried calling their business support and
> talked to someone that is actually local to explain your problem and
> see if they possibly have a transparent cache in place and if it would
> be possible to exclude you? I also seem to recall you're located in NY
> (I could be wrong), but again being in a heavily populated area they
> could be doing the caching to try and alleviate an over-saturated
> local network. I don't think in reality the BOS cloudflare was always
> behind, I think there *has* to be some other caching going on that
> simply makes it *look* like it's behind. It also makes sense in that
> if others in your area were requesting files from the BOS server, that
> would be in the cache... But none would be manually requesting from
> another one (IAD) so there wouldn't be an existing cached copy and
> thus you get the latest version.
> 
> Many years ago the company I worked for used Akami for caching static
> content. Their caches were smart and knew when a file was changed
> (even if the name was the same), however, web browsers on the other
> hand typically had issues caching the older version. Even worse was
> when a transparent proxy was somewhere in the mix doing its own
> caching and ignoring things like when a file's date changes or
> no-cache headers. We found out our company had one in place, and we
> had to get our department excluded as it severely interfered with
> development work.
> 
> I also believe you said that one of the other cloudflare servers had
> the correct file when your local one didn't. Did you try changing your
> freshclam.conf to point to said other server(s) instead of letting it
> geo-locate you to your local cache that has caused you problems?
> 
> Third... Have you done a cost-benefit analysis? I know you said you
> wanted to help reduce bandwidth, but when you are downloading the
> entire daily.cvd file each time there is an update, that's currently a
> little over 50MB each update. I downloaded the last 10 cdiff files and
> they look to average about 15k... So by that math (I'm still drinking
> my coffee this morning, so I could be wildly wrong)... You would need
> to have over 3,333 machines to be saving any bandwidth...
> 
> Dennis posted what I was thinking about once (but didn't post about
> since I've never tried it with clamav). Once you have the data you
> need on your local network, you can push it out to clients however you
> wish. I was thinking just basic rsync, followed by a notify command
> for clamd... Or whatever newer and fancier program you might want to
> use.
> 
> Lastly, another route would be to setup your own transparent proxy, so
> even if X machines were requesting a cdiff, it only gets downloaded
> once and your local proxy caches it for all the others... You can do
> it even with HTTPS traffic so in theory it should work.

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Micah Snyder (micasnyd)]

I was actually wondering about this part too.  You would need quite a few 
machines downstream of your local mirror to make up the difference switching 
from cdiffs for each machine to CVD's, at least given the current size of 
daily.cvd.  It probably is about time for us to fold daily into main, and start 
fresh with a smaller daily.

I do want to say, since I'm not sure I've said it before, thank-you to everyone 
who is making an effort to reduce bandwidth usage.  Despite being a part of a 
huge corporation - we are an open source project that doesn't have a 
subscription service or anything to make money for the company.  As a result, 
we have very limited funds year to year and your efforts do make a difference.  
Thanks!

-Micah


On Dec 15, 2018, at 10:14 AM, J.R. 
mailto:themadbea...@gmail.com>> wrote:

Third... Have you done a cost-benefit analysis? I know you said you
wanted to help reduce bandwidth, but when you are downloading the
entire daily.cvd file each time there is an update, that's currently a
little over 50MB each update. I downloaded the last 10 cdiff files and
they look to average about 15k... So by that math (I'm still drinking
my coffee this morning, so I could be wildly wrong)... You would need
to have over 3,333 machines to be saving any bandwidth...

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[J.R.]

I seem to recall you said you had comcast, and I'm assuming it is a
business account. Have you tried calling their business support and
talked to someone that is actually local to explain your problem and
see if they possibly have a transparent cache in place and if it would
be possible to exclude you? I also seem to recall you're located in NY
(I could be wrong), but again being in a heavily populated area they
could be doing the caching to try and alleviate an over-saturated
local network. I don't think in reality the BOS cloudflare was always
behind, I think there *has* to be some other caching going on that
simply makes it *look* like it's behind. It also makes sense in that
if others in your area were requesting files from the BOS server, that
would be in the cache... But none would be manually requesting from
another one (IAD) so there wouldn't be an existing cached copy and
thus you get the latest version.

Many years ago the company I worked for used Akami for caching static
content. Their caches were smart and knew when a file was changed
(even if the name was the same), however, web browsers on the other
hand typically had issues caching the older version. Even worse was
when a transparent proxy was somewhere in the mix doing its own
caching and ignoring things like when a file's date changes or
no-cache headers. We found out our company had one in place, and we
had to get our department excluded as it severely interfered with
development work.

I also believe you said that one of the other cloudflare servers had
the correct file when your local one didn't. Did you try changing your
freshclam.conf to point to said other server(s) instead of letting it
geo-locate you to your local cache that has caused you problems?

Third... Have you done a cost-benefit analysis? I know you said you
wanted to help reduce bandwidth, but when you are downloading the
entire daily.cvd file each time there is an update, that's currently a
little over 50MB each update. I downloaded the last 10 cdiff files and
they look to average about 15k... So by that math (I'm still drinking
my coffee this morning, so I could be wildly wrong)... You would need
to have over 3,333 machines to be saving any bandwidth...

Dennis posted what I was thinking about once (but didn't post about
since I've never tried it with clamav). Once you have the data you
need on your local network, you can push it out to clients however you
wish. I was thinking just basic rsync, followed by a notify command
for clamd... Or whatever newer and fancier program you might want to
use.

Lastly, another route would be to setup your own transparent proxy, so
even if X machines were requesting a cdiff, it only gets downloaded
once and your local proxy caches it for all the others... You can do
it even with HTTPS traffic so in theory it should work.
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] No good deed goes unpunished, or, why CVD files don't work

[Dennis Peterson]

From a best practices perspective it is best to use freshclam when talking to 
ClamAV resources. Once you have what you need from them you can do anything you 
like internally. You don't have to be nice to them at this point. I had a couple 
hundred RedHat servers to manage and they all required scanning software because 
of the industry I was in and because of HIPPA, credit card, social security, 
phone numbers and other personal information rules we were bound to. I created a 
lot of locally generated signatures to look for this information. This was 
before smart file systems that would do this for us.


When I built the local private mirror I used the cdiff files (scripted downloads 
were permitted) to create local patched .cld files. These had to be distributed 
to the hundreds of other machines and for that I initially used rsync because it 
is just bullet proof, and later I moved it all to CFengine (predecessor to 
puppet, chef).


The CFengine master server received the cld files from a snapshot file system 
(freshclam triggered the snapshot before and after an update) so new updates 
would not corrupt existing signature files, and it then immediately informed all 
the clients they had work to do to become conformal (in CFengine  terms). 
CFengine is smart enough to know to transmit differences between local and 
remote files on the fly (rsync) so net traffic is minimized as the daily and 
bytecode files don't change much. And because of the way the process works 
(creating hidden files until the differences are resolved), the hidden files are 
renamed to the original names which is very close to an atomic operation, so 
problems working with files in transport were prevented. The CFengine client 
would notify the local clamd instance when the files were ready. Clamd has to be 
told not to reload when it detects signature change. All very clean, fast, and 
secure owing to using secure processes at each step and hands-free on my part. 
It also passed federal government security audits which was the best part.


Short answer - don't use freshclam to get the signature files from your mirror 
to your clients and it won't matter if they are cld, cvd, cud, etc., and it 
doesn't burden the ClamAV servers by pulling full copies of CVD files.


As for the cdiff files not changing, that is by design because each cdiff file 
brings the local cld file to the cdiff version, and because it can't be known 
how many cdiffs have been created between user updates, they are retained for a 
period of time and freshclam applies them in order until the final cdiff matches 
the current DNS TXT record.


dp


On 12/14/18 6:58 PM, Paul Kosinski wrote:

The Good Deed

When we started using ClamAV, we wanted to distribute the database
to the several machines on our LAN in order to reduce the load on the
volunteer servers and minimize the load on our old DSL (now gone). The
best way to do this, it seemed, was to set up a trivial HTTP server to
mirror and deliver the new files. And, of course, they had to be cvd
files which, according to the FAQ, precluded "Scripted Updates" and the
much smaller cdiff files.


The Punishment

This all worked quite well until ClamAV switched to distributing the
updates via Cloudflare: then The Delays started. The Delays initially
exhibited themselves when freshclam itself(!) found that the DNS TXT
record said that a new daily.cvd was available but upon trying to
retrieve it freshclam failed, complaining about network problems. This
eventually would cause all the mirrors to be disabled.

After much investigation (documented at length in previous posts) I
noticed that the daily.cvd from the BOS Cloudflare server was often far
behind that from the IAD Cloudflare server (which always seemed to
match the DNS TXT advertisement). I began to suspect that this was
perhaps caused by a caching web proxy, probably a transparent one
"helpfully" interposed by Comcast.

While all this was going on, Joel stated that nobody else was having
(or at least reporting) these Delay problems.

Now I think I know why.


The Explanation

Most everybody (I would guess) uses the Scripted Update feature, which
is enabled by default. So, I ran an experiment. On one machine I
bypassed local mirroring, enabled Scripted Update *and* captured the
HTTP traffic to/from Cloudflare via dumpcap. What I found was that
Scripted Update does HTTP GETs for one or more daily-12345.cdiff
files in sequence, each, presumably, updating "daily" from the
numerically previous version.

Now it became clear! Each daily-12345.cdiff *always* has the same
content, no matter when it is retrieved. The content of daily.cvd, on
the other hand varies over time. That makes *any* caching of daily.cvd
files susceptible to cause versioning problems, whereas the cdiff files
(such as daily-12345.cdiff) are totally invulnerable to any caching
whatsoever: web caches work according to file *name*, not file content.

This problem is exacerbated by the fact that the