Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-13 Thread Micah Snyder (micasnyd)
If you're looking at the CLD it will be bigger, because the CLD is not 
compressed and the CVD is compressed.  When you use diffs, it will store the 
database in CLD format.


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.


On Dec 12, 2018, at 11:23 PM, Dennis Peterson 
mailto:denni...@inetnw.com>> wrote:

I wonder if the file size changed when Joel regenerated the daily.cvd file  (or 
I had in unexplainable file size error). I still use all the technology but no 
longer for big dot coms. The patched files are larger because they have a lot 
of unneeded bits in them.

dp

On 12/12/18 7:43 AM, Paul Kosinski wrote:
The daily.cvd is still less than half as big as main.cvd:

  -rw-r--r-- 1 clamav clamav 117892267 Jun  7  2017 main.cvd
  -rw-r--r-- 1 clamav clamav  53147013 Dec 11 14:03 daily.cvd

but indeed using the cdiffs could save bandwidth.

I never tried using cdiffs since the FAQ said "Let freshclam download
the *.cvd files", and I wasn't sure if "scripted update" would actually
create a proper cvd for both local mirroring *and* HAVP. Also, I
figured that we were already saving lots of bandwidth by doing local
mirroring instead of N separate freshclam external downloads.

P.S. After retirement there is less pressure, but the technology I deal
with daily (for my own purposes, rather than for pay) doesn't seem to
get any simpler.


___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-12 Thread Dennis Peterson
I wonder if the file size changed when Joel regenerated the daily.cvd file  (or 
I had in unexplainable file size error). I still use all the technology but no 
longer for big dot coms. The patched files are larger because they have a lot of 
unneeded bits in them.


dp

On 12/12/18 7:43 AM, Paul Kosinski wrote:

The daily.cvd is still less than half as big as main.cvd:

   -rw-r--r-- 1 clamav clamav 117892267 Jun  7  2017 main.cvd
   -rw-r--r-- 1 clamav clamav  53147013 Dec 11 14:03 daily.cvd

but indeed using the cdiffs could save bandwidth.

I never tried using cdiffs since the FAQ said "Let freshclam download
the *.cvd files", and I wasn't sure if "scripted update" would actually
create a proper cvd for both local mirroring *and* HAVP. Also, I
figured that we were already saving lots of bandwidth by doing local
mirroring instead of N separate freshclam external downloads.

P.S. After retirement there is less pressure, but the technology I deal
with daily (for my own purposes, rather than for pay) doesn't seem to
get any simpler.



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-12 Thread Paul Kosinski
The daily.cvd is still less than half as big as main.cvd:

  -rw-r--r-- 1 clamav clamav 117892267 Jun  7  2017 main.cvd
  -rw-r--r-- 1 clamav clamav  53147013 Dec 11 14:03 daily.cvd

but indeed using the cdiffs could save bandwidth.

I never tried using cdiffs since the FAQ said "Let freshclam download
the *.cvd files", and I wasn't sure if "scripted update" would actually
create a proper cvd for both local mirroring *and* HAVP. Also, I
figured that we were already saving lots of bandwidth by doing local
mirroring instead of N separate freshclam external downloads.

P.S. After retirement there is less pressure, but the technology I deal
with daily (for my own purposes, rather than for pay) doesn't seem to
get any simpler.


On Tue, 11 Dec 2018 14:34:17 -0800
Dennis Peterson  wrote:

> You know the daily.cvd file is now larger than the main.cvd file, so
> you are burning up a lot of bandwidth if your world-facing ClamAV
> mirror is ignoring cdiff files. If it is using freshclam then it is
> using cdiffs and merging them as part of the process of mirroring. In
> that case your clients won't see the cdiff files which is perfectly
> acceptable. I used to use a proxy when many systems were co-located
> and it was very effective and was also being used for other purposes.
> Life is much simpler now that I'm retired.
> 
> dp
> 
> On 12/11/18 11:45 AM, Paul Kosinski wrote:
> > Ever since we set up a local mirror on our LAN, we have not been
> > using cdiffs. The reason for this is that I followed the procedure
> > outlined on the ClamAV website (about 2/3 down the page) at:
> >
> >http://www.clamav.net/documents/clamav-virus-database-faq
> >
> > where it says:
> >
> > [Q] I’m running ClamAV on a lot of clients on my local network.
> > Can I serve the cvd files from a local server so that each client
> > doesn’t have to download them from your servers? 
> > [A] Sure, you can find more details on our Mirror page.
> >
> > If you want to take advantage of incremental updates, install a
> > proxy server and then configure your freshclam clients to use it
> > (watch for the HTTPProxyServer parameter in man freshclam.conf). 
> > The second possible solution is to:
> >
> >Configure a local webserver on one of your machines (say
> > machine1.mylan) 
> >Let freshclam download the *.cvd files from
> > http://database.clamav.net to the webserver’s DocumentRoot. 
> >Finally, change freshclam.conf on your clients so that it
> > includes: 
> >DatabaseMirror machine1.mylan
> >
> >ScriptedUpdates off
> >
> >First the database will be downloaded to the local webserver
> > and then the other clients on the network will update their copy of
> > the database from it. 
> >Important: For this to work, you have to add ScriptedUpdates
> > off on all of your machines!
> >
> > Since I didn't want to set up a proxy server for this purpose, I
> > used the 2nd solution (and a very trivial web server). Thus, cvd
> > files only.
> >
> > P.S. I am now thinking about trying the BOS vs IAD test for cdiff
> > files. But, even if cdiff files always work without any delays,
> > doesn't "scripted update" on occasion have to back off to
> > downloading full cvds?
> >
> > P.P.S. Thanks for the curl help!
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-11 Thread Al Varnell
I have to support you in that this guidance has been there for many years now, 
but I've never really understood why that was necessary. Obviously this method 
is part of the problem that Joel has been describing about the number of users 
always downloading the .cvd and it also greatly increases local network 
traffic. 

I'm not in a position to come up with a better solution, but it would seem 
there should be a more cost-effective solution in cases where a local mirror is 
required.

-Al-
ClamXAV User

On Tue, Dec 11, 2018 at 11:45 AM, Paul Kosinski wrote:
> Ever since we set up a local mirror on our LAN, we have not been using
> cdiffs. The reason for this is that I followed the procedure outlined
> on the ClamAV website (about 2/3 down the page) at:
> 
>  http://www.clamav.net/documents/clamav-virus-database-faq 
> 
> 
> where it says:
> 
> [Q] I’m running ClamAV on a lot of clients on my local network.  Can I serve 
> the cvd files from a local server
>so that each client doesn’t have to download them from your servers?
> 
> [A] Sure, you can find more details on our Mirror page.
> 
>   If you want to take advantage of incremental updates, install a proxy 
> server and then
>configure your freshclam clients to use it (watch for the HTTPProxyServer 
> parameter in man freshclam.conf).
> 
>   The second possible solution is to:
> 
>  Configure a local webserver on one of your machines (say machine1.mylan)
> 
>  Let freshclam download the *.cvd files from http://database.clamav.net 
>  to the webserver’s DocumentRoot.
> 
>  Finally, change freshclam.conf on your clients so that it includes:
> 
>  DatabaseMirror machine1.mylan
> 
>  ScriptedUpdates off
> 
>  First the database will be downloaded to the local webserver and then 
> the other clients
>on the network will update their copy of the database from it.
> 
>  Important: For this to work, you have to add ScriptedUpdates off on all 
> of your machines!
> 
> Since I didn't want to set up a proxy server for this purpose, I used
> the 2nd solution (and a very trivial web server). Thus, cvd files only.
> 
> P.S. I am now thinking about trying the BOS vs IAD test for cdiff
> files. But, even if cdiff files always work without any delays, doesn't
> "scripted update" on occasion have to back off to downloading full cvds?
> 
> P.P.S. Thanks for the curl help!
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-11 Thread Joel Esler (jesler)
Cloudflare's cache timeout is set to 5 seconds.  So, I would doubt that 
Cloudflare's cache is the issue, it may be an ISP thing in the middle doing the 
caching, which is what Paul is guessing at this point, if I am following the 
thread correctly.

Out of an abundance of caution I did a worldwide flush of daily.cvd yesterday.  
Which caused everyone to get a new copy if it didn't match what they had.  This 
resulted in about 3TB of traffic in 10 minutes, but after that it settled back 
down.  We're still a bit higher than normal, as I eased some of the "you're 
going to fast" restrictions.  (I have a rate limiter set up, if you are 
downloading 100 cdiffs in 10 seconds, to rate limit the offender...)  I've 
disabled this for now

We're up to about 71TB a day right now and it seems to be holding steady.  Give 
it a couple more days and see if it comes back down.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com

> On Dec 10, 2018, at 9:56 PM, Eric Tykwinski  wrote:
> 
> Paul,
> 
> Sorry some of this confusion is probably my fault trying to help without 
> going back to the whole thread.
> 
>> On Dec 10, 2018, at 9:34 PM, Paul Kosinski  wrote:
>> 
>> We ARE using freshclam to perform the actual update. And always have
>> been!
>> 
>> We've only been using curl (not wget, if that matters) to pull the first
>> few bytes of the cvd to see if its version number matches what the DNS
>> TXT query said.
>> 
>> We do this because, after the conversion to Cloudflare, we were getting
>> lots of FAILURES where *freshclam* said things were out of sync (and
>> eventually disabled all the mirrors).
> 
> Have you tried what I did below?  I.E. curl/wget/telnet whatever your flavor 
> of the day, and pull the newest cdiff?
> If you’re getting a 404, that’s definitely an issue.  
> 
> My guess is that it’s actually timing out though, and could be more of an 
> issue troubleshooting.
> Is it local, ie an IDP getting stuck scanning the files, or remotely 
> freshclam itself is timing out on BOS pulling the update from ClamAV and 
> caching it before you can download it.
> 
>> And we have recently seen that our Web server sometimes can get the new
>> updates (from IAD) *hours* before our main LAN does (from BOS).
> 
> Those hours before are only checking the CVDs, which can and probably are 
> cached on CloudFlare so not up to date.
> My guess is that there are just more people in Boston using Clam, so the 
> cache last the longest.
> 
>> P.S. It's been quite frustrating getting some replies seemingly based on
>> assumptions that we are doing things we shouldn't, when we aren't in
>> fact doing those things. (Like not using freshclam.)
> 
> I would agree, this has gone on a long time from my recollection, which is 
> why I jumped in and started looking at it.
> Definitely, I did hop on without all the facts and was just trying to figure 
> out on the fly what’s going on, so my bad on that.
> 
> When in doubt, I usually pull a pcap on a server.  There’s a lot of factors 
> that can come into play, but actually with clam only using http, this 
> actually makes it a lot easier.
> 
> Sincerely,
> 
> Eric Tykwinski
> 
> 
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml



smime.p7s
Description: S/MIME cryptographic signature
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-11 Thread Dennis Peterson
You know the daily.cvd file is now larger than the main.cvd file, so you are 
burning up a lot of bandwidth if your world-facing ClamAV mirror is ignoring 
cdiff files. If it is using freshclam then it is using cdiffs and merging them 
as part of the process of mirroring. In that case your clients won't see the 
cdiff files which is perfectly acceptable. I used to use a proxy when many 
systems were co-located and it was very effective and was also being used for 
other purposes. Life is much simpler now that I'm retired.


dp

On 12/11/18 11:45 AM, Paul Kosinski wrote:

Ever since we set up a local mirror on our LAN, we have not been using
cdiffs. The reason for this is that I followed the procedure outlined
on the ClamAV website (about 2/3 down the page) at:

   http://www.clamav.net/documents/clamav-virus-database-faq

where it says:

[Q] I’m running ClamAV on a lot of clients on my local network.  Can I serve 
the cvd files from a local server
 so that each client doesn’t have to download them from your servers?
   
[A] Sure, you can find more details on our Mirror page.
   
If you want to take advantage of incremental updates, install a proxy server and then

 configure your freshclam clients to use it (watch for the HTTPProxyServer 
parameter in man freshclam.conf).
   
The second possible solution is to:
   
   Configure a local webserver on one of your machines (say machine1.mylan)
   
   Let freshclam download the *.cvd files from http://database.clamav.net to the webserver’s DocumentRoot.
   
   Finally, change freshclam.conf on your clients so that it includes:
   
   DatabaseMirror machine1.mylan
   
   ScriptedUpdates off
   
   First the database will be downloaded to the local webserver and then the other clients

 on the network will update their copy of the database from it.
   
   Important: For this to work, you have to add ScriptedUpdates off on all of your machines!


Since I didn't want to set up a proxy server for this purpose, I used
the 2nd solution (and a very trivial web server). Thus, cvd files only.

P.S. I am now thinking about trying the BOS vs IAD test for cdiff
files. But, even if cdiff files always work without any delays, doesn't
"scripted update" on occasion have to back off to downloading full cvds?

P.P.S. Thanks for the curl help!



On Mon, 10 Dec 2018 20:34:45 -0800
Dennis Peterson  wrote:


You were using curl (I did remember that after I posted as I'd helped
you sort out curl options to do what you wanted) to explore what was
available on the servers compared to what was on the DNS TXT record,
and that was outside process. It also ignored cdiff files that may
have been available in a version that matched the TXT record. The
purpose of the cdiff files is to cut down on bandwidth.

dp

On 12/10/18 6:34 PM, Paul Kosinski wrote:

We ARE using freshclam to perform the actual update. And always have
been!

We've only been using curl (not wget, if that matters) to pull the
first few bytes of the cvd to see if its version number matches
what the DNS TXT query said.

We do this because, after the conversion to Cloudflare, we were
getting lots of FAILURES where *freshclam* said things were out of
sync (and eventually disabled all the mirrors).

And we have recently seen that our Web server sometimes can get the
new updates (from IAD) *hours* before our main LAN does (from BOS).

P.S. It's been quite frustrating getting some replies seemingly
based on assumptions that we are doing things we shouldn't, when we
aren't in fact doing those things. (Like not using freshclam.)



On Mon, 10 Dec 2018 16:46:42 -0800
Dennis Peterson  wrote:


Exactly right. We can't be blaming the ClamAV process when we don't
use the ClamAV process. People that don't use freshclam should have
no expectation of high reliability. In fact any expectations are
baseless when the wrong tools are employed.

dp

On 12/9/18 5:44 AM, Joel Esler (jesler) wrote:

As it should be.  No one should be downloading the daily and main,
(although thousands are), cdiffs were created for a reason.

Sent from my  iPhone


On Dec 9, 2018, at 06:58, Eric Tykwinski 
wrote:

   From back in archives, I think he’s using wget to just pull the
files, but freshclam would just pull the cdiffs and keep you up
to date on the next check.

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-11 Thread Paul Kosinski
Ever since we set up a local mirror on our LAN, we have not been using
cdiffs. The reason for this is that I followed the procedure outlined
on the ClamAV website (about 2/3 down the page) at:

  http://www.clamav.net/documents/clamav-virus-database-faq

where it says:

[Q] I’m running ClamAV on a lot of clients on my local network.  Can I serve 
the cvd files from a local server
so that each client doesn’t have to download them from your servers?
  
[A] Sure, you can find more details on our Mirror page.
  
   If you want to take advantage of incremental updates, install a proxy server 
and then
configure your freshclam clients to use it (watch for the HTTPProxyServer 
parameter in man freshclam.conf).
  
   The second possible solution is to:
  
  Configure a local webserver on one of your machines (say machine1.mylan)
  
  Let freshclam download the *.cvd files from http://database.clamav.net to 
the webserver’s DocumentRoot.
  
  Finally, change freshclam.conf on your clients so that it includes:
  
  DatabaseMirror machine1.mylan
  
  ScriptedUpdates off
  
  First the database will be downloaded to the local webserver and then the 
other clients
on the network will update their copy of the database from it.
  
  Important: For this to work, you have to add ScriptedUpdates off on all 
of your machines!

Since I didn't want to set up a proxy server for this purpose, I used
the 2nd solution (and a very trivial web server). Thus, cvd files only.

P.S. I am now thinking about trying the BOS vs IAD test for cdiff
files. But, even if cdiff files always work without any delays, doesn't
"scripted update" on occasion have to back off to downloading full cvds?

P.P.S. Thanks for the curl help!



On Mon, 10 Dec 2018 20:34:45 -0800
Dennis Peterson  wrote:

> You were using curl (I did remember that after I posted as I'd helped
> you sort out curl options to do what you wanted) to explore what was
> available on the servers compared to what was on the DNS TXT record,
> and that was outside process. It also ignored cdiff files that may
> have been available in a version that matched the TXT record. The
> purpose of the cdiff files is to cut down on bandwidth.
> 
> dp
> 
> On 12/10/18 6:34 PM, Paul Kosinski wrote:
> > We ARE using freshclam to perform the actual update. And always have
> > been!
> >
> > We've only been using curl (not wget, if that matters) to pull the
> > first few bytes of the cvd to see if its version number matches
> > what the DNS TXT query said.
> >
> > We do this because, after the conversion to Cloudflare, we were
> > getting lots of FAILURES where *freshclam* said things were out of
> > sync (and eventually disabled all the mirrors).
> >
> > And we have recently seen that our Web server sometimes can get the
> > new updates (from IAD) *hours* before our main LAN does (from BOS).
> >
> > P.S. It's been quite frustrating getting some replies seemingly
> > based on assumptions that we are doing things we shouldn't, when we
> > aren't in fact doing those things. (Like not using freshclam.)
> >
> >
> >
> > On Mon, 10 Dec 2018 16:46:42 -0800
> > Dennis Peterson  wrote:
> >
> >> Exactly right. We can't be blaming the ClamAV process when we don't
> >> use the ClamAV process. People that don't use freshclam should have
> >> no expectation of high reliability. In fact any expectations are
> >> baseless when the wrong tools are employed.
> >>
> >> dp
> >>
> >> On 12/9/18 5:44 AM, Joel Esler (jesler) wrote:
> >>> As it should be.  No one should be downloading the daily and main,
> >>> (although thousands are), cdiffs were created for a reason.
> >>>
> >>> Sent from my  iPhone
> >>>
>  On Dec 9, 2018, at 06:58, Eric Tykwinski 
>  wrote:
> 
>    From back in archives, I think he’s using wget to just pull the
>  files, but freshclam would just pull the cdiffs and keep you up
>  to date on the next check.

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-10 Thread Dennis Peterson
You were using curl (I did remember that after I posted as I'd helped you sort 
out curl options to do what you wanted) to explore what was available on the 
servers compared to what was on the DNS TXT record, and that was outside 
process. It also ignored cdiff files that may have been available in a version 
that matched the TXT record. The purpose of the cdiff files is to cut down on 
bandwidth.


dp

On 12/10/18 6:34 PM, Paul Kosinski wrote:

We ARE using freshclam to perform the actual update. And always have
been!

We've only been using curl (not wget, if that matters) to pull the first
few bytes of the cvd to see if its version number matches what the DNS
TXT query said.

We do this because, after the conversion to Cloudflare, we were getting
lots of FAILURES where *freshclam* said things were out of sync (and
eventually disabled all the mirrors).

And we have recently seen that our Web server sometimes can get the new
updates (from IAD) *hours* before our main LAN does (from BOS).

P.S. It's been quite frustrating getting some replies seemingly based on
assumptions that we are doing things we shouldn't, when we aren't in
fact doing those things. (Like not using freshclam.)



On Mon, 10 Dec 2018 16:46:42 -0800
Dennis Peterson  wrote:


Exactly right. We can't be blaming the ClamAV process when we don't
use the ClamAV process. People that don't use freshclam should have
no expectation of high reliability. In fact any expectations are
baseless when the wrong tools are employed.

dp

On 12/9/18 5:44 AM, Joel Esler (jesler) wrote:

As it should be.  No one should be downloading the daily and main,
(although thousands are), cdiffs were created for a reason.

Sent from my  iPhone


On Dec 9, 2018, at 06:58, Eric Tykwinski 
wrote:

  From back in archives, I think he’s using wget to just pull the
files, but freshclam would just pull the cdiffs and keep you up to
date on the next check.

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-10 Thread Eric Tykwinski
Paul,

Sorry some of this confusion is probably my fault trying to help without going 
back to the whole thread.

> On Dec 10, 2018, at 9:34 PM, Paul Kosinski  wrote:
> 
> We ARE using freshclam to perform the actual update. And always have
> been!
> 
> We've only been using curl (not wget, if that matters) to pull the first
> few bytes of the cvd to see if its version number matches what the DNS
> TXT query said.
> 
> We do this because, after the conversion to Cloudflare, we were getting
> lots of FAILURES where *freshclam* said things were out of sync (and
> eventually disabled all the mirrors).

Have you tried what I did below?  I.E. curl/wget/telnet whatever your flavor of 
the day, and pull the newest cdiff?
If you’re getting a 404, that’s definitely an issue.  

My guess is that it’s actually timing out though, and could be more of an issue 
troubleshooting.
Is it local, ie an IDP getting stuck scanning the files, or remotely freshclam 
itself is timing out on BOS pulling the update from ClamAV and caching it 
before you can download it.

> And we have recently seen that our Web server sometimes can get the new
> updates (from IAD) *hours* before our main LAN does (from BOS).

Those hours before are only checking the CVDs, which can and probably are 
cached on CloudFlare so not up to date.
My guess is that there are just more people in Boston using Clam, so the cache 
last the longest.

> P.S. It's been quite frustrating getting some replies seemingly based on
> assumptions that we are doing things we shouldn't, when we aren't in
> fact doing those things. (Like not using freshclam.)

I would agree, this has gone on a long time from my recollection, which is why 
I jumped in and started looking at it.
Definitely, I did hop on without all the facts and was just trying to figure 
out on the fly what’s going on, so my bad on that.

When in doubt, I usually pull a pcap on a server.  There’s a lot of factors 
that can come into play, but actually with clam only using http, this actually 
makes it a lot easier.

Sincerely,

Eric Tykwinski


___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-10 Thread Paul Kosinski
We ARE using freshclam to perform the actual update. And always have
been!

We've only been using curl (not wget, if that matters) to pull the first
few bytes of the cvd to see if its version number matches what the DNS
TXT query said.

We do this because, after the conversion to Cloudflare, we were getting
lots of FAILURES where *freshclam* said things were out of sync (and
eventually disabled all the mirrors).

And we have recently seen that our Web server sometimes can get the new
updates (from IAD) *hours* before our main LAN does (from BOS).

P.S. It's been quite frustrating getting some replies seemingly based on
assumptions that we are doing things we shouldn't, when we aren't in
fact doing those things. (Like not using freshclam.)



On Mon, 10 Dec 2018 16:46:42 -0800
Dennis Peterson  wrote:

> Exactly right. We can't be blaming the ClamAV process when we don't
> use the ClamAV process. People that don't use freshclam should have
> no expectation of high reliability. In fact any expectations are
> baseless when the wrong tools are employed.
> 
> dp
> 
> On 12/9/18 5:44 AM, Joel Esler (jesler) wrote:
> > As it should be.  No one should be downloading the daily and main,
> > (although thousands are), cdiffs were created for a reason.
> >
> > Sent from my  iPhone
> >
> >> On Dec 9, 2018, at 06:58, Eric Tykwinski 
> >> wrote:
> >>
> >>  From back in archives, I think he’s using wget to just pull the
> >> files, but freshclam would just pull the cdiffs and keep you up to
> >> date on the next check.

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-10 Thread Eric Tykwinski
Dennis,

> On Dec 10, 2018, at 8:26 PM, Dennis Peterson  wrote:
> 
> Helps too to read the entire thread and the thread that preceded this one. 
> The OP has used combinations of dig and wget in diagnosing his problems.
> 
> dp

Seriously, then he should be just trying to pull the new cdiffs to see if they 
are propagated to the various Cloudflare hosts.

>> 
>> Sigh.
>> 
>> Does no one actually READ THE MESSAGES???
>> 
>> The OP's problem is:
>> 
>> FRESHCLAM FAILS, REPEATEDLY, UNTIL ALL MIRRORS ARE MARKED AS BAD
>> AND NO UPDATES CAN OCCUR.
>> 
>> Pissing up a rope about "you shouldn't do various work-arounds" is a waste 
>> of time and bandwidth.
>> 
>> The OP has shown that different Cloudflare nodes give (him) different 
>> results, someone should be asking CLoudflare about how this can be 
>> addressed, not dismissing the very valid and basic problem.
>> 
>> This sort of behaviour just proves that Dunning-Kruger is alive and involved 
>> in far too many OSS projects.
>> 
>> Cheers,
>> GaryB-)

Gary,

I haven’t really followed the whole thread, but I’ve been seeing it for months 
that I recall, definitely a waste of bandwidth, and probably should be solved 
to some extent.

Looking at his logs, the headers are only for a CVD, so he’s not trying updates.

Example of a cdiff pull from telnet:
telnet database.clamav.net 80
Trying 104.16.186.138...
Connected to database.clamav.net.cdn.cloudflare.net.
Escape character is '^]'.
GET /daily-25195.cdiff HTTP/1.1
host: database.clamav.net

?o??_}??/~?uЯ?|??~?f?l??Ox~??O6/??_?>??Ϸ_7?~??̯???ߢ?ӏ~???B??{}~?[A???7ņ?>???


You don’t get those nice header parts to the file, so you wouldn’t know the 
last update as it’s apart of the file itself.  Looking at manager.c on 
freshclam, he should have been posting something like: "^getfile: %s not found 
on %s (IP: %s)\n" which gets posted to the logs when the file doesn’t exist.

I’m not positive on this so Micah can chime in, but I do believe you get the 
cdiff files from the DNS TXT somehow.

If anything it’s a good lesson on how exactly freshclam works.

Sincerely,

Eric Tykwinski___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-10 Thread Dennis Peterson
Helps too to read the entire thread and the thread that preceded this one. The 
OP has used combinations of dig and wget in diagnosing his problems.


dp

On 12/10/18 5:22 PM, Gary R. Schmidt wrote:

On 11/12/2018 11:46, Dennis Peterson wrote:
Exactly right. We can't be blaming the ClamAV process when we don't use the 
ClamAV process. People that don't use freshclam should have no expectation of 
high reliability. In fact any expectations are baseless when the wrong tools 
are employed.




Sigh.

Does no one actually READ THE MESSAGES???

The OP's problem is:

FRESHCLAM FAILS, REPEATEDLY, UNTIL ALL MIRRORS ARE MARKED AS BAD
AND NO UPDATES CAN OCCUR.

Pissing up a rope about "you shouldn't do various work-arounds" is a waste of 
time and bandwidth.


The OP has shown that different Cloudflare nodes give (him) different results, 
someone should be asking CLoudflare about how this can be addressed, not 
dismissing the very valid and basic problem.


This sort of behaviour just proves that Dunning-Kruger is alive and involved 
in far too many OSS projects.


Cheers,
    Gary    B-)
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-10 Thread Gary R. Schmidt

On 11/12/2018 11:46, Dennis Peterson wrote:
Exactly right. We can't be blaming the ClamAV process when we don't use 
the ClamAV process. People that don't use freshclam should have no 
expectation of high reliability. In fact any expectations are baseless 
when the wrong tools are employed.




Sigh.

Does no one actually READ THE MESSAGES???

The OP's problem is:

FRESHCLAM FAILS, REPEATEDLY, UNTIL ALL MIRRORS ARE MARKED AS BAD
AND NO UPDATES CAN OCCUR.

Pissing up a rope about "you shouldn't do various work-arounds" is a 
waste of time and bandwidth.


The OP has shown that different Cloudflare nodes give (him) different 
results, someone should be asking CLoudflare about how this can be 
addressed, not dismissing the very valid and basic problem.


This sort of behaviour just proves that Dunning-Kruger is alive and 
involved in far too many OSS projects.


Cheers,
GaryB-)
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-10 Thread Dennis Peterson
Exactly right. We can't be blaming the ClamAV process when we don't use the 
ClamAV process. People that don't use freshclam should have no expectation of 
high reliability. In fact any expectations are baseless when the wrong tools are 
employed.


dp

On 12/9/18 5:44 AM, Joel Esler (jesler) wrote:

As it should be.  No one should be downloading the daily and main, (although 
thousands are), cdiffs were created for a reason.

Sent from my  iPhone


On Dec 9, 2018, at 06:58, Eric Tykwinski  wrote:

 From back in archives, I think he’s using wget to just pull the files, but 
freshclam would just pull the cdiffs and keep you up to date on the next check.

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-09 Thread Joel Esler (jesler)
As it should be.  No one should be downloading the daily and main, (although 
thousands are), cdiffs were created for a reason.  

Sent from my  iPhone

> On Dec 9, 2018, at 06:58, Eric Tykwinski  wrote:
> 
> From back in archives, I think he’s using wget to just pull the files, but 
> freshclam would just pull the cdiffs and keep you up to date on the next 
> check.


smime.p7s
Description: S/MIME cryptographic signature
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-09 Thread Eric Tykwinski
Joel,

> On Dec 8, 2018, at 11:21 PM, Joel Esler (jesler)  wrote:
> 
> Not sure what you’re saying here.  Are you saying that the daily on the cache 
> is out of date?
> 

I haven’t really noticed it, but that was Paul Kosinski’s observation from what 
I’m reading in the first email.
So it looks like IAD updated at 14:14:30 GMT, but BOS didn’t update till 
17:09:01 GMT from his email.

From back in archives, I think he’s using wget to just pull the files, but 
freshclam would just pull the cdiffs and keep you up to date on the next check.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-08 Thread Joel Esler (jesler)
Not sure what you’re saying here.  Are you saying that the daily on the cache 
is out of date?

Sent from my  iPhone

> On Dec 8, 2018, at 20:30, Eric Tykwinski  wrote:
> 
> J.R.
> 
> You are falling into the same trap I followed.  The txt record is:
> current.cvd.clamav.net.1749INTXT
> "0.101.0:58:25189:1544315340:1:63:48210:327"
> 
> But host headers is what he’s looking at:
> telnet database.clamav.net 80
> Trying 104.16.185.138...
> Connected to database.clamav.net.cdn.cloudflare.net.
> Escape character is '^]'.
> GET /daily.cvd HTTP/1.1
> host: database.clamav.net
> 
> HTTP/1.1 200 OK
> Date: Sun, 09 Dec 2018 01:18:51 GMT
> Content-Type: application/octet-stream
> Content-Length: 53110330
> Connection: keep-alive
> Set-Cookie: __cfduid=ddc4d2ab2a13638c99a90bb14c12128971544318331; 
> expires=Mon, 09-Dec-19 01:18:51 GMT; path=/; domain=.clamav.net; HttpOnly
> Last-Modified: Sat, 08 Dec 2018 18:18:00 GMT
> ETag: "5c0c0ad8-32a663a"
> Expires: Sun, 09 Dec 2018 05:05:51 GMT
> Cache-Control: public, max-age=13620
> CF-Cache-Status: HIT
> Accept-Ranges: bytes
> Server: cloudflare
> CF-RAY: 4863a3a9553bc5d2-EWR
> 
> ClamAV-VDB:08 Dec 2018 13-18 
> -0500:25189:2177974:63:2e2e28a4556e83e2df68c40fa61566d4:nWqDCF65xA9fMhiKYOtZhH8Up6lAHLrl6VyCrXRAXCB7aMf7WqSPrwMz/YHhdgKSNjxGiL8Z2ORQ2aPm23KwqwyJUpOZv94+soWx+NibPlKBPJ6/ZAt9Z5UrhgDbgz0IVQsHX998ZjBE6NY6xtqfzboOPNKyeFINLeAUL5hSpzj:neo:1544293134
> 
> So daily.cvd is being cached on cloudflare for the first update and you might 
> need to be running a freshclam right after a new install since it’s out of 
> date due to caching on cloudflare’s server.  
> 
> Sincerely,
> 
> Eric Tykwinski
> TrueNet, Inc.
> P: 610-429-8300
> 
>> On Dec 8, 2018, at 7:30 PM, J.R.  wrote:
>> 
>> I've kind of been reading this thread about the delay at one location
>> vs the other.
>> 
>> Maybe I missed it, but I don't seem to recall which DNS servers you
>> were querying. I remember you saying the one location you were having
>> the issues was Comcast as the ISP, but were you always using the
>> Comcast DNS or did you try others like 1.1.1.1 or 8.8.8.8 ?
>> 
>> Or was the DNS saying there was a newer version but when you queried
>> cloudflare it was reporting differently?
>> ___
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>> 
>> 
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>> 
>> http://www.clamav.net/contact.html#ml
> 
> 
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml


smime.p7s
Description: S/MIME cryptographic signature
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-08 Thread Eric Tykwinski
J.R.

You are falling into the same trap I followed.  The txt record is:
 current.cvd.clamav.net.1749IN  TXT 
"0.101.0:58:25189:1544315340:1:63:48210:327"

But host headers is what he’s looking at:
telnet database.clamav.net 80
Trying 104.16.185.138...
Connected to database.clamav.net.cdn.cloudflare.net.
Escape character is '^]'.
GET /daily.cvd HTTP/1.1
host: database.clamav.net

HTTP/1.1 200 OK
Date: Sun, 09 Dec 2018 01:18:51 GMT
Content-Type: application/octet-stream
Content-Length: 53110330
Connection: keep-alive
Set-Cookie: __cfduid=ddc4d2ab2a13638c99a90bb14c12128971544318331; expires=Mon, 
09-Dec-19 01:18:51 GMT; path=/; domain=.clamav.net; HttpOnly
Last-Modified: Sat, 08 Dec 2018 18:18:00 GMT
ETag: "5c0c0ad8-32a663a"
Expires: Sun, 09 Dec 2018 05:05:51 GMT
Cache-Control: public, max-age=13620
CF-Cache-Status: HIT
Accept-Ranges: bytes
Server: cloudflare
CF-RAY: 4863a3a9553bc5d2-EWR

ClamAV-VDB:08 Dec 2018 13-18 
-0500:25189:2177974:63:2e2e28a4556e83e2df68c40fa61566d4:nWqDCF65xA9fMhiKYOtZhH8Up6lAHLrl6VyCrXRAXCB7aMf7WqSPrwMz/YHhdgKSNjxGiL8Z2ORQ2aPm23KwqwyJUpOZv94+soWx+NibPlKBPJ6/ZAt9Z5UrhgDbgz0IVQsHX998ZjBE6NY6xtqfzboOPNKyeFINLeAUL5hSpzj:neo:1544293134

So daily.cvd is being cached on cloudflare for the first update and you might 
need to be running a freshclam right after a new install since it’s out of date 
due to caching on cloudflare’s server.  

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

> On Dec 8, 2018, at 7:30 PM, J.R.  wrote:
> 
> I've kind of been reading this thread about the delay at one location
> vs the other.
> 
> Maybe I missed it, but I don't seem to recall which DNS servers you
> were querying. I remember you saying the one location you were having
> the issues was Comcast as the ISP, but were you always using the
> Comcast DNS or did you try others like 1.1.1.1 or 8.8.8.8 ?
> 
> Or was the DNS saying there was a newer version but when you queried
> cloudflare it was reporting differently?
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml


___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-08 Thread J.R.
I've kind of been reading this thread about the delay at one location
vs the other.

Maybe I missed it, but I don't seem to recall which DNS servers you
were querying. I remember you saying the one location you were having
the issues was Comcast as the ISP, but were you always using the
Comcast DNS or did you try others like 1.1.1.1 or 8.8.8.8 ?

Or was the DNS saying there was a newer version but when you queried
cloudflare it was reporting differently?
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-08 Thread Eric Tykwinski
Paul,

Sorry I got it backwards, I thought you were saying the TXT record was 
different which would be effected by DNS caching.
The CloudFlare cache would definitely effect daily.cvd, but updates are new.

Only way I could see you get around it yourself is to create your own cdiff 
program from the source and use the updates, 
which pretty much is using freshclam.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

> On Dec 8, 2018, at 10:37 AM, Paul Kosinski  wrote:
> 
> Not sure what DNS caching would have to do with this. As I understand
> "anycast", it happens at the IP address level. An anycast IP address
> gets routed differently depending are where you are -- different
> (regional) routers have different "next hops" for the IP address, and
> it eventually ends up at a "nearby" server. This is in addition to the
> fact that database.clamav.net resolves to 5 different IP addresses (all
> of which are anycast IPs, I would guess).
> 
> The Cloudflare servers, although they have the same IP address(es) seem
> to identify themselves by means of an HTTP header that they return:
> 
>  CF-RAY: 4852e02c35ae5a5c-BOS
>  CF-RAY: 48526af5624356c3-IAD
> 
> Finally, AFAIK, I always get the same result for 
> 
>  "dig TXT current.cvd.clamav.net"
> 
> no matter where I 'dig' (or 'host') from.
> 
> 
> 
> On Fri, 7 Dec 2018 19:27:20 -0500
> Eric Tykwinski  wrote:
> 
>> This is getting rather technical, and probably some of CloudFlare’s
>> secret sauce. It sounds like the anycast DNS that cloudflare hosts
>> isn’t really working, or at least I would assume that they are using
>> anycast.
>> 
>> So you query current.cvd.clamav.net 
>> but are getting different results at IAD and BOS.  Now next is the
>> inclusion of Comcast, which may and probably is caching DNS records
>> beyond normal TTLs which could cause the difference.  I personally
>> always run an Unbound cache server on my mailserver networks to cache
>> dns for at least an hour for rbls that I’m not rsyncing, but that
>> could cause an issue with Microsoft’s wonderful 10 second MX
>> records.  So that’s where I’ve run into this issue, but not often
>> enough since I’m just caching for an hour and probably MS expects it.
>> 
>> So my guess, is probably not anycast, but a caching DNS server that
>> is still giving older records.
>> 
>> Sincerely,
>> 
>> Eric Tykwinski
>> TrueNet, Inc.
>> P: 610-429-8300
>> 
>>> On Dec 7, 2018, at 6:20 PM, Paul Kosinski 
>>> wrote:
>>> 
>>> As some of you may be aware, ever since ClamAV began using
>>> Cloudflare, we have seen many occasions when files like daily.cvd
>>> were not available to our LAN until well after the DNS TXT record
>>> implied they should be.
>>> 
>>> However, we discovered that these same files *are* available to our
>>> Web/email server right away. So what is the difference? The first
>>> difference is that our Web server (a VM) is offsite, and is served
>>> by the "IAD" Cloudflare complex, whereas our local setup is served
>>> by the "BOS" Cloudflare complex.
>>> 
>>> The second, and likely explanatory difference, is that our local
>>> setup is connected via Comcast (a dynamic IP and all that), while
>>> our Web server (with its static IP etc.) is almost certainly more
>>> directly connected to the Internet as a whole.
>>> 
>>> The workaround we have adopted is as follows: we installed a
>>> "tinyproxy" server on our offsite VM. To ensure it only proxys for
>>> us, it listens on the encrypted OpenVPN tunnel we already had in
>>> place for FTP uploads etc. Then, instead of directly accessing
>>> database.clamav.net, freshclam uses our remote VM as a proxy,so
>>> that the cvd files are downloaded indirectly from Cloudflare's IAD
>>> server complex (via tinyproxy) rather than directly from
>>> Cloudflare's BOS server complex.
>>> 
>>> Since switching to this workaround a few days ago, we haven't
>>> observed any delays: the cvd files are available right away when
>>> the DNS TXT query says they should be.
>>> 
>>> I strongly suspect that Comcast is the culprit in the delays that
>>> had plagued us. This is especially suggested by the fact that
>>> Cloudflare returns a "Cache-Control:" header similar to:
>>> 
>>> Cache-Control: public, max-age=13672
>>> 
>>> where the max-age value varies, but is often several hours.
>>> 
>>> In my opinion, for data like ClamAV virus updates, the
>>> "Cache-Control:" should specify "no-cache". Can Cloudflare do this
>>> for ClamAV?
>>> 
>>> -
>>> 
>>> Below is a pair of recent (pre-workaround) log excerpts. They show a
>>> delay of over 2.5 hours experienced from BOS (via Comcast) vs no
>>> delay from IAD.
>>> 
>>> Note that the BOS "Date:" timestamp of 16:49:01 GMT *still* shows
>>> a "Last-Modified:" timestamp of 06:15:18 GMT, while IAD already
>>> shows the up-to-date "Last-Modified:" timestamp of 14:14:30 GMT at
>>> the much earlier "Date:" of 14:29:01 GMT!
>>> 
>>> 

Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-08 Thread Paul Kosinski
Not sure what DNS caching would have to do with this. As I understand
"anycast", it happens at the IP address level. An anycast IP address
gets routed differently depending are where you are -- different
(regional) routers have different "next hops" for the IP address, and
it eventually ends up at a "nearby" server. This is in addition to the
fact that database.clamav.net resolves to 5 different IP addresses (all
of which are anycast IPs, I would guess).

The Cloudflare servers, although they have the same IP address(es) seem
to identify themselves by means of an HTTP header that they return:

  CF-RAY: 4852e02c35ae5a5c-BOS
  CF-RAY: 48526af5624356c3-IAD

Finally, AFAIK, I always get the same result for 

  "dig TXT current.cvd.clamav.net"

no matter where I 'dig' (or 'host') from.



On Fri, 7 Dec 2018 19:27:20 -0500
Eric Tykwinski  wrote:

> This is getting rather technical, and probably some of CloudFlare’s
> secret sauce. It sounds like the anycast DNS that cloudflare hosts
> isn’t really working, or at least I would assume that they are using
> anycast.
> 
> So you query current.cvd.clamav.net 
> but are getting different results at IAD and BOS.  Now next is the
> inclusion of Comcast, which may and probably is caching DNS records
> beyond normal TTLs which could cause the difference.  I personally
> always run an Unbound cache server on my mailserver networks to cache
> dns for at least an hour for rbls that I’m not rsyncing, but that
> could cause an issue with Microsoft’s wonderful 10 second MX
> records.  So that’s where I’ve run into this issue, but not often
> enough since I’m just caching for an hour and probably MS expects it.
> 
> So my guess, is probably not anycast, but a caching DNS server that
> is still giving older records.
> 
> Sincerely,
> 
> Eric Tykwinski
> TrueNet, Inc.
> P: 610-429-8300
> 
> > On Dec 7, 2018, at 6:20 PM, Paul Kosinski 
> > wrote:
> > 
> > As some of you may be aware, ever since ClamAV began using
> > Cloudflare, we have seen many occasions when files like daily.cvd
> > were not available to our LAN until well after the DNS TXT record
> > implied they should be.
> > 
> > However, we discovered that these same files *are* available to our
> > Web/email server right away. So what is the difference? The first
> > difference is that our Web server (a VM) is offsite, and is served
> > by the "IAD" Cloudflare complex, whereas our local setup is served
> > by the "BOS" Cloudflare complex.
> > 
> > The second, and likely explanatory difference, is that our local
> > setup is connected via Comcast (a dynamic IP and all that), while
> > our Web server (with its static IP etc.) is almost certainly more
> > directly connected to the Internet as a whole.
> > 
> > The workaround we have adopted is as follows: we installed a
> > "tinyproxy" server on our offsite VM. To ensure it only proxys for
> > us, it listens on the encrypted OpenVPN tunnel we already had in
> > place for FTP uploads etc. Then, instead of directly accessing
> > database.clamav.net, freshclam uses our remote VM as a proxy,so
> > that the cvd files are downloaded indirectly from Cloudflare's IAD
> > server complex (via tinyproxy) rather than directly from
> > Cloudflare's BOS server complex.
> > 
> > Since switching to this workaround a few days ago, we haven't
> > observed any delays: the cvd files are available right away when
> > the DNS TXT query says they should be.
> > 
> > I strongly suspect that Comcast is the culprit in the delays that
> > had plagued us. This is especially suggested by the fact that
> > Cloudflare returns a "Cache-Control:" header similar to:
> > 
> >  Cache-Control: public, max-age=13672
> > 
> > where the max-age value varies, but is often several hours.
> > 
> > In my opinion, for data like ClamAV virus updates, the
> > "Cache-Control:" should specify "no-cache". Can Cloudflare do this
> > for ClamAV?
> > 
> > -
> > 
> > Below is a pair of recent (pre-workaround) log excerpts. They show a
> > delay of over 2.5 hours experienced from BOS (via Comcast) vs no
> > delay from IAD.
> > 
> > Note that the BOS "Date:" timestamp of 16:49:01 GMT *still* shows
> > a "Last-Modified:" timestamp of 06:15:18 GMT, while IAD already
> > shows the up-to-date "Last-Modified:" timestamp of 14:14:30 GMT at
> > the much earlier "Date:" of 14:29:01 GMT!
> > 
> > 
> >  IAD
> > 
> >Date: Sun, 02 Dec 2018 14:09:01 GMT
> >Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
> >ClamAV-VDB:02 Dec 2018 01-14
> > -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
> > 
> >Date: Sun, 02 Dec 2018 14:29:01 GMT
> >Last-Modified: Sun, 02 Dec 2018 14:14:30 GMT
> >ClamAV-VDB:02 Dec 2018 09-13
> > -0500:25173:2167842:63:ba557f61737b9d4b66acc96f7044b524:3nBAOxo97ssSNZb
> > 
> > 
> >  BOS
> > 
> >Date: Sun, 02 Dec 2018 14:09:01 GMT
> >Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT

Re: [clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

2018-12-07 Thread Eric Tykwinski
This is getting rather technical, and probably some of CloudFlare’s secret 
sauce.
It sounds like the anycast DNS that cloudflare hosts isn’t really working, or 
at least I would assume that they are using anycast.

So you query current.cvd.clamav.net  but are 
getting different results at IAD and BOS.  Now next is the inclusion of 
Comcast, which may and probably is caching DNS records beyond normal TTLs which 
could cause the difference.  I personally always run an Unbound cache server on 
my mailserver networks to cache dns for at least an hour for rbls that I’m not 
rsyncing, but that could cause an issue with Microsoft’s wonderful 10 second MX 
records.  So that’s where I’ve run into this issue, but not often enough since 
I’m just caching for an hour and probably MS expects it.

So my guess, is probably not anycast, but a caching DNS server that is still 
giving older records.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

> On Dec 7, 2018, at 6:20 PM, Paul Kosinski  wrote:
> 
> As some of you may be aware, ever since ClamAV began using Cloudflare,
> we have seen many occasions when files like daily.cvd were not
> available to our LAN until well after the DNS TXT record implied they
> should be.
> 
> However, we discovered that these same files *are* available to our
> Web/email server right away. So what is the difference? The first
> difference is that our Web server (a VM) is offsite, and is served by
> the "IAD" Cloudflare complex, whereas our local setup is served by the
> "BOS" Cloudflare complex.
> 
> The second, and likely explanatory difference, is that our local setup
> is connected via Comcast (a dynamic IP and all that), while our Web
> server (with its static IP etc.) is almost certainly more directly
> connected to the Internet as a whole.
> 
> The workaround we have adopted is as follows: we installed a "tinyproxy"
> server on our offsite VM. To ensure it only proxys for us, it listens on
> the encrypted OpenVPN tunnel we already had in place for FTP uploads
> etc. Then, instead of directly accessing database.clamav.net, freshclam
> uses our remote VM as a proxy,so that the cvd files are downloaded
> indirectly from Cloudflare's IAD server complex (via tinyproxy) rather
> than directly from Cloudflare's BOS server complex.
> 
> Since switching to this workaround a few days ago, we haven't observed
> any delays: the cvd files are available right away when the DNS TXT
> query says they should be.
> 
> I strongly suspect that Comcast is the culprit in the delays that had
> plagued us. This is especially suggested by the fact that Cloudflare
> returns a "Cache-Control:" header similar to:
> 
>  Cache-Control: public, max-age=13672
> 
> where the max-age value varies, but is often several hours.
> 
> In my opinion, for data like ClamAV virus updates, the "Cache-Control:"
> should specify "no-cache". Can Cloudflare do this for ClamAV?
> 
> -
> 
> Below is a pair of recent (pre-workaround) log excerpts. They show a
> delay of over 2.5 hours experienced from BOS (via Comcast) vs no delay
> from IAD.
> 
> Note that the BOS "Date:" timestamp of 16:49:01 GMT *still* shows
> a "Last-Modified:" timestamp of 06:15:18 GMT, while IAD already shows
> the up-to-date "Last-Modified:" timestamp of 14:14:30 GMT at the much
> earlier "Date:" of 14:29:01 GMT!
> 
> 
>  IAD
> 
>Date: Sun, 02 Dec 2018 14:09:01 GMT
>Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>ClamAV-VDB:02 Dec 2018 01-14 
> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
> 
>Date: Sun, 02 Dec 2018 14:29:01 GMT
>Last-Modified: Sun, 02 Dec 2018 14:14:30 GMT
>ClamAV-VDB:02 Dec 2018 09-13 
> -0500:25173:2167842:63:ba557f61737b9d4b66acc96f7044b524:3nBAOxo97ssSNZb
> 
> 
>  BOS
> 
>Date: Sun, 02 Dec 2018 14:09:01 GMT
>Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>ClamAV-VDB:02 Dec 2018 01-14 
> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
> 
>Date: Sun, 02 Dec 2018 14:29:01 GMT
>Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>ClamAV-VDB:02 Dec 2018 01-14 
> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
> 
>Date: Sun, 02 Dec 2018 14:49:01 GMT
>Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>ClamAV-VDB:02 Dec 2018 01-14 
> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
> 
>Date: Sun, 02 Dec 2018 15:09:01 GMT
>Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>ClamAV-VDB:02 Dec 2018 01-14 
> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
> 
>Date: Sun, 02 Dec 2018 15:29:02 GMT
>Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>ClamAV-VDB:02 Dec 2018 01-14 
> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
> 
>Date: Sun, 02 Dec 2018 15:49:02 GMT
>Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>ClamAV-VDB:02 Dec 2018