Paul,

Sorry I got it backwards, I thought you were saying the TXT record was 
different which would be effected by DNS caching.
The CloudFlare cache would definitely effect daily.cvd, but updates are new.

Only way I could see you get around it yourself is to create your own cdiff 
program from the source and use the updates, 
which pretty much is using freshclam.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

> On Dec 8, 2018, at 10:37 AM, Paul Kosinski <clamav-us...@iment.com> wrote:
> 
> Not sure what DNS caching would have to do with this. As I understand
> "anycast", it happens at the IP address level. An anycast IP address
> gets routed differently depending are where you are -- different
> (regional) routers have different "next hops" for the IP address, and
> it eventually ends up at a "nearby" server. This is in addition to the
> fact that database.clamav.net resolves to 5 different IP addresses (all
> of which are anycast IPs, I would guess).
> 
> The Cloudflare servers, although they have the same IP address(es) seem
> to identify themselves by means of an HTTP header that they return:
> 
>  CF-RAY: 4852e02c35ae5a5c-BOS
>  CF-RAY: 48526af5624356c3-IAD
> 
> Finally, AFAIK, I always get the same result for 
> 
>  "dig TXT current.cvd.clamav.net"
> 
> no matter where I 'dig' (or 'host') from.
> 
> 
> 
> On Fri, 7 Dec 2018 19:27:20 -0500
> Eric Tykwinski <eric-l...@truenet.com> wrote:
> 
>> This is getting rather technical, and probably some of CloudFlare’s
>> secret sauce. It sounds like the anycast DNS that cloudflare hosts
>> isn’t really working, or at least I would assume that they are using
>> anycast.
>> 
>> So you query current.cvd.clamav.net <http://current.cvd.clamav.net/>
>> but are getting different results at IAD and BOS.  Now next is the
>> inclusion of Comcast, which may and probably is caching DNS records
>> beyond normal TTLs which could cause the difference.  I personally
>> always run an Unbound cache server on my mailserver networks to cache
>> dns for at least an hour for rbls that I’m not rsyncing, but that
>> could cause an issue with Microsoft’s wonderful 10 second MX
>> records.  So that’s where I’ve run into this issue, but not often
>> enough since I’m just caching for an hour and probably MS expects it.
>> 
>> So my guess, is probably not anycast, but a caching DNS server that
>> is still giving older records.
>> 
>> Sincerely,
>> 
>> Eric Tykwinski
>> TrueNet, Inc.
>> P: 610-429-8300
>> 
>>> On Dec 7, 2018, at 6:20 PM, Paul Kosinski <clamav-us...@iment.com>
>>> wrote:
>>> 
>>> As some of you may be aware, ever since ClamAV began using
>>> Cloudflare, we have seen many occasions when files like daily.cvd
>>> were not available to our LAN until well after the DNS TXT record
>>> implied they should be.
>>> 
>>> However, we discovered that these same files *are* available to our
>>> Web/email server right away. So what is the difference? The first
>>> difference is that our Web server (a VM) is offsite, and is served
>>> by the "IAD" Cloudflare complex, whereas our local setup is served
>>> by the "BOS" Cloudflare complex.
>>> 
>>> The second, and likely explanatory difference, is that our local
>>> setup is connected via Comcast (a dynamic IP and all that), while
>>> our Web server (with its static IP etc.) is almost certainly more
>>> directly connected to the Internet as a whole.
>>> 
>>> The workaround we have adopted is as follows: we installed a
>>> "tinyproxy" server on our offsite VM. To ensure it only proxys for
>>> us, it listens on the encrypted OpenVPN tunnel we already had in
>>> place for FTP uploads etc. Then, instead of directly accessing
>>> database.clamav.net, freshclam uses our remote VM as a proxy,so
>>> that the cvd files are downloaded indirectly from Cloudflare's IAD
>>> server complex (via tinyproxy) rather than directly from
>>> Cloudflare's BOS server complex.
>>> 
>>> Since switching to this workaround a few days ago, we haven't
>>> observed any delays: the cvd files are available right away when
>>> the DNS TXT query says they should be.
>>> 
>>> I strongly suspect that Comcast is the culprit in the delays that
>>> had plagued us. This is especially suggested by the fact that
>>> Cloudflare returns a "Cache-Control:" header similar to:
>>> 
>>> Cache-Control: public, max-age=13672
>>> 
>>> where the max-age value varies, but is often several hours.
>>> 
>>> In my opinion, for data like ClamAV virus updates, the
>>> "Cache-Control:" should specify "no-cache". Can Cloudflare do this
>>> for ClamAV?
>>> 
>>> ---------------------------------------------------------------------
>>> 
>>> Below is a pair of recent (pre-workaround) log excerpts. They show a
>>> delay of over 2.5 hours experienced from BOS (via Comcast) vs no
>>> delay from IAD.
>>> 
>>> Note that the BOS "Date:" timestamp of 16:49:01 GMT *still* shows
>>> a "Last-Modified:" timestamp of 06:15:18 GMT, while IAD already
>>> shows the up-to-date "Last-Modified:" timestamp of 14:14:30 GMT at
>>> the much earlier "Date:" of 14:29:01 GMT!
>>> 
>>> 
>>> IAD
>>> 
>>>   Date: Sun, 02 Dec 2018 14:09:01 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>>>   ClamAV-VDB:02 Dec 2018 01-14
>>> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
>>> 
>>>   Date: Sun, 02 Dec 2018 14:29:01 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 14:14:30 GMT
>>>   ClamAV-VDB:02 Dec 2018 09-13
>>> -0500:25173:2167842:63:ba557f61737b9d4b66acc96f7044b524:3nBAOxo97ssSNZb
>>> 
>>> 
>>> BOS
>>> 
>>>   Date: Sun, 02 Dec 2018 14:09:01 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>>>   ClamAV-VDB:02 Dec 2018 01-14
>>> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
>>> 
>>>   Date: Sun, 02 Dec 2018 14:29:01 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>>>   ClamAV-VDB:02 Dec 2018 01-14
>>> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
>>> 
>>>   Date: Sun, 02 Dec 2018 14:49:01 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>>>   ClamAV-VDB:02 Dec 2018 01-14
>>> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
>>> 
>>>   Date: Sun, 02 Dec 2018 15:09:01 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>>>   ClamAV-VDB:02 Dec 2018 01-14
>>> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
>>> 
>>>   Date: Sun, 02 Dec 2018 15:29:02 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>>>   ClamAV-VDB:02 Dec 2018 01-14
>>> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
>>> 
>>>   Date: Sun, 02 Dec 2018 15:49:02 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>>>   ClamAV-VDB:02 Dec 2018 01-14
>>> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
>>> 
>>>   Date: Sun, 02 Dec 2018 16:09:01 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>>>   ClamAV-VDB:02 Dec 2018 01-14
>>> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
>>> 
>>>   Date: Sun, 02 Dec 2018 16:29:01 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>>>   ClamAV-VDB:02 Dec 2018 01-14
>>> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
>>> 
>>>   Date: Sun, 02 Dec 2018 16:49:01 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
>>>   ClamAV-VDB:02 Dec 2018 01-14
>>> -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
>>> 
>>>   Date: Sun, 02 Dec 2018 17:09:01 GMT
>>>   Last-Modified: Sun, 02 Dec 2018 14:14:30 GMT
>>>   ClamAV-VDB:02 Dec 2018 09-13
>>> -0500:25173:2167842:63:ba557f61737b9d4b66acc96f7044b524:3nBAOxo97ssSNZb
> 
> _______________________________________________
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Reply via email to