I made a lot of postings last year about our ultimately unsuccessful attempts to continue using a private mirror after ClamAV switched to Cloudflare.
In summary, I had set up a wrapper for freshclam that would periodically use the DNS TXT record, followed by a curl of the first ~100 bytes of the CVDs, to determine when to download new CVDs to our mirror server. Unfortunately, the CVD files on *our* local Cloudflare server (BOS) were, as often as not, behind the CVD version numbers the DNS TXT reported. (Other Cloudflare servers, like IAD, were better.) But, since we only have a few machines running ClamAV, a local mirror didn't really save bandwidth, so I gave up and just download the CDIFFs to each machine. The downside to this is that I had to allow (strictly limited) direct Internet access from a machine that previously had none. I also tried using a simple HTTP proxy (in the IAD Anycast region) for a while, but I suspect the new HTTPS stuff would make this impractical. On Mon, 23 Mar 2020 20:57:16 +0000 "Scott A. Wozny via clamav-users" <[email protected]> wrote: > That's a very valid point. I hesitated to mention it only because my > experience with distributed mirrors has been that the files on each > mirror are timestamped when the mirror you happen to be talking to at > that moment downloaded the file which is inconsistent over the pool. > I wondered if, perhaps, the clamav mirroring structure had found a > way around that issue, but when I see freshclam does either a DNS TXT > check or a "first 512 byte header comparison" check it made me think > this was still an issue and if-modified-since couldn't be relied upon > to work until you got to the absolute last mirror to pull down the > absolute newest version which could still be pretty inefficient > overall (depending on the size of the pool compared to how often > files change). AND this issue would percolate down to the clients of > the PrivateMirror unless, of course, the issue with differing > timestamps from mirror to mirror has been resolved in some way. > > So, I'm curious if your wget --timestamping approach has been tested > in this context as truly downloading each file only once or if your > point was to note the capability of the wget command? > > Thanks, > > Scott > > ________________________________ > From: clamav-users <[email protected]> on behalf > of Arjen de Korte via clamav-users <[email protected]> > Sent: March 23, 2020 4:35 PM To: ClamAV users ML > <[email protected]> Cc: Arjen de Korte > <[email protected]> Subject: Re: [clamav-users] PrivateMirror > set on client machine. Disable cld downloads > > Citeren "Scott A. Wozny via clamav-users" > <[email protected]>: > > > One caveat with that suggestion is that if you move off of freshclam > > to do your signature retrieval with wget, you give up the > > efficiencies of just downloading the first 512 bytes of each DB file > > to see if it's been updated and, if not, going back to sleep until > > the next check. Using wget you go from a few KB of bandwidth per > > server per check to hundreds of MB for each update on each server. > > If you don't care about bandwidth and disk I/O, then this difference > > won't matter to you, but if it does, just be aware of the > > difference. > > The --timestamping option of wget will fix that for you and will only > download the file if the timestamp of the remote file changes: > > wget --timestamping http://database.clamav.net/daily.cvd > > > Alternatively, you could emulate what freshclam does and curl the > > first 512 bytes, do the comparative version checking and then > > sleeping or doing the full download. But once you get to that > > point, you're practically rewriting freshclam and then the option of > > reaching out to the devel list to modify freshclam to fit your needs > > for a custom build starts to become a comparable amount of work. > > > > Scott > > > > ________________________________ > > From: clamav-users <[email protected]> on behalf > > of G.W. Haywood via clamav-users <[email protected]> > > Sent: March 23, 2020 12:32 PM > > To: ClamAV users ML <[email protected]> > > Cc: G.W. Haywood <[email protected]> > > Subject: Re: [clamav-users] PrivateMirror set on client machine. > > Disable cld downloads > > > > Hi there, > > > > On Mon, 23 Mar 2020, vin9999 via clamav-users wrote: > > > >> ... how can we disable .cld downloads? ... > > > > Use something else to download the files? > > > > Presumably you use freshclam to keep the private mirror up to date, > > but there is nothing which forces you to use freshclam to copy files > > from the private mirror to its clients. A utility like 'wget' could > > do what you need. > > > > -- > > > > 73, > > Ged. _______________________________________________ clamav-users mailing list [email protected] https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
