Re: [clamav-users] ClamAV mirrors have gotten worse!

[Dennis Peterson]

I think these reports don't tell you what you think they mean. In fact they're 
pretty much meaningless. The two different servers have different versions of 
the signature. That is perfectly normal - there is simply zero chance and it is 
naive to think they will always be fully synced in the same second of time of 
day. You can infer nothing when this occurs.


In any event these signature serial numbers are associated with the DNS txt 
record. The designed process is entirely serial - freshclam knows your installed 
signature file serial number, it knows the DNS txt record, and it requests 
updates from any of the signature servers if the local version is different from 
the DNS txt record. It will try all the mirrors until success or the list of 
mirrors is exhausted. Other things that mess with the fully synchronized state 
is that DNS caching, TTL, local system clock differences, and policies of 
various name service admins to ignore authoritative TTL suggestions.


The database.clamav.net dns is a round robin of 5 different servers and you 
cannot predict what you will receive. In fact in the best case the list be 
reordered each time you request the A record. And the chances of two different 
clients getting the same A record is very low.


Your own local resolver looks in its own cache to see if it has expired. The TTL 
record for the TXT record is 1800 seconds. If you use the dig command retrieve 
the TXT record you can watch the TTL count down:


    dig  txt current.cvd.clamav.net |grep TXT

To eliminate this as a problem source you can always use host table entries 
rather than dns for your tests. The round robin records ensure reliability for 
the client and crude load balancing for the server farm.


So worst case is the record you see can be 1800 seconds behind an updated TXT 
record. Obviously polling the current.cvd.clamav.net server directly will return 
an uncached record at the expense of recursing queries (use the IP instead of 
the hostname to avoid this).


Because these variables exist, freshclam is somewhat fault tolerant and will 
retry 3 times per mirror (default and is configurable), and if a mirror is in a 
failed state freshclam will map it out of the servers to try next time 
(mirrors.dat). The other variable is some of the sync process is demand-driven. 
In very busy systems (which these are) stale files should not exist very long. 
Your request just might be a trigger to refresh a stale file, and the next 
person to hit that server will retrieve the updated file and your system will 
move to another mirror. This scenario presumes files are pulled to the mirrors, 
not pushed.


I do believe your angst over not having complete system synchronization is 
unwarranted as there are too many uncontrollable variables and it's really not 
critical if the first mirror doesn't respond.


Finally - the current cloudflare process is pretty solid - it is a vast 
improvement over the historical mirror collaboration


On 11/26/18 4:19 PM, Paul Kosinski wrote:

I believe that the delays we have been observing are due to some
problem with the Boston Cloudflare servers, or, perhaps, Comcast has a
"transparent" caching proxy which is causing us trouble.

I recently installed the same build and configuration of ClamAV 0.100.2
on our Web server, a virtual machine hosted in NYC. It runs the same
extra code (curl etc.) to check the cvd version number that we have
locally. Since Friday, there have been no delays there, although there
have been several significant delays locally. They check at exactly
the same time as each other (i.e., via NTP synced cron jobs).

I also am now running, at each location, simple curls to read the first
few bytes of the cvd files (to get the version number), *and* to log
all the headers sent and received. These are also run at exactly the
same time (as each other) via cron.

The headers show that our local system uses the 'BOS' Cloudflare server,
while the remote one uses the 'IAD' server:

   CF-RAY: 47fd0b7af79dae32-BOS
   CF-RAY: 47fd0b8064d9c1b8-IAD

Interestingly, these two cron jobs sometimes show that the BOS server
is out of date relative to the IAD server. For example, the following
curls show that one cvd file served by the BOS server is one version
behind that served by the IAD server at the *same* time. The files'
"Last-modified" lines are of particular interest. The BOS server says
the file was last modified on Mon, 26 Nov 2018 at 06:19:22 GMT, while
the IAD server says the file was last modified on Mon, 26 Nov 2018 at
14:15:24 GMT.

In particular, the BOS "Date:" header says it's already about 14 mins
*later* than the IAD "Last-modified:" timestamp indicates. In other
words, the file delivered by the BOS server is, at time of *delivery*,
already about 14 minutes out of date.



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us 

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Joel Esler (jesler)]

The "out of date at one mirror" issue you are speaking of is fine.  Once 
someone requests a file, it is cached at that POP site.  All further requests 
to other POPs then check "sister" POP sites to see if the other POP sites have 
the file first, then download it laterally from one POP to another.

So, if you are hitting two POPs at once, you may just be the first one at that 
POP to request the file.

After the first request it should grab the file and serve it.  

> On Nov 26, 2018, at 7:19 PM, Paul Kosinski  wrote:
> 
> I believe that the delays we have been observing are due to some
> problem with the Boston Cloudflare servers, or, perhaps, Comcast has a
> "transparent" caching proxy which is causing us trouble.
> 
> I recently installed the same build and configuration of ClamAV 0.100.2
> on our Web server, a virtual machine hosted in NYC. It runs the same
> extra code (curl etc.) to check the cvd version number that we have
> locally. Since Friday, there have been no delays there, although there
> have been several significant delays locally. They check at exactly
> the same time as each other (i.e., via NTP synced cron jobs).
> 
> I also am now running, at each location, simple curls to read the first
> few bytes of the cvd files (to get the version number), *and* to log
> all the headers sent and received. These are also run at exactly the
> same time (as each other) via cron.
> 
> The headers show that our local system uses the 'BOS' Cloudflare server,
> while the remote one uses the 'IAD' server:
> 
>  CF-RAY: 47fd0b7af79dae32-BOS
>  CF-RAY: 47fd0b8064d9c1b8-IAD
> 
> Interestingly, these two cron jobs sometimes show that the BOS server
> is out of date relative to the IAD server. For example, the following
> curls show that one cvd file served by the BOS server is one version
> behind that served by the IAD server at the *same* time. The files'
> "Last-modified" lines are of particular interest. The BOS server says
> the file was last modified on Mon, 26 Nov 2018 at 06:19:22 GMT, while
> the IAD server says the file was last modified on Mon, 26 Nov 2018 at
> 14:15:24 GMT. 
> 
> In particular, the BOS "Date:" header says it's already about 14 mins
> *later* than the IAD "Last-modified:" timestamp indicates. In other
> words, the file delivered by the BOS server is, at time of *delivery*,
> already about 14 minutes out of date.
> 
> --- BOS server ---
> 
>--  Monday 26 November 2018 at 09:29:01  
> --
> 
>/usr/bin/curl -4 -0 -s -v -r 0-99  -H Connection:close  -A ClamAV/0.100.2  
>   database.clamav.net/daily.cvd  2>&1
>* About to connect() to database.clamav.net port 80 (#0)
>*   Trying 104.16.187.138...
>* connected
>* Connected to database.clamav.net (104.16.187.138) port 80 (#0)
>> GET /daily.cvd HTTP/1.0
>> Range: bytes=0-99
>> User-Agent: ClamAV/0.100.2
>> Host: database.clamav.net
>> Accept: */*
>> Connection:close
>> 
>* additional stuff not fine transfer.c:1042: 0 0
>* HTTP 1.1 or later with persistent connection, pipelining supported
>< HTTP/1.1 206 Partial Content
>< Date: Mon, 26 Nov 2018 14:29:01 GMT
>< Content-Type: application/octet-stream
>< Content-Length: 100
>< Connection: close
>< Set-Cookie: __cfduid=d6eb82c36b149e2a4d07b430117606e581543242541; 
> expires=Tue, 26-Nov-19 14:29:01 GMT; path=/; domain=.clamav.net; HttpOnly
>< Last-Modified: Mon, 26 Nov 2018 06:19:22 GMT
>< ETag: "5bfb906a-321a420"
>< Expires: Mon, 26 Nov 2018 18:23:39 GMT
>< Cache-Control: public, max-age=14078
>< CF-Cache-Status: HIT
>< Content-Range: bytes 0-99/52536352
>< Server: cloudflare
>< CF-RAY: 47fd0b7af79dae32-BOS
><
>{ [data not shown]
>* Closing connection #0
>ClamAV-VDB:26 Nov 2018 01-15 
> -0500:25154:2160594:63:083b8d4aa3824a865ac0e3ebeb3f7ce0:MT9OksyAAA34bbI
> 
>--  Monday 26 November 2018 at 09:29:01  
> --
> 
> 
> --- IAD server ---
> 
>--  Monday 26 November 2018 at 09:29:02  
> --
> 
>/usr/bin/curl -4 -0 -s -v -r 0-99  -H Connection:close  -A ClamAV/0.100.2  
>   database.clamav.net/daily.cvd  2>&1
>* About to connect() to database.clamav.net port 80 (#0)
>*   Trying 104.16.185.138...
>* Connected to database.clamav.net (104.16.185.138) port 80 (#0)
>> GET /daily.cvd HTTP/1.0
>> Range: bytes=0-99
>> User-Agent: ClamAV/0.100.2
>> Host: database.clamav.net
>> Accept: */*
>> Connection:close
>> 
>< HTTP/1.1 206 Partial Content
>< Date: Mon, 26 Nov 2018 14:29:02 GMT
>< Content-Type: application/octet-stream
>< Content-Length: 100
>< Connection: close
>< Set-Cookie: __cfduid=d426fd78ff1d6c6e42029baf939e5bbee1543242542; 
> expires=Tue, 26-Nov-19 14:29:02 GMT; path=/; domain=.clamav.net; HttpOnly
>< Last-Modified: Mon, 26 Nov 2018 14:15:24 GMT
>< ETag: "5bfbfffc-321bb54"
>

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Paul Kosinski]

I believe that the delays we have been observing are due to some
problem with the Boston Cloudflare servers, or, perhaps, Comcast has a
"transparent" caching proxy which is causing us trouble.

I recently installed the same build and configuration of ClamAV 0.100.2
on our Web server, a virtual machine hosted in NYC. It runs the same
extra code (curl etc.) to check the cvd version number that we have
locally. Since Friday, there have been no delays there, although there
have been several significant delays locally. They check at exactly
the same time as each other (i.e., via NTP synced cron jobs).

I also am now running, at each location, simple curls to read the first
few bytes of the cvd files (to get the version number), *and* to log
all the headers sent and received. These are also run at exactly the
same time (as each other) via cron.

The headers show that our local system uses the 'BOS' Cloudflare server,
while the remote one uses the 'IAD' server:

  CF-RAY: 47fd0b7af79dae32-BOS
  CF-RAY: 47fd0b8064d9c1b8-IAD

Interestingly, these two cron jobs sometimes show that the BOS server
is out of date relative to the IAD server. For example, the following
curls show that one cvd file served by the BOS server is one version
behind that served by the IAD server at the *same* time. The files'
"Last-modified" lines are of particular interest. The BOS server says
the file was last modified on Mon, 26 Nov 2018 at 06:19:22 GMT, while
the IAD server says the file was last modified on Mon, 26 Nov 2018 at
14:15:24 GMT. 

In particular, the BOS "Date:" header says it's already about 14 mins
*later* than the IAD "Last-modified:" timestamp indicates. In other
words, the file delivered by the BOS server is, at time of *delivery*,
already about 14 minutes out of date.

--- BOS server ---

--  Monday 26 November 2018 at 09:29:01  
--

/usr/bin/curl -4 -0 -s -v -r 0-99  -H Connection:close  -A ClamAV/0.100.2   
 database.clamav.net/daily.cvd  2>&1
* About to connect() to database.clamav.net port 80 (#0)
*   Trying 104.16.187.138...
* connected
* Connected to database.clamav.net (104.16.187.138) port 80 (#0)
> GET /daily.cvd HTTP/1.0
> Range: bytes=0-99
> User-Agent: ClamAV/0.100.2
> Host: database.clamav.net
> Accept: */*
> Connection:close
>
* additional stuff not fine transfer.c:1042: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 206 Partial Content
< Date: Mon, 26 Nov 2018 14:29:01 GMT
< Content-Type: application/octet-stream
< Content-Length: 100
< Connection: close
< Set-Cookie: __cfduid=d6eb82c36b149e2a4d07b430117606e581543242541; 
expires=Tue, 26-Nov-19 14:29:01 GMT; path=/; domain=.clamav.net; HttpOnly
< Last-Modified: Mon, 26 Nov 2018 06:19:22 GMT
< ETag: "5bfb906a-321a420"
< Expires: Mon, 26 Nov 2018 18:23:39 GMT
< Cache-Control: public, max-age=14078
< CF-Cache-Status: HIT
< Content-Range: bytes 0-99/52536352
< Server: cloudflare
< CF-RAY: 47fd0b7af79dae32-BOS
<
{ [data not shown]
* Closing connection #0
ClamAV-VDB:26 Nov 2018 01-15 
-0500:25154:2160594:63:083b8d4aa3824a865ac0e3ebeb3f7ce0:MT9OksyAAA34bbI

--  Monday 26 November 2018 at 09:29:01  
--


--- IAD server ---

--  Monday 26 November 2018 at 09:29:02  
--

/usr/bin/curl -4 -0 -s -v -r 0-99  -H Connection:close  -A ClamAV/0.100.2   
 database.clamav.net/daily.cvd  2>&1
* About to connect() to database.clamav.net port 80 (#0)
*   Trying 104.16.185.138...
* Connected to database.clamav.net (104.16.185.138) port 80 (#0)
> GET /daily.cvd HTTP/1.0
> Range: bytes=0-99
> User-Agent: ClamAV/0.100.2
> Host: database.clamav.net
> Accept: */*
> Connection:close
>
< HTTP/1.1 206 Partial Content
< Date: Mon, 26 Nov 2018 14:29:02 GMT
< Content-Type: application/octet-stream
< Content-Length: 100
< Connection: close
< Set-Cookie: __cfduid=d426fd78ff1d6c6e42029baf939e5bbee1543242542; 
expires=Tue, 26-Nov-19 14:29:02 GMT; path=/; domain=.clamav.net; HttpOnly
< Last-Modified: Mon, 26 Nov 2018 14:15:24 GMT
< ETag: "5bfbfffc-321bb54"
< Expires: Mon, 26 Nov 2018 18:23:46 GMT
< Cache-Control: public, max-age=14084
< CF-Cache-Status: HIT
< Content-Range: bytes 0-99/52542292
< Server: cloudflare
< CF-RAY: 47fd0b8064d9c1b8-IAD
<
{ [data not shown]
* Closing connection 0
ClamAV-VDB:26 Nov 2018 09-14 
-0500:25155:2160841:63:9817036334370e1482f3fc58c6ed745a:MDvX2VW3tQr3ba4

--  Monday 26 November 2018 at 09:29:02  
--


P.S. As far as I can tell, there are no Cloudflare "blocks".


===

On 

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Paul Kosinski]

"I might be a little late to the party here, but are you saying that
10.11.14.160 is the IP address which the Cloudflare servers see?"

Sorry, I left out a bit. The 10.11.14.160 is the address assigned to
a NIC on our firewall / gateway / internal router machine, but that NIC
is connected to a Netgear C7100V cable modem which runs in NAT/routing
mode. The actual public IP address, as provided by Comcast via DHCP, is
currently 66.31.152.192 (as it has been since well before Cloudflare).

The reason for this complicated setup is partly historical: until last
month we *also* had a DSL connection to the Internet. So the gateway did
the routing (including "Policy Routing") for that. And we have a few
other 10.x.x.x LANs internally, which we will keep. Plus, the gateway
runs iptables, HAVP (which uses libclamav), Privoxy, some OpenVPN
tunnels and occasionally captures packets to/from the Internet for
analysis.

Although we *could* run the cable modem in bridge mode, that would mean
our gateway machine would be hit with lots of IPv6 packets, which we
would just have to drop. (Our internal LANs aren't set up for IPv6 and,
until some crucial Internet services won't work over IPv4, I have too
many other things to do.)

P.S. I think we've been using ClamAV since 0.86.2, back in July 2005
(how time flies), and I've generally been very happy with it.


On Fri, 23 Nov 2018 18:32:00 + (GMT)
"G.W. Haywood"  wrote:

> Hi there,
> 
> On Thu, 22 Nov 2018, Paul Kosinski wrote:
> 
> > I wonder how many users of ClamAV actually log their freshclam
> > updates.
> 
> I've been using ClamAV for more than a decade.  I've already said on
> the list that I log all freshclam updates and that in general my
> experience is that the mirrors are very reliable.  During the switch
> to Cloudflare I experienced no problems.  There was an issue back in
> March/April this year which I believe was the first that ran for more
> than a day for as long as I can remember.
> 
> Like Mr. Peterson (and *un*like this mailing list's configuration :) I
> run freshclam so as to avoid pile-ups.  Typically I fetch three
> updates per day and they succeed as regularly as clockwork.
> 
> On Wed, 21 Nov 2018, Paul Kosinski wrote:
> 
> > ... 'LocalIPAddress' is the *outgoing* IP address ...
> > .. To sum up, the "LocalIPAddress 10.11.14.160" is ...
> 
> I might be a little late to the party here, but are you saying that
> 10.11.14.160 is the IP address which the Cloudflare servers see?  Some
> ISPs do issue RFC1918 addresses to their clients but I'd expect a NAT
> gateway somewhere between your interface and Cloudflare, so that the
> Cloudflare servers see an Internet-routable IP.  If there is such a
> gateway, maybe it's worth a look on the other side of it.  If there
> isn't, I'd expect problems because packets from/to RFC1918 addresses
> should never be allowed onto the public Internet.  Please forgive me
> for teaching granny; this *might* be news to some who are reading now.
> 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Paul Kosinski]

"But the OP has *refused* to consider trying that."

Nonsense!

I did, of course, do that for a while, in a desperate attempt to get any
updates at all. But it seemed like an incredibly crude way to try to
get back to the pre-Cloudflare behavior when updates didn't regularly
fail (and eventually block all updates).

Merely getting updates was not the problem, since that could be "fixed"
by deleting mirrors.dat every so often, but I wanted to get them when
they were advertised (via the DNS TXT) as being available. (On occasion,
a delay in getting virus updates may be dangerous.)

That's when I started to do something more sophisticated (the curl read
of the cvd header), to gather some data as to exactly what was going on.

Even if it's only our installation that has this problem, I'd still like
to fully solve it.


On Fri, 23 Nov 2018 23:00:20 +1100
"Gary R. Schmidt"  wrote:

> On 23/11/2018 22:45, Gene Heskett wrote:
> > On Friday 23 November 2018 03:43:40 Dennis Peterson wrote:
> > 
> >> On 11/22/18 8:51 PM, Paul Kosinski wrote:
> >>> I wonder how many users of ClamAV actually log their freshclam
> >>> updates. Those who don't likely won't notice freshclam temporary
> >>> failures due to an out-of-sync condition.
> >>
> >> I just checked logs on two systems dating from July 1 and see no
> >> failures. I isolated the signature serial numbers and time tags and
> >> all were received with clock like precision. Freshclam is launched
> >> every three hours from cron.d and incorporates a randomizer to
> >> create a delay to help avoid pileups on common cardinal clock
> >> positions. No serial numbers were missed within the time slot.
> >>
> >> dp
> > 
> > I did keep a tail on mine, but it Just Works(TM)  That is what its
> > supposed to do I believe. :)
> > 
> As does mine, as it has since I don't know when, and my
> freshclam.conf has logging turned on.
> 
> It *didn't* work during the switch over to Cloudflare, but has been
> fine since that stabilised, but when it wasn't working I had a cron
> job that deleted "mirrors.dat" every hour, which reduced the failure
> rate.
> 
> But the OP has refused to consider trying that.
> 
>   Cheers,
>   GaryB-)

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[G.W. Haywood]

Hi there,

On Thu, 22 Nov 2018, Paul Kosinski wrote:


I wonder how many users of ClamAV actually log their freshclam updates.


I've been using ClamAV for more than a decade.  I've already said on
the list that I log all freshclam updates and that in general my
experience is that the mirrors are very reliable.  During the switch
to Cloudflare I experienced no problems.  There was an issue back in
March/April this year which I believe was the first that ran for more
than a day for as long as I can remember.

Like Mr. Peterson (and *un*like this mailing list's configuration :) I
run freshclam so as to avoid pile-ups.  Typically I fetch three updates
per day and they succeed as regularly as clockwork.

On Wed, 21 Nov 2018, Paul Kosinski wrote:


... 'LocalIPAddress' is the *outgoing* IP address ...
.. To sum up, the "LocalIPAddress 10.11.14.160" is ...


I might be a little late to the party here, but are you saying that
10.11.14.160 is the IP address which the Cloudflare servers see?  Some
ISPs do issue RFC1918 addresses to their clients but I'd expect a NAT
gateway somewhere between your interface and Cloudflare, so that the
Cloudflare servers see an Internet-routable IP.  If there is such a
gateway, maybe it's worth a look on the other side of it.  If there
isn't, I'd expect problems because packets from/to RFC1918 addresses
should never be allowed onto the public Internet.  Please forgive me
for teaching granny; this *might* be news to some who are reading now.

--

73,
Ged.
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Joel Esler (jesler)]

The “be” error was my fault.  Plain and simple.  I misconfigured a dns entry.  

Sent from my  iPhone

On Nov 23, 2018, at 04:28, Pierre Dehaen  wrote:

>> On 11/22/18 8:51 PM, Paul Kosinski wrote:
>> I wonder how many users of ClamAV actually log their freshclam updates.
>> Those who don't likely won't notice freshclam temporary failures due
>> to an out-of-sync condition.
> 
> I do log and do analyze all logs on all servers everyday, sometimes every 
> hour (a little script 
> sends me an email if anything abnormal happens).
> 
> If you mean "Mirror not synchronized" messages, I've received some since 2016 
> (list 
> attached) but there was no big issue excepted the recent problem with the 
> "be" mirror, now 
> fixed, that I submitted here. The logs show that the errors generally happen 
> in a row, maybe 
> some temporary issues on some servers?
> 
> Pierre
> 
> 
> 
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml


smime.p7s
Description: S/MIME cryptographic signature
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Gary R. Schmidt]

On 23/11/2018 22:45, Gene Heskett wrote:

On Friday 23 November 2018 03:43:40 Dennis Peterson wrote:


On 11/22/18 8:51 PM, Paul Kosinski wrote:

I wonder how many users of ClamAV actually log their freshclam
updates. Those who don't likely won't notice freshclam temporary
failures due to an out-of-sync condition.


I just checked logs on two systems dating from July 1 and see no
failures. I isolated the signature serial numbers and time tags and
all were received with clock like precision. Freshclam is launched
every three hours from cron.d and incorporates a randomizer to create
a delay to help avoid pileups on common cardinal clock positions. No
serial numbers were missed within the time slot.

dp


I did keep a tail on mine, but it Just Works(TM)  That is what its
supposed to do I believe. :)

As does mine, as it has since I don't know when, and my freshclam.conf 
has logging turned on.


It *didn't* work during the switch over to Cloudflare, but has been fine 
since that stabilised, but when it wasn't working I had a cron job that 
deleted "mirrors.dat" every hour, which reduced the failure rate.


But the OP has refused to consider trying that.

Cheers,
GaryB-)
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Gene Heskett]

On Friday 23 November 2018 03:43:40 Dennis Peterson wrote:

> On 11/22/18 8:51 PM, Paul Kosinski wrote:
> > I wonder how many users of ClamAV actually log their freshclam
> > updates. Those who don't likely won't notice freshclam temporary
> > failures due to an out-of-sync condition.
>
> I just checked logs on two systems dating from July 1 and see no
> failures. I isolated the signature serial numbers and time tags and
> all were received with clock like precision. Freshclam is launched
> every three hours from cron.d and incorporates a randomizer to create
> a delay to help avoid pileups on common cardinal clock positions. No
> serial numbers were missed within the time slot.
>
> dp

I did keep a tail on mine, but it Just Works(TM)  That is what its 
supposed to do I believe. :)

> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml



-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Pierre Dehaen]

> On 11/22/18 8:51 PM, Paul Kosinski wrote:
> I wonder how many users of ClamAV actually log their freshclam updates.
> Those who don't likely won't notice freshclam temporary failures due
> to an out-of-sync condition.

I do log and do analyze all logs on all servers everyday, sometimes every hour 
(a little script 
sends me an email if anything abnormal happens).

If you mean "Mirror not synchronized" messages, I've received some since 2016 
(list 
attached) but there was no big issue excepted the recent problem with the "be" 
mirror, now 
fixed, that I submitted here. The logs show that the errors generally happen in 
a row, maybe 
some temporary issues on some servers?

Pierre


<>
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Dennis Peterson]

On 11/22/18 8:51 PM, Paul Kosinski wrote:

I wonder how many users of ClamAV actually log their freshclam updates.
Those who don't likely won't notice freshclam temporary failures due
to an out-of-sync condition.


I just checked logs on two systems dating from July 1 and see no failures. I 
isolated the signature serial numbers and time tags and all were received with 
clock like precision. Freshclam is launched every three hours from cron.d and 
incorporates a randomizer to create a delay to help avoid pileups on common 
cardinal clock positions. No serial numbers were missed within the time slot.


dp

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Paul Kosinski]

I was just looking at freshclam.conf.sample in 0.101.2, and it looks
like *all* logging is disabled by default  (back to 0.98.6, at least).
I wonder how many users of ClamAV actually log their freshclam updates.
Those who don't likely won't notice freshclam temporary failures due
to an out-of-sync condition.

Also, timestamps are a logging option which is separate, and disabled
by default. Thus even people who enable some logging may not see the
details of any delays. (I, personally, don't understand how logging
without timestamps is of much use.)

In any case, we do eventually get (hopefully all) the virus DB updates,
assuming at least one mirror remains enabled in mirrors.dat.


On Tue, 20 Nov 2018 22:39:44 +
"Joel Esler (jesler)"  wrote:

> It's possible.  But, unless there is a vocal minority that no one is
> chiming in about, you are the only person/group that I have heard
> complain about the issue...
> 
> Millions of people are getting updates from Cloudflare a day, so
> something is working correctly, and there's been no configuration
> changes on our side.
> 
> If you receive Cloudflare blocks then that's a different story.
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Joel Esler (jesler)]

It's possible.  But, unless there is a vocal minority that no one is chiming in 
about, you are the only person/group that I have heard complain about the 
issue...

Millions of people are getting updates from Cloudflare a day, so something is 
working correctly, and there's been no configuration changes on our side.

If you receive Cloudflare blocks then that's a different story.

On Nov 20, 2018, at 1:55 PM, Paul Kosinski 
mailto:clamav-us...@iment.com>> wrote:

We are using a local mirror to reduce Internet traffic and (mainly) to
reduce load on the ClamAV servers.

It is *only* the "master" (Internet-connected) ClamAV that sees these
delays, where the DNS TXT record advertises updates before whatever
Cloudflare server we (are unlucky enough to) actually hit has the files
available for download. The other ClamAVs on our LAN use an approach
wherein the mirror server (on the "master" ClamAV machine) simply
reports when new cvd (etc.) files are available locally.

It is only the "master" ClamAV that periodically does a DNS TXT query to
decide whether to run freshclam, whose source is the ClamAV (Clouflare)
server. If it does this immediately when the DNS TXT records suggests,
freshclam sometimes fails, complaining about things being out of sync.
(That's when I added the curl prefetch to see if the file really was
what the DNS TXT said.) All this has nothing to do with our local
mirroring,

Somebody suggested that our ISP (Comcast) may be proxying / caching the
ClamAV files -- and doing it badly. If that's the case, I don't know
what we can do about it.


On Tue, 20 Nov 2018 13:09:54 +
"Joel Esler (jesler)" mailto:jes...@cisco.com>> wrote:

Any particular reason that you are using a local mirror?  I mean, if
not strictly necessary, just point it at our mirrors and call it a
day.

I've talked to a couple people off list in the last few days that
were experiencing errors or delays, and 100% of them were using local
proxies or mirrors.

One was because the proxy didn't know how to address
"HTTP/1.0" (Instead of "HTTP/1.1").

So it could be the smallest of issues, eliminate any issues that are
not strictly relevant.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com

On Nov 19, 2018, at 9:25 PM, Paul Kosinski 
wrote:

Our Internet-facing ClamAV sits on our gateway/firewall and serves
as our local mirror. It accesses the Internet via the NIC whose IP
address is 10.11.14.160. (We used to have two NICs connected to the
Internet, but now only have one, so this is historical only.)

The msg "Using ip '10.11.14.160' for fetching" is produced by the
freshclam binary itself and derives from our freshclam.conf entry:

# Use aaa.bbb.ccc.ddd as client address for downloading databases.
# Useful for multi-homed systems.
# Default: Use OS'es default outgoing IP address.
LocalIPAddress 10.11.14.160

No matter, are we so unlucky -- only 1 out of 3M -- in having the
sync errors reappear? Or are we simply one of far fewer users who
log (and actually examine) their entire freshclam output?

P.S. I have very recently updated our clamavs to 0.100.2. I wonder
if that will improve things in this regard.



On Thu, 15 Nov 2018 19:40:43 +
"Joel Esler (jesler)"  wrote:

Judging by the 60+TB of traffic we are transferring a day, it's
working for at least 3M+ users.

On Nov 15, 2018, at 1:34 PM, Dennis Peterson 
wrote:

On 11/13/18 12:04 PM, Paul Kosinski wrote:
"Why are you looking at October reports?"

It was the first one. And it also shows that the problem began
*before* 0.100.1 was deemed OUTDATED.

So, here's one from this morning.

I also have 4 from yesterday, 3 from Sunday Nov 11 etc. Posting
them all would be a bit tedious.

What does this line mean - that is, what is fetching from that IP?
Local mirror?

Using ip '10.11.14.160' for fetching.

And we're having a completely different experience here with
reliability over the same time span:

Mirror #1
IP: 104.16.189.138
Successes: 19
Failures: 0
Last access: Thu Nov 15 07:01:02 2018
Ignore: No
-
Mirror #2
IP: 104.16.186.138
Successes: 19
Failures: 0
Last access: Wed Nov 14 23:01:03 2018
Ignore: No
-
Mirror #3
IP: 104.16.185.138
Successes: 18
Failures: 0
Last access: Mon Nov 12 21:05:32 2018
Ignore: No
-
Mirror #4
IP: 104.16.187.138
Successes: 18
Failures: 0
Last access: Sun Nov 11 01:07:46 2018
Ignore: No
-
Mirror #5
IP: 104.16.188.138
Successes: 19
Failures: 0
Last access: Mon Nov 12 14:03:05 2018
Ignore: No


___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Paul Kosinski]

We are using a local mirror to reduce Internet traffic and (mainly) to
reduce load on the ClamAV servers.

It is *only* the "master" (Internet-connected) ClamAV that sees these
delays, where the DNS TXT record advertises updates before whatever
Cloudflare server we (are unlucky enough to) actually hit has the files
available for download. The other ClamAVs on our LAN use an approach
wherein the mirror server (on the "master" ClamAV machine) simply
reports when new cvd (etc.) files are available locally.

It is only the "master" ClamAV that periodically does a DNS TXT query to
decide whether to run freshclam, whose source is the ClamAV (Clouflare)
server. If it does this immediately when the DNS TXT records suggests,
freshclam sometimes fails, complaining about things being out of sync.
(That's when I added the curl prefetch to see if the file really was
what the DNS TXT said.) All this has nothing to do with our local
mirroring,

Somebody suggested that our ISP (Comcast) may be proxying / caching the
ClamAV files -- and doing it badly. If that's the case, I don't know
what we can do about it.


On Tue, 20 Nov 2018 13:09:54 +
"Joel Esler (jesler)"  wrote:

> Any particular reason that you are using a local mirror?  I mean, if
> not strictly necessary, just point it at our mirrors and call it a
> day.
> 
> I've talked to a couple people off list in the last few days that
> were experiencing errors or delays, and 100% of them were using local
> proxies or mirrors.
> 
> One was because the proxy didn't know how to address
> "HTTP/1.0" (Instead of "HTTP/1.1").
> 
> So it could be the smallest of issues, eliminate any issues that are
> not strictly relevant.
> 
> --
> Joel Esler
> Manager, Communities Division
> Cisco Talos Intelligence Group
> http://www.talosintelligence.com
> 
> > On Nov 19, 2018, at 9:25 PM, Paul Kosinski 
> > wrote:
> > 
> > Our Internet-facing ClamAV sits on our gateway/firewall and serves
> > as our local mirror. It accesses the Internet via the NIC whose IP
> > address is 10.11.14.160. (We used to have two NICs connected to the
> > Internet, but now only have one, so this is historical only.) 
> > 
> > The msg "Using ip '10.11.14.160' for fetching" is produced by the
> > freshclam binary itself and derives from our freshclam.conf entry:
> > 
> >  # Use aaa.bbb.ccc.ddd as client address for downloading databases.
> >  # Useful for multi-homed systems. 
> >  # Default: Use OS'es default outgoing IP address. 
> >  LocalIPAddress 10.11.14.160
> > 
> > No matter, are we so unlucky -- only 1 out of 3M -- in having the
> > sync errors reappear? Or are we simply one of far fewer users who
> > log (and actually examine) their entire freshclam output?
> > 
> > P.S. I have very recently updated our clamavs to 0.100.2. I wonder
> > if that will improve things in this regard.
> > 
> > 
> > 
> > On Thu, 15 Nov 2018 19:40:43 +
> > "Joel Esler (jesler)"  wrote:
> > 
> >> Judging by the 60+TB of traffic we are transferring a day, it's
> >> working for at least 3M+ users.
> >> 
> >>> On Nov 15, 2018, at 1:34 PM, Dennis Peterson 
> >>> wrote:
> >>> 
> >>> On 11/13/18 12:04 PM, Paul Kosinski wrote:
>  "Why are you looking at October reports?"
>  
>  It was the first one. And it also shows that the problem began
>  *before* 0.100.1 was deemed OUTDATED.
>  
>  So, here's one from this morning.
>  
>  I also have 4 from yesterday, 3 from Sunday Nov 11 etc. Posting
>  them all would be a bit tedious.
> >>> 
> >>> What does this line mean - that is, what is fetching from that IP?
> >>> Local mirror?
> >>> 
> >>> Using ip '10.11.14.160' for fetching.
> >>> 
> >>> And we're having a completely different experience here with
> >>> reliability over the same time span:
> >>> 
> >>> Mirror #1
> >>> IP: 104.16.189.138
> >>> Successes: 19
> >>> Failures: 0
> >>> Last access: Thu Nov 15 07:01:02 2018
> >>> Ignore: No
> >>> -
> >>> Mirror #2
> >>> IP: 104.16.186.138
> >>> Successes: 19
> >>> Failures: 0
> >>> Last access: Wed Nov 14 23:01:03 2018
> >>> Ignore: No
> >>> -
> >>> Mirror #3
> >>> IP: 104.16.185.138
> >>> Successes: 18
> >>> Failures: 0
> >>> Last access: Mon Nov 12 21:05:32 2018
> >>> Ignore: No
> >>> -
> >>> Mirror #4
> >>> IP: 104.16.187.138
> >>> Successes: 18
> >>> Failures: 0
> >>> Last access: Sun Nov 11 01:07:46 2018
> >>> Ignore: No
> >>> -
> >>> Mirror #5
> >>> IP: 104.16.188.138
> >>> Successes: 19
> >>> Failures: 0
> >>> Last access: Mon Nov 12 14:03:05 2018
> >>> Ignore: No

> 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Paul Kosinski]

I think you misunderstand. The 'LocalIPAddress' is the *outgoing* IP
address: i.e., the address assigned to the NIC. (This used to be
important when we had two Internet connections.)

The 'LocalIPAddress' has nothing to do with the IP address that
freshclam tries to get the cvds etc. *from*. (Otherwise, the OS would
be expected to have a default IP address as the source of ClamAV data?)

The *only* freshclam that is experiencing delays is the one pulling its
updates directly from the ClamAV (Cloudflare) servers. The other
machines on our LAN don't use the DNS TXT mechanism, but rather are
signaled by the "master" (Internet-connected) ClamAV that new cvd files
are now available locally. Then they run freshclam using a 10.x.x.x
mirror address.

To sum up, the "LocalIPAddress 10.11.14.160" is exactly what it should
be, the address assigned to a NIC. It is *not* where freshclam tries to
get updates from.


On Tue, 20 Nov 2018 03:39:12 +0100
Benny Pedersen  wrote:

> Paul Kosinski skrev den 2018-11-20 03:25:
> 
> >   # Use aaa.bbb.ccc.ddd as client address for downloading databases.
> >   # Useful for multi-homed systems.
> >   # Default: Use OS'es default outgoing IP address.
> >   LocalIPAddress 10.11.14.160
> 
> comment that line
> 
> > No matter, are we so unlucky -- only 1 out of 3M -- in having the
> > sync errors reappear? Or are we simply one of far fewer users who
> > log (and actually examine) their entire freshclam output?
> 
> you did not make localmirror correct, whole 10.x.x.x/8 is non routed
> in rfc1918, so not external problem to solve other then make
> freshclam get a localmirror that works, if unsure how, comment that
> localipaddress

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Joel Esler (jesler)]

Any particular reason that you are using a local mirror?  I mean, if not 
strictly necessary, just point it at our mirrors and call it a day.

I've talked to a couple people off list in the last few days that were 
experiencing errors or delays, and 100% of them were using local proxies or 
mirrors.

One was because the proxy didn't know how to address "HTTP/1.0" (Instead of 
"HTTP/1.1").

So it could be the smallest of issues, eliminate any issues that are not 
strictly relevant.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com

> On Nov 19, 2018, at 9:25 PM, Paul Kosinski  wrote:
> 
> Our Internet-facing ClamAV sits on our gateway/firewall and serves as
> our local mirror. It accesses the Internet via the NIC whose IP address
> is 10.11.14.160. (We used to have two NICs connected to the Internet,
> but now only have one, so this is historical only.) 
> 
> The msg "Using ip '10.11.14.160' for fetching" is produced by the
> freshclam binary itself and derives from our freshclam.conf entry:
> 
>  # Use aaa.bbb.ccc.ddd as client address for downloading databases.
>  # Useful for multi-homed systems. 
>  # Default: Use OS'es default outgoing IP address. 
>  LocalIPAddress 10.11.14.160
> 
> No matter, are we so unlucky -- only 1 out of 3M -- in having the sync
> errors reappear? Or are we simply one of far fewer users who log (and
> actually examine) their entire freshclam output?
> 
> P.S. I have very recently updated our clamavs to 0.100.2. I wonder if
> that will improve things in this regard.
> 
> 
> 
> On Thu, 15 Nov 2018 19:40:43 +
> "Joel Esler (jesler)"  wrote:
> 
>> Judging by the 60+TB of traffic we are transferring a day, it's
>> working for at least 3M+ users.
>> 
>>> On Nov 15, 2018, at 1:34 PM, Dennis Peterson 
>>> wrote:
>>> 
>>> On 11/13/18 12:04 PM, Paul Kosinski wrote:
 "Why are you looking at October reports?"
 
 It was the first one. And it also shows that the problem began
 *before* 0.100.1 was deemed OUTDATED.
 
 So, here's one from this morning.
 
 I also have 4 from yesterday, 3 from Sunday Nov 11 etc. Posting
 them all would be a bit tedious.
>>> 
>>> What does this line mean - that is, what is fetching from that IP?
>>> Local mirror?
>>> 
>>> Using ip '10.11.14.160' for fetching.
>>> 
>>> And we're having a completely different experience here with
>>> reliability over the same time span:
>>> 
>>> Mirror #1
>>> IP: 104.16.189.138
>>> Successes: 19
>>> Failures: 0
>>> Last access: Thu Nov 15 07:01:02 2018
>>> Ignore: No
>>> -
>>> Mirror #2
>>> IP: 104.16.186.138
>>> Successes: 19
>>> Failures: 0
>>> Last access: Wed Nov 14 23:01:03 2018
>>> Ignore: No
>>> -
>>> Mirror #3
>>> IP: 104.16.185.138
>>> Successes: 18
>>> Failures: 0
>>> Last access: Mon Nov 12 21:05:32 2018
>>> Ignore: No
>>> -
>>> Mirror #4
>>> IP: 104.16.187.138
>>> Successes: 18
>>> Failures: 0
>>> Last access: Sun Nov 11 01:07:46 2018
>>> Ignore: No
>>> -
>>> Mirror #5
>>> IP: 104.16.188.138
>>> Successes: 19
>>> Failures: 0
>>> Last access: Mon Nov 12 14:03:05 2018
>>> Ignore: No
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Benny Pedersen]

Paul Kosinski skrev den 2018-11-20 03:25:


  # Use aaa.bbb.ccc.ddd as client address for downloading databases.
  # Useful for multi-homed systems.
  # Default: Use OS'es default outgoing IP address.
  LocalIPAddress 10.11.14.160


comment that line


No matter, are we so unlucky -- only 1 out of 3M -- in having the sync
errors reappear? Or are we simply one of far fewer users who log (and
actually examine) their entire freshclam output?


you did not make localmirror correct, whole 10.x.x.x/8 is non routed in 
rfc1918, so not external problem to solve other then make freshclam get 
a localmirror that works, if unsure how, comment that localipaddress

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Paul Kosinski]

Our Internet-facing ClamAV sits on our gateway/firewall and serves as
our local mirror. It accesses the Internet via the NIC whose IP address
is 10.11.14.160. (We used to have two NICs connected to the Internet,
but now only have one, so this is historical only.) 

The msg "Using ip '10.11.14.160' for fetching" is produced by the
freshclam binary itself and derives from our freshclam.conf entry:

  # Use aaa.bbb.ccc.ddd as client address for downloading databases.
  # Useful for multi-homed systems. 
  # Default: Use OS'es default outgoing IP address. 
  LocalIPAddress 10.11.14.160

No matter, are we so unlucky -- only 1 out of 3M -- in having the sync
errors reappear? Or are we simply one of far fewer users who log (and
actually examine) their entire freshclam output?

P.S. I have very recently updated our clamavs to 0.100.2. I wonder if
that will improve things in this regard.



On Thu, 15 Nov 2018 19:40:43 +
"Joel Esler (jesler)"  wrote:

> Judging by the 60+TB of traffic we are transferring a day, it's
> working for at least 3M+ users.
> 
> > On Nov 15, 2018, at 1:34 PM, Dennis Peterson 
> > wrote:
> > 
> > On 11/13/18 12:04 PM, Paul Kosinski wrote:
> >> "Why are you looking at October reports?"
> >> 
> >> It was the first one. And it also shows that the problem began
> >> *before* 0.100.1 was deemed OUTDATED.
> >> 
> >> So, here's one from this morning.
> >> 
> >> I also have 4 from yesterday, 3 from Sunday Nov 11 etc. Posting
> >> them all would be a bit tedious.
> > 
> > What does this line mean - that is, what is fetching from that IP?
> > Local mirror?
> > 
> > Using ip '10.11.14.160' for fetching.
> > 
> > And we're having a completely different experience here with
> > reliability over the same time span:
> > 
> > Mirror #1
> > IP: 104.16.189.138
> > Successes: 19
> > Failures: 0
> > Last access: Thu Nov 15 07:01:02 2018
> > Ignore: No
> > -
> > Mirror #2
> > IP: 104.16.186.138
> > Successes: 19
> > Failures: 0
> > Last access: Wed Nov 14 23:01:03 2018
> > Ignore: No
> > -
> > Mirror #3
> > IP: 104.16.185.138
> > Successes: 18
> > Failures: 0
> > Last access: Mon Nov 12 21:05:32 2018
> > Ignore: No
> > -
> > Mirror #4
> > IP: 104.16.187.138
> > Successes: 18
> > Failures: 0
> > Last access: Sun Nov 11 01:07:46 2018
> > Ignore: No
> > -
> > Mirror #5
> > IP: 104.16.188.138
> > Successes: 19
> > Failures: 0
> > Last access: Mon Nov 12 14:03:05 2018
> > Ignore: No
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Joel Esler (jesler)]

Judging by the 60+TB of traffic we are transferring a day, it's working for at 
least 3M+ users.

> On Nov 15, 2018, at 1:34 PM, Dennis Peterson  wrote:
> 
> On 11/13/18 12:04 PM, Paul Kosinski wrote:
>> "Why are you looking at October reports?"
>> 
>> It was the first one. And it also shows that the problem began *before*
>> 0.100.1 was deemed OUTDATED.
>> 
>> So, here's one from this morning.
>> 
>> I also have 4 from yesterday, 3 from Sunday Nov 11 etc. Posting them
>> all would be a bit tedious.
> 
> What does this line mean - that is, what is fetching from that IP? Local 
> mirror?
> 
> Using ip '10.11.14.160' for fetching.
> 
> And we're having a completely different experience here with reliability over 
> the same time span:
> 
> Mirror #1
> IP: 104.16.189.138
> Successes: 19
> Failures: 0
> Last access: Thu Nov 15 07:01:02 2018
> Ignore: No
> -
> Mirror #2
> IP: 104.16.186.138
> Successes: 19
> Failures: 0
> Last access: Wed Nov 14 23:01:03 2018
> Ignore: No
> -
> Mirror #3
> IP: 104.16.185.138
> Successes: 18
> Failures: 0
> Last access: Mon Nov 12 21:05:32 2018
> Ignore: No
> -
> Mirror #4
> IP: 104.16.187.138
> Successes: 18
> Failures: 0
> Last access: Sun Nov 11 01:07:46 2018
> Ignore: No
> -
> Mirror #5
> IP: 104.16.188.138
> Successes: 19
> Failures: 0
> Last access: Mon Nov 12 14:03:05 2018
> Ignore: No
> 
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Dennis Peterson]

On 11/13/18 12:04 PM, Paul Kosinski wrote:

"Why are you looking at October reports?"

It was the first one. And it also shows that the problem began *before*
0.100.1 was deemed OUTDATED.

So, here's one from this morning.

I also have 4 from yesterday, 3 from Sunday Nov 11 etc. Posting them
all would be a bit tedious.


What does this line mean - that is, what is fetching from that IP? Local mirror?

Using ip '10.11.14.160' for fetching.

And we're having a completely different experience here with reliability over 
the same time span:

Mirror #1
IP: 104.16.189.138
Successes: 19
Failures: 0
Last access: Thu Nov 15 07:01:02 2018
Ignore: No
-
Mirror #2
IP: 104.16.186.138
Successes: 19
Failures: 0
Last access: Wed Nov 14 23:01:03 2018
Ignore: No
-
Mirror #3
IP: 104.16.185.138
Successes: 18
Failures: 0
Last access: Mon Nov 12 21:05:32 2018
Ignore: No
-
Mirror #4
IP: 104.16.187.138
Successes: 18
Failures: 0
Last access: Sun Nov 11 01:07:46 2018
Ignore: No
-
Mirror #5
IP: 104.16.188.138
Successes: 19
Failures: 0
Last access: Mon Nov 12 14:03:05 2018
Ignore: No

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Paul Kosinski]

"Why are you looking at October reports?"

It was the first one. And it also shows that the problem began *before*
0.100.1 was deemed OUTDATED.

So, here's one from this morning.

I also have 4 from yesterday, 3 from Sunday Nov 11 etc. Posting them
all would be a bit tedious.

--  Tuesday 13 November 2018 at 10:33:01  
--

/opt/clamav/bin/testclam-external
-->  UPD  D 25117/25117/25116  B 327/327/327  M 58/58/58

/opt/clamav/bin/freshclam -v --stdout --on-update-execute=EXIT_1
Current working dir is /opt/clamav.d/clamav.0.100.1/share/clamav
Max retries == 1
ClamAV update process started at Tue Nov 13 10:33:02 2018
Using IPv6 aware code
Querying current.cvd.clamav.net
TTL: 1799
Software version from DNS: 0.100.2
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.100.1 Recommended version: 0.100.2
DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
main.cvd version from DNS: 58
main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: 
sigmgr)
daily.cvd version from DNS: 25117
Retrieving http://database.clamav.net/daily.cvd
Using ip '10.11.14.160' for fetching.
Trying to download http://database.clamav.net/daily.cvd (IP: 104.16.189.138)
Downloading daily.cvd [100%]
WARNING: Mirror 104.16.189.138 is not synchronized.
Querying daily.0.92.0.0.6810BD8A.ping.clamav.net
Giving up on database.clamav.net...
Update failed. Your network may be down or none of the mirrors listed in 
/opt/clamav.d/clamav.0.100.1/etc/freshclam.conf is working. Check 
https://www.clamav.net/documents/official-mirror-faq for possible reasons.

/opt/clamav/bin/freshclam --list-mirrors

Mirror #1
IP: 104.16.187.138
Successes: 79
Failures: 7
Last access: Mon Nov 12 19:03:04 2018
Ignore: No
-
Mirror #2
IP: 104.16.189.138
Successes: 87
Failures: 7
Last access: Tue Nov 13 10:33:07 2018
Ignore: Yes
-
Mirror #3
IP: 104.16.188.138
Successes: 86
Failures: 6
Last access: Tue Nov 13 02:03:06 2018
Ignore: No
-
Mirror #4
IP: 104.16.185.138
Successes: 88
Failures: 6
Last access: Mon Nov 12 18:03:07 2018
Ignore: Yes
-
Mirror #5
IP: 104.16.186.138
Successes: 79
Failures: 7
Last access: Sun Nov 11 09:33:04 2018
Ignore: No

--  Tuesday 13 November 2018 at 10:33:08  
--



On Tue, 13 Nov 2018 09:49:54 -0800
Dennis Peterson  wrote:

> On 11/12/18 6:28 PM, Paul Kosinski wrote:
> > As some of you may remember, I "solved" the problems of the
> > Cloudflare mirrors being out of sync by not relying on what version
> > the DNS TXT record reports, but double checking it by retrieving
> > the head of the CVD file via curl.
> >
> Why are you looking at October reports?
> 
> 
> dp
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV mirrors have gotten worse!

[Dennis Peterson]

On 11/12/18 6:28 PM, Paul Kosinski wrote:

As some of you may remember, I "solved" the problems of the Cloudflare
mirrors being out of sync by not relying on what version the DNS TXT
record reports, but double checking it by retrieving the head of the
CVD file via curl.


Why are you looking at October reports?


dp


___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[clamav-users] ClamAV mirrors have gotten worse!

[Paul Kosinski]

As some of you may remember, I "solved" the problems of the Cloudflare
mirrors being out of sync by not relying on what version the DNS TXT
record reports, but double checking it by retrieving the head of the
CVD file via curl.

Now that I have replaced our dead (hardware, 32-bit) Web and email
server with a totally new server (virtual, 64-bit), which involved
restoring most files from rsync backups, followed by *lots* of
reconfiguring for the new Centos 7 and Apache 2.4.6, plus fighting with
SeLinux (which makes configuring systemd seem really easy), I again
have some time to deal with ClamAV.

So, the curl workaround seems still to work OK. But ...

Since Oct 20, we are occasionally getting the old error msgs from
freshclam (although the next freshclam run usually works):

  Update failed. Your network may be down or none of the mirrors listed
  in /opt/clamav.d/clamav.0.100.1/etc/freshclam.conf is working. Check
  https://www.clamav.net/documents/official-mirror\-faq for possible
  reasons.

This is followed by a mirror report (see below).

I suspect this is due to curl retrieving the head of the cvd from one
mirror (IP or anycast physical) while freshclam downloads the full cvd
from a different mirror.

We are running ClamAV version 0.100.1. Freshclam started reporting that
0.100.1 was OUTDATED on Oct 3, but the new batch of sync errors didn't
start until Oct 20, so it can't be that simple. 

Will these synchronization problems never end? We have seen about 2
dozen of these failures in the past 3 weeks (since Oct 20), but none
before that -- once I had added the curl workaround.



Here is a typical detailed freshclam error report (which happens to be
the first):

--  Saturday 20 October 2018 at 09:46:01  
--

/opt/clamav/bin/testclam-external
-->  UPD  D 25054/25054/25053  B 327/327/327  M 58/58/58

/opt/clamav/bin/freshclam -v --stdout --on-update-execute=EXIT_1
Current working dir is /opt/clamav.d/clamav.0.100.1/share/clamav
Max retries == 1
ClamAV update process started at Sat Oct 20 09:46:02 2018
Using IPv6 aware code
Querying current.cvd.clamav.net
TTL: 1319
Software version from DNS: 0.100.2
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.100.1 Recommended version: 0.100.2
DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
main.cvd version from DNS: 58
main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: 
sigmgr)
daily.cvd version from DNS: 25054
Retrieving http://database.clamav.net/daily.cvd
Using ip '10.11.14.160' for fetching.
Trying to download http://database.clamav.net/daily.cvd (IP: 104.16.189.138)
Downloading daily.cvd [100%]
WARNING: Mirror 104.16.189.138 is not synchronized.
Querying daily.0.92.0.0.6810BD8A.ping.clamav.net
Giving up on database.clamav.net...
Update failed. Your network may be down or none of the mirrors listed in 
/opt/clamav.d/clamav.0.100.1/etc/freshclam.conf is working. Check 
https://www.clamav.net/documents/official-mirror-faq for possible reasons.

/opt/clamav/bin/freshclam --list-mirrors

Mirror #1
IP: 104.16.187.138
Successes: 52
Failures: 2
Last access: Fri Oct 19 10:33:07 2018
Ignore: No
-
Mirror #2
IP: 104.16.189.138
Successes: 70
Failures: 2
Last access: Sat Oct 20 09:46:06 2018
Ignore: Yes
-
Mirror #3
IP: 104.16.188.138
Successes: 70
Failures: 1
Last access: Sat Oct 20 02:52:05 2018
Ignore: No
-
Mirror #4
IP: 104.16.185.138
Successes: 70
Failures: 1
Last access: Fri Oct 19 02:49:07 2018
Ignore: No
-
Mirror #5
IP: 104.16.186.138
Successes: 70
Failures: 1
Last access: Fri Oct 19 17:18:05 2018
Ignore: No

--  Saturday 20 October 2018 at 09:46:06  
--
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml