Re: [clamav-users] Database not updating
On Tue, 13 Sep 2011 07:57:35 -0700 Al Varnell alvarn...@mac.com wrote: On 9/12/11 6:59 PM, Dan dantear...@gmail.com wrote: At 11:12 AM -0700 9/3/2011, Al Varnell wrote: Both current.cvd.clamav.net and your home page say the latest version of daily.cvd is 13538, but according to Twitter there have been seven updates in the last 24 hours. It's doin it again. I'm seein 13602 but on Twitter it says 13605 is newest -- FIVE HOURS AGO. What's with the gigantic lag? My morning update just now caught up to 13608, so I guess it's fixed. The lag was forty hours for me. I'd feel a lot better if there was some sort of explanation. There was a problem with our internal file distribution mechanism, which should be fixed now. Sorry for the inconvenience. -- oo. Tomasz Kojm tk...@clamav.net (\/)\. http://www.ClamAV.net/gpg/tkojm.gpg \..._ 0DCA5A08407D5288279DB43454822DC8985A444B //\ /\ Tue Sep 13 17:09:25 CEST 2011 ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
Yet more failure on 88.198.67.125, this morning. This one is a double. Shouldn't Freshclam be smart enough to avoid the same failing server at least within the same run? ClamAV update process started at Tue Sep 13 10:45:01 2011 main.cvd is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven) connect_error: getsockopt(SO_ERROR): fd=6 error=61: Connection refused Can't connect to port 80 of host database.clamav.net (IP: 88.198.67.125) Trying host database.clamav.net (65.19.179.67)... Downloading daily-13603.cdiff [100%] Downloading daily-13604.cdiff [100%] nonblock_recv: recv timing out (30 secs) connect_error: getsockopt(SO_ERROR): fd=6 error=61: Connection refused Can't connect to port 80 of host database.clamav.net (IP: 88.198.67.125) Trying host database.clamav.net (207.57.106.31)... Downloading daily-13605.cdiff [100%] Downloading daily-13606.cdiff [100%] Downloading daily-13607.cdiff [100%] Downloading daily-13608.cdiff [100%] daily.cld updated (version: 13608, sigs: 192488, f-level: 60, builder: neo) bytecode.cld is up to date (version: 144, sigs: 41, f-level: 60, builder: edwin) Database updated (1038743 signatures) from database.clamav.net (IP: 207.57.106.31) Clamd successfully notified about the update. - Dan. -- - Psychoceramic Emeritus; South Jersey, USA, Earth. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On Sep 13, 2011, at 8:15 AM, Dan dantear...@gmail.com wrote: Yet more failure on 88.198.67.125, this morning. This one is a double. I was going to wait a few more days to mention this, but since you bring it up... I have seen this twice a day almost every day since 29 Aug. The only times I didn't see this was when the database was reported to be up-to-date. During that same period, I was _never_ able to successfully connect to it. This can't be just my bad luck. Also, why was this mirror the first one checked from 2-10 Sep? I thought there was supposed to be more randomness in the list. This morning was the first time a different server appeared first this month. Sent from Janet's iPad -Al- -- Al Varnell ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
Noone has suggested maximum. The issue is that the mirrors are so overloaded that it's often taking freshclam an excessive amount of time to do its thing, because of the time-outs / connection failures. No big deal if it's the update run in the background. But if it's on-demand update preceding a user-driven scan, it's making the user sit there, twiddling its thumbs, for up to a minute or two. Are we really having this protracted discussion, because we don't want someone to have to sit for up to a minute or two? This problem seems overstated. I mean, are we talking about on-demand scans perhaps a dozen or more times per day, every day? i.e. is this adding up to hours of lost time every week? If so, is it really such a problem to have a database that is *at most* 2 hours out-of-date (the default)? Do you need to do an update before *every* on-demand scan? And why can't that be solved (if it is, in fact, an issue) by increasing the check frequency to, say, every hour? I'm not trying to stifle the idea of distributing the databases via torrent, but some of this discussion seems to be trying to solve a fabricated issue. As for the torrent, I think we can stop the discussion given the following: 1. The ClamAV team has said they will not support torrents. 2. The question about the local directory has been addressed. 3. Torrents can be easily created by anyone. Is there really anything more to discuss, except perhaps some more details of the local directory answer? -- Bryan Burke IT Administrator Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville bbu...@eecs.utk.edu (865) 974-4694 ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
Am 13.09.2011 18:01, schrieb Al Varnell: On Sep 13, 2011, at 8:15 AM, Dan dantear...@gmail.com wrote: Yet more failure on 88.198.67.125, this morning. This one is a double. I was going to wait a few more days to mention this, but since you bring it up... I have seen this twice a day almost every day since 29 Aug. The only times I didn't see this was when the database was reported to be up-to-date. During that same period, I was _never_ able to successfully connect to it. This can't be just my bad luck. just your bad luck 2011/09/05 - 297638 connects 2011/09/06 - 265677 connects 2011/09/07 - 265228 connects 2011/09/08 - 210367 connects 2011/09/09 - 230462 connects 2011/09/10 - 142702 connects 2011/09/11 - 120486 connects 2011/09/12 - 207272 connects 2011/09/13 - 129521 connetcs until now - 1916 CET as mentioned a few days befor, YOU have a very slow connection to my system. just use another mirror instead of crying all the time about your bad setup. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/2011 1:18 PM, sys...@ra-schaal.de wrote: Am 13.09.2011 18:01, schrieb Al Varnell: On Sep 13, 2011, at 8:15 AM, Dan dantear...@gmail.com wrote: Yet more failure on 88.198.67.125, this morning. This one is a double. I was going to wait a few more days to mention this, but since you bring it up... I have seen this twice a day almost every day since 29 Aug. The only times I didn't see this was when the database was reported to be up-to-date. During that same period, I was _never_ able to successfully connect to it. This can't be just my bad luck. just your bad luck 2011/09/05 - 297638 connects 2011/09/06 - 265677 connects 2011/09/07 - 265228 connects 2011/09/08 - 210367 connects 2011/09/09 - 230462 connects 2011/09/10 - 142702 connects 2011/09/11 - 120486 connects 2011/09/12 - 207272 connects 2011/09/13 - 129521 connetcs until now - 1916 CET as mentioned a few days befor, YOU have a very slow connection to my system. Not just him. I don't hit your mirror every time, but the last time I was able to successfully update from it was Aug 28, which matches what Al reported. Since then, I have seen 23 errors: Can't connect to port 80 of host db.us.clamav.net (IP: 88.198.67.125) Trying it manually today, I can ping the server, but cannot connect to port 80. Seems like something changed on Aug 28 or 29 which is causing connection problems for some people. -- Bowie ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 9:36 AM, Bryan Burke bbu...@eecs.utk.edu wrote: Noone has suggested maximum. The issue is that the mirrors are so overloaded that it's often taking freshclam an excessive amount of time to do its thing, because of the time-outs / connection failures. No big deal if it's the update run in the background. But if it's on-demand update preceding a user-driven scan, it's making the user sit there, twiddling its thumbs, for up to a minute or two. Are we really having this protracted discussion, because we don't want someone to have to sit for up to a minute or two? That was the original intent, but we seem to have hit a couple of other nerves. This problem seems overstated. I mean, are we talking about on-demand scans perhaps a dozen or more times per day, every day? i.e. is this adding up to hours of lost time every week? If so, is it really such a problem to have a database that is *at most* 2 hours out-of-date (the default)? Do you need to do an update before *every* on-demand scan? I don't know the frequency, but it was enough of a problem for him to complain...three times before I brought it up here. And why can't that be solved (if it is, in fact, an issue) by increasing the check frequency to, say, every hour? That's not a user option with ClamXav, although I realize it could be done by hacking the LaunchAgent (formerly cron) event. I will probably recommend to Mark that he include multiple updates as a user preference one of these days, but there are a couple of other features I'd like to see first. ... Is there really anything more to discuss, except perhaps some more details of the local directory answer? As I mentioned earlier today, I believe the issue with this particular mirror is bigger than what has been stated. I understand the need to limit access but why do we have a mirror: - Supporting users half way around the world - Which always seems to be the first one checked - And has never successfully connected for over two weeks If it was just one of these I could accept it, but there has to be something else going on with it. My guess is that if the network was working as designed the user would never had lodged his initial complaint. -Al- -- Al Varnell Mountain View, CA ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 10:18 AM, sys...@ra-schaal.de sys...@ra-schaal.de wrote: Am 13.09.2011 18:01, schrieb Al Varnell: On Sep 13, 2011, at 8:15 AM, Dan dantear...@gmail.com wrote: Yet more failure on 88.198.67.125, this morning. This one is a double. I was going to wait a few more days to mention this, but since you bring it up... I have seen this twice a day almost every day since 29 Aug. The only times I didn't see this was when the database was reported to be up-to-date. During that same period, I was _never_ able to successfully connect to it. This can't be just my bad luck. just your bad luck 2011/09/05 - 297638 connects 2011/09/06 - 265677 connects 2011/09/07 - 265228 connects 2011/09/08 - 210367 connects 2011/09/09 - 230462 connects 2011/09/10 - 142702 connects 2011/09/11 - 120486 connects 2011/09/12 - 207272 connects 2011/09/13 - 129521 connetcs until now - 1916 CET as mentioned a few days befor, YOU have a very slow connection to my system. I'm half a world away from you, so I'm not really surprised by that, but what difference should it make? just use another mirror instead of crying all the time about your bad setup. What are you talking about? I have no choice whatsoever on the mirror I connect to! -Al- -- Al Varnell Mountain View, CA ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
I don't know the frequency, but it was enough of a problem for him to complain...three times before I brought it up here. So is this issue specifically with ClamXav? i.e. is ClamXav forcing an update each time it's run? I know that the regular clamav does not do this, and if that's the product in question, my point still seems valid: aren't we crying over spilled milk here? I mean, it would seem that the user's desired case/functionality is unreasonable, and as a result, that asking the ClamAV team to do anything about it is also unreasonable. If the issue is, however, with ClamXav, then this isn't the correct mailing list to be having this discussion, correct? That's not a user option with ClamXav, although I realize it could be done by hacking the LaunchAgent (formerly cron) event. I will probably recommend to Mark that he include multiple updates as a user preference one of these days, but there are a couple of other features I'd like to see first. Fair enough. - Supporting users half way around the world Don't see a problem with this. - Which always seems to be the first one checked Actual issue. Perhaps DNS caching is a factor? If freshclam checks often enough, then perhaps the cache entry never dies, and you get the same order every time? - And has never successfully connected for over two weeks Other than an announcement to the list that there may be problems with one of the mirrors, this seems to be an issue primarily between those users who encountered said error (and caused them distress) and the mirror admins, not the whole list. However, maybe I'm wrong and many readers of the list appreciate seeing the back-and-forth. P.S. - My goal is to try to limit the scope of this thread a little more, so it stays focused and relevant. As a side-line user on this list, I feel it had long since gotten out-of-hand. -- Bryan Burke IT Administrator Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville bbu...@eecs.utk.edu (865) 974-4694 ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
Hi-- On Sep 13, 2011, at 12:49 PM, Bryan Burke wrote: - Which always seems to be the first one checked Actual issue. Perhaps DNS caching is a factor? If freshclam checks often enough, then perhaps the cache entry never dies, and you get the same order every time? Running dig db.us.clamav.net a few times shows that the nameserver responses are rotating the resource records; and even if it didn't, well-behaved resolver clients ought to rotate through multiple valid IPs returned by gethostbyname()/getaddrinfo() for a hostname anyway. - And has never successfully connected for over two weeks Other than an announcement to the list that there may be problems with one of the mirrors, this seems to be an issue primarily between those users who encountered said error (and caused them distress) and the mirror admins, not the whole list. However, maybe I'm wrong and many readers of the list appreciate seeing the back-and-forth. P.S. - My goal is to try to limit the scope of this thread a little more, so it stays focused and relevant. As a side-line user on this list, I feel it had long since gotten out-of-hand. I admire your goal of focussing on the problem, which I why I'll reply to this rather than other emails. :-) This being said, there is definitely a recurring issue with this particular mirror. Since Aug 22, I've seen: % grep Can't connect to port 80 of host database.clamav.net (IP: 88.198.67.125) /var/log/freshclam.log | wc -l 27 ...with zero successful connections to that IP. The connectivity failure is entirely reproducible by hand: % telnet 88.198.67.125 80 Trying 88.198.67.125... telnet: connect to address 88.198.67.125: Connection refused telnet: Unable to connect to remote host I don't consider this to be a significant problem since other mirrors are up, but it's not a matter of bandwidth or connectivity on my side. As it happens, I'm testing from Cupertino, CA via Apple's 17.0.0.0/8 network, and from a Time-Warner cable link from NYC, NY on 24.103.0.0/16. However, as a workaround it should be possible for folks to manually set DatabaseMirror in freshclam.conf to specific IPs from db.us.clamav.net, or perhaps switch to using db.ca.clamav.net, db.mx.clamav.net, or similar. Regards, -- -Chuck ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On Sep 13, 2011, at 2:28 PM, Bryan Burke wrote: ...with zero successful connections to that IP. The connectivity failure is entirely reproducible by hand: % telnet 88.198.67.125 80 Trying 88.198.67.125... telnet: connect to address 88.198.67.125: Connection refused telnet: Unable to connect to remote host I should say that when I did this, I got the same, but the connection seemed to be timing out, not being refused (despite what telnet says). Was it the same for you? No, I get an immediate connection refused and an ICMP port unreachable back: # tcpdump -nq host 88.198.67.125 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes 14:32:31.222347 IP 17.209.4.71.55899 88.198.67.125.80: tcp 0 14:32:31.397480 IP 88.198.67.125 17.209.4.71: ICMP 88.198.67.125 tcp port 80 unreachable, length 72 ^C 2 packets captured I ask because that would indicate either that the web server on that IP is down, or that some firewall is silently dropping packets. The webserver appears down from here; while a firewall could be configured to return ICMP_UNREACH_PORT, normally they just drop the traffic and you get connection timeouts as you've described... Regards, -- -Chuck ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/2011 12:47 AM, Henrik K wrote: If you are an individual not able to put $15-$100 a month, then yes, it's not in your capability. $15 - $100 extra / month would go to higher priority tasks / needs. Some of our servers are nearly old enough to vote. :-) As an individual, or small company, it just isn't within our current capabilities. When it is, we plan to get involved there. For now, we do what we can with what we have. No one thinks any less of you for trying to help, on the contrary. But if you can't even get any facts straight etc, it's just messing up the thread. Let's not forget that ClamAV is backed by a commercial organization?? If they wanted US bandwidth badly, they can get it. If not by buying, then probably just by asking around or even on the web page? Why do you think it's not mentioned there. Probably very few users read this list. Very good point. They could get it if they really needed it. Asking the user base for it is kind of sad. -- Sincerely, Nathan Gibbs Systems Administrator Christ Media http://www.cmpublishers.com signature.asc Description: OpenPGP digital signature ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
No, I get an immediate connection refused and an ICMP port unreachable back: # tcpdump -nq host 88.198.67.125 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes 14:32:31.222347 IP 17.209.4.71.55899 88.198.67.125.80: tcp 0 14:32:31.397480 IP 88.198.67.125 17.209.4.71: ICMP 88.198.67.125 tcp port 80 unreachable, length 72 My fault; just different telnet behaviors: I was using BSD telnet, which apparently kept trying to connect. When I used linux telnet, it ends immediately. So no discrepancy there. And I momentarily forgot the behavior of so-called closed ports (not blocked by firewall, but nothing running on them... thought the packets were dropped). So assuming a common firewall setup, it would appear the webserver is down. For potential aid in comparing notes and diagnosing the problem, I'm attaching some network information (whois and traceroute). If no firewall rule at the remote site explains this, then I can only surmise that some hop along the way is blocking the connections. If, however, this is due to some rate-limiting rule at the end point, is that acceptable? I don't know if ClamAV has a policy they ask their mirror hosts to adhere to, but if so, would this constitute grounds for removal from the pool? If not, then at this point, I'm guessing there's enough data here for the team to make a decision one way or the other concerning this host. Even if removed, it can always be re-added when the cause of this issue is tracked down and fixed. At least concerning this issue, is there anything more to be done? -- Bryan Burke IT Administrator Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville bbu...@eecs.utk.edu (865) 974-4694 WHOIS: The University of Tennessee Health Science Center UTK-NET (NET-160-36-0-0-1) 160.36.0.0 - 160.36.255.255 Various Registries (Maintained by ARIN) NET160 (NET-160-0-0-0-0) 160.0.0.0 - 160.255.255.255 traceroute: 1 chm01v150.ns.utk.edu (160.36.56.1) 0.383 ms 0.430 ms 0.371 ms 2 10.8.2.30 (10.8.2.30) 0.605 ms 0.547 ms 0.477 ms 3 bsm01v20.ns.utk.edu (160.36.128.133) 0.962 ms 0.967 ms 0.975 ms 4 bhm01ge3-3.ns.utk.edu (160.36.2.74) 0.671 ms 0.940 ms 0.869 ms 5 gi1-8.mpd01.atl04.atlas.cogentco.com (38.104.182.37) 6.564 ms 6.551 ms 6.580 ms 6 te0-1-0-1.mpd22.atl01.atlas.cogentco.com (154.54.3.169) 18.520 ms te0-1-0-1.ccr22.atl01.atlas.cogentco.com (154.54.6.121) 18.685 ms 18.603 ms 7 te0-4-0-7.mpd22.dca01.atlas.cogentco.com (154.54.27.93) 18.552 ms te0-1-0-2.ccr22.dca01.atlas.cogentco.com (154.54.28.230) 18.521 ms te0-2-0-3.mpd22.dca01.atlas.cogentco.com (154.54.2.102) 18.642 ms 8 te0-1-0-1.ccr22.iad02.atlas.cogentco.com (154.54.26.138) 19.529 ms te0-1-0-1.mpd22.iad02.atlas.cogentco.com (154.54.26.122) 19.656 ms te0-3-0-5.ccr22.iad02.atlas.cogentco.com (154.54.41.238) 19.922 ms 9 te1-8.ccr02.iad01.atlas.cogentco.com (154.54.31.174) 19.450 ms te2-7.ccr02.iad01.atlas.cogentco.com (154.54.31.214) 19.676 ms te1-2.ccr02.iad01.atlas.cogentco.com (154.54.31.194) 19.713 ms 10 kpn.iad01.atlas.cogentco.com (154.54.10.242) 19.364 ms 19.434 ms 19.377 ms 11 nyk-s2-rou-1021.US.eurorings.net (134.222.227.133) 26.53 ms 25.576 ms 25.506 ms 12 nntr-s1-rou-1022.FR.eurorings.net (134.222.226.162) 101.182 ms 103.179 ms 101.83 ms 13 ffm-s1-rou-1022.DE.eurorings.net (134.222.229.30) 117.550 ms 117.294 ms 117.393 ms 14 ffm-s1-rou-1021.DE.eurorings.net (134.222.228.85) 118.820 ms 116.595 ms 118.851 ms 15 nbg-s1-rou-1001.DE.eurorings.net (134.222.225.26) 119.864 ms 120.319 ms 120.34 ms 16 kpn-gw.hetzner.de (134.222.107.21) 121.689 ms 121.654 ms 121.642 ms 17 hos-bb2.juniper1.fs.hetzner.de (213.239.240.146) 122.426 ms hos-bb2.juniper2.rz14.hetzner.de (213.239.240.151) 123.412 ms 123.453 ms 18 hos-tr2.ex3k4.rz14.hetzner.de (213.239.224.165) 124.146 ms hos-tr1.ex3k4.rz14.hetzner.de (213.239.224.133) 128.706 ms 127.250 ms 19 mx00.akxnet.de (88.198.67.125) 122.800 ms 122.781 ms 122.707 ms traceroute -n: 1 160.36.56.1 0.456 ms 2.169 ms 2.226 ms 2 10.8.2.30 7.586 ms 0.622 ms 0.563 ms 3 160.36.128.133 0.541 ms 0.529 ms 0.566 ms 4 160.36.2.74 0.594 ms 0.580 ms 0.630 ms 5 38.104.182.37 6.674 ms 6.600 ms 6.551 ms 6 154.54.3.169 18.612 ms 154.54.6.121 18.850 ms 19.305 ms 7 154.54.3.66 18.513 ms 154.54.1.122 18.616 ms 154.54.27.97 18.489 ms 8 154.54.30.126 19.643 ms 154.54.30.118 19.548 ms 154.54.7.158 19.570 ms 9 154.54.31.214 19.513 ms 154.54.31.174 19.478 ms 154.54.31.234 19.504 ms 10 154.54.10.242 19.359 ms 19.324 ms 19.288 ms 11 134.222.227.133 42.719 ms 33.734 ms 32.88 ms 12 134.222.226.162 101.309 ms 101.216 ms 112.846 ms 13 134.222.231.145 118.146 ms 118.101 ms 118.99 ms 14 134.222.228.89 120.349 ms 118.313 ms 124.437 ms 15 134.222.225.26 119.494 ms 119.264 ms 119.573 ms 16
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 12:49 PM, Bryan Burke bbu...@eecs.utk.edu wrote: I don't know the frequency, but it was enough of a problem for him to complain...three times before I brought it up here. So is this issue specifically with ClamXav? i.e. is ClamXav forcing an update each time it's run? No, the option to check updates at launch defaults to off, but this particular user prefers to have the most recent updates available when running manual checks, so he has toggled the option on. I know that the regular clamav does not do this, and if that's the product in question, my point still seems valid: aren't we crying over spilled milk here? I mean, it would seem that the user's desired case/functionality is unreasonable, and as a result, that asking the ClamAV team to do anything about it is also unreasonable. If the issue is, however, with ClamXav, then this isn't the correct mailing list to be having this discussion, correct? Correct and it has been extensively discussed on the ClamXav Forum long before I brought it here. He has tried all the suggestions we made and still feels like he's wasting a log of time. If one user isn't enough to justify making any changes, fair enough, but I firmly believe we have a systemic problem that affects all US users here that needs to be resolved. That's not a user option with ClamXav, although I realize it could be done by hacking the LaunchAgent (formerly cron) event. I will probably recommend to Mark that he include multiple updates as a user preference one of these days, but there are a couple of other features I'd like to see first. Fair enough. - Supporting users half way around the world Don't see a problem with this. Not under normal circumstances, but from the Traceroutes I and others have done there does seem to be a significant delay in the Trans Atlantic segment. If that's what's causing the failure to connects, then maybe we need to take a look at the viability of where we go for off-shore mirrors. - Which always seems to be the first one checked Actual issue. Perhaps DNS caching is a factor? If freshclam checks often enough, then perhaps the cache entry never dies, and you get the same order every time? Interesting thought. - And has never successfully connected for over two weeks Other than an announcement to the list that there may be problems with one of the mirrors, this seems to be an issue primarily between those users who encountered said error (and caused them distress) and the mirror admins, not the whole list. However, maybe I'm wrong and many readers of the list appreciate seeing the back-and-forth. I'm more than willing to take this off-line if someone can give me a list of everybody that needs to be part of the discussion. P.S. - My goal is to try to limit the scope of this thread a little more, so it stays focused and relevant. As a side-line user on this list, I feel it had long since gotten out-of-hand. -Al- -- Al Varnell Mountain View, CA ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 2:28 PM, Bryan Burke bbu...@eecs.utk.edu wrote: % grep Can't connect to port 80 of host database.clamav.net (IP: 88.198.67.125) /var/log/freshclam.log | wc -l 27 Interesting. When I just grep for the IP in my logs: ib /var/log # grep 88.198.67.125 maillog* | wc -l 12 ren /var/log # grep 88.198.67.125 maillog* | wc -l 5 ba /var/log # grep 88.198.67.125 maillog* | wc -l 12 That represents 7 days worth of logs, across three servers. That averages to ~10/day. Note that my systems are configured for the default, which is 12 DB update checks per day. Since freshclam doesn't seem to log the IP (by default, at least) when the update succeeds (or there is no update), I have no good way of checking how many times 88.198.67.125 is queried. My logs show successful update sources in the last line, but not when there is no update. For instance, here is the one that just occurred: -- ClamAV update process started at Tue Sep 13 15:45:07 2011 main.cld is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven) connect_error: getsockopt(SO_ERROR): fd=4 error=61: Connection refused Can't connect to port 80 of host db.US.clamav.net (IP: 88.198.67.125) Downloading daily-13609.cdiff [100%] daily.cld updated (version: 13609, sigs: 192584, f-level: 60, builder: neo) bytecode.cld is up to date (version: 144, sigs: 41, f-level: 60, builder: edwin) Database updated (1038839 signatures) from db.US.clamav.net (IP: 194.8.197.22) -- -Al- -- Al Varnell Mountain View, CA ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/2011 7:07 PM, Al Varnell wrote: On 9/13/11 2:28 PM, Bryan Burke bbu...@eecs.utk.edu wrote: Since freshclam doesn't seem to log the IP (by default, at least) when the update succeeds (or there is no update), I have no good way of checking how many times 88.198.67.125 is queried. My logs show successful update sources in the last line, but not when there is no update. Which log messages need the IP? I'm testing the next CCEE patch set, so I could possibly slip those changes in before release. :-) -- Sincerely, Nathan Gibbs Systems Administrator Christ Media http://www.cmpublishers.com signature.asc Description: OpenPGP digital signature ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
My logs show successful update sources in the last line, but not when there is no update. Ok, well I did check the output of the grep before posting the number of lines on this list, and all log entries mentioning that IP were failures. So there's still *technically* some gray area, in that, if it happened to query that IP successfully, and there was no update, we'd never know, but I'm guessing that would reveal a similar outcome. Another side note: My ping latency times were about half of those posted earlier in the thread and I can't connect (about 122ms average). Either way, I really doubt the high latency of 250ms would cause any sort of issue. -- Bryan Burke IT Administrator Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville bbu...@eecs.utk.edu (865) 974-4694 ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 6:31 PM, Nathan Gibbs nat...@cmpublishers.com wrote: On 9/13/2011 7:07 PM, Al Varnell wrote: On 9/13/11 2:28 PM, Bryan Burke bbu...@eecs.utk.edu wrote: Since freshclam doesn't seem to log the IP (by default, at least) when the update succeeds (or there is no update), I have no good way of checking how many times 88.198.67.125 is queried. My logs show successful update sources in the last line, but not when there is no update. Which log messages need the IP? I was trying to say that using this command: freshclam --stdout --quiet --no-warnings --log=/usr/local/clamXav/share/clamav/freshclam.log I can determine the IP address of a successful update in the last line, e.g. Database updated (1038839 signatures) from db.US.clamav.net (IP: 194.8.197.22) If the database is already up-to-date then there is no attempt to access a mirror, so it would not be possible to provide an IP. But appreciate the offer. -Al- -- Al Varnell Mountain View, CA ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
Eliminate some unknowns - like maybe your DNS doesn't like big packets. Add this *temporarily* to your host table: 88.198.67.125 db.us.big.clamav.net And try again - and try with your browser, too. It should show you a web page indentifying the site you connected to and then after a short time you will be sent to clamav.net. Running host db.us.big.clamav.net multiple times seems to reveal 15 servers in the pool, and the order changes each time; as I mentioned earlier, in my case at least, the random pool idea is working, even if over a 7-day period, 1/7 attempts to update seemed to try the IP in question... just the nature of randomness, I suppose. Also, how would this reveal anything more than what telnet 88.198.67.125 80 getting a connection refused tells us? However, I did just discover something bizarre and interesting: telnet 88.198.67.125 80 Trying 88.198.67.125... telnet: connect to address 88.198.67.125: Connection refused host 88.198.67.125 125.67.198.88.in-addr.arpa domain name pointer mx00.akxnet.de. host mx00.akxnet.de mx00.akxnet.de has address 88.198.67.99 mx00.akxnet.de has IPv6 address 2a01:4f8:140:4301::2 telnet 88.198.67.99 80 Trying 88.198.67.99... Connected to 88.198.67.99. Escape character is '^]'. ^] telnet quit Connection closed. Is it possible this is caused by a master DNS issue? Of course, I tried to see the page and didn't get much, but I'm not all that familiar with HTTP: curl -H Host: db.us.clamav.net 88.198.67.99 !DOCTYPE HTML PUBLIC -//IETF//DTD HTML 2.0//EN htmlhead title403 Forbidden/title /headbody h1Forbidden/h1 pYou don't have permission to access / on this server./p hr addressApache/2.2.15 (Linux/SUSE) Server at db.us.clamav.net Port 80/address /body/html -- Bryan Burke IT Administrator Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville bbu...@eecs.utk.edu (865) 974-4694 ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/2011 9:03 PM, Bryan Burke wrote: My logs show successful update sources in the last line, but not when there is no update. Ok, well I did check the output of the grep before posting the number of lines on this list, and all log entries mentioning that IP were failures. So there's still *technically* some gray area, in that, if it happened to query that IP successfully, and there was no update, we'd never know, but I'm guessing that would reveal a similar outcome. There is no grey area. All connections are logged, both successful and unsuccessful. When DNS reports there is no update available, no connection is attempted and consequently there is no IP to log. From a well-connected host near Nashville TN USA: # tcping 88.198.67.125 80 88.198.67.125 port 80 closed. I get identical port 80 closed results from several hosts on various major USA ISPs. Logs going back a couple weeks show several failures each day and zero successful downloads from this host for us. While I certainly appreciate the donation of hardware and bandwidth by the owners of 88.198.67.125, a host that is consistently unavailable should be removed from the pool until it can be reliably accessed. -- Noel Jones ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 7:53 PM, Noel Jones wrote: On 9/13/2011 9:03 PM, Bryan Burke wrote: My logs show successful update sources in the last line, but not when there is no update. Ok, well I did check the output of the grep before posting the number of lines on this list, and all log entries mentioning that IP were failures. So there's still *technically* some gray area, in that, if it happened to query that IP successfully, and there was no update, we'd never know, but I'm guessing that would reveal a similar outcome. There is no grey area. All connections are logged, both successful and unsuccessful. When DNS reports there is no update available, no connection is attempted and consequently there is no IP to log. From a well-connected host near Nashville TN USA: # tcping 88.198.67.125 80 88.198.67.125 port 80 closed. I get identical port 80 closed results from several hosts on various major USA ISPs. I've just sent the URL to validator.wc3.org and got the same problem with this message: I got the following unexpected response when trying to retrieve http://88.198.67.125: 500 Can't connect to 88.198.67.125:80 (connect: Connection refused) I'm satisfied that site should be pulled from the list. If you have your own DNS server you can create your own round-robin authorative DNS server pointing to known to be reliable signature servers and which are located where ever they may be. It takes very little time to set one up. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 8:05 PM, Dennis Peterson wrote: I've just sent the URL to validator.wc3.org and got the same problem with this message: My fat fingers intended to type http://validator.wc.org and not what they did type. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 6:58 PM, Dennis Peterson denni...@inetnw.com wrote: On 9/13/11 3:15 PM, Bryan Burke wrote: At least concerning this issue, is there anything more to be done? Eliminate some unknowns - like maybe your DNS doesn't like big packets. Add this *temporarily* to your host table: 88.198.67.125 db.us.big.clamav.net And try again - and try with your browser, too. It should show you a web page indentifying the site you connected to and then after a short time you will be sent to clamav.net. Sounds like the server will be pulled, so you may not care, but since I went through the effort. Made changes to the hosts file. Ran dig $ db.us.clamav.net ; DiG 9.4.3-P3 db.us.clamav.net ;; global options: printcmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 61401 ;; flags: qr rd ra; QUERY: 1, ANSWER: 16, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;db.us.clamav.net.INA ;; ANSWER SECTION: db.us.clamav.net.1190INCNAMEdb.us.big.clamav.net. db.us.big.clamav.net.50INA194.47.250.218 db.us.big.clamav.net.50INA194.186.47.19 db.us.big.clamav.net.50INA200.236.31.1 db.us.big.clamav.net.50INA204.109.62.22 db.us.big.clamav.net.50INA207.57.106.31 db.us.big.clamav.net.50INA208.72.56.53 db.us.big.clamav.net.50INA64.246.134.219 db.us.big.clamav.net.50INA65.19.179.67 db.us.big.clamav.net.50INA69.12.162.28 db.us.big.clamav.net.50INA69.163.100.14 db.us.big.clamav.net.50INA88.198.67.125 db.us.big.clamav.net.50INA150.214.142.197 db.us.big.clamav.net.50INA155.98.64.87 db.us.big.clamav.net.50INA168.143.19.95 db.us.big.clamav.net.50INA194.8.197.22 ;; Query time: 91 msec ;; SERVER: 10.0.1.1#53(10.0.1.1) ;; WHEN: Tue Sep 13 19:37:53 2011 ;; MSG SIZE rcvd: 298 Note that 88.198.67.125 is far down the list, so I immediately ran $ sudo /usr/local/clamXav/bin/freshclam --stdout --quiet --no-warnings --log=/usr/local/clamXav/share/clamav/freshclam.log With the following results: -- ClamAV update process started at Tue Sep 13 19:40:13 2011 main.cld is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven) connect_error: getsockopt(SO_ERROR): fd=4 error=61: Connection refused Can't connect to port 80 of host db.US.clamav.net (IP: 88.198.67.125) Downloading daily-13610.cdiff [100%] Downloading daily-13611.cdiff [100%] daily.cld updated (version: 13611, sigs: 192595, f-level: 60, builder: guitar) bytecode.cld is up to date (version: 144, sigs: 41, f-level: 60, builder: edwin) Database updated (1038850 signatures) from db.US.clamav.net (IP: 69.163.100.14) So how could old 88 have possibly worked is way back to the top? Sending my browser to db.US.clamav.net gives me Safari can¹t open the page ³http://db.us.big.clamav.net/² because Safari can¹t connect to the server ³db.us.big.clamav.net². No matter how many times I try it. Was there anything else I need to try before restoring the hosts file? -Al- -- Al Varnell Mountain View, CA ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 8:07 PM, Dennis Peterson denni...@inetnw.com wrote: On 9/13/11 8:05 PM, Dennis Peterson wrote: I've just sent the URL to validator.wc3.org and got the same problem with this message: My fat fingers intended to type http://validator.wc.org and not what they did type. Or possibly http://validator.w3.org? -Al- -- Al Varnell Mountain View, CA ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 8:31 PM, Al Varnell wrote: Sounds like the server will be pulled, so you may not care, but since I went through the effort. Made changes to the hosts file. Ran dig $ db.us.clamav.net Does your dig use the host table? Mine does not. Same with nslookup. I can't imagine why they would, in fact. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 8:34 PM, Al Varnell wrote: On 9/13/11 8:07 PM, Dennis Petersondenni...@inetnw.com wrote: On 9/13/11 8:05 PM, Dennis Peterson wrote: I've just sent the URL to validator.wc3.org and got the same problem with this message: My fat fingers intended to type http://validator.wc.org and not what they did type. Or possibly http://validator.w3.org? -Al- Thank you, Al - I knew the truth would out! The lesson learned is if you can avoid it, don't work 48 hour shifts and then try to think and type at the same time :) dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/13/11 8:34 PM, Dennis Peterson denni...@inetnw.com wrote: On 9/13/11 8:31 PM, Al Varnell wrote: Sounds like the server will be pulled, so you may not care, but since I went through the effort. Made changes to the hosts file. Ran dig $ db.us.clamav.net Does your dig use the host table? Mine does not. Same with nslookup. I can't imagine why they would, in fact. Apparently not. I re-launched Terminal, just in case that was necessary, but it still didn't make any difference. -Al- -- Al Varnell Mountain View, CA ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 09/13/2011 12:33 PM, Al Varnell wrote: On 9/13/11 10:18 AM, sys...@ra-schaal.desys...@ra-schaal.de wrote: Am 13.09.2011 18:01, schrieb Al Varnell: On Sep 13, 2011, at 8:15 AM, Dandantear...@gmail.com wrote: Yet more failure on 88.198.67.125, this morning. This one is a double. I was going to wait a few more days to mention this, but since you bring it up... I have seen this twice a day almost every day since 29 Aug. The only times I didn't see this was when the database was reported to be up-to-date. During that same period, I was _never_ able to successfully connect to it. This can't be just my bad luck. just your bad luck 2011/09/05 - 297638 connects 2011/09/06 - 265677 connects 2011/09/07 - 265228 connects 2011/09/08 - 210367 connects 2011/09/09 - 230462 connects 2011/09/10 - 142702 connects 2011/09/11 - 120486 connects 2011/09/12 - 207272 connects 2011/09/13 - 129521 connetcs until now - 1916 CET as mentioned a few days befor, YOU have a very slow connection to my system. I'm half a world away from you, so I'm not really surprised by that, but what difference should it make? just use another mirror instead of crying all the time about your bad setup. What are you talking about? I have no choice whatsoever on the mirror I connect to! -Al- Well here I have to take exception. You have every option to choose mirrors that suit your liking better. If the US servers are not meeting your needs, pick a different region. If the US round-robin are using mirrors half way around the world, then. there is no detraction to picking default mirrors that are half way around the world but choosing something other then US as the location. The fact that ClamXav HAS chosen to make it inconvenient for users to change update frequency or setting of db mirrors is NOT a clamav fault. The mechanism exists in freschclam but the port to OS X has chosen to ignore this very important feature. Would you like me to write a user interface application so OS X users can do this very simple preference setting? And don't get me started on some of the stupid approaches Apple has taken to a very simple to manage OS like FreeBSD. Although I choose express no opinion on the MACH kernel versus other kernels, the MACH kernel choice, is not issue that has detracted from the ability to easily set preferences. Apple has chosen to go the Microsoft route of our users are too stupid to be allowed to do their own customization and as such we OS X users have to suffer as we do with the choices made in Redmond. -- Jim Preston jimli...@commspeed.net ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 09/13/2011 01:16 PM, Chuck Swiger wrote: This being said, there is definitely a recurring issue with this particular mirror. Since Aug 22, I've seen: % grep Can't connect to port 80 of host database.clamav.net (IP: 88.198.67.125) /var/log/freshclam.log | wc -l 27 ...with zero successful connections to that IP. The connectivity failure is entirely reproducible by hand: % telnet 88.198.67.125 80 Trying 88.198.67.125... telnet: connect to address 88.198.67.125: Connection refused telnet: Unable to connect to remote host Well I wonder if it is a configuration issue on the web server of thus mirror. Others have reported that it responds to pings but will not accept connections on port 80. Maybe the config is unrealistically limiting connections.. I don't consider this to be a significant problem since other mirrors are up, but it's not a matter of bandwidth or connectivity on my side. As it happens, I'm testing from Cupertino, CA via Apple's 17.0.0.0/8 network, and from a Time-Warner cable link from NYC, NY on 24.103.0.0/16. Is Apple running an ISP on 17.0.0.0/8? If so, maybe my objection to Apple having a class A pubic subnet is unjustified. -- Jim Preston jimli...@commspeed.net ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
