Re: [clamav-users] Yet Another US Mirror Issue
On Sun, Sep 11, 2011 at 04:11:07PM -0400, Dan wrote: At 11:40 PM +0200 9/7/2011, Luca Gibelli wrote: Traffic is around 5TB/month on each mirror. Short of a paid service, which I doubt any of us want, few have such bandwidth available to donate. First of all, I think this whole thread is overreacting. I seriously doubt the mirror capacity is at maximum. Anyways, 5TB comes at 2MB/s average, which is not that much. I can do it with my $15 OVH/Kimsufi box and so do probably thousands of others. Clam needs to leverage the power of the Internet - as it is now, not yesterday. The simple, semi-linear propagate thru a few mirrors design has obviously reached a limit... 5 TB *per mirror* per month!!!??? Just to maintain a tiny 36 MB database? d'oh! It does sound a bit much for all the cdiffs etc, but maybe I'm underestimating the number of ClamAV users.. It may have worked just fine yesterday, but, seriously, just a model that's waiting to fall on its face as Clam becomes more popular. I don't think it can suddenly come _that_ much more popular, since it's already quite popular. So, I'm thinking that leaves two choices: 1) a cloud, a la Amazon S3. 2) p2p. Maybe, someday, when the well-cached cloud services are fully propagated *and* reliable world-wide, using a cloud in leiu of the traditional mirror set-up might be viable. But IMO that's years away and too expensive. There's nothing wrong with the current method. It's simple and cheap. You are underestimating the bandwidth available in the world. Either there really is no problem and ClamAV is just lazily fishing for more mirrors, or then they are just clueless and/or not having the substantial financial and engineering resources of a much larger organization (advertised in faq). Heck, even I could buy few boxes for mirrors, but I'm not going to do that as a private person since there are bazillion commercial entities that have or can get the bandwidth if needed, including Sourcefire itself. Right now, IMO, a p2p set-up would be the most viable. Continue to propagate via mirrors. *ADD* the torrent. Together, we clam users have many times the bandwidth needed! Is there a way to make freshclam grab and verify database files from a local directory? If there is, creating a torrent set-up would be fairly easy, even on an ad-hoc basis. I think it would be interesting to get a test going... WRT the reputation of p2p/torrents... There are quite a few legit uses for p2p. A number of open source products are even distributed via bittorrent. Yes, some ISPs are blocking the protocol -- but when shown that it's a legit use, they're usually willing to fix that. I like the idea of some 3rd party offering torrent service for the p2p-minded. What I don't want to see is freshclam bloated with some torrent libraries and stuff. You do realize that torrents actually need to have central servers for the .torrent files themselves? That's just the first step (freshclam would have already downloaded cdiffs at the same step). Then you actually need to have some trackers also, unless you are relying on DHT. Hopefully it's not the main database you end up downloading from some guys slow ADSL link.. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] improving ClamAV private mirroring?
On Fri, 15 Jul 2011 13:58:43 +0200 Tomasz Kojm tk...@clamav.net wrote: On Thu, 14 Jul 2011 23:15:20 -0400 James Ralston qralston+ml.clamav-us...@andrew.cmu.edu wrote: But freshclam falls over fairly badly if you try to use it to update internal clients from a private mirror, even though it has some options to help adjust its behavior for that purpose. Now I'm thinking that if freshclam is going to be a good tool for updating clients behind a private mirror, there really needs to be an option to specify that directly. Something like: # If PrivateMirror is set, freshclam assumes that the server it # names is a private mirror. In this case, freshclam does not # attempt to use DNS to determine whether its databases are # out-of-date, but instead downloads the database files from the # private mirror every time it runs. For each database, freshclam # first attempts to download the CVD file. If that fails, # freshclam tries to grab the CLD file. If the attempt to # download the CVD file and attempt to download the CLD file both # fail, then that counts as a single attempt against MaxAttempts. # Default: disabled. #PrivateMirror clam-update.example.org I see no problem adding such a feature, just open a feature request at bugs.clamav.net The option has been implemented in clamav-devel. This is how it works: # This option allows you to easily point freshclam to private mirrors. # If PrivateMirror is set, freshclam does not attempt to use DNS # to determine whether its databases are out-of-date, instead it will # use the If-Modified-Since request or directly check the headers of the # remote database files. For each database, freshclam first attempts # to download the CLD file. If that fails, it tries to download the # CVD file. This option overrides DatabaseMirror, DNSDatabaseInfo # and ScriptedUpdates. It can be used multiple times to provide # fall-back mirrors. # Default: disabled #PrivateMirror mirror1.mynetwork.com #PrivateMirror mirror2.mynetwork.com Regards, -- oo. Tomasz Kojm tk...@clamav.net (\/)\. http://www.ClamAV.net/gpg/tkojm.gpg \..._ 0DCA5A08407D5288279DB43454822DC8985A444B //\ /\ Mon Sep 12 16:31:07 CEST 2011 ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
At 9:22 AM +0300 9/12/2011, Henrik K wrote: On Sun, Sep 11, 2011 at 04:11:07PM -0400, Dan wrote: At 11:40 PM +0200 9/7/2011, Luca Gibelli wrote: Traffic is around 5TB/month on each mirror. Short of a paid service, which I doubt any of us want, few have such bandwidth available to donate. First of all, I think this whole thread is overreacting. I seriously doubt the mirror capacity is at maximum. Noone has suggested maximum. The issue is that the mirrors are so overloaded that it's often taking freshclam an excessive amount of time to do its thing, because of the time-outs / connection failures. No big deal if it's the update run in the background. But if it's on-demand update preceding a user-driven scan, it's making the user sit there, twiddling its thumbs, for up to a minute or two. Luca's response to the problem is that more mirror capacity is needed. Hence the discussion of alternatives... Anyways, 5TB comes at 2MB/s average, which is not that much. I can do it with my $15 OVH/Kimsufi box and so do probably thousands of others. Perhaps, where you live. Here, in the good'ole USofA, if I set up a server to feed 170 GB/day, my ISP would shut me down and bill me big. So, I'm thinking that leaves two choices: 1) a cloud, a la Amazon S3. 2) p2p. Maybe, someday, when the well-cached cloud services are fully propagated *and* reliable world-wide, using a cloud in leiu of the traditional mirror set-up might be viable. But IMO that's years away and too expensive. There's nothing wrong with the current method. It's simple and cheap. You are underestimating the bandwidth available in the world. I didn't say there's anything wrong with the current method. It's just overwhelmed, and I doubt that adding a mirror or two will fix it now or even in the long term. I'm looking to explore ways of supplementing the current infrastructure. You do realize that torrents actually need to have central servers for the .torrent files themselves? Are you saying that including a 30 KB file in the Clam distro is too heavy of a burden? That's just the first step (freshclam would have already downloaded cdiffs at the same step). Then you actually need to have some trackers also, unless you are relying on DHT. Hopefully it's not the main database you end up downloading from some guys slow ADSL link.. The point of a torrent is that noone provides all the data from one source. It's *distributed*. - Dan. -- - Psychoceramic Emeritus; South Jersey, USA, Earth. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
G.W. Haywood wrote: The ClamAV database mirrors appear to have a growing capacity problem. Torrents are intended to alleviate the problem, and it takes, oh, ten minutes to set one up. Scripts already exist which could be adapted fairly easily to use torrents instead of mirrors to download the data. The DNS tells us the filenames to ask for. Anybody can run a torrent, the torrent software can control the data rates used by clients, and a network of torrents is a much more challenging target for the Bad Guys than a few mirrors. So what's the problem? Maybe I just don't understand enough about how torrents actually work... but wouldn't you need to update the .torrent every time the virus database changed? I don't think the standard torrent protocol includes any support for something like that... -kgd ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 9/12/2011 11:05 AM, Dan wrote: At 9:22 AM +0300 9/12/2011, Henrik K wrote: On Sun, Sep 11, 2011 at 04:11:07PM -0400, Dan wrote: At 11:40 PM +0200 9/7/2011, Luca Gibelli wrote: Traffic is around 5TB/month on each mirror. Short of a paid service, which I doubt any of us want, few have such bandwidth available to donate. First of all, I think this whole thread is overreacting. I seriously doubt the mirror capacity is at maximum. Noone has suggested maximum. The issue is that the mirrors are so overloaded that it's often taking freshclam an excessive amount of time to do its thing, because of the time-outs / connection failures. No big deal if it's the update run in the background. But if it's on-demand update preceding a user-driven scan, it's making the user sit there, twiddling its thumbs, for up to a minute or two. Luca's response to the problem is that more mirror capacity is needed. Hence the discussion of alternatives... Anyways, 5TB comes at 2MB/s average, which is not that much. I can do it with my $15 OVH/Kimsufi box and so do probably thousands of others. Perhaps, where you live. Here, in the good'ole USofA, if I set up a server to feed 170 GB/day, my ISP would shut me down and bill me big. HERE HERE! My ISP is pretty cool about letting users do what they want. However, if I started moving 170GB / day they would definitely be chasing me down to have a chat. :-) When they start offering inexpensive 10Mbit links to the net, a mirror would be an option, but not right now. -- Sincerely, Nathan Gibbs Systems Administrator Christ Media http://www.cmpublishers.com signature.asc Description: OpenPGP digital signature ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On Mon, Sep 12, 2011 at 12:41:14PM -0400, Nathan Gibbs wrote: On 9/12/2011 11:05 AM, Dan wrote: At 9:22 AM +0300 9/12/2011, Henrik K wrote: On Sun, Sep 11, 2011 at 04:11:07PM -0400, Dan wrote: At 11:40 PM +0200 9/7/2011, Luca Gibelli wrote: Traffic is around 5TB/month on each mirror. Short of a paid service, which I doubt any of us want, few have such bandwidth available to donate. First of all, I think this whole thread is overreacting. I seriously doubt the mirror capacity is at maximum. Noone has suggested maximum. The issue is that the mirrors are so overloaded that it's often taking freshclam an excessive amount of time to do its thing, because of the time-outs / connection failures. No big deal if it's the update run in the background. But if it's on-demand update preceding a user-driven scan, it's making the user sit there, twiddling its thumbs, for up to a minute or two. Luca's response to the problem is that more mirror capacity is needed. Hence the discussion of alternatives... Anyways, 5TB comes at 2MB/s average, which is not that much. I can do it with my $15 OVH/Kimsufi box and so do probably thousands of others. Perhaps, where you live. Here, in the good'ole USofA, if I set up a server to feed 170 GB/day, my ISP would shut me down and bill me big. HERE HERE! My ISP is pretty cool about letting users do what they want. However, if I started moving 170GB / day they would definitely be chasing me down to have a chat. :-) When they start offering inexpensive 10Mbit links to the net, a mirror would be an option, but not right now. Guys, I'm not talking about some home or office ISP lines. I'm talking about rented dedicated servers that have huge bandwidth by contract. Why do you make pointless arguments? Depending on where you live or want the servers to be located, they can be cheap or amazingly cheap. And Dan, please familiarize yourself first on how torrents work. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On Sep 12, 2011, at 10:58 AM, Henrik K h...@hege.li wrote: I'm not talking about some home or office ISP lines. I'm talking about rented dedicated servers that have huge bandwidth by contract. Why do you make pointless arguments? Has anybody talked to Apple? Every box of Server software they sell comes with clavav, so they are already invested and have plenty of capacity world-wide. Sent from Janet's iPad -Al- -- Al Varnell ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
At 8:58 PM +0300 9/12/2011, Henrik K wrote: Guys, I'm not talking about some home or office ISP lines. I'm talking about rented dedicated servers that have huge bandwidth by contract. Why do you make pointless arguments? Excuse me? Pointless? Is that your way of disagreeing intelligently or just trying to shut the conversation down? In YOUR opinion individuals and even small businesses are incapable of contributing to Clam's strained infrastructure? So OUR suggestions and inquiries on this USER mailing list are ... pointless? And Dan, please familiarize yourself first on how torrents work. I know pretty much how they work. What's your point here? Is there some design issue that invalidates the idea of using a p2p/torrent type distribution method to supplement the mirrors? I just love having a design idea shot down with no discussion because it's POINTLESS. Or perhaps I've made the error here? Is there some heresy in asking my question yesterday: Is there a way to make freshclam grab and verify database files from a local directory? - Dan. -- - Psychoceramic Emeritus; South Jersey, USA, Earth. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 09/12/2011 10:54 PM, Dan wrote: Is there a way to make freshclam grab and verify database files from a local directory? Yes, but they don't work for fetching incremental updates from local dir (DatabaseCustomURL, PrivateMirror). What you could try is set DatabaseMirror to a local webserver, which fetches CDIFFs/CVDs from torrents on demand. FWIW fetching small cdiffs (1kb) via torrents is probably a bad idea as it'll take a lot more for you to find peers than to download from a mirror. Best regards, --Edwin ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On 09/12/2011 12:20 PM, Al Varnell wrote: On Sep 12, 2011, at 10:58 AM, Henrik Kh...@hege.li wrote: I'm not talking about some home or office ISP lines. I'm talking about rented dedicated servers that have huge bandwidth by contract. Why do you make pointless arguments? Has anybody talked to Apple? Every box of Server software they sell comes with clavav, so they are already invested and have plenty of capacity world-wide. Sent from Janet's iPad -Al- And Apple (along with several other large corporations) has an over abundance of public IP addresses to assign to their own hosted servers 017/8 (16,777,216 IP Addresses). -- Jim Preston jimli...@commspeed.net ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On Sep 12, 2011, at 3:20 PM, Al Varnell wrote: On Sep 12, 2011, at 10:58 AM, Henrik K h...@hege.li wrote: I'm not talking about some home or office ISP lines. I'm talking about rented dedicated servers that have huge bandwidth by contract. Why do you make pointless arguments? Has anybody talked to Apple? Yes. ...and you know that's all I can say about it. -- Joel Esler OpenSource Community Manager Sourcefire ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On Mon, Sep 12, 2011 at 03:54:44PM -0400, Dan wrote: At 8:58 PM +0300 9/12/2011, Henrik K wrote: Guys, I'm not talking about some home or office ISP lines. I'm talking about rented dedicated servers that have huge bandwidth by contract. Why do you make pointless arguments? Excuse me? Pointless? Is that your way of disagreeing intelligently or just trying to shut the conversation down? In YOUR opinion individuals and even small businesses are incapable of contributing to Clam's strained infrastructure? So OUR suggestions and inquiries on this USER mailing list are ... pointless? I'm sorry but that's the fact. If mirrors need bandwidth, it's not going to work on some slow home connection. Why do you take it so personally? If you want to help, buy a server and host a mirror. And Dan, please familiarize yourself first on how torrents work. I know pretty much how they work. What's your point here? Is there some design issue that invalidates the idea of using a p2p/torrent type distribution method to supplement the mirrors? Obviously you didn't think how you are going to download all those cdiffs. You do realize that all of them need .torrent files also? It's pointless overhead. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [clamav-users] Yet Another US Mirror Issue
On Mon, Sep 12, 2011 at 05:57:24PM -0400, Nathan Gibbs wrote: On 9/12/2011 1:58 PM, Henrik K wrote: Guys, I'm not talking about some home or office ISP lines. I'm talking about rented dedicated servers that have huge bandwidth by contract. OK, but what the rest of us are talking about is taking load off the global clamav mirror infrastructure. Particularly the US section. And I'm not?? But a da*n US server and host a mirror. Even as a individual if you like. Depending on where you live Because it is our section of the infrastructure that is having issues. Please read the thread title. Even I can buy some US servers if I want. There are lots of providers to choose from. or want the servers to be located, they can be cheap or amazingly cheap. I don't care where the servers are as long as I can get the current DBs. Rehash 1. The Clamav Project needs more capacity especially in the US zone. 2. Many of us have gone to a local mirror configuration to use as little of the capacity as possible. 3. The Clamav Project still needs more capacity. 4. Many of us would step up to the plate and provide this capacity if it were within our ability to do so. If you are an individual not able to put $15-$100 a month, then yes, it's not in your capability. 5. Barring that we are asking about torrent because we would step up to the plate and provide what is within our ability to provide. I could easily provide 20MB of transfer a month initially and maybe more. However 5TB / month is definitely out of the question. No one thinks any less of you for trying to help, on the contrary. But if you can't even get any facts straight etc, it's just messing up the thread. Let's not forget that ClamAV is backed by a commercial organization?? If they wanted US bandwidth badly, they can get it. If not by buying, then probably just by asking around or even on the web page? Why do you think it's not mentioned there. Probably very few users read this list. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
