Just to add my 2 cents, for the most part if you have a decent nic card you could issue OS commands to drop the port rate of your interface to 10mbit and not waste cpu cycles on shaping/proxying.
Although i do recommend squid for this since i too use it to further filter/offload regex/hostname blocks as well. -byron --- Jay Pound <[EMAIL PROTECTED]> wrote: > there are a number of linux packages for QOS/traffic > shaping, my favorite is > wondershaper, I havent set it up since the 2.4 > kernel but it works well, > also if your not inclined to do something that > involved, your isp can give > that machine's ip address a car statement in > your/their cisco router > preventing that particular machine from using a max > of x bandwidth. or the > do it yourself solution buy the cheapest POS router > (linksys, generic) they > wont be able to route 20mbit of data through the > nat, at least older ones > couldent get much more than 5mbit or so (newer ones > can do 9mbit +) so there > are some solutions to your problem. > -J > ----- Original Message ----- > From: "Insurance Squared Inc." > <[EMAIL PROTECTED]> > To: <[email protected]> > Sent: Monday, January 16, 2006 6:02 PM > Subject: throttling bandwidth > > > > My ISP called and said my nutch crawler is chewing > up 20mbits on a line > > he's only supposed to be using 10. Is there an > easy way to tinker with > > how much bandwidth we're using at once? I know we > can change the number > > of open threads the crawler has, but it seems to > me this won't make a > > huge difference. If I chop the number of open > threads in half, it'll > > just download half the pages, twice as fast? I > stand to be corrected on > > this. > > > > Any other thoughts? doesn't have to be correct or > elegant as long as it > > works. > > > > Failing a reasonable solution in nutch, is there > some sort of linux > > level tool that will easily allow me to throttle > how much bandwidth the > > crawl is using at once? > > > > Thanks. > > > > > > > > -- > > This message has been scanned for viruses and > > dangerous content by MailScanner, and is > > believed to be clean. > > > > > > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > >
