there are a number of linux packages for QOS/traffic shaping, my favorite is wondershaper, I havent set it up since the 2.4 kernel but it works well, also if your not inclined to do something that involved, your isp can give that machine's ip address a car statement in your/their cisco router preventing that particular machine from using a max of x bandwidth. or the do it yourself solution buy the cheapest POS router (linksys, generic) they wont be able to route 20mbit of data through the nat, at least older ones couldent get much more than 5mbit or so (newer ones can do 9mbit +) so there are some solutions to your problem. -J ----- Original Message ----- From: "Insurance Squared Inc." <[EMAIL PROTECTED]> To: <[email protected]> Sent: Monday, January 16, 2006 6:02 PM Subject: throttling bandwidth
> My ISP called and said my nutch crawler is chewing up 20mbits on a line > he's only supposed to be using 10. Is there an easy way to tinker with > how much bandwidth we're using at once? I know we can change the number > of open threads the crawler has, but it seems to me this won't make a > huge difference. If I chop the number of open threads in half, it'll > just download half the pages, twice as fast? I stand to be corrected on > this. > > Any other thoughts? doesn't have to be correct or elegant as long as it > works. > > Failing a reasonable solution in nutch, is there some sort of linux > level tool that will easily allow me to throttle how much bandwidth the > crawl is using at once? > > Thanks. > > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
