I made small assumption/mistake in a previous post. Not all of you are using Transport-Layer-Routers (aka Firewalls, or layer-4-Router)
But, small in-house companies are almost always using SHDSL etc., IP over ATM, IP over Frame Relay, ... Hardware between Crawler and Web-Site always has limitations such as CPU, RAM; and IP packets (layer 3 of OSI), and in some cases TCP (layer 4) are randomly/evenly distributed... If hardware allows to send 1,000,000 of IP packets per second, and you are trying to send 1,999,999 of IP packets per second, no one else can get access to Internet but you, even if you are using just 10% of the total available bandwidth. In some cases equipment gets overloaded even with 55-60% of the total channel loading. -----Original Message----- From: Fuad Efendi ... hardware allows to remember (due to RAM and CPU limitations) up to 1,000,000 of IP addresses, and 20,000 TCP ports for each "handshake". And his hardware randomize bandwidth evenly between 1,000,000 x 20,000 = 20,000,000,000 TCP connections... ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642 _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
