Hi Thomas,
for this crawl setup we have a test environment of nutch 0.8,
10xAMD's, custom linux build, 100Mbit eth1, 1Gb eth0, each box has a
'caching' dns server.
Stefan
Am 06.03.2006 um 15:59 schrieb TDLN:
Stefan.
I know people having >500 mio pages index and I personal run
crawls with
~300 pages per second.
Sorry, but I have to ask: what kind of setup do you have (network,
hw, nutch
version) that you manage so many pages per second?
Unless this is a "company secret", it would be very nice to know
how you
manage this.
Rgrds, Thomas
---------------------------------------------------------------
company: http://www.media-style.com
forum: http://www.text-mining.org
blog: http://www.find23.net
