I'm doing some experimenting with NDFS to see if it will work
for my nutch cluster. It seems to do fine until my data nodes start
to run out of memory. This is a pilot project, so I'm using some
older systems -- P4/1.9ghz/512MB/20GB. Right now they're booting off
of a ramdisk image, but I'm going to be moving to a disk-based system
to see if that helps things.
Anyhow, my database node was able to generate the database and
import about 4 million URLs just fine. It generated segments from the
database just fine. I generated it for 4 fetchers and ran a fetcher
on each of my compute nodes.
To clarify, I've got one master node that maintains the
database,
and acts as the NDFS name node. I've got four compute nodes that perform
fetches, and act as data nodes.
Things went fine for a good 36 hours or so, then I noticed that
the systems were starting to swap and their performance started to tank.
After awhile, each node's fetch process started to do die off with
errors like:
SEVERE error writing output:java.io.IOException: Could not obtain new
output block for file /user/root/segments/20050919121742-2/content/data
... stack trace ...
SEVERE error writing output:java.io.IOException: key out of order:
312670 after 312670
... stack trace ...
... A few more of these errors ...
Exception in thread "main" java.lang.RunTimeException: SEVERE error
logged. Exiting fetcher.
And then the fetch dies.
Is there anything that can be done to prevent this, short of
adding
more RAM to these systems?
Thanks,
Ryan
--
Ryan Ordway
Unix Systems Administrator E-mail:
[EMAIL PROTECTED]
Oregon State University Libraries
[EMAIL PROTECTED]
121 The Valley Library
Corvallis, OR 97331 Desk: 541.737.8972
