Hi,
I've been running the latest trunk nutch version on a cluster of 10
machines. Fetch mappers always finish without any problem over 4,000,000
pages, but some reducers fails because of "Task failed to report status for
602 seconds. Killing." Once this task fails even if it gets assigned to
another machine that fails again.
I checked the reducer of the fetcher class, and it seems to be an identity
reducer that gets stuck for one key and doesn't move any further. I am not
storing any http contents or files, why reducer should take this long for a
key which is a URL and its content is limited to 100,000 bytes.
These faulty reducers do the copying and sorting (up to 66%) without any
problem, and then they get stuck in the reduce stage.
Thank. Mike
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
