Hi list,
I downloaded the nightly build of nutch+hadoop, and have been trying to
get it working on a small cluster of machines.
I have it working properly on a single machine, however when I try to
have my map and reduce tasks run on the cluster slaves, I get the
following exception:
060405 000219 SEVERE Can't open map
output:/home2/nutch/filesystem/mapreduce/local/part-2.out/task_m_1v749p
java.io.FileNotFoundException:
/home2/nutch/filesystem/mapreduce/local/part-2.out/task_m_1v749p
at
org.apache.hadoop.fs.LocalFileSystem.openRaw(LocalFileSystem.java:114)
(rest of stack trace snipped)
Oddly enough, this map task ran on the same machine which produced the
above error message. This it the output from the map task on the same
machine:
060405 000210 task_m_1v749p Child starting
060405 000211 Server connection on port 50050 from 127.0.0.1: starting
060405 000211 task_m_1v749p Client connection to 0.0.0.0:50050: starting
060405 000211 task_m_1v749p Client connection to 10.10.0.3:9000: starting
060405 000211 task_m_1v749p Using URL normalizer:
org.apache.nutch.net.BasicUrlNormalizer
060405 000211 Server connection on port 50050 from 127.0.0.1: starting
060405 000211 task_m_1v749p Client connection to 0.0.0.0:50050: starting
060405 000211 task_m_1v749p 1.0% /user/nutch/urls/urllist.txt:2+2
060405 000211 Task task_m_1v749p is done.
(parsing lines snipped for brevity)
All map tasks finish with the output above, however none of my reduce
tasks are finishing. The problem exists when the map task, and the
corresponding reduce task which depends on the map's output, are run on
the same machine or different machines. In both cases I see an IPC
timeout Exception being thrown, 1 minute (60,000 ms, as specified in the
hadoop-default.xml file) after the above "FileNotFound" exception is
generated.
Does anyone have any pointers as to where I should look to determine the
reason the map output is not being generated, or is not able to be accessed?
Regards,
-Shawn
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
