Hi, I have some junit tests for a MapReduce job that run fine from my IDE, but fail with the following error when using Maven (mvn test):
2011-08-17 15:08:49,191 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id : attempt_20110817150608891_0003_m_000027_0, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229) .... 2011-08-17 15:08:49,196 [32680187@qtp-19417347-2] WARN org.apache.hadoop.mapred.TaskLog - Failed to retrieve stderr log for task: attempt_20110817150608891_0003_m_000027_0 java.io.FileNotFoundException: /home/cionescu/dev/newsearch/newsearch-deleter/../logs/userlogs/job_20110817150608891_0003/attempt_20110817150608891_0003_m_000027_0/log.index (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:120) at java.io.FileReader.<init>(FileReader.java:55) at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:110) I have searched the archive and came over this: http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201107.mbox/browser but none of the suggestions there could solve my problem: - the disk is not full - the /userlogs directory doesn't contain too many files (also tried renaming the directory and it got recreated but error is still there) - permissions ok, /userlogs and its subdirectories are writeable (mkdir successfully with the user running the job). When examining the file system I could see that the stdout and stderr logs are created but there is no log.index. Stderr contains what seems to be a classpath error: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory at org.apache.hadoop.mapred.Child.<clinit>(Child.java:61) Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory at java.net.URLClassLoader$1.run(URLClassLoader.java:202) ... Could not find the main class: org.apache.hadoop.mapred.Child. Program will exit. I'm guessing this is the actual problem here. Does anyone have any idea on why this happens? Thanks, Cristina