Hi all,
I'm trying to get Hadoop 0.13 running from Intellij so I can test my
own Map and Reduce classes on the local job runner. To do that I'm
first trying to get the WordCount sample to work and I'm already
running into trouble:
I copied the WordCount class into my own package from my own project,
created a directory 'in' containing the file from which I want to
count the words. I copied the 'conf' dir from the hadoop dir that I
know contains a configuration that allows me to run the WordCount
sample.
This is the output from my console:
******************************
C:\java\jdk1.5.0\bin\java
-Xmx1000m
-Dhadoop.log.dir=logs
-Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=.
-Dhadoop.id.str=
-Dhadoop.root.logger=DEBUG,console
-Didea.launcher.port=7534
"-Didea.launcher.bin.path=C:\java\IntelliJ IDEA 5.1\bin"
-Dfile.encoding=windows-1252
-classpath "
C:\java\jdk1.5.0\jre\lib\charsets.jar;
C:\java\jdk1.5.0\jre\lib\deploy.jar;
C:\java\jdk1.5.0\jre\lib\javaws.jar;
C:\java\jdk1.5.0\jre\lib\jce.jar;
C:\java\jdk1.5.0\jre\lib\jsse.jar;
C:\java\jdk1.5.0\jre\lib\plugin.jar;
C:\java\jdk1.5.0\jre\lib\rt.jar;
C:\java\jdk1.5.0\jre\lib\ext\dnsns.jar;
C:\java\jdk1.5.0\jre\lib\ext\localedata.jar;
C:\java\jdk1.5.0\jre\lib\ext\sunjce_provider.jar;
C:\java\jdk1.5.0\jre\lib\ext\sunpkcs11.jar;
C:\mycomp\myproject\core\target\test-classes;
C:\mycomp\myproject\core\target\classes;
C:\mycomp\myproject\core\lib\hadoop-0.13.0-core.jar;
C:\java\IntelliJ IDEA 5.1\lib\idea_rt.jar"
com.intellij.rt.execution.application.AppMain nl.mycomp.TestHadoop in
target/out
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
at nl.mycomp.TestHadoop.main(TestHadoop.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
Process finished with exit code 1
******************************
And this is the content of the log4j log file:
******************************
DEBUG 15:36:11,813 main org.apache.hadoop.conf.Configuration -
java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:93)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:66)
at nl.mycomp.TestHadoop.main(TestHadoop.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
DEBUG 15:36:12,313 main org.apache.hadoop.mapred.JobClient - default
FileSystem: file:///
INFO 15:36:12,344 main org.apache.hadoop.mapred.FileInputFormat -
Total input paths to process : 1
DEBUG 15:36:12,344 main org.apache.hadoop.mapred.JobClient - Creating
splits at file:/tmp/hadoop-jeroen/mapred/system/submit_em16mi/job.split
DEBUG 15:36:12,360 main org.apache.hadoop.mapred.FileInputFormat -
Total # of splits: 2
DEBUG 15:36:12,579 main org.apache.hadoop.conf.Configuration -
java.io.IOException: config(config)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:103)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:77)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:83)
at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:260)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:376)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:543)
at nl.mycomp.TestHadoop.main(TestHadoop.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
DEBUG 15:36:12,657 main org.apache.hadoop.conf.Configuration -
java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:93)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:107)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:87)
at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:260)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:376)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:543)
at nl.mycomp.TestHadoop.main(TestHadoop.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
INFO 15:36:12,719 main org.apache.hadoop.mapred.JobClient - Running
job: job_8wasq7
DEBUG 15:36:12,719 Thread-0 org.apache.hadoop.mapred.FileInputFormat -
Total # of splits: 1
DEBUG 15:36:12,735 Thread-0 org.apache.hadoop.conf.Configuration -
java.io.IOException: config(config)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:103)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:77)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126)
DEBUG 15:36:12,782 Thread-0 org.apache.hadoop.mapred.MapTask - Writing
local split to /tmp/hadoop-jeroen/mapred/system/submit_em16mi/split.dta
INFO 15:36:12,782 Thread-0 org.apache.hadoop.mapred.MapTask - numReduceTasks: 1
INFO 15:36:13,719 main org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
INFO 15:36:13,751 Thread-0 org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
INFO 15:36:14,735 main org.apache.hadoop.mapred.JobClient - map 9% reduce 0%
INFO 15:36:14,766 Thread-0 org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
INFO 15:36:15,735 main org.apache.hadoop.mapred.JobClient - map 23% reduce 0%
INFO 15:36:15,969 Thread-0 org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
INFO 15:36:16,735 main org.apache.hadoop.mapred.JobClient - map 36% reduce 0%
INFO 15:36:17,063 Thread-0 org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
INFO 15:36:17,735 main org.apache.hadoop.mapred.JobClient - map 54% reduce 0%
INFO 15:36:18,079 Thread-0 org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
INFO 15:36:18,798 main org.apache.hadoop.mapred.JobClient - map 73% reduce 0%
INFO 15:36:19,094 Thread-0 org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
DEBUG 15:36:19,751 Sort progress reporter for task map_0000
org.apache.hadoop.mapred.MapTask - Started thread: Sort progress
reporter for task map_0000
WARN 15:36:19,766 Thread-0 org.apache.hadoop.mapred.LocalJobRunner - job_8wasq7
java.io.IOException: CreateProcess: df -k
C:\tmp\hadoop-jeroen\mapred\local error=2
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:81)
at java.lang.ProcessImpl.start(ProcessImpl.java:30)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
at java.lang.Runtime.exec(Runtime.java:591)
at java.lang.Runtime.exec(Runtime.java:464)
at org.apache.hadoop.fs.DF.doDF(DF.java:60)
at org.apache.hadoop.fs.DF.<init>(DF.java:53)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:181)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:218)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:88)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(MapTask.java:405)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:620)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:187)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:131)
******************************
Could anyone please help me and tell me what's causing this problem?
On a sidenote: Why is hadoop creating tmp files in C:\tmp? I did not
configure C:\tmp to be my tempdir, that's
C:\DOCUME~1\jeroen\LOCALS~1\Temp which is set through an environment
variable.
Thanks and regards,
Jeroen
