Hello,
I have a four node hadoop cluster running hadoop v.0.20.2 on CentOS 5.6.
Here is my layout:
Name01.hadoop.stage (namenode)
Name02.hadoop.stage (sec namenode / jobtracker)
Data01.hadoop.stage (data node)
Data02.hadoop.stage (data node)
When trying to run a benchmark test for this newly-stood up cluster I'm
getting errors. This is the command (run as the hadoop user on my
name01.hadoop.stage node):
# /opt/hadoop/bin/hadoop jar /opt/hadoop/hadoop-0.20.2-test.jar TestDFSIO
-write -nrFiles 1 -fileSize 10
Here is the output:
{{BEGIN}}
TestFDSIO.0.0.4
11/05/12 09:35:52 INFO mapred.FileInputFormat: nrFiles = 1
11/05/12 09:35:52 INFO mapred.FileInputFormat: fileSize (MB) = 10
11/05/12 09:35:52 INFO mapred.FileInputFormat: bufferSize = 1000000
11/05/12 09:35:52 INFO mapred.FileInputFormat: creating control file: 10
mega bytes, 1 files
11/05/12 09:35:52 INFO mapred.FileInputFormat: created control files for: 1
files
11/05/12 09:35:52 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
11/05/12 09:35:52 INFO mapred.FileInputFormat: Total input paths to process
: 1
11/05/12 09:35:52 INFO mapred.JobClient: Running job: job_201105120935_0001
11/05/12 09:35:53 INFO mapred.JobClient: map 0% reduce 0%
11/05/12 09:35:59 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:35:59 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_0&filter=stdout
11/05/12 09:35:59 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_0&filter=stderr
11/05/12 09:36:05 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_r_000002_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:05 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000002_0&filter=stdout
11/05/12 09:36:05 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000002_0&filter=stderr
11/05/12 09:36:14 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_1, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:14 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_1&filter=stdout
11/05/12 09:36:14 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_1&filter=stderr
11/05/12 09:36:20 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_2, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:20 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_2&filter=stdout
11/05/12 09:36:20 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_2&filter=stderr
11/05/12 09:36:33 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:33 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_0&filter=stdout
11/05/12 09:36:33 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_0&filter=stderr
11/05/12 09:36:39 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_r_000001_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:39 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000001_0&filter=stdout
11/05/12 09:36:39 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000001_0&filter=stderr
11/05/12 09:36:48 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_1, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:48 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_1&filter=stdout
11/05/12 09:36:48 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_1&filter=stderr
11/05/12 09:36:54 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_2, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:54 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_2&filter=stdout
11/05/12 09:36:54 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_2&filter=stderr
11/05/12 09:37:00 INFO mapred.JobClient: Job complete: job_201105120935_0001
11/05/12 09:37:00 INFO mapred.JobClient: Counters: 0
java.io.IOException: Job failed!
at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at
org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:236)
at
org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:218)
at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:354)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver
.java:68)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at
org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
{{END}}
Looking at some older posts I thought it may be related to DNS so I added
the hosts/ips of my four-node cluster to /etc/hosts but that didn't help.
Turning DEBUG on on my data01 node I see exceptions similar to:
{{BEGIN}}
TestFDSIO.0.0.4
11/05/12 09:35:52 INFO mapred.FileInputFormat: nrFiles = 1
11/05/12 09:35:52 INFO mapred.FileInputFormat: fileSize (MB) = 10
11/05/12 09:35:52 INFO mapred.FileInputFormat: bufferSize = 1000000
11/05/12 09:35:52 INFO mapred.FileInputFormat: creating control file: 10
mega bytes, 1 files
11/05/12 09:35:52 INFO mapred.FileInputFormat: created control files for: 1
files
11/05/12 09:35:52 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
11/05/12 09:35:52 INFO mapred.FileInputFormat: Total input paths to process
: 1
11/05/12 09:35:52 INFO mapred.JobClient: Running job: job_201105120935_0001
11/05/12 09:35:53 INFO mapred.JobClient: map 0% reduce 0%
11/05/12 09:35:59 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:35:59 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_0&filter=stdout
11/05/12 09:35:59 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_0&filter=stderr
11/05/12 09:36:05 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_r_000002_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:05 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000002_0&filter=stdout
11/05/12 09:36:05 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000002_0&filter=stderr
11/05/12 09:36:14 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_1, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:14 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_1&filter=stdout
11/05/12 09:36:14 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_1&filter=stderr
11/05/12 09:36:20 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_2, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:20 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_2&filter=stdout
11/05/12 09:36:20 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_2&filter=stderr
11/05/12 09:36:33 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:33 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_0&filter=stdout
11/05/12 09:36:33 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_0&filter=stderr
11/05/12 09:36:39 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_r_000001_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:39 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000001_0&filter=stdout
11/05/12 09:36:39 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000001_0&filter=stderr
11/05/12 09:36:48 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_1, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:48 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_1&filter=stdout
11/05/12 09:36:48 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_1&filter=stderr
11/05/12 09:36:54 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_2, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
11/05/12 09:36:54 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_2&filter=stdout
11/05/12 09:36:54 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_2&filter=stderr
11/05/12 09:37:00 INFO mapred.JobClient: Job complete: job_201105120935_0001
11/05/12 09:37:00 INFO mapred.JobClient: Counters: 0
java.io.IOException: Job failed!
at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at
org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:236)
at
org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:218)
at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:354)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver
.java:68)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at
org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
{{END}
Another thread mentioned a full FS being a possibility - but my FS usage is
near 0%. Any help/pointers would be much appreciated.
Thanks,
Matt
