It occures in our hadoop cluster)(0.20.1 version) from time to time(the log is below). Anyone could help? what hadoop is doing in the course of(after) committing before the map task is done? joWhat kind of reasons could make communication error. The job is still regarded as succeed even with this communication error. Thanks for the help.
2009-11-25 01:21:30,476 WARN org.apache.hadoop.fs.FileSystem: "master1:9000" is a deprecated filesystem name. Use "hdfs://master1:9000/" instead. 2009-11-25 01:21:30,541 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId= 2009-11-25 01:21:30,559 WARN org.apache.hadoop.fs.FileSystem: "master1:9000" is a deprecated filesystem name. Use "hdfs://master1:9000/" instead. ... 2009-11-25 01:21:31,616 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 27 2009-11-25 01:21:31,624 WARN org.apache.hadoop.fs.FileSystem: "master1:9000" is a deprecated filesystem name. Use "hdfs://master1:9000/" instead. 2009-11-25 01:21:31,627 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100 2009-11-25 01:21:31,799 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720 2009-11-25 01:21:31,799 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680 2009-11-25 01:21:33,266 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2009-11-25 01:21:33,277 WARN org.apache.hadoop.fs.FileSystem: "master1:9000" is a deprecated filesystem name. Use "hdfs://master1:9000/" instead. 2009-11-25 01:21:37,187 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2009-11-25 01:21:37,236 WARN org.apache.hadoop.fs.FileSystem: "master1:9000" is a deprecated filesystem name. Use "hdfs://master1:9000/" instead. 2009-11-25 01:21:37,271 INFO org.apache.hadoop.mapred.TaskRunner: Task:attempt_200911171829_0181_m_000008_0 is done. And is in the process of commiting 2009-11-25 01:22:37,323 INFO org.apache.hadoop.mapred.TaskRunner: Communication exception: java.io.IOException: Call to /127.0.0.1:53674failed on local exception: java.nio.channels.ClosedByInterruptException at org.apache.hadoop.ipc.Client.wrapException(Client.java:774) at org.apache.hadoop.ipc.Client.call(Client.java:742) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at org.apache.hadoop.mapred.$Proxy0.statusUpdate(Unknown Source) at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:543) at java.lang.Thread.run(Thread.java:619) Caused by: java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341) at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at java.io.DataOutputStream.flush(DataOutputStream.java:106) at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:480) at org.apache.hadoop.ipc.Client.call(Client.java:720) ... 4 more 2009-11-25 01:22:37,338 INFO org.apache.hadoop.mapred.TaskRunner: Task 'attempt_200911171829_0181_m_000008_0' done.
