I ran into the following problem running a hadoop job written in pig.Pls help
check what caused the issue. As I could tell, it seems to me the job/task
tracker failed for some reason but
name/data nodes still functioning.
The job simply seems to make no progress at all (no output, no log). But couple
of other hadoop jobs ran successfully before this one. hadoop fs -ls can still
list files. But I did "Hadoop job -list", it took too long and then failed with
error message as follows.
Exception in thread "main" java.io.IOException: Call to
hostname/ip-address:50002 failed on
local exception: Connection reset by peer at
org.apache.hadoop.ipc.Client.call(Client.java:699) at
org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at
org.apache.hadoop.mapred.$Proxy0.getProtocolVersion(Unknown Source) at
org.apache.hadoop.ipc.RPC.getProxy(RPC.java:319) at
org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:435) at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:429) at
org.apache.hadoop.mapred.JobClient.run(JobClient.java:1512) at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at
org.apache.hadoop.mapred.JobClient.main(JobClient.java:1727)Caused
by: java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcher.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at
sun.nio.ch.IOUtil.read(IOUtil.java:206) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:150)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:123)
at
java.io.FilterInputStream.read(FilterInputStream.java:116) at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:271)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at
java.io.BufferedInputStream.read(BufferedInputStream.java:237) at
java.io.DataInputStream.readInt(DataInputStream.java:370) at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:493)
at
org.apache.hadoop.ipc.Client$Connection.run(Client.java:438)
Web interface to job trac...@50030 simply came with no response at all.
By checking netstat, sometimes it shows 50030 and sometimes not. connections
and ports with data nodes were shown there.
Then, if I ran another pig, it failed with the following error:
Error before Pig is launched----------------------------ERROR
6009: Failed to create job client:Call to hostname/ip-address:50002 failed on
local exception: Connection reset by peer
org.apache.pig.backend.executionengine.ExecException:
ERROR 6009: Failed to create job client:Call to hostname/ip-address:50002
failed on
local exception: Connection reset by peer at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:217)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:137)
at
org.apache.pig.impl.PigContext.connect(PigContext.java:199) at
org.apache.pig.PigServer.<init>(PigServer.java:169) at
org.apache.pig.PigServer.<init>(PigServer.java:158) at
org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:54) at
org.apache.pig.Main.main(Main.java:395)Caused by:
java.io.IOException: Call to hostname/ip-address:50002 failed on
local exception: Connection reset by peer at
org.apache.hadoop.ipc.Client.call(Client.java:699) at
org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at
org.apache.hadoop.mapred.$Proxy1.getProtocolVersion(Unknown Source) at
org.apache.hadoop.ipc.RPC.getProxy(RPC.java:319) at
org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:435) at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:429) at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:398) at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:212)
... 6 moreCaused
by: java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcher.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at
sun.nio.ch.IOUtil.read(IOUtil.java:206) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:150)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:123)
at
java.io.FilterInputStream.read(FilterInputStream.java:116) at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:271)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at
java.io.BufferedInputStream.read(BufferedInputStream.java:237) at
java.io.DataInputStream.readInt(DataInputStream.java:370) at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:493)
at
org.apache.hadoop.ipc.Client$Connection.run(Client.java:438)================================================================================
Thank,
Michael
