Hi , all
I wrote a pig script and it works fine. After I add more computation to
the existing pig script, the hadoop job count exceed 70, and I get this error
from job tracker:
2012-08-08 00:35:22,648 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5
on 8012, call submitJob(job_201208071030_3456,
hdfs://DC-hadoop01:9000/home/hadoop/hdfs/hdfstmp/mapred/staging/hadoop/.staging/job_201208071030_3456,
org.apache.hadoop.security.Credentials@4ff3b079) from 10.20.100.34:19728:
error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3644)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3611)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeThreads(DFSClient.java:4026)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:4069)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:4004)
at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.close(DFSClient.java:1514)
at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:383)
at
org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:339)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:1499)
at org.apache.hadoop.fs.FileSystem.closeAllForUGI(FileSystem.java:289)
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:491)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3642)
It seems like the DFSOutputStream the job tracker is trying to closing is
already closed, so the streamer field is already null.
But I can not turn off the fs cache, since I'm afraid the leaked
DFSOutputstream may cause other problems.
Is there any other solution?
Thanks.
Haitao Yao
[email protected]
weibo: @haitao_yao
Skype: haitao.yao.final
