No. These are production jobs which were working pretty fine and suddenly, we started seeing these issues. And, if you see the error log, the jobs are failing at the time of submission itself while copying the application jar. And, when I see the client machine disk size and also HDFS, it is only 60% full.
Thanks Pallavi -----Original Message----- From: prashant ullegaddi [mailto:[email protected]] Sent: Monday, August 03, 2009 10:10 AM To: [email protected] Subject: Re: No Space Left On Device though space is available Are you using any space on local nodes? When we indexed 1TB on 8 nodes, we were creating index on local file system and then copying the same to DFS. It so happened that there wasn't any space left. After that we started moving the index instead of copying it. Everything worked fine. Probably that could be a problem with your application as well. Thanks, Prashant. On Mon, Aug 3, 2009 at 10:05 AM, Pallavi Palleti < [email protected]> wrote: > Hi all, > > We are having a 60 node cluster running hadoop-0.18.2. We are seeing "No > Space Left On Device" and the detailed error is > org.apache.hadoop.ipc.RemoteException: java.io.IOException: > java.lang.RuntimeException: javax.xml.transfor > m.TransformerException: java.io.IOException: No space left on device > at > org.apache.hadoop.conf.Configuration.write(Configuration.java:996) > at > org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java :530) > at > org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:196) > at > org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1783) > at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888) > > at org.apache.hadoop.ipc.Client.call(Client.java:715) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) > at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:788) > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026) > > Surprisingly, there is no space issue. Still, it is giving above error. Can > someone kindly let me know what could be the issue? > > Thanks > Pallavi >
