Hi All,
I am firing a pig script using PigServer from a web application (tomcat 7.0).
The script runs every five minutes via PigServer. There are around ten jar
files that are being written in {tomcat.home}/temp directory repeatedly which
is talking lot of space. After a while, I start seeing following exceptions in
tomcat log:
WARN : org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher -
There is no log file to write to.ERROR:
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Backend
error message during job submissionjava.io.IOException: Cannot run program
"chmod": java.io.IOException: error=12, Cannot allocate memory at
java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at
org.apache.hadoop.util.Shell.runCommand(Shell.java:200) at
org.apache.hadoop.util.Shell.run(Shell.java:182) at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:461) at
org.apache.hadoop.util.Shell.execCommand(Shell.java:444) at
org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:508)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:319)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:839)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:396) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807) at
org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:619)Caused by: java.io.IOException:
java.io.IOException: error=12, Cannot allocate memory at
java.lang.UNIXProcess.<init>(UNIXProcess.java:148) at
java.lang.ProcessImpl.start(ProcessImpl.java:65) at
java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 21 more
Hadoop logs look fine and the same script on same data set works fine when I
run it manually. Multiple parallel instances (I tried 3 at same time) also
works fine when ran manually.
I am using cloudera build of pig (pig-0.8.1-cdh3u2) and hadoop(0.20.2-cdh3u2)
(running on remote cluster). Any suggestions?
Thanks,Rakesh
