I figured out the problem. There is a JIRA opened on the same issue (https://issues.apache.org/jira/browse/PIG-1313). The bug was closed as it was fixed in 0.8. However, I see it happening in cloudera build pig-0.8.1-cdh3u2. My app is running fine for 12 hours now after I started spawning a separate JVM for every pig script.
> From: [email protected] > To: [email protected] > Subject: RE: Issue with PigServer in web application (tomcat container) > Date: Mon, 2 Apr 2012 01:07:59 +0000 > > > I am using PigServer.registerScript() to launch my script. However, the post > http://search-hadoop.com/m/N8Owj1uu0131&subj=Re+PigServer+vs+PigRunner says > that it should not be used. I am wondering if it could be contributing to > this problem. > > > From: [email protected] > > To: [email protected] > > Subject: Issue with PigServer in web application (tomcat container) > > Date: Sun, 1 Apr 2012 20:32:17 +0000 > > > > > > Hi All, > > I am firing a pig script using PigServer from a web application (tomcat > > 7.0). The script runs every five minutes via PigServer. There are around > > ten jar files that are being written in {tomcat.home}/temp directory > > repeatedly which is talking lot of space. After a while, I start seeing > > following exceptions in tomcat log: > > WARN : > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - > > There is no log file to write to.ERROR: > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - > > Backend error message during job submissionjava.io.IOException: Cannot run > > program "chmod": java.io.IOException: error=12, Cannot allocate memory > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at > > org.apache.hadoop.util.Shell.runCommand(Shell.java:200) at > > org.apache.hadoop.util.Shell.run(Shell.java:182) at > > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375) > > at org.apache.hadoop.util.Shell.execCommand(Shell.java:461) at > > org.apache.hadoop.util.Shell.execCommand(Shell.java:444) at > > org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:508) > > at > > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499) > > at > > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:319) > > at > > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) > > at > > org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126) > > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:839) > > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) > > at java.security.AccessController.doPrivileged(Native Method) at > > javax.security.auth.Subject.doAs(Subject.java:396) at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) > > at > > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833) > > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807) > > at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) > > at > > org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) > > at > > org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) > > at java.lang.Thread.run(Thread.java:619)Caused by: java.io.IOException: > > java.io.IOException: error=12, Cannot allocate memory at > > java.lang.UNIXProcess.<init>(UNIXProcess.java:148) at > > java.lang.ProcessImpl.start(ProcessImpl.java:65) at > > java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 21 more > > Hadoop logs look fine and the same script on same data set works fine when > > I run it manually. Multiple parallel instances (I tried 3 at same time) > > also works fine when ran manually. > > I am using cloudera build of pig (pig-0.8.1-cdh3u2) and > > hadoop(0.20.2-cdh3u2) (running on remote cluster). Any suggestions? > > Thanks,Rakesh >
