I think it's trying to find the staging directory set in your configuration, not finding it, and isn't able to create it. depending on your configs, that could be in different places, but usually it's looking under /tmp/mapred . Check permissions there.
D On Mon, Oct 15, 2012 at 4:35 PM, lei tang <[email protected]> wrote: > Yes. it is there. Tried the full path as well, still not working. > > On Mon, Oct 15, 2012 at 3:45 PM, Prashant Kommireddi > <[email protected]>wrote: > >> Is this file present on your local FS? >> file:///data1/home/ltang01/mf/prep/in/toy.txt >> >> On Mon, Oct 15, 2012 at 3:43 PM, lei tang <[email protected]> wrote: >> >> > Hi, >> > >> > I'm using Pig for my daily job. Pig dose a good job in mapreduce mode in >> > our internal hadoop cluster. But it signals an error whenever I want to >> > run it in local mode. I believe this is due to some permission or config >> > issue. Does anybody know how I can address this problem? Any idea is >> > appreciated. >> > >> > [ltang01@stg-trgt00 in]$ pig -x local >> > 2012-10-11 17:29:25,707 [main] INFO org.apache.pig.Main - Apache Pig >> > version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12 >> > 2012-10-11 17:29:25,708 [main] INFO org.apache.pig.Main - Logging error >> > messages to: /somewhere/in/pig_1350001765703.log >> > 2012-10-11 17:29:26,068 [main] INFO >> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - >> Connecting >> > to hadoop file system at: file:/// >> > grunt> A = load 'toy.txt'; >> > grunt> dump A; >> > 2012-10-11 17:29:36,056 [main] INFO >> > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the >> > script: UNKNOWN >> > 2012-10-11 17:29:36,218 [main] INFO >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - >> > File concatenation threshold: 100 optimistic? false >> > 2012-10-11 17:29:36,249 [main] INFO >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer >> > - MR plan size before optimization: 1 >> > 2012-10-11 17:29:36,250 [main] INFO >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer >> > - MR plan size after optimization: 1 >> > 2012-10-11 17:29:36,382 [main] INFO >> > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added >> > to the job >> > 2012-10-11 17:29:36,403 [main] INFO >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler >> > - mapred.job.reduce.markreset.buffer.percent is not set, set to default >> 0.3 >> > 2012-10-11 17:29:36,445 [main] INFO >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler >> > - Setting up single store job >> > 2012-10-11 17:29:36,486 [main] INFO >> > org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with >> > processName=JobTracker, sessionId= >> > 2012-10-11 17:29:36,493 [main] INFO >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher >> > - 1 map-reduce job(s) waiting for submission. >> > 2012-10-11 17:29:36,995 [main] INFO >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher >> > - 0% complete >> > 2012-10-11 17:29:37,002 [main] INFO >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher >> > - job null has failed! Stop running all dependent jobs >> > 2012-10-11 17:29:37,002 [main] INFO >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher >> > - 100% complete >> > 2012-10-11 17:29:37,007 [main] ERROR >> > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to >> > recreate exception from backend error: ENOENT: No such file or directory >> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) >> > at >> > >> > >> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513) >> > at >> > >> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325) >> > at >> > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) >> > at >> > >> > >> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126) >> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873) >> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAs(Subject.java:416) >> > at >> > >> > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109) >> > at >> > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867) >> > at >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841) >> > at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) >> > at >> > >> > >> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) >> > at >> > org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) >> > at java.lang.Thread.run(Thread.java:679) >> > at >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260) >> > >> > 2012-10-11 17:29:37,007 [main] ERROR >> > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed! >> > 2012-10-11 17:29:37,007 [main] INFO >> > org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats >> > reported below may be incomplete >> > 2012-10-11 17:29:37,009 [main] INFO >> > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: >> > >> > HadoopVersion PigVersion UserId StartedAt FinishedAt >> > Features >> > 0.20.2-dev 0.10.0 ltang01 2012-10-11 17:29:36 2012-10-11 >> > 17:29:37 UNKNOWN >> > >> > Failed! >> > >> > Failed Jobs: >> > JobId Alias Feature Message Outputs >> > N/A A MAP_ONLY Message: ENOENT: No such file or >> directory >> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) >> > at >> > >> > >> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513) >> > at >> > >> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325) >> > at >> > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) >> > at >> > >> > >> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126) >> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873) >> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAs(Subject.java:416) >> > at >> > >> > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109) >> > at >> > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867) >> > at >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841) >> > at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) >> > at >> > >> > >> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) >> > at >> > org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) >> > at java.lang.Thread.run(Thread.java:679) >> > at >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260) >> > file:/tmp/temp-916111882/tmp1833113178, >> > >> > Input(s): >> > Failed to read data from "file:///data1/home/ltang01/mf/prep/in/toy.txt" >> > >> > Output(s): >> > Failed to produce result in "file:/tmp/temp-916111882/tmp1833113178" >> > >> > Job DAG: >> > null >> > >> > >> > 2012-10-11 17:29:37,009 [main] INFO >> > >> > >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher >> > - Failed! >> > 2012-10-11 17:29:37,015 [main] ERROR org.apache.pig.tools.grunt.Grunt - >> > ERROR 1066: Unable to open iterator for alias A >> > Details at logfile: /data1/home/ltang01/mf/prep/in/pig_1350001765703.log >> > >>
