I checked the hadoop config and found that the staging dir specified does not exist in local path (only in hdfs) and neither do I have write access. So I changed the staging dir as below, but encounter some weird error message:
[ltang01@stg-trgt00 ~]$ pig -Dmapreduce.jobtracker.staging.root.dir=~/tmp/staging -x local 2012-10-19 11:06:49,624 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12 2012-10-19 11:06:49,625 [main] INFO org.apache.pig.Main - Logging error messages to: /data1/home/ltang01/pig_1350670009621.log 2012-10-19 11:06:50,009 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:/// grunt> A = load 'toy.txt'; grunt> dump A; 2012-10-19 11:06:53,485 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN 2012-10-19 11:06:53,660 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false 2012-10-19 11:06:53,693 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 2012-10-19 11:06:53,693 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 2012-10-19 11:06:53,886 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job 2012-10-19 11:06:53,908 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2012-10-19 11:06:53,953 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2012-10-19 11:06:53,996 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId= 2012-10-19 11:06:54,000 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2012-10-19 11:06:54,125 [Thread-5] WARN org.apache.hadoop.mapred.JobClient - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 2012-10-19 11:06:54,207 [Thread-5] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 2012-10-19 11:06:54,207 [Thread-5] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 2012-10-19 11:06:54,222 [Thread-5] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 2012-10-19 11:06:54,501 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2012-10-19 11:06:55,005 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001 2012-10-19 11:06:55,268 [Thread-6] INFO org.apache.hadoop.mapreduce.util.ProcessTree - setsid exited with exit code 0 2012-10-19 11:06:55,824 [Thread-6] WARN org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree - /proc/<pid>/status does not have information about swap space used(VmSwap). Can not track swap usage of a task. 2012-10-19 11:06:55,825 [Thread-6] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@52cab854 2012-10-19 11:06:55,853 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/data1/home/ltang01/toy.txt:0+8 2012-10-19 11:06:55,914 [Thread-6] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting 2012-10-19 11:06:55,921 [Thread-6] INFO org.apache.hadoop.mapred.LocalJobRunner - 2012-10-19 11:06:55,922 [Thread-6] INFO org.apache.hadoop.mapred.Task - Task attempt_local_0001_m_000000_0 is allowed to commit now 2012-10-19 11:06:55,927 [Thread-6] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000000_0' to file:/tmp/temp487011820/tmp-1248641571 2012-10-19 11:06:55,927 [Thread-6] INFO org.apache.hadoop.mapred.LocalJobRunner - 2012-10-19 11:06:55,928 [Thread-6] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000000_0' done. 2012-10-19 11:06:55,929 [Thread-6] WARN org.apache.hadoop.mapred.FileOutputCommitter - Output path is null in cleanup 2012-10-19 11:06:56,380 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: localhost/127.0.0.1:9001. Already tried 0 time(s). 2012-10-19 11:06:57,382 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: localhost/127.0.0.1:9001. Already tried 1 time(s). 2012-10-19 11:06:58,383 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: localhost/127.0.0.1:9001. Already tried 2 time(s). 2012-10-19 11:06:59,385 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: localhost/127.0.0.1:9001. Already tried 3 time(s). It keeps trying to connect to the local server. I have no idea what this is about. Can you help? - Lei On Wed, Oct 17, 2012 at 9:05 PM, Dmitriy Ryaboy <[email protected]> wrote: > I think it's trying to find the staging directory set in your > configuration, not finding it, and isn't able to create it. > depending on your configs, that could be in different places, but > usually it's looking under /tmp/mapred . Check permissions there. > > D > > On Mon, Oct 15, 2012 at 4:35 PM, lei tang <[email protected]> wrote: > > Yes. it is there. Tried the full path as well, still not working. > > > > On Mon, Oct 15, 2012 at 3:45 PM, Prashant Kommireddi < > [email protected]>wrote: > > > >> Is this file present on your local FS? > >> file:///data1/home/ltang01/mf/prep/in/toy.txt > >> > >> On Mon, Oct 15, 2012 at 3:43 PM, lei tang <[email protected]> wrote: > >> > >> > Hi, > >> > > >> > I'm using Pig for my daily job. Pig dose a good job in mapreduce mode > in > >> > our internal hadoop cluster. But it signals an error whenever I want > to > >> > run it in local mode. I believe this is due to some permission or > config > >> > issue. Does anybody know how I can address this problem? Any idea is > >> > appreciated. > >> > > >> > [ltang01@stg-trgt00 in]$ pig -x local > >> > 2012-10-11 17:29:25,707 [main] INFO org.apache.pig.Main - Apache Pig > >> > version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12 > >> > 2012-10-11 17:29:25,708 [main] INFO org.apache.pig.Main - Logging > error > >> > messages to: /somewhere/in/pig_1350001765703.log > >> > 2012-10-11 17:29:26,068 [main] INFO > >> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > >> Connecting > >> > to hadoop file system at: file:/// > >> > grunt> A = load 'toy.txt'; > >> > grunt> dump A; > >> > 2012-10-11 17:29:36,056 [main] INFO > >> > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > >> > script: UNKNOWN > >> > 2012-10-11 17:29:36,218 [main] INFO > >> > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > >> > File concatenation threshold: 100 optimistic? false > >> > 2012-10-11 17:29:36,249 [main] INFO > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > >> > - MR plan size before optimization: 1 > >> > 2012-10-11 17:29:36,250 [main] INFO > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > >> > - MR plan size after optimization: 1 > >> > 2012-10-11 17:29:36,382 [main] INFO > >> > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are > added > >> > to the job > >> > 2012-10-11 17:29:36,403 [main] INFO > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > >> > - mapred.job.reduce.markreset.buffer.percent is not set, set to > default > >> 0.3 > >> > 2012-10-11 17:29:36,445 [main] INFO > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > >> > - Setting up single store job > >> > 2012-10-11 17:29:36,486 [main] INFO > >> > org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics > with > >> > processName=JobTracker, sessionId= > >> > 2012-10-11 17:29:36,493 [main] INFO > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > >> > - 1 map-reduce job(s) waiting for submission. > >> > 2012-10-11 17:29:36,995 [main] INFO > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > >> > - 0% complete > >> > 2012-10-11 17:29:37,002 [main] INFO > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > >> > - job null has failed! Stop running all dependent jobs > >> > 2012-10-11 17:29:37,002 [main] INFO > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > >> > - 100% complete > >> > 2012-10-11 17:29:37,007 [main] ERROR > >> > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to > >> > recreate exception from backend error: ENOENT: No such file or > directory > >> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) > >> > at > >> > > >> > > >> > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513) > >> > at > >> > > >> > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325) > >> > at > >> > > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) > >> > at > >> > > >> > > >> > org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126) > >> > at > org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873) > >> > at > org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867) > >> > at java.security.AccessController.doPrivileged(Native Method) > >> > at javax.security.auth.Subject.doAs(Subject.java:416) > >> > at > >> > > >> > > >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109) > >> > at > >> > > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867) > >> > at > >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841) > >> > at > org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) > >> > at > >> > > >> > > >> > org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) > >> > at > >> > > org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) > >> > at java.lang.Thread.run(Thread.java:679) > >> > at > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260) > >> > > >> > 2012-10-11 17:29:37,007 [main] ERROR > >> > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) > failed! > >> > 2012-10-11 17:29:37,007 [main] INFO > >> > org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. > Stats > >> > reported below may be incomplete > >> > 2012-10-11 17:29:37,009 [main] INFO > >> > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: > >> > > >> > HadoopVersion PigVersion UserId StartedAt FinishedAt > >> > Features > >> > 0.20.2-dev 0.10.0 ltang01 2012-10-11 17:29:36 2012-10-11 > >> > 17:29:37 UNKNOWN > >> > > >> > Failed! > >> > > >> > Failed Jobs: > >> > JobId Alias Feature Message Outputs > >> > N/A A MAP_ONLY Message: ENOENT: No such file or > >> directory > >> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) > >> > at > >> > > >> > > >> > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513) > >> > at > >> > > >> > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325) > >> > at > >> > > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) > >> > at > >> > > >> > > >> > org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126) > >> > at > org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873) > >> > at > org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867) > >> > at java.security.AccessController.doPrivileged(Native Method) > >> > at javax.security.auth.Subject.doAs(Subject.java:416) > >> > at > >> > > >> > > >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109) > >> > at > >> > > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867) > >> > at > >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841) > >> > at > org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) > >> > at > >> > > >> > > >> > org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) > >> > at > >> > > org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) > >> > at java.lang.Thread.run(Thread.java:679) > >> > at > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260) > >> > file:/tmp/temp-916111882/tmp1833113178, > >> > > >> > Input(s): > >> > Failed to read data from > "file:///data1/home/ltang01/mf/prep/in/toy.txt" > >> > > >> > Output(s): > >> > Failed to produce result in "file:/tmp/temp-916111882/tmp1833113178" > >> > > >> > Job DAG: > >> > null > >> > > >> > > >> > 2012-10-11 17:29:37,009 [main] INFO > >> > > >> > > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > >> > - Failed! > >> > 2012-10-11 17:29:37,015 [main] ERROR org.apache.pig.tools.grunt.Grunt > - > >> > ERROR 1066: Unable to open iterator for alias A > >> > Details at logfile: > /data1/home/ltang01/mf/prep/in/pig_1350001765703.log > >> > > >> >
