Hello
I am trying to run a pig script which is stored on s3 . The cluster
scenario is as follows:
* Cluster is installed on EC2 using Cloudera Manager 4.5 Automatic
Installation
* Installed version: CDH4
* Script location on - s3:/pigfiles
* running as workflow: -> pig -> script file: s3://panshulpigfiles/
nysesamples/nysesamplesaws/countGroups_daily.pig
*The Pig Script*:
set fs.s3.awsAccessKeyId xxxxxxxxxxxxxxxxxx
set fs.s3.awsSecretAccessKey xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
--load the sample input file
data = load 's3://steamdata/nysedata/NYSE_daily.txt' as
(exchange:chararray, symbol:chararray, date:chararray, open:float,
high:float, low:float, close:float, volume:int, adj_close:float);
--group data by symbols
symbolgrp = group data by symbol;
--count data in every group
symcount = foreach symbolgrp generate group,COUNT(data);
--order the counted list by count
symcountordered = order symcount by $1;
store symcountordered into 's3://steamdata/nyseoutput/daily';
*Error:*
JA008: File does not exist:
/nysesamples/nysesamplesaws/countGroups_daily.pig
The log file is attached.
Please help me, what am I doing wrong. I can assure you that the input
path/file exists on s3 and the AWS key and secret key entered are correct.
Thanking You,
--
Regards,
Ouch Whisper
010101010101
2013-04-07 16:37:23,160 INFO org.apache.oozie.command.wf.ActionStartXCommand:
USER[admin] GROUP[-] TOKEN[] APP[nysesample]
JOB[0000000-130406121414552-oozie-oozi-W]
ACTION[0000000-130406121414552-oozie-oozi-W@:start:] Start action
[0000000-130406121414552-oozie-oozi-W@:start:] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-04-07 16:37:23,172 WARN org.apache.oozie.command.wf.ActionStartXCommand:
USER[admin] GROUP[-] TOKEN[] APP[nysesample]
JOB[0000000-130406121414552-oozie-oozi-W]
ACTION[0000000-130406121414552-oozie-oozi-W@:start:]
[***0000000-130406121414552-oozie-oozi-W@:start:***]Action status=DONE
2013-04-07 16:37:23,172 WARN org.apache.oozie.command.wf.ActionStartXCommand:
USER[admin] GROUP[-] TOKEN[] APP[nysesample]
JOB[0000000-130406121414552-oozie-oozi-W]
ACTION[0000000-130406121414552-oozie-oozi-W@:start:]
[***0000000-130406121414552-oozie-oozi-W@:start:***]Action updated in DB!
2013-04-07 16:37:23,687 INFO org.apache.oozie.command.wf.ActionStartXCommand:
USER[admin] GROUP[-] TOKEN[] APP[nysesample]
JOB[0000000-130406121414552-oozie-oozi-W]
ACTION[0000000-130406121414552-oozie-oozi-W@nyse_countgroups_daliy] Start
action [0000000-130406121414552-oozie-oozi-W@nyse_countgroups_daliy] with
user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-04-07 16:37:27,479 WARN org.apache.oozie.command.wf.ActionStartXCommand:
USER[admin] GROUP[-] TOKEN[] APP[nysesample]
JOB[0000000-130406121414552-oozie-oozi-W]
ACTION[0000000-130406121414552-oozie-oozi-W@nyse_countgroups_daliy] Error
starting action [nyse_countgroups_daliy]. ErrorType [ERROR], ErrorCode [JA008],
Message [JA008: File does not exist:
/nysesamples/nysesamplesaws/countGroups_daily.pig]
org.apache.oozie.action.ActionExecutorException: JA008: File does not exist:
/nysesamples/nysesamplesaws/countGroups_daily.pig
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:760)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:911)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: File does not exist:
/nysesamples/nysesamplesaws/countGroups_daily.pig
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
at
org.apache.hadoop.filecache.DistributedCache.getFileStatus(DistributedCache.java:185)
at
org.apache.hadoop.filecache.TrackerDistributedCacheManager.getFileStatus(TrackerDistributedCacheManager.java:786)
at
org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineTimestamps(TrackerDistributedCacheManager.java:855)
at
org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineTimestampsAndCacheVisibilities(TrackerDistributedCacheManager.java:818)
at
org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:853)
at
org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:743)
at org.apache.hadoop.mapred.JobClient.access$400(JobClient.java:174)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:961)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:946)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:946)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:920)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:745)
... 10 more
2013-04-07 16:37:27,482 WARN org.apache.oozie.command.wf.ActionStartXCommand:
USER[admin] GROUP[-] TOKEN[] APP[nysesample]
JOB[0000000-130406121414552-oozie-oozi-W]
ACTION[0000000-130406121414552-oozie-oozi-W@nyse_countgroups_daliy] Setting
Action Status to [DONE]
2013-04-07 16:37:27,723 INFO org.apache.oozie.command.wf.ActionEndXCommand:
USER[admin] GROUP[-] TOKEN[] APP[nysesample]
JOB[0000000-130406121414552-oozie-oozi-W]
ACTION[0000000-130406121414552-oozie-oozi-W@nyse_countgroups_daliy] ERROR is
considered as FAILED for SLA
2013-04-07 16:37:27,942 INFO org.apache.oozie.command.wf.ActionStartXCommand:
USER[admin] GROUP[-] TOKEN[] APP[nysesample]
JOB[0000000-130406121414552-oozie-oozi-W]
ACTION[0000000-130406121414552-oozie-oozi-W@kill] Start action
[0000000-130406121414552-oozie-oozi-W@kill] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-04-07 16:37:27,943 WARN org.apache.oozie.command.wf.ActionStartXCommand:
USER[admin] GROUP[-] TOKEN[] APP[nysesample]
JOB[0000000-130406121414552-oozie-oozi-W]
ACTION[0000000-130406121414552-oozie-oozi-W@kill]
[***0000000-130406121414552-oozie-oozi-W@kill***]Action status=DONE
2013-04-07 16:37:27,943 WARN org.apache.oozie.command.wf.ActionStartXCommand:
USER[admin] GROUP[-] TOKEN[] APP[nysesample]
JOB[0000000-130406121414552-oozie-oozi-W]
ACTION[0000000-130406121414552-oozie-oozi-W@kill]
[***0000000-130406121414552-oozie-oozi-W@kill***]Action updated in DB!
2013-04-07 16:37:28,468 WARN
org.apache.oozie.command.coord.CoordActionUpdateXCommand: USER[admin] GROUP[-]
TOKEN[] APP[nysesample] JOB[0000000-130406121414552-oozie-oozi-W] ACTION[-]
E1100: Command precondition does not hold before execution, [, coord action is
null], Error Code: E1100
