I added don-flag as log.txt and kept this file in each folder on HDFS path
with configuration as
<coordinator-app name="coord-pig" frequency="${coord:minutes(1)}"
start="${start}" end="${end}" timezone="UTC"
xmlns="uri:oozie:coordinator:0.2">
<controls>
<concurrency>1</concurrency>
<execution>FIFO</execution>
<throttle>3</throttle>
</controls>
<datasets>
<dataset name="input" frequency="${coord:minutes(1)}"
initial-instance="${start}" timezone="UTC">
<uri-template>${nameNode}/user/${coord:user()}/${examplesRoot}/input-data/rawLogs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}/log.txt</uri-template>
</dataset>
</datasets>
<input-events>
<data-in name="inputEvent" dataset="input">
<instance>${coord:current(0)}</instance>
</data-in>
</input-events>
<action>
<workflow>
<app-path>${nameNode}/user/${coord:user()}/${examplesRoot}/apps/cordpig</app-path>
<configuration>
<property>
<name>jobTracker</name>
<value>${jobTracker}</value>
</property>
<property>
<name>nameNode</name>
<value>${nameNode}</value>
</property>
<property>
<name>queueName</name>
<value>${queueName}</value>
</property>
<property>
<name>inputData</name>
<value>${coord:dataIn('inputEvent')}</value>
</property>
<property>
<name>outputData</name>
<value>${nameNode}/user/${coord:user()}/${examplesRoot}/output-data/pig</value>
</property>
</configuration>
</workflow>
</action>
</coordinator-app>
Job .properties as :
nameNode=hdfs://indmtx260.corp.amdocs.com:9000
jobTracker=indmtx260.corp.amdocs.com:9001
queueName=default
examplesRoot=examples
oozie.use.system.libpath=true
oozie.coord.application.path=${nameNode}/user/${user.name
}/${examplesRoot}/apps/cordpig/coordinator.xml
#oozie.wf.application.path=${nameNode}/user/${user.name
}/${examplesRoot}/apps/cordpig
start=2014-01-27T02:10Z
end=2014-02-15T13:57Z
Now oozie started the job and all the worklfow jobs are in waiting state,
they stay in waiting. I an not sure why this data-event is not working for
me.
Please find attached snapshot from oozie portal and HDFS.
-Chandramohan
On Mon, Jan 27, 2014 at 7:24 PM, Chandramohan <[email protected]>wrote:
> Yes I do have the directory available on HDFS with path hdfs://
> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07
> .
> There are *no *_SUCCESS files in any of the folder. Do we need to have a
> _SUCCESS file, is there any way not to have it.
>
>
> On Thu, Jan 16, 2014 at 4:40 PM, Chandramohan <[email protected]>wrote:
>
>> Hi All,
>>
>> I have sample coordinator application where log files needs to be moved
>> on HDFS from one location to another every minute.
>>
>> Once the coordination application started, it keeps running while
>> workflow jobs stays in WAITING state forever.
>>
>> pig script and only workflow job is tested separately and it's working
>> fine.
>>
>> Please find attached application and inlined log and help me out.
>>
>> Log file is as below
>> 2014-01-16 16:37:10,632 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@3]
>> [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: Action is
>> in WAITING state.
>> 2014-01-16 16:37:10,634 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@3]
>> [0000357-140113142545858-oozie-biad-C@3]::CoordActionInputCheck::
>> Missing deps:hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS
>> 2014-01-16 16:37:10,636 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@3]
>> [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In
>> checkResolvedUris...
>> 2014-01-16 16:37:10,636 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@3]
>> [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In
>> checkListOfPaths: hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESSis
>> Missing.
>> 2014-01-16 16:37:10,663 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@3]
>> [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: File:hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS,
>> Exists? :false
>> 2014-01-16 16:37:10,884 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2]
>> [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: Action is
>> in WAITING state.
>> 2014-01-16 16:37:10,886 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2]
>> [0000357-140113142545858-oozie-biad-C@2]::CoordActionInputCheck::
>> Missing deps:hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS
>> 2014-01-16 16:37:10,888 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2]
>> [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In
>> checkResolvedUris...
>> 2014-01-16 16:37:10,888 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2]
>> [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In
>> checkListOfPaths: hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESSis
>> Missing.
>> 2014-01-16 16:37:10,916 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2]
>> [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: File:hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS,
>> Exists? :false
>> 2014-01-16 16:37:10,943 INFO PauseTransitService:539 - USER[-] GROUP[-]
>> Acquired lock for [org.apache.oozie.service.PauseTransitService]
>> 2014-01-16 16:37:10,962 INFO PauseTransitService:539 - USER[-] GROUP[-]
>> Released lock for [org.apache.oozie.service.PauseTransitService]
>> 2014-01-16 16:37:50,061 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired
>> lock for [org.apache.oozie.service.StatusTransitService]
>> 2014-01-16 16:37:50,061 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running
>> coordinator status service from last instance time = 2014-01-16T11:06Z
>> 2014-01-16 16:37:50,079 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set
>> coordinator job [0000357-140113142545858-oozie-biad-C] status to
>> RUNNINGWITHERROR' from 'RUNNING'
>> 2014-01-16 16:37:50,079 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Coord job
>> [0000357-140113142545858-oozie-biad-C] Pending set to FALSE
>> 2014-01-16 16:37:50,083 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running
>> bundle status service from last instance time = 2014-01-16T11:06Z
>> 2014-01-16 16:37:50,084 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released
>> lock for [org.apache.oozie.service.StatusTransitService]
>> 2014-01-16 16:38:10,713 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@3]
>> [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: Action is
>> in WAITING state.
>> 2014-01-16 16:38:10,716 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@3]
>> [0000357-140113142545858-oozie-biad-C@3]::CoordActionInputCheck::
>> Missing deps:hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS
>> 2014-01-16 16:38:10,718 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@3]
>> [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In
>> checkResolvedUris...
>> 2014-01-16 16:38:10,718 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@3]
>> [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In
>> checkListOfPaths: hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESSis
>> Missing.
>> 2014-01-16 16:38:10,745 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@3]
>> [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: File:hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS,
>> Exists? :false
>> 2014-01-16 16:38:10,963 INFO PauseTransitService:539 - USER[-] GROUP[-]
>> Acquired lock for [org.apache.oozie.service.PauseTransitService]
>> 2014-01-16 16:38:10,972 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2]
>> [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: Action is
>> in WAITING state.
>> 2014-01-16 16:38:10,975 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2]
>> [0000357-140113142545858-oozie-biad-C@2]::CoordActionInputCheck::
>> Missing deps:hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS
>> 2014-01-16 16:38:10,977 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2]
>> [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In
>> checkResolvedUris...
>> 2014-01-16 16:38:10,977 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2]
>> [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In
>> checkListOfPaths: hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESSis
>> Missing.
>> 2014-01-16 16:38:10,983 INFO PauseTransitService:539 - USER[-] GROUP[-]
>> Released lock for [org.apache.oozie.service.PauseTransitService]
>> 2014-01-16 16:38:11,001 INFO CoordActionInputCheckXCommand:539 - USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2]
>> [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: File:hdfs://
>> indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS,
>> Exists? :false
>> 2014-01-16 16:38:11,276 INFO CoordActionNotificationXCommand:539 -
>> USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2] STARTED Coordinator
>> Notification actionId=0000357-140113142545858-oozie-biad-C@2 : TIMEDOUT
>> 2014-01-16 16:38:11,278 INFO CoordActionNotificationXCommand:539 -
>> USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2] No Notification URL is
>> defined. Therefore nothing to notify for job
>> 0000357-140113142545858-oozie-biad-C action ID
>> 0000357-140113142545858-oozie-biad-C@2
>> 2014-01-16 16:38:11,279 INFO CoordActionNotificationXCommand:539 -
>> USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
>> ACTION[0000357-140113142545858-oozie-biad-C@2] ENDED Coordinator
>> Notification actionId=0000357-140113142545858-oozie-biad-C@2
>> 2014-01-16 16:38:50,085 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired
>> lock for [org.apache.oozie.service.StatusTransitService]
>> 2014-01-16 16:38:50,085 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running
>> coordinator status service from last instance time = 2014-01-16T11:07Z
>> 2014-01-16 16:38:50,109 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set
>> coordinator job [0000357-140113142545858-oozie-biad-C] status to
>> RUNNINGWITHERROR' from 'RUNNING'
>> 2014-01-16 16:38:50,109 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Coord job
>> [0000357-140113142545858-oozie-biad-C] Pending set to FALSE
>> 2014-01-16 16:38:50,111 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running
>> bundle status service from last instance time = 2014-01-16T11:07Z
>> 2014-01-16 16:38:50,112 INFO
>> StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released
>> lock for [org.apache.oozie.service.StatusTransitService]
>>
>>
>>
>> Thanks and Regards,
>> Chandramohan
>>
>>
>
