Hi,
Do you have the directory and file there in HDFS?
For example, do you have this directory ?
hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07
Secondly, are you adding _SUCCESS file in that directory? By default, Oozie
looks for _SUCCESS file in the directory. This file is generally created by MR
job.
If you want to override this please add <done-flag></done-flag> in data set
definition.
For example,
<datasets>
<datasetname="input"frequency="${coord:minutes(1)}"initial-instance="${start}"timezone="UTC">
<uri-template>${nameNode}/user/${coord:user()}/${examplesRoot}/input-data/rawLogs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
<done-flag></done-flag>
</dataset>
</datasets>
More could be found at :
http://oozie.apache.org/docs/4.0.0/CoordinatorFunctionalSpec.html#a5.1._Synchronous_Datasets
Regards,
Mohammad
On Thursday, January 16, 2014 11:55 AM, Chandramohan <[email protected]>
wrote:
Hi All,
I have sample coordinator application where log files needs to be moved on HDFS
from one location to another every minute.
Once the coordination application started, it keeps running while workflow jobs
stays in WAITING state forever.
pig script and only workflow job is tested separately and it's working fine.
Please find attached application and inlined log and help me out.
Log file is as below
2014-01-16 16:37:10,632 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@3]
[0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: Action is in
WAITING state.
2014-01-16 16:37:10,634 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@3]
[0000357-140113142545858-oozie-biad-C@3]::CoordActionInputCheck:: Missing
deps:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS
2014-01-16 16:37:10,636 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@3]
[0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In
checkResolvedUris...
2014-01-16 16:37:10,636 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@3]
[0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In
checkListOfPaths:
hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS
is Missing.
2014-01-16 16:37:10,663 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@3]
[0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck::
File:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS,
Exists? :false
2014-01-16 16:37:10,884 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2]
[0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: Action is in
WAITING state.
2014-01-16 16:37:10,886 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2]
[0000357-140113142545858-oozie-biad-C@2]::CoordActionInputCheck:: Missing
deps:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS
2014-01-16 16:37:10,888 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2]
[0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In
checkResolvedUris...
2014-01-16 16:37:10,888 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2]
[0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In
checkListOfPaths:
hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS
is Missing.
2014-01-16 16:37:10,916 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2]
[0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck::
File:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS,
Exists? :false
2014-01-16 16:37:10,943 INFO PauseTransitService:539 - USER[-] GROUP[-]
Acquired lock for [org.apache.oozie.service.PauseTransitService]
2014-01-16 16:37:10,962 INFO PauseTransitService:539 - USER[-] GROUP[-]
Released lock for [org.apache.oozie.service.PauseTransitService]
2014-01-16 16:37:50,061 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Acquired lock for
[org.apache.oozie.service.StatusTransitService]
2014-01-16 16:37:50,061 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Running coordinator status service from last instance time =
2014-01-16T11:06Z
2014-01-16 16:37:50,079 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Set coordinator job [0000357-140113142545858-oozie-biad-C]
status to RUNNINGWITHERROR' from 'RUNNING'
2014-01-16 16:37:50,079 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Coord job [0000357-140113142545858-oozie-biad-C] Pending set
to FALSE
2014-01-16 16:37:50,083 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Running bundle status service from last instance time =
2014-01-16T11:06Z
2014-01-16 16:37:50,084 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Released lock for
[org.apache.oozie.service.StatusTransitService]
2014-01-16 16:38:10,713 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@3]
[0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: Action is in
WAITING state.
2014-01-16 16:38:10,716 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@3]
[0000357-140113142545858-oozie-biad-C@3]::CoordActionInputCheck:: Missing
deps:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS
2014-01-16 16:38:10,718 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@3]
[0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In
checkResolvedUris...
2014-01-16 16:38:10,718 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@3]
[0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In
checkListOfPaths:
hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS
is Missing.
2014-01-16 16:38:10,745 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@3]
[0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck::
File:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS,
Exists? :false
2014-01-16 16:38:10,963 INFO PauseTransitService:539 - USER[-] GROUP[-]
Acquired lock for [org.apache.oozie.service.PauseTransitService]
2014-01-16 16:38:10,972 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2]
[0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: Action is in
WAITING state.
2014-01-16 16:38:10,975 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2]
[0000357-140113142545858-oozie-biad-C@2]::CoordActionInputCheck:: Missing
deps:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS
2014-01-16 16:38:10,977 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2]
[0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In
checkResolvedUris...
2014-01-16 16:38:10,977 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2]
[0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In
checkListOfPaths:
hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS
is Missing.
2014-01-16 16:38:10,983 INFO PauseTransitService:539 - USER[-] GROUP[-]
Released lock for [org.apache.oozie.service.PauseTransitService]
2014-01-16 16:38:11,001 INFO CoordActionInputCheckXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2]
[0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck::
File:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS,
Exists? :false
2014-01-16 16:38:11,276 INFO CoordActionNotificationXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2] STARTED Coordinator Notification
actionId=0000357-140113142545858-oozie-biad-C@2 : TIMEDOUT
2014-01-16 16:38:11,278 INFO CoordActionNotificationXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2] No Notification URL is defined.
Therefore nothing to notify for job 0000357-140113142545858-oozie-biad-C action
ID 0000357-140113142545858-oozie-biad-C@2
2014-01-16 16:38:11,279 INFO CoordActionNotificationXCommand:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C]
ACTION[0000357-140113142545858-oozie-biad-C@2] ENDED Coordinator Notification
actionId=0000357-140113142545858-oozie-biad-C@2
2014-01-16 16:38:50,085 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Acquired lock for
[org.apache.oozie.service.StatusTransitService]
2014-01-16 16:38:50,085 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Running coordinator status service from last instance time =
2014-01-16T11:07Z
2014-01-16 16:38:50,109 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Set coordinator job [0000357-140113142545858-oozie-biad-C]
status to RUNNINGWITHERROR' from 'RUNNING'
2014-01-16 16:38:50,109 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Coord job [0000357-140113142545858-oozie-biad-C] Pending set
to FALSE
2014-01-16 16:38:50,111 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Running bundle status service from last instance time =
2014-01-16T11:07Z
2014-01-16 16:38:50,112 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Released lock for
[org.apache.oozie.service.StatusTransitService]
Thanks and Regards,
Chandramohan
