[
https://issues.apache.org/jira/browse/OOZIE-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shwetha G S updated OOZIE-1622:
-------------------------------
Description:
We saw a weird instance where multiple coords were created for same bundle id
when the bundle was supposed to have just 1 coordinator. Here are the oozie
logs:
{noformat}
2013-11-19 09:09:46,473 INFO BundleStartXCommand:539 - USER[fetl] GROUP[-]
TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
Bundle 0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in :
RUNNING
2013-11-19 09:09:46,473 WARN BundleStartXCommand:542 - USER[fetl] GROUP[-]
TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
E1100: Command precondition does not hold before execution, [Bundle
0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in :
RUNNING], Error Code: E1100
2013-11-19 09:09:46,473 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED
Coordinator Submit
2013-11-19 09:09:46,483 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
configDefault Doesn't exist
2013-11-19 09:09:46,515 INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-]
TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-]
ENDED Coordinator Submit jobId=0484437-131016085136608-oozie-oozi-C
2013-11-19 09:09:46,529 INFO BundleStatusUpdateXCommand:539 - USER[fetl]
GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C]
ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app
name>] from prev status [PREP] to current coord status [PREP], and new bundle
action pending [0]
2013-11-19 09:09:46,535 INFO CoordMaterializeTransitionXCommand:539 -
USER[fetl] GROUP[-] TOKEN[] APP[<app name>]
JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] materialize actions for
tz=Coordinated Universal Time,
2013-11-19 09:09:54,590 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status
to 'RUNNING' from 'RUNNING'
2013-11-19 09:09:54,590 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set
to FALSE
2013-11-19 09:10:16,326 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:12:57,246 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status
to 'SUSPENDED' from 'SUSPENDED'
2013-11-19 09:12:57,246 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set
to TRUE
2013-11-19 09:13:16,410 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:16:16,446 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:17:00,913 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status
to 'RUNNING' from 'RUNNING'
2013-11-19 09:17:00,914 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set
to TRUE
2013-11-19 09:19:16,490 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:22:16,907 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:25:17,086 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:26:49,373 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED
Coordinator Submit
2013-11-19 09:26:49,383 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
configDefault Doesn't exist
2013-11-19 09:26:49,438 INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-]
TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C] ACTION[-]
ENDED Coordinator Submit jobId=0484598-131016085136608-oozie-oozi-C
2013-11-19 09:26:49,445 INFO BundleStatusUpdateXCommand:539 - USER[fetl]
GROUP[-] TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C]
ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app
name>] from prev status [PREP] to current coord status [PREP], and new bundle
action pending [1]
{noformat}
was:
We saw a weird instance where multiple coords were created for same bundle id
when the bundle was supposed to have just 1 coordinator. Here are the oozie
logs:
{noformat}
2013-11-19 09:09:46,473 INFO BundleStartXCommand:539 - USER[fetl] GROUP[-]
TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
Bundle 0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in :
RUNNING
2013-11-19 09:09:46,473 WARN BundleStartXCommand:542 - USER[fetl] GROUP[-]
TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
E1100: Command precondition does not hold before execution, [Bundle
0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in :
RUNNING], Error Code: E1100
2013-11-19 09:09:46,473 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED
Coordinator Submit
2013-11-19 09:09:46,483 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
configDefault Doesn't exist
2013-11-19 09:09:46,515 INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-]
TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-]
ENDED Coordinator Submit jobId=0484437-131016085136608-oozie-oozi-C
2013-11-19 09:09:46,529 INFO BundleStatusUpdateXCommand:539 - USER[fetl]
GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C]
ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app
name>] from prev status [PREP] to current coord status [PREP], and new bundle
action pending [0]
2013-11-19 09:09:46,535 INFO CoordMaterializeTransitionXCommand:539 -
USER[fetl] GROUP[-] TOKEN[] APP[<app name>]
JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] materialize actions for
tz=Coordinated Universal Time,
2013-11-19 09:09:54,590 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status
to 'RUNNING' from 'RUNNING'
2013-11-19 09:09:54,590 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set
to FALSE
2013-11-19 09:10:16,326 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:12:57,246 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status
to 'SUSPENDED' from 'SUSPENDED'
2013-11-19 09:12:57,246 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set
to TRUE
2013-11-19 09:13:16,410 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:16:16,446 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:17:00,913 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status
to 'RUNNING' from 'RUNNING'
2013-11-19 09:17:00,914 INFO StatusTransitService$StatusTransitRunnable:539 -
USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set
to TRUE
2013-11-19 09:19:16,490 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:22:16,907 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:25:17,086 INFO
CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
:0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
2013-11-19 09:26:49,373 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED
Coordinator Submit
2013-11-19 09:26:49,383 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
configDefault Doesn't exist
hdfs://emerald-nn.grid.lhr1.inmobi.com:8020/projects/ivory/staging/falcon/workflows/process/FETL2-click-enhancer/82440d66a679cb223df8380367624281/DEFAULT/coord-config-default.xml
2013-11-19 09:26:49,438 INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-]
TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C] ACTION[-]
ENDED Coordinator Submit jobId=0484598-131016085136608-oozie-oozi-C
2013-11-19 09:26:49,445 INFO BundleStatusUpdateXCommand:539 - USER[fetl]
GROUP[-] TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C]
ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app
name>] from prev status [PREP] to current coord status [PREP], and new bundle
action pending [1]
{noformat}
> Multiple CoordSubmit for same bundle
> ------------------------------------
>
> Key: OOZIE-1622
> URL: https://issues.apache.org/jira/browse/OOZIE-1622
> Project: Oozie
> Issue Type: Bug
> Reporter: Shwetha G S
>
> We saw a weird instance where multiple coords were created for same bundle id
> when the bundle was supposed to have just 1 coordinator. Here are the oozie
> logs:
> {noformat}
> 2013-11-19 09:09:46,473 INFO BundleStartXCommand:539 - USER[fetl] GROUP[-]
> TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
> Bundle 0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in :
> RUNNING
> 2013-11-19 09:09:46,473 WARN BundleStartXCommand:542 - USER[fetl] GROUP[-]
> TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
> E1100: Command precondition does not hold before execution, [Bundle
> 0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in :
> RUNNING], Error Code: E1100
> 2013-11-19 09:09:46,473 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
> TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED
> Coordinator Submit
> 2013-11-19 09:09:46,483 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
> TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
> configDefault Doesn't exist
> 2013-11-19 09:09:46,515 INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-]
> TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-]
> ENDED Coordinator Submit jobId=0484437-131016085136608-oozie-oozi-C
> 2013-11-19 09:09:46,529 INFO BundleStatusUpdateXCommand:539 - USER[fetl]
> GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C]
> ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app
> name>] from prev status [PREP] to current coord status [PREP], and new bundle
> action pending [0]
> 2013-11-19 09:09:46,535 INFO CoordMaterializeTransitionXCommand:539 -
> USER[fetl] GROUP[-] TOKEN[] APP[<app name>]
> JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] materialize actions for
> tz=Coordinated Universal Time,
> 2013-11-19 09:09:54,590 INFO StatusTransitService$StatusTransitRunnable:539
> - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B]
> status to 'RUNNING' from 'RUNNING'
> 2013-11-19 09:09:54,590 INFO StatusTransitService$StatusTransitRunnable:539
> - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending
> set to FALSE
> 2013-11-19 09:10:16,326 INFO
> CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
> GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
> :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:12:57,246 INFO StatusTransitService$StatusTransitRunnable:539
> - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B]
> status to 'SUSPENDED' from 'SUSPENDED'
> 2013-11-19 09:12:57,246 INFO StatusTransitService$StatusTransitRunnable:539
> - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending
> set to TRUE
> 2013-11-19 09:13:16,410 INFO
> CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
> GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
> :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:16:16,446 INFO
> CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
> GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
> :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:17:00,913 INFO StatusTransitService$StatusTransitRunnable:539
> - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B]
> status to 'RUNNING' from 'RUNNING'
> 2013-11-19 09:17:00,914 INFO StatusTransitService$StatusTransitRunnable:539
> - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending
> set to TRUE
> 2013-11-19 09:19:16,490 INFO
> CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
> GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
> :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:22:16,907 INFO
> CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
> GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
> :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:25:17,086 INFO
> CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-]
> GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job
> :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:26:49,373 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
> TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED
> Coordinator Submit
> 2013-11-19 09:26:49,383 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-]
> TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-]
> configDefault Doesn't exist
> 2013-11-19 09:26:49,438 INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-]
> TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C] ACTION[-]
> ENDED Coordinator Submit jobId=0484598-131016085136608-oozie-oozi-C
> 2013-11-19 09:26:49,445 INFO BundleStatusUpdateXCommand:539 - USER[fetl]
> GROUP[-] TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C]
> ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app
> name>] from prev status [PREP] to current coord status [PREP], and new bundle
> action pending [1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
