Hi Team,
Appreciate your help in this. Thanks and Regards Jaboy *From:* Jaboy Mathai [mailto:[email protected]] *Sent:* 17 June 2016 15:18 *To:* '[email protected]' *Subject:* Reg: Oozie workflow jobs with worng ID's spawn automatically Hi Team, Please help to resolve this issue: Present parent job is started with job ID: 0000490-160616150539152-oozie-hado-C But the child jobs are getting generated with different ID’s and failing stating wrong credential issue. I am attaching some screenshots for reference. Below mentioned is the portion from oozie.log which shows same errors. 2016-06-17 11:25:15,099 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService] 2016-06-17 11:25:15,099 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time = 2016-06-17T09:24Z 2016-06-17 11:25:15,122 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time = 2016-06-17T09:24Z 2016-06-17 11:25:15,124 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService] 2016-06-17 11:25:28,000 INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService] 2016-06-17 11:25:28,014 INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService] 2016-06-17 11:25:53,548 INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Recover a WAITTING coord action and resubmit CoordActionInputCheckXCommand :0000052-160616150539152-oozie-hado-C@88 2016-06-17 11:25:53,549 INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Recover a WAITTING coord action and resubmit CoordActionInputCheckXCommand :0000054-160616150539152-oozie-hado-C@88 2016-06-17 11:25:53,549 INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Recover a WAITTING coord action and resubmit CoordActionInputCheckXCommand :0000056-160616150539152-oozie-hado-C@108 2016-06-17 11:25:53,549 INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Recover a WAITTING coord action and resubmit CoordActionInputCheckXCommand :0000060-160616150539152-oozie-hado-C@108 2016-06-17 11:25:53,568 WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[VODACOM_MZ_RECHARGE_TRIGGER] JOB[0000089-160615115019988-oozie-hado-W] ACTION[0000089-160615115019988-oozie-hado-W@GetProcessedFileDetailsRecharge] E0810: Job state is not [RUNNING]. Skipping ActionStart Execution, Error Code: E0810 2016-06-17 11:25:53,569 WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[VODACOM_MZ_RECHARGE_TRIGGER] JOB[0001647-160527134104689-oozie-root-W] ACTION[0001647-160527134104689-oozie-root-W@GetProcessedFileDetailsRecharge] E0810: Job state is not [RUNNING]. Skipping ActionStart Execution, Error Code: E0810 2016-06-17 11:25:53,575 WARN KillXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[VODACOM_MZ_MOC_USAGE_TRIGGER] JOB[0017024-160527134104689-oozie-root-W] ACTION[-] E0725: Workflow instance can not be killed, 0017024-160527134104689-oozie-root-W, Error Code: E0725 2016-06-17 11:25:53,575 ERROR KillXCommand:536 - USER[hadoop] GROUP[-] TOKEN[] APP[VODACOM_MZ_MOC_USAGE_TRIGGER] JOB[0017024-160527134104689-oozie-root-W] ACTION[-] Exception, java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:317) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:992) at org.apache.oozie.util.MemoryLocks$LockToken.release(MemoryLocks.java:59) at org.apache.oozie.command.XCommand.releaseLock(XCommand.java:222) at org.apache.oozie.command.XCommand.call(XCommand.java:295) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2016-06-17 11:25:53,575 WARN CallableQueueService$CompositeCallable:542 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] exception callable [kill], E0607: Other error in operation [java.lang.IllegalMonitorStateException], {1} org.apache.oozie.command.CommandException: E0607: Other error in operation [java.lang.IllegalMonitorStateException], {1} at org.apache.oozie.command.XCommand.call(XCommand.java:317) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:317) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:992) at org.apache.oozie.util.MemoryLocks$LockToken.release(MemoryLocks.java:59) at org.apache.oozie.command.XCommand.releaseLock(XCommand.java:222) at org.apache.oozie.command.XCommand.call(XCommand.java:295) ... 6 more Here the parent job for ‘0017024-160527134104689-oozie-root-W’ was killed long back. Please note: Here no service has been restarted, the workflow with different job ID’s are getting spawned automatically. Thanks & Regards, *Jaboy Mathai* *Tech Lead, TAC-Hadoop* *Flytxt* Skype ID: jaboymathai *E-mail: **[email protected]* <[email protected]> *Mob: +91-7025244727*
