Hi Team,
Please help to resolve this issue:
Present parent job is started with job ID:
0000490-160616150539152-oozie-hado-C
But the child jobs are getting generated with different ID’s and failing
stating wrong credential issue.
I am attaching some screenshots for reference. Below mentioned is the
portion from oozie.log which shows same errors.
2016-06-17 11:25:15,099 INFO
StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired
lock for [org.apache.oozie.service.StatusTransitService]
2016-06-17 11:25:15,099 INFO
StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running
coordinator status service from last instance time = 2016-06-17T09:24Z
2016-06-17 11:25:15,122 INFO
StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running
bundle status service from last instance time = 2016-06-17T09:24Z
2016-06-17 11:25:15,124 INFO
StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released
lock for [org.apache.oozie.service.StatusTransitService]
2016-06-17 11:25:28,000 INFO PauseTransitService:539 - USER[-] GROUP[-]
Acquired lock for [org.apache.oozie.service.PauseTransitService]
2016-06-17 11:25:28,014 INFO PauseTransitService:539 - USER[-] GROUP[-]
Released lock for [org.apache.oozie.service.PauseTransitService]
2016-06-17 11:25:53,548 INFO RecoveryService$RecoveryRunnable:539 -
USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Recover a WAITTING coord
action and resubmit CoordActionInputCheckXCommand
:0000052-160616150539152-oozie-hado-C@88
2016-06-17 11:25:53,549 INFO RecoveryService$RecoveryRunnable:539 -
USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Recover a WAITTING coord
action and resubmit CoordActionInputCheckXCommand
:0000054-160616150539152-oozie-hado-C@88
2016-06-17 11:25:53,549 INFO RecoveryService$RecoveryRunnable:539 -
USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Recover a WAITTING coord
action and resubmit CoordActionInputCheckXCommand
:0000056-160616150539152-oozie-hado-C@108
2016-06-17 11:25:53,549 INFO RecoveryService$RecoveryRunnable:539 -
USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Recover a WAITTING coord
action and resubmit CoordActionInputCheckXCommand
:0000060-160616150539152-oozie-hado-C@108
2016-06-17 11:25:53,568 WARN ActionStartXCommand:542 - USER[hadoop]
GROUP[-] TOKEN[] APP[VODACOM_MZ_RECHARGE_TRIGGER]
JOB[0000089-160615115019988-oozie-hado-W]
ACTION[0000089-160615115019988-oozie-hado-W@GetProcessedFileDetailsRecharge]
E0810: Job state is not [RUNNING]. Skipping ActionStart Execution, Error
Code: E0810
2016-06-17 11:25:53,569 WARN ActionStartXCommand:542 - USER[hadoop]
GROUP[-] TOKEN[] APP[VODACOM_MZ_RECHARGE_TRIGGER]
JOB[0001647-160527134104689-oozie-root-W]
ACTION[0001647-160527134104689-oozie-root-W@GetProcessedFileDetailsRecharge]
E0810: Job state is not [RUNNING]. Skipping ActionStart Execution, Error
Code: E0810
2016-06-17 11:25:53,575 WARN KillXCommand:542 - USER[hadoop] GROUP[-]
TOKEN[] APP[VODACOM_MZ_MOC_USAGE_TRIGGER]
JOB[0017024-160527134104689-oozie-root-W] ACTION[-] E0725: Workflow
instance can not be killed, 0017024-160527134104689-oozie-root-W, Error
Code: E0725
2016-06-17 11:25:53,575 ERROR KillXCommand:536 - USER[hadoop] GROUP[-]
TOKEN[] APP[VODACOM_MZ_MOC_USAGE_TRIGGER]
JOB[0017024-160527134104689-oozie-root-W] ACTION[-] Exception,
java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:317)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:992)
at
org.apache.oozie.util.MemoryLocks$LockToken.release(MemoryLocks.java:59)
at org.apache.oozie.command.XCommand.releaseLock(XCommand.java:222)
at org.apache.oozie.command.XCommand.call(XCommand.java:295)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2016-06-17 11:25:53,575 WARN CallableQueueService$CompositeCallable:542 -
USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] exception callable
[kill], E0607: Other error in operation
[java.lang.IllegalMonitorStateException], {1}
org.apache.oozie.command.CommandException: E0607: Other error in operation
[java.lang.IllegalMonitorStateException], {1}
at org.apache.oozie.command.XCommand.call(XCommand.java:317)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:317)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:992)
at
org.apache.oozie.util.MemoryLocks$LockToken.release(MemoryLocks.java:59)
at org.apache.oozie.command.XCommand.releaseLock(XCommand.java:222)
at org.apache.oozie.command.XCommand.call(XCommand.java:295)
... 6 more
Here the parent job for ‘0017024-160527134104689-oozie-root-W’ was killed
long back.
Please note: Here no service has been restarted, the workflow with
different job ID’s are getting spawned automatically.
Thanks & Regards,
*Jaboy Mathai*
*Tech Lead, TAC-Hadoop*
*Flytxt*
Skype ID: jaboymathai
*E-mail: **[email protected]* <[email protected]>
*Mob: +91-7025244727*
