Hi, What version of Oozie are you using? Do you have any coordinator job running? You could bumped up the value for this two properties: oozie.service.CallableQueueService.threads = 150 oozie.service.CallableQueueService.callable.concurrency = 60 Please let us know how does it go. Regards, Mohammad
________________________________ From: Siva Subramanian <[email protected]> To: [email protected] Sent: Thursday, February 21, 2013 11:41 PM Subject: Callable Queue is Full Hey, We have a complex workflow with about lots of parellel actions (around 300 - a mix of MR, FS & java actions). Frequently we notice that, even though the MR job corresponding to the action (this mostly happens with Java actions) finishes in less than 10 seconds, the corresponding oozie action takes more than 10 minutes to finish. In the oozie log, I see these warnings: 2013-02-21 22:33:08,259 INFO CallbackServlet:525 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000229-130215175004882-oozie-oozi-W] ACTION[0000229-130215175004882-oozie-oozi-W@move_data] callback for action [0000229-130215175004882-oozie-oozi-W@move_data] 2013-02-21 22:33:08,259 WARN CallableQueueService:528 - USER[-] GROUP[-] queue if full, ignoring queuing for [callback,2] 2013-02-21 22:33:08,259 WARN CallableQueueService:528 - USER[-] GROUP[-] Could not queue callable 2013-02-21 22:33:08,260 WARN DagEngine:528 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000229-130215175004882-oozie-oozi-W] ACTION[0000229-130215175004882-oozie-oozi-W@move_data] queue is full or system is in SAFEMODE, ignoring callback This particular job was started at 22:32:44 and the MR job ended in 10 seconds. We have the following config: oozie.service.CallableQueueService.queue.size = 10000 oozie.service.CallableQueueService.threads = 10 oozie.service.CallableQueueService.callable.concurrency = 3 What parameter(s) can we tune to scale? Thanks in advance, Siva
