> On 2011-12-09 21:59:30, Alejandro Abdelnur wrote: > > I'm not understanding how this patch handles interruptions by just adding a > > new method to command. > > > > From the description of the problem, my logical solution would be: > > > > * kill/suspend/modify commands would be only commands with highest priority > > (either by bumping everybody else one level or create a higher priority) > > (logically this is the same as interruptions but using the current priority > > logic which we know it works fine) > > * all commands should check in their precondition that the job is in the > > expected state (job status and that the command in the case of coordinator > > jobs is for an action before the end time. The later is the case when the > > endtime has been shorten) > > > >
I attached the design document to the Jira [I added it to the Jira couple of weeks ago], giving more clarity about the problem and the proposed solution The interrupt is mainly handled at the interrupt map, which will hold all the interrupt driven commands and give them higher priority over the other commands that belong to the same lock key. it has been done at similar fashion of the priority queue but taking care of which job "lock key" this command belongs The method is added to get the lock key of each command which is used by the interrupt driven map. The idea you proposed will work, but it may cause starvation for the other jobs as you are not going to promote any commands to the higher priority queue and you are not executing it based on the jobs. so few jobs that have some interrupt commands "kill, suspend,modify" may cause the other jobs to starve while waiting to execute their commands in the higher priority queue. - Mohamed ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3030/#review3801 ----------------------------------------------------------- On 2011-12-09 18:58:56, Mohamed Battisha wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/3030/ > ----------------------------------------------------------- > > (Updated 2011-12-09 18:58:56) > > > Review request for oozie. > > > Summary > ------- > > Oozie continues to materialize new actions after end date modification. The > main issue is related how Oozie materialized the actions based on a FIFO > priority queue. Changing the status of a bundle/coordinator job should take > higher priority than executing this job. > > The main idea is to enable the queue to handle interruptions. Mainly, once > you finished what you are working on currently, you should focus on executing > this next action. > > The issue can be illustrated as follow: > > 1. Configure a pipeline to run for 1 hour > 2. Start the pipeline > 3. After it starts materializing new actions, change the end time (in my > example - to 10 minutes after the pipeline > starts) > 4. Monitor the coordinator apps - they will continue to materialize new > actions past the end time. > > > This addresses bug OOZIE-591. > https://issues.apache.org/jira/browse/OOZIE-591 > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java > 1209829 > > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java > 1209829 > > Diff: https://reviews.apache.org/r/3030/diff > > > Testing > ------- > > Regression for all the Commands in addition for specific testing for the > Interrupt Driven Map > > > Thanks, > > Mohamed > >
