[ 
https://issues.apache.org/jira/browse/OOZIE-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-1118:
---------------------------------

    Attachment: OOZIE-1118.patch

The patch:
\- Replaced PurgeXCommand, CoordPurgeXCommand, and BundlePurgeXCommand with a 
single PurgeXCommand class that handles all 3 types
\- Parental relationships are taken into account: the children will be purged 
or not purged according to their parents (e.g. if a WF is old enough to be 
purged, but its coord parent isn't, neither will be purged until the coord 
parent is ready); also handles subworkflows
\- moved all of the tests from TestPurgeXCommand, TestCoordPurgeXCommand, and 
TestBundlePurgeXCommand to a new TestPurgeXCommand and added some more tests 
for the parental stuff
\- updated some of the methods in XDataTestCase for inserting jobs into the 
tables to set parent/child relationships properly

There's 3 lines longer than 132 characters, but they're queries, which we 
typically keep on one line even if they are too long.  

I tried to manually test it as best as I could on an actual Oozie server by 
messing with the clock, but its hard to say if that actually worked 100%.  

It's kinda a large patch, so I'll put it on reviewboard shortly.  
                
> improve logic of purge service
> ------------------------------
>
>                 Key: OOZIE-1118
>                 URL: https://issues.apache.org/jira/browse/OOZIE-1118
>             Project: Oozie
>          Issue Type: Improvement
>          Components: bundle, coordinator, workflow
>    Affects Versions: 3.3.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Robert Kanter
>         Attachments: OOZIE-1118.patch
>
>
> The current logic of the purge service is flat. I.e., WF purging only takes 
> into account WF end time, it does not take into account that the WF was 
> started by a COORD job. This means that completed WFs of a running COORD job 
> could be purge if the COORD job runs for longer that the purge age.
> One way of addressing this would be:
> * WF purging only purges WF jobs started directly by a client call.
> * COORD purging purges COORD jobs started directly by a client call. It also 
> purges the WF jobs created by the COORD jobs being purged.
> * BUNDLE purging purges BUNDLE jobs, and the corresponding COORD jobs and WF 
> jobs.
> This could be handled by a new property in the job beans 'job-owner'. Set to 
> 'self' it would mean it can be purged by the same job type purger. If set to 
> other value, then it is a higher level purger the one responsible for purging 
> it.
> This means that for a WF job started by COORD job started by a BUNDLE job, 
> the WF job and the COORD job would have the BUNDLE job as owner, while the 
> BUNDLE with have 'self' as owner.
> This ownership propagation would also have
> A caveat here would be how to handle sub-workflows. 
> I guess we should check if the wf was created from  coord, and if then let 
> the coord purge take care of that, meaning wf purge does not purge wf started 
> by coords.
> Similarly, the same should also apply for sub-WFs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to