[
https://issues.apache.org/jira/browse/OOZIE-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966598#comment-13966598
]
Rohini Palaniswamy commented on OOZIE-1401:
-------------------------------------------
bq. But when you have more than a million work flow it will run very slow and
create extra load on DB.
Yes. PurgeService is very inefficient and should be rewritten. Please feel
free to take up that work if you are interested. At Y! we use custom optimized
PL/SQL scripts using Oracle cursors run as cron jobs to perform bulk deletion
and have the PurgeService turned off. So have not focused on the PurgeService.
With lot of 1 min jobs, you must be facing problems with this at Inmobi. You
can even probably enhance it to clean up more frequently based on the frequency
of the coordinator. i.e Something like keep only 10 days worth if frequency is
1 min, 15 days worth if frequency is 5 min, 30 days worth if frequency is 1
days, etc. Remember [~sriksun] talking about such a feature.
> PurgeCommand should purge the workflow jobs w/o end_time
> --------------------------------------------------------
>
> Key: OOZIE-1401
> URL: https://issues.apache.org/jira/browse/OOZIE-1401
> Project: Oozie
> Issue Type: Sub-task
> Components: bundle, coordinator, workflow
> Affects Versions: trunk
> Reporter: Mona Chitnis
> Fix For: trunk
>
>
> Currently, Purge logic is not working with those workflow jobs with
> end_time=null. This command needs to take care of those jobs as well. This
> happens in the case of long stuck jobs after Hadoop restarts or DB failures.
> It could be done by checking created_time if end_time is not available.
> The current query:
> select w from WorkflowJobBean w where w.endTimestamp < :endTime
--
This message was sent by Atlassian JIRA
(v6.2#6252)
