If you turn on DEBUG level in your log configuration, it should print out whenever the purge service starts and finishes; in the message where it finishes, it should say how many jobs it deleted. With info level logging, I don't think it prints anything.
The purge behavior won't purge a workflow/coordinator/bundle until all of its parents/children are ready to be purged. This is to protect the consistency of the data. So for example, if you have a long-running coordinator job launching workflows, none of those workflows will be purged until the coordinator is ready to be purged. Perhaps this is what you're seeing? <!-- We need it because of NPE in PurgeService. It needs all Purge > properties to be explicitly set--> > <property> > <name>oozie.service.PurgeService.bundle.older.than</name> > <value>14</value> > <description> > Completed bundle jobs older than this value, in days, will be > purged by the PurgeService. > </description> > </property> This shouldn't be the case; if you don't specify this, it should just default to 7. Can you paste the NPE so I can look into what's causing it? - Robert On Mon, Oct 7, 2013 at 1:30 AM, Serega Sheypak <[email protected]>wrote: > Nothing helps. Looks like I'm missing somthing. Oozie works, but there is > no purge happens. > > > 2013/10/4 Serega Sheypak <[email protected]> > > > purge.executions= 11 > > in counters section, but nothing happens or I don't see i > > > > > > 2013/10/4 Serega Sheypak <[email protected]> > > > >> Hi, I've tried this one: > >> > >> <property> > >> <name>oozie.services.ext</name> > >> <value> > >> org.apache.oozie.service.PurgeService > >> </value> > >> </property> > >> > >> <!-- Purging old materializations--> > >> <!-- Keep them for three months --> > >> <property> > >> <name>oozie.service.PurgeService.older.than</name> > >> <value>90</value> > >> </property> > >> > >> <property> > >> <name>oozie.service.PurgeService.coord.older.than</name> > >> <value>90</value> > >> </property> > >> > >> <property> > >> <name>oozie.service.PurgeService.purge.limit</name> > >> <value>100</value> > >> </property> > >> > >> <property> > >> <name>oozie.service.PurgeService.purge.interval</name> > >> <value>100</value> > >> </property> > >> > >> <!-- We need it because of NPE in PurgeService. It needs all Purge > >> properties to be explicitly set--> > >> <property> > >> <name>oozie.service.PurgeService.bundle.older.than</name> > >> <value>14</value> > >> <description> > >> Completed bundle jobs older than this value, in days, will > be > >> purged by the PurgeService. > >> </description> > >> </property> > >> > >> Oozie is started, cool, I forgot about oozie-default.xml. Thank you. > >> The problem is here I still see last 900+ coordinator materializations. > >> Nothing has been deleted. > >> 100 seconds (oozie.service.PurgeService.purge.interval) did pass. > >> > >> There is nothing related to PurgeSerivce in log. > >> > >> 1. What do I do wrong? > >> 2. oozie.service.PurgeService.purge.limit=100 and > >> oozie.service.PurgeService.older.than=90 (days) > >> What has more power? Or one of these conditions should trigger > >> PurgeService to cleanup oozie DB? > >> > >> > >> > >> 2013/10/2 Robert Kanter <[email protected]> > >> > >>> CDH does have that property. In both CDH and Apache distributions, > that > >>> property (and some others) is specified in oozie-default.xml. There > >>> should > >>> be a copy of its somewhere. Oozie has its own internal copy of > >>> oozie-default.xml, so don't bother editing the one you find; Oozie > loads > >>> its oozie-default.xml first and then loads oozie-site.xml with any > >>> properties you set (Hadoop works the same way with core-default.xml and > >>> core-site.xml). > >>> > >>> The NPE is because you only have the two services, PurgeService and > >>> SchedulerService, > >>> loaded because you overrode the oozie.services property. The NPE > occurs > >>> because Oozie does need the ConfigurationService for sure to work > >>> properly; > >>> and without many of the other services, it will likely have other > issues. > >>> > >>> You should remove your override of oozie.services from oozie-site.xml. > >>> Check oozie-default.xml and see if it lists PurgeService under > >>> oozie.services. If it does not, then you can add it in your > >>> oozie-site.xml > >>> in the oozie.services.ext property; if it does, then the PurgeService > is > >>> enabled. > >>> > >>> - Robert > >>> > >>> > >>> > >>> > >>> On Wed, Oct 2, 2013 at 10:49 AM, Serega Sheypak < > >>> [email protected]>wrote: > >>> > >>> > Hi Robert, thanks for reply. > >>> > > >>> > Cloudera distirbution doesn't have property oozie.services > >>> > 1. I've tried to set all props for purge service with purge period = > >>> 100 > >>> > (seconds???) Nothing happened for half an hour > >>> > > >>> > 2. I've tired to explicitly declare PurgeService and got NPE. > >>> > > >>> > I don't understand how it works at all... I don't see expected > >>> behaviour. > >>> > > >>> > > >>> > 2013/10/2 Robert Kanter <[email protected]> > >>> > > >>> > > Hi Serega, > >>> > > > >>> > > oozie-default.xml specifies a number of oozie services to load via > >>> the > >>> > > "oozie.services" property; but specifying > >>> > > <property> <name>oozie.services</name> <value> > >>> > > org.apache.oozie.service.PurgeService, > >>> > > org.apache.oozie.service.SchedulerService </value> </property> > >>> > > you've now removed all but 2 of them. You should remove that > >>> property > >>> > from > >>> > > your oozie-site.xml. > >>> > > > >>> > > If you want to add additional services, you should use the > >>> > > "oozie.services.ext" property instead. That said, the PurgeService > >>> > should > >>> > > already be listed under "oozie.services" so it should be running by > >>> > > default. > >>> > > > >>> > > - Robert > >>> > > > >>> > > > >>> > > On Wed, Oct 2, 2013 at 2:06 AM, Serega Sheypak < > >>> [email protected] > >>> > > >wrote: > >>> > > > >>> > > > Hi, we are using Oozie vervsion Oozie BUILD_VERSION > >>> [3.3.2-cdh4.3.0] > >>> > > > compiled by [jenkins] on [2013.05.28-04:29:38GMT] > >>> > > > > >>> > > > I want to PurgeService work. > >>> > > > I did set these props: > >>> > > > <!-- Purging old materializations--> <!-- Keep them for three > >>> months > >>> > --> > >>> > > > <property> <name>oozie.service.PurgeService.older.than</name> > >>> > > > <value>90</value> </property> <property> > >>> > > > <name>oozie.service.PurgeService.coord.older.than</name> > >>> > > <value>90</value> > >>> > > > </property> <property> > >>> > > <name>oozie.service.PurgeService.purge.limit</name> > >>> > > > <value>100</value> </property> <property> > >>> > > > <name>oozie.service.PurgeService.purge.interval</name> > >>> > <value>100</value> > >>> > > > </property> > >>> > > > > >>> > > > > >>> > > > The idea is: keep actions data for 90 dayes or not more than last > >>> 100 > >>> > > > items. > >>> > > > oozie.service.PurgeService.purge.interval is set to 100 seconds. > >>> > > > Nothing happened after 100 seconds. > >>> > > > > >>> > > > I've tried to add this section: > >>> > > > <property> <name>oozie.services</name> <value> > >>> > > > org.apache.oozie.service.PurgeService, > >>> > > > org.apache.oozie.service.SchedulerService </value> </property> > >>> > > > > >>> > > > Now Ooize is not starting: > >>> > > > > >>> > > > > >>> > > > org.apache.oozie.service.ServiceException: E0103: Could not load > >>> > > > service classes, null > >>> > > > at > >>> > > > org.apache.oozie.service.Services.loadServices(Services.java:291) > >>> > > > at > >>> org.apache.oozie.service.Services.init(Services.java:208) > >>> > > > > >>> > > > at > >>> > > > > >>> > > > >>> > > >>> > org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39) > >>> > > > at > >>> > > > > >>> > > > >>> > > >>> > org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206) > >>> > > > at > >>> > > > > >>> > > >>> > org.apache.catalina.core.StandardContext.start(StandardContext.java:4705) > >>> > > > > >>> > > > at > >>> > > > > >>> > > > >>> > > >>> > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799) > >>> > > > at > >>> > > > > >>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) > >>> > > > at > >>> > > > > >>> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601) > >>> > > > > >>> > > > at > >>> > > > > >>> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943) > >>> > > > at > >>> > > > > >>> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778) > >>> > > > at > >>> > > > > >>> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504) > >>> > > > > >>> > > > at > >>> > > > > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317) > >>> > > > at > >>> > > > > >>> > > > >>> > > >>> > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324) > >>> > > > at > >>> > > > > >>> > > > >>> > > >>> > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) > >>> > > > > >>> > > > at > >>> > > > > >>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065) > >>> > > > at > >>> > > > > org.apache.catalina.core.StandardHost.start(StandardHost.java:840) > >>> > > > at > >>> > > > > >>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057) > >>> > > > > >>> > > > at > >>> > > > > >>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) > >>> > > > at > >>> > > > > >>> > > >>> > org.apache.catalina.core.StandardService.start(StandardService.java:525) > >>> > > > at > >>> > > > > >>> org.apache.catalina.core.StandardServer.start(StandardServer.java:754) > >>> > > > > >>> > > > at > >>> > org.apache.catalina.startup.Catalina.start(Catalina.java:595) > >>> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>> Method) > >>> > > > at > >>> > > > > >>> > > > >>> > > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >>> > > > > >>> > > > at > >>> > > > > >>> > > > >>> > > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >>> > > > at java.lang.reflect.Method.invoke(Method.java:597) > >>> > > > at > >>> > > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) > >>> > > > at > >>> > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) > >>> > > > > >>> > > > Caused by: java.lang.NullPointerException > >>> > > > at > >>> > > org.apache.oozie.service.PurgeService.init(PurgeService.java:76) > >>> > > > at > >>> > > > > >>> org.apache.oozie.service.Services.setServiceInternal(Services.java:368) > >>> > > > at > >>> > > org.apache.oozie.service.Services.setService(Services.java:354) > >>> > > > > >>> > > > at > >>> > > > org.apache.oozie.service.Services.loadServices(Services.java:287) > >>> > > > ... 26 more > >>> > > > > >>> > > > > >>> > > > What do I do wrong? > >>> > > > > >>> > > > >>> > > >>> > >> > >> > > >
