Balu Vellanki created FALCON-1580:
-------------------------------------
Summary: Feed Retention corner case : does not clean up feed
instances
Key: FALCON-1580
URL: https://issues.apache.org/jira/browse/FALCON-1580
Project: Falcon
Issue Type: Bug
Components: feed
Affects Versions: 0.8
Reporter: Balu Vellanki
If a Feed entity is defined as follows,
{code}
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="rawEmailFeed" description="Raw customer email feed"
xmlns="uri:falcon:feed:0.1">
<tags>externalSystem=USWestEmailServers</tags>
<groups>churnAnalysisDataPipeline</groups>
<frequency>hours(1)</frequency>
<timezone>UTC</timezone>
<late-arrival cut-off="hours(1)"/>
<clusters>
<cluster name="primaryCluster" type="source">
<validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
<retention limit="days(30)" action="delete"/>
</cluster>
</clusters>
<locations>
<location type="data"
path="/user/ambari-qa/falcon/demo/primary/input/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
<location type="stats" path="/"/>
<location type="meta" path="/"/>
</locations>
<ACL owner="ambari-qa" group="users" permission="0x755"/>
<schema location="/none" provider="/none"/>
</feed>
{code}
The feed is valid for one day, hence the feed bundle is valid for a day. The
retention coordinator completes after a day with status SUCCEEDED. During this
one day, there are no feed instances older than 30 days. So no feed instances
are deleted.
After 30 days, the user expects feed instances to be deleted. But there is no
retention coordinator running to delete the feed instances. The solution is to
set the "endtime" for retention coordinator to "feed validityEndTime +
retentionAgeLimit".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
