[
https://issues.apache.org/jira/browse/NIFI-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura updated NIFI-3373:
--------------------------------
Assignee: Koji Kawamura
Status: Patch Available (was: Open)
> Add nifi.flow.configuration.archive.max.count property
> ------------------------------------------------------
>
> Key: NIFI-3373
> URL: https://issues.apache.org/jira/browse/NIFI-3373
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
>
> Currently we can limit the number of flow.xml.gz archive files by:
> * total archive size (nifi.flow.configuration.archive.max.storage)
> * archive file age (nifi.flow.configuration.archive.max.time)
> In addition to these conditions to manage old archives, there's a demand that
> simply limiting number of archive files regardless time or size constraint.
> https://lists.apache.org/thread.html/4d2d9cec46ee896318a5492bf020f60c28396e2850c077dad40d45d2@%3Cusers.nifi.apache.org%3E
> We can provide that by adding new property
> 'nifi.flow.configuration.archive.max.count', so that If specified, only N
> latest config files can be archived.
> Make those properties optional, and process in following order:
> - If max.count is specified, any archive other than the latest (N-1) is
> removed
> - If max.time is specified, any archive that is older than max.time is removed
> - If max.storage is specified, old archives are deleted while total size is
> greater than the configuration
> - Create new archive, keep the latest archive regardless of above limitations
> To illustrate how flow.xml archiving works, here are simulations with the
> updated logic, where the size of flow.xml keeps increasing:
> h3. CASE-1
> archive.max.storage=10MB
> archive.max.count = 5
> ||Time || flow.xml || archives || archive total ||
> |t1 | f1 5MB | f1 | 5MB|
> |t2 | f2 5MB | f1, f2 | 10MB|
> |t3 | f3 5MB | f2, f3 | 10MB|
> |t4 | f4 10MB | f4 | 10MB|
> |t5 | f5 15MB | f5 | 15MB|
> |t6 | f6 20MB | f6 | 20MB|
> |t7 | f7 25MB | t7 | 25MB|
> * t3: The oldest f1 is removed, because f1 + f2 + f3 > 10MB.
> * t5: Even if flow.xml size exceeds max.storage, the latest archive is
> created. f4 is removed because f4 + f5 > 10MB. WAR message is logged because
> f5 is greater than 10MB.
> In this case, NiFi will keep logging WAR message
> indicating archive storage size is exceeding limit, from t5.
> After t5, NiFi will only keep the latest flow.xml.
> h3. CASE-2
> If at least 5 archives need to be kept no matter what, then set
> blank max.storage and max.time.
> archive.max.storage=
> archive.max.time=
> archive.max.count = 5 // Only limit archives by count
> |Time || flow.xml || archives || archive total ||
> |t1 | f1 5MB | f1 | 5MB|
> |t2 | f2 5MB | f1, f2 | 10MB|
> |t3 | f3 5MB | f1, f2, f3 | 15MB|
> |t4 | f4 10MB | f1, f2, f3, f4 | 25MB|
> |t5 | f5 15MB | f1, f2, f3, f4, f5 | 40MB|
> |t6 | f6 20MB | f2, f3, f4, f5, f6 | 55MB|
> |t7 | f7 25MB | f3, f4, f5, f6, (f7) | 50MB, (75MB)|
> |t8 | f8 30MB | f3, f4, f5, f6 | 50MB|
> * From t6, oldest archive is removed to keep number of archives <= 5
> * At t7, if the disk has only 60MB space, f7 won't be archived. And
> after this point, archive mechanism stop working (Trying to create new
> archive, but keep getting exception: no space left on device).
> In either case above, once flow.xml has grown to that size, some human
> intervention would be needed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
