[
https://issues.apache.org/jira/browse/NIFI-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309981#comment-16309981
]
ASF GitHub Bot commented on NIFI-3039:
--------------------------------------
Github user jskora commented on the issue:
https://github.com/apache/nifi/pull/1241
@joewitt Since 0.x is end-of-life, this is OBE and should be closed.
IIRC, the root problem was writing records first and then checking
afterward to see if storage limits were exceeded and a rollover needed. If the
storage limits don't leave a margin before reaching the physical capacity,
writes can fill the disk and crash the NiFi and possible the system. I looked
at the 1.x repositories in depth since they were re-written, but if the same
write-then-check behavior exists this could still be a problem.
> Provenance Repository - Fix PurgeOldEvent and Rollover Size Limits
> ------------------------------------------------------------------
>
> Key: NIFI-3039
> URL: https://issues.apache.org/jira/browse/NIFI-3039
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.0.0, 1.1.0, 0.8.0, 0.7.1
> Reporter: Joe Skora
> Assignee: Joe Skora
>
> Current {purgeOldEvents} logic triggers cleanup when 90% of space is used,
> but it only removes one file if usage is under 100%, causing thrashing around
> 100% usage. In testing, cleanup up to 70% after hitting 90% makes the system
> run more smoothly.
> Also, {rollover} will not trigger cleanup unless 110% of the allowed space is
> in use, changing this to 100% also make a difference in testing.
> Before these changes, a test system that generates huge amounts of provenance
> would become unstable and stop processing provenance until restarted. With
> these changes, the system consistently recovers even under heavy load.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
