[
https://issues.apache.org/jira/browse/NIFI-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15684749#comment-15684749
]
Joe Skora commented on NIFI-3039:
---------------------------------
[~markap14] I see your points but on loaded systems I've seen a lot of
provenance issues that I was trying to alleviate, the worst being the case
where provenance freezes until the node is restarted. I've never completed
sorted out what is hung in that case.
I'd like to see some or all three of these become configurable properties,
especially if there were some form of advanced parameters implying "change at
your own risk". Further, I think anything exceeding the 100% is contrary to
allowing the user to configure a maximum storage use, unless we clearly
disclose that we could use 110% for performance or efficiency reasons,
otherwise it ends up overrunning available storage unless a buffer is built in
by the user.
For the low water mark anything less than 90% should have a positive impact.
The existing implementation triggers at 90% but only deletes 1 file until usage
is over 100%, that's what I mean by thrashing since it doesn't actually cleanup
to below the trigger threshold. And calculating the file sizes looks like it
is more expensive than I expected, but any alternatives I brainstormed with
[~mosermw] had their own drawbacks.
For the rollover cutoff, some use cases value throughput over provenance but I
think others expect provenance to be as reliable the chain of evidence in a
court trial. I'd never expect it to use more than 100% of the user configured
maximum storage and if slowing the flow to allow provenance to clear is
necessary that seems reasonable in situations where provenance isn't optional,
much like a database won't close a transaction until it's 100% certain to exist
in a restart.
I'm fine bumping the low water to 80% or 85% and I set the rollover cutoff to
90% to be consistent with the purge cutoff. How do you feel about those limit
changes? What about making these configurable properties, since they have such
an impact on the performance of a loaded system but the factors depend so much
on the individual data flows and systems involved?
> Provenance Repository - Fix PurgeOldEvent and Rollover Size Limits
> ------------------------------------------------------------------
>
> Key: NIFI-3039
> URL: https://issues.apache.org/jira/browse/NIFI-3039
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.0.0, 1.1.0, 0.8.0, 0.7.1
> Reporter: Joe Skora
> Assignee: Joe Skora
>
> Current {purgeOldEvents} logic triggers cleanup when 90% of space is used,
> but it only removes one file if usage is under 100%, causing thrashing around
> 100% usage. In testing, cleanup up to 70% after hitting 90% makes the system
> run more smoothly.
> Also, {rollover} will not trigger cleanup unless 110% of the allowed space is
> in use, changing this to 100% also make a difference in testing.
> Before these changes, a test system that generates huge amounts of provenance
> would become unstable and stop processing provenance until restarted. With
> these changes, the system consistently recovers even under heavy load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
