[
https://issues.apache.org/jira/browse/NIFI-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096477#comment-16096477
]
Mark Payne commented on NIFI-3376:
----------------------------------
[~m-hogue] [~mosermw] - I don't believe this is a road that we should really
venture down. Trying to compact it would mean that the data would have to be
copied, first off, which is potentially incredibly expensive. It means that any
FlowFile would have to also be updated to point to the new content claim, which
would require a stop-the-world lock in order to do. It would also mean that we
would have some very complex inter-dependencies in locking because this could
not happen while a Processor has access to a FlowFile that references a Content
Claim that would be compacted. That means that we'd also have to store a
tremendous amount of state in the Processor/queue about what is in-flight. In
addition, it would mean that Provenance events that are pointing to the content
would also no longer be pointing to the right place. Also, of note, this would
have to be done in such a way that we write the new, 'compacted' resource claim
parallel to the old content claim so that a failure can be rolled back; if
already concerned about over-utilization of the repo this may cause even more
tension. And all of this doesn't even taken into account the complexity of code
maintenance.
To get this to work properly, I'd estimate that it would take several months of
focused effort by the people who know the inner workings of the framework, and
I think that with NIFI-3736 resolved, it provides a fairly small benefit, so
I'm a -1 on this proposal.
> Implement content repository ResourceClaim compaction
> -----------------------------------------------------
>
> Key: NIFI-3376
> URL: https://issues.apache.org/jira/browse/NIFI-3376
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 0.7.1, 1.1.1
> Reporter: Michael Moser
> Assignee: Michael Hogue
>
> On NiFi systems that deal with many files whose size is less than 1 MB, we
> often see that the actual disk usage of the content_repository is much
> greater than the size of flowfiles that NiFi reports are in its queues. As
> an example, NiFi may report "50,000 / 12.5 GB" but the content_repository
> takes up 240 GB of its file system. This leads to scenarios where a 500 GB
> content_repository file system gets 100% full, but "I only had 40 GB of data
> in my NiFi!"
> When several content claims exist in a single resource claim, and most but
> not all content claims are terminated, the entire resource claim is still not
> eligible for deletion or archive. This could mean that only one 10 KB
> content claim out of a 1 MB resource claim is counted by NiFi as existing in
> its queues.
> If a particular flow has a slow egress point where flowfiles could back up
> and remain on the system longer than expected, this problem is exacerbated.
> A potential solution is to compact resource claim files on disk. A background
> thread could examine all resource claims, and for those that get "old" and
> whose active content claim usage drops below a threshold, then rewrite the
> resource claim file.
> A potential work-around is to allow modification of the FileSystemRepository
> MAX_APPENDABLE_CLAIM_LENGTH to make it a smaller number. This would increase
> the probability that the content claims reference count in a resource claim
> would reach 0 and the resource claim becomes eligible for deletion/archive.
> Let users trade-off performance for more accurate accounting of NiFi queue
> size to content repository size.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
