[
https://issues.apache.org/jira/browse/NIFI-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156414#comment-15156414
]
Joseph Witt commented on NIFI-1527:
-----------------------------------
Applied latest patch. Seeing better behavior. There does seem to be some sort
of off-by-one kind of thing.
To recreate
1) Have a content repo on a partition that is already using more space than the
content archive allows.
2) Have a GenerateFlowFile feeding UpdateAttribute.
3) Let it run for a few seconds making tons of data and removing it. Then wait
until checkpointing runs.
4) Check how many files are left in the content repository. I consistently end
up with one file. It goes away on restart but will not otherwise go away.
In looking at the info in the content repo i see tons of files with this sort
of info:
- 1049600 Feb 21 21:56 ./95/1456109782914-9
and one single file with this info:
- 343040 Feb 21 21:56 ./786/1456109799488-786
So it looks like the file created that didn't trigger the max entries in a
segment is the one that won't go away until restart. Seems interesting.
> Resource Claim counts not incremented on restart for FlowFiles that are
> swapped out
> -----------------------------------------------------------------------------------
>
> Key: NIFI-1527
> URL: https://issues.apache.org/jira/browse/NIFI-1527
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Blocker
> Fix For: 0.6.0, 0.5.1
>
> Attachments:
> 0001-NIFI-1527-Ensure-that-we-increment-Claimant-Counts-f.patch,
> 0001-NIFI-1527-Ensure-that-we-increment-Claimant-Counts-f.patch,
> 0001-NIFI-1527-Ensure-that-we-increment-Claimant-Counts-f.patch,
> 0001-NIFI-1527-Fixed-issue-that-resulted-in-resource-clai.patch
>
>
> When NiFi starts, it tallies the count of how many FlowFiles reference each
> Resource Claim and then removes or archives any file in the Content
> Repository for which there is no claim.
> However, the claim counts are not incremented for data that is swapped out.
> As a result, the content repository could delete or archive data on restart
> that it should not. This could potentially result in data loss, if NiFi is
> restarted while both of the following conditions are met:
> * All FlowFiles that reference a Resource Claim are swapped out
> * Content Repository's archive is disabled, or the archival threshold is
> already exceeded from data outside of the Content Repository
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
