[
https://issues.apache.org/jira/browse/NIFI-9704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17493998#comment-17493998
]
Mark Payne commented on NIFI-9704:
----------------------------------
I attached the flow that I used to reproduce this issue. It has two
GenerateFlowFile processors. The first one generates large numbers of tiny
FlowFiles and keeps 1/1000 of them in the flow. The second generates large
FlowFiles (5 MB each). This causes the files in the content repo to become much
larger, exacerbating the problem. To reproduce: Start all processors. Wait for
the queue whose destination is the funnel to fill up. Stop the generateFlowFile
processors. Optionally, empty the queue between GenerateFlowFile ->
RouteOnAttribute. Wait for the FlowFile Repository to checkpoint (happens every
20 seconds by default). Check the size of the content repo ( {{{}du -h -d 0
content_repository{}}}) on Linux / OSX. Change the value of the property in
nifi.properties, restarted, drop all flowfiles, and repeat test.
> Improve verbose diagnostics and change default value of
> "nifi.content.claim.max.appendable.size" from 1 MB to 50 KB
> -------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-9704
> URL: https://issues.apache.org/jira/browse/NIFI-9704
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Configuration, Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Labels: ageoff, content-repo, content-repository, disk,
> disk-space, full, space
> Fix For: 1.16.0
>
> Attachments: Use_Excessive_Disk_Space.json
>
>
> We sometimes see users (especially those with a mix of flows where some
> produce very large FlowFiles and some produce tons of tiny FlowFiles) run
> into an issue where the UI shows very little space is used up by FlowFiles
> but the content repository fills up.
> Yesterday I was on a call with such a team. Their NiFi UI showed one node had
> about 200,000 FlowFiles totaling dozens of MB. However, the content
> repository was 300 GB in size (which was the entire content repo). As a
> result, their NiFi instance stopped processing data because the content repo
> was completely full.
> We did some analysis to check if there were "orphaned" flowfiles filling the
> content repository, but there were not. Instead, the {{nifi.sh diagnostics
> --verbose}} command showed us that a handful of queues were causing the
> content repo to retain those 100's of GB of data, even though the FlowFiles
> themselves only amounted to a few MB.
> This is a known issue and is caused by how we write FlowFile Content to disk,
> using the same file on disk for many content claims. By default, we allow up
> to 1 MB to be written to a file before we conclude that we should no longer
> write additional FlowFiles to it. This is controlled by the
> "nifi.content.claim.max.appendable.size" property.
> The support team indicates that this happens frequently. We need to change
> the default value of this property from "1 MB" to "50 KB". This will
> dramatically decrease the incidence rate.
> I setup a flow to test this locally. Queued up 5,000 FlowFiles totaling 610
> KB, and the Content Repo was taking 45 GB of disk space. I then dropped all
> data, changed this property from the default 1 MB to 50 KB and repeated the
> test. As expected, I queued up the same number of files (610 KB worth), and
> the content repo occupied 2.6 GB of disk space. I.e., making the value 5% of
> the original value resulted in occupying only 5% as much "unnecessary" disk
> space.
> Performance tests indicate that the performance was approximately the same,
> regardless of whether I used "1 MB" or "50 KB"
> Additionally, when running the {{nifi.sh diagnostics --verbose}} command, the
> information that was necessary for tracking down the root cause of this was
> made available but took tremendous effort to decipher. We should update the
> diagnostics output when scanning the content repo to show the amount of data
> in the content repo that is being retained by each queue in the flow.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
