[
https://issues.apache.org/jira/browse/NIFI-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17401214#comment-17401214
]
Andrew Heys commented on NIFI-9056:
-----------------------------------
Thanks for the info about the diagnostics, I did not know about that feature.
Unfortunately, our Nifi cluster operates on a private company network, so I am
unable to directly share the file with you, but I would be happy to look for
anything of note and relay the contents to you. Some things I noticed:
* The number of Timer-Driven threads is configured to 50 with 8 available
cores (not sure if this could cause an issue with the updated nifi version)
* Even with servers that have 0 queued flowfiles/bytes, the content repository
is still retaining a large amount GBs.
One thing that I just thought of, is that our Nifi content repositories are not
created with the same amount of total space. Some nodes have 3x the size of
others for total disk space in the content repository. Here is a representation
of our cluster content storage:
||Node Address||Total Space||Usage Space||Utilization||
|Node 1|525 GB|80GB|15%|
|Node 2|175GB|56GB|32%|
|Node 3|175GB|80GB|46%|
|Node 4|175GB|51GB|29%|
|Node 5|525GB|95GB|18%|
|Node 6|175GB|46GB|27%|
|Node 7|525GB|30GB|6%|
This is with the cluster reporting 22k (180MB) of flowfiles on the canvas. Is
it possible that Nifi does not think the content repository needs to be cleaned
up because the larger nodes are below the threshold properties?
> Content Repository Filling Up
> -----------------------------
>
> Key: NIFI-9056
> URL: https://issues.apache.org/jira/browse/NIFI-9056
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.13.2
> Reporter: Andrew Heys
> Priority: Major
>
> We have a clustered nifi setup that has recently been upgraded to 1.13.2 from
> 1.11.4. After upgrading, one of the issues we have run into is that the
> Content Repository will fill up to the
> nifi.content.repository.archive.backpressure.percentage mark and lock the
> processing & canvas. The only solution is to restart nifi at this point. We
> have the following properties set:
> nifi.content.repository.archive.backpressure.percentage=95%
> nifi.content.repository.archive.max.usage.percentage=25%
> nifi.content.repository.archive.max.retention.period=2 hours
> The max usage property seems to be completed ignored. Monitoring the nifi
> cluster disk % for content repository shows that it slowly fills up over time
> and never decreasing. If we pause the input to entire nifi flow and let all
> the processing clear out with 0 flowfiles remaining on the canvas for 15+
> minutes, the content repository disk usage does not decrease. Currently, our
> only solution is to restart nifi on a daily cron schedule. After restarting
> the nifi, it will clear out the 80+ GB of the content repository and usage
> falls down to 0%.
>
> There seems to be an issue removing the older content claims in 1.13.2.
> Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
