Hey Alan, Any chance that you're seeing any OutOfMemory errors or anything like that? Any error logs about FlowFile Repository?
Can you check the total size of your FlowFile Repository? The reason that I ask is that someone was reporting a problem the other day about an OOME preventing the FlowFile Repo from checkpointing. As a result, if this were to happen, it would result in Content Claims not being properly cleaned up. Thanks -Mark Sent from my iPhone > On Sep 1, 2017, at 5:24 PM, Alan Jackoway <[email protected]> wrote: > > Hello, > > We have had issues with a few nifi instances recently where the content > repository grew too large and we couldn't get nifi to clean it up. > > In our current instance of this, we added a second disk to keep nifi from > consuming 100% of the available space, and now it is close to consuming all > space on both disks. > > Our nifi.properties looks like this: > # Content Repository > nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository > nifi.content.claim.max.appendable.size=10 MB > nifi.content.claim.max.flow.files=100 > nifi.content.repository.directory.default=/data/3/nifi_storage/edh-production-ingestion/content_repository > nifi.content.repository.directory.content2=/data/5/nifi_storage/edh-production-ingestion/content_repository > nifi.content.repository.archive.max.retention.period=1 hours > nifi.content.repository.archive.max.usage.percentage=50% > nifi.content.repository.archive.enabled=false > nifi.content.repository.always.sync=false > nifi.content.viewer.url=/nifi-content-viewer/ > > Each of those content_repository directories are now consuming about 750GB on > their disks, which are 1TB. The nifi's queue/size is 53,084 / 3.42 MB but it > does process a fairly large amount of data. > > How do we keep nifi from consuming our entire disks with content repository? > We turned off archive enabled in the hope that it would bring the content > repository size down to close to 0, but that has not worked for us. > > Thanks, > Alan > > PS. Sorry for the Friday afternoon email. That's when disks always want to > get full.
