Re: Content Claims Filling Disk - Best practice for small files?

2020-09-18 Thread Ryan Hendrickson
nts, etc. It seems to cap >>>> out around 2-3GB/5 minutes >>>> >>>> @Mark I'll check the diagnostics. >>>> >>>> @Jim definitely disk space 100% used. >>>> >>>> [1] https://issues.apache.org/jira/browse/NIFI

Re: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Mark Payne
ailto:jwilli...@alertlogic.com> | alertlogic.com<http://www.alertlogic.com/> <https://twitter.com/alertlogic><https://www.linkedin.com/company/alert-logic> From: Joe Witt mailto:joe.w...@gmail.com>> Sent: Thursday, September 17, 2020 10:19 AM To: users@nifi.apache.org<mai

Re: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Ryan Hendrickson
t;>> >>> Ryan >>> >>> On Thu, Sep 17, 2020 at 11:33 AM Williams, Jim >>> wrote: >>> >>>> Ryan, >>>> >>>> >>>> >>>> Is this this maybe a case of exhausting inodes on the filesystem rather >>>> than ex

Re: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Ryan Hendrickson
er >>> than exhausting the space available? If you do a ‘df -I’ on the system >>> what do you see for inode usage? >>> >>> >>> >>> Warm regards, >>> >>> >>> >>> <https://www.alertlogic.com/&

Re: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Ryan Hendrickson
Engineering >> >> O: +1 713.341.7812 | C: +1 919.523.8767 | jwilli...@alertlogic.com | >> alertlogic.com <http://www.alertlogic.com/> >> <https://twitter.com/alertlogic> >> <https://www.linkedin.com/company/alert-logic> >> >> >> >> &

Re: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Mark Payne
l.com>> Sent: Thursday, September 17, 2020 10:19 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Content Claims Filling Disk - Best practice for small files? can you share your flow.xml.gz? On Thu, Sep 17, 2020 at 8:08 AM Ryan Hendrickson mailto:ryan.andre

Re: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Ryan Hendrickson
gt; > <https://www.linkedin.com/company/alert-logic> > > > > > > *From:* Joe Witt > *Sent:* Thursday, September 17, 2020 10:19 AM > *To:* users@nifi.apache.org > *Subject:* Re: Content Claims Filling Disk - Best practice for small > files? > > > > c

Re: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Mark Payne
Ryan, Thanks. So 1.12.0 has no known issues with content repo not being cleaned up properly. As you pointed out, nifi.content.claim.max.appendable.size is intended to cap the maximum number of FlowFiles that will be written to a single file. However, it does come with a couple of caveats.

RE: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Williams, Jim
D68CDD.FC463950] <https://www.linkedin.com/company/alert-logic> [cid:image004.png@01D68CDD.FC463950] From: Joe Witt Sent: Thursday, September 17, 2020 10:19 AM To: users@nifi.apache.org Subject: Re: Content Claims Filling Disk - Best practice for small files? can you share your flow.xml.gz? On Thu

Re: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Joe Witt
can you share your flow.xml.gz? On Thu, Sep 17, 2020 at 8:08 AM Ryan Hendrickson < ryan.andrew.hendrick...@gmail.com> wrote: > 1.12.0 > > Thanks, > Ryan > > On Thu, Sep 17, 2020 at 11:04 AM Joe Witt wrote: > >> Ryan >> >> What version are you using? I do think we had an issue that kept items >>

Re: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Ryan Hendrickson
1.12.0 Thanks, Ryan On Thu, Sep 17, 2020 at 11:04 AM Joe Witt wrote: > Ryan > > What version are you using? I do think we had an issue that kept items > around longer than intended that has been addressed. > > Thanks > > On Thu, Sep 17, 2020 at 7:58 AM Ryan Hendrickson < >

Re: Content Claims Filling Disk - Best practice for small files?

2020-09-17 Thread Joe Witt
Ryan What version are you using? I do think we had an issue that kept items around longer than intended that has been addressed. Thanks On Thu, Sep 17, 2020 at 7:58 AM Ryan Hendrickson < ryan.andrew.hendrick...@gmail.com> wrote: > Hello, > I've got ~15 million FlowFiles, each roughly 4KB,