Dear NiFiers, we have been bugged so much by overflowing logfiles, and nothing has ever helped. I thought it was just my lack of skills...especially when NiFi has some issues and keeps on spilling stacktraces with high frequency to disk, it eats up space quickly. I have created cronjobs that rotate logs every minute iff required, and when almost no space is left, it simply deletes old files. Will try totalCapSize etc. Thank you for the pointers! Best, Lars
On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed" <jmkofoed....@gmail.com> wrote: >Hi > >Please have a look at this old jira: >https://issues.apache.org/jira/browse/NIFI-2203 >I have had issues where a processor create a log message ever 10ms >resulting in the disk is being full. For me it seems like the maxHistory >settings only effect how many files defined by the rolling patten to be >kept. If you have defined it like this: ><fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern> >MaxHistory only effect the days not the increments file %i per day. So you >can stille have thousands of files in one day. >The totalSizeCap will delete the oldes files if the total size hits the cap >settings. > >The totalSizeCap have been added in the logback.xml file for nifi-registry >where it has been added inside the rollingPolicy section. I cound not get >it to work inside the rollingPolicy section in nifi but just added >in appender section. See my comment in the jira: >https://issues.apache.org/jira/browse/NIFI-2203 > >Kind regards >Jens M. Kofoed > >Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen <mikerthom...@gmail.com>: > >> Yeah, I'm working through some of it where I have time. I plan to have a >> Jira up this weekend. I'm wondering, though, if we shouldn't consider a >> spike for switching to log4j2 in 2.X because I saw a lot of complaints >> about logback being inconsistent in honoring its settings. >> >> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt <joe.w...@gmail.com> wrote: >> >>> Hmmmm. Interesting. Can you capture these bits of fun in a jira? >>> >>> Thanks >>> >>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <mikerthom...@gmail.com> >>> wrote: >>> >>>> After doing some research, it appears that <maxFileSize/> is a wonky >>>> setting WRT how well it's honored by logback. I let a GenerateFlowFile > >>>> LogAttribute flow run for a long time, and it just kept filling up. When I >>>> added <totalSizeCap/> that appeared to force expected behavior on total log >>>> size. We might want to add the following: >>>> >>>> <cleanHistoryOnStart>true</cleanHistoryOnStart> >>>> <totalSizeCap>50GB</totalSizeCap> >>>> >>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <moser...@gmail.com> >>>> wrote: >>>> >>>>> Hi Mike, >>>>> >>>>> You aren't alone in experiencing this. I think logback uses a pattern >>>>> matcher on filename to discover files to delete. If "something" happens >>>>> which causes a gap in the date pattern, then the matcher will then fail to >>>>> pick up and delete files on the other side of that gap. >>>>> >>>>> Regards, >>>>> -- Mike M >>>>> >>>>> >>>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <mikerthom...@gmail.com> >>>>> wrote: >>>>> >>>>>> We are using the stock configuration, and have noticed that we have a >>>>>> lot of nifi-app* logs that are well beyond the historic data cap of 30 >>>>>> days >>>>>> in logback.xml; some of those logs go back to April. We also have a bunch >>>>>> of 0 byte nifi-user logs and some of the other logs are 0 bytes as well. >>>>>> It >>>>>> looks like logback is rotating based on time, but isn't cleaning up. Is >>>>>> this expected behavior or a problem with the configuration? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Mike >>>>>> >>>>>
