Lars, You should also experiment with cleanHistoryOnStart. I did some experimentation this morning where I set the maxHistory to 1 (1 day vs the default of 30 which is 30 days), created a few fake log files from previous days and NiFi immediately cleared out those "old files" on startup. I have a Jira ticket up to fix this for 1.x and 2.x and will likely have it up today. Should definitely be ready for 1.23
On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling <lars.winderl...@posteo.de> wrote: > Dear NiFiers, we have been bugged so much by overflowing logfiles, and > nothing has ever helped. I thought it was just my lack of > skills...especially when NiFi has some issues and keeps on spilling > stacktraces with high frequency to disk, it eats up space quickly. I have > created cronjobs that rotate logs every minute iff required, and when > almost no space is left, it simply deletes old files. Will try totalCapSize > etc. Thank you for the pointers! Best, Lars > > > On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed" <jmkofoed....@gmail.com> > wrote: > >> Hi >> >> Please have a look at this old jira: >> https://issues.apache.org/jira/browse/NIFI-2203 >> I have had issues where a processor create a log message ever 10ms >> resulting in the disk is being full. For me it seems like the maxHistory >> settings only effect how many files defined by the rolling patten to be >> kept. If you have defined it like this: >> >> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern> >> MaxHistory only effect the days not the increments file %i per day. So >> you can stille have thousands of files in one day. >> The totalSizeCap will delete the oldes files if the total size hits the >> cap settings. >> >> The totalSizeCap have been added in the logback.xml file for >> nifi-registry where it has been added inside the rollingPolicy section. I >> cound not get it to work inside the rollingPolicy section in nifi but just >> added in appender section. See my comment in the jira: >> https://issues.apache.org/jira/browse/NIFI-2203 >> >> Kind regards >> Jens M. Kofoed >> >> Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen < >> mikerthom...@gmail.com>: >> >>> Yeah, I'm working through some of it where I have time. I plan to have a >>> Jira up this weekend. I'm wondering, though, if we shouldn't consider a >>> spike for switching to log4j2 in 2.X because I saw a lot of complaints >>> about logback being inconsistent in honoring its settings. >>> >>> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt <joe.w...@gmail.com> wrote: >>> >>>> Hmmmm. Interesting. Can you capture these bits of fun in a jira? >>>> >>>> Thanks >>>> >>>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <mikerthom...@gmail.com> >>>> wrote: >>>> >>>>> After doing some research, it appears that <maxFileSize/> is a wonky >>>>> setting WRT how well it's honored by logback. I let a GenerateFlowFile > >>>>> LogAttribute flow run for a long time, and it just kept filling up. When I >>>>> added <totalSizeCap/> that appeared to force expected behavior on total >>>>> log >>>>> size. We might want to add the following: >>>>> >>>>> <cleanHistoryOnStart>true</cleanHistoryOnStart> >>>>> <totalSizeCap>50GB</totalSizeCap> >>>>> >>>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <moser...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Mike, >>>>>> >>>>>> You aren't alone in experiencing this. I think logback uses a >>>>>> pattern matcher on filename to discover files to delete. If "something" >>>>>> happens which causes a gap in the date pattern, then the matcher will >>>>>> then >>>>>> fail to pick up and delete files on the other side of that gap. >>>>>> >>>>>> Regards, >>>>>> -- Mike M >>>>>> >>>>>> >>>>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <mikerthom...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> We are using the stock configuration, and have noticed that we have >>>>>>> a lot of nifi-app* logs that are well beyond the historic data cap of 30 >>>>>>> days in logback.xml; some of those logs go back to April. We also have a >>>>>>> bunch of 0 byte nifi-user logs and some of the other logs are 0 bytes as >>>>>>> well. It looks like logback is rotating based on time, but isn't >>>>>>> cleaning >>>>>>> up. Is this expected behavior or a problem with the configuration? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Mike >>>>>>> >>>>>>
