Hi Mike, thanks for the advice. Our NiFi instances are running for week, if not months. Often times until the next update, so the startup option will bring much benefit, I fear, or am I mistaken. But looking forward to 1.23!
On 8 July 2023 13:40:15 CEST, Mike Thomsen <[email protected]> wrote: >Lars, > >You should also experiment with cleanHistoryOnStart. I did some >experimentation this morning where I set the maxHistory to 1 (1 day vs the >default of 30 which is 30 days), created a few fake log files from previous >days and NiFi immediately cleared out those "old files" on startup. I have >a Jira ticket up to fix this for 1.x and 2.x and will likely have it up >today. Should definitely be ready for 1.23 > >On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling <[email protected]> >wrote: > >> Dear NiFiers, we have been bugged so much by overflowing logfiles, and >> nothing has ever helped. I thought it was just my lack of >> skills...especially when NiFi has some issues and keeps on spilling >> stacktraces with high frequency to disk, it eats up space quickly. I have >> created cronjobs that rotate logs every minute iff required, and when >> almost no space is left, it simply deletes old files. Will try totalCapSize >> etc. Thank you for the pointers! Best, Lars >> >> >> On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed" <[email protected]> >> wrote: >> >>> Hi >>> >>> Please have a look at this old jira: >>> https://issues.apache.org/jira/browse/NIFI-2203 >>> I have had issues where a processor create a log message ever 10ms >>> resulting in the disk is being full. For me it seems like the maxHistory >>> settings only effect how many files defined by the rolling patten to be >>> kept. If you have defined it like this: >>> >>> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern> >>> MaxHistory only effect the days not the increments file %i per day. So >>> you can stille have thousands of files in one day. >>> The totalSizeCap will delete the oldes files if the total size hits the >>> cap settings. >>> >>> The totalSizeCap have been added in the logback.xml file for >>> nifi-registry where it has been added inside the rollingPolicy section. I >>> cound not get it to work inside the rollingPolicy section in nifi but just >>> added in appender section. See my comment in the jira: >>> https://issues.apache.org/jira/browse/NIFI-2203 >>> >>> Kind regards >>> Jens M. Kofoed >>> >>> Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen < >>> [email protected]>: >>> >>>> Yeah, I'm working through some of it where I have time. I plan to have a >>>> Jira up this weekend. I'm wondering, though, if we shouldn't consider a >>>> spike for switching to log4j2 in 2.X because I saw a lot of complaints >>>> about logback being inconsistent in honoring its settings. >>>> >>>> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt <[email protected]> wrote: >>>> >>>>> Hmmmm. Interesting. Can you capture these bits of fun in a jira? >>>>> >>>>> Thanks >>>>> >>>>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <[email protected]> >>>>> wrote: >>>>> >>>>>> After doing some research, it appears that <maxFileSize/> is a wonky >>>>>> setting WRT how well it's honored by logback. I let a GenerateFlowFile > >>>>>> LogAttribute flow run for a long time, and it just kept filling up. When >>>>>> I >>>>>> added <totalSizeCap/> that appeared to force expected behavior on total >>>>>> log >>>>>> size. We might want to add the following: >>>>>> >>>>>> <cleanHistoryOnStart>true</cleanHistoryOnStart> >>>>>> <totalSizeCap>50GB</totalSizeCap> >>>>>> >>>>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Mike, >>>>>>> >>>>>>> You aren't alone in experiencing this. I think logback uses a >>>>>>> pattern matcher on filename to discover files to delete. If "something" >>>>>>> happens which causes a gap in the date pattern, then the matcher will >>>>>>> then >>>>>>> fail to pick up and delete files on the other side of that gap. >>>>>>> >>>>>>> Regards, >>>>>>> -- Mike M >>>>>>> >>>>>>> >>>>>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> We are using the stock configuration, and have noticed that we have >>>>>>>> a lot of nifi-app* logs that are well beyond the historic data cap of >>>>>>>> 30 >>>>>>>> days in logback.xml; some of those logs go back to April. We also have >>>>>>>> a >>>>>>>> bunch of 0 byte nifi-user logs and some of the other logs are 0 bytes >>>>>>>> as >>>>>>>> well. It looks like logback is rotating based on time, but isn't >>>>>>>> cleaning >>>>>>>> up. Is this expected behavior or a problem with the configuration? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Mike >>>>>>>> >>>>>>>
