Hi, The discussion about the configurable option to have rsyslog remove state files where no file exists to avoid the 'leak' of state files can be found under the following link: https://github.com/rsyslog/rsyslog/pull/4500 . The PR should be updated, if there is a plan to deliver it within rsyslog. Advantages/disadvantages of both approaches are discussed there.
Attila Lakatos On Sat, Mar 26, 2022 at 12:19 AM David Lang via rsyslog < [email protected]> wrote: > new parameter that was added deleteStateOnFileDelete > > you will need to be running a pretty current version to have that. > > David Lang > > On Fri, 25 Mar 2022, David Lang wrote: > > > Date: Fri, 25 Mar 2022 11:48:02 -0700 (PDT) > > From: David Lang <[email protected]> > > To: John Chivian <[email protected]> > > Cc: David Lang <[email protected]>, > > "Cosmas, Cossy via rsyslog" <[email protected]>, > > "Cosmas, Cossy" <[email protected]> > > Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... > > > > I would suggest that your cleanup process also delete old state files, > then > > if you restart rsyslog weekly, any state files that have been deleted, > but > > that rsyslog had open, will be purged by the OS (they are relatively > small, > > so infrequent restarts should work, you don't want full restarts > frequently, > > because there is a window during the restart where rsyslog cannot > proccess > > logs) > > > > David Lang > > > > On Fri, 25 Mar 2022, John Chivian wrote: > > > >> Date: Fri, 25 Mar 2022 13:42:33 -0500 > >> From: John Chivian <[email protected]> > >> To: David Lang <[email protected]> > >> Cc: "Cosmas, Cossy via rsyslog" <[email protected]>, > >> "Cosmas, Cossy" <[email protected]> > >> Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... > >> > >> Excellent, thank you David! Our rotation methodology is confirmed as > >> required until rsyslog supports deleting orphan state files at either > >> startup or shutdown, which to my way of thinking is a high priority > add. > >> We have some cleanup tasks that spin every six hours because clients > drop > >> dated files as often as every minute. > >> > >> Regards, > >> > >> > >>> On Mar 25, 2022, at 13:30, David Lang <[email protected]> wrote: > >>> > >>> rsyslog doesn't delete the old state files because it doesn't know if > the > >>> file is going to come back in a few seconds or not (such things > happen), > >>> so the decision was made to keep the files around as a lesser evil > than > >>> re-ingesting an old file that reappears. > >>> > >>> for exactly this duplicate ingestion issue, I would suggest you either > >>> rotate the file to a directory that rsyslog is not watching, or rotate > it > >>> to a filename that rsyslog is not watching. > >>> > >>> If you app is creating files that have a date in their name, and > creating > >>> new files over time, this approach doesn't work, but that's a perfect > >>> example of where rsyslog may think it's done with a file, but that it > may > >>> reappear (either because it's a new file created because the app is > >>> confused with the date, or because someone restored it from elsewhere > to > >>> look at it) > >>> > >>> There is discussion of this problem elsewhere and talk of adding an > option > >>> to have rsyslog remove state files where no file exists to avoid the > >>> 'leak' of state files, but opening up the risk of duplicate ingestion. > >>> > >>> David Lang > >>> > >>> On Fri, 25 Mar 2022, Cosmas, Cossy via rsyslog wrote: > >>> > >>>> Date: Fri, 25 Mar 2022 15:31:12 +0000 > >>>> From: "Cosmas, Cossy via rsyslog" <[email protected]> > >>>> To: John Chivian <[email protected]> > >>>> Cc: "Cosmas, Cossy" <[email protected]>, > >>>> rsyslog-users <[email protected]> > >>>> Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... > >>>> > >>>> Hi John, > >>>> > >>>> I stopped (systemctl stop rsyslog.service) and then started(systemctl > >>>> start rsyslog.service) the rsyslog service but that hasn't reduced > the > >>>> number of state files. > >>>> > >>>> When you say rotate the files are you referring to the cron tab job > that > >>>> is deleting all of the monitored audit log files? > >>>> > >>>> That activity takes place overnight. > >>>> > >>>> Regards, > >>>> Cossy > >>>> > >>>> -----Original Message----- > >>>> From: Cosmas, Cossy > >>>> Sent: 25 March 2022 15:22 > >>>> To: John Chivian <[email protected]> > >>>> Cc: rsyslog-users <[email protected]> > >>>> Subject: RE: [rsyslog] Imifile-state File Housekeeping Query... > >>>> > >>>> > >>>> Hi John, > >>>> > >>>> Thanks for the advice, appreciate it. > >>>> > >>>> Ill give it a go now. > >>>> > >>>> Regards, > >>>> Cossy > >>>> > >>>> -----Original Message----- > >>>> From: John Chivian <[email protected]> > >>>> Sent: 25 March 2022 15:21 > >>>> To: rsyslog-users <[email protected]> > >>>> Cc: Cosmas, Cossy <[email protected]> > >>>> Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... > >>>> > >>>> ! EXTERNAL MESSAGE - Think Before You Click or Download > >>>> > >>>> My best advice is to stop and restart rsyslog after rotating files (a > HUP > >>>> won’t do it). This makes rsyslog close and verify state files at > >>>> shutdown, and the ones for non-existent files will then get removed > at > >>>> startup. > >>>> > >>>> Regards, > >>>> > >>>>> On Mar 25, 2022, at 10:10, Cosmas, Cossy via rsyslog > >>>>> <[email protected]> wrote: > >>>>> > >>>>> > >>>>> PS. We are running rsyslog 8.2006. > >>>>> > >>>>> -----Original Message----- > >>>>> From: rsyslog <[email protected]> On Behalf Of > Cosmas, > >>>>> Cossy via rsyslog > >>>>> Sent: 25 March 2022 10:02 > >>>>> To: [email protected] > >>>>> Cc: Cosmas, Cossy <[email protected]> > >>>>> Subject: [rsyslog] Imifile-state File Housekeeping Query... > >>>>> > >>>>> ! EXTERNAL MESSAGE - Think Before You Click or Download > >>>>> > >>>>> > >>>>> Dear Rsyslog Forum Users, > >>>>> > >>>>> A quick question from a relative newbie... > >>>>> > >>>>> I have configured rsyslog to monitor my applications audit log > files. I > >>>>> have also implemented a cron based housekeeping script to delete the > >>>>> application audit log files when they are over a week old. > >>>>> > >>>>> I would have expected the rsyslog imifile-state files to > automatically > >>>>> reduce in number as the number of audit files decreases due to the > above > >>>>> housekeeping task but this is not the case. > >>>>> > >>>>> The number of imifile-state files just keeps on increasing and this > is > >>>>> problematic as we have limits around the number of open files that > >>>>> rsyslog can maintain. > >>>>> > >>>>> I would just like to know what should be happening here and what is > >>>>> normal. > >>>>> > >>>>> Does rsyslog ever automatically remove redundant state files or are > they > >>>>> left in place with the user expected to implement a cron based > routine > >>>>> to delete them manually? > >>>>> > >>>>> Any advice appreciated. > >>>>> > >>>>> Thank you. > >>>>> > >>>>> Regards, > >>>>> > >>>>> Cossy Cosmas > >>>>> Payments and Transaction Management Services Diebold Nixdorf > >>>>> > >>>>> Advanced notice of annual leave: > >>>>> 20th June - 24th June > >>>>> 22nd August - 4th September > >>>>> > >>>>> Mobile: +44 7717 863755 > >>>>> One The Boulevard, Cain Road, > >>>>> Bracknell, Berkshire, RG12 1WP > >>>>> > >>>>> [email protected]<mailto: > [email protected] > >>>>>> > >>>>> DieboldNixdorf.com > >>>>> > >>>>> [1_twitter_logo_24px]< > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=LUATZyF1IN8aMKCSuxYkfho4Vg6eU041XiNrdpyD3so&e= > >>>>> > [2_facebook_logo_24px] > >>>>> < > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=X2WIo3MSLhpeBcW6VFX4Sy2SM0WbkIVr7xwShNubav4&e= > >>>>> > [3_youtube_logo_24px] > >>>>> < > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=6wy_XQvegYq7h1tm5hOX8BXR_r-hKk9D3osb2Lz0Nro&e= > >>>>> > [4_linkedin_logo_24px] > >>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A_ > > _w > >>> ww. > >>>>> > linkedin.com_company_diebold&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=j7-qHkmmXOVmynEwlaLnBhLfDyOBUuON7TbZ8mexxOw&e= > > >>>>> > [5_blog_logo_24px] <http://blog.dieboldnixdorf.com/> > >>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net > >>>>> > _mailman_listinfo_rsyslog&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZ > >>>>> > bAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHT > >>>>> > VWykmkZNx0bHELIbg9Zzft9Q6GbmLh4t&s=FY_T0UNmcLGdLKOQg3GcZNwP-6fDEVWHKy0 > >>>>> aplakdEE&e= > >>>>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_pr > >>>>> > ofessional-2Dservices_&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAd > >>>>> > baH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHTVWy > >>>>> > kmkZNx0bHELIbg9Zzft9Q6GbmLh4t&s=UF7Hr7h98tGBxyIDcwVhf3axMFQhzWkQSAh6T5 > >>>>> 45tFs&e= What's up with rsyslog? Follow > >>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerh > >>>>> > ards&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_ > >>>>> > Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHTVWykmkZNx0bHELIbg9Zzf > >>>>> t9Q6GbmLh4t&s=3pzUn1DEzQh35h5Uyo_0LY3g6k70GXY3BXy3KSLL55k&e= > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>> DON'T LIKE THAT. > >>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>> DON'T LIKE THAT. > >> > > > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
