Excellent, thank you David! Our rotation methodology is confirmed as required until rsyslog supports deleting orphan state files at either startup or shutdown, which to my way of thinking is a high priority add. We have some cleanup tasks that spin every six hours because clients drop dated files as often as every minute.
Regards, > On Mar 25, 2022, at 13:30, David Lang <[email protected]> wrote: > > rsyslog doesn't delete the old state files because it doesn't know if the > file is going to come back in a few seconds or not (such things happen), so > the decision was made to keep the files around as a lesser evil than > re-ingesting an old file that reappears. > > for exactly this duplicate ingestion issue, I would suggest you either rotate > the file to a directory that rsyslog is not watching, or rotate it to a > filename that rsyslog is not watching. > > If you app is creating files that have a date in their name, and creating new > files over time, this approach doesn't work, but that's a perfect example of > where rsyslog may think it's done with a file, but that it may reappear > (either because it's a new file created because the app is confused with the > date, or because someone restored it from elsewhere to look at it) > > There is discussion of this problem elsewhere and talk of adding an option to > have rsyslog remove state files where no file exists to avoid the 'leak' of > state files, but opening up the risk of duplicate ingestion. > > David Lang > > On Fri, 25 Mar 2022, Cosmas, Cossy via rsyslog wrote: > >> Date: Fri, 25 Mar 2022 15:31:12 +0000 >> From: "Cosmas, Cossy via rsyslog" <[email protected]> >> To: John Chivian <[email protected]> >> Cc: "Cosmas, Cossy" <[email protected]>, >> rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... >> >> Hi John, >> >> I stopped (systemctl stop rsyslog.service) and then started(systemctl start >> rsyslog.service) the rsyslog service but that hasn't reduced the number of >> state files. >> >> When you say rotate the files are you referring to the cron tab job that is >> deleting all of the monitored audit log files? >> >> That activity takes place overnight. >> >> Regards, >> Cossy >> >> -----Original Message----- >> From: Cosmas, Cossy >> Sent: 25 March 2022 15:22 >> To: John Chivian <[email protected]> >> Cc: rsyslog-users <[email protected]> >> Subject: RE: [rsyslog] Imifile-state File Housekeeping Query... >> >> >> Hi John, >> >> Thanks for the advice, appreciate it. >> >> Ill give it a go now. >> >> Regards, >> Cossy >> >> -----Original Message----- >> From: John Chivian <[email protected]> >> Sent: 25 March 2022 15:21 >> To: rsyslog-users <[email protected]> >> Cc: Cosmas, Cossy <[email protected]> >> Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... >> >> ! EXTERNAL MESSAGE - Think Before You Click or Download >> >> My best advice is to stop and restart rsyslog after rotating files (a HUP >> won’t do it). This makes rsyslog close and verify state files at shutdown, >> and the ones for non-existent files will then get removed at startup. >> >> Regards, >> >>> On Mar 25, 2022, at 10:10, Cosmas, Cossy via rsyslog >>> <[email protected]> wrote: >>> >>> >>> PS. We are running rsyslog 8.2006. >>> >>> -----Original Message----- >>> From: rsyslog <[email protected]> On Behalf Of Cosmas, >>> Cossy via rsyslog >>> Sent: 25 March 2022 10:02 >>> To: [email protected] >>> Cc: Cosmas, Cossy <[email protected]> >>> Subject: [rsyslog] Imifile-state File Housekeeping Query... >>> >>> ! EXTERNAL MESSAGE - Think Before You Click or Download >>> >>> >>> Dear Rsyslog Forum Users, >>> >>> A quick question from a relative newbie... >>> >>> I have configured rsyslog to monitor my applications audit log files. I >>> have also implemented a cron based housekeeping script to delete the >>> application audit log files when they are over a week old. >>> >>> I would have expected the rsyslog imifile-state files to automatically >>> reduce in number as the number of audit files decreases due to the above >>> housekeeping task but this is not the case. >>> >>> The number of imifile-state files just keeps on increasing and this is >>> problematic as we have limits around the number of open files that rsyslog >>> can maintain. >>> >>> I would just like to know what should be happening here and what is normal. >>> >>> Does rsyslog ever automatically remove redundant state files or are they >>> left in place with the user expected to implement a cron based routine to >>> delete them manually? >>> >>> Any advice appreciated. >>> >>> Thank you. >>> >>> Regards, >>> >>> Cossy Cosmas >>> Payments and Transaction Management Services Diebold Nixdorf >>> >>> Advanced notice of annual leave: >>> 20th June - 24th June >>> 22nd August - 4th September >>> >>> Mobile: +44 7717 863755 >>> One The Boulevard, Cain Road, >>> Bracknell, Berkshire, RG12 1WP >>> >>> [email protected]<mailto:[email protected] >>>> >>> DieboldNixdorf.com >>> >>> [1_twitter_logo_24px]<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=LUATZyF1IN8aMKCSuxYkfho4Vg6eU041XiNrdpyD3so&e= >>> > [2_facebook_logo_24px] >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=X2WIo3MSLhpeBcW6VFX4Sy2SM0WbkIVr7xwShNubav4&e= >>> > [3_youtube_logo_24px] >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=6wy_XQvegYq7h1tm5hOX8BXR_r-hKk9D3osb2Lz0Nro&e= >>> > [4_linkedin_logo_24px] >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__w > ww. >>> linkedin.com_company_diebold&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=j7-qHkmmXOVmynEwlaLnBhLfDyOBUuON7TbZ8mexxOw&e= >>> > [5_blog_logo_24px] <http://blog.dieboldnixdorf.com/> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net >>> _mailman_listinfo_rsyslog&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZ >>> bAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHT >>> VWykmkZNx0bHELIbg9Zzft9Q6GbmLh4t&s=FY_T0UNmcLGdLKOQg3GcZNwP-6fDEVWHKy0 >>> aplakdEE&e= >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_pr >>> ofessional-2Dservices_&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAd >>> baH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHTVWy >>> kmkZNx0bHELIbg9Zzft9Q6GbmLh4t&s=UF7Hr7h98tGBxyIDcwVhf3axMFQhzWkQSAh6T5 >>> 45tFs&e= What's up with rsyslog? Follow >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerh >>> ards&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_ >>> Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHTVWykmkZNx0bHELIbg9Zzf >>> t9Q6GbmLh4t&s=3pzUn1DEzQh35h5Uyo_0LY3g6k70GXY3BXy3KSLL55k&e= >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> >> _______________________________________________ >> rsyslog mailing list >> https://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
