new parameter that was added deleteStateOnFileDelete
you will need to be running a pretty current version to have that.
David Lang
On Fri, 25 Mar 2022, David Lang wrote:
Date: Fri, 25 Mar 2022 11:48:02 -0700 (PDT)
From: David Lang <[email protected]>
To: John Chivian <[email protected]>
Cc: David Lang <[email protected]>,
"Cosmas, Cossy via rsyslog" <[email protected]>,
"Cosmas, Cossy" <[email protected]>
Subject: Re: [rsyslog] Imifile-state File Housekeeping Query...
I would suggest that your cleanup process also delete old state files, then
if you restart rsyslog weekly, any state files that have been deleted, but
that rsyslog had open, will be purged by the OS (they are relatively small,
so infrequent restarts should work, you don't want full restarts frequently,
because there is a window during the restart where rsyslog cannot proccess
logs)
David Lang
On Fri, 25 Mar 2022, John Chivian wrote:
Date: Fri, 25 Mar 2022 13:42:33 -0500
From: John Chivian <[email protected]>
To: David Lang <[email protected]>
Cc: "Cosmas, Cossy via rsyslog" <[email protected]>,
"Cosmas, Cossy" <[email protected]>
Subject: Re: [rsyslog] Imifile-state File Housekeeping Query...
Excellent, thank you David! Our rotation methodology is confirmed as
required until rsyslog supports deleting orphan state files at either
startup or shutdown, which to my way of thinking is a high priority add.
We have some cleanup tasks that spin every six hours because clients drop
dated files as often as every minute.
Regards,
On Mar 25, 2022, at 13:30, David Lang <[email protected]> wrote:
rsyslog doesn't delete the old state files because it doesn't know if the
file is going to come back in a few seconds or not (such things happen),
so the decision was made to keep the files around as a lesser evil than
re-ingesting an old file that reappears.
for exactly this duplicate ingestion issue, I would suggest you either
rotate the file to a directory that rsyslog is not watching, or rotate it
to a filename that rsyslog is not watching.
If you app is creating files that have a date in their name, and creating
new files over time, this approach doesn't work, but that's a perfect
example of where rsyslog may think it's done with a file, but that it may
reappear (either because it's a new file created because the app is
confused with the date, or because someone restored it from elsewhere to
look at it)
There is discussion of this problem elsewhere and talk of adding an option
to have rsyslog remove state files where no file exists to avoid the
'leak' of state files, but opening up the risk of duplicate ingestion.
David Lang
On Fri, 25 Mar 2022, Cosmas, Cossy via rsyslog wrote:
Date: Fri, 25 Mar 2022 15:31:12 +0000
From: "Cosmas, Cossy via rsyslog" <[email protected]>
To: John Chivian <[email protected]>
Cc: "Cosmas, Cossy" <[email protected]>,
rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Imifile-state File Housekeeping Query...
Hi John,
I stopped (systemctl stop rsyslog.service) and then started(systemctl
start rsyslog.service) the rsyslog service but that hasn't reduced the
number of state files.
When you say rotate the files are you referring to the cron tab job that
is deleting all of the monitored audit log files?
That activity takes place overnight.
Regards,
Cossy
-----Original Message-----
From: Cosmas, Cossy
Sent: 25 March 2022 15:22
To: John Chivian <[email protected]>
Cc: rsyslog-users <[email protected]>
Subject: RE: [rsyslog] Imifile-state File Housekeeping Query...
Hi John,
Thanks for the advice, appreciate it.
Ill give it a go now.
Regards,
Cossy
-----Original Message-----
From: John Chivian <[email protected]>
Sent: 25 March 2022 15:21
To: rsyslog-users <[email protected]>
Cc: Cosmas, Cossy <[email protected]>
Subject: Re: [rsyslog] Imifile-state File Housekeeping Query...
! EXTERNAL MESSAGE - Think Before You Click or Download
My best advice is to stop and restart rsyslog after rotating files (a HUP
won’t do it). This makes rsyslog close and verify state files at
shutdown, and the ones for non-existent files will then get removed at
startup.
Regards,
On Mar 25, 2022, at 10:10, Cosmas, Cossy via rsyslog
<[email protected]> wrote:
PS. We are running rsyslog 8.2006.
-----Original Message-----
From: rsyslog <[email protected]> On Behalf Of Cosmas,
Cossy via rsyslog
Sent: 25 March 2022 10:02
To: [email protected]
Cc: Cosmas, Cossy <[email protected]>
Subject: [rsyslog] Imifile-state File Housekeeping Query...
! EXTERNAL MESSAGE - Think Before You Click or Download
Dear Rsyslog Forum Users,
A quick question from a relative newbie...
I have configured rsyslog to monitor my applications audit log files. I
have also implemented a cron based housekeeping script to delete the
application audit log files when they are over a week old.
I would have expected the rsyslog imifile-state files to automatically
reduce in number as the number of audit files decreases due to the above
housekeeping task but this is not the case.
The number of imifile-state files just keeps on increasing and this is
problematic as we have limits around the number of open files that
rsyslog can maintain.
I would just like to know what should be happening here and what is
normal.
Does rsyslog ever automatically remove redundant state files or are they
left in place with the user expected to implement a cron based routine
to delete them manually?
Any advice appreciated.
Thank you.
Regards,
Cossy Cosmas
Payments and Transaction Management Services Diebold Nixdorf
Advanced notice of annual leave:
20th June - 24th June
22nd August - 4th September
Mobile: +44 7717 863755
One The Boulevard, Cain Road,
Bracknell, Berkshire, RG12 1WP
[email protected]<mailto:[email protected]
DieboldNixdorf.com
[1_twitter_logo_24px]<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=LUATZyF1IN8aMKCSuxYkfho4Vg6eU041XiNrdpyD3so&e=
> [2_facebook_logo_24px]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=X2WIo3MSLhpeBcW6VFX4Sy2SM0WbkIVr7xwShNubav4&e=
> [3_youtube_logo_24px]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=6wy_XQvegYq7h1tm5hOX8BXR_r-hKk9D3osb2Lz0Nro&e=
> [4_linkedin_logo_24px]
<https://urldefense.proofpoint.com/v2/url?u=https-3A_
_w
ww.
linkedin.com_company_diebold&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=j7-qHkmmXOVmynEwlaLnBhLfDyOBUuON7TbZ8mexxOw&e=
> [5_blog_logo_24px] <http://blog.dieboldnixdorf.com/>
_______________________________________________
rsyslog mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net
_mailman_listinfo_rsyslog&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZ
bAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHT
VWykmkZNx0bHELIbg9Zzft9Q6GbmLh4t&s=FY_T0UNmcLGdLKOQg3GcZNwP-6fDEVWHKy0
aplakdEE&e=
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_pr
ofessional-2Dservices_&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAd
baH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHTVWy
kmkZNx0bHELIbg9Zzft9Q6GbmLh4t&s=UF7Hr7h98tGBxyIDcwVhf3axMFQhzWkQSAh6T5
45tFs&e= What's up with rsyslog? Follow
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerh
ards&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_
Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHTVWykmkZNx0bHELIbg9Zzf
t9Q6GbmLh4t&s=3pzUn1DEzQh35h5Uyo_0LY3g6k70GXY3BXy3KSLL55k&e=
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
