Hmm, commenting out the module imjournal "fixed" it. Emanuel
From: Emanuel Machado Sent: Friday, June 20, 2025 11:44 AM To: [email protected] Subject: systemd vs rsyslogd message trimming and stopping Hi, I wonder if you have noticed this before. We are migrating our apps from Centos 6 and 7 to RHEL 9.x. Our apps, because of their interplay, spit out many log messages via syslog(). Our apps are running in two computers, and the messages are not transferred between them, i.e. these are all local logs. At one time, in the past, we noticed that a fraction of the messages were being skipped, so we dialed the settings so that that would not happen. So, all it was well then. In setting up things for RHEL 9.x, we noticed that when our apps were running normally, spewing their normal, but high, quantity of log messages, the messages would stop after a while. All of them. Initially I saw that there was a significant trimming of messages, which we fixed with tweaking the rate limiting knobs (see below rsyslog.conf relevant part). But I couldn't understand why all messages would stop after a while. Restarting rsyslogd through systemctl gets it running again. In looking at the documentation, I saw the option to start rsyslogd in interactive mode. So, I stopped rsyslogd through systemctl and started it interactively (-n). Now everything is great, as I don't see the loss of messages anymore. This is seen with the same rsyslogd configuration settings as before. So, it seems to me that systemd is doing something that affects rsyslogd. Now, I know that this is not strictly a rsyslog problem, but am I alone in noticing this? Here is the relevant information from rsyslog.conf: #### GLOBAL DIRECTIVES #### # Where to place auxiliary files global(workDirectory="/var/lib/rsyslog") # Use default timestamp format module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat") #### MODULES #### module(load="imuxsock" # provides support for local system logging (e.g. via logger command) #SysSock.Use="off" # Turn off message reception via local log socket; SysSock.RateLimit.Interval="0" # turn of rate limiting SysSock.RateLimit.Burst="0") # turn of rate limiting # local messages are retrieved through imjournal now. module(load="imjournal" # provides access to the systemd journal UsePid="system" # PID nummber is retrieved as the ID of the process the journal entry originates from FileCreateMode="0644" # Set the access permissions for the state file StateFile="imjournal.state") # File to store the position in the journal And here is the rsyslog.service information: [Unit] Description=System Logging Service ;Requires=syslog.socket Wants=network.target network-online.target After=network.target network-online.target Documentation=man:rsyslogd(8) Documentation=https://www.rsyslog.com/doc/ [Service] Type=notify EnvironmentFile=-/etc/sysconfig/rsyslog ExecStart=/usr/sbin/rsyslogd -n $SYSLOGD_OPTIONS ExecReload=/usr/bin/kill -HUP $MAINPID UMask=0066 StandardOutput=null Restart=on-failure RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX RestrictNamespaces=net NoNewPrivileges=yes ProtectControlGroups=yes ProtectHome=read-only ProtectKernelModules=yes ProtectKernelTunables=yes RestrictSUIDSGID=yes SystemCallArchitectures=native SystemCallFilter=~@clock @debug @module @raw-io @reboot @swap @cpu-emulation @obsolete LockPersonality=yes MemoryDenyWriteExecute=yes # Increase the default a bit in order to allow many simultaneous # files to be monitored, we might need a lot of fds. LimitNOFILE=16384 [Install] WantedBy=multi-user.target ;Alias=syslog.service Any information, suggestions on where to look, or other recommendations would be greatly appreciated. Many thanks in advance, Emanuel Please note the new addressing info [cid:[email protected]] Emanuel Machado, PhD | Senior Electrical Engineering Lead 10 Maguire Rd, Building 3, Suite 310, Lexington, MA 02421, USA Desk: +1 (617) 693-9983 | [email protected] www.cytonome.com<http://www.cytonome.com/> This email message may contain confidential and/or privileged information concerning Cytonome/ST, LLC, dba Cytonome. If the reader of this message is not the intended recipient, you are hereby notified that any review, distribution, retention, or copying of this communication is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you. Please consider the environment before printing this email. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Disclaimer: This electronic transmission (and/or the documents accompanying it) may contain CONFIDENTIAL information belonging to the sender that is protected by the Electronic Communications Privacy Act, 18 U.S.C. Sections 2510 and 2521 and may be proprietary. This message(and any attached files) is INTENDED ONLY FOR THE USE OF THE INTENDED INDIVIDUAL or entity to which it is addressed and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, use, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this communication in error, please notify Inguran LLC, dba Sexing Technologies immediately by telephone (936-870-3960) and destroy the original message and any related attachments. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
_______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
