What I found was that I had to do this in 50-default.conf: $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName srvrfwd # set file name, also enables disk mode $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1 $ActionResumeRetryCount -1 # infinite retries on insert failure $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
local7.* :omrelp:will:2514 I believe that's what you meant here, yes? >I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use >in the end) action alone. When I did that, everything started to work properly. I can see the retries happening when rsyslogd is disabled on the server. Thanks for all your help. I wish I understood the configuration better. I have to admit, I find the documentation really confusing. -----Original Message----- From: rsyslog <[email protected]> On Behalf Of Mariusz Kruk via rsyslog Sent: Friday, February 18, 2022 3:22 PM To: [email protected] Cc: Mariusz Kruk <[email protected]> Subject: Re: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog If you run a client as rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry You should see some text blob at the start but then, when the server is running, the client should not emit any more messages. But when you stop the server, the client should start emiting messages like 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c: actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0, ResumeInRow 1 rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c: actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c: actionDoRetry: action-0-builtin:omfwd check for max retries, iResumeRetryCount -1, iRetries 0 And if you look for the string '<somenumber> messages' in debug log, if you close the client some time after stopping the server and pushing some more messages to the client, you should get something like rsyslog internal message (6,-2041): action-0-builtin:omfwd queue: queue holds 2 messages after shutdown of workers. queue.saveonshutdown is set, so data will now be spooled to disk [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2041 ] I'm not fully sure, however, since you use the legacy config format what's the interaction between both actions within the same queue. In order to be sure to have proper queueing _on the forwarding action_ I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use in the end) action alone. On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote: > So, following your advice, I've confirmed the following > > 1. I switched to RELP. as per the following: > > add the following to server rsyslog.conf > > module(load="imrelp") > input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on") > > add the following to server 50-default.conf: > > local7.* -/var/log/local7.log > > add the following to client 50-default.conf > > local7.* -/var/log/local7.log > local7.* :omrelp:<server>:2514 > > 2. I've confirmed that /var/spool/rsyslog exists; however, I was only > buffering one or two messages so the queue file would never be created. > > 3. On my client, $RepeatedMsgReduction defaults to "on". I had to > explicitly turn it off in rsyslog.conf so duplicates do not get rolled up > > Here's exactly how I tested: > > 1. log a message from the client, verify that it shows up on the server > # logger -p local7.info -s 'hello world' > > shows up in /var/log/local7.log on the server > shows up in /var/log/local7.log on the client > > 2. disable rsyslog on the server > # systemctl stop syslog.socket rsyslog.service > > 3. log a message on the client > # logger -p local7.info -s 'hello world 2' > > shows up in /var/log/local7.log on the client > > 4. enable rsyslog on the server > # systemctl start syslog.socket rsyslog.service > > 5. log a message on the client > # logger -p local7.info -s 'hello world 3' > > shows up in /var/log/local7.log on the server > shows up in /var/log/local7.log on the client > > "hello world 3" comes out on the server. "hello world 2" does not. > Note that the server is only down for a few seconds in this scenario. > > I tried setting $ActionResumeInterval 1 on the client, and I've tried running > syslogd in debug mode, but frankly I don't understand the output very well > and have no idea what I'm looking for. I don't see anything that would > suggest the message is being queued on the client when the server is down as > in step 3, but again, I'm not sure how that would show up in the debug trace. > > There must be something I'm doing wrong, but what? > > -----Original Message----- > From: rsyslog <[email protected]> On Behalf Of Mariusz > Kruk via rsyslog > Sent: Friday, February 18, 2022 4:18 AM > To: [email protected] > Cc: Mariusz Kruk <[email protected]> > Subject: Re: [rsyslog] setting up reliable forwarding of syslog > Messages with Rsyslog > > Firstly, after you confirm that your queueing works properly, I'd advise you > to switch to RELP so you have "more reliability". > > But regarding your setup - as you defined > > $WorkDirectory /var/spool/rsyslog > > Your queue should be placed there. > > Question is whether you do indeed have such directory in your system. > Because if you don't, the rsyslog daemon won't be able to save the queue > contents. > > But in case of just a few messages you shouldn't be saving the contents do > disk at all. (it would be saved when you have unsent messages and shut down > the rsyslog daemon). > > Also, notice that > https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repeat > edmsgreduction.html "This parameter models old sysklogd legacy. *Note > that many people, including the rsyslog authors, consider this to be a > misfeature.* See /Discussion/ below to learn why." > > But in general, the setup should work... with one caveat. Your "never" > might in fact not be "never". You didn't tweak the settings that control > action resuming so they are at default 30 second initial interval which is > getting raised after every 10 tries up to a default 1800 seconds. So if the > server was off for long enough, the client might simply have paused sending > for a really significant time. > > See the description of parameters at > https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters. > > You might set (just for test! you probably don't want to set it in > prod for that often) > > $ActionResumeInterval 1 > > And then run your client instance in debug mode to see interactively what > it's trying to do. > > rsyslogd -f rsyslog.conf -i NONE -n -d > > > > > On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote: >> I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog >> messages with rsyslog according to these instructions: >> >> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.html >> >> I confirm that remote logging is working initially by doing >> >> # logger "hello, world" >> >> on the client, and verifying that this message shows up in the server >> (in this case in /var/log/syslog) >> >> I then shut down the rsyslog server, and log a few more messages on the >> client. As expected, these are not showing up on the server side any more. >> On the client, they seem to be going to its /var/log/syslog file; I have no >> idea where (if) they're being queued. >> >> I then re-enable the rsyslog server, but the entries that I wrote on the >> client never seem to make it back to the server. What am I doing wrong? >> >> Some configuration files: >> >> --------------------------------------------------------------------- >> - >> ---------------------- >> client rsyslog.conf file: >> >> # /etc/rsyslog.conf Configuration file for rsyslog. >> # >> # For more information see >> # >> /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html >> # >> # Default logging rules can be found in >> /etc/rsyslog.d/50-default.conf >> >> >> ################# >> #### MODULES #### >> ################# >> >> module(load="imuxsock") # provides support for local system logging >> #module(load="immark") # provides --MARK-- message capability >> >> # provides UDP syslog reception >> #module(load="imudp") >> #input(type="imudp" port="514") >> >> # provides TCP syslog reception >> #module(load="imtcp") >> #input(type="imtcp" port="514") >> >> # provides kernel logging support and enable non-kernel klog messages >> module(load="imklog" permitnonkernelfacility="on") >> >> ########################### >> #### GLOBAL DIRECTIVES #### >> ########################### >> >> # >> # Use traditional timestamp format. >> # To enable high precision timestamps, comment out the following line. >> # >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> >> # Filter duplicated messages >> $RepeatedMsgReduction on >> >> # >> # Set the default permissions for all log files. >> # >> $FileOwner syslog >> $FileGroup adm >> $FileCreateMode 0640 >> $DirCreateMode 0755 >> $Umask 0022 >> $PrivDropToUser syslog >> $PrivDropToGroup syslog >> >> # >> # Where to place spool and state files # $WorkDirectory >> /var/spool/rsyslog >> >> # >> # setup reliable local buffering >> # >> $ActionQueueType LinkedList # use asynchronous processing >> $ActionQueueFileName srvrfwd # set file name, also enables disk mode >> $ActionResumeRetryCount -1 # infinite retries on insert failure >> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts >> down >> >> # >> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig >> /etc/rsyslog.d/*.conf >> *.* @@<redacted>:514 >> >> ------------------------------------------------------------------ >> server rsyslog.conf file >> >> # /etc/rsyslog.conf Configuration file for rsyslog. >> # >> # For more information see >> # >> /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html >> # >> # Default logging rules can be found in >> /etc/rsyslog.d/50-default.conf >> >> >> ################# >> #### MODULES #### >> ################# >> >> module(load="imuxsock") # provides support for local system logging >> #module(load="immark") # provides --MARK-- message capability >> >> # provides UDP syslog reception >> #module(load="imudp") >> #input(type="imudp" port="514") >> >> # provides TCP syslog reception >> module(load="imtcp") >> input(type="imtcp" port="514") >> >> # provides kernel logging support and enable non-kernel klog messages >> module(load="imklog" permitnonkernelfacility="on") >> >> ########################### >> #### GLOBAL DIRECTIVES #### >> ########################### >> >> # >> # Use traditional timestamp format. >> # To enable high precision timestamps, comment out the following line. >> # >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> >> # Filter duplicated messages >> $RepeatedMsgReduction on >> >> # >> # Set the default permissions for all log files. >> # >> $FileOwner syslog >> $FileGroup adm >> $FileCreateMode 0640 >> $DirCreateMode 0755 >> $Umask 0022 >> $PrivDropToUser syslog >> $PrivDropToGroup syslog >> >> # >> # Where to place spool and state files # $WorkDirectory >> /var/spool/rsyslog >> >> # >> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig >> /etc/rsyslog.d/*.conf >> >> ------------------------------------------------------------------ >> version info for rsyslogd (both machines running Ubuntu 18.04, FWIW) >> >> # rsyslogd -version (same version for both client and server) >> >> rsyslogd 8.32.0, compiled with: >> PLATFORM: x86_64-pc-linux-gnu >> PLATFORM (lsb_release -d): >> FEATURE_REGEXP: Yes >> GSSAPI Kerberos 5 support: Yes >> FEATURE_DEBUG (debug build, slow code): No >> 32bit Atomic operations supported: Yes >> 64bit Atomic operations supported: Yes >> memory allocator: system default >> Runtime Instrumentation (slow code): No >> uuid support: Yes >> systemd support: Yes >> Number of Bits in RainerScript integers: 64 >> _______________________________________________ >> rsyslog mailing list >> https://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Followhttps://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This > is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our > control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
