Ah, I did not expect this. That is in fact what is happening. May I ask why this is the case? This is not a desirable behavior in my application - I'd have to attach sequence numbers to each message and reorder them later. Are there any options that would force in-order message delivery?
-----Original Message----- From: David Lang <[email protected]> Sent: Tuesday, February 22, 2022 2:16 PM To: David Lang <[email protected]> Cc: MACGREGOR Will <[email protected]>; MACGREGOR Will via rsyslog <[email protected]> Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with Rsyslog it's worth noting that the logs do not come out in the order that they went in. The logs in the memory queue will go out very quickly but the logs in the disk queue will go out much more slowly, so if you have the logs with a number in them, and just look at the end of the desination, the last number may be 1152, but numbers 1153-2000 may still be in the file, just much earlier in the file. David Lang On Tue, 22 Feb 2022, David Lang wrote: > Date: Tue, 22 Feb 2022 11:14:18 -0800 (PST) > From: David Lang <[email protected]> > To: MACGREGOR Will <[email protected]> > Cc: David Lang <[email protected]>, > MACGREGOR Will via rsyslog <[email protected]> > Subject: RE: [rsyslog] setting up reliable forwarding of syslog Messages with > Rsyslog > > I'll have to look at the attachment later, but what does it show about > the number of items processed by action 4? and can you get a similar > stats dump from the system it's sending to? > > if you switch to the new format, one advantage is that the action() > statement lets you give it a name, much easier to figure out what's > what rather than just 'action 4' > > David Lang > > On Tue, 22 Feb 2022, MACGREGOR Will wrote: > >> Date: Tue, 22 Feb 2022 18:38:05 +0000 >> From: MACGREGOR Will <[email protected]> >> To: David Lang <[email protected]> >> Cc: MACGREGOR Will via rsyslog <[email protected]> >> Subject: RE: [rsyslog] setting up reliable forwarding of syslog >> Messages with >> Rsyslog >> >> So in this case, I _think_ this shows the queue was holding 1152 >> messages, the memory queue was holding 848, then after starting the >> server, the memory queue appears to get emptied - if that's what this line >> means: >> >> Feb 22 13:18:03 AA3945 rsyslogd-pstats: action 4 queue: >> origin=core.queue >> size=0 enqueued=4000 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=925 >> >> But only the first 1152 messages ever come out on the server. >> >> -----Original Message----- >> From: David Lang <[email protected]> >> Sent: Tuesday, February 22, 2022 12:58 PM >> To: MACGREGOR Will <[email protected]> >> Cc: David Lang <[email protected]>; MACGREGOR Will via rsyslog >> <[email protected]> >> Subject: RE: [rsyslog] setting up reliable forwarding of syslog >> Messages with Rsyslog >> >> if you look there is an action 4 queue that also has 880 items in it, >> that's the rest of them. That's the memory queue. those should also >> be delivered once the link comes back up. >> >> what does pstats show after you bring the server back up? >> >> David Lang >> >> On Tue, 22 Feb 2022, MACGREGOR Will wrote: >> >>> Date: Tue, 22 Feb 2022 17:37:40 +0000 >>> From: MACGREGOR Will <[email protected]> >>> To: David Lang <[email protected]>, >>> MACGREGOR Will via rsyslog <[email protected]> >>> Subject: RE: [rsyslog] setting up reliable forwarding of syslog >>> Messages with >>> Rsyslog >>> >>> I've attached the output of the impstat module for the following scenario: >>> >>> 1. impstat update rate is 30 seconds >>> >>> 2. restarted rsyslog on client, with server rsyslog is disabled >>> >>> 3. attempt to queue 2000 messages (just a simple 'C' program that >>> calls syslog repeatedly) >>> >>> I can see where the DA queue only gets 1120 messages, in these two >>> entries >>> here: >>> >>> Feb 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue[DA]: >>> origin=core.queue size=3117 enqueued=1120 full=0 discarded.full=0 >>> discarded.nf=0 maxqsize=3117 Feb 22 12:24:21 AA3945 rsyslogd-pstats: >>> action 4 queue: origin=core.queue size=880 enqueued=2000 full=0 >>> discarded.full=0 discarded.nf=0 maxqsize=901 >>> >>> ---------------impstat output---------------- >>> >>> Feb 22 12:24:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb >>> 22 >>> 12:24:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock >>> submitted=2009 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb >>> 22 12:24:21 AA3945 rsyslogd-pstats: action 0: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:24:21 AA3945 rsyslogd-pstats: action 1: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:24:21 AA3945 rsyslogd-pstats: action 2: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:24:21 AA3945 rsyslogd-pstats: action 3: origin=core.action >>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 >>> Feb >>> 22 12:24:21 AA3945 rsyslogd-pstats: action 4: origin=core.action >>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 >>> Feb >>> 22 12:24:21 AA3945 rsyslogd-pstats: action 5: origin=core.action >>> processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0 >>> Feb >>> 22 12:24:21 AA3945 rsyslogd-pstats: action 6: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:24:21 AA3945 rsyslogd-pstats: action 7: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:24:21 AA3945 rsyslogd-pstats: action 8: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:24:21 AA3945 rsyslogd-pstats: action 9: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:24:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats >>> utime=29543 stime=33764 maxrss=6844 minflt=711 majflt=0 inblock=0 >>> oublock=1248 nvcsw=4870 nivcsw=313 openfiles=13 Feb 22 12:24:21 >>> AA3945 >>> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 >>> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 >>> Feb >>> 22 12:24:21 AA3945 rsyslogd-pstats: action 4 queue: >>> origin=core.queue >>> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 >>> maxqsize=901 Feb 22 12:24:21 AA3945 rsyslogd-pstats: main Q: >>> origin=core.queue size=15 enqueued=2024 full=0 discarded.full=0 >>> discarded.nf=0 maxqsize=41 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> global: origin=dynstats Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> imuxsock: origin=imuxsock submitted=2009 ratelimit.discarded=0 >>> ratelimit.numratelimiters=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> action 0: origin=core.action processed=0 failed=0 suspended=0 >>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> action 1: origin=core.action processed=16 failed=0 suspended=0 >>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> action 2: origin=core.action processed=0 failed=0 suspended=0 >>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> action 3: origin=core.action processed=2000 failed=0 suspended=0 >>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> action 4: origin=core.action processed=2000 failed=0 suspended=0 >>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> action 5: origin=core.action processed=2025 failed=0 suspended=0 >>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> action 6: origin=core.action processed=0 failed=0 suspended=0 >>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> action 7: origin=core.action processed=0 failed=0 suspended=0 >>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> action 8: origin=core.action processed=0 failed=0 suspended=0 >>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> action 9: origin=core.action processed=0 failed=0 suspended=0 >>> suspended.duration=0 resumed=0 Feb 22 12:24:51 AA3945 rsyslogd-pstats: >>> resource-usage: origin=impstats utime=39977 stime=43975 maxrss=6844 >>> minflt=717 majflt=0 inblock=0 oublock=1256 nvcsw=4992 nivcsw=313 >>> openfiles=14 Feb 22 12:24:51 AA3945 rsyslogd-pstats: action 4 >>> queue[DA]: origin=core.queue size=3117 enqueued=1120 full=0 >>> discarded.full=0 discarded.nf=0 maxqsize=3117 Feb 22 12:24:51 AA3945 >>> rsyslogd-pstats: action 4 queue: origin=core.queue size=880 >>> enqueued=2000 full=0 discarded.full=0 discarded.nf=0 maxqsize=901 >>> Feb >>> 22 12:24:51 AA3945 rsyslogd-pstats: main Q: origin=core.queue >>> size=15 >>> enqueued=2040 full=0 discarded.full=0 discarded.nf=0 maxqsize=41 Feb >>> 22 12:25:21 AA3945 rsyslogd-pstats: global: origin=dynstats Feb 22 >>> 12:25:21 AA3945 rsyslogd-pstats: imuxsock: origin=imuxsock >>> submitted=2010 ratelimit.discarded=0 ratelimit.numratelimiters=0 Feb >>> 22 12:25:21 AA3945 rsyslogd-pstats: action 0: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:25:21 AA3945 rsyslogd-pstats: action 1: origin=core.action >>> processed=32 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 12:25:21 AA3945 rsyslogd-pstats: action 2: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:25:21 AA3945 rsyslogd-pstats: action 3: origin=core.action >>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 >>> Feb >>> 22 12:25:21 AA3945 rsyslogd-pstats: action 4: origin=core.action >>> processed=2000 failed=0 suspended=0 suspended.duration=0 resumed=0 >>> Feb >>> 22 12:25:21 AA3945 rsyslogd-pstats: action 5: origin=core.action >>> processed=2042 failed=0 suspended=0 suspended.duration=0 resumed=0 >>> Feb >>> 22 12:25:21 AA3945 rsyslogd-pstats: action 6: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:25:21 AA3945 rsyslogd-pstats: action 7: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:25:21 AA3945 rsyslogd-pstats: action 8: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:25:21 AA3945 rsyslogd-pstats: action 9: origin=core.action >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Feb >>> 22 >>> 12:25:21 AA3945 rsyslogd-pstats: resource-usage: origin=impstats >>> utime=48322 stime=56376 maxrss=6844 minflt=720 majflt=0 inblock=0 >>> oublock=1280 nvcsw=5116 nivcsw=313 openfiles=14 Feb 22 12:25:21 >>> AA3945 >>> rsyslogd-pstats: action 4 queue[DA]: origin=core.queue size=3117 >>> enqueued=1120 full=0 discarded.full=0 discarded.nf=0 maxqsize=3117 >>> Feb >>> 22 12:25:21 AA3945 rsyslogd-pstats: action 4 queue: >>> origin=core.queue >>> size=880 enqueued=2000 full=0 discarded.full=0 discarded.nf=0 >>> maxqsize=901 Feb 22 12:25:21 AA3945 rsyslogd-pstats: main Q: >>> origin=core.queue size=15 enqueued=2057 full=0 discarded.full=0 >>> discarded.nf=0 maxqsize=41 >>> >>> -----Original Message----- >>> From: David Lang <[email protected]> >>> Sent: Tuesday, February 22, 2022 11:47 AM >>> To: MACGREGOR Will via rsyslog <[email protected]> >>> Cc: MACGREGOR Will <[email protected]> >>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog >>> Messages with Rsyslog >>> >>> enable impstats and post the results so that we can see what's >>> happening with the queues >>> >>> with a DA queue you have both a memory queue and a disk queue. did >>> you restart the sending system while the server was down? >>> >>> David Lang >>> >>> On Tue, 22 Feb 2022, MACGREGOR Will via rsyslog wrote: >>> >>>> Date: Tue, 22 Feb 2022 16:44:58 +0000 >>>> From: MACGREGOR Will via rsyslog <[email protected]> >>>> To: rsyslog-users <[email protected]> >>>> Cc: MACGREGOR Will <[email protected]> >>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog >>>> Messages with >>>> Rsyslog >>>> >>>> So there's still something I'm not understanding about DA queues. >>>> >>>> In my configuration, I have >>>> $ActionQueueSize 1000 >>>> $WorkDirectory /var/spool/rsyslog >>>> $ActionQueueFileName srvrfwd # set file name, also enables disk >>>> mode >>>> >>>> >>>> If I disable the server, queue < 1000 messages, then re-enable the >>>> server, all messages are delivered. >>>> >>>> If I disable the server, queue 2000 messages, then re-enable the >>>> server, only 1120 messages get delivered. >>>> >>>> I can confirm that file /var/spool/rsyslog/srvrfwd.00000001 gets >>>> created, but it seems as if it does not contain anything beyond message >>>> 1120. >>>> It's like a lot of the messages didn't get flushed to the disk queue... >>>> >>>> -----Original Message----- >>>> From: rsyslog <[email protected]> On Behalf Of >>>> Mariusz Kruk via rsyslog >>>> Sent: Tuesday, February 22, 2022 8:44 AM >>>> To: rsyslog-users <[email protected]> >>>> Cc: Mariusz Kruk <[email protected]> >>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog >>>> Messages with Rsyslog >>>> >>>> Not exactly, because with "creating a spearate ruleset" I meant a >>>> completely different RainerScript-based configuration but this one >>>> should also work as I wrote "somewhere around". >>>> >>>> Anyway, as David wrote somewhere in this thread - legacy config >>>> format is OK for simple setups where it's more readable than Rainer >>>> Script but if your config requires multiple directives modifying >>>> functionality of the action, it's probably easier to write it as >>>> (in your case) >>>> >>>> if ($syslogfacility == "local7") then >>>> action(type="omfwd" Target="wll" Port="2514" >>>> action.resumeRetryCount="0" [... more action.parameters and >>>> queue.parameters ...] ) >>>> >>>> It's more obvious then what the parameters are for and you don't >>>> have them scattered around (possibly intertwining with other >>>> parameters modifying the resulting config). >>>> >>>> MK >>>> >>>> PS: I'm not sure if this condition will work this way; there was >>>> some bug lately about textual representation but I don't recall if >>>> it was facility or severity or both. >>>> >>>> On 22.02.2022 14:31, MACGREGOR Will wrote: >>>>> What I found was that I had to do this in 50-default.conf: >>>>> >>>>> $ActionQueueType LinkedList # use asynchronous processing >>>>> $ActionQueueFileName srvrfwd # set file name, also enables disk >>>>> mode $ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1 >>>>> $ActionResumeRetryCount -1 # infinite retries on insert failure >>>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog >>>>> shuts down >>>>> >>>>> local7.* :omrelp:will:2514 >>>>> >>>>> I believe that's what you meant here, yes? >>>>>> I'd do a separate queue for this omfwd (or omrelp or whatever >>>>>> you're gonna use in the end) action alone. >>>>> When I did that, everything started to work properly. I can see >>>>> the retries happening when rsyslogd is disabled on the server. >>>>> Thanks for all your help. >>>>> >>>>> I wish I understood the configuration better. I have to admit, I >>>>> find the documentation really confusing. >>>>> >>>>> -----Original Message----- >>>>> From: rsyslog <[email protected]> On Behalf Of >>>>> Mariusz Kruk via rsyslog >>>>> Sent: Friday, February 18, 2022 3:22 PM >>>>> To: [email protected] >>>>> Cc: Mariusz Kruk <[email protected]> >>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog >>>>> Messages with Rsyslog >>>>> >>>>> If you run a client as >>>>> >>>>> rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry >>>>> >>>>> You should see some text blob at the start but then, when the >>>>> server is running, the client should not emit any more messages. >>>>> >>>>> But when you stop the server, the client should start emiting >>>>> messages like >>>>> >>>>> 5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c: >>>>> actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0, >>>>> ResumeInRow 1 >>>>> rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused >>>>> [v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ] >>>>> 5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c: >>>>> actionDoRetry: action-0-builtin:omfwd action->tryResume returned >>>>> -2007 5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c: >>>>> actionDoRetry: action-0-builtin:omfwd check for max retries, >>>>> iResumeRetryCount -1, iRetries 0 >>>>> >>>>> And if you look for the string '<somenumber> messages' in debug >>>>> log, if you close the client some time after stopping the server >>>>> and pushing some more messages to the client, you should get >>>>> something like >>>>> >>>>> rsyslog internal message (6,-2041): action-0-builtin:omfwd queue: >>>>> queue holds 2 messages after shutdown of workers. >>>>> queue.saveonshutdown is set, so data will now be spooled to disk >>>>> [v8.2102.0-4.fc35 try >>>>> https://www.rsyslog.com/e/2041 ] >>>>> >>>>> I'm not fully sure, however, since you use the legacy config >>>>> format what's the interaction between both actions within the same >>>>> queue. In order to be sure to have proper queueing _on the >>>>> forwarding action_ I'd do a separate queue for this omfwd (or >>>>> omrelp or whatever you're gonna use in the end) action alone. >>>>> >>>>> On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote: >>>>>> So, following your advice, I've confirmed the following >>>>>> >>>>>> 1. I switched to RELP. as per the following: >>>>>> >>>>>> add the following to server rsyslog.conf >>>>>> >>>>>> module(load="imrelp") >>>>>> input(type="imrelp" port="2514" maxDataSize="10k" >>>>>> keepAlive="on") >>>>>> >>>>>> add the following to server 50-default.conf: >>>>>> >>>>>> local7.* -/var/log/local7.log >>>>>> >>>>>> add the following to client 50-default.conf >>>>>> >>>>>> local7.* -/var/log/local7.log >>>>>> local7.* :omrelp:<server>:2514 >>>>>> >>>>>> 2. I've confirmed that /var/spool/rsyslog exists; however, I was >>>>>> only buffering one or two messages so the queue file would never >>>>>> be created. >>>>>> >>>>>> 3. On my client, $RepeatedMsgReduction defaults to "on". I had >>>>>> to explicitly turn it off in rsyslog.conf so duplicates do not >>>>>> get rolled up >>>>>> >>>>>> Here's exactly how I tested: >>>>>> >>>>>> 1. log a message from the client, verify that it shows up on the >>>>>> server >>>>>> # logger -p local7.info -s 'hello world' >>>>>> >>>>>> shows up in /var/log/local7.log on the server >>>>>> shows up in /var/log/local7.log on the client >>>>>> >>>>>> 2. disable rsyslog on the server >>>>>> # systemctl stop syslog.socket rsyslog.service >>>>>> >>>>>> 3. log a message on the client >>>>>> # logger -p local7.info -s 'hello world 2' >>>>>> >>>>>> shows up in /var/log/local7.log on the client >>>>>> >>>>>> 4. enable rsyslog on the server >>>>>> # systemctl start syslog.socket rsyslog.service >>>>>> >>>>>> 5. log a message on the client >>>>>> # logger -p local7.info -s 'hello world 3' >>>>>> >>>>>> shows up in /var/log/local7.log on the server >>>>>> shows up in /var/log/local7.log on the client >>>>>> >>>>>> "hello world 3" comes out on the server. "hello world 2" does not. >>>>>> Note that the server is only down for a few seconds in this scenario. >>>>>> >>>>>> I tried setting $ActionResumeInterval 1 on the client, and I've >>>>>> tried running syslogd in debug mode, but frankly I don't >>>>>> understand the output very well and have no idea what I'm looking >>>>>> for. I don't see anything that would suggest the message is >>>>>> being queued on the client when the server is down as in step 3, >>>>>> but again, I'm not sure how that would show up in the debug trace. >>>>>> >>>>>> There must be something I'm doing wrong, but what? >>>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog <[email protected]> On Behalf Of >>>>>> Mariusz Kruk via rsyslog >>>>>> Sent: Friday, February 18, 2022 4:18 AM >>>>>> To: [email protected] >>>>>> Cc: Mariusz Kruk <[email protected]> >>>>>> Subject: Re: [rsyslog] setting up reliable forwarding of syslog >>>>>> Messages with Rsyslog >>>>>> >>>>>> Firstly, after you confirm that your queueing works properly, I'd >>>>>> advise you to switch to RELP so you have "more reliability". >>>>>> >>>>>> But regarding your setup - as you defined >>>>>> >>>>>> $WorkDirectory /var/spool/rsyslog >>>>>> >>>>>> Your queue should be placed there. >>>>>> >>>>>> Question is whether you do indeed have such directory in your system. >>>>>> Because if you don't, the rsyslog daemon won't be able to save >>>>>> the queue contents. >>>>>> >>>>>> But in case of just a few messages you shouldn't be saving the >>>>>> contents do disk at all. (it would be saved when you have unsent >>>>>> messages and shut down the rsyslog daemon). >>>>>> >>>>>> Also, notice that >>>>>> https://www.rsyslog.com/doc/master/configuration/action/rsconf1_r >>>>>> ep e a t edmsgreduction.html "This parameter models old sysklogd >>>>>> legacy. >>>>>> *Note that many people, including the rsyslog authors, consider >>>>>> this to be a misfeature.* See /Discussion/ below to learn why." >>>>>> >>>>>> But in general, the setup should work... with one caveat. Your "never" >>>>>> might in fact not be "never". You didn't tweak the settings that >>>>>> control action resuming so they are at default 30 second initial >>>>>> interval which is getting raised after every 10 tries up to a >>>>>> default >>>>>> 1800 seconds. So if the server was off for long enough, the >>>>>> client might simply have paused sending for a really significant time. >>>>>> >>>>>> See the description of parameters at >>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters. >>>>>> >>>>>> You might set (just for test! you probably don't want to set it >>>>>> in prod for that often) >>>>>> >>>>>> $ActionResumeInterval 1 >>>>>> >>>>>> And then run your client instance in debug mode to see >>>>>> interactively what it's trying to do. >>>>>> >>>>>> rsyslogd -f rsyslog.conf -i NONE -n -d >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote: >>>>>>> I'm new to rsyslog, and I'm trying to set up reliable forwarding >>>>>>> of syslog messages with rsyslog according to these instructions: >>>>>>> >>>>>>> https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding >>>>>>> .h >>>>>>> t >>>>>>> m >>>>>>> l >>>>>>> >>>>>>> I confirm that remote logging is working initially by doing >>>>>>> >>>>>>> # logger "hello, world" >>>>>>> >>>>>>> on the client, and verifying that this message shows up in the >>>>>>> server (in this case in /var/log/syslog) >>>>>>> >>>>>>> I then shut down the rsyslog server, and log a few more messages >>>>>>> on the client. As expected, these are not showing up on the >>>>>>> server side any more. On the client, they seem to be going to >>>>>>> its /var/log/syslog file; I have no idea where (if) they're being >>>>>>> queued. >>>>>>> >>>>>>> I then re-enable the rsyslog server, but the entries that I >>>>>>> wrote on the client never seem to make it back to the server. >>>>>>> What am I doing wrong? >>>>>>> >>>>>>> Some configuration files: >>>>>>> >>>>>>> ---------------------------------------------------------------- >>>>>>> -- >>>>>>> - >>>>>>> - >>>>>>> - >>>>>>> - >>>>>>> ---------------------- >>>>>>> client rsyslog.conf file: >>>>>>> >>>>>>> # /etc/rsyslog.conf Configuration file for rsyslog. >>>>>>> # >>>>>>> # For more information see >>>>>>> # >>>>>>> /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html >>>>>>> # >>>>>>> # Default logging rules can be found in >>>>>>> /etc/rsyslog.d/50-default.conf >>>>>>> >>>>>>> >>>>>>> ################# >>>>>>> #### MODULES #### >>>>>>> ################# >>>>>>> >>>>>>> module(load="imuxsock") # provides support for local system >>>>>>> logging >>>>>>> #module(load="immark") # provides --MARK-- message capability >>>>>>> >>>>>>> # provides UDP syslog reception >>>>>>> #module(load="imudp") >>>>>>> #input(type="imudp" port="514") >>>>>>> >>>>>>> # provides TCP syslog reception >>>>>>> #module(load="imtcp") >>>>>>> #input(type="imtcp" port="514") >>>>>>> >>>>>>> # provides kernel logging support and enable non-kernel klog >>>>>>> messages module(load="imklog" permitnonkernelfacility="on") >>>>>>> >>>>>>> ########################### >>>>>>> #### GLOBAL DIRECTIVES #### >>>>>>> ########################### >>>>>>> >>>>>>> # >>>>>>> # Use traditional timestamp format. >>>>>>> # To enable high precision timestamps, comment out the following line. >>>>>>> # >>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >>>>>>> >>>>>>> # Filter duplicated messages >>>>>>> $RepeatedMsgReduction on >>>>>>> >>>>>>> # >>>>>>> # Set the default permissions for all log files. >>>>>>> # >>>>>>> $FileOwner syslog >>>>>>> $FileGroup adm >>>>>>> $FileCreateMode 0640 >>>>>>> $DirCreateMode 0755 >>>>>>> $Umask 0022 >>>>>>> $PrivDropToUser syslog >>>>>>> $PrivDropToGroup syslog >>>>>>> >>>>>>> # >>>>>>> # Where to place spool and state files # $WorkDirectory >>>>>>> /var/spool/rsyslog >>>>>>> >>>>>>> # >>>>>>> # setup reliable local buffering # $ActionQueueType LinkedList # >>>>>>> use asynchronous processing $ActionQueueFileName srvrfwd # set >>>>>>> file name, also enables disk mode $ActionResumeRetryCount -1 # >>>>>>> infinite retries on insert failure $ActionQueueSaveOnShutdown on >>>>>>> # save in-memory data if rsyslog shuts down >>>>>>> >>>>>>> # >>>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig >>>>>>> /etc/rsyslog.d/*.conf >>>>>>> *.* @@<redacted>:514 >>>>>>> >>>>>>> ---------------------------------------------------------------- >>>>>>> -- >>>>>>> server rsyslog.conf file >>>>>>> >>>>>>> # /etc/rsyslog.conf Configuration file for rsyslog. >>>>>>> # >>>>>>> # For more information see >>>>>>> # >>>>>>> /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html >>>>>>> # >>>>>>> # Default logging rules can be found in >>>>>>> /etc/rsyslog.d/50-default.conf >>>>>>> >>>>>>> >>>>>>> ################# >>>>>>> #### MODULES #### >>>>>>> ################# >>>>>>> >>>>>>> module(load="imuxsock") # provides support for local system >>>>>>> logging >>>>>>> #module(load="immark") # provides --MARK-- message capability >>>>>>> >>>>>>> # provides UDP syslog reception >>>>>>> #module(load="imudp") >>>>>>> #input(type="imudp" port="514") >>>>>>> >>>>>>> # provides TCP syslog reception >>>>>>> module(load="imtcp") >>>>>>> input(type="imtcp" port="514") >>>>>>> >>>>>>> # provides kernel logging support and enable non-kernel klog >>>>>>> messages module(load="imklog" permitnonkernelfacility="on") >>>>>>> >>>>>>> ########################### >>>>>>> #### GLOBAL DIRECTIVES #### >>>>>>> ########################### >>>>>>> >>>>>>> # >>>>>>> # Use traditional timestamp format. >>>>>>> # To enable high precision timestamps, comment out the following line. >>>>>>> # >>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >>>>>>> >>>>>>> # Filter duplicated messages >>>>>>> $RepeatedMsgReduction on >>>>>>> >>>>>>> # >>>>>>> # Set the default permissions for all log files. >>>>>>> # >>>>>>> $FileOwner syslog >>>>>>> $FileGroup adm >>>>>>> $FileCreateMode 0640 >>>>>>> $DirCreateMode 0755 >>>>>>> $Umask 0022 >>>>>>> $PrivDropToUser syslog >>>>>>> $PrivDropToGroup syslog >>>>>>> >>>>>>> # >>>>>>> # Where to place spool and state files # $WorkDirectory >>>>>>> /var/spool/rsyslog >>>>>>> >>>>>>> # >>>>>>> # Include all config files in /etc/rsyslog.d/ # $IncludeConfig >>>>>>> /etc/rsyslog.d/*.conf >>>>>>> >>>>>>> ---------------------------------------------------------------- >>>>>>> -- version info for rsyslogd (both machines running Ubuntu >>>>>>> 18.04, >>>>>>> FWIW) >>>>>>> >>>>>>> # rsyslogd -version (same version for both client and server) >>>>>>> >>>>>>> rsyslogd 8.32.0, compiled with: >>>>>>> PLATFORM: x86_64-pc-linux-gnu >>>>>>> PLATFORM (lsb_release -d): >>>>>>> FEATURE_REGEXP: Yes >>>>>>> GSSAPI Kerberos 5 support: Yes >>>>>>> FEATURE_DEBUG (debug build, slow code): No >>>>>>> 32bit Atomic operations supported: Yes >>>>>>> 64bit Atomic operations supported: Yes >>>>>>> memory allocator: system default >>>>>>> Runtime Instrumentation (slow code): No >>>>>>> uuid support: Yes >>>>>>> systemd support: Yes >>>>>>> Number of Bits in RainerScript integers: 64 >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Followhttps://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >>>>>>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO >>>>>>> NOT POST if you DON'T LIKE THAT. >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >>>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>> you DON'T LIKE THAT. >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>> POST if you DON'T LIKE THAT. >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>> you DON'T LIKE THAT. >>>> _______________________________________________ >>>> rsyslog mailing list >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> _______________________________________________ >>>> rsyslog mailing list >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you DON'T LIKE THAT. >>> > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
