What I found was that I had to do this in 50-default.conf:
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionQueueMaxDiskSpace 1g $ActionResumeInterval 1
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts
down
local7.* :omrelp:will:2514
I believe that's what you meant here, yes?
I'd do a separate queue for this omfwd (or omrelp or whatever you're gonna use
in the end) action alone.
When I did that, everything started to work properly. I can see the retries
happening when rsyslogd is disabled on the server. Thanks for all your help.
I wish I understood the configuration better. I have to admit, I find the
documentation really confusing.
-----Original Message-----
From: rsyslog <[email protected]> On Behalf Of
Mariusz Kruk via rsyslog
Sent: Friday, February 18, 2022 3:22 PM
To: [email protected]
Cc: Mariusz Kruk <[email protected]>
Subject: Re: [rsyslog] setting up reliable forwarding of syslog
Messages with Rsyslog
If you run a client as
rsyslogd -f rsyslog.conf -i NONE -n -d | grep actionDoRetry
You should see some text blob at the start but then, when the server is
running, the client should not emit any more messages.
But when you stop the server, the client should start emiting
messages like
5207.132709967:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
actionDoRetry: action-0-builtin:omfwd enter loop, iRetries=0,
ResumeInRow 1
rsyslogd: cannot connect to 127.0.0.1:10514: Connection refused
[v8.2102.0-4.fc35 try https://www.rsyslog.com/e/2027 ]
5207.133205763:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
actionDoRetry: action-0-builtin:omfwd action->tryResume returned -2007
5207.133209346:action-0-builtin:omfwd queue:Reg/w0: ../action.c:
actionDoRetry: action-0-builtin:omfwd check for max retries,
iResumeRetryCount -1, iRetries 0
And if you look for the string '<somenumber> messages' in debug log,
if you close the client some time after stopping the server and
pushing some more messages to the client, you should get something
like
rsyslog internal message (6,-2041): action-0-builtin:omfwd queue:
queue holds 2 messages after shutdown of workers.
queue.saveonshutdown is set, so data will now be spooled to disk
[v8.2102.0-4.fc35 try
https://www.rsyslog.com/e/2041 ]
I'm not fully sure, however, since you use the legacy config format what's the
interaction between both actions within the same queue. In order to be sure to
have proper queueing _on the forwarding action_ I'd do a separate queue for
this omfwd (or omrelp or whatever you're gonna use in the end) action alone.
On 18.02.2022 17:47, MACGREGOR Will via rsyslog wrote:
So, following your advice, I've confirmed the following
1. I switched to RELP. as per the following:
add the following to server rsyslog.conf
module(load="imrelp")
input(type="imrelp" port="2514" maxDataSize="10k" keepAlive="on")
add the following to server 50-default.conf:
local7.* -/var/log/local7.log
add the following to client 50-default.conf
local7.* -/var/log/local7.log
local7.* :omrelp:<server>:2514
2. I've confirmed that /var/spool/rsyslog exists; however, I was only
buffering one or two messages so the queue file would never be created.
3. On my client, $RepeatedMsgReduction defaults to "on". I had to
explicitly turn it off in rsyslog.conf so duplicates do not get rolled up
Here's exactly how I tested:
1. log a message from the client, verify that it shows up on the server
# logger -p local7.info -s 'hello world'
shows up in /var/log/local7.log on the server
shows up in /var/log/local7.log on the client
2. disable rsyslog on the server
# systemctl stop syslog.socket rsyslog.service
3. log a message on the client
# logger -p local7.info -s 'hello world 2'
shows up in /var/log/local7.log on the client
4. enable rsyslog on the server
# systemctl start syslog.socket rsyslog.service
5. log a message on the client
# logger -p local7.info -s 'hello world 3'
shows up in /var/log/local7.log on the server
shows up in /var/log/local7.log on the client
"hello world 3" comes out on the server. "hello world 2" does not.
Note that the server is only down for a few seconds in this scenario.
I tried setting $ActionResumeInterval 1 on the client, and I've tried running
syslogd in debug mode, but frankly I don't understand the output very well and
have no idea what I'm looking for. I don't see anything that would suggest the
message is being queued on the client when the server is down as in step 3, but
again, I'm not sure how that would show up in the debug trace.
There must be something I'm doing wrong, but what?
-----Original Message-----
From: rsyslog <[email protected]> On Behalf Of
Mariusz Kruk via rsyslog
Sent: Friday, February 18, 2022 4:18 AM
To: [email protected]
Cc: Mariusz Kruk <[email protected]>
Subject: Re: [rsyslog] setting up reliable forwarding of syslog
Messages with Rsyslog
Firstly, after you confirm that your queueing works properly, I'd advise you to switch to
RELP so you have "more reliability".
But regarding your setup - as you defined
$WorkDirectory /var/spool/rsyslog
Your queue should be placed there.
Question is whether you do indeed have such directory in your system.
Because if you don't, the rsyslog daemon won't be able to save the queue
contents.
But in case of just a few messages you shouldn't be saving the contents do disk
at all. (it would be saved when you have unsent messages and shut down the
rsyslog daemon).
Also, notice that
https://www.rsyslog.com/doc/master/configuration/action/rsconf1_repe
a t edmsgreduction.html "This parameter models old sysklogd legacy.
*Note that many people, including the rsyslog authors, consider this
to be a misfeature.* See /Discussion/ below to learn why."
But in general, the setup should work... with one caveat. Your "never"
might in fact not be "never". You didn't tweak the settings that control action
resuming so they are at default 30 second initial interval which is getting raised after
every 10 tries up to a default 1800 seconds. So if the server was off for long enough,
the client might simply have paused sending for a really significant time.
See the description of parameters at
https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#general-action-parameters.
You might set (just for test! you probably don't want to set it in
prod for that often)
$ActionResumeInterval 1
And then run your client instance in debug mode to see interactively what it's
trying to do.
rsyslogd -f rsyslog.conf -i NONE -n -d
On 17.02.2022 18:03, MACGREGOR Will via rsyslog wrote:
I'm new to rsyslog, and I'm trying to set up reliable forwarding of syslog
messages with rsyslog according to these instructions:
https://www.rsyslog.com/doc/master/tutorials/reliable_forwarding.ht
m
l
I confirm that remote logging is working initially by doing
# logger "hello, world"
on the client, and verifying that this message shows up in the
server (in this case in /var/log/syslog)
I then shut down the rsyslog server, and log a few more messages on the client.
As expected, these are not showing up on the server side any more. On the
client, they seem to be going to its /var/log/syslog file; I have no idea where
(if) they're being queued.
I then re-enable the rsyslog server, but the entries that I wrote on the client
never seem to make it back to the server. What am I doing wrong?
Some configuration files:
-------------------------------------------------------------------
-
-
-
----------------------
client rsyslog.conf file:
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
#
/usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in
/etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
# provides kernel logging support and enable non-kernel klog
messages module(load="imklog" permitnonkernelfacility="on")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files # $WorkDirectory
/var/spool/rsyslog
#
# setup reliable local buffering
#
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk
mode $ActionResumeRetryCount -1 # infinite retries on insert
failure $ActionQueueSaveOnShutdown on # save in-memory data if
rsyslog shuts down
#
# Include all config files in /etc/rsyslog.d/ # $IncludeConfig
/etc/rsyslog.d/*.conf
*.* @@<redacted>:514
------------------------------------------------------------------
server rsyslog.conf file
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
#
/usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in
/etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
# provides kernel logging support and enable non-kernel klog
messages module(load="imklog" permitnonkernelfacility="on")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files # $WorkDirectory
/var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/ # $IncludeConfig
/etc/rsyslog.d/*.conf
------------------------------------------------------------------
version info for rsyslogd (both machines running Ubuntu 18.04,
FWIW)
# rsyslogd -version (same version for both client and server)
rsyslogd 8.32.0, compiled with:
PLATFORM: x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Number of Bits in RainerScript integers: 64
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
