One issue you are probably running into is the fact that TCP is not reliable
enough. TCP is 'reliable' in that if a packet gets dropped it will detect and
re-send it as part of the same connection, but there is a problem when the
connection is interrupted (as it is in your test by the firewall)
When the sending rsyslog hands the message to the OS on the sending system, it
is told "yes, I have this and will deliver it" by the OS. But if the OS is
unable to deliver it (because the network connection breaks), there is no way
for the OS to go back to the sending application and say "oops, I really can't
get this through to the other end" so the messages are lost.
This is what the RELP protocol was developed to fix. It does application level
acks for the messages so that if the network is interrupted, it knows what
messages didn't get through and will re-send them. switch to omrelp/imrelp
instead of omfwd/imtcp and you should avoid loosing any messages
If you make the firewall send resets rather than dropping packets, the sending
machine will detect the problem faster.
This would account for some messages being lost between the time that you ahve
the firewall block the connection and the time that rsyslog detects this. But
once rsyslog detects this and starts queueing messages, those messages should be
sent when rsyslog re-establishes the connection.
enable impstats writing stats rapidly (say every 10 seconds or so for testing)
then block the connection and send a small number of messages until rsyslog logs
locally that it's unable to send and the queue for the action starts filling.
then do your write of 40000 test messages
then open the firewall and you should see all the messages get sent once rsyslog
retries and notices that the connection is open again.
David Lang
On Tue, 14 Jul
2015, Gerhardus Geldenhuis wrote:
Date: Tue, 14 Jul 2015 11:20:46 +0100
From: Gerhardus Geldenhuis <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Disk queue to flush after restart
Hi,
I don't seem to be making as much progress as I would hope for. I now have
the following client config:
action( type="omfwd"
target="192.168.8.134"
port="514"
protocol="tcp"
queue.size="50000"
queue.type="LinkedList"
queue.spoolDirectory="/var/lib/rsyslog"
queue.filename="testforwardingqueue"
queue.lowwatermark="20000"
queue.highwatermark="35000"
queue.discardmark="50000"
queue.maxfilesize="1g"
queue.saveonshutdown="on"
# queue.fulldelaymark="2500"
)
I have made the numbers smaller to be able to do testing that will finish
in a reasonable amount of time.
My process is:
1. Start firewall on master server to block all access.
2. On client server start logging using: for i in $(seq 1 40000); do
logger -p error "############### TESTING ########### SEQ001-${i}"; done
3. Wait for for loop to finish and check if all entries are contained
locally
4. watch 'ls -l /var/lib/rsyslog' to see if queue gets created.
5. Disable firewall on master
6. run logger -p error with one message to get logs going again
7. Wait for logs to arrive on master server
8. grep 'SEQ001' /var/log/messages -c
The problem I am getting is that the number of messages I receive on the
remote server does not correspond with what I get on the master server.
I only get 24768 messages on the master server.
Is there anything else that I have missed out of the configuration? I can't
see anything that would cause my messages to not arrive. The queue file I
specified grows in size but interestingly when I grep'ed it for SEQ005 I
only get 15232 matches. I did the grep after re-enabling the firewall.
I thought that it might be because syslog thinks my messages are the same
and then marks them as duplicates. I then added:
for i in $(seq 1 40000); do logger -p error "############### TESTING
########### SEQ005-${i}-$(openssl rand -base64 32)"; done
to try and prevent that. However if the firewall is not enabled then all
40k messages arrive on the master server.
Any help would be appreciated.
Regards
On 13 July 2015 at 14:20, Rainer Gerhards <[email protected]> wrote:
2015-07-13 15:18 GMT+02:00 Gerhardus Geldenhuis
<[email protected]>:
Hi,
Thanks for the replies. I think the bulk of my problem was mixing old and
new config and I much further along to get something working. I have
discovered a few other niggles but will report back once I have something
working properly. As far as pull requests go, I would really consider
doing
so but as always time is a factor.
yup - for everyone ;) And that is what it is like it currently is ;)
It does bug me so much that I must just
end up doing it for the documentation. I will give debug mode a go.
Regards
On 13 July 2015 at 13:17, Rainer Gerhards <[email protected]>
wrote:
2015-07-10 13:40 GMT+02:00 Gerhardus Geldenhuis
<[email protected]>:
Hi
I am struggling a bit to get rsyslog to work as described.
<rant>
Firstly the documentation is a struggle. There is some reference to
old
and
new style configuration but no clear differentiation between the two.
What
makes it more confusing is that documents like
http://www.rsyslog.com/doc/queues.html then refers to what looks like
the
old style of config and none of the examples contains new syntax
examples.
There was also an expectation that the new rsyslog package would
install
a
new style config but that turned out to be not the case. I deleted the
config file and did a yum reinstall just to be sure.
</rant>
well, this is open source. Pull requests are always appreciated,
anything else happens as time permits ;)
OS: CentOS 7
RSyslog:
rsyslog-8.11.0-1.el7.x86_64
rsyslog-relp-8.11.0-1.el7.x86_64
rsyslog-gnutls-8.11.0-1.el7.x86_64
So basically what I am trying to achieve is the following:
- Log remotely to a rsyslog server
- Turn off the remote server ( via firewall )
- Have logs be cached locally and saved to disk
- Restart client server
- Turn remote server back on
- See cached logs appear in the remote server
It does not work...
- So more specifically, if I turn the firewall off, log a few
messages
and turn it back on then the caching works and I get the messages.
- If however I restart the client server, the logs never make it to
the
remote sever, I can see the logs in the cached file but it does not
get
send to the remote server.
My config on the client:
#### MODULES ####
module(load="imuxsock") # provides support for local system logging
(e.g.
via logger command)
module(load="imklog") # provides kernel logging support (previously
done
by rklogd)
#### GLOBAL DIRECTIVES ####
$IncludeConfig /etc/rsyslog.d/*.conf
#### RULES ####
*.info;mail.none;authpriv.none;cron.none
/var/log/messages
authpriv.*
/var/log/secure
mail.*
/var/log/maillog
cron.* /var/log/cron
*.emerg :omusrmsg:*
uucp,news.crit
/var/log/spooler
local7.*
/var/log/boot.log
# ### begin forwarding rule ###
$WorkDirectory /var/lib/rsyslog # where to place spool files
$MainMsgQueueFileName LocalCaching # unique name prefix for spool
files
$MainMsgQueueSaveOnShutdown on # save messages to disk on shutdown
# $MainMsgQueueType LinkedList
$MainMsgQueueType Disk
$MainMsgResumeRetryCount -1 # infinite retries if host is down
*.* @@192.168.8.253:514
# ### end of the forwarding rule ###
My config on the remote server:
module(load="imuxsock") # provides support for local system logging
(e.g.
via logger command)
module(load="imklog") # provides kernel logging support (previously
done
by rklogd)
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
*.info;mail.none;authpriv.none;cron.none
/var/log/messages
authpriv.*
/var/log/secure
mail.*
/var/log/maillog
cron.* /var/log/cron
*.emerg :omusrmsg:*
uucp,news.crit
/var/log/spooler
local7.*
Any pointers would be appreciated. I am hoping I am missing something
obvious or misunderstanding what I am suppose to be doing.
You should run rsyslog in such a situation in debug mode.From the
debug log, we can see why it thinks it can't deliver to the remote
system (well, hopefully ;)).
HTH
Rainer
Regards
--
Gerhardus Geldenhuis
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
--
Gerhardus Geldenhuis
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
