Hi
I am attaching a pstats file here as requested.
I configured pstats to log every 10 seconds and the queue that forwards to
the remote host is called:
action( type="omrelp"
name="logtoserver"
target="192.168.8.134"
port="514"
queue.size="50000"
queue.type="LinkedList"
queue.spoolDirectory="/var/lib/rsyslog"
queue.filename="testforwardingqueue"
queue.lowwatermark="20000"
queue.highwatermark="35000"
queue.discardmark="50000"
queue.maxfilesize="1g"
queue.saveonshutdown="on"
# queue.fulldelaymark="2500"
)
My process were:
Start firewall on remote server to block syslog
Run script:
#!/bin/bash
echo '' > /var/log/pstats
echo "" >> /var/log/pstats
echo "BEGINING of TEST with SEQ-${1}" >> /var/log/pstats
echo "" >> /var/log/pstats
echo 'Starting log storm'
for i in $(seq 1 40000); do logger -p error "############### TESTING
########### SEQ${1}-${i}"; done
echo "" >> /var/log/pstats
echo "END of TEST with SEQ-${1}" >> /var/log/pstats
echo "" >> /var/log/pstats
after having run the script, I then stopped the firewall and waited for the
logs to flush. Some logs flushed and after I saw them appear in the remote
server I stopped rsyslog on the client and took a copy of the pstats file.
I am trawling through the pstats file now but hopefully you might have some
useful tool that can make it easier on your side and if not experience will
always reveal things quicker.
Regards
On 14 July 2015 at 17:12, Rainer Gerhards <[email protected]> wrote:
> 2015-07-14 17:32 GMT+02:00 Gerhardus Geldenhuis
> <[email protected]>:
> > Thanks David,
> > I configured RELP but it has not made a difference...
> >
> > More specifically if I turn on the firewall by
> >
> > 1. running: systemctl start firewalld on the master server
> > 2. and then start generating 40 000 log entries on the client, and
> wait
> > for it to finish
> > 3. run: systemctl stop firewalld
> > 4. Wait for logs to come over
> > 5. Count logs
> >
> > I still get 24767 messages instead of 40000.
>
> can you share the impstats data with us? We may get a better
> understanding of what's going on based on it...
>
> Rainer
> >
> > If however I physically turn of the network on the master machine using
> > VMWare, I do get 40 000 messages when I connect the network back again.
> >
> > So why is RELP handling it like this surely if it does not get any
> response
> > from the firewall for if the firewall sends it to a black hole then it
> > should not mark messages as delivered. I am also willing to admit that my
> > testing methodology is flawed but then I would like to understand why
> > exactly and at the moment I feel my testing methodology represents a real
> > world scenario which would make loosing logs a reality even if you use
> RELP.
> >
> > Importantly I enable the firewall before starting the log test so it
> should
> > all be blocked from the very beginning.
> >
> > Regards
> >
> > On 14 July 2015 at 12:44, David Lang <[email protected]> wrote:
> >
> >> One issue you are probably running into is the fact that TCP is not
> >> reliable enough. TCP is 'reliable' in that if a packet gets dropped it
> will
> >> detect and re-send it as part of the same connection, but there is a
> >> problem when the connection is interrupted (as it is in your test by the
> >> firewall)
> >>
> >> When the sending rsyslog hands the message to the OS on the sending
> >> system, it is told "yes, I have this and will deliver it" by the OS.
> But if
> >> the OS is unable to deliver it (because the network connection breaks),
> >> there is no way for the OS to go back to the sending application and say
> >> "oops, I really can't get this through to the other end" so the messages
> >> are lost.
> >>
> >> This is what the RELP protocol was developed to fix. It does application
> >> level acks for the messages so that if the network is interrupted, it
> knows
> >> what messages didn't get through and will re-send them. switch to
> >> omrelp/imrelp instead of omfwd/imtcp and you should avoid loosing any
> >> messages
> >>
> >> If you make the firewall send resets rather than dropping packets, the
> >> sending machine will detect the problem faster.
> >>
> >>
> >> This would account for some messages being lost between the time that
> you
> >> ahve the firewall block the connection and the time that rsyslog detects
> >> this. But once rsyslog detects this and starts queueing messages, those
> >> messages should be sent when rsyslog re-establishes the connection.
> >>
> >> enable impstats writing stats rapidly (say every 10 seconds or so for
> >> testing)
> >>
> >> then block the connection and send a small number of messages until
> >> rsyslog logs locally that it's unable to send and the queue for the
> action
> >> starts filling.
> >>
> >> then do your write of 40000 test messages
> >>
> >> then open the firewall and you should see all the messages get sent once
> >> rsyslog retries and notices that the connection is open again.
> >>
> >> David Lang
> >>
> >> On Tue, 14 Jul 2015, Gerhardus Geldenhuis wrote:
> >>
> >> Date: Tue, 14 Jul 2015 11:20:46 +0100
> >>> From: Gerhardus Geldenhuis <[email protected]>
> >>> Reply-To: rsyslog-users <[email protected]>
> >>> To: rsyslog-users <[email protected]>
> >>> Subject: Re: [rsyslog] Disk queue to flush after restart
> >>>
> >>> Hi,
> >>> I don't seem to be making as much progress as I would hope for. I now
> have
> >>> the following client config:
> >>> action( type="omfwd"
> >>> target="192.168.8.134"
> >>> port="514"
> >>> protocol="tcp"
> >>> queue.size="50000"
> >>> queue.type="LinkedList"
> >>> queue.spoolDirectory="/var/lib/rsyslog"
> >>> queue.filename="testforwardingqueue"
> >>> queue.lowwatermark="20000"
> >>> queue.highwatermark="35000"
> >>> queue.discardmark="50000"
> >>> queue.maxfilesize="1g"
> >>> queue.saveonshutdown="on"
> >>> # queue.fulldelaymark="2500"
> >>> )
> >>>
> >>> I have made the numbers smaller to be able to do testing that will
> finish
> >>> in a reasonable amount of time.
> >>>
> >>> My process is:
> >>>
> >>> 1. Start firewall on master server to block all access.
> >>> 2. On client server start logging using: for i in $(seq 1 40000); do
> >>> logger -p error "############### TESTING ########### SEQ001-${i}";
> done
> >>> 3. Wait for for loop to finish and check if all entries are contained
> >>> locally
> >>> 4. watch 'ls -l /var/lib/rsyslog' to see if queue gets created.
> >>> 5. Disable firewall on master
> >>> 6. run logger -p error with one message to get logs going again
> >>> 7. Wait for logs to arrive on master server
> >>> 8. grep 'SEQ001' /var/log/messages -c
> >>>
> >>>
> >>> The problem I am getting is that the number of messages I receive on
> the
> >>> remote server does not correspond with what I get on the master server.
> >>> I only get 24768 messages on the master server.
> >>>
> >>> Is there anything else that I have missed out of the configuration? I
> >>> can't
> >>> see anything that would cause my messages to not arrive. The queue
> file I
> >>> specified grows in size but interestingly when I grep'ed it for SEQ005
> I
> >>> only get 15232 matches. I did the grep after re-enabling the firewall.
> >>>
> >>> I thought that it might be because syslog thinks my messages are the
> same
> >>> and then marks them as duplicates. I then added:
> >>> for i in $(seq 1 40000); do logger -p error "############### TESTING
> >>> ########### SEQ005-${i}-$(openssl rand -base64 32)"; done
> >>>
> >>> to try and prevent that. However if the firewall is not enabled then
> all
> >>> 40k messages arrive on the master server.
> >>>
> >>> Any help would be appreciated.
> >>>
> >>> Regards
> >>>
> >>> On 13 July 2015 at 14:20, Rainer Gerhards <[email protected]>
> >>> wrote:
> >>>
> >>> 2015-07-13 15:18 GMT+02:00 Gerhardus Geldenhuis
> >>>> <[email protected]>:
> >>>>
> >>>>> Hi,
> >>>>> Thanks for the replies. I think the bulk of my problem was mixing old
> >>>>> and
> >>>>> new config and I much further along to get something working. I have
> >>>>> discovered a few other niggles but will report back once I have
> >>>>> something
> >>>>> working properly. As far as pull requests go, I would really consider
> >>>>>
> >>>> doing
> >>>>
> >>>>> so but as always time is a factor.
> >>>>>
> >>>>
> >>>> yup - for everyone ;) And that is what it is like it currently is ;)
> >>>>
> >>>> It does bug me so much that I must just
> >>>>> end up doing it for the documentation. I will give debug mode a go.
> >>>>>
> >>>>> Regards
> >>>>>
> >>>>> On 13 July 2015 at 13:17, Rainer Gerhards <[email protected]>
> >>>>>
> >>>> wrote:
> >>>>
> >>>>>
> >>>>> 2015-07-10 13:40 GMT+02:00 Gerhardus Geldenhuis
> >>>>>> <[email protected]>:
> >>>>>>
> >>>>>>> Hi
> >>>>>>> I am struggling a bit to get rsyslog to work as described.
> >>>>>>>
> >>>>>>> <rant>
> >>>>>>> Firstly the documentation is a struggle. There is some reference to
> >>>>>>>
> >>>>>> old
> >>>>
> >>>>> and
> >>>>>>
> >>>>>>> new style configuration but no clear differentiation between the
> two.
> >>>>>>>
> >>>>>> What
> >>>>>>
> >>>>>>> makes it more confusing is that documents like
> >>>>>>> http://www.rsyslog.com/doc/queues.html then refers to what looks
> like
> >>>>>>>
> >>>>>> the
> >>>>>>
> >>>>>>> old style of config and none of the examples contains new syntax
> >>>>>>>
> >>>>>> examples.
> >>>>>>
> >>>>>>>
> >>>>>>> There was also an expectation that the new rsyslog package would
> >>>>>>>
> >>>>>> install
> >>>>
> >>>>> a
> >>>>>>
> >>>>>>> new style config but that turned out to be not the case. I deleted
> the
> >>>>>>> config file and did a yum reinstall just to be sure.
> >>>>>>> </rant>
> >>>>>>>
> >>>>>>
> >>>>>> well, this is open source. Pull requests are always appreciated,
> >>>>>> anything else happens as time permits ;)
> >>>>>>
> >>>>>>
> >>>>>>> OS: CentOS 7
> >>>>>>> RSyslog:
> >>>>>>> rsyslog-8.11.0-1.el7.x86_64
> >>>>>>> rsyslog-relp-8.11.0-1.el7.x86_64
> >>>>>>> rsyslog-gnutls-8.11.0-1.el7.x86_64
> >>>>>>>
> >>>>>>> So basically what I am trying to achieve is the following:
> >>>>>>>
> >>>>>>> - Log remotely to a rsyslog server
> >>>>>>> - Turn off the remote server ( via firewall )
> >>>>>>> - Have logs be cached locally and saved to disk
> >>>>>>> - Restart client server
> >>>>>>> - Turn remote server back on
> >>>>>>> - See cached logs appear in the remote server
> >>>>>>>
> >>>>>>> It does not work...
> >>>>>>>
> >>>>>>> - So more specifically, if I turn the firewall off, log a few
> >>>>>>>
> >>>>>> messages
> >>>>
> >>>>> and turn it back on then the caching works and I get the messages.
> >>>>>>> - If however I restart the client server, the logs never make
> it to
> >>>>>>>
> >>>>>> the
> >>>>>>
> >>>>>>> remote sever, I can see the logs in the cached file but it does
> not
> >>>>>>>
> >>>>>> get
> >>>>>>
> >>>>>>> send to the remote server.
> >>>>>>>
> >>>>>>> My config on the client:
> >>>>>>> #### MODULES ####
> >>>>>>> module(load="imuxsock") # provides support for local system logging
> >>>>>>>
> >>>>>> (e.g.
> >>>>
> >>>>> via logger command)
> >>>>>>> module(load="imklog") # provides kernel logging support
> (previously
> >>>>>>>
> >>>>>> done
> >>>>>>
> >>>>>>> by rklogd)
> >>>>>>>
> >>>>>>> #### GLOBAL DIRECTIVES ####
> >>>>>>> $IncludeConfig /etc/rsyslog.d/*.conf
> >>>>>>>
> >>>>>>> #### RULES ####
> >>>>>>>
> >>>>>>> *.info;mail.none;authpriv.none;cron.none
> >>>>>>>
> >>>>>> /var/log/messages
> >>>>
> >>>>> authpriv.*
> >>>>>>>
> >>>>>> /var/log/secure
> >>>>
> >>>>> mail.*
> >>>>>>>
> >>>>>> /var/log/maillog
> >>>>
> >>>>> cron.* /var/log/cron
> >>>>>>> *.emerg :omusrmsg:*
> >>>>>>> uucp,news.crit
> >>>>>>>
> >>>>>> /var/log/spooler
> >>>>
> >>>>> local7.*
> >>>>>>>
> >>>>>> /var/log/boot.log
> >>>>
> >>>>>
> >>>>>>> # ### begin forwarding rule ###
> >>>>>>> $WorkDirectory /var/lib/rsyslog # where to place spool files
> >>>>>>> $MainMsgQueueFileName LocalCaching # unique name prefix for spool
> >>>>>>>
> >>>>>> files
> >>>>
> >>>>> $MainMsgQueueSaveOnShutdown on # save messages to disk on shutdown
> >>>>>>> # $MainMsgQueueType LinkedList
> >>>>>>> $MainMsgQueueType Disk
> >>>>>>> $MainMsgResumeRetryCount -1 # infinite retries if host is down
> >>>>>>>
> >>>>>>> *.* @@192.168.8.253:514
> >>>>>>>
> >>>>>>> # ### end of the forwarding rule ###
> >>>>>>>
> >>>>>>> My config on the remote server:
> >>>>>>> module(load="imuxsock") # provides support for local system logging
> >>>>>>>
> >>>>>> (e.g.
> >>>>
> >>>>> via logger command)
> >>>>>>> module(load="imklog") # provides kernel logging support
> (previously
> >>>>>>>
> >>>>>> done
> >>>>>>
> >>>>>>> by rklogd)
> >>>>>>> module(load="imtcp") # needs to be done just once
> >>>>>>> input(type="imtcp" port="514")
> >>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> >>>>>>> $IncludeConfig /etc/rsyslog.d/*.conf
> >>>>>>> *.info;mail.none;authpriv.none;cron.none
> >>>>>>>
> >>>>>> /var/log/messages
> >>>>
> >>>>> authpriv.*
> >>>>>>>
> >>>>>> /var/log/secure
> >>>>
> >>>>> mail.*
> >>>>>>>
> >>>>>> /var/log/maillog
> >>>>
> >>>>> cron.* /var/log/cron
> >>>>>>> *.emerg :omusrmsg:*
> >>>>>>> uucp,news.crit
> >>>>>>>
> >>>>>> /var/log/spooler
> >>>>
> >>>>> local7.*
> >>>>>>>
> >>>>>>> Any pointers would be appreciated. I am hoping I am missing
> something
> >>>>>>> obvious or misunderstanding what I am suppose to be doing.
> >>>>>>>
> >>>>>>>
> >>>>>> You should run rsyslog in such a situation in debug mode.From the
> >>>>>> debug log, we can see why it thinks it can't deliver to the remote
> >>>>>> system (well, hopefully ;)).
> >>>>>>
> >>>>>> HTH
> >>>>>> Rainer
> >>>>>>
> >>>>>> Regards
> >>>>>>>
> >>>>>>> --
> >>>>>>> Gerhardus Geldenhuis
> >>>>>>> _______________________________________________
> >>>>>>> rsyslog mailing list
> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>> http://www.rsyslog.com/professional-services/
> >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>>>>>
> >>>>>> myriad
> >>>>
> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> >>>>>> DON'T LIKE THAT.
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.com/professional-services/
> >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>>>> myriad
> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> >>>>>> DON'T LIKE THAT.
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Gerhardus Geldenhuis
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com/professional-services/
> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >>>>>
> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >>>> DON'T LIKE THAT.
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com/professional-services/
> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >>>> DON'T LIKE THAT.
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> >
> >
> >
> > --
> > Gerhardus Geldenhuis
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
--
Gerhardus Geldenhuis
pstats
Description: Binary data
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
