I have made a bit of progress... 2015-07-14T19:53:26.374223+01:00 rclient rsyslogd-pstats: logtoserver queue[DA]: origin=core.queue size=0 enqueued=15232 full=0 discarded.full=0 discarded.nf=0 maxqsize=15232 2015-07-14T19:53:26.374227+01:00 rclient rsyslogd-pstats: logtoserver queue: origin=core.queue size=24642 enqueued=56608 full=0 discarded.full=0 discarded.nf=0 maxqsize=35002
Those lines show that the DA queue has a smaller max queue size of 15232. If I subtract that from 40 000 I get 24 768 which is the exact amount of messages that I get on the remote server. The max queue size value is thus the number of messages that I seem to loose or not get on the remote server. I don't have a solution yet but I think that will give some guidance as to where things are going wrong ( I suspect in my config and/or assumptions ) Regards On 15 July 2015 at 09:58, Gerhardus Geldenhuis < [email protected]> wrote: > Hi > I am attaching a pstats file here as requested. > > I configured pstats to log every 10 seconds and the queue that forwards to > the remote host is called: > action( type="omrelp" > name="logtoserver" > target="192.168.8.134" > port="514" > queue.size="50000" > queue.type="LinkedList" > queue.spoolDirectory="/var/lib/rsyslog" > queue.filename="testforwardingqueue" > queue.lowwatermark="20000" > queue.highwatermark="35000" > queue.discardmark="50000" > queue.maxfilesize="1g" > queue.saveonshutdown="on" > # queue.fulldelaymark="2500" > ) > > My process were: > Start firewall on remote server to block syslog > > Run script: > > #!/bin/bash > > echo '' > /var/log/pstats > > echo "" >> /var/log/pstats > echo "BEGINING of TEST with SEQ-${1}" >> /var/log/pstats > echo "" >> /var/log/pstats > > echo 'Starting log storm' > for i in $(seq 1 40000); do logger -p error "############### TESTING > ########### SEQ${1}-${i}"; done > > echo "" >> /var/log/pstats > echo "END of TEST with SEQ-${1}" >> /var/log/pstats > echo "" >> /var/log/pstats > > after having run the script, I then stopped the firewall and waited for > the logs to flush. Some logs flushed and after I saw them appear in the > remote server I stopped rsyslog on the client and took a copy of the pstats > file. > > I am trawling through the pstats file now but hopefully you might have > some useful tool that can make it easier on your side and if not experience > will always reveal things quicker. > > Regards > > On 14 July 2015 at 17:12, Rainer Gerhards <[email protected]> > wrote: > >> 2015-07-14 17:32 GMT+02:00 Gerhardus Geldenhuis >> <[email protected]>: >> > Thanks David, >> > I configured RELP but it has not made a difference... >> > >> > More specifically if I turn on the firewall by >> > >> > 1. running: systemctl start firewalld on the master server >> > 2. and then start generating 40 000 log entries on the client, and >> wait >> > for it to finish >> > 3. run: systemctl stop firewalld >> > 4. Wait for logs to come over >> > 5. Count logs >> > >> > I still get 24767 messages instead of 40000. >> >> can you share the impstats data with us? We may get a better >> understanding of what's going on based on it... >> >> Rainer >> > >> > If however I physically turn of the network on the master machine using >> > VMWare, I do get 40 000 messages when I connect the network back again. >> > >> > So why is RELP handling it like this surely if it does not get any >> response >> > from the firewall for if the firewall sends it to a black hole then it >> > should not mark messages as delivered. I am also willing to admit that >> my >> > testing methodology is flawed but then I would like to understand why >> > exactly and at the moment I feel my testing methodology represents a >> real >> > world scenario which would make loosing logs a reality even if you use >> RELP. >> > >> > Importantly I enable the firewall before starting the log test so it >> should >> > all be blocked from the very beginning. >> > >> > Regards >> > >> > On 14 July 2015 at 12:44, David Lang <[email protected]> wrote: >> > >> >> One issue you are probably running into is the fact that TCP is not >> >> reliable enough. TCP is 'reliable' in that if a packet gets dropped it >> will >> >> detect and re-send it as part of the same connection, but there is a >> >> problem when the connection is interrupted (as it is in your test by >> the >> >> firewall) >> >> >> >> When the sending rsyslog hands the message to the OS on the sending >> >> system, it is told "yes, I have this and will deliver it" by the OS. >> But if >> >> the OS is unable to deliver it (because the network connection breaks), >> >> there is no way for the OS to go back to the sending application and >> say >> >> "oops, I really can't get this through to the other end" so the >> messages >> >> are lost. >> >> >> >> This is what the RELP protocol was developed to fix. It does >> application >> >> level acks for the messages so that if the network is interrupted, it >> knows >> >> what messages didn't get through and will re-send them. switch to >> >> omrelp/imrelp instead of omfwd/imtcp and you should avoid loosing any >> >> messages >> >> >> >> If you make the firewall send resets rather than dropping packets, the >> >> sending machine will detect the problem faster. >> >> >> >> >> >> This would account for some messages being lost between the time that >> you >> >> ahve the firewall block the connection and the time that rsyslog >> detects >> >> this. But once rsyslog detects this and starts queueing messages, those >> >> messages should be sent when rsyslog re-establishes the connection. >> >> >> >> enable impstats writing stats rapidly (say every 10 seconds or so for >> >> testing) >> >> >> >> then block the connection and send a small number of messages until >> >> rsyslog logs locally that it's unable to send and the queue for the >> action >> >> starts filling. >> >> >> >> then do your write of 40000 test messages >> >> >> >> then open the firewall and you should see all the messages get sent >> once >> >> rsyslog retries and notices that the connection is open again. >> >> >> >> David Lang >> >> >> >> On Tue, 14 Jul 2015, Gerhardus Geldenhuis wrote: >> >> >> >> Date: Tue, 14 Jul 2015 11:20:46 +0100 >> >>> From: Gerhardus Geldenhuis <[email protected]> >> >>> Reply-To: rsyslog-users <[email protected]> >> >>> To: rsyslog-users <[email protected]> >> >>> Subject: Re: [rsyslog] Disk queue to flush after restart >> >>> >> >>> Hi, >> >>> I don't seem to be making as much progress as I would hope for. I now >> have >> >>> the following client config: >> >>> action( type="omfwd" >> >>> target="192.168.8.134" >> >>> port="514" >> >>> protocol="tcp" >> >>> queue.size="50000" >> >>> queue.type="LinkedList" >> >>> queue.spoolDirectory="/var/lib/rsyslog" >> >>> queue.filename="testforwardingqueue" >> >>> queue.lowwatermark="20000" >> >>> queue.highwatermark="35000" >> >>> queue.discardmark="50000" >> >>> queue.maxfilesize="1g" >> >>> queue.saveonshutdown="on" >> >>> # queue.fulldelaymark="2500" >> >>> ) >> >>> >> >>> I have made the numbers smaller to be able to do testing that will >> finish >> >>> in a reasonable amount of time. >> >>> >> >>> My process is: >> >>> >> >>> 1. Start firewall on master server to block all access. >> >>> 2. On client server start logging using: for i in $(seq 1 40000); do >> >>> logger -p error "############### TESTING ########### SEQ001-${i}"; >> done >> >>> 3. Wait for for loop to finish and check if all entries are >> contained >> >>> locally >> >>> 4. watch 'ls -l /var/lib/rsyslog' to see if queue gets created. >> >>> 5. Disable firewall on master >> >>> 6. run logger -p error with one message to get logs going again >> >>> 7. Wait for logs to arrive on master server >> >>> 8. grep 'SEQ001' /var/log/messages -c >> >>> >> >>> >> >>> The problem I am getting is that the number of messages I receive on >> the >> >>> remote server does not correspond with what I get on the master >> server. >> >>> I only get 24768 messages on the master server. >> >>> >> >>> Is there anything else that I have missed out of the configuration? I >> >>> can't >> >>> see anything that would cause my messages to not arrive. The queue >> file I >> >>> specified grows in size but interestingly when I grep'ed it for >> SEQ005 I >> >>> only get 15232 matches. I did the grep after re-enabling the firewall. >> >>> >> >>> I thought that it might be because syslog thinks my messages are the >> same >> >>> and then marks them as duplicates. I then added: >> >>> for i in $(seq 1 40000); do logger -p error "############### TESTING >> >>> ########### SEQ005-${i}-$(openssl rand -base64 32)"; done >> >>> >> >>> to try and prevent that. However if the firewall is not enabled then >> all >> >>> 40k messages arrive on the master server. >> >>> >> >>> Any help would be appreciated. >> >>> >> >>> Regards >> >>> >> >>> On 13 July 2015 at 14:20, Rainer Gerhards <[email protected]> >> >>> wrote: >> >>> >> >>> 2015-07-13 15:18 GMT+02:00 Gerhardus Geldenhuis >> >>>> <[email protected]>: >> >>>> >> >>>>> Hi, >> >>>>> Thanks for the replies. I think the bulk of my problem was mixing >> old >> >>>>> and >> >>>>> new config and I much further along to get something working. I have >> >>>>> discovered a few other niggles but will report back once I have >> >>>>> something >> >>>>> working properly. As far as pull requests go, I would really >> consider >> >>>>> >> >>>> doing >> >>>> >> >>>>> so but as always time is a factor. >> >>>>> >> >>>> >> >>>> yup - for everyone ;) And that is what it is like it currently is ;) >> >>>> >> >>>> It does bug me so much that I must just >> >>>>> end up doing it for the documentation. I will give debug mode a go. >> >>>>> >> >>>>> Regards >> >>>>> >> >>>>> On 13 July 2015 at 13:17, Rainer Gerhards <[email protected] >> > >> >>>>> >> >>>> wrote: >> >>>> >> >>>>> >> >>>>> 2015-07-10 13:40 GMT+02:00 Gerhardus Geldenhuis >> >>>>>> <[email protected]>: >> >>>>>> >> >>>>>>> Hi >> >>>>>>> I am struggling a bit to get rsyslog to work as described. >> >>>>>>> >> >>>>>>> <rant> >> >>>>>>> Firstly the documentation is a struggle. There is some reference >> to >> >>>>>>> >> >>>>>> old >> >>>> >> >>>>> and >> >>>>>> >> >>>>>>> new style configuration but no clear differentiation between the >> two. >> >>>>>>> >> >>>>>> What >> >>>>>> >> >>>>>>> makes it more confusing is that documents like >> >>>>>>> http://www.rsyslog.com/doc/queues.html then refers to what looks >> like >> >>>>>>> >> >>>>>> the >> >>>>>> >> >>>>>>> old style of config and none of the examples contains new syntax >> >>>>>>> >> >>>>>> examples. >> >>>>>> >> >>>>>>> >> >>>>>>> There was also an expectation that the new rsyslog package would >> >>>>>>> >> >>>>>> install >> >>>> >> >>>>> a >> >>>>>> >> >>>>>>> new style config but that turned out to be not the case. I >> deleted the >> >>>>>>> config file and did a yum reinstall just to be sure. >> >>>>>>> </rant> >> >>>>>>> >> >>>>>> >> >>>>>> well, this is open source. Pull requests are always appreciated, >> >>>>>> anything else happens as time permits ;) >> >>>>>> >> >>>>>> >> >>>>>>> OS: CentOS 7 >> >>>>>>> RSyslog: >> >>>>>>> rsyslog-8.11.0-1.el7.x86_64 >> >>>>>>> rsyslog-relp-8.11.0-1.el7.x86_64 >> >>>>>>> rsyslog-gnutls-8.11.0-1.el7.x86_64 >> >>>>>>> >> >>>>>>> So basically what I am trying to achieve is the following: >> >>>>>>> >> >>>>>>> - Log remotely to a rsyslog server >> >>>>>>> - Turn off the remote server ( via firewall ) >> >>>>>>> - Have logs be cached locally and saved to disk >> >>>>>>> - Restart client server >> >>>>>>> - Turn remote server back on >> >>>>>>> - See cached logs appear in the remote server >> >>>>>>> >> >>>>>>> It does not work... >> >>>>>>> >> >>>>>>> - So more specifically, if I turn the firewall off, log a few >> >>>>>>> >> >>>>>> messages >> >>>> >> >>>>> and turn it back on then the caching works and I get the >> messages. >> >>>>>>> - If however I restart the client server, the logs never make >> it to >> >>>>>>> >> >>>>>> the >> >>>>>> >> >>>>>>> remote sever, I can see the logs in the cached file but it >> does not >> >>>>>>> >> >>>>>> get >> >>>>>> >> >>>>>>> send to the remote server. >> >>>>>>> >> >>>>>>> My config on the client: >> >>>>>>> #### MODULES #### >> >>>>>>> module(load="imuxsock") # provides support for local system >> logging >> >>>>>>> >> >>>>>> (e.g. >> >>>> >> >>>>> via logger command) >> >>>>>>> module(load="imklog") # provides kernel logging support >> (previously >> >>>>>>> >> >>>>>> done >> >>>>>> >> >>>>>>> by rklogd) >> >>>>>>> >> >>>>>>> #### GLOBAL DIRECTIVES #### >> >>>>>>> $IncludeConfig /etc/rsyslog.d/*.conf >> >>>>>>> >> >>>>>>> #### RULES #### >> >>>>>>> >> >>>>>>> *.info;mail.none;authpriv.none;cron.none >> >>>>>>> >> >>>>>> /var/log/messages >> >>>> >> >>>>> authpriv.* >> >>>>>>> >> >>>>>> /var/log/secure >> >>>> >> >>>>> mail.* >> >>>>>>> >> >>>>>> /var/log/maillog >> >>>> >> >>>>> cron.* >> /var/log/cron >> >>>>>>> *.emerg >> :omusrmsg:* >> >>>>>>> uucp,news.crit >> >>>>>>> >> >>>>>> /var/log/spooler >> >>>> >> >>>>> local7.* >> >>>>>>> >> >>>>>> /var/log/boot.log >> >>>> >> >>>>> >> >>>>>>> # ### begin forwarding rule ### >> >>>>>>> $WorkDirectory /var/lib/rsyslog # where to place spool files >> >>>>>>> $MainMsgQueueFileName LocalCaching # unique name prefix for spool >> >>>>>>> >> >>>>>> files >> >>>> >> >>>>> $MainMsgQueueSaveOnShutdown on # save messages to disk on shutdown >> >>>>>>> # $MainMsgQueueType LinkedList >> >>>>>>> $MainMsgQueueType Disk >> >>>>>>> $MainMsgResumeRetryCount -1 # infinite retries if host is down >> >>>>>>> >> >>>>>>> *.* @@192.168.8.253:514 >> >>>>>>> >> >>>>>>> # ### end of the forwarding rule ### >> >>>>>>> >> >>>>>>> My config on the remote server: >> >>>>>>> module(load="imuxsock") # provides support for local system >> logging >> >>>>>>> >> >>>>>> (e.g. >> >>>> >> >>>>> via logger command) >> >>>>>>> module(load="imklog") # provides kernel logging support >> (previously >> >>>>>>> >> >>>>>> done >> >>>>>> >> >>>>>>> by rklogd) >> >>>>>>> module(load="imtcp") # needs to be done just once >> >>>>>>> input(type="imtcp" port="514") >> >>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> >>>>>>> $IncludeConfig /etc/rsyslog.d/*.conf >> >>>>>>> *.info;mail.none;authpriv.none;cron.none >> >>>>>>> >> >>>>>> /var/log/messages >> >>>> >> >>>>> authpriv.* >> >>>>>>> >> >>>>>> /var/log/secure >> >>>> >> >>>>> mail.* >> >>>>>>> >> >>>>>> /var/log/maillog >> >>>> >> >>>>> cron.* >> /var/log/cron >> >>>>>>> *.emerg >> :omusrmsg:* >> >>>>>>> uucp,news.crit >> >>>>>>> >> >>>>>> /var/log/spooler >> >>>> >> >>>>> local7.* >> >>>>>>> >> >>>>>>> Any pointers would be appreciated. I am hoping I am missing >> something >> >>>>>>> obvious or misunderstanding what I am suppose to be doing. >> >>>>>>> >> >>>>>>> >> >>>>>> You should run rsyslog in such a situation in debug mode.From the >> >>>>>> debug log, we can see why it thinks it can't deliver to the remote >> >>>>>> system (well, hopefully ;)). >> >>>>>> >> >>>>>> HTH >> >>>>>> Rainer >> >>>>>> >> >>>>>> Regards >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Gerhardus Geldenhuis >> >>>>>>> _______________________________________________ >> >>>>>>> rsyslog mailing list >> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>>>> http://www.rsyslog.com/professional-services/ >> >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>>>>> >> >>>>>> myriad >> >>>> >> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >>>>>> DON'T LIKE THAT. >> >>>>>> _______________________________________________ >> >>>>>> rsyslog mailing list >> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>>> http://www.rsyslog.com/professional-services/ >> >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>>>> myriad >> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >>>>>> DON'T LIKE THAT. >> >>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Gerhardus Geldenhuis >> >>>>> _______________________________________________ >> >>>>> rsyslog mailing list >> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>> http://www.rsyslog.com/professional-services/ >> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>>>> >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >>>> DON'T LIKE THAT. >> >>>> _______________________________________________ >> >>>> rsyslog mailing list >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>> http://www.rsyslog.com/professional-services/ >> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >>>> DON'T LIKE THAT. >> >>>> >> >>>> >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> >> > >> > >> > >> > -- >> > Gerhardus Geldenhuis >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > > > -- > Gerhardus Geldenhuis > -- Gerhardus Geldenhuis _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
