2015-07-15 11:33 GMT+02:00 Gerhardus Geldenhuis <[email protected]>: > I have made a bit of progress... > 2015-07-14T19:53:26.374223+01:00 rclient rsyslogd-pstats: logtoserver > queue[DA]: origin=core.queue size=0 enqueued=15232 full=0 discarded.full=0 > discarded.nf=0 maxqsize=15232 > 2015-07-14T19:53:26.374227+01:00 rclient rsyslogd-pstats: logtoserver > queue: origin=core.queue size=24642 enqueued=56608 full=0 discarded.full=0 > discarded.nf=0 maxqsize=35002 > > Those lines show that the DA queue has a smaller max queue size of 15232. > If I subtract that from 40 000 I get 24 768 which is the exact amount of > messages that I get on the remote server. The max queue size value is thus > the number of messages that I seem to loose or not get on the remote server. > > I don't have a solution yet but I think that will give some guidance as to > where things are going wrong ( I suspect in my config and/or assumptions )
Not a solution, but a thought: This looks like the remote end does not accept the additional connection the DA queue tries to make. I have seen similar effects during testing when running against nc, which only permits one connection. Something in the firewall that limits that? Rainer > > Regards > > On 15 July 2015 at 09:58, Gerhardus Geldenhuis < > [email protected]> wrote: > >> Hi >> I am attaching a pstats file here as requested. >> >> I configured pstats to log every 10 seconds and the queue that forwards to >> the remote host is called: >> action( type="omrelp" >> name="logtoserver" >> target="192.168.8.134" >> port="514" >> queue.size="50000" >> queue.type="LinkedList" >> queue.spoolDirectory="/var/lib/rsyslog" >> queue.filename="testforwardingqueue" >> queue.lowwatermark="20000" >> queue.highwatermark="35000" >> queue.discardmark="50000" >> queue.maxfilesize="1g" >> queue.saveonshutdown="on" >> # queue.fulldelaymark="2500" >> ) >> >> My process were: >> Start firewall on remote server to block syslog >> >> Run script: >> >> #!/bin/bash >> >> echo '' > /var/log/pstats >> >> echo "" >> /var/log/pstats >> echo "BEGINING of TEST with SEQ-${1}" >> /var/log/pstats >> echo "" >> /var/log/pstats >> >> echo 'Starting log storm' >> for i in $(seq 1 40000); do logger -p error "############### TESTING >> ########### SEQ${1}-${i}"; done >> >> echo "" >> /var/log/pstats >> echo "END of TEST with SEQ-${1}" >> /var/log/pstats >> echo "" >> /var/log/pstats >> >> after having run the script, I then stopped the firewall and waited for >> the logs to flush. Some logs flushed and after I saw them appear in the >> remote server I stopped rsyslog on the client and took a copy of the pstats >> file. >> >> I am trawling through the pstats file now but hopefully you might have >> some useful tool that can make it easier on your side and if not experience >> will always reveal things quicker. >> >> Regards >> >> On 14 July 2015 at 17:12, Rainer Gerhards <[email protected]> >> wrote: >> >>> 2015-07-14 17:32 GMT+02:00 Gerhardus Geldenhuis >>> <[email protected]>: >>> > Thanks David, >>> > I configured RELP but it has not made a difference... >>> > >>> > More specifically if I turn on the firewall by >>> > >>> > 1. running: systemctl start firewalld on the master server >>> > 2. and then start generating 40 000 log entries on the client, and >>> wait >>> > for it to finish >>> > 3. run: systemctl stop firewalld >>> > 4. Wait for logs to come over >>> > 5. Count logs >>> > >>> > I still get 24767 messages instead of 40000. >>> >>> can you share the impstats data with us? We may get a better >>> understanding of what's going on based on it... >>> >>> Rainer >>> > >>> > If however I physically turn of the network on the master machine using >>> > VMWare, I do get 40 000 messages when I connect the network back again. >>> > >>> > So why is RELP handling it like this surely if it does not get any >>> response >>> > from the firewall for if the firewall sends it to a black hole then it >>> > should not mark messages as delivered. I am also willing to admit that >>> my >>> > testing methodology is flawed but then I would like to understand why >>> > exactly and at the moment I feel my testing methodology represents a >>> real >>> > world scenario which would make loosing logs a reality even if you use >>> RELP. >>> > >>> > Importantly I enable the firewall before starting the log test so it >>> should >>> > all be blocked from the very beginning. >>> > >>> > Regards >>> > >>> > On 14 July 2015 at 12:44, David Lang <[email protected]> wrote: >>> > >>> >> One issue you are probably running into is the fact that TCP is not >>> >> reliable enough. TCP is 'reliable' in that if a packet gets dropped it >>> will >>> >> detect and re-send it as part of the same connection, but there is a >>> >> problem when the connection is interrupted (as it is in your test by >>> the >>> >> firewall) >>> >> >>> >> When the sending rsyslog hands the message to the OS on the sending >>> >> system, it is told "yes, I have this and will deliver it" by the OS. >>> But if >>> >> the OS is unable to deliver it (because the network connection breaks), >>> >> there is no way for the OS to go back to the sending application and >>> say >>> >> "oops, I really can't get this through to the other end" so the >>> messages >>> >> are lost. >>> >> >>> >> This is what the RELP protocol was developed to fix. It does >>> application >>> >> level acks for the messages so that if the network is interrupted, it >>> knows >>> >> what messages didn't get through and will re-send them. switch to >>> >> omrelp/imrelp instead of omfwd/imtcp and you should avoid loosing any >>> >> messages >>> >> >>> >> If you make the firewall send resets rather than dropping packets, the >>> >> sending machine will detect the problem faster. >>> >> >>> >> >>> >> This would account for some messages being lost between the time that >>> you >>> >> ahve the firewall block the connection and the time that rsyslog >>> detects >>> >> this. But once rsyslog detects this and starts queueing messages, those >>> >> messages should be sent when rsyslog re-establishes the connection. >>> >> >>> >> enable impstats writing stats rapidly (say every 10 seconds or so for >>> >> testing) >>> >> >>> >> then block the connection and send a small number of messages until >>> >> rsyslog logs locally that it's unable to send and the queue for the >>> action >>> >> starts filling. >>> >> >>> >> then do your write of 40000 test messages >>> >> >>> >> then open the firewall and you should see all the messages get sent >>> once >>> >> rsyslog retries and notices that the connection is open again. >>> >> >>> >> David Lang >>> >> >>> >> On Tue, 14 Jul 2015, Gerhardus Geldenhuis wrote: >>> >> >>> >> Date: Tue, 14 Jul 2015 11:20:46 +0100 >>> >>> From: Gerhardus Geldenhuis <[email protected]> >>> >>> Reply-To: rsyslog-users <[email protected]> >>> >>> To: rsyslog-users <[email protected]> >>> >>> Subject: Re: [rsyslog] Disk queue to flush after restart >>> >>> >>> >>> Hi, >>> >>> I don't seem to be making as much progress as I would hope for. I now >>> have >>> >>> the following client config: >>> >>> action( type="omfwd" >>> >>> target="192.168.8.134" >>> >>> port="514" >>> >>> protocol="tcp" >>> >>> queue.size="50000" >>> >>> queue.type="LinkedList" >>> >>> queue.spoolDirectory="/var/lib/rsyslog" >>> >>> queue.filename="testforwardingqueue" >>> >>> queue.lowwatermark="20000" >>> >>> queue.highwatermark="35000" >>> >>> queue.discardmark="50000" >>> >>> queue.maxfilesize="1g" >>> >>> queue.saveonshutdown="on" >>> >>> # queue.fulldelaymark="2500" >>> >>> ) >>> >>> >>> >>> I have made the numbers smaller to be able to do testing that will >>> finish >>> >>> in a reasonable amount of time. >>> >>> >>> >>> My process is: >>> >>> >>> >>> 1. Start firewall on master server to block all access. >>> >>> 2. On client server start logging using: for i in $(seq 1 40000); do >>> >>> logger -p error "############### TESTING ########### SEQ001-${i}"; >>> done >>> >>> 3. Wait for for loop to finish and check if all entries are >>> contained >>> >>> locally >>> >>> 4. watch 'ls -l /var/lib/rsyslog' to see if queue gets created. >>> >>> 5. Disable firewall on master >>> >>> 6. run logger -p error with one message to get logs going again >>> >>> 7. Wait for logs to arrive on master server >>> >>> 8. grep 'SEQ001' /var/log/messages -c >>> >>> >>> >>> >>> >>> The problem I am getting is that the number of messages I receive on >>> the >>> >>> remote server does not correspond with what I get on the master >>> server. >>> >>> I only get 24768 messages on the master server. >>> >>> >>> >>> Is there anything else that I have missed out of the configuration? I >>> >>> can't >>> >>> see anything that would cause my messages to not arrive. The queue >>> file I >>> >>> specified grows in size but interestingly when I grep'ed it for >>> SEQ005 I >>> >>> only get 15232 matches. I did the grep after re-enabling the firewall. >>> >>> >>> >>> I thought that it might be because syslog thinks my messages are the >>> same >>> >>> and then marks them as duplicates. I then added: >>> >>> for i in $(seq 1 40000); do logger -p error "############### TESTING >>> >>> ########### SEQ005-${i}-$(openssl rand -base64 32)"; done >>> >>> >>> >>> to try and prevent that. However if the firewall is not enabled then >>> all >>> >>> 40k messages arrive on the master server. >>> >>> >>> >>> Any help would be appreciated. >>> >>> >>> >>> Regards >>> >>> >>> >>> On 13 July 2015 at 14:20, Rainer Gerhards <[email protected]> >>> >>> wrote: >>> >>> >>> >>> 2015-07-13 15:18 GMT+02:00 Gerhardus Geldenhuis >>> >>>> <[email protected]>: >>> >>>> >>> >>>>> Hi, >>> >>>>> Thanks for the replies. I think the bulk of my problem was mixing >>> old >>> >>>>> and >>> >>>>> new config and I much further along to get something working. I have >>> >>>>> discovered a few other niggles but will report back once I have >>> >>>>> something >>> >>>>> working properly. As far as pull requests go, I would really >>> consider >>> >>>>> >>> >>>> doing >>> >>>> >>> >>>>> so but as always time is a factor. >>> >>>>> >>> >>>> >>> >>>> yup - for everyone ;) And that is what it is like it currently is ;) >>> >>>> >>> >>>> It does bug me so much that I must just >>> >>>>> end up doing it for the documentation. I will give debug mode a go. >>> >>>>> >>> >>>>> Regards >>> >>>>> >>> >>>>> On 13 July 2015 at 13:17, Rainer Gerhards <[email protected] >>> > >>> >>>>> >>> >>>> wrote: >>> >>>> >>> >>>>> >>> >>>>> 2015-07-10 13:40 GMT+02:00 Gerhardus Geldenhuis >>> >>>>>> <[email protected]>: >>> >>>>>> >>> >>>>>>> Hi >>> >>>>>>> I am struggling a bit to get rsyslog to work as described. >>> >>>>>>> >>> >>>>>>> <rant> >>> >>>>>>> Firstly the documentation is a struggle. There is some reference >>> to >>> >>>>>>> >>> >>>>>> old >>> >>>> >>> >>>>> and >>> >>>>>> >>> >>>>>>> new style configuration but no clear differentiation between the >>> two. >>> >>>>>>> >>> >>>>>> What >>> >>>>>> >>> >>>>>>> makes it more confusing is that documents like >>> >>>>>>> http://www.rsyslog.com/doc/queues.html then refers to what looks >>> like >>> >>>>>>> >>> >>>>>> the >>> >>>>>> >>> >>>>>>> old style of config and none of the examples contains new syntax >>> >>>>>>> >>> >>>>>> examples. >>> >>>>>> >>> >>>>>>> >>> >>>>>>> There was also an expectation that the new rsyslog package would >>> >>>>>>> >>> >>>>>> install >>> >>>> >>> >>>>> a >>> >>>>>> >>> >>>>>>> new style config but that turned out to be not the case. I >>> deleted the >>> >>>>>>> config file and did a yum reinstall just to be sure. >>> >>>>>>> </rant> >>> >>>>>>> >>> >>>>>> >>> >>>>>> well, this is open source. Pull requests are always appreciated, >>> >>>>>> anything else happens as time permits ;) >>> >>>>>> >>> >>>>>> >>> >>>>>>> OS: CentOS 7 >>> >>>>>>> RSyslog: >>> >>>>>>> rsyslog-8.11.0-1.el7.x86_64 >>> >>>>>>> rsyslog-relp-8.11.0-1.el7.x86_64 >>> >>>>>>> rsyslog-gnutls-8.11.0-1.el7.x86_64 >>> >>>>>>> >>> >>>>>>> So basically what I am trying to achieve is the following: >>> >>>>>>> >>> >>>>>>> - Log remotely to a rsyslog server >>> >>>>>>> - Turn off the remote server ( via firewall ) >>> >>>>>>> - Have logs be cached locally and saved to disk >>> >>>>>>> - Restart client server >>> >>>>>>> - Turn remote server back on >>> >>>>>>> - See cached logs appear in the remote server >>> >>>>>>> >>> >>>>>>> It does not work... >>> >>>>>>> >>> >>>>>>> - So more specifically, if I turn the firewall off, log a few >>> >>>>>>> >>> >>>>>> messages >>> >>>> >>> >>>>> and turn it back on then the caching works and I get the >>> messages. >>> >>>>>>> - If however I restart the client server, the logs never make >>> it to >>> >>>>>>> >>> >>>>>> the >>> >>>>>> >>> >>>>>>> remote sever, I can see the logs in the cached file but it >>> does not >>> >>>>>>> >>> >>>>>> get >>> >>>>>> >>> >>>>>>> send to the remote server. >>> >>>>>>> >>> >>>>>>> My config on the client: >>> >>>>>>> #### MODULES #### >>> >>>>>>> module(load="imuxsock") # provides support for local system >>> logging >>> >>>>>>> >>> >>>>>> (e.g. >>> >>>> >>> >>>>> via logger command) >>> >>>>>>> module(load="imklog") # provides kernel logging support >>> (previously >>> >>>>>>> >>> >>>>>> done >>> >>>>>> >>> >>>>>>> by rklogd) >>> >>>>>>> >>> >>>>>>> #### GLOBAL DIRECTIVES #### >>> >>>>>>> $IncludeConfig /etc/rsyslog.d/*.conf >>> >>>>>>> >>> >>>>>>> #### RULES #### >>> >>>>>>> >>> >>>>>>> *.info;mail.none;authpriv.none;cron.none >>> >>>>>>> >>> >>>>>> /var/log/messages >>> >>>> >>> >>>>> authpriv.* >>> >>>>>>> >>> >>>>>> /var/log/secure >>> >>>> >>> >>>>> mail.* >>> >>>>>>> >>> >>>>>> /var/log/maillog >>> >>>> >>> >>>>> cron.* >>> /var/log/cron >>> >>>>>>> *.emerg >>> :omusrmsg:* >>> >>>>>>> uucp,news.crit >>> >>>>>>> >>> >>>>>> /var/log/spooler >>> >>>> >>> >>>>> local7.* >>> >>>>>>> >>> >>>>>> /var/log/boot.log >>> >>>> >>> >>>>> >>> >>>>>>> # ### begin forwarding rule ### >>> >>>>>>> $WorkDirectory /var/lib/rsyslog # where to place spool files >>> >>>>>>> $MainMsgQueueFileName LocalCaching # unique name prefix for spool >>> >>>>>>> >>> >>>>>> files >>> >>>> >>> >>>>> $MainMsgQueueSaveOnShutdown on # save messages to disk on shutdown >>> >>>>>>> # $MainMsgQueueType LinkedList >>> >>>>>>> $MainMsgQueueType Disk >>> >>>>>>> $MainMsgResumeRetryCount -1 # infinite retries if host is down >>> >>>>>>> >>> >>>>>>> *.* @@192.168.8.253:514 >>> >>>>>>> >>> >>>>>>> # ### end of the forwarding rule ### >>> >>>>>>> >>> >>>>>>> My config on the remote server: >>> >>>>>>> module(load="imuxsock") # provides support for local system >>> logging >>> >>>>>>> >>> >>>>>> (e.g. >>> >>>> >>> >>>>> via logger command) >>> >>>>>>> module(load="imklog") # provides kernel logging support >>> (previously >>> >>>>>>> >>> >>>>>> done >>> >>>>>> >>> >>>>>>> by rklogd) >>> >>>>>>> module(load="imtcp") # needs to be done just once >>> >>>>>>> input(type="imtcp" port="514") >>> >>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >>> >>>>>>> $IncludeConfig /etc/rsyslog.d/*.conf >>> >>>>>>> *.info;mail.none;authpriv.none;cron.none >>> >>>>>>> >>> >>>>>> /var/log/messages >>> >>>> >>> >>>>> authpriv.* >>> >>>>>>> >>> >>>>>> /var/log/secure >>> >>>> >>> >>>>> mail.* >>> >>>>>>> >>> >>>>>> /var/log/maillog >>> >>>> >>> >>>>> cron.* >>> /var/log/cron >>> >>>>>>> *.emerg >>> :omusrmsg:* >>> >>>>>>> uucp,news.crit >>> >>>>>>> >>> >>>>>> /var/log/spooler >>> >>>> >>> >>>>> local7.* >>> >>>>>>> >>> >>>>>>> Any pointers would be appreciated. I am hoping I am missing >>> something >>> >>>>>>> obvious or misunderstanding what I am suppose to be doing. >>> >>>>>>> >>> >>>>>>> >>> >>>>>> You should run rsyslog in such a situation in debug mode.From the >>> >>>>>> debug log, we can see why it thinks it can't deliver to the remote >>> >>>>>> system (well, hopefully ;)). >>> >>>>>> >>> >>>>>> HTH >>> >>>>>> Rainer >>> >>>>>> >>> >>>>>> Regards >>> >>>>>>> >>> >>>>>>> -- >>> >>>>>>> Gerhardus Geldenhuis >>> >>>>>>> _______________________________________________ >>> >>>>>>> rsyslog mailing list >>> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> >>>>>>> http://www.rsyslog.com/professional-services/ >>> >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> >>>>>>> >>> >>>>>> myriad >>> >>>> >>> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you >>> >>>>>> DON'T LIKE THAT. >>> >>>>>> _______________________________________________ >>> >>>>>> rsyslog mailing list >>> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> >>>>>> http://www.rsyslog.com/professional-services/ >>> >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> >>>>>> myriad >>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you >>> >>>>>> DON'T LIKE THAT. >>> >>>>>> >>> >>>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> Gerhardus Geldenhuis >>> >>>>> _______________________________________________ >>> >>>>> rsyslog mailing list >>> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> >>>>> http://www.rsyslog.com/professional-services/ >>> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> >>>>> >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you >>> >>>> DON'T LIKE THAT. >>> >>>> _______________________________________________ >>> >>>> rsyslog mailing list >>> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> >>>> http://www.rsyslog.com/professional-services/ >>> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you >>> >>>> DON'T LIKE THAT. >>> >>>> >>> >>>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >> rsyslog mailing list >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> >> http://www.rsyslog.com/professional-services/ >>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> >> DON'T LIKE THAT. >>> >> >>> > >>> > >>> > >>> > -- >>> > Gerhardus Geldenhuis >>> > _______________________________________________ >>> > rsyslog mailing list >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > http://www.rsyslog.com/professional-services/ >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> >> >> >> -- >> Gerhardus Geldenhuis >> > > > > -- > Gerhardus Geldenhuis > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
