2015-07-14 17:32 GMT+02:00 Gerhardus Geldenhuis <[email protected]>: > Thanks David, > I configured RELP but it has not made a difference... > > More specifically if I turn on the firewall by > > 1. running: systemctl start firewalld on the master server > 2. and then start generating 40 000 log entries on the client, and wait > for it to finish > 3. run: systemctl stop firewalld > 4. Wait for logs to come over > 5. Count logs > > I still get 24767 messages instead of 40000.
can you share the impstats data with us? We may get a better understanding of what's going on based on it... Rainer > > If however I physically turn of the network on the master machine using > VMWare, I do get 40 000 messages when I connect the network back again. > > So why is RELP handling it like this surely if it does not get any response > from the firewall for if the firewall sends it to a black hole then it > should not mark messages as delivered. I am also willing to admit that my > testing methodology is flawed but then I would like to understand why > exactly and at the moment I feel my testing methodology represents a real > world scenario which would make loosing logs a reality even if you use RELP. > > Importantly I enable the firewall before starting the log test so it should > all be blocked from the very beginning. > > Regards > > On 14 July 2015 at 12:44, David Lang <[email protected]> wrote: > >> One issue you are probably running into is the fact that TCP is not >> reliable enough. TCP is 'reliable' in that if a packet gets dropped it will >> detect and re-send it as part of the same connection, but there is a >> problem when the connection is interrupted (as it is in your test by the >> firewall) >> >> When the sending rsyslog hands the message to the OS on the sending >> system, it is told "yes, I have this and will deliver it" by the OS. But if >> the OS is unable to deliver it (because the network connection breaks), >> there is no way for the OS to go back to the sending application and say >> "oops, I really can't get this through to the other end" so the messages >> are lost. >> >> This is what the RELP protocol was developed to fix. It does application >> level acks for the messages so that if the network is interrupted, it knows >> what messages didn't get through and will re-send them. switch to >> omrelp/imrelp instead of omfwd/imtcp and you should avoid loosing any >> messages >> >> If you make the firewall send resets rather than dropping packets, the >> sending machine will detect the problem faster. >> >> >> This would account for some messages being lost between the time that you >> ahve the firewall block the connection and the time that rsyslog detects >> this. But once rsyslog detects this and starts queueing messages, those >> messages should be sent when rsyslog re-establishes the connection. >> >> enable impstats writing stats rapidly (say every 10 seconds or so for >> testing) >> >> then block the connection and send a small number of messages until >> rsyslog logs locally that it's unable to send and the queue for the action >> starts filling. >> >> then do your write of 40000 test messages >> >> then open the firewall and you should see all the messages get sent once >> rsyslog retries and notices that the connection is open again. >> >> David Lang >> >> On Tue, 14 Jul 2015, Gerhardus Geldenhuis wrote: >> >> Date: Tue, 14 Jul 2015 11:20:46 +0100 >>> From: Gerhardus Geldenhuis <[email protected]> >>> Reply-To: rsyslog-users <[email protected]> >>> To: rsyslog-users <[email protected]> >>> Subject: Re: [rsyslog] Disk queue to flush after restart >>> >>> Hi, >>> I don't seem to be making as much progress as I would hope for. I now have >>> the following client config: >>> action( type="omfwd" >>> target="192.168.8.134" >>> port="514" >>> protocol="tcp" >>> queue.size="50000" >>> queue.type="LinkedList" >>> queue.spoolDirectory="/var/lib/rsyslog" >>> queue.filename="testforwardingqueue" >>> queue.lowwatermark="20000" >>> queue.highwatermark="35000" >>> queue.discardmark="50000" >>> queue.maxfilesize="1g" >>> queue.saveonshutdown="on" >>> # queue.fulldelaymark="2500" >>> ) >>> >>> I have made the numbers smaller to be able to do testing that will finish >>> in a reasonable amount of time. >>> >>> My process is: >>> >>> 1. Start firewall on master server to block all access. >>> 2. On client server start logging using: for i in $(seq 1 40000); do >>> logger -p error "############### TESTING ########### SEQ001-${i}"; done >>> 3. Wait for for loop to finish and check if all entries are contained >>> locally >>> 4. watch 'ls -l /var/lib/rsyslog' to see if queue gets created. >>> 5. Disable firewall on master >>> 6. run logger -p error with one message to get logs going again >>> 7. Wait for logs to arrive on master server >>> 8. grep 'SEQ001' /var/log/messages -c >>> >>> >>> The problem I am getting is that the number of messages I receive on the >>> remote server does not correspond with what I get on the master server. >>> I only get 24768 messages on the master server. >>> >>> Is there anything else that I have missed out of the configuration? I >>> can't >>> see anything that would cause my messages to not arrive. The queue file I >>> specified grows in size but interestingly when I grep'ed it for SEQ005 I >>> only get 15232 matches. I did the grep after re-enabling the firewall. >>> >>> I thought that it might be because syslog thinks my messages are the same >>> and then marks them as duplicates. I then added: >>> for i in $(seq 1 40000); do logger -p error "############### TESTING >>> ########### SEQ005-${i}-$(openssl rand -base64 32)"; done >>> >>> to try and prevent that. However if the firewall is not enabled then all >>> 40k messages arrive on the master server. >>> >>> Any help would be appreciated. >>> >>> Regards >>> >>> On 13 July 2015 at 14:20, Rainer Gerhards <[email protected]> >>> wrote: >>> >>> 2015-07-13 15:18 GMT+02:00 Gerhardus Geldenhuis >>>> <[email protected]>: >>>> >>>>> Hi, >>>>> Thanks for the replies. I think the bulk of my problem was mixing old >>>>> and >>>>> new config and I much further along to get something working. I have >>>>> discovered a few other niggles but will report back once I have >>>>> something >>>>> working properly. As far as pull requests go, I would really consider >>>>> >>>> doing >>>> >>>>> so but as always time is a factor. >>>>> >>>> >>>> yup - for everyone ;) And that is what it is like it currently is ;) >>>> >>>> It does bug me so much that I must just >>>>> end up doing it for the documentation. I will give debug mode a go. >>>>> >>>>> Regards >>>>> >>>>> On 13 July 2015 at 13:17, Rainer Gerhards <[email protected]> >>>>> >>>> wrote: >>>> >>>>> >>>>> 2015-07-10 13:40 GMT+02:00 Gerhardus Geldenhuis >>>>>> <[email protected]>: >>>>>> >>>>>>> Hi >>>>>>> I am struggling a bit to get rsyslog to work as described. >>>>>>> >>>>>>> <rant> >>>>>>> Firstly the documentation is a struggle. There is some reference to >>>>>>> >>>>>> old >>>> >>>>> and >>>>>> >>>>>>> new style configuration but no clear differentiation between the two. >>>>>>> >>>>>> What >>>>>> >>>>>>> makes it more confusing is that documents like >>>>>>> http://www.rsyslog.com/doc/queues.html then refers to what looks like >>>>>>> >>>>>> the >>>>>> >>>>>>> old style of config and none of the examples contains new syntax >>>>>>> >>>>>> examples. >>>>>> >>>>>>> >>>>>>> There was also an expectation that the new rsyslog package would >>>>>>> >>>>>> install >>>> >>>>> a >>>>>> >>>>>>> new style config but that turned out to be not the case. I deleted the >>>>>>> config file and did a yum reinstall just to be sure. >>>>>>> </rant> >>>>>>> >>>>>> >>>>>> well, this is open source. Pull requests are always appreciated, >>>>>> anything else happens as time permits ;) >>>>>> >>>>>> >>>>>>> OS: CentOS 7 >>>>>>> RSyslog: >>>>>>> rsyslog-8.11.0-1.el7.x86_64 >>>>>>> rsyslog-relp-8.11.0-1.el7.x86_64 >>>>>>> rsyslog-gnutls-8.11.0-1.el7.x86_64 >>>>>>> >>>>>>> So basically what I am trying to achieve is the following: >>>>>>> >>>>>>> - Log remotely to a rsyslog server >>>>>>> - Turn off the remote server ( via firewall ) >>>>>>> - Have logs be cached locally and saved to disk >>>>>>> - Restart client server >>>>>>> - Turn remote server back on >>>>>>> - See cached logs appear in the remote server >>>>>>> >>>>>>> It does not work... >>>>>>> >>>>>>> - So more specifically, if I turn the firewall off, log a few >>>>>>> >>>>>> messages >>>> >>>>> and turn it back on then the caching works and I get the messages. >>>>>>> - If however I restart the client server, the logs never make it to >>>>>>> >>>>>> the >>>>>> >>>>>>> remote sever, I can see the logs in the cached file but it does not >>>>>>> >>>>>> get >>>>>> >>>>>>> send to the remote server. >>>>>>> >>>>>>> My config on the client: >>>>>>> #### MODULES #### >>>>>>> module(load="imuxsock") # provides support for local system logging >>>>>>> >>>>>> (e.g. >>>> >>>>> via logger command) >>>>>>> module(load="imklog") # provides kernel logging support (previously >>>>>>> >>>>>> done >>>>>> >>>>>>> by rklogd) >>>>>>> >>>>>>> #### GLOBAL DIRECTIVES #### >>>>>>> $IncludeConfig /etc/rsyslog.d/*.conf >>>>>>> >>>>>>> #### RULES #### >>>>>>> >>>>>>> *.info;mail.none;authpriv.none;cron.none >>>>>>> >>>>>> /var/log/messages >>>> >>>>> authpriv.* >>>>>>> >>>>>> /var/log/secure >>>> >>>>> mail.* >>>>>>> >>>>>> /var/log/maillog >>>> >>>>> cron.* /var/log/cron >>>>>>> *.emerg :omusrmsg:* >>>>>>> uucp,news.crit >>>>>>> >>>>>> /var/log/spooler >>>> >>>>> local7.* >>>>>>> >>>>>> /var/log/boot.log >>>> >>>>> >>>>>>> # ### begin forwarding rule ### >>>>>>> $WorkDirectory /var/lib/rsyslog # where to place spool files >>>>>>> $MainMsgQueueFileName LocalCaching # unique name prefix for spool >>>>>>> >>>>>> files >>>> >>>>> $MainMsgQueueSaveOnShutdown on # save messages to disk on shutdown >>>>>>> # $MainMsgQueueType LinkedList >>>>>>> $MainMsgQueueType Disk >>>>>>> $MainMsgResumeRetryCount -1 # infinite retries if host is down >>>>>>> >>>>>>> *.* @@192.168.8.253:514 >>>>>>> >>>>>>> # ### end of the forwarding rule ### >>>>>>> >>>>>>> My config on the remote server: >>>>>>> module(load="imuxsock") # provides support for local system logging >>>>>>> >>>>>> (e.g. >>>> >>>>> via logger command) >>>>>>> module(load="imklog") # provides kernel logging support (previously >>>>>>> >>>>>> done >>>>>> >>>>>>> by rklogd) >>>>>>> module(load="imtcp") # needs to be done just once >>>>>>> input(type="imtcp" port="514") >>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >>>>>>> $IncludeConfig /etc/rsyslog.d/*.conf >>>>>>> *.info;mail.none;authpriv.none;cron.none >>>>>>> >>>>>> /var/log/messages >>>> >>>>> authpriv.* >>>>>>> >>>>>> /var/log/secure >>>> >>>>> mail.* >>>>>>> >>>>>> /var/log/maillog >>>> >>>>> cron.* /var/log/cron >>>>>>> *.emerg :omusrmsg:* >>>>>>> uucp,news.crit >>>>>>> >>>>>> /var/log/spooler >>>> >>>>> local7.* >>>>>>> >>>>>>> Any pointers would be appreciated. I am hoping I am missing something >>>>>>> obvious or misunderstanding what I am suppose to be doing. >>>>>>> >>>>>>> >>>>>> You should run rsyslog in such a situation in debug mode.From the >>>>>> debug log, we can see why it thinks it can't deliver to the remote >>>>>> system (well, hopefully ;)). >>>>>> >>>>>> HTH >>>>>> Rainer >>>>>> >>>>>> Regards >>>>>>> >>>>>>> -- >>>>>>> Gerhardus Geldenhuis >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> >>>>>> myriad >>>> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Gerhardus Geldenhuis >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> >>> >>> >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > > > -- > Gerhardus Geldenhuis > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
