Hi David and Rainer, I hope you both are doing fine. Could you please point me to the link where I can find how to write a Ruby plugin in Rsyslog. I remember you mentioned in one of your email that is quite possible. Did not find in my email box but later found here :D
http://lists.adiscon.net/pipermail/rsyslog/2014-April/037237.html I just want to give it a try that is to include all the Ruby filters within the Rsyslog. Please guide. Thanks. On Mon, May 19, 2014 at 7:33 PM, David Lang <[email protected]> wrote: > the pstats output should also show the deliveries to logstash (at least if > you are using anything close to a current version of rsyslog) > > This isn't the logstash support mailing list :-) but we've been hearing a > lot of people talking about how logstash will drop logs when it gets > behind, so that could be the problem. If you use RELP instead of just TCP, > that will guarantee all the logs get to logstash, even if the TCP > connection gets cut and has to be reestablished. But if logstash is > dropping them after that, we won't be able to help much. > > it's frequently useful in situations like this to log exactly what you are > sending to the remote system to a local file. > > Why is it that you are going through logstash to get to mongodb instead of > having rsyslog deliver directly to mongodb? > > You mention that you have about 15 filters in logstash, have you looked at > putting those filters in rsyslog instead? > > David Lang > > > On Mon, 19 May 2014, masoom alam wrote: > > We have verified that logtash is the main culprit in dropping the logs. >> >> >> >> we are generating logs from LOIC with 800 logs per second and sending it >> to >> rsyslog. >> >> rsyslog agent is receiving all the logs even it is showing some count on >> the upper side , i mean if i sent 3000 logs from LOIC it always shows me >> that it receives 3020 logs , don't know about these 20.(According to my >> imstat file) >> >> Now again rsyslog passes these logs to logstash which at the moment has 15 >> filters. The metric to count the logs recived at logstash is >> http://logstash.net/docs/1.4.1/filters/metrics#flush_interval , but it >> always showed me the less number as i received in rsyslog. At certain >> level >> for example ,,3000 logs/5 sec, count of rsyslog and logstash is same but >> after 3500 , 4000 there is a difference of 30 -40 logs between rsyslog and >> logstash , our connection between rsyslog to logstash is TCP , so there >> should be no reason for this difference , either logstash is unable to >> parse all the msgs or rsyslog is sending less msgs to logstash. >> >> Any clue? >> >> On Sunday, May 18, 2014, David Lang <[email protected]> wrote: >> >>> If you are sending the right contents in the packet, LOIC should be just >>> >> fine. >> >>> >>> Now, you haven't said what version of rsyslog you are using, your >>> >> configuration, or even talked about what speed network you are on (let >> alone system specs), so figuring out what's wrong is not possible yet :-) >> >>> >>> That said, we've had people getting around 400,000 packets/sec through >>> >> rsyslog, so you are probably not hitting any fundamental limit at 10,000 >> packets/sec. But there are a lot of things to look at to figure this out. >> >>> >>> start by configuring impstats (set it to dump stats every 10 sec for now >>> >> at these traffic levels) so that we can see the what's happening inside >> rsyslog (to see if the problem is getting the logs in, or out of rsyslog) >> >>> >>> version info (rsyslog, distro, kernel) >>> >>> when you say you put things on a switch, is this a 100Mb switch, Gb >>> >> switch, managed (so you can get stats from the switch)???? >> >>> >>> what's your rsyslog config? >>> >>> what do you use for name resolution (/etc/hosts, local DNS, nearby DNS, >>> >> ISP DNS, LDAP, ????) >> >>> >>> UDP isn't necessarily faster than TCP, but it's a whole lot easier to >>> >> setup a UDP test, so let's stick with that for now. There's nothing in the >> syslog protocol to add reliability to UDP >> >>> >>> David Lang >>> >>> >>> On Sun, 18 May 2014, masoom alam wrote: >>> >>> Dear All, >>>> >>>> I hope every one is doing fine. We were doing stress testing of Rsyslog >>>> >>> and >> >>> found few problems (in our setup and not in Rsyslog :)) that I want to >>>> discuss at this forum. >>>> >>>> >>>> 1. We were using LOIC (LOIC is DDOS attack tool - your anti virus will >>>> delete it :) - disclaimer: handle it with care) for generation of UDP >>>> packets. We created a customized log message. The speed of Packets >>>> >>> sent by >> >>> LOIC is very very high, that is some thing 20,000 packets in 2 sec. for >>>> example. Every thing is fine if we use point to point connection >>>> >>> between >> >>> Rsyslog machine and MangoDB machine. Point to point means connection >>>> >>> via >> >>> cross cable and not through a switch. We calculated the no. of packets >>>> >>> sent >> >>> by LOIC and no. of packets received by Rsyslog and then written by >>>> >>> MongoDB >> >>> after parsing by Logtash is fairly equal. >>>> 2. However, if we connect both MongoDB and Rsyslog through a switch >>>> >>> LAN, >> >>> there is a packet drop at the Rsyslog end, some what between 300-500 >>>> packets depending on the speed of LOIC - thus 300-500 lesser logs are >>>> written to MongoDB. >>>> >>>> >>>> What we concluded and I want your expert opinion on this: >>>> >>>> >>>> 1. It seems LOIC is the not the right choice for traffic generation >>>> for >>>> Rsyslog - that is stress testing of Rsyslog. It sends packets via UDP >>>> >>> 514, >> >>> but essentially it does not follow Syslog Protocol. Now, we are trying >>>> >>> to >> >>> understand: Is there some sort of reliability achieved in Syslog >>>> >>> protocol >> >>> even if packets are sent on UDP 514? BTW, i am well aware that UDP is >>>> >>> for >> >>> faster communication but at the expense of reliability. Why we are >>>> >>> saying >> >>> that there is a problem at the LOIC - that is traffic generation end - >>>> because when we selected to send traffic via TCP on LOIC, due to speed >>>> >>> it >> >>> combines log packets and Rsyslog reports an error in its log. The net >>>> effect of this operation is that Rsyslog combines arbitrary no. of >>>> logs >>>> together and then give to Logtash, which does not understand and leave >>>> >>> it >> >>> un parsed. >>>> 2. What options do we have, either we use a python library to generate >>>> traffic, write it to /var/log/messages and ask the Rsyslog to send >>>> that >>>> traffic to another Rsyslog?. Does using this way guarantees that there >>>> >>> will >> >>> be no drop of Log messages even in UDP? >>>> 3. But what I am interested in understanding what is the reliability >>>> mechanism provided by Rsyslog in general in the case of UDP. After all >>>> >>> each >> >>> n every log is a very important piece of information and can destroy >>>> >>> the >> >>> reputation of an organization. >>>> 4. Even if the some reliability can be achieved at the Rsyslog end, >>>> how >>>> can we avoid - up to maximum extent - the possibility of log drop >>>> >>> between >> >>> Rsyslog to Logtash. Logtash is a very small program than Rsyslog. >>>> >>> Rsyslog >> >>> in our setup is used only for Buffering - thus, what parameters in the >>>> .conf file of Rsyslog should be changed to achieve this reliability. >>>> >>> Please >> >>> note that, our Rsyslog and Logtash are setup at the same system - so no >>>> network latency at this end. >>>> 5. In all this setup, the performance of LogAnalyzer was very pathetic >>>> in filtering and running other queries. So we choose to write a simple >>>> >>> PHP >> >>> web page for displaying logs and it is running much much faster than >>>> LogAnalyzer. >>>> 6. Are we on the right path for checking reliability and stress >>>> >>> testing? >> >>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >>> >>> >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
