Nothing here: http://www.rsyslog.com/plugins/
On Tue, May 20, 2014 at 6:31 AM, masoom alam <[email protected]> wrote: > Hi David and Rainer, > > I hope you both are doing fine. Could you please point me to the link > where I can find how to write a Ruby plugin in Rsyslog. I remember you > mentioned in one of your email that is quite possible. Did not find in my > email box but later found here :D > > http://lists.adiscon.net/pipermail/rsyslog/2014-April/037237.html > > I just want to give it a try that is to include all the Ruby filters > within the Rsyslog. Please guide. > > Thanks. > > > > > > > On Mon, May 19, 2014 at 7:33 PM, David Lang <[email protected]> wrote: > >> the pstats output should also show the deliveries to logstash (at least >> if you are using anything close to a current version of rsyslog) >> >> This isn't the logstash support mailing list :-) but we've been hearing a >> lot of people talking about how logstash will drop logs when it gets >> behind, so that could be the problem. If you use RELP instead of just TCP, >> that will guarantee all the logs get to logstash, even if the TCP >> connection gets cut and has to be reestablished. But if logstash is >> dropping them after that, we won't be able to help much. >> >> it's frequently useful in situations like this to log exactly what you >> are sending to the remote system to a local file. >> >> Why is it that you are going through logstash to get to mongodb instead >> of having rsyslog deliver directly to mongodb? >> >> You mention that you have about 15 filters in logstash, have you looked >> at putting those filters in rsyslog instead? >> >> David Lang >> >> >> On Mon, 19 May 2014, masoom alam wrote: >> >> We have verified that logtash is the main culprit in dropping the logs. >>> >>> >>> >>> we are generating logs from LOIC with 800 logs per second and sending it >>> to >>> rsyslog. >>> >>> rsyslog agent is receiving all the logs even it is showing some count on >>> the upper side , i mean if i sent 3000 logs from LOIC it always shows me >>> that it receives 3020 logs , don't know about these 20.(According to my >>> imstat file) >>> >>> Now again rsyslog passes these logs to logstash which at the moment has >>> 15 >>> filters. The metric to count the logs recived at logstash is >>> http://logstash.net/docs/1.4.1/filters/metrics#flush_interval , but it >>> always showed me the less number as i received in rsyslog. At certain >>> level >>> for example ,,3000 logs/5 sec, count of rsyslog and logstash is same but >>> after 3500 , 4000 there is a difference of 30 -40 logs between rsyslog >>> and >>> logstash , our connection between rsyslog to logstash is TCP , so there >>> should be no reason for this difference , either logstash is unable to >>> parse all the msgs or rsyslog is sending less msgs to logstash. >>> >>> Any clue? >>> >>> On Sunday, May 18, 2014, David Lang <[email protected]> wrote: >>> >>>> If you are sending the right contents in the packet, LOIC should be just >>>> >>> fine. >>> >>>> >>>> Now, you haven't said what version of rsyslog you are using, your >>>> >>> configuration, or even talked about what speed network you are on (let >>> alone system specs), so figuring out what's wrong is not possible yet :-) >>> >>>> >>>> That said, we've had people getting around 400,000 packets/sec through >>>> >>> rsyslog, so you are probably not hitting any fundamental limit at 10,000 >>> packets/sec. But there are a lot of things to look at to figure this out. >>> >>>> >>>> start by configuring impstats (set it to dump stats every 10 sec for now >>>> >>> at these traffic levels) so that we can see the what's happening inside >>> rsyslog (to see if the problem is getting the logs in, or out of rsyslog) >>> >>>> >>>> version info (rsyslog, distro, kernel) >>>> >>>> when you say you put things on a switch, is this a 100Mb switch, Gb >>>> >>> switch, managed (so you can get stats from the switch)???? >>> >>>> >>>> what's your rsyslog config? >>>> >>>> what do you use for name resolution (/etc/hosts, local DNS, nearby DNS, >>>> >>> ISP DNS, LDAP, ????) >>> >>>> >>>> UDP isn't necessarily faster than TCP, but it's a whole lot easier to >>>> >>> setup a UDP test, so let's stick with that for now. There's nothing in >>> the >>> syslog protocol to add reliability to UDP >>> >>>> >>>> David Lang >>>> >>>> >>>> On Sun, 18 May 2014, masoom alam wrote: >>>> >>>> Dear All, >>>>> >>>>> I hope every one is doing fine. We were doing stress testing of Rsyslog >>>>> >>>> and >>> >>>> found few problems (in our setup and not in Rsyslog :)) that I want to >>>>> discuss at this forum. >>>>> >>>>> >>>>> 1. We were using LOIC (LOIC is DDOS attack tool - your anti virus >>>>> will >>>>> delete it :) - disclaimer: handle it with care) for generation of UDP >>>>> packets. We created a customized log message. The speed of Packets >>>>> >>>> sent by >>> >>>> LOIC is very very high, that is some thing 20,000 packets in 2 sec. >>>>> for >>>>> example. Every thing is fine if we use point to point connection >>>>> >>>> between >>> >>>> Rsyslog machine and MangoDB machine. Point to point means connection >>>>> >>>> via >>> >>>> cross cable and not through a switch. We calculated the no. of packets >>>>> >>>> sent >>> >>>> by LOIC and no. of packets received by Rsyslog and then written by >>>>> >>>> MongoDB >>> >>>> after parsing by Logtash is fairly equal. >>>>> 2. However, if we connect both MongoDB and Rsyslog through a switch >>>>> >>>> LAN, >>> >>>> there is a packet drop at the Rsyslog end, some what between 300-500 >>>>> packets depending on the speed of LOIC - thus 300-500 lesser logs are >>>>> written to MongoDB. >>>>> >>>>> >>>>> What we concluded and I want your expert opinion on this: >>>>> >>>>> >>>>> 1. It seems LOIC is the not the right choice for traffic generation >>>>> for >>>>> Rsyslog - that is stress testing of Rsyslog. It sends packets via UDP >>>>> >>>> 514, >>> >>>> but essentially it does not follow Syslog Protocol. Now, we are trying >>>>> >>>> to >>> >>>> understand: Is there some sort of reliability achieved in Syslog >>>>> >>>> protocol >>> >>>> even if packets are sent on UDP 514? BTW, i am well aware that UDP is >>>>> >>>> for >>> >>>> faster communication but at the expense of reliability. Why we are >>>>> >>>> saying >>> >>>> that there is a problem at the LOIC - that is traffic generation end - >>>>> because when we selected to send traffic via TCP on LOIC, due to >>>>> speed >>>>> >>>> it >>> >>>> combines log packets and Rsyslog reports an error in its log. The net >>>>> effect of this operation is that Rsyslog combines arbitrary no. of >>>>> logs >>>>> together and then give to Logtash, which does not understand and >>>>> leave >>>>> >>>> it >>> >>>> un parsed. >>>>> 2. What options do we have, either we use a python library to >>>>> generate >>>>> traffic, write it to /var/log/messages and ask the Rsyslog to send >>>>> that >>>>> traffic to another Rsyslog?. Does using this way guarantees that >>>>> there >>>>> >>>> will >>> >>>> be no drop of Log messages even in UDP? >>>>> 3. But what I am interested in understanding what is the reliability >>>>> mechanism provided by Rsyslog in general in the case of UDP. After >>>>> all >>>>> >>>> each >>> >>>> n every log is a very important piece of information and can destroy >>>>> >>>> the >>> >>>> reputation of an organization. >>>>> 4. Even if the some reliability can be achieved at the Rsyslog end, >>>>> how >>>>> can we avoid - up to maximum extent - the possibility of log drop >>>>> >>>> between >>> >>>> Rsyslog to Logtash. Logtash is a very small program than Rsyslog. >>>>> >>>> Rsyslog >>> >>>> in our setup is used only for Buffering - thus, what parameters in the >>>>> .conf file of Rsyslog should be changed to achieve this reliability. >>>>> >>>> Please >>> >>>> note that, our Rsyslog and Logtash are setup at the same system - so >>>>> no >>>>> network latency at this end. >>>>> 5. In all this setup, the performance of LogAnalyzer was very >>>>> pathetic >>>>> in filtering and running other queries. So we choose to write a >>>>> simple >>>>> >>>> PHP >>> >>>> web page for displaying logs and it is running much much faster than >>>>> LogAnalyzer. >>>>> 6. Are we on the right path for checking reliability and stress >>>>> >>>> testing? >>> >>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>>> >>>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>>> >>>> >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
