Can you post the complete pstats? Sent from phone, thus brief. Am 19.05.2014 15:07 schrieb "masoom alam" <[email protected]>:
> We have verified that logtash is the main culprit in dropping the logs. > > > > we are generating logs from LOIC with 800 logs per second and sending it to > rsyslog. > > rsyslog agent is receiving all the logs even it is showing some count on > the upper side , i mean if i sent 3000 logs from LOIC it always shows me > that it receives 3020 logs , don't know about these 20.(According to my > imstat file) > > Now again rsyslog passes these logs to logstash which at the moment has 15 > filters. The metric to count the logs recived at logstash is > http://logstash.net/docs/1.4.1/filters/metrics#flush_interval , but it > always showed me the less number as i received in rsyslog. At certain level > for example ,,3000 logs/5 sec, count of rsyslog and logstash is same but > after 3500 , 4000 there is a difference of 30 -40 logs between rsyslog and > logstash , our connection between rsyslog to logstash is TCP , so there > should be no reason for this difference , either logstash is unable to > parse all the msgs or rsyslog is sending less msgs to logstash. > > Any clue? > > On Sunday, May 18, 2014, David Lang <[email protected]> wrote: > > If you are sending the right contents in the packet, LOIC should be just > fine. > > > > Now, you haven't said what version of rsyslog you are using, your > configuration, or even talked about what speed network you are on (let > alone system specs), so figuring out what's wrong is not possible yet :-) > > > > That said, we've had people getting around 400,000 packets/sec through > rsyslog, so you are probably not hitting any fundamental limit at 10,000 > packets/sec. But there are a lot of things to look at to figure this out. > > > > start by configuring impstats (set it to dump stats every 10 sec for now > at these traffic levels) so that we can see the what's happening inside > rsyslog (to see if the problem is getting the logs in, or out of rsyslog) > > > > version info (rsyslog, distro, kernel) > > > > when you say you put things on a switch, is this a 100Mb switch, Gb > switch, managed (so you can get stats from the switch)???? > > > > what's your rsyslog config? > > > > what do you use for name resolution (/etc/hosts, local DNS, nearby DNS, > ISP DNS, LDAP, ????) > > > > UDP isn't necessarily faster than TCP, but it's a whole lot easier to > setup a UDP test, so let's stick with that for now. There's nothing in the > syslog protocol to add reliability to UDP > > > > David Lang > > > > > > On Sun, 18 May 2014, masoom alam wrote: > > > >> Dear All, > >> > >> I hope every one is doing fine. We were doing stress testing of Rsyslog > and > >> found few problems (in our setup and not in Rsyslog :)) that I want to > >> discuss at this forum. > >> > >> > >> 1. We were using LOIC (LOIC is DDOS attack tool - your anti virus will > >> delete it :) - disclaimer: handle it with care) for generation of UDP > >> packets. We created a customized log message. The speed of Packets > sent by > >> LOIC is very very high, that is some thing 20,000 packets in 2 sec. > for > >> example. Every thing is fine if we use point to point connection > between > >> Rsyslog machine and MangoDB machine. Point to point means connection > via > >> cross cable and not through a switch. We calculated the no. of packets > sent > >> by LOIC and no. of packets received by Rsyslog and then written by > MongoDB > >> after parsing by Logtash is fairly equal. > >> 2. However, if we connect both MongoDB and Rsyslog through a switch > LAN, > >> there is a packet drop at the Rsyslog end, some what between 300-500 > >> packets depending on the speed of LOIC - thus 300-500 lesser logs are > >> written to MongoDB. > >> > >> > >> What we concluded and I want your expert opinion on this: > >> > >> > >> 1. It seems LOIC is the not the right choice for traffic generation > for > >> Rsyslog - that is stress testing of Rsyslog. It sends packets via UDP > 514, > >> but essentially it does not follow Syslog Protocol. Now, we are trying > to > >> understand: Is there some sort of reliability achieved in Syslog > protocol > >> even if packets are sent on UDP 514? BTW, i am well aware that UDP is > for > >> faster communication but at the expense of reliability. Why we are > saying > >> that there is a problem at the LOIC - that is traffic generation end - > >> because when we selected to send traffic via TCP on LOIC, due to speed > it > >> combines log packets and Rsyslog reports an error in its log. The net > >> effect of this operation is that Rsyslog combines arbitrary no. of > logs > >> together and then give to Logtash, which does not understand and leave > it > >> un parsed. > >> 2. What options do we have, either we use a python library to generate > >> traffic, write it to /var/log/messages and ask the Rsyslog to send > that > >> traffic to another Rsyslog?. Does using this way guarantees that there > will > >> be no drop of Log messages even in UDP? > >> 3. But what I am interested in understanding what is the reliability > >> mechanism provided by Rsyslog in general in the case of UDP. After all > each > >> n every log is a very important piece of information and can destroy > the > >> reputation of an organization. > >> 4. Even if the some reliability can be achieved at the Rsyslog end, > how > >> can we avoid - up to maximum extent - the possibility of log drop > between > >> Rsyslog to Logtash. Logtash is a very small program than Rsyslog. > Rsyslog > >> in our setup is used only for Buffering - thus, what parameters in the > >> .conf file of Rsyslog should be changed to achieve this reliability. > Please > >> note that, our Rsyslog and Logtash are setup at the same system - so > no > >> network latency at this end. > >> 5. In all this setup, the performance of LogAnalyzer was very pathetic > >> in filtering and running other queries. So we choose to write a simple > PHP > >> web page for displaying logs and it is running much much faster than > >> LogAnalyzer. > >> 6. Are we on the right path for checking reliability and stress > testing? > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > > -- > Sent from noir > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
