Hi Rainer and David n every one, While doing troubleshooting, we have found the following:
1. Firstly, LOIC is extremely unpredictable. We ran many times overnight experiments and when we come in the morning, the count on the LOIC machine to Rsyslog impstats significantly varies. We noted that LOIC is not built for log stress testing.... 2. We then found tcpflood - native Rsyslog utility and tested our setup with it. Ironically following are the results (see the figure). We also setup tcpdump both at the tcpflood sending system and receiving Rsyslog system. You will see that tcpdump even at source is not showing the correct number of packet sent. We have sent 0.1 million packets of logs. We noted that TCPdump at source is showing less than what was sent by tcpflood. Plus the destination tcpdump shows even less, but the comedy point is that Rsyslog impstats are showing higher packets received, even higher than source. tcpdump :). 3. However, the good thing is that the no. of packets received by Rsyslog and stored by MongoDB after parsing by Logtash are exactly equal. 4. We are trying to understand why packets disappear within the network. Our network is Gigabit ethernet, with manageable D-Linux switch. [image: Inline image 1] Please comment on the overall experiment and tell us where are we doing wrong.... On Wed, May 21, 2014 at 3:39 PM, Rainer Gerhards <[email protected]>wrote: > As a side note you should temporarily remove logstash from the picture. > Ship to rsyslog. If the problem persists, we can much better > troubleshoot. If it does not persist, you know you need to talk to the > logstash folks. > > Rainer > > Sent from phone, thus brief. > Am 21.05.2014 12:25 schrieb "Rainer Gerhards" <[email protected]>: > > > This sounds like logstash isn't fast enough in some cases and the relp > > window fills up. You can try to increase it: > > > > http://www.rsyslog.com/doc/omrelp.html > > > > But that won't help if logstash can't keep up in general. > > > > The original problem went away? > > > > Rainer > > > > Sent from phone, thus brief. > > Am 21.05.2014 12:18 schrieb "Muhammad Asif" <[email protected]>: > > > >> Hi Everyone, > >> > >> We have configured rsyslog 7.6.3 and logstash on Desktop Machine (CPU: > >> core > >> i3, RAM: 4GB). We have configured statistics on both rsyslog and > logstash. > >> RELP is being used between both. When we sent 30716 logs in 10 sec from > >> LOIC, there is no loss on logstash but rsyslog sends 256 logs chunk to > >> logstash and a delay of about 3 sec is noticed between 256 logs chunk. > >> Similarly when we sent 68529, again no loss of logs on logstash and same > >> delay is noticed. Please find the attached pstats files of Desktop. > >> > >> We have performed the same activity on a Server machine (CPU: Xeon Quad > >> core, RAM 32 GB), no delay is noticed in this case. Everything is > fine.On > >> server we are using rsyslogd 8.2. Why delay is not being noticed during > >> consecutive 256 logs chunk. > >> > >> Can you elaborate the reason of delay in Desktop case and can we > increase > >> 256 logs chunk which rsyslog send to logstash. > >> > >> The server and desktop stats files are attached. > >> > >> Thanks > >> > >> > >> > >> > >> On Tue, May 20, 2014 at 7:23 PM, Rainer Gerhards > >> <[email protected]>wrote: > >> > >> > On Tue, May 20, 2014 at 4:10 PM, masoom alam <[email protected]> > >> > wrote: > >> > > >> > > are you asking for this? > >> > > > >> > > > >> > no. When you run impstats, you get statistics records. Without knowing > >> your > >> > config, I do not know how exactly they look and where they are written > >> to, > >> > but they contain the statistics data. I though that you quoted these > >> lines > >> > when you said "n records where received by rsyslog". What I would like > >> to > >> > see is how many records were pushed to the forwarding action. That way > >> we > >> > know how many were actually sent to logstash. > >> > > >> > If the stats are a bit unclear to you, I suggest reading: > >> > > >> > http://www.rsyslog.com/doc/impstats.html > >> > > >> > and - at least as important - the links that go out to the page that > >> > describe the actual counters. There is a wealth of information in them > >> in > >> > cases like these. > >> > > >> > Rainer > >> > > >> > see attachement > >> > > > >> > > > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> > DON'T LIKE THAT. > >> > > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. >
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
