Thanks David, All the experiments Pstats will be shared on Monday for further guidance.
Thanks again. On Fri, May 23, 2014 at 10:58 PM, David Lang <[email protected]> wrote: > tcpdump will drop packets when it can't keep up. The best way to use > tcpdump is to not have it output to the screen, but write to a file > (parsing the packets to format them for the screen is _very_ expensive) > > the very low performance of tcpdump on the receiver is probably caused by > it trying to do DNS lookup for it's output, you can reduce this by adding > -n, but as noted above, you are far better off writing the output to a file > (-o ) and then reading it with another tcpdump process and having that > second process do all the output formatting. > > you can have filters on the first process, and you can configure the > smaplength (the amount of data to capture for each packet) to reduce the > volume of data to be written to disk > > I've also had good success using tcpreplay to generate traffic, I capture > data UDP data with tcpdump, and then replay it with tcpreplay, which allows > me to tune how fast I'm sending data. > > since you have a managed switch, and are using UDP, if you can make the > network quiet otherwise the switch counters will tell you how many packets > it's seeing on each port (similarly, the ifconfig data on the sender will > tell you how many packets are being sent) > > It would be extremely useful to see the pstats data from rsyslog. > > David Lang > > > On Fri, 23 May 2014, masoom alam wrote: > > Hi Rainer and David n every one, >> >> While doing troubleshooting, we have found the following: >> >> >> 1. Firstly, LOIC is extremely unpredictable. We ran many times overnight >> >> experiments and when we come in the morning, the count on the LOIC >> machine >> to Rsyslog impstats significantly varies. We noted that LOIC is not >> built >> for log stress testing.... >> 2. We then found tcpflood - native Rsyslog utility and tested our setup >> >> with it. Ironically following are the results (see the figure). We also >> setup tcpdump both at the tcpflood sending system and receiving Rsyslog >> system. You will see that tcpdump even at source is not showing the >> correct >> number of packet sent. We have sent 0.1 million packets of logs. We >> noted >> that TCPdump at source is showing less than what was sent by tcpflood. >> Plus >> the destination tcpdump shows even less, but the comedy point is that >> Rsyslog impstats are showing higher packets received, even higher than >> source. tcpdump :). >> 3. However, the good thing is that the no. of packets received by >> >> Rsyslog and stored by MongoDB after parsing by Logtash are exactly >> equal. >> 4. We are trying to understand why packets disappear within the network. >> >> Our network is Gigabit ethernet, with manageable D-Linux switch. >> >> [image: Inline image 1] >> >> Please comment on the overall experiment and tell us where are we doing >> wrong.... >> >> >> >> >> >> On Wed, May 21, 2014 at 3:39 PM, Rainer Gerhards >> <[email protected]>wrote: >> >> As a side note you should temporarily remove logstash from the picture. >>> Ship to rsyslog. If the problem persists, we can much better >>> troubleshoot. If it does not persist, you know you need to talk to the >>> logstash folks. >>> >>> Rainer >>> >>> Sent from phone, thus brief. >>> Am 21.05.2014 12:25 schrieb "Rainer Gerhards" <[email protected] >>> >: >>> >>> This sounds like logstash isn't fast enough in some cases and the relp >>>> window fills up. You can try to increase it: >>>> >>>> http://www.rsyslog.com/doc/omrelp.html >>>> >>>> But that won't help if logstash can't keep up in general. >>>> >>>> The original problem went away? >>>> >>>> Rainer >>>> >>>> Sent from phone, thus brief. >>>> Am 21.05.2014 12:18 schrieb "Muhammad Asif" <[email protected]>: >>>> >>>> Hi Everyone, >>>>> >>>>> We have configured rsyslog 7.6.3 and logstash on Desktop Machine (CPU: >>>>> core >>>>> i3, RAM: 4GB). We have configured statistics on both rsyslog and >>>>> >>>> logstash. >>> >>>> RELP is being used between both. When we sent 30716 logs in 10 sec from >>>>> LOIC, there is no loss on logstash but rsyslog sends 256 logs chunk to >>>>> logstash and a delay of about 3 sec is noticed between 256 logs chunk. >>>>> Similarly when we sent 68529, again no loss of logs on logstash and >>>>> same >>>>> delay is noticed. Please find the attached pstats files of Desktop. >>>>> >>>>> We have performed the same activity on a Server machine (CPU: Xeon Quad >>>>> core, RAM 32 GB), no delay is noticed in this case. Everything is >>>>> >>>> fine.On >>> >>>> server we are using rsyslogd 8.2. Why delay is not being noticed during >>>>> consecutive 256 logs chunk. >>>>> >>>>> Can you elaborate the reason of delay in Desktop case and can we >>>>> >>>> increase >>> >>>> 256 logs chunk which rsyslog send to logstash. >>>>> >>>>> The server and desktop stats files are attached. >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, May 20, 2014 at 7:23 PM, Rainer Gerhards >>>>> <[email protected]>wrote: >>>>> >>>>> On Tue, May 20, 2014 at 4:10 PM, masoom alam <[email protected]> >>>>>> wrote: >>>>>> >>>>>> are you asking for this? >>>>>>> >>>>>>> >>>>>>> no. When you run impstats, you get statistics records. Without >>>>>> knowing >>>>>> >>>>> your >>>>> >>>>>> config, I do not know how exactly they look and where they are written >>>>>> >>>>> to, >>>>> >>>>>> but they contain the statistics data. I though that you quoted these >>>>>> >>>>> lines >>>>> >>>>>> when you said "n records where received by rsyslog". What I would like >>>>>> >>>>> to >>>>> >>>>>> see is how many records were pushed to the forwarding action. That way >>>>>> >>>>> we >>>>> >>>>>> know how many were actually sent to logstash. >>>>>> >>>>>> If the stats are a bit unclear to you, I suggest reading: >>>>>> >>>>>> http://www.rsyslog.com/doc/impstats.html >>>>>> >>>>>> and - at least as important - the links that go out to the page that >>>>>> describe the actual counters. There is a wealth of information in them >>>>>> >>>>> in >>>>> >>>>>> cases like these. >>>>>> >>>>>> Rainer >>>>>> >>>>>> see attachement >>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> >>>>> myriad >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
