Hi every one We have performed our testing in office time and off time as well.
On sending Server --------------------------------------------------------------------------- ./tcpflood -t 172.20.16.8 -p 514 -m 500000 -M ‘We are here for stress testing of rsysog ' -T udp –W 10 tcpdump –i eth0 –n -O –w src.pcap On receiving Rsyslog --------------------------------------------------------------------------------------- Tcpdump –i eth0 –n –O –w dest.pcap During office time, we use delay (-W 10). We send 500000 msgs in 12.4 sec, and receive equal logs on Rsyslog and tcpdump (on both sides) without any loss. But after office time, we successfully send the same amount without delay (-W 10) and receive exact on Rsyslog and tcpdump. But when we start Logstash on same (rsyslog) machine for parsing, and send same 500000 msgs in 12.4 sec, Rsyslog receive only 100300 messages, tcpdump on sending machine was exact but on receiving machine there was significant dropping of logs. We perform the same test with 100000 msgs in 2.4 sec, this time Rsyslog count was exact, tcpdump on sending machine was also exact but on receiving machine 100 messages were dropped. Any clue? Thanks David, All the experiments Pstats will be shared on Monday for further guidance. Thanks again. On Fri, May 23, 2014 at 10:58 PM, David Lang <[email protected]> wrote: > tcpdump will drop packets when it can't keep up. The best way to use > tcpdump is to not have it output to the screen, but write to a file > (parsing the packets to format them for the screen is _very_ expensive) > > the very low performance of tcpdump on the receiver is probably caused by > it trying to do DNS lookup for it's output, you can reduce this by adding > -n, but as noted above, you are far better off writing the output to a file > (-o ) and then reading it with another tcpdump process and having that > second process do all the output formatting. > > you can have filters on the first process, and you can configure the > smaplength (the amount of data to capture for each packet) to reduce the > volume of data to be written to disk > > I've also had good success using tcpreplay to generate traffic, I capture > data UDP data with tcpdump, and then replay it with tcpreplay, which allows > me to tune how fast I'm sending data. > > since you have a managed switch, and are using UDP, if you can make the > network quiet otherwise the switch counters will tell you how many packets > it's seeing on each port (similarly, the ifconfig data on the sender will > tell you how many packets are being sent) > > It would be extremely useful to see the pstats data from rsyslog. > > David Lang > > > On Fri, 23 May 2014, masoom alam wrote: > > Hi Rainer and David n every one, >> >> While doing troubleshooting, we have found the following: >> >> >> 1. Firstly, LOIC is extremely unpredictable. We ran many times overnight >> >> experiments and when we come in the morning, the count on the LOIC >> machine >> to Rsyslog impstats significantly varies. We noted that LOIC is not >> built >> for log stress testing.... >> 2. We then found tcpflood - native Rsyslog utility and tested our setup >> >> with it. Ironically following are the results (see the figure). We also >> setup tcpdump both at the tcpflood sending system and receiving Rsyslog >> system. You will see that tcpdump even at source is not showing the >> correct >> number of packet sent. We have sent 0.1 million packets of logs. We >> noted >> that TCPdump at source is showing less than what was sent by tcpflood. >> Plus >> the destination tcpdump shows even less, but the comedy point is that >> Rsyslog impstats are showing higher packets received, even higher than >> source. tcpdump :). >> 3. However, the good thing is that the no. of packets received by >> >> Rsyslog and stored by MongoDB after parsing by Logtash are exactly >> equal. >> 4. We are trying to understand why packets disappear within the network. >> >> Our network is Gigabit ethernet, with manageable D-Linux switch. >> >> [image: Inline image 1] >> >> Please comment on the overall experiment and tell us where are we doing >> wrong.... >> >> >> >> >> >> On Wed, May 21, 2014 at 3:39 PM, Rainer Gerhards >> <[email protected]>wrote: >> >> As a side note you should temporarily remove logstash from the picture. >>> Ship to rsyslog. If the problem persists, we can much better >>> troubleshoot. If it does not persist, you know you need to talk to the >>> logstash folks. >>> >>> Rainer >>> >>> Sent from phone, thus brief. >>> Am 21.05.2014 12:25 schrieb "Rainer Gerhards" <[email protected] >>> >: >>> >>> This sounds like logstash isn't fast enough in some cases and the relp >>>> window fills up. You can try to increase it: >>>> >>>> http://www.rsyslog.com/doc/omrelp.html >>>> >>>> But that won't help if logstash can't keep up in general. >>>> >>>> The original problem went away? >>>> >>>> Rainer >>>> >>>> Sent from phone, thus brief. >>>> Am 21.05.2014 12:18 schrieb "Muhammad Asif" <[email protected]>: >>>> >>>> Hi Everyone, >>>>> >>>>> We have configured rsyslog 7.6.3 and logstash on Desktop Machine (CPU: >>>>> core >>>>> i3, RAM: 4GB). We have configured statistics on both rsyslog and >>>>> >>>> logstash. >>> >>>> RELP is being used between both. When we sent 30716 logs in 10 sec from >>>>> LOIC, there is no loss on logstash but rsyslog sends 256 logs chunk to >>>>> logstash and a delay of about 3 sec is noticed between 256 logs chunk. >>>>> Similarly when we sent 68529, again no loss of logs on logstash and >>>>> same >>>>> delay is noticed. Please find the attached pstats files of Desktop. >>>>> >>>>> We have performed the same activity on a Server machine (CPU: Xeon Quad >>>>> core, RAM 32 GB), no delay is noticed in this case. Everything is >>>>> >>>> fine.On >>> >>>> server we are using rsyslogd 8.2. Why delay is not being noticed during >>>>> consecutive 256 logs chunk. >>>>> >>>>> Can you elaborate the reason of delay in Desktop case and can we >>>>> >>>> increase >>> >>>> 256 logs chunk which rsyslog send to logstash. >>>>> >>>>> The server and desktop stats files are attached. >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, May 20, 2014 at 7:23 PM, Rainer Gerhards >>>>> <[email protected]>wrote: >>>>> >>>>> On Tue, May 20, 2014 at 4:10 PM, masoom alam <[email protected]> >>>>>> wrote: >>>>>> >>>>>> are you asking for this? >>>>>>> >>>>>>> >>>>>>> no. When you run impstats, you get statistics records. Without >>>>>> knowing >>>>>> >>>>> your >>>>> >>>>>> config, I do not know how exactly they look and where they are written >>>>>> >>>>> to, >>>>> >>>>>> but they contain the statistics data. I though that you quoted these >>>>>> >>>>> lines >>>>> >>>>>> when you said "n records where received by rsyslog". What I would like >>>>>> >>>>> to >>>>> >>>>>> see is how many records were pushed to the forwarding action. That way >>>>>> >>>>> we >>>>> >>>>>> know how many were actually sent to logstash. >>>>>> >>>>>> If the stats are a bit unclear to you, I suggest reading: >>>>>> >>>>>> http://www.rsyslog.com/doc/impstats.html >>>>>> >>>>>> and - at least as important - the links that go out to the page that >>>>>> describe the actual counters. There is a wealth of information in them >>>>>> >>>>> in >>>>> >>>>>> cases like these. >>>>>> >>>>>> Rainer >>>>>> >>>>>> see attachement >>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> >>>>> myriad >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
