Dear All, I hope every one is doing fine. We were doing stress testing of Rsyslog and found few problems (in our setup and not in Rsyslog :)) that I want to discuss at this forum.
1. We were using LOIC (LOIC is DDOS attack tool - your anti virus will delete it :) - disclaimer: handle it with care) for generation of UDP packets. We created a customized log message. The speed of Packets sent by LOIC is very very high, that is some thing 20,000 packets in 2 sec. for example. Every thing is fine if we use point to point connection between Rsyslog machine and MangoDB machine. Point to point means connection via cross cable and not through a switch. We calculated the no. of packets sent by LOIC and no. of packets received by Rsyslog and then written by MongoDB after parsing by Logtash is fairly equal. 2. However, if we connect both MongoDB and Rsyslog through a switch LAN, there is a packet drop at the Rsyslog end, some what between 300-500 packets depending on the speed of LOIC - thus 300-500 lesser logs are written to MongoDB. What we concluded and I want your expert opinion on this: 1. It seems LOIC is the not the right choice for traffic generation for Rsyslog - that is stress testing of Rsyslog. It sends packets via UDP 514, but essentially it does not follow Syslog Protocol. Now, we are trying to understand: Is there some sort of reliability achieved in Syslog protocol even if packets are sent on UDP 514? BTW, i am well aware that UDP is for faster communication but at the expense of reliability. Why we are saying that there is a problem at the LOIC - that is traffic generation end - because when we selected to send traffic via TCP on LOIC, due to speed it combines log packets and Rsyslog reports an error in its log. The net effect of this operation is that Rsyslog combines arbitrary no. of logs together and then give to Logtash, which does not understand and leave it un parsed. 2. What options do we have, either we use a python library to generate traffic, write it to /var/log/messages and ask the Rsyslog to send that traffic to another Rsyslog?. Does using this way guarantees that there will be no drop of Log messages even in UDP? 3. But what I am interested in understanding what is the reliability mechanism provided by Rsyslog in general in the case of UDP. After all each n every log is a very important piece of information and can destroy the reputation of an organization. 4. Even if the some reliability can be achieved at the Rsyslog end, how can we avoid - up to maximum extent - the possibility of log drop between Rsyslog to Logtash. Logtash is a very small program than Rsyslog. Rsyslog in our setup is used only for Buffering - thus, what parameters in the .conf file of Rsyslog should be changed to achieve this reliability. Please note that, our Rsyslog and Logtash are setup at the same system - so no network latency at this end. 5. In all this setup, the performance of LogAnalyzer was very pathetic in filtering and running other queries. So we choose to write a simple PHP web page for displaying logs and it is running much much faster than LogAnalyzer. 6. Are we on the right path for checking reliability and stress testing? _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
