are you sure that you didn't have logstash listening on port 514 as well as
rsyslog? having two programs listening can cause them to fight over receiving
messages and each one only getting a few of them.
I'm understanding you to be saying that in each case tcpdump sees all the
messages, but with both rsyslog and logstash running, rsyslog is only seeing
some of them.
David Lang
On Tue, 27 May 2014, masoom
alam wrote:
Date: Tue, 27 May 2014 06:11:44 +0500
From: masoom alam <[email protected]>
To: David Lang <[email protected]>
Cc: [email protected], rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Fwd: Re: Log Drop Issue
Hi every one
We have performed our testing in office time and off time as well.
On sending Server
---------------------------------------------------------------------------
./tcpflood -t 172.20.16.8 -p 514 -m 500000 -M ‘We are here for stress
testing of rsysog ' -T udp –W 10
tcpdump –i eth0 –n -O –w src.pcap
On receiving Rsyslog
---------------------------------------------------------------------------------------
Tcpdump –i eth0 –n –O –w dest.pcap
During office time, we use delay (-W 10). We send 500000 msgs in 12.4 sec,
and receive equal logs on Rsyslog and tcpdump (on both sides) without any
loss.
But after office time, we successfully send the same amount without delay
(-W 10) and receive exact on Rsyslog and tcpdump. But when we start
Logstash on same (rsyslog) machine for parsing, and send same 500000 msgs
in 12.4 sec, Rsyslog receive only 100300 messages, tcpdump on sending
machine was exact but on receiving machine there was significant dropping
of logs. We perform the same test with 100000 msgs in 2.4 sec, this time
Rsyslog count was exact, tcpdump on sending machine was also exact but on
receiving machine 100 messages were dropped.
Any clue?
Thanks David,
All the experiments Pstats will be shared on Monday for further guidance.
Thanks again.
On Fri, May 23, 2014 at 10:58 PM, David Lang <[email protected]> wrote:
tcpdump will drop packets when it can't keep up. The best way to use
tcpdump is to not have it output to the screen, but write to a file
(parsing the packets to format them for the screen is _very_ expensive)
the very low performance of tcpdump on the receiver is probably caused by
it trying to do DNS lookup for it's output, you can reduce this by adding
-n, but as noted above, you are far better off writing the output to a file
(-o ) and then reading it with another tcpdump process and having that
second process do all the output formatting.
you can have filters on the first process, and you can configure the
smaplength (the amount of data to capture for each packet) to reduce the
volume of data to be written to disk
I've also had good success using tcpreplay to generate traffic, I capture
data UDP data with tcpdump, and then replay it with tcpreplay, which allows
me to tune how fast I'm sending data.
since you have a managed switch, and are using UDP, if you can make the
network quiet otherwise the switch counters will tell you how many packets
it's seeing on each port (similarly, the ifconfig data on the sender will
tell you how many packets are being sent)
It would be extremely useful to see the pstats data from rsyslog.
David Lang
On Fri, 23 May 2014, masoom alam wrote:
Hi Rainer and David n every one,
While doing troubleshooting, we have found the following:
1. Firstly, LOIC is extremely unpredictable. We ran many times overnight
experiments and when we come in the morning, the count on the LOIC
machine
to Rsyslog impstats significantly varies. We noted that LOIC is not
built
for log stress testing....
2. We then found tcpflood - native Rsyslog utility and tested our setup
with it. Ironically following are the results (see the figure). We also
setup tcpdump both at the tcpflood sending system and receiving Rsyslog
system. You will see that tcpdump even at source is not showing the
correct
number of packet sent. We have sent 0.1 million packets of logs. We
noted
that TCPdump at source is showing less than what was sent by tcpflood.
Plus
the destination tcpdump shows even less, but the comedy point is that
Rsyslog impstats are showing higher packets received, even higher than
source. tcpdump :).
3. However, the good thing is that the no. of packets received by
Rsyslog and stored by MongoDB after parsing by Logtash are exactly
equal.
4. We are trying to understand why packets disappear within the network.
Our network is Gigabit ethernet, with manageable D-Linux switch.
[image: Inline image 1]
Please comment on the overall experiment and tell us where are we doing
wrong....
On Wed, May 21, 2014 at 3:39 PM, Rainer Gerhards
<[email protected]>wrote:
As a side note you should temporarily remove logstash from the picture.
Ship to rsyslog. If the problem persists, we can much better
troubleshoot. If it does not persist, you know you need to talk to the
logstash folks.
Rainer
Sent from phone, thus brief.
Am 21.05.2014 12:25 schrieb "Rainer Gerhards" <[email protected]
:
This sounds like logstash isn't fast enough in some cases and the relp
window fills up. You can try to increase it:
http://www.rsyslog.com/doc/omrelp.html
But that won't help if logstash can't keep up in general.
The original problem went away?
Rainer
Sent from phone, thus brief.
Am 21.05.2014 12:18 schrieb "Muhammad Asif" <[email protected]>:
Hi Everyone,
We have configured rsyslog 7.6.3 and logstash on Desktop Machine (CPU:
core
i3, RAM: 4GB). We have configured statistics on both rsyslog and
logstash.
RELP is being used between both. When we sent 30716 logs in 10 sec from
LOIC, there is no loss on logstash but rsyslog sends 256 logs chunk to
logstash and a delay of about 3 sec is noticed between 256 logs chunk.
Similarly when we sent 68529, again no loss of logs on logstash and
same
delay is noticed. Please find the attached pstats files of Desktop.
We have performed the same activity on a Server machine (CPU: Xeon Quad
core, RAM 32 GB), no delay is noticed in this case. Everything is
fine.On
server we are using rsyslogd 8.2. Why delay is not being noticed during
consecutive 256 logs chunk.
Can you elaborate the reason of delay in Desktop case and can we
increase
256 logs chunk which rsyslog send to logstash.
The server and desktop stats files are attached.
Thanks
On Tue, May 20, 2014 at 7:23 PM, Rainer Gerhards
<[email protected]>wrote:
On Tue, May 20, 2014 at 4:10 PM, masoom alam <[email protected]>
wrote:
are you asking for this?
no. When you run impstats, you get statistics records. Without
knowing
your
config, I do not know how exactly they look and where they are written
to,
but they contain the statistics data. I though that you quoted these
lines
when you said "n records where received by rsyslog". What I would like
to
see is how many records were pushed to the forwarding action. That way
we
know how many were actually sent to logstash.
If the stats are a bit unclear to you, I suggest reading:
http://www.rsyslog.com/doc/impstats.html
and - at least as important - the links that go out to the page that
describe the actual counters. There is a wealth of information in them
in
cases like these.
Rainer
see attachement
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
