tcpdump will drop packets when it can't keep up. The best way to use tcpdump is
to not have it output to the screen, but write to a file (parsing the packets to
format them for the screen is _very_ expensive)
the very low performance of tcpdump on the receiver is probably caused by it
trying to do DNS lookup for it's output, you can reduce this by adding -n, but
as noted above, you are far better off writing the output to a file (-o ) and
then reading it with another tcpdump process and having that second process do
all the output formatting.
you can have filters on the first process, and you can configure the smaplength
(the amount of data to capture for each packet) to reduce the volume of data to
be written to disk
I've also had good success using tcpreplay to generate traffic, I capture data
UDP data with tcpdump, and then replay it with tcpreplay, which allows me to
tune how fast I'm sending data.
since you have a managed switch, and are using UDP, if you can make the network
quiet otherwise the switch counters will tell you how many packets it's seeing
on each port (similarly, the ifconfig data on the sender will tell you how many
packets are being sent)
It would be extremely useful to see the pstats data from rsyslog.
David Lang
On Fri, 23 May 2014, masoom alam wrote:
Hi Rainer and David n every one,
While doing troubleshooting, we have found the following:
1. Firstly, LOIC is extremely unpredictable. We ran many times overnight
experiments and when we come in the morning, the count on the LOIC machine
to Rsyslog impstats significantly varies. We noted that LOIC is not built
for log stress testing....
2. We then found tcpflood - native Rsyslog utility and tested our setup
with it. Ironically following are the results (see the figure). We also
setup tcpdump both at the tcpflood sending system and receiving Rsyslog
system. You will see that tcpdump even at source is not showing the correct
number of packet sent. We have sent 0.1 million packets of logs. We noted
that TCPdump at source is showing less than what was sent by tcpflood. Plus
the destination tcpdump shows even less, but the comedy point is that
Rsyslog impstats are showing higher packets received, even higher than
source. tcpdump :).
3. However, the good thing is that the no. of packets received by
Rsyslog and stored by MongoDB after parsing by Logtash are exactly equal.
4. We are trying to understand why packets disappear within the network.
Our network is Gigabit ethernet, with manageable D-Linux switch.
[image: Inline image 1]
Please comment on the overall experiment and tell us where are we doing
wrong....
On Wed, May 21, 2014 at 3:39 PM, Rainer Gerhards
<[email protected]>wrote:
As a side note you should temporarily remove logstash from the picture.
Ship to rsyslog. If the problem persists, we can much better
troubleshoot. If it does not persist, you know you need to talk to the
logstash folks.
Rainer
Sent from phone, thus brief.
Am 21.05.2014 12:25 schrieb "Rainer Gerhards" <[email protected]>:
This sounds like logstash isn't fast enough in some cases and the relp
window fills up. You can try to increase it:
http://www.rsyslog.com/doc/omrelp.html
But that won't help if logstash can't keep up in general.
The original problem went away?
Rainer
Sent from phone, thus brief.
Am 21.05.2014 12:18 schrieb "Muhammad Asif" <[email protected]>:
Hi Everyone,
We have configured rsyslog 7.6.3 and logstash on Desktop Machine (CPU:
core
i3, RAM: 4GB). We have configured statistics on both rsyslog and
logstash.
RELP is being used between both. When we sent 30716 logs in 10 sec from
LOIC, there is no loss on logstash but rsyslog sends 256 logs chunk to
logstash and a delay of about 3 sec is noticed between 256 logs chunk.
Similarly when we sent 68529, again no loss of logs on logstash and same
delay is noticed. Please find the attached pstats files of Desktop.
We have performed the same activity on a Server machine (CPU: Xeon Quad
core, RAM 32 GB), no delay is noticed in this case. Everything is
fine.On
server we are using rsyslogd 8.2. Why delay is not being noticed during
consecutive 256 logs chunk.
Can you elaborate the reason of delay in Desktop case and can we
increase
256 logs chunk which rsyslog send to logstash.
The server and desktop stats files are attached.
Thanks
On Tue, May 20, 2014 at 7:23 PM, Rainer Gerhards
<[email protected]>wrote:
On Tue, May 20, 2014 at 4:10 PM, masoom alam <[email protected]>
wrote:
are you asking for this?
no. When you run impstats, you get statistics records. Without knowing
your
config, I do not know how exactly they look and where they are written
to,
but they contain the statistics data. I though that you quoted these
lines
when you said "n records where received by rsyslog". What I would like
to
see is how many records were pushed to the forwarding action. That way
we
know how many were actually sent to logstash.
If the stats are a bit unclear to you, I suggest reading:
http://www.rsyslog.com/doc/impstats.html
and - at least as important - the links that go out to the page that
describe the actual counters. There is a wealth of information in them
in
cases like these.
Rainer
see attachement
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
