I think that for what he's needing, he needs info on the message modification
interface (I am needing to dig that up as well)
David Lang
On Tue, 20 May 2014, Rainer Gerhards wrote:
That's the shortest intro:
http://www.slideshare.net/rainergerhards1/writing-rsyslog-p
I points you to more resources.
Rainer
On Tue, May 20, 2014 at 3:31 AM, masoom alam <[email protected]> wrote:
Hi David and Rainer,
I hope you both are doing fine. Could you please point me to the link where
I can find how to write a Ruby plugin in Rsyslog. I remember you mentioned
in one of your email that is quite possible. Did not find in my email box
but later found here :D
http://lists.adiscon.net/pipermail/rsyslog/2014-April/037237.html
I just want to give it a try that is to include all the Ruby filters within
the Rsyslog. Please guide.
Thanks.
On Mon, May 19, 2014 at 7:33 PM, David Lang <[email protected]> wrote:
the pstats output should also show the deliveries to logstash (at least
if
you are using anything close to a current version of rsyslog)
This isn't the logstash support mailing list :-) but we've been hearing a
lot of people talking about how logstash will drop logs when it gets
behind, so that could be the problem. If you use RELP instead of just
TCP,
that will guarantee all the logs get to logstash, even if the TCP
connection gets cut and has to be reestablished. But if logstash is
dropping them after that, we won't be able to help much.
it's frequently useful in situations like this to log exactly what you
are
sending to the remote system to a local file.
Why is it that you are going through logstash to get to mongodb instead
of
having rsyslog deliver directly to mongodb?
You mention that you have about 15 filters in logstash, have you looked
at
putting those filters in rsyslog instead?
David Lang
On Mon, 19 May 2014, masoom alam wrote:
We have verified that logtash is the main culprit in dropping the logs.
we are generating logs from LOIC with 800 logs per second and sending it
to
rsyslog.
rsyslog agent is receiving all the logs even it is showing some count on
the upper side , i mean if i sent 3000 logs from LOIC it always shows me
that it receives 3020 logs , don't know about these 20.(According to my
imstat file)
Now again rsyslog passes these logs to logstash which at the moment has
15
filters. The metric to count the logs recived at logstash is
http://logstash.net/docs/1.4.1/filters/metrics#flush_interval , but it
always showed me the less number as i received in rsyslog. At certain
level
for example ,,3000 logs/5 sec, count of rsyslog and logstash is same but
after 3500 , 4000 there is a difference of 30 -40 logs between rsyslog
and
logstash , our connection between rsyslog to logstash is TCP , so there
should be no reason for this difference , either logstash is unable to
parse all the msgs or rsyslog is sending less msgs to logstash.
Any clue?
On Sunday, May 18, 2014, David Lang <[email protected]> wrote:
If you are sending the right contents in the packet, LOIC should be
just
fine.
Now, you haven't said what version of rsyslog you are using, your
configuration, or even talked about what speed network you are on (let
alone system specs), so figuring out what's wrong is not possible yet
:-)
That said, we've had people getting around 400,000 packets/sec through
rsyslog, so you are probably not hitting any fundamental limit at 10,000
packets/sec. But there are a lot of things to look at to figure this
out.
start by configuring impstats (set it to dump stats every 10 sec for
now
at these traffic levels) so that we can see the what's happening inside
rsyslog (to see if the problem is getting the logs in, or out of
rsyslog)
version info (rsyslog, distro, kernel)
when you say you put things on a switch, is this a 100Mb switch, Gb
switch, managed (so you can get stats from the switch)????
what's your rsyslog config?
what do you use for name resolution (/etc/hosts, local DNS, nearby DNS,
ISP DNS, LDAP, ????)
UDP isn't necessarily faster than TCP, but it's a whole lot easier to
setup a UDP test, so let's stick with that for now. There's nothing in
the
syslog protocol to add reliability to UDP
David Lang
On Sun, 18 May 2014, masoom alam wrote:
Dear All,
I hope every one is doing fine. We were doing stress testing of
Rsyslog
and
found few problems (in our setup and not in Rsyslog :)) that I want to
discuss at this forum.
1. We were using LOIC (LOIC is DDOS attack tool - your anti virus
will
delete it :) - disclaimer: handle it with care) for generation of
UDP
packets. We created a customized log message. The speed of Packets
sent by
LOIC is very very high, that is some thing 20,000 packets in 2 sec.
for
example. Every thing is fine if we use point to point connection
between
Rsyslog machine and MangoDB machine. Point to point means connection
via
cross cable and not through a switch. We calculated the no. of
packets
sent
by LOIC and no. of packets received by Rsyslog and then written by
MongoDB
after parsing by Logtash is fairly equal.
2. However, if we connect both MongoDB and Rsyslog through a switch
LAN,
there is a packet drop at the Rsyslog end, some what between 300-500
packets depending on the speed of LOIC - thus 300-500 lesser logs
are
written to MongoDB.
What we concluded and I want your expert opinion on this:
1. It seems LOIC is the not the right choice for traffic generation
for
Rsyslog - that is stress testing of Rsyslog. It sends packets via
UDP
514,
but essentially it does not follow Syslog Protocol. Now, we are
trying
to
understand: Is there some sort of reliability achieved in Syslog
protocol
even if packets are sent on UDP 514? BTW, i am well aware that UDP is
for
faster communication but at the expense of reliability. Why we are
saying
that there is a problem at the LOIC - that is traffic generation end
-
because when we selected to send traffic via TCP on LOIC, due to
speed
it
combines log packets and Rsyslog reports an error in its log. The net
effect of this operation is that Rsyslog combines arbitrary no. of
logs
together and then give to Logtash, which does not understand and
leave
it
un parsed.
2. What options do we have, either we use a python library to
generate
traffic, write it to /var/log/messages and ask the Rsyslog to send
that
traffic to another Rsyslog?. Does using this way guarantees that
there
will
be no drop of Log messages even in UDP?
3. But what I am interested in understanding what is the reliability
mechanism provided by Rsyslog in general in the case of UDP. After
all
each
n every log is a very important piece of information and can destroy
the
reputation of an organization.
4. Even if the some reliability can be achieved at the Rsyslog end,
how
can we avoid - up to maximum extent - the possibility of log drop
between
Rsyslog to Logtash. Logtash is a very small program than Rsyslog.
Rsyslog
in our setup is used only for Buffering - thus, what parameters in
the
.conf file of Rsyslog should be changed to achieve this reliability.
Please
note that, our Rsyslog and Logtash are setup at the same system - so
no
network latency at this end.
5. In all this setup, the performance of LogAnalyzer was very
pathetic
in filtering and running other queries. So we choose to write a
simple
PHP
web page for displaying logs and it is running much much faster than
LogAnalyzer.
6. Are we on the right path for checking reliability and stress
testing?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
