On Wed, 3 Jun 2015, Muhammad Asif wrote:
@David Lang
We are sending these logs to fluent (Testing fluent to replace logstash) to
parse logs.
not the question I was asking.
the slide deck shows basic logstash doing 4K logs/sec, having it mutate the logs
as well _increases_ performance to 5K/sec, and adding grok parsing improves it
to 8K/sec (admittedly while using significantly more CPU), what causes this.
Is there any good webinar on rsyslog parsing and template. There should be
a series of about 5 webinars on rsyslog. Rsyslog is lacking on youtube.
I don't know of anyone who has recoreded webinars on the topic, but what is it
you want to know about parsing and templates (two rather different topics)?
The template side of things is output and allows you to craft messages in
whatever format you want. This can also be done via string modules in C (for a
significant performance improvement)
for parsing, mmnormalize is the tol, which uses liblognorm. This is under rapid
development right now with a good number of new features that have nto yet made
it into a release.
The core of how it works is that you create a set of rules to match your logs
and extract values into variables and then rsyslog compiles all these rules into
a parse tree that allows it to do this work extremely fast.
Rsyslog stores this parsed data as JSON and the templates can include parts or
all of this JSON data in outputs.
Rsyslog comes with many input and output moduels written in C, but it also
supports modules written in any scripting language.
so, with this background, where do you want to start?
David Lang
*.* @127.0.0.1:5120;msgonly
On Wed, Jun 3, 2015 at 3:58 AM, David Lang <[email protected]> wrote:
On Tue, 2 Jun 2015, Otis Gospodnetic wrote:
To answer your question about rsyslog in place of Logstash. Short answer:
yes.
See
http://blog.sematext.com/2015/05/18/tuning-elasticsearch-indexing-pipeline-for-logs/
for some info about rsyslog and Logstash comparison when it comes to
performance and footprint.
a question about this link. Why is it showing logstash performance getting
better as it starts doing more complicated things?
David Lang
This may also be of interest to you:
http://blog.sematext.com/2015/04/13/monitoring-rsyslogs-performance-with-impstats-and-elasticsearch/
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Jun 2, 2015 at 8:33 AM, Muhammad Asif <[email protected]>
wrote:
Is there any default queue mechanism in rsyslog. I did not configure any
main and action queue but when i checked imstat statistics. It shows the
following output.
Tue Jun 2 17:30:17 2015: main Q: size=0 enqueued=60933 full=0
discarded.full=0 discarded.nf=0 maxqsize=409
Tue Jun 2 17:31:17 2015: imuxsock: submitted=4 ratelimit.discarded=0
ratelimit.numratelimiters=3
Tue Jun 2 17:31:17 2015: action 1: processed=60933 failed=0
Tue Jun 2 17:31:17 2015: action 2: processed=0 failed=0
Tue Jun 2 17:31:17 2015: action 3: processed=20 failed=0
Tue Jun 2 17:31:17 2015: action 4: processed=60913 failed=0
Tue Jun 2 17:31:17 2015: action 5: processed=0 failed=0
Tue Jun 2 17:31:17 2015: action 6: processed=0 failed=0
Tue Jun 2 17:31:17 2015: action 7: processed=0 failed=0
Tue Jun 2 17:31:17 2015: action 8: processed=0 failed=0
Tue Jun 2 17:31:17 2015: action 9: processed=0 failed=0
Tue Jun 2 17:31:17 2015: action 10: processed=0 failed=0
Tue Jun 2 17:31:17 2015: action 11: processed=0 failed=0
*Tue Jun 2 17:31:17 2015: action 12: processed=60932 failed=60932*
Tue Jun 2 17:31:17 2015: action 13: processed=60933 failed=0
Tue Jun 2 17:31:17 2015: imudp(*:514): submitted=0
Tue Jun 2 17:31:17 2015: imudp(*:514): submitted=0
Tue Jun 2 17:31:17 2015: imtcp(514): submitted=0
Tue Jun 2 17:31:17 2015: main Q: size=0 enqueued=60933 full=0
discarded.full=0 discarded.nf=0 maxqsize=409
Why it is showing as many failed messages as many processed.
Can we use rsyslog in place of logstash.
Thanks
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
