Hehe, thanks for the kind introduction, Rainer! Now I have to step up to it :p
My answers inline. 2013/10/29 masoom alam <[email protected]> > [...] > > *Log Collection through RSyslog or Logstash* > > I want to collect log and then store it in elasticsearch, I have already > have did this btw, I can collect log through logstash, store it in > elasticsearch and then display it through Kibana, my Question is why use > Logstash? Can't RSyslog provide all these functionality of log collection? > The only reason I see is that Logstash have filters, RSyslog can store logs > in elasticsearch through OMELASTICSEARCH, why using logstash is so highly > recommended? > You seem to know the answers already. Or at least know the same stuff as me. So I'll just phrase them differently and write the details I'm aware of, and hopefully it will help. Logstash is very popular now. I guess that's what makes it "highly recommended". I assume the reasons for being popular are: - it's been around for a while, like James mentioned Kind of repeating what Michael said here: - it integrates nicely with Kibana. Actually, it's the other way around :) - Elasticsearch (the company) now takes care of Logstash, ES and Kibana, so they all work together nicely - it has lots of plugins, and it's easy to write plugins. I'm a Ruby newbie (but I make rhymes :p) and I could contribute a Solr output plugin. In case you're interested in how output plugins may look like and/or you prefer Solr over ES, have a look: https://github.com/logstash/logstash/pull/675 So you'd probably use Logstash if you need some functionality that rsyslog doesn't have. For example, if you need to parse unstructured data via user-friendly regular expressions (see grok<http://logstash.net/docs/1.2.2/filters/grok>). Or, if you want to pipe metrics to something like Graphite or Ganglia or whatever exotic plugins rsyslog doesn't have yet. On the other hand, Logstash does take A LOT more resources than rsyslog. We're actually using both in production. James mentioned a factor of 10. It all depends on how you configure them but, in my experience, the gap is even wider than that. So you'd probably use rsyslog+omelasticsearch for performance and for keeping your setup simple: if you already have rsyslog in the design, it makes little sense to add extra moving pieces. Also, please note that rsyslog can also do a bunch of stuff that is typically invoked by Logstash users as a reason for moving away from syslog. There's some truth to those claims, but rsyslog evolved in time: - structured logging. If you can control what is logged from your app, you can log in JSON using standard syslog libraries. Here's a blog post I wrote a while ago that show how you can do that all the way to Elasticsearch: http://logstash.net/docs/1.2.2/filters/grok - parse unstructured logs and make them structured. Some of that can be done via mmnormalize <http://www.rsyslog.com/doc/mmnormalize.html>, although it looks like it needs some contributions and/or sponsoring<http://www.rsyslog.com/sponsors/>to step up to its potential in terms of features and performance - sending data to various destinations. For low traffic, like sending alerts to Nagios or something like that, you can always format logs the way you want using templates<http://www.rsyslog.com/doc/rsyslog_conf_templates.html>and pipe them to external programs via omprog <http://www.rsyslog.com/doc/omprog.html>. If you need more performance, you can always contribute/sponsor a new plugin. But have a look first: there are some plugins that aren't documented yet (I just "discovered the wheel" a few days ago when there was a mention of omhiredis). Finally, two links: - a blog I wrote about using rsyslog+omelasticsearch+ES+Kibana: http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/ - a talk I gave at Monitorama EU, where I talk a lot about rsyslog and syslog in general, Logstash, and tuning ES for logs: http://blog.sematext.com/2013/10/03/video-presentation-on-centralizing-logs/ > > *Kibana vs Elasticsearch UI* > > This question does not concern this mailing list but I will ask it anyway > may be in near future someone like me will come looking to answer the same > questions :p which one should I prefer? Kibana or the UIs for > elasticsearch? My main concern is to minimize rely my dependance on > external tools as much as possible without weakening my Log-server. > > I don't really get what you mean by "UIs for Elasticsearch". Kibana is the de-facto "UI for Elasticsearch", but you can always use some other UI or build your own. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
