I am currently indexing 100 million events a day with elastic search through the rsyslog-based event processing system I've built here. I use kibana as one of the search front ends and send the output from rsyslog to logstash to parse them and inject them into ES.
I log about 2/3 of the events in CEE format (which basically just get routed straight into elastic search with minimal parsing) and the legacy event syntax that needs heavy parsing goes through a rather elaborate logstash filter. Logstash works a treat with rsyslog and it's built to output directly to ES in the format Kibana needs with no tweaking. You might think about adding logstash int the middle of your route if you need more elaborate parsing. To the original poster… One thing you're going to want to avoid if you have any sort of traffic load is dumping everything into a single index like you've got. You want to spread out your indexing or you'll eventually give your elastic search cluster a hernia. -- Gary F. On May 29, 2013, at 10:52 AM, Todd Mortensen <[email protected]> wrote: > I am using the following and I see my data in kibana, I imported the > logstash template into kibana. > > I would like to parse out more fields but it is a start. > > template(name="ElasticLogStash" type="string" > string="{%timestamp:::date-rfc3339,jsonf:@timestamp > %,%source:::jsonf:@source_host > %,\"@source\":\"syslog://%fromhost-ip:::json%\",\"@message\":\"%msg:::json%\",\"@fields\":{%syslogfacility-text:::jsonf:facility%,%syslogseverity-text:::jsonf:severity%,%app-name:::jsonf:program%,%procid:::jsonf:processid%}}") > > # ES index name > template(name="ESLSidx" type="string" > string="logstash-%timereported:1:10:date-rfc3339%") > > action(name="Elastic" Template="ElasticLogStash" type="omelasticsearch" > server="elasticsearch.example.org" > searchIndex="ESLSidx" > dynSearchIndex="on" > bulkmode="on" > queue.dequeuebatchsize="200" > queue.type="linkedlist" > queue.filename="elasticlsq" > queue.highwatermark="500000" > queue.lowwatermark="400000" > queue.discardmark="5000000" > queue.timeoutenqueue="0" > queue.maxdiskspace="5g" > queue.size="2000000" > queue.saveonshutdown="on" > action.resumeretrycount="-1") > > I do see strange results from pstats, but I have not had time to track > down why it is reporting so many failed when I do see the data in > elasticsearch. > > > 2013-05-29T10:45:35.330398-07:00 central.example.org rsyslogd-pstats: > elasticsearch: connfail=0 submits=6951573 failed=6963852 success=0 > > > > > On Wed, May 29, 2013 at 9:55 AM, Jason A. Johnson <[email protected]>wrote: > >> Hello, >> >> I'm currently working on a central logging solution which seems to be >> working great. Rsyslog forwarding logs to the central logging server >> which has been upgrade to version 7.2.7 and elasticsearch storing the >> logs. Searching elasticsearch I can see that logs are being sent/stored. >> However I would like to get the frontend working which is kibana 3. I'm >> wondering if anyone has been able to get kibana 3 working with rsyslog >> or could point me in the direction of what I would need to change on >> kibnana side to have the logs visible. Logstash setting are the default >> for kibana so it works out of the box. I have looked at the defaults for >> kibana and the only difference I can see that needs to be updated is the >> timestamps defaults index: >> >> "index": "[logstash-]YYYY.MM.DD" I have changed that to "system" which >> is what shows up in elasticsearch when logs are forward from rsyslog to >> be stored. However no logs are being displayed in kibnana. >> >> If someone has any idea what I'm missing would be hopeful. >> Thank You, >> Jason >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
