I just remembered the other piece I had to change, I used this as a template for kibana3
https://gist.github.com/anonymous/36ae0b37025c31dbf344 Because the rsyslog date format uses "-" as a seperator and logstash uses ".", I had to change the following lines in the kibana3 template. "index": "\"logstash-\"yyyy.mm.dd", to "index": "\"logstash-\"yyyy-mm-dd", Sorry if this is getting off topic for an rsyslog list. On Wed, May 29, 2013 at 2:20 PM, Jason A. Johnson <[email protected]>wrote: > Hi Gary, > > I started out with logstash in the middle taking the logs from the > rsyslog clients. I wanted to test out having all rsyslog setup as > currently I'm not doing a lot of parsing but this solution is still in a > testing phase. > > From your reply it seems I need to figure out what format kibana is > looking for and create a rsyslog template that will write that into > elastic before kibana will be able to search and pull the information > back ? > > Thank You, > -Jason > > On 05/29/2013 01:02 PM, Gary Foster wrote: > > I am currently indexing 100 million events a day with elastic search > through the rsyslog-based event processing system I've built here. I use > kibana as one of the search front ends and send the output from rsyslog to > logstash to parse them and inject them into ES. > > > > I log about 2/3 of the events in CEE format (which basically just get > routed straight into elastic search with minimal parsing) and the legacy > event syntax that needs heavy parsing goes through a rather elaborate > logstash filter. > > > > Logstash works a treat with rsyslog and it's built to output directly to > ES in the format Kibana needs with no tweaking. You might think about > adding logstash int the middle of your route if you need more elaborate > parsing. > > > > To the original poster… One thing you're going to want to avoid if you > have any sort of traffic load is dumping everything into a single index > like you've got. You want to spread out your indexing or you'll eventually > give your elastic search cluster a hernia. > > > > -- Gary F. > > > > On May 29, 2013, at 10:52 AM, Todd Mortensen <[email protected]> > wrote: > > > >> I am using the following and I see my data in kibana, I imported the > >> logstash template into kibana. > >> > >> I would like to parse out more fields but it is a start. > >> > >> template(name="ElasticLogStash" type="string" > >> string="{%timestamp:::date-rfc3339,jsonf:@timestamp > >> %,%source:::jsonf:@source_host > >> > %,\"@source\":\"syslog://%fromhost-ip:::json%\",\"@message\":\"%msg:::json%\",\"@fields\":{%syslogfacility-text:::jsonf:facility%,%syslogseverity-text:::jsonf:severity%,%app-name:::jsonf:program%,%procid:::jsonf:processid%}}") > >> > >> # ES index name > >> template(name="ESLSidx" type="string" > >> string="logstash-%timereported:1:10:date-rfc3339%") > >> > >> action(name="Elastic" Template="ElasticLogStash" type="omelasticsearch" > >> server="elasticsearch.example.org" > >> searchIndex="ESLSidx" > >> dynSearchIndex="on" > >> bulkmode="on" > >> queue.dequeuebatchsize="200" > >> queue.type="linkedlist" > >> queue.filename="elasticlsq" > >> queue.highwatermark="500000" > >> queue.lowwatermark="400000" > >> queue.discardmark="5000000" > >> queue.timeoutenqueue="0" > >> queue.maxdiskspace="5g" > >> queue.size="2000000" > >> queue.saveonshutdown="on" > >> action.resumeretrycount="-1") > >> > >> I do see strange results from pstats, but I have not had time to track > >> down why it is reporting so many failed when I do see the data in > >> elasticsearch. > >> > >> > >> 2013-05-29T10:45:35.330398-07:00 central.example.org rsyslogd-pstats: > >> elasticsearch: connfail=0 submits=6951573 failed=6963852 success=0 > >> > >> > >> > >> > >> On Wed, May 29, 2013 at 9:55 AM, Jason A. Johnson <[email protected] > >wrote: > >> > >>> Hello, > >>> > >>> I'm currently working on a central logging solution which seems to be > >>> working great. Rsyslog forwarding logs to the central logging server > >>> which has been upgrade to version 7.2.7 and elasticsearch storing the > >>> logs. Searching elasticsearch I can see that logs are being > sent/stored. > >>> However I would like to get the frontend working which is kibana 3. I'm > >>> wondering if anyone has been able to get kibana 3 working with rsyslog > >>> or could point me in the direction of what I would need to change on > >>> kibnana side to have the logs visible. Logstash setting are the default > >>> for kibana so it works out of the box. I have looked at the defaults > for > >>> kibana and the only difference I can see that needs to be updated is > the > >>> timestamps defaults index: > >>> > >>> "index": "[logstash-]YYYY.MM.DD" I have changed that to "system" which > >>> is what shows up in elasticsearch when logs are forward from rsyslog to > >>> be stored. However no logs are being displayed in kibnana. > >>> > >>> If someone has any idea what I'm missing would be hopeful. > >>> Thank You, > >>> Jason > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
