Hi Gary, I started out with logstash in the middle taking the logs from the rsyslog clients. I wanted to test out having all rsyslog setup as currently I'm not doing a lot of parsing but this solution is still in a testing phase.
>From your reply it seems I need to figure out what format kibana is looking for and create a rsyslog template that will write that into elastic before kibana will be able to search and pull the information back ? Thank You, -Jason On 05/29/2013 01:02 PM, Gary Foster wrote: > I am currently indexing 100 million events a day with elastic search through > the rsyslog-based event processing system I've built here. I use kibana as > one of the search front ends and send the output from rsyslog to logstash to > parse them and inject them into ES. > > I log about 2/3 of the events in CEE format (which basically just get routed > straight into elastic search with minimal parsing) and the legacy event > syntax that needs heavy parsing goes through a rather elaborate logstash > filter. > > Logstash works a treat with rsyslog and it's built to output directly to ES > in the format Kibana needs with no tweaking. You might think about adding > logstash int the middle of your route if you need more elaborate parsing. > > To the original poster… One thing you're going to want to avoid if you have > any sort of traffic load is dumping everything into a single index like > you've got. You want to spread out your indexing or you'll eventually give > your elastic search cluster a hernia. > > -- Gary F. > > On May 29, 2013, at 10:52 AM, Todd Mortensen <[email protected]> wrote: > >> I am using the following and I see my data in kibana, I imported the >> logstash template into kibana. >> >> I would like to parse out more fields but it is a start. >> >> template(name="ElasticLogStash" type="string" >> string="{%timestamp:::date-rfc3339,jsonf:@timestamp >> %,%source:::jsonf:@source_host >> %,\"@source\":\"syslog://%fromhost-ip:::json%\",\"@message\":\"%msg:::json%\",\"@fields\":{%syslogfacility-text:::jsonf:facility%,%syslogseverity-text:::jsonf:severity%,%app-name:::jsonf:program%,%procid:::jsonf:processid%}}") >> >> # ES index name >> template(name="ESLSidx" type="string" >> string="logstash-%timereported:1:10:date-rfc3339%") >> >> action(name="Elastic" Template="ElasticLogStash" type="omelasticsearch" >> server="elasticsearch.example.org" >> searchIndex="ESLSidx" >> dynSearchIndex="on" >> bulkmode="on" >> queue.dequeuebatchsize="200" >> queue.type="linkedlist" >> queue.filename="elasticlsq" >> queue.highwatermark="500000" >> queue.lowwatermark="400000" >> queue.discardmark="5000000" >> queue.timeoutenqueue="0" >> queue.maxdiskspace="5g" >> queue.size="2000000" >> queue.saveonshutdown="on" >> action.resumeretrycount="-1") >> >> I do see strange results from pstats, but I have not had time to track >> down why it is reporting so many failed when I do see the data in >> elasticsearch. >> >> >> 2013-05-29T10:45:35.330398-07:00 central.example.org rsyslogd-pstats: >> elasticsearch: connfail=0 submits=6951573 failed=6963852 success=0 >> >> >> >> >> On Wed, May 29, 2013 at 9:55 AM, Jason A. Johnson >> <[email protected]>wrote: >> >>> Hello, >>> >>> I'm currently working on a central logging solution which seems to be >>> working great. Rsyslog forwarding logs to the central logging server >>> which has been upgrade to version 7.2.7 and elasticsearch storing the >>> logs. Searching elasticsearch I can see that logs are being sent/stored. >>> However I would like to get the frontend working which is kibana 3. I'm >>> wondering if anyone has been able to get kibana 3 working with rsyslog >>> or could point me in the direction of what I would need to change on >>> kibnana side to have the logs visible. Logstash setting are the default >>> for kibana so it works out of the box. I have looked at the defaults for >>> kibana and the only difference I can see that needs to be updated is the >>> timestamps defaults index: >>> >>> "index": "[logstash-]YYYY.MM.DD" I have changed that to "system" which >>> is what shows up in elasticsearch when logs are forward from rsyslog to >>> be stored. However no logs are being displayed in kibnana. >>> >>> If someone has any idea what I'm missing would be hopeful. >>> Thank You, >>> Jason >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
