There is a logstash plugin named logstash-filter-de_dot. And because logstash has a force queue to send a larger bulk_size to ES which is better in ES recommandition, I suggest to use rsyslog-omelasticsearch only for not so high load -- but need to watch it as quickly-- use case.
2016-11-18 18:53 GMT+08:00 Rainer Gerhards <[email protected]>: > 2016-11-18 11:51 GMT+01:00 David Lang <[email protected]>: > > On Fri, 18 Nov 2016, Simon Tiong wrote: > > > >> On 18/11/2016 5:39 PM, David Lang wrote: > >>> > >>> On Fri, 18 Nov 2016, Peter Viskup wrote: > >>> > >>>> Dear all, > >>>> at the moment our message flow is setup like this: > >>>> rsyslog remote client->rsyslog server->local > >>>> file->Logstash->ElasticSearch > >>>> > >>>> Logstash does any special message processing, just forwards messages > to > >>>> ES. > >>>> We are considering the use of omelasticsearch module for direct > >>>> forward to ES (skip local file store and LogStash). > >>>> > >>>> What could be the consequences or drawbacks? What are your experiences > >>>> you can share? > >>> > >>> > >>> going directly to ES will be MUCH faster :-) > >>> > >>> one problem with recent versions of ES is that it won't accept any json > >>> that has a '.' in the object name. I don't think Logstash fixes this, > but I > >>> don't know what it does in those cases. > >>> > >>> David Lang > >> > >> > >> Hi Peter, > >> > >> ES not accepting '.', but here is a workaround way. Under logstash.conf > >> you might do some converting, for example. Hope it helps. > > > > > > the question was how to not use logstash and what the > problems/advantages of > > not using logstash are. > > > >> > >> mutate { > >> rename => { > >> "id.orig_h" => "id_orig_h" > >> "id.orig_p" => "id_orig_p" > >> "id.resp_h" => "id_resp_h" > >> "id.resp_p" => "id_resp_p" > >> } > >> } > > > > > > this requires that you know all the values that have '.' in their name. > If > > you know that you can also manually do the same thing in rsyslog. > > recent omelasticsearch handles that itself: > > https://github.com/rsyslog/rsyslog/commit/0f5595c0742fce2f5fcfe86fda22bc > 091330f368 > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
