And, as I wrote, rsyslog already removes dots... Rainer Sent from phone, thus brief.
Am 18.11.2016 15:55 schrieb "Rich Megginson" <[email protected]>: > On 11/18/2016 05:27 AM, Ryan Ward wrote: > >> Supporting dots are back in the latest elasticsearch 5.0 >> > > And, if you don't want to use 5.0 just yet, 2.4.1 supports dots in field > names if you use the -Dmapper.allow_dots_in_name=true flag. > > >> On Fri, Nov 18, 2016 at 6:16 AM, chenlin rao <[email protected]> >> wrote: >> >> There is a logstash plugin named logstash-filter-de_dot. >>> And because logstash has a force queue to send a larger bulk_size to ES >>> which is better in ES recommandition, I suggest to use >>> rsyslog-omelasticsearch only for not so high load -- but need to watch it >>> as quickly-- use case. >>> >>> 2016-11-18 18:53 GMT+08:00 Rainer Gerhards <[email protected]>: >>> >>> 2016-11-18 11:51 GMT+01:00 David Lang <[email protected]>: >>>> >>>>> On Fri, 18 Nov 2016, Simon Tiong wrote: >>>>> >>>>> On 18/11/2016 5:39 PM, David Lang wrote: >>>>>> >>>>>>> On Fri, 18 Nov 2016, Peter Viskup wrote: >>>>>>> >>>>>>> Dear all, >>>>>>>> at the moment our message flow is setup like this: >>>>>>>> rsyslog remote client->rsyslog server->local >>>>>>>> file->Logstash->ElasticSearch >>>>>>>> >>>>>>>> Logstash does any special message processing, just forwards messages >>>>>>>> >>>>>>> to >>>> >>>>> ES. >>>>>>>> We are considering the use of omelasticsearch module for direct >>>>>>>> forward to ES (skip local file store and LogStash). >>>>>>>> >>>>>>>> What could be the consequences or drawbacks? What are your >>>>>>>> >>>>>>> experiences >>> >>>> you can share? >>>>>>>> >>>>>>> >>>>>>> going directly to ES will be MUCH faster :-) >>>>>>> >>>>>>> one problem with recent versions of ES is that it won't accept any >>>>>>> >>>>>> json >>> >>>> that has a '.' in the object name. I don't think Logstash fixes this, >>>>>>> >>>>>> but I >>>> >>>>> don't know what it does in those cases. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>> >>>>>> Hi Peter, >>>>>> >>>>>> ES not accepting '.', but here is a workaround way. Under >>>>>> >>>>> logstash.conf >>> >>>> you might do some converting, for example. Hope it helps. >>>>>> >>>>> >>>>> the question was how to not use logstash and what the >>>>> >>>> problems/advantages of >>>> >>>>> not using logstash are. >>>>> >>>>> mutate { >>>>>> rename => { >>>>>> "id.orig_h" => "id_orig_h" >>>>>> "id.orig_p" => "id_orig_p" >>>>>> "id.resp_h" => "id_resp_h" >>>>>> "id.resp_p" => "id_resp_p" >>>>>> } >>>>>> } >>>>>> >>>>> >>>>> this requires that you know all the values that have '.' in their name. >>>>> >>>> If >>>> >>>>> you know that you can also manually do the same thing in rsyslog. >>>>> >>>> recent omelasticsearch handles that itself: >>>> >>>> https://github.com/rsyslog/rsyslog/commit/0f5595c0742fce2f5f >>>> cfe86fda22bc >>>> 091330f368 >>>> >>>> Rainer >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
