Can you file a feature request for melasticsesarch on github. I guess a quick but useful back could be done there.
Rainer Am 05.12.2015 16:15 schrieb "Brian Knox" <[email protected]>: > David - yes, that exactly describes the situation that I'm in. If I can't > find a short term solution with existing capabilities, I may look into > providing a load balanced pool of sanitization workers that I connect to > over the zeromq plugins I've been working on as a more near term solution. > Ideally, I'd like to be able to handle the sanitization within rsyslog > itself. > > For a quick hack, a template on my output from my aggregators replacing "." > characters with "_" might work and I'll give that a spin. I still have an > elasticsearch 1.5 cluster that is our production cluster in parallel with > the new 2.1 cluster, so I have some room to experiment. > > As an aside - does anyone have a link to a config example using a regex > replace on a property using the new v8 template format? > > Peter - I'd be very interested if you have an approach to this problem that > works with existing syslog capability. > > Cheers, > Brian > > > > > On Fri, Dec 4, 2015 at 3:28 PM, Peter Portante <[email protected] > > > wrote: > > > On Fri, Dec 4, 2015 at 3:00 PM, David Lang <[email protected]> wrote: > > > > > On Fri, 4 Dec 2015, Peter Portante wrote: > > > > > > On Fri, Dec 4, 2015 at 12:40 PM, Brian Knox <[email protected]> > > >> wrote: > > >> > > >> In my case, I have "flat" ( 1 level deep ) CEE JSON logs with field > > names > > >>> that are dot delimited ( @cee { "resp.duration_ms" : 10000, > > >>> "resp.code" : > > >>> 200 } ). > > >>> > > >>> > > >> So if you have a "flat" namespace where the fields include dots in > them, > > >> then if you move to a hierarchical namespace then won't the field name > > >> references still work? > > >> > > > > > > the problem he's having is the the field names in his incoming logs are > > > not hierarchical. He's not hand-crafting the structure the way you are, > > > he's parsing incoming logs and then outputting $! to ES (or something > > > similar) > > > > > > As such, he's pretty much stuck with the names on the incoming data. > > > > > > > We are using rsyslog to normalize the data. I'll post an example config > > file for what we are doing shortly (prolly on github). > > > > -peter > > > > > > > > > > Rsyslog hasn't had a requirement before now to change/sanitize the > field > > > names, so there's nothing setup to do this. > > > > > > the work-around that I can think of basically involved re-parsing the > > > message after manipulating it. > > > > > > you could use omexternal to pass the json data to an external script > that > > > can muck with the names and pass them back. unfortunantly this > interface > > > can't delete fields, just alter or add them, so you would want to do > > > something along the lines of moving everything down a level so instead > of > > > $!blah you have $!fixed!blah (or in json instead of { 'blah': 'value', > > > 'foo': 'value' } you would have { "fixed": { "blah": "value", "foo": > > > "value" } } > > > > > > another possibility would be to do something in rsyslog where you use a > > > template to replace all '.' with some other character, and then parse > the > > > result with mmnormalize, but this is ugly as well. > > > > > > We've got a few cases where field names just don't work (case > sensitivity > > > , () in field names, etc), so it may be a good idea for someone to > write > > a > > > mm (message modification) module that goes through all the field names > > and > > > sanitizes them, with several options as to what to do (and especially > > what > > > to do if the sanitized version already exists, overwrite, try a > different > > > name, ??) > > > > > > David Lang > > > > > > > > > > > >> GIven my lack of control over the incoming logs, I think the simplest > > >>> solution to this issue would be a way to change the attribute names > > >>> themselves ( "resp_duration_ms", "resp_code" ). > > >>> Given that I don't know the total space of all possible keys, I'd > like > > >>> this > > >>> to work with the $!all-json property. > > >>> > > >>> If there's not already a way to do this that I'm missing, I think > given > > >>> the > > >>> change in elasticsearch and that the suggested solution to this > problem > > >>> is > > >>> "use logstash", I'd like to look at the possibility of adding a > > property > > >>> formatter that could handle this sanitization. > > >>> > > >>> > > >>> On Fri, Dec 4, 2015 at 11:37 AM, Peter Portante < > > >>> [email protected]> > > >>> wrote: > > >>> > > >>> We are using sub-objects: > > >>>> > > >>>> # this is for index names to be like: logstash-YYYY.MM.DD > > >>>> # WARNING: any rsyslog collecting host MUST be running UTC > > >>>> # if the proper index is to be chosen to hold the > > >>>> # log entry. If you are running EDT, e.g., then > > >>>> # the previous day's index will be chosen even > > >>>> # though the UTC value is the current day, because > > >>>> # the pattern logic does not convert "timereported" > > >>>> # to a UTC value before pulling data out of it. > > >>>> template(name="logstash-index-pattern" type="list") { > > >>>> constant(value="logstash-") > > >>>> property(name="timereported" dateFormat="rfc3339" > > >>>> position.from="1" position.to="4") > > >>>> constant(value=".") > > >>>> property(name="timereported" dateFormat="rfc3339" > > >>>> position.from="6" position.to="7") > > >>>> constant(value=".") > > >>>> property(name="timereported" dateFormat="rfc3339" > > >>>> position.from="9" position.to="10") > > >>>> } > > >>>> # this is for formatting our syslog data in JSON with @timestamp > using > > >>>> a "hierarchical" metdata namespace > > >>>> template(name="com-redhat-rsyslog-hier" > > >>>> type="list") { > > >>>> constant(value="{") > > >>>> constant(value="\"@timestamp\":\"") > > >>>> property(name="timereported" dateFormat="rfc3339") > > >>>> constant(value="\",\"@version\":\"2015.09.24-0") > > >>>> constant(value="\",\"message\":\"") > > >>>> property(name="$.msg" format="json") > > >>>> constant(value="\",\"hostname\":\"") > > >>>> property(name="$.hostname") > > >>>> constant(value="\",\"level\":\"") > > >>>> property(name="$.level") > > >>>> constant(value="\",\"pid\":\"") > > >>>> property(name="$.pid") > > >>>> constant(value="\",\"tags\":\"") > > >>>> property(name="$.tags") > > >>>> constant(value="\",\"CEE\":") > > >>>> property(name="$!all-json") > > >>>> constant(value=",\"systemd\":") > > >>>> property(name="$.systemd") > > >>>> constant(value=",\"rsyslog\":") > > >>>> property(name="$.rsyslog") > > >>>> constant(value="}\n") > > >>>> } > > >>>> > > >>>> > > >>>> > > >>>> On Fri, Dec 4, 2015 at 10:44 AM, Brian Knox <[email protected] > > > > >>>> wrote: > > >>>> > > >>>> I found out today that elasticsearch 2.x does not allow field names > to > > >>>>> > > >>>> have > > >>>> > > >>>>> the period character in them. This is making my life interesting > as > > I > > >>>>> > > >>>> use > > >>>> > > >>>>> elasticsearch with rsyslog end to end (no logstash), and a lot of > our > > >>>>> > > >>>> field > > >>>> > > >>>>> names have "." as a delimiter in them. > > >>>>> > > >>>>> In a perfect world, I'd like an "elasticsearch" property formatter > > that > > >>>>> could look for and replace "." in field names with "_", that would > > also > > >>>>> work with the all-json property, something like: > > >>>>> > > >>>>> property(name="$!all-json" format="elasticsearch") > > >>>>> > > >>>>> Or, if this is to ES specific for rsyslog core, perhaps we could > add > > >>>>> > > >>>> this > > >>> > > >>>> functionality to the omelasticsearch output itself (I'll look over > the > > >>>>> > > >>>> code > > >>>> > > >>>>> today). > > >>>>> > > >>>>> I'd like to not have to introduce logstash to my environment just > to > > >>>>> > > >>>> regex > > >>>> > > >>>>> a character in field names. I'm open to other ideas as well, just > > >>>>> > > >>>> wanted > > >>> > > >>>> to start the conversation. > > >>>>> > > >>>>> Cheers, > > >>>>> BRian > > >>>>> _______________________________________________ > > >>>>> rsyslog mailing list > > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>> http://www.rsyslog.com/professional-services/ > > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >>>>> > > >>>> myriad > > >>> > > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > >>>>> DON'T LIKE THAT. > > >>>>> > > >>>>> _______________________________________________ > > >>>> rsyslog mailing list > > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>> http://www.rsyslog.com/professional-services/ > > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > >>>> DON'T LIKE THAT. > > >>>> > > >>>> _______________________________________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com/professional-services/ > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > >>> DON'T LIKE THAT. > > >>> > > >>> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com/professional-services/ > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >> DON'T LIKE THAT. > > >> > > >> _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
