On Fri, Dec 4, 2015 at 3:00 PM, David Lang <[email protected]> wrote:
> On Fri, 4 Dec 2015, Peter Portante wrote:
>
> On Fri, Dec 4, 2015 at 12:40 PM, Brian Knox <[email protected]>
>> wrote:
>>
>> In my case, I have "flat" ( 1 level deep ) CEE JSON logs with field names
>>> that are dot delimited ( @cee { "resp.duration_ms" : 10000,
>>> "resp.code" :
>>> 200 } ).
>>>
>>>
>> So if you have a "flat" namespace where the fields include dots in them,
>> then if you move to a hierarchical namespace then won't the field name
>> references still work?
>>
>
> the problem he's having is the the field names in his incoming logs are
> not hierarchical. He's not hand-crafting the structure the way you are,
> he's parsing incoming logs and then outputting $! to ES (or something
> similar)
>
> As such, he's pretty much stuck with the names on the incoming data.
>
We are using rsyslog to normalize the data. I'll post an example config
file for what we are doing shortly (prolly on github).
-peter
>
> Rsyslog hasn't had a requirement before now to change/sanitize the field
> names, so there's nothing setup to do this.
>
> the work-around that I can think of basically involved re-parsing the
> message after manipulating it.
>
> you could use omexternal to pass the json data to an external script that
> can muck with the names and pass them back. unfortunantly this interface
> can't delete fields, just alter or add them, so you would want to do
> something along the lines of moving everything down a level so instead of
> $!blah you have $!fixed!blah (or in json instead of { 'blah': 'value',
> 'foo': 'value' } you would have { "fixed": { "blah": "value", "foo":
> "value" } }
>
> another possibility would be to do something in rsyslog where you use a
> template to replace all '.' with some other character, and then parse the
> result with mmnormalize, but this is ugly as well.
>
> We've got a few cases where field names just don't work (case sensitivity
> , () in field names, etc), so it may be a good idea for someone to write a
> mm (message modification) module that goes through all the field names and
> sanitizes them, with several options as to what to do (and especially what
> to do if the sanitized version already exists, overwrite, try a different
> name, ??)
>
> David Lang
>
>
>
>> GIven my lack of control over the incoming logs, I think the simplest
>>> solution to this issue would be a way to change the attribute names
>>> themselves ( "resp_duration_ms", "resp_code" ).
>>> Given that I don't know the total space of all possible keys, I'd like
>>> this
>>> to work with the $!all-json property.
>>>
>>> If there's not already a way to do this that I'm missing, I think given
>>> the
>>> change in elasticsearch and that the suggested solution to this problem
>>> is
>>> "use logstash", I'd like to look at the possibility of adding a property
>>> formatter that could handle this sanitization.
>>>
>>>
>>> On Fri, Dec 4, 2015 at 11:37 AM, Peter Portante <
>>> [email protected]>
>>> wrote:
>>>
>>> We are using sub-objects:
>>>>
>>>> # this is for index names to be like: logstash-YYYY.MM.DD
>>>> # WARNING: any rsyslog collecting host MUST be running UTC
>>>> # if the proper index is to be chosen to hold the
>>>> # log entry. If you are running EDT, e.g., then
>>>> # the previous day's index will be chosen even
>>>> # though the UTC value is the current day, because
>>>> # the pattern logic does not convert "timereported"
>>>> # to a UTC value before pulling data out of it.
>>>> template(name="logstash-index-pattern" type="list") {
>>>> constant(value="logstash-")
>>>> property(name="timereported" dateFormat="rfc3339"
>>>> position.from="1" position.to="4")
>>>> constant(value=".")
>>>> property(name="timereported" dateFormat="rfc3339"
>>>> position.from="6" position.to="7")
>>>> constant(value=".")
>>>> property(name="timereported" dateFormat="rfc3339"
>>>> position.from="9" position.to="10")
>>>> }
>>>> # this is for formatting our syslog data in JSON with @timestamp using
>>>> a "hierarchical" metdata namespace
>>>> template(name="com-redhat-rsyslog-hier"
>>>> type="list") {
>>>> constant(value="{")
>>>> constant(value="\"@timestamp\":\"")
>>>> property(name="timereported" dateFormat="rfc3339")
>>>> constant(value="\",\"@version\":\"2015.09.24-0")
>>>> constant(value="\",\"message\":\"")
>>>> property(name="$.msg" format="json")
>>>> constant(value="\",\"hostname\":\"")
>>>> property(name="$.hostname")
>>>> constant(value="\",\"level\":\"")
>>>> property(name="$.level")
>>>> constant(value="\",\"pid\":\"")
>>>> property(name="$.pid")
>>>> constant(value="\",\"tags\":\"")
>>>> property(name="$.tags")
>>>> constant(value="\",\"CEE\":")
>>>> property(name="$!all-json")
>>>> constant(value=",\"systemd\":")
>>>> property(name="$.systemd")
>>>> constant(value=",\"rsyslog\":")
>>>> property(name="$.rsyslog")
>>>> constant(value="}\n")
>>>> }
>>>>
>>>>
>>>>
>>>> On Fri, Dec 4, 2015 at 10:44 AM, Brian Knox <[email protected]>
>>>> wrote:
>>>>
>>>> I found out today that elasticsearch 2.x does not allow field names to
>>>>>
>>>> have
>>>>
>>>>> the period character in them. This is making my life interesting as I
>>>>>
>>>> use
>>>>
>>>>> elasticsearch with rsyslog end to end (no logstash), and a lot of our
>>>>>
>>>> field
>>>>
>>>>> names have "." as a delimiter in them.
>>>>>
>>>>> In a perfect world, I'd like an "elasticsearch" property formatter that
>>>>> could look for and replace "." in field names with "_", that would also
>>>>> work with the all-json property, something like:
>>>>>
>>>>> property(name="$!all-json" format="elasticsearch")
>>>>>
>>>>> Or, if this is to ES specific for rsyslog core, perhaps we could add
>>>>>
>>>> this
>>>
>>>> functionality to the omelasticsearch output itself (I'll look over the
>>>>>
>>>> code
>>>>
>>>>> today).
>>>>>
>>>>> I'd like to not have to introduce logstash to my environment just to
>>>>>
>>>> regex
>>>>
>>>>> a character in field names. I'm open to other ideas as well, just
>>>>>
>>>> wanted
>>>
>>>> to start the conversation.
>>>>>
>>>>> Cheers,
>>>>> BRian
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>
>>>> myriad
>>>
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
