In my case, I have "flat" ( 1 level deep ) CEE JSON logs with field names
that are dot delimited ( @cee { "resp.duration_ms" : 10000, "resp.code" :
200 } ).
GIven my lack of control over the incoming logs, I think the simplest
solution to this issue would be a way to change the attribute names
themselves ( "resp_duration_ms", "resp_code" ).
Given that I don't know the total space of all possible keys, I'd like this
to work with the $!all-json property.
If there's not already a way to do this that I'm missing, I think given the
change in elasticsearch and that the suggested solution to this problem is
"use logstash", I'd like to look at the possibility of adding a property
formatter that could handle this sanitization.
On Fri, Dec 4, 2015 at 11:37 AM, Peter Portante <[email protected]>
wrote:
> We are using sub-objects:
>
> # this is for index names to be like: logstash-YYYY.MM.DD
> # WARNING: any rsyslog collecting host MUST be running UTC
> # if the proper index is to be chosen to hold the
> # log entry. If you are running EDT, e.g., then
> # the previous day's index will be chosen even
> # though the UTC value is the current day, because
> # the pattern logic does not convert "timereported"
> # to a UTC value before pulling data out of it.
> template(name="logstash-index-pattern" type="list") {
> constant(value="logstash-")
> property(name="timereported" dateFormat="rfc3339"
> position.from="1" position.to="4")
> constant(value=".")
> property(name="timereported" dateFormat="rfc3339"
> position.from="6" position.to="7")
> constant(value=".")
> property(name="timereported" dateFormat="rfc3339"
> position.from="9" position.to="10")
> }
> # this is for formatting our syslog data in JSON with @timestamp using
> a "hierarchical" metdata namespace
> template(name="com-redhat-rsyslog-hier"
> type="list") {
> constant(value="{")
> constant(value="\"@timestamp\":\"")
> property(name="timereported" dateFormat="rfc3339")
> constant(value="\",\"@version\":\"2015.09.24-0")
> constant(value="\",\"message\":\"")
> property(name="$.msg" format="json")
> constant(value="\",\"hostname\":\"")
> property(name="$.hostname")
> constant(value="\",\"level\":\"")
> property(name="$.level")
> constant(value="\",\"pid\":\"")
> property(name="$.pid")
> constant(value="\",\"tags\":\"")
> property(name="$.tags")
> constant(value="\",\"CEE\":")
> property(name="$!all-json")
> constant(value=",\"systemd\":")
> property(name="$.systemd")
> constant(value=",\"rsyslog\":")
> property(name="$.rsyslog")
> constant(value="}\n")
> }
>
>
>
> On Fri, Dec 4, 2015 at 10:44 AM, Brian Knox <[email protected]>
> wrote:
>
> > I found out today that elasticsearch 2.x does not allow field names to
> have
> > the period character in them. This is making my life interesting as I
> use
> > elasticsearch with rsyslog end to end (no logstash), and a lot of our
> field
> > names have "." as a delimiter in them.
> >
> > In a perfect world, I'd like an "elasticsearch" property formatter that
> > could look for and replace "." in field names with "_", that would also
> > work with the all-json property, something like:
> >
> > property(name="$!all-json" format="elasticsearch")
> >
> > Or, if this is to ES specific for rsyslog core, perhaps we could add this
> > functionality to the omelasticsearch output itself (I'll look over the
> code
> > today).
> >
> > I'd like to not have to introduce logstash to my environment just to
> regex
> > a character in field names. I'm open to other ideas as well, just wanted
> > to start the conversation.
> >
> > Cheers,
> > BRian
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
