On Fri, Dec 4, 2015 at 12:40 PM, Brian Knox <[email protected]> wrote:
> In my case, I have "flat" ( 1 level deep ) CEE JSON logs with field names
> that are dot delimited ( @cee { "resp.duration_ms" : 10000, "resp.code" :
> 200 } ).
>
So if you have a "flat" namespace where the fields include dots in them,
then if you move to a hierarchical namespace then won't the field name
references still work?
> GIven my lack of control over the incoming logs, I think the simplest
> solution to this issue would be a way to change the attribute names
> themselves ( "resp_duration_ms", "resp_code" ).
> Given that I don't know the total space of all possible keys, I'd like this
> to work with the $!all-json property.
>
> If there's not already a way to do this that I'm missing, I think given the
> change in elasticsearch and that the suggested solution to this problem is
> "use logstash", I'd like to look at the possibility of adding a property
> formatter that could handle this sanitization.
>
>
> On Fri, Dec 4, 2015 at 11:37 AM, Peter Portante <
> [email protected]>
> wrote:
>
> > We are using sub-objects:
> >
> > # this is for index names to be like: logstash-YYYY.MM.DD
> > # WARNING: any rsyslog collecting host MUST be running UTC
> > # if the proper index is to be chosen to hold the
> > # log entry. If you are running EDT, e.g., then
> > # the previous day's index will be chosen even
> > # though the UTC value is the current day, because
> > # the pattern logic does not convert "timereported"
> > # to a UTC value before pulling data out of it.
> > template(name="logstash-index-pattern" type="list") {
> > constant(value="logstash-")
> > property(name="timereported" dateFormat="rfc3339"
> > position.from="1" position.to="4")
> > constant(value=".")
> > property(name="timereported" dateFormat="rfc3339"
> > position.from="6" position.to="7")
> > constant(value=".")
> > property(name="timereported" dateFormat="rfc3339"
> > position.from="9" position.to="10")
> > }
> > # this is for formatting our syslog data in JSON with @timestamp using
> > a "hierarchical" metdata namespace
> > template(name="com-redhat-rsyslog-hier"
> > type="list") {
> > constant(value="{")
> > constant(value="\"@timestamp\":\"")
> > property(name="timereported" dateFormat="rfc3339")
> > constant(value="\",\"@version\":\"2015.09.24-0")
> > constant(value="\",\"message\":\"")
> > property(name="$.msg" format="json")
> > constant(value="\",\"hostname\":\"")
> > property(name="$.hostname")
> > constant(value="\",\"level\":\"")
> > property(name="$.level")
> > constant(value="\",\"pid\":\"")
> > property(name="$.pid")
> > constant(value="\",\"tags\":\"")
> > property(name="$.tags")
> > constant(value="\",\"CEE\":")
> > property(name="$!all-json")
> > constant(value=",\"systemd\":")
> > property(name="$.systemd")
> > constant(value=",\"rsyslog\":")
> > property(name="$.rsyslog")
> > constant(value="}\n")
> > }
> >
> >
> >
> > On Fri, Dec 4, 2015 at 10:44 AM, Brian Knox <[email protected]>
> > wrote:
> >
> > > I found out today that elasticsearch 2.x does not allow field names to
> > have
> > > the period character in them. This is making my life interesting as I
> > use
> > > elasticsearch with rsyslog end to end (no logstash), and a lot of our
> > field
> > > names have "." as a delimiter in them.
> > >
> > > In a perfect world, I'd like an "elasticsearch" property formatter that
> > > could look for and replace "." in field names with "_", that would also
> > > work with the all-json property, something like:
> > >
> > > property(name="$!all-json" format="elasticsearch")
> > >
> > > Or, if this is to ES specific for rsyslog core, perhaps we could add
> this
> > > functionality to the omelasticsearch output itself (I'll look over the
> > code
> > > today).
> > >
> > > I'd like to not have to introduce logstash to my environment just to
> > regex
> > > a character in field names. I'm open to other ideas as well, just
> wanted
> > > to start the conversation.
> > >
> > > Cheers,
> > > BRian
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > > DON'T LIKE THAT.
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
