On Thu, 2 Apr 2015, Rainer Gerhards wrote:
Sent from phone, thus brief.
Am 02.04.2015 22:30 schrieb "David Lang" <[email protected]>:
I have a couple systems that are sending raw JSON, but I'm also looking
at interacting with logstash and nxlog systems, and they like to use raw
JSON on a TCP port
Do you know about the framing? Is it all one line?
in some cases yes, but not in all cases. When used as an input parser, there
would need to be an option to treat newlines like whitespace and continue until
the parser either completed or no longer matched. In some cases the newlines are
going to be in the content not just in the whitespace around the JSON elements
an example of such messages is at http://bladesdev.com/samplejson.htm
I can send you some of the ones I've run into in the field from other
sources directly.
That would be good. A handful different ones would be sufficient. Just
want to see the actual details.
David Lang
On Thu, 2 Apr 2015, Rainer Gerhards wrote:
Date: Thu, 2 Apr 2015 22:15:25 +0200
From: Rainer Gerhards <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] JSON in rsyslog
Side-note: do you have some examples?
Sent from phone, thus brief.
Am 02.04.2015 22:09 schrieb "Rainer Gerhards" <[email protected]>:
The non obvious one sounds interesting. But bear a bit with me, the
backlog is currently really bad and thus week had a lot of surprises;)
Rainer
Sent from phone, thus brief.
Am 02.04.2015 20:24 schrieb "David Lang" <[email protected]>:
cee appears to be defunct, but I am seeing more use of JSON in logs,
but
the way it's being used isn't meshing well with stock rsyslog.
I'm seeing several tools that are sending raw JSON to the log port (no
pri, timestamp, hostname, syslogtag or cee cookie, just {stuff}
I see two obvious approaches to better handling of this sort of thing
and
one not-so-obvious approach that I think may actually be a better
long-term
option.
1. modify mmjsonparse to have the @cee cookie be optional, and gain the
source and destination options that mmlognormalize has, then have
rules in
rsyslog to detect that $rawmsg startswith "{" and invoke the
mmjsonparse
against $rawmsg
2. create a new pmjson module that invokes json parsing against the raw
message and populates the other standard properties
then the non-obvious solution is in two parts (both of which I think
are
useful independently, and potentially extremely powerful when combined)
two parts.
1. add an extension to liblognorm to let it be able to parse JSON into
a
tree (the way it does with descent, but without the current bug there)
2. allow a parser() definition to be used to create a parser that uses
liblognorm (in addition to populating the rawmsg, fromhost-ip,
fromhost,
inputname, and timegenerated properties) and allow that to be bound to
a
ruleset/input for parsing.
thoughts?
David Lang
P.S. I'm also wondering if it may be that in the longer term, it could
be
a win to replace some of the existing internal parsers with liblognorm
definitions, since the parse tree is such an efficient parser.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
