Sent from phone, thus brief. Am 02.04.2015 23:15 schrieb "David Lang" <[email protected]>: > > On Thu, 2 Apr 2015, Rainer Gerhards wrote: > >> Sent from phone, thus brief. >> Am 02.04.2015 22:30 schrieb "David Lang" <[email protected]>: >>> >>> >>> I have a couple systems that are sending raw JSON, but I'm also looking >> >> at interacting with logstash and nxlog systems, and they like to use raw >> JSON on a TCP port >> >> Do you know about the framing? Is it all one line? > > > in some cases yes, but not in all cases. When used as an input parser, there would need to be an option to treat newlines like whitespace and continue until the parser either completed or no longer matched. In some cases the newlines are going to be in the content not just in the whitespace around the JSON elements
Then it's more than just parsers, it's a different protocol. Rainer > > >>> >>> an example of such messages is at http://bladesdev.com/samplejson.htm >>> >>> I can send you some of the ones I've run into in the field from other >> >> sources directly. >> >> That would be good. A handful different ones would be sufficient. Just >> want to see the actual details. >> >>> >>> David Lang >>> >>> On Thu, 2 Apr 2015, Rainer Gerhards wrote: >>> >>>> Date: Thu, 2 Apr 2015 22:15:25 +0200 >>>> From: Rainer Gerhards <[email protected]> >>>> Reply-To: rsyslog-users <[email protected]> >>>> To: rsyslog-users <[email protected]> >>>> Subject: Re: [rsyslog] JSON in rsyslog >>>> >>>> >>>> Side-note: do you have some examples? >>>> >>>> Sent from phone, thus brief. >>>> Am 02.04.2015 22:09 schrieb "Rainer Gerhards" <[email protected] >: >>>> >>>>> The non obvious one sounds interesting. But bear a bit with me, the >>>>> backlog is currently really bad and thus week had a lot of surprises;) >>>>> >>>>> Rainer >>>>> >>>>> Sent from phone, thus brief. >>>>> Am 02.04.2015 20:24 schrieb "David Lang" <[email protected]>: >>>>> >>>>>> cee appears to be defunct, but I am seeing more use of JSON in logs, >> >> but >>>>>> >>>>>> the way it's being used isn't meshing well with stock rsyslog. >>>>>> >>>>>> I'm seeing several tools that are sending raw JSON to the log port (no >>>>>> pri, timestamp, hostname, syslogtag or cee cookie, just {stuff} >>>>>> >>>>>> I see two obvious approaches to better handling of this sort of thing >> >> and >>>>>> >>>>>> one not-so-obvious approach that I think may actually be a better >> >> long-term >>>>>> >>>>>> option. >>>>>> >>>>>> 1. modify mmjsonparse to have the @cee cookie be optional, and gain the >>>>>> source and destination options that mmlognormalize has, then have >> >> rules in >>>>>> >>>>>> rsyslog to detect that $rawmsg startswith "{" and invoke the >> >> mmjsonparse >>>>>> >>>>>> against $rawmsg >>>>>> >>>>>> 2. create a new pmjson module that invokes json parsing against the raw >>>>>> message and populates the other standard properties >>>>>> >>>>>> then the non-obvious solution is in two parts (both of which I think >> >> are >>>>>> >>>>>> useful independently, and potentially extremely powerful when combined) >>>>>> >>>>>> two parts. >>>>>> >>>>>> 1. add an extension to liblognorm to let it be able to parse JSON into >> >> a >>>>>> >>>>>> tree (the way it does with descent, but without the current bug there) >>>>>> >>>>>> 2. allow a parser() definition to be used to create a parser that uses >>>>>> liblognorm (in addition to populating the rawmsg, fromhost-ip, >> >> fromhost, >>>>>> >>>>>> inputname, and timegenerated properties) and allow that to be bound to >> >> a >>>>>> >>>>>> ruleset/input for parsing. >>>>>> >>>>>> thoughts? >>>>>> >>>>>> David Lang >>>>>> >>>>>> P.S. I'm also wondering if it may be that in the longer term, it could >> >> be >>>>>> >>>>>> a win to replace some of the existing internal parsers with liblognorm >>>>>> definitions, since the parse tree is such an efficient parser. >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> myriad >>>>>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
