On Apr 14, 2015, at 11:43 PM, David Lang <[email protected]> wrote: > > On Wed, 15 Apr 2015, Ezell, Matthew A. wrote: > >> >> Hello- >> >> What is the current "best practice" for a portable application to get >> structured data to rsyslog? >> >> ... gets JSON printed to the log. That may be undesirable in the common >> case. > > the question is why it is undesirable and how much effort you are willing to > do > to fix the issue. ... > What I do is I ask for the apps to output in JSON wherever possible, and I > don't > worry about creating a human friendly message in a text file. I write the > JSON > (or a subset of it) to the text file and if someone needs it prettier, they > can > read the JSON and convert it.
For example: as long as there aren’t character set conversion issues such as writing Windows-1252 encoded strings into what should be UTF-8 JSON encoding, then tools like 'jq' <http://stedolan.github.io/jq/> are very helpful for pulling fields out of JSON-format logs. It can be as simple as: pipe the file to "jq -r '.msg'" to get the raw logs back out. > what I do is to take whatever message was output and then run mmjsonparse > against it. If it's cee JSON (insert grumble about the requirement for the > cee > cookie ;-) I have all the variables, but no $!msg field. If I have a $!msg > field, then I parse it using mmnormalize to extract variables from it. If > there > isn't a $!msg field, I set $!msg=$mesg so that I have something I can spit > out > when I'm doing a 'plain' logfile. > > I also add metadata to the JSON (fromhost-ip, received time, hostname of > relay, > and an environment tag so that later on I can trivially tell the difference > between dev and prod copies of the same software) We do something very similar to this, and I suspect so do other high-volume Rsyslog users such as Radu at Sematext. I feel this should just be Rsyslog’s recommended operational practice. If you’re building a log monitoring infrastructure today, this is how you should be doing it. Free-form text syslog should be considered a legacy encoding that is of course still supported as an input format (and if you must, an output format). Maybe we should put such a recommended config prominently on the Rsyslog web site to help overcome any lingering impressions that syslog is a legacy logging format that has been replaced by journald. -- Dave Caplinger, Director of Architecture | Solutionary — An NTT Group Security Company _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
