Ah, OK.
Have you looked at what's sent on the wire then via tcpdump or
ngrep?
BR,
- Simon
On Tue, 2023-09-19 at 09:16:45 +0200, Lennon, Sean (UK) wrote:
>
>
>
>
> This email may contain proprietary information of BAE Systems and/or third
> parties.
>
> Hi Simon, thanks for your response. I am using logstash in foreground mode
> to view the rsyslog output. I'm also monitoring the output of the mmexternal
> bespoke code via the mmextenal debug option, this goes into a file and I am
> happy with that output. It's what logstash receives that is the problem. I
> was initially sending the complete json message to a file but this also
> exhibited the problem. Therefore my conclusion is that the problem occurs
> between the output of the mmextenal code and the process that generates the
> json output.
>
> -----Original Message-----
> From: Simon Lundström <si...@su.se>
> Sent: 19 September 2023 06:38
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Rainer Gerhards <rgerha...@hq.adiscon.com>; Lennon, Sean (UK)
> <sean.lenn...@baesystems.com>
> Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with escaped
> quotations and additional quotations
>
> ----------------------------- PHISHING ALERT -----------------------------
> This email has been sent from an account outside of the BAE Systems network.
>
> Please treat the email with caution, especially if you are requested to click
> on a link or open an attachment.
> For further information on how to spot and report a phishing email please
> access the Global Intranet, then select <Functions> / <IT>.
>
> ------------------------------------------------------------------------------------
>
> Morning Sean,
>
> Are you using the stdout output plugin to view the logs from logstash?
> IIRC it tries to escape the data.
>
> Try outputing the logs to a file.
>
> Using tcpdump to look at the syslog data after rsyslog sends it and/or before
> rsyslog receives it might also help.
>
> BR,
> - Simon
>
> On Mon, 2023-09-18 at 17:04:25 +0200, Lennon, Sean (UK) via rsyslog wrote:
> > This email may contain proprietary information of BAE Systems and/or third
> > parties.
> >
> > Sorry, but for ‘reasons’ I can only give you a severely edited version, I
> > have used debug output from mmexternal first and the received message from
> > logstash second:
> >
> >
> > 1. mexternal debug output – I am satisfied with this.
> > { “msg” :
> > {“messageGroup”:[{“field1”:1,”field2”:2},{“field1”:3,”field2”:4}]}}
> >
> > 2. what logstash receives
> > “message” => “{ \“msg\” :
> > {\“messageGroup\”:[{\“field1\”:1,\”field2\”:2},{\“field1\”:3,\”field2\”:4}]}}”
> >
> >
> > From: Rainer Gerhards <rgerha...@hq.adiscon.com>
> > Sent: 18 September 2023 15:47
> > To: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
> > Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> > Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with
> > escaped quotations and additional quotations
> >
> >
> > PHISHING ALERT
> > This email has been sent from an account outside of the BAE Systems network.
> >
> > Please treat the email with caution, especially if you are requested to
> > click on a link or open an attachment.
> > For further information on how to spot and report a phishing email please
> > access the Global Intranet then select <Functions> / <IT>.
> > If you think this is a phishing email, please report it by using the
> > "Report Phishing" button in Outlook.
> >
> >
> > Output the message with RSYSLOG_DebugFormat template. I need to see which
> > data msg actually has.
> >
> > Rainer
> > Sent from phone, thus brief.
> >
> > Lennon, Sean (UK)
> > <sean.lenn...@baesystems.com<mailto:sean.lenn...@baesystems.com>> schrieb
> > am Mo., 18. Sept. 2023, 16:41:
> >
> >
> >
> >
> > This email may contain proprietary information of BAE Systems and/or third
> > parties.
> >
> > Thanks for your response Rainer. I don't think it answers my question, I
> > have property fields from the Rsyslog message that are fine, they get
> > formatted correctly, for example 'timereported' or 'syslogseverity-text'.
> > So, the output json for these and others are correct, it's the msg field
> > that is returned from my custom code (using mmexternal) that is the problem.
> >
> > I have created a newer template that is more upto date and looks something
> > similar to this:
> >
> > template(name="json-template" type="list" option.jsonf="on") {
> > property(outname="@timestamp" name="timereported"
> > dataformat="rfc3339" format="jsonf")
> > property(outname="message" name="msg" format="jsonf") }
> >
> > -----Original Message-----
> > From: Rainer Gerhards
> > <rgerha...@hq.adiscon.com<mailto:rgerha...@hq.adiscon.com>>
> > Sent: 18 September 2023 15:26
> > To: rsyslog-users
> > <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>
> > Cc: Lennon, Sean (UK)
> > <sean.lenn...@baesystems.com<mailto:sean.lenn...@baesystems.com>>
> > Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with
> > escaped quotations and additional quotations
> >
> > ----------------------------- PHISHING ALERT
> > ----------------------------- This email has been sent from an account
> > outside of the BAE Systems network.
> >
> > Please treat the email with caution, especially if you are requested to
> > click on a link or open an attachment.
> > For further information on how to spot and report a phishing email please
> > access the Global Intranet, then select <Functions> / <IT>.
> >
> > ----------------------------------------------------------------------
> > --------------
> >
> > Does this example from the rsyslog testbench help?
> >
> > https://github.com/rsyslog/rsyslog/blob/761cb2bc51e3046b242b45994cff11
> > ff8be3990e/tests/json-nonstring.sh#L4
> >
> > Rainer
> >
> > El lun, 18 sept 2023 a las 15:10, Lennon, Sean (UK) via rsyslog
> > (<rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>) escribió:
> > >
> > >
> > >
> > >
> > >
> > > This email may contain proprietary information of BAE Systems and/or
> > > third parties.
> > >
> > > This is the one I meant.
> > >
> > > -----Original Message-----
> > > From: rsyslog
> > > <rsyslog-boun...@lists.adiscon.com<mailto:rsyslog-boun...@lists.adis
> > > con.com>> On Behalf Of Lennon, Sean (UK) via rsyslog
> > > Sent: 29 August 2023 17:39
> > > To: rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>
> > > Cc: Lennon, Sean (UK)
> > > <sean.lenn...@baesystems.com<mailto:sean.lenn...@baesystems.com>>
> > > Subject: [rsyslog] rsyslog mmextenal logstash json output with
> > > escaped quotations and additional quotations
> > >
> > > ----------------------------- PHISHING ALERT
> > > ----------------------------- This email has been sent from an account
> > > outside of the BAE Systems network.
> > >
> > > Please treat the email with caution, especially if you are requested to
> > > click on a link or open an attachment.
> > > For further information on how to spot and report a phishing email please
> > > access the Global Intranet, then select <Functions> / <IT>.
> > >
> > > --------------------------------------------------------------------
> > > --
> > > --------------
> > >
> > > This email may contain proprietary information of BAE Systems and/or
> > > third parties.
> > >
> > > Hi all,
> > >
> > > I've encountered an issue with formatting json output to logstash. I'm
> > > using mmexternal to reformat data received from a remote system, the data
> > > is project specific and needs to be massaged into json for use with
> > > logstash. The intention is to create a json message for logstash with
> > > the mmexternal output being part of that message. I'm able to receive
> > > this json output at logstash but the message field (which contains the
> > > mmexternal output) is encapsulated within double quotes and all json
> > > fields within have escaped double quotes. This means that logstash is
> > > not able to interpret part of the message. If I take the raw output of
> > > the mmextenal code and send it to a omfile then it looks perfectly fine.
> > >
> > > I have asked a more detailed question, on Stackoverflow:
> > > https://stackoverflow.com/questions/77001549/rsyslog-mmextenal-logst
> > > as h-json-output-with-escaped-quotations-and-additional-qu
> > >
> > > What am I missing?
> > >
> > > I appreciate your help.
> > >
> > > Sean
> > >
> > > ********************************************************************
> > > This email and any attachments are confidential to the intended recipient
> > > and may also be privileged. If you are not the intended recipient please
> > > delete it from your system and notify the sender.
> > > You should not copy it or use it for any purpose nor disclose or
> > > distribute its contents to any other person.
> > > ********************************************************************
> > >
> > > BAE Systems may process information about you that may be subject to
> > > data protection laws. For more information about how we use your
> > > personal information, how we protect your information, our legal
> > > basis for using your information, your rights and who you can
> > > contact, please refer to our Privacy Notice at
> > > www.baesystems.com/en/privacy<http://www.baesystems.com/en/privacy>
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
> > > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
> > > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
> > > THAT.
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > > LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
