James,
Thanks for bringing this up. I've added expanded SD support to our roadmap,
currently we only support an RFC5424 compliant header with our own Structured
Data block.
Leila.
Loggly, Inc.
On Sep 12, 2013, at 3:58 AM, James Crowley wrote:
> Thanks Botond. For some reason Loggly doesn't support the IETF structured
> data format, nor do they support BSD. Will give the other routes a try, thank
> you.
>
>
> On 10 September 2013 12:43, Botond Botyanszki <[email protected]> wrote:
> Hi James,
>
> Not sure why this is necessary since rfc 5424 syslog already has
> structured data. Can't Loggly parse this without requiring JSON?
> If you still need to go with JSON, here are some possible solutions:
>
> 1. Try using bsd syslog:
> $Message = to_json(); to_syslog_bsd();
> Not sure what happens with the customer token in this case.
> 2. Use a regexp replacement operator s/// to yank the structured data in
> $raw_event .
> 3. You can explicitly remove fields so that they will not be included in
> the structured data part before it gets converted to ietf syslog:
> $Message = to_json(); delete($RecordNumber); delete(...); \
> to_syslog_ietf();
> 4. Construct the $raw_event manually without using to_syslog_ietf().
> Possibly there are other formats which loggly will accept which include
> a simple header and then the JSON message part.
>
> Regards,
> Botond
>
>
> On Mon, 9 Sep 2013 18:36:29 +0100
> James Crowley <[email protected]> wrote:
>
> > Does anyone know how I can keep the data fields that I've converted to JSON
> > out of the structured data element of syslog_ietf?
> >
> > I'm currently outputting data like this:
> >
> > Exec $Message = to_json(); to_syslog_ietf(); $raw_event =
> > replace($raw_event, 'NXLOG@14506', '%CUSTOMER_TOKEN%@41058 tag="windows"]
> > [', 1);
> >
> > which gives me output like the below, note how the fields are supplied
> > twice once in the [FieldName="X" FieldName="Y"] format and once in the {
> > FieldName: "X", FieldName: "Y" } format? The consumer (Loggly) at the other
> > end requires structured data in JSON, but I just want to avoid doubling the
> > size of our logs unnecessarily!
> >
> > Any ideas? Many thanks
> >
> > <14>1 2013-09-09T17:15:50.507658+00:00 fa-testdevelop
> > Microsoft-Windows-WMI-Activity 12260 -
> > [a3c50c76-0fc1-412e-a10f-efe6e7b6e09f@41058 tag="windows"] [
> > Keywords="4611686018427387904" EventType="INFO" EventID="5857"
> > ProviderGuid="{1418EF04-B0B4-4623-BF7E-D74AB47BBDAA}" Version="0" Task="0"
> > OpcodeValue="0" RecordNumber="15145" ThreadID="5592"
> > Channel="Microsoft-Windows-WMI-Activity/Operational" Domain="NT AUTHORITY"
> > AccountName="NETWORK SERVICE" UserID="NETWORK SERVICE" AccountType="Well
> > Known Group" Opcode="Info" EventReceivedTime="2013-09-09 17:15:51"
> > SourceModuleName="eventlog" SourceModuleType="im_msvistalog"]
> > {"EventTime":"2013-09-09
> > 17:15:50","Hostname":"fa-testdevelop","Keywords":4611686018427387904,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":5857,"SourceName":"Microsoft-Windows-WMI-Activity","ProviderGuid":"{1418EF04-B0B4-4623-BF7E-D74AB47BBDAA}","Version":0,"Task":0,"OpcodeValue":0,"RecordNumber":15145,"ProcessID":12260,"ThreadID":5592,"Channel":"Microsoft-Windows-WMI-Activity/Operational","Domain":"NT
> > AUTHORITY","AccountName":"NETWORK SERVICE","UserID":"NETWORK
> > SERVICE","AccountType":"Well Known
> > Group","Message":"Win32_WIN32_TERMINALSERVICE_Prov provider started with
> > result code 0x0. HostProcess = wmiprvse.exe; ProcessID = 12260;
> > ProviderPath =
> > %SystemRoot%\\system32\\tscfgwmi.dll","Opcode":"Info","EventReceivedTime":"2013-09-09
> > 17:15:51","SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog"}
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. Consolidate legacy IT systems to a single system of record for IT
> 2. Standardize and globalize service processes across IT
> 3. Implement zero-touch automation to replace manual, redundant tasks
> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
> _______________________________________________
> nxlog-ce-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
>
>
>
> --
>
> ---
> James Crowley
> CTO, FundApps - a new generation in financial services software -
> http://www.fundapps.co/
> Founder, developerFusion - the global developer community -
> http://www.developerfusion.com/
>
> linkedin: http://linkedin.com/in/jamescrowley
> twitter: http://twitter.com/jamescrowley
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. Consolidate legacy IT systems to a single system of record for IT
> 2. Standardize and globalize service processes across IT
> 3. Implement zero-touch automation to replace manual, redundant tasks
> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk_______________________________________________
> nxlog-ce-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. Consolidate legacy IT systems to a single system of record for IT
2. Standardize and globalize service processes across IT
3. Implement zero-touch automation to replace manual, redundant tasks
http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
_______________________________________________
nxlog-ce-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
