Re: Syslog Message Normalization

2019-12-05 Thread Matt Hogstrom 
Thanks all, this is helpful.  The second part is to parse the message apart 
into elements for some analysis.  I’ll process through the suggestions.

Matt Hogstrom
m...@hogstrom.org
+1-919-656-0564
PGP Key: 0x90ECB270
Facebook <https://facebook.com/matt.hogstrom>  LinkedIn 
<https://linkedin/in/mhogstrom>  Twitter <https://twitter.com/hogstrom>

“It may be cognitive, but, it ain’t intuitive."
— Hogstrom

> On Dec 5, 2019, at 1:15 PM, Leonardo Vaz  wrote:
> 
> If you are writing to OPERLOG I consider using that, for each record, you'll 
> have the type (Single-line message, First line of a multi-line message, Data 
> line of a multi-line message, Data end line of a multi-line message) as 
> well as the multiline ID to guarantee you are not mixing lines.
> 
> IBM has a sample to read the operlog on SAMPLIB(IEAMDBLG)
> 
> Regards,
> Leonardo Vaz
> 
> -Original Message-
> From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On 
> Behalf Of Vernooij, Kees (ITOP NM) - KLM
> Sent: Thursday, December 05, 2019 3:45 AM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Syslog Message Normalization
> 
> I think there is a difference in multi-line messages (where each line is 
> identified by an id, '808' in the example below) and long messages that are 
> spread in syslog over more than 1 line. I think your example belongs to the 
> latter.
> 
> MR000 MVSC 19339 09:30:47.49 ACTWRK02   .HASP003 RC=(52),D 
> 808
> DR808   .HASP003 RC=(52),D 
> JOBQ  - NO SELECTABLE ENTRIES FOUND MATCHING   
> ER808   .HASP003   
> SPECIFICATION  
> 
> Kees.
> 
> -Original Message-
> From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On 
> Behalf Of Matt Hogstrom
> Sent: 05 December 2019 02:28
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Syslog Message Normalization
> 
> I’m processing syslog messages and I’d like to combine multi-line messages 
> into a single entry before processing the entries.  For instance, these 
> messages
> 
> N 002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND PROGRAM 
> FOR 02 SECONDS. ===
> N 0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER   
> 00  4  0  0.20  0.000.0   
> S31  JES2 
> 0 0 0 0
> 
> 
> Would become
> 002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND PROGRAM 
> FOR 02 SECONDS. ===
> 0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER   
> 00  4  0  0.20  0.000.031  JES2 0 0   
>   0 0
> 
> Given there are a number of subtle rules I was wondering if anyone had 
> written or was aware of a general purpose normalizer.
> 
> 
> Matt Hogstrom
> m...@hogstrom.org
> +1-919-656-0564
> PGP Key: 0x90ECB270
> Facebook <https://facebook.com/matt.hogstrom>  LinkedIn 
> <https://linkedin/in/mhogstrom>  Twitter <https://twitter.com/hogstrom>
> 
> “It may be cognitive, but, it ain’t intuitive."
> — Hogstrom
> 
> 
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
> lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> 
> For information, services and offers, please visit our web site: 
> http://www.klm.com. This e-mail and any attachment may contain confidential 
> and privileged material intended for the addressee only. If you are not the 
> addressee, you are notified that no part of the e-mail or any attachment may 
> be disclosed, copied or distributed, and that any other action related to 
> this e-mail or attachment is strictly prohibited, and may be unlawful. If you 
> have received this e-mail by error, please notify the sender immediately by 
> return e-mail, and delete this message.
> 
> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its 
> employees shall not be liable for the incorrect or incomplete transmission of 
> this e-mail or any attachments, nor responsible for any delay in receipt.
> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch 
> Airlines) is registered in Amstelveen, The Netherlands, with registered 
> number 33014286
> 
> 
> 
> --
&

Re: Syslog Message Normalization

2019-12-05 Thread Leonardo Vaz 
If you are writing to OPERLOG I consider using that, for each record, you'll 
have the type (Single-line message, First line of a multi-line message, Data 
line of a multi-line message, Data end line of a multi-line message) as 
well as the multiline ID to guarantee you are not mixing lines.

IBM has a sample to read the operlog on SAMPLIB(IEAMDBLG)

Regards,
Leonardo Vaz

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf 
Of Vernooij, Kees (ITOP NM) - KLM
Sent: Thursday, December 05, 2019 3:45 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Syslog Message Normalization

I think there is a difference in multi-line messages (where each line is 
identified by an id, '808' in the example below) and long messages that are 
spread in syslog over more than 1 line. I think your example belongs to the 
latter.

MR000 MVSC 19339 09:30:47.49 ACTWRK02   .HASP003 RC=(52),D 808  
  
DR808   .HASP003 RC=(52),D JOBQ 
 - NO SELECTABLE ENTRIES FOUND MATCHING   
ER808   .HASP003   
SPECIFICATION  

Kees.

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf 
Of Matt Hogstrom
Sent: 05 December 2019 02:28
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Syslog Message Normalization

I’m processing syslog messages and I’d like to combine multi-line messages into 
a single entry before processing the entries.  For instance, these messages

N 002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND PROGRAM 
FOR 02 SECONDS. ===
N 0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER   
00  4  0  0.20  0.000.0   
S31  JES2 0 
0 0 0


Would become
002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND PROGRAM FOR 
02 SECONDS. ===
0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER   00  
4  0  0.20  0.000.031  JES2 0 0 0   
  0

Given there are a number of subtle rules I was wondering if anyone had written 
or was aware of a general purpose normalizer.


Matt Hogstrom
m...@hogstrom.org
+1-919-656-0564
PGP Key: 0x90ECB270
Facebook <https://facebook.com/matt.hogstrom>  LinkedIn 
<https://linkedin/in/mhogstrom>  Twitter <https://twitter.com/hogstrom>

“It may be cognitive, but, it ain’t intuitive."
— Hogstrom


--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

For information, services and offers, please visit our web site: 
http://www.klm.com. This e-mail and any attachment may contain confidential and 
privileged material intended for the addressee only. If you are not the 
addressee, you are notified that no part of the e-mail or any attachment may be 
disclosed, copied or distributed, and that any other action related to this 
e-mail or attachment is strictly prohibited, and may be unlawful. If you have 
received this e-mail by error, please notify the sender immediately by return 
e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its 
employees shall not be liable for the incorrect or incomplete transmission of 
this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch 
Airlines) is registered in Amstelveen, The Netherlands, with registered number 
33014286



--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Syslog Message Normalization

2019-12-05 Thread Sri h Kolusu 
I’m processing syslog messages and I’d like to combine multi-line messages
into a single entry before processing the entries.

Matt,

DFSORT has the capability of combining multiple lines into a single line
using WHEN=GROUP.   You can easily identify the continuation line as it
will NOT have the Timestamp at position 21 ( assuming your syslog dataset
is FBM 133 LRECL) . You validate that value and push the records and
finally create 1 single line of data. I already have a working example of
combining up to 20 lines into a single record. We can automate this job to
combine up to 999 records also.

If you are interested I can post the solution if you let me know

1. Your Syslog DCB properties (LRECL, RECFM)
2. Maximum number of continuation lines that you want to combine (2 to 999)
3. Do you want the intermittent spaces to be squeezed out and have the data
as a continuous text?

Thanks,
Kolusu
DFSORT Development
IBM Corporation

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Syslog Message Normalization

2019-12-05 Thread Vernooij, Kees (ITOP NM) - KLM 
I think there is a difference in multi-line messages (where each line is 
identified by an id, '808' in the example below) and long messages that are 
spread in syslog over more than 1 line. I think your example belongs to the 
latter.

MR000 MVSC 19339 09:30:47.49 ACTWRK02   .HASP003 RC=(52),D 808  
  
DR808   .HASP003 RC=(52),D JOBQ 
 - NO SELECTABLE ENTRIES FOUND MATCHING   
ER808   .HASP003   
SPECIFICATION  

Kees.

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf 
Of Matt Hogstrom
Sent: 05 December 2019 02:28
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Syslog Message Normalization

I’m processing syslog messages and I’d like to combine multi-line messages into 
a single entry before processing the entries.  For instance, these messages

N 002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND PROGRAM 
FOR 02 SECONDS. ===
N 0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER   
00  4  0  0.20  0.000.0   
S31  JES2 0 
0 0 0


Would become
002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND PROGRAM FOR 
02 SECONDS. ===
0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER   00  
4  0  0.20  0.000.031  JES2 0 0 0   
  0

Given there are a number of subtle rules I was wondering if anyone had written 
or was aware of a general purpose normalizer.


Matt Hogstrom
m...@hogstrom.org
+1-919-656-0564
PGP Key: 0x90ECB270
Facebook <https://facebook.com/matt.hogstrom>  LinkedIn 
<https://linkedin/in/mhogstrom>  Twitter <https://twitter.com/hogstrom>

“It may be cognitive, but, it ain’t intuitive."
— Hogstrom


--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

For information, services and offers, please visit our web site: 
http://www.klm.com. This e-mail and any attachment may contain confidential and 
privileged material intended for the addressee only. If you are not the 
addressee, you are notified that no part of the e-mail or any attachment may be 
disclosed, copied or distributed, and that any other action related to this 
e-mail or attachment is strictly prohibited, and may be unlawful. If you have 
received this e-mail by error, please notify the sender immediately by return 
e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its 
employees shall not be liable for the incorrect or incomplete transmission of 
this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch 
Airlines) is registered in Amstelveen, The Netherlands, with registered number 
33014286



--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Syslog Message Normalization

2019-12-05 Thread Vernooij, Kees (ITOP NM) - KLM 
No, 31 is the continuation of the message text. The id is always 3 digits and 
is more to the left. "808" in the example below.


MR000 MVSC 19339 09:30:47.49 ACTWRK02   .HASP003 RC=(52),D 808  
  
DR808   .HASP003 RC=(52),D JOBQ 
 - NO SELECTABLE ENTRIES FOUND MATCHING   
ER808   .HASP003   
SPECIFICATION  

Met vriendelijke groet,
Kees Vernooij
KLM Information Services
z/OS Systems
Tel +31 6 10 14 58 78


-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf 
Of ITschak Mugzach
Sent: 05 December 2019 05:26
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Syslog Message Normalization

"31" in your sample is the correlation id of the original message. I can't see 
it in the original line in your sample, but it is there. it is not part of the 
message and you have to drop it from the concatenated line.

ITschak

On Thu, Dec 5, 2019 at 3:28 AM Matt Hogstrom  wrote:

> I’m processing syslog messages and I’d like to combine multi-line 
> messages into a single entry before processing the entries.  For 
> instance, these messages
>
> N 002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND
> PROGRAM FOR 02 SECONDS. ===
> N 0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER
>00  4  0  0.20  0.000.0
> S31  JES2
>0 0 0 0
>
>
> Would become
> 002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND PROGRAM
> FOR 02 SECONDS. ===
> 0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER
>  00  4  0  0.20  0.000.031  JES2 0
>  0 0 0
>
> Given there are a number of subtle rules I was wondering if anyone had 
> written or was aware of a general purpose normalizer.
>
>
> Matt Hogstrom
> m...@hogstrom.org
> +1-919-656-0564
> PGP Key: 0x90ECB270
> Facebook <https://facebook.com/matt.hogstrom>  LinkedIn < 
> https://linkedin/in/mhogstrom>  Twitter <https://twitter.com/hogstrom>
>
> “It may be cognitive, but, it ain’t intuitive."
> — Hogstrom
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send 
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>


--
ITschak Mugzach
*|** IronSphere Platform* *|* *Information Security Contiguous Monitoring for 
Legacy **|  *

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

For information, services and offers, please visit our web site: 
http://www.klm.com. This e-mail and any attachment may contain confidential and 
privileged material intended for the addressee only. If you are not the 
addressee, you are notified that no part of the e-mail or any attachment may be 
disclosed, copied or distributed, and that any other action related to this 
e-mail or attachment is strictly prohibited, and may be unlawful. If you have 
received this e-mail by error, please notify the sender immediately by return 
e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its 
employees shall not be liable for the incorrect or incomplete transmission of 
this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch 
Airlines) is registered in Amstelveen, The Netherlands, with registered number 
33014286



--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Syslog Message Normalization

2019-12-04 Thread ITschak Mugzach 
"31" in your sample is the correlation id of the original message. I can't
see it in the original line in your sample, but it is there. it is not part
of the message and you have to drop it from the concatenated line.

ITschak

On Thu, Dec 5, 2019 at 3:28 AM Matt Hogstrom  wrote:

> I’m processing syslog messages and I’d like to combine multi-line messages
> into a single entry before processing the entries.  For instance, these
> messages
>
> N 002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND
> PROGRAM FOR 02 SECONDS. ===
> N 0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER
>00  4  0  0.20  0.000.0
> S31  JES2
>0 0 0 0
>
>
> Would become
> 002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND PROGRAM
> FOR 02 SECONDS. ===
> 0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER
>  00  4  0  0.20  0.000.031  JES2 0
>  0 0 0
>
> Given there are a number of subtle rules I was wondering if anyone had
> written or was aware of a general purpose normalizer.
>
>
> Matt Hogstrom
> m...@hogstrom.org
> +1-919-656-0564
> PGP Key: 0x90ECB270
> Facebook   LinkedIn <
> https://linkedin/in/mhogstrom>  Twitter 
>
> “It may be cognitive, but, it ain’t intuitive."
> — Hogstrom
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>


-- 
ITschak Mugzach
*|** IronSphere Platform* *|* *Information Security Contiguous Monitoring
for Legacy **|  *

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Syslog Message Normalization

2019-12-04 Thread Matt Hogstrom 
I’m processing syslog messages and I’d like to combine multi-line messages into 
a single entry before processing the entries.  For instance, these messages

N 002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND PROGRAM 
FOR 02 SECONDS. ===
N 0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER   
00  4  0  0.20  0.000.0   
S31  JES2 0 
0 0 0


Would become
002 PROD 19111 16:00:40.08 JOB08657 0090  +=== SUSPEND PROGRAM FOR 
02 SECONDS. ===
0004000 PROD 19111 16:00:40.08 JOB08657 0290  -STIMER   00  
4  0  0.20  0.000.031  JES2 0 0 0   
  0

Given there are a number of subtle rules I was wondering if anyone had written 
or was aware of a general purpose normalizer.


Matt Hogstrom
m...@hogstrom.org
+1-919-656-0564
PGP Key: 0x90ECB270
Facebook   LinkedIn 
  Twitter 

“It may be cognitive, but, it ain’t intuitive."
— Hogstrom


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN