Re: thought: z/OS structured logging

2014-12-06 Thread Scott Ford
When you build hardware its logical to try to sell it or push it for solutions ...does mean you have to like it or be involved if you don't need to or want to Scott ford www.identityforge.com from my IPAD On Dec 5, 2014, at 11:25 AM, David Crayford dcrayf...@gmail.com wrote: On

Re: thought: z/OS structured logging

2014-12-06 Thread Rob Schramm
Not really sure that the format matters if you are going to have a self described layout/format/etc. Sure. Basic legibility is good.. But having XML json (or something similar) plus a general engine for massive searches.. Not to get on a band wagon..but Hadoop would do the trick. ;-) Rob

Re: thought: z/OS structured logging

2014-12-06 Thread Shmuel Metz (Seymour J.)
In CAAJSdjib6t_m-9iKZOshxHvXBz0=683tfnt-hu4jxfj01wg...@mail.gmail.com, on 12/05/2014 at 07:22 AM, John McKown john.archie.mck...@gmail.com said: Hum, I was thinking more of the UNIX syslog daemon stuff. Which is harder to parse than, e.g., SMF. Wouldn't including both UTC local time in

Re: thought: z/OS structured logging

2014-12-06 Thread Charles Mills
- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Shmuel Metz (Seymour J.) Sent: Saturday, December 06, 2014 3:49 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: thought: z/OS structured logging In CAAJSdjib6t_m-9iKZOshxHvXBz0=683tfnt-hu4jxfj01wg

Re: thought: z/OS structured logging

2014-12-05 Thread Shmuel Metz (Seymour J.)
In caajsdjhqt77sx8xckcezrekvfbmgfo1vouxyd7w7zmn_prh...@mail.gmail.com, on 12/04/2014 at 08:06 AM, John McKown john.archie.mck...@gmail.com said: But I've been thinking about the z/OS syslog for some reason lately. Given what it was originally designed for, review by a human, it is a decent

Re: thought: z/OS structured logging

2014-12-05 Thread John McKown
On Thu, Dec 4, 2014 at 7:57 PM, Shmuel Metz (Seymour J.) shmuel+ibm-m...@patriot.net wrote: ​snip​ And I will admit that my mind has been corrupted by using Linux too much lately. Isn't the Linux equivalent to, e.g., SMF, harder to parse? ​Hum, I was thinking more of the UNIX syslog daemon

Re: thought: z/OS structured logging

2014-12-05 Thread Art Celestini
vendor pitch You might want to take a look at IronStream from Syncsort: (http://www.syncsort.com/en/Solutions/Mainframe-Solutions/Ironstream It captures SYSLOG/OPERLOG messages in real time and sends them to a Splunk server (http://www.splunk.com/) where you can search and report based on

Re: thought: z/OS structured logging

2014-12-05 Thread John McKown
On Fri, Dec 5, 2014 at 7:25 AM, Art Celestini ibmm...@celestini.com wrote: vendor pitch You might want to take a look at IronStream from Syncsort: (http://www.syncsort.com/en/Solutions/Mainframe-Solutions/Ironstream It captures SYSLOG/OPERLOG messages in real time and sends them to a Splunk

Re: thought: z/OS structured logging

2014-12-05 Thread Staller, Allan
snip I really like some of the new centralized logging systems like http://logstash.net/. It can handle loads of different sources and sinks and when you throw in the full power of elasticsearch searching for interesting data is an order of magnitude more powerful then what we currently have on

Re: thought: z/OS structured logging

2014-12-05 Thread David Crayford
On 5/12/2014 11:23 PM, Staller, Allan wrote: snip I really like some of the new centralized logging systems like http://logstash.net/. It can handle loads of different sources and sinks and when you throw in the full power of elasticsearch searching for interesting data is an order of

Re: thought: z/OS structured logging

2014-12-05 Thread Staller, Allan
By the way, ISTR the HP product was OPEN-VIEW. There were others players besides CA and HP in the space as well. All of the products were predicated on a single log of events (not processed on z hardware), and were doing more than just logging/storing messages. As the number of messages

Re: thought: z/OS structured logging

2014-12-05 Thread John Gilmore
David Crayford wrote: begin extract I know it's heresy on this list, but in the distributed world they would just add another server and/or add more grunt to the network. /end extract The granularity of mainframes is of course greater, but additional storage and CPEs are available. This

Re: thought: z/OS structured logging

2014-12-05 Thread David Crayford
On 6/12/2014 12:08 AM, John Gilmore wrote: David Crayford wrote: begin extract I know it's heresy on this list, but in the distributed world they would just add another server and/or add more grunt to the network. /end extract The granularity of mainframes is of course greater, but additional

thought: z/OS structured logging

2014-12-04 Thread John McKown
This is just my mind wandering around loose again. You kind indulgence is appreciated. But I've been thinking about the z/OS syslog for some reason lately. Given what it was originally designed for, review by a human, it is a decent design. But is it really as helpful as it could be in today's

Re: thought: z/OS structured logging

2014-12-04 Thread Doug Henry
Hi John, While I am sure this not exactly what you are dreaming about we do run IBM's z/Aware with does consume operlogs and provides an api that uses xml. http://www-01.ibm.com/support/docview.wss?uid=isg24f9114255d7d1f3285257a6a0077c2ca Doug On Thu, 4 Dec 2014 08:06:33 -0600, John McKown

Re: thought: z/OS structured logging

2014-12-04 Thread Elardus Engelbrecht
John McKown wrote: This is just my mind wandering around loose again. Catch it! Catch it! ;-) ( There is an old Afrikaans song 'Catch it!' which says how difficult is it to catch chicken/pig/sheep/etc. ;-D ) But I've been thinking about the z/OS syslog for some reason lately. Given what it

Re: thought: z/OS structured logging

2014-12-04 Thread John McKown
On Thu, Dec 4, 2014 at 8:44 AM, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: ​snip​ And I will admit that my mind has been corrupted by using Linux too much lately. grin/ Please refresh my mind about how is Linux version of log(s) working? ​Well, the normal UNIX syslogd data

Re: thought: z/OS structured logging

2014-12-04 Thread Joel Ewing
While conceptually XML sounds nice, the problem would seem to be the extreme volume of data involved, millions of messages daily for large installations. Uncompressed XML is incredibly inefficient in storage requirements, and compressing/uncompressing XML has processing costs. From my viewpoint I

Re: thought: z/OS structured logging

2014-12-04 Thread Charles Mills
, 2014 9:16 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: thought: z/OS structured logging On Thu, Dec 4, 2014 at 8:44 AM, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: ​snip​ And I will admit that my mind has been corrupted by using Linux too much lately. grin/ Please refresh

Re: thought: z/OS structured logging

2014-12-04 Thread R.S.
My €0.02 The idea of structured logging smells like WIndows Event Log which I hate deeply. JSON ans XML are the format which I like in similar manner like the above. For human and script-powered review we have syslog. Of structured logging we have SMF which can be exported to XML if someone

Re: thought: z/OS structured logging

2014-12-04 Thread David Crayford
I really like some of the new centralized logging systems like http://logstash.net/. It can handle loads of different sources and sinks and when you throw in the full power of elasticsearch searching for interesting data is an order of magnitude more powerful then what we currently have on

Re: thought: z/OS structured logging

2014-12-04 Thread Shane Ginnane
On Fri, 5 Dec 2014 11:15:58 +0800, David Crayford dcrayf...@gmail.com wrote: I really like some of the new centralized logging systems like http://logstash.net/. It can handle loads of different sources and sinks and when you throw in the full power of elasticsearch searching for interesting data

Re: thought: z/OS structured logging

2014-12-04 Thread David Crayford
On 5/12/2014 2:30 PM, Shane Ginnane wrote: Yeah, but how do you get it there ?. Sysadmins want everything*now*. Not tomorrow after syslog has been archived off. Takes us back to the tail syslog thread last month. Surely the Netview (or whatever) developers must be able to knock up code to push