Re: thought: z/OS structured logging
When you build hardware its logical to try to sell it or push it for solutions ...does mean you have to like it or be involved if you don't need to or want to Scott ford www.identityforge.com from my IPAD On Dec 5, 2014, at 11:25 AM, David Crayford dcrayf...@gmail.com wrote: On 6/12/2014 12:08 AM, John Gilmore wrote: David Crayford wrote: begin extract I know it's heresy on this list, but in the distributed world they would just add another server and/or add more grunt to the network. /end extract The granularity of mainframes is of course greater, but additional storage and CPEs are available. Yes, and they're very expensive! This strategy, that of throwing hardware at problems, has been around in the mainframe world for decades. IBM salesmen used to call the customer CIOs who used it routinely hardware hawks, perhaps still do. They were much appreciated, though not much respected. IBM salesmen will be queuing up to sell you an IBM SmartCloud Analytics solution for System z that doesn't actually run on system z. Well, maybe the agent does. It's all going off host these days and that makes sense. You only want to pay $$$ for running your legacy COBOL apps. John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
Not really sure that the format matters if you are going to have a self described layout/format/etc. Sure. Basic legibility is good.. But having XML json (or something similar) plus a general engine for massive searches.. Not to get on a band wagon..but Hadoop would do the trick. ;-) Rob Schramm On Dec 4, 2014 2:38 PM, R.S. r.skoru...@bremultibank.com.pl wrote: My €0.02 The idea of structured logging smells like WIndows Event Log which I hate deeply. JSON ans XML are the format which I like in similar manner like the above. For human and script-powered review we have syslog. Of structured logging we have SMF which can be exported to XML if someone want it. -- Radoslaw Skorupka Lodz, Poland --- Treść tej wiadomości może zawierać informacje prawnie chronione Banku przeznaczone wyłącznie do użytku służbowego adresata. Odbiorcą może być jedynie jej adresat z wyłączeniem dostępu osób trzecich. Jeżeli nie jesteś adresatem niniejszej wiadomości lub pracownikiem upoważnionym do jej przekazania adresatowi, informujemy, że jej rozpowszechnianie, kopiowanie, rozprowadzanie lub inne działanie o podobnym charakterze jest prawnie zabronione i może być karalne. Jeżeli otrzymałeś tę wiadomość omyłkowo, prosimy niezwłocznie zawiadomić nadawcę wysyłając odpowiedź oraz trwale usunąć tę wiadomość włączając w to wszelkie jej kopie wydrukowane lub zapisane na dysku. This e-mail may contain legally privileged information of the Bank and is intended solely for business use of the addressee. This e-mail may only be received by the addressee and may not be disclosed to any third parties. If you are not the intended addressee of this e-mail or the employee authorized to forward it to the addressee, be advised that any dissemination, copying, distribution or any other similar activity is legally prohibited and may be punishable. If you received this e-mail by mistake please advise the sender immediately by using the reply facility in your e-mail software and delete permanently this e-mail including any copies of it either printed or saved to hard drive. mBank S.A. z siedzibą w Warszawie, ul. Senatorska 18, 00-950 Warszawa, www.mBank.pl, e-mail: kont...@mbank.pl Sąd Rejonowy dla m. st. Warszawy XII Wydział Gospodarczy Krajowego Rejestru Sądowego, nr rejestru przedsiębiorców KRS 025237, NIP: 526-021-50-88. Według stanu na dzień 01.01.2014 r. kapitał zakładowy mBanku S.A. (w całości wpłacony) wynosi 168.696.052 złote. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
In CAAJSdjib6t_m-9iKZOshxHvXBz0=683tfnt-hu4jxfj01wg...@mail.gmail.com, on 12/05/2014 at 07:22 AM, John McKown john.archie.mck...@gmail.com said: Hum, I was thinking more of the UNIX syslog daemon stuff. Which is harder to parse than, e.g., SMF. Wouldn't including both UTC local time in ISO8601 be redundant? Only if ISO 8601 requires including the offset. Also, how does ISO 8601 handle fractional time zones? But perhaps, since you like binary, it would be better to make this the STCKE value and the CVTLDTO field, and maybe the CVTLSO field as well. What is important is that the information be recorded and recorded in a consistent fashion. -- Shmuel (Seymour J.) Metz, SysProg and JOAT ISO position; see http://patriot.net/~shmuel/resume/brief.html We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
Which is harder to parse than, e.g., SMF. Depending on your background. You and I think fixed-length binary and character fields cheek-to-jowl are peachy-keen; the UNIX-y folks are appalled and want something in delimited character form and more-or-less human-readable. Only if ISO 8601 requires including the offset. Also, how does ISO 8601 handle fractional time zones? ISO 8601 does not *require* an offset but it does *permit* the designator Z (for Zulu=UTC) or an offset. Offsets are specified in hours and minutes. (No provision for fractional minutes g.) Charles -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Shmuel Metz (Seymour J.) Sent: Saturday, December 06, 2014 3:49 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: thought: z/OS structured logging In CAAJSdjib6t_m-9iKZOshxHvXBz0=683tfnt-hu4jxfj01wg...@mail.gmail.com, on 12/05/2014 at 07:22 AM, John McKown john.archie.mck...@gmail.com said: Hum, I was thinking more of the UNIX syslog daemon stuff. Which is harder to parse than, e.g., SMF. Wouldn't including both UTC local time in ISO8601 be redundant? Only if ISO 8601 requires including the offset. Also, how does ISO 8601 handle fractional time zones? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
In caajsdjhqt77sx8xckcezrekvfbmgfo1vouxyd7w7zmn_prh...@mail.gmail.com, on 12/04/2014 at 08:06 AM, John McKown john.archie.mck...@gmail.com said: But I've been thinking about the z/OS syslog for some reason lately. Given what it was originally designed for, review by a human, it is a decent design. But is it really as helpful as it could be in today's z/OS environment? Should z/OS have a more generalized logging facility? It already does. The syslog is intended solely to capture console[1] messages; it is not and never has been a general event log. What is relevant is facilities like CT, GTF, LOGREC and SMF. Now those have issues, but they are a lot closer to general than syslog was ever intended to be. Is there really a need for the z/OS system log anymore? Yes. And I will admit that my mind has been corrupted by using Linux too much lately. Isn't the Linux equivalent to, e.g., SMF, harder to parse? So, if such a thing is even needed any more, what might it look like? IMHO a new generalized event log, replacing SMF et all, should timestamp in both global and local time, possibly UTC plus offset. It definitely should support LOGGER. I know most will moan, Indeed. but I _like_ structured, textual, information. I hate it. I prefer well structured binary. And no encoded binary, OK? Not okay. program name in the RB ITYM CDE or LPDE pointede to by the RB. JS TCB What if the event is logged from an SRB? How should character data be encoded? EBCDIC would be reasonable if there was only a single EBCDIC code page, but as it is UTF-8 might be better. -- Shmuel (Seymour J.) Metz, SysProg and JOAT ISO position; see http://patriot.net/~shmuel/resume/brief.html We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
On Thu, Dec 4, 2014 at 7:57 PM, Shmuel Metz (Seymour J.) shmuel+ibm-m...@patriot.net wrote: snip And I will admit that my mind has been corrupted by using Linux too much lately. Isn't the Linux equivalent to, e.g., SMF, harder to parse? Hum, I was thinking more of the UNIX syslog daemon stuff. I've never touched the accounting information. I don't think that many do in UNIX. So, if such a thing is even needed any more, what might it look like? IMHO a new generalized event log, replacing SMF et all, should timestamp in both global and local time, possibly UTC plus offset. It definitely should support LOGGER. Wouldn't including both UTC local time in ISO8601 be redundant? But perhaps, since you like binary, it would be better to make this the STCKE value and the CVTLDTO field, and maybe the CVTLSO field as well. I know most will moan, Indeed. but I _like_ structured, textual, information. I hate it. I prefer well structured binary. Well, the reason for my preference for textual data is the unfortunate fact that where I am now has _NO_ tools on the z/OS platform for data reduction and reporting. Well, other that what I personally code in HLASM, REXX, or COBOL. When all you have is a hammer, everything looks like a nail. But I don't even have a hammer on z/OS, I only have a rock. Hammers are too high tech, expensive, and unnecessary. (not grinning) program name in the RB ITYM CDE or LPDE pointede to by the RB. Yes, I guess I should have said associated with rather than pointed to. Poor choice of words on my part. JS TCB What if the event is logged from an SRB? Hum, I don't do much with SRBs personally so that just slipped past me. How should character data be encoded? EBCDIC would be reasonable if there was only a single EBCDIC code page, but as it is UTF-8 might be better. OK, I was hoping for something that I could ftp to my Linux box _easily_ (which is why I prefer no binary), which I have set up using the en_US.UTF8 locale. So UTF-8 would be nice. I'm just not sure of the translation from EBCDIC to UTF-8. Perhaps we will agree _NOT_ use UTF-EBCDIC? -- Shmuel (Seymour J.) Metz, SysProg and JOAT ISO position; see http://patriot.net/~shmuel/resume/brief.html -- The temperature of the aqueous content of an unremittingly ogled culinary vessel will not achieve 100 degrees on the Celsius scale. Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
vendor pitch You might want to take a look at IronStream from Syncsort: (http://www.syncsort.com/en/Solutions/Mainframe-Solutions/Ironstream It captures SYSLOG/OPERLOG messages in real time and sends them to a Splunk server (http://www.splunk.com/) where you can search and report based on keyword, message number, etc. IronStream also supports the capture of SMF records in real time, and interprets (converts to JSON and translates to ASCII) those records before sending them to Splunk. Data can be observed in Splunk within a few seconds of its creation on the mainframe. /vendor pitch Art Celestini Technology Architect SyncSort Inc. At 09:06 AM 12/4/2014, John McKown wrote: This is just my mind wandering around loose again. You kind indulgence is appreciated. But I've been thinking about the z/OS syslog for some reason lately. Given what it was originally designed for, review by a human, it is a decent design. But is it really as helpful as it could be in today's z/OS environment? Should z/OS have a more generalized logging facility? I will grant that subsystems have various logs, but they each basically have their own structure. Is there really a need for the z/OS system log anymore? I really don't know. And I will admit that my mind has been corrupted by using Linux too much lately. grin/ So, if such a thing is even needed any more, what might it look like? Should it go to SPOOL? Should it be more like the OPERLOG and go to a LOGGER destination? Or should it go somewhere else? So what would I like? I know most will moan, but I _like_ structured, textual, information. So I would prefer that the output be in something like XML or JSON structure, not column based. And no encoded binary, OK? Now I'm trying to come up with what sort of data should be in the system header type data. These are just some fields that _I_ think would be useful in a good, generic, logging facility. First would be the current date in ISO8601 format, something like 2014-12-04T07:34:03-06:00 which is the date/time as I am typing this near Dallas, TX. This tells us the local time and gives us enough information to determine the UTC for comparison or conversion. I would also like the z/OS sysplex name, the system name, the CPU serial number, LPAR number, z/VM guest name (if applicable), job name (address space name), RACF owner, step name, proc step name, program name in the RB which issued the logging service call, program name in the first RB chained to the JS TCB (which I think should be the EXEC PGM=... name in most cases for batch jobs), ASID number, UNIX process id (==0 if not dubbed because there is no PID of 0 in UNIX, or maybe -1), step number (as used in SMF), substep number (again as defined in some SMF records). Product specific data would be formally encoded as designed by the product. Preferably, if in XML, with a DTD to describe it. And done so that standard XML facilities such as XSLT and XPath can process it. Which I one reason that I like XML a bit better than JSON at this point in time. There are a lot of XML utilities around. And, lastly, I do realize that the above would be very costly. Not necessarily to just implement into z/OS, but to actually change z/OS code to start using it. And that may be the real killer. IMO, one of the biggest obstructions to designing new facility which enhance existing facilities is the cost of implementing them. This combined with the current emphasis on immediate return on investment. I.e. if I invest a million dollars in something, I expect to get back 2 million in 6 months or less. Well, I guess that I've bored you enough with this bit of weirdness. Like many of my ideas, they sound good to me until others point out that they are just silly/stupid/unnecessary. -- The temperature of the aqueous content of an unremittingly ogled culinary vessel will not achieve 100 degrees on the Celsius scale. Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
On Fri, Dec 5, 2014 at 7:25 AM, Art Celestini ibmm...@celestini.com wrote: vendor pitch You might want to take a look at IronStream from Syncsort: (http://www.syncsort.com/en/Solutions/Mainframe-Solutions/Ironstream It captures SYSLOG/OPERLOG messages in real time and sends them to a Splunk server (http://www.splunk.com/) where you can search and report based on keyword, message number, etc. IronStream also supports the capture of SMF records in real time, and interprets (converts to JSON and translates to ASCII) those records before sending them to Splunk. Data can be observed in Splunk within a few seconds of its creation on the mainframe. /vendor pitch Ah, no money. Nice to know of it, however. We actually have code in CA-OPS/MVS which traps some SYSLOG messages (mostly from CA-7 browse log, via CAGTS) and sends SNMP messages to SolarWinds' Orion (our LAN software for this sort of thing), by using the CAOPSWTO facility. I have actually used this, in testing only, to send SNMP traps to my Linux/Intel desktop. Since CA-OPS/MVS uses REXX, I _could_ reformat the interesting z/OS SYSLOG messages in order to send them, via REXX sockets, to a UNIX syslog daemon. Or as I did in my testing, use CA-OPS/MVS to an SNMP server which could relay to the UNIX syslog daemon. Combine this with Co:Z to send UNIX commands back over SSH to z/OS and I could do weird and wonderful(?) things. Well, except that I'm not allowed because z/OS is going away and management wants it stable and so don't make any changes, possibly because they don't understand what I'm doing. Art Celestini Technology Architect SyncSort Inc. -- The temperature of the aqueous content of an unremittingly ogled culinary vessel will not achieve 100 degrees on the Celsius scale. Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
snip I really like some of the new centralized logging systems like http://logstash.net/. It can handle loads of different sources and sinks and when you throw in the full power of elasticsearch searching for interesting data is an order of magnitude more powerful then what we currently have on z/OS. You can throw your distributed systems into the mix for a nice holistic view of your entire stack. /snip The last time I was involved in this in a serious way, all of the central loggers (Ca-UNICENTER, HP-???,.), regardless of the underlying HW platform, all ran into scaling issues as the quantity of data being processed increased. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
On 5/12/2014 11:23 PM, Staller, Allan wrote: snip I really like some of the new centralized logging systems like http://logstash.net/. It can handle loads of different sources and sinks and when you throw in the full power of elasticsearch searching for interesting data is an order of magnitude more powerful then what we currently have on z/OS. You can throw your distributed systems into the mix for a nice holistic view of your entire stack. /snip The last time I was involved in this in a serious way, all of the central loggers (Ca-UNICENTER, HP-???,.), regardless of the underlying HW platform, all ran into scaling issues as the quantity of data being processed increased. Can you remember what the bottleneck was? When you consider that you can buy a 32-core server with SSD disk for peanuts and network latencies are dramatically shrinking it may not still be the case. I know it's heresy on this list but in the distributed world they would just add another server and/or add more grunt to the network. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
By the way, ISTR the HP product was OPEN-VIEW. There were others players besides CA and HP in the space as well. All of the products were predicated on a single log of events (not processed on z hardware), and were doing more than just logging/storing messages. As the number of messages processed increased, performance degraded (non-linearly and rapidly). There were operating system, application, and hardware bottlenecks, (common to all vendors, just at different choke points). Of course, ours is the best, came from all of the vendors as well. HTH, snip On 5/12/2014 11:23 PM, Staller, Allan wrote: snip I really like some of the new centralized logging systems like http://logstash.net/. It can handle loads of different sources and sinks and when you throw in the full power of elasticsearch searching for interesting data is an order of magnitude more powerful then what we currently have on z/OS. You can throw your distributed systems into the mix for a nice holistic view of your entire stack. /snip The last time I was involved in this in a serious way, all of the central loggers (Ca-UNICENTER, HP-???,.), regardless of the underlying HW platform, all ran into scaling issues as the quantity of data being processed increased. Can you remember what the bottleneck was? When you consider that you can buy a 32-core server with SSD disk for peanuts and network latencies are dramatically shrinking it may not still be the case. I know it's heresy on this list but in the distributed world they would just add another server and/or add more grunt to the network. /snip -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
David Crayford wrote: begin extract I know it's heresy on this list, but in the distributed world they would just add another server and/or add more grunt to the network. /end extract The granularity of mainframes is of course greater, but additional storage and CPEs are available. This strategy, that of throwing hardware at problems, has been around in the mainframe world for decades. IBM salesmen used to call the customer CIOs who used it routinely hardware hawks, perhaps still do. They were much appreciated, though not much respected. John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
On 6/12/2014 12:08 AM, John Gilmore wrote: David Crayford wrote: begin extract I know it's heresy on this list, but in the distributed world they would just add another server and/or add more grunt to the network. /end extract The granularity of mainframes is of course greater, but additional storage and CPEs are available. Yes, and they're very expensive! This strategy, that of throwing hardware at problems, has been around in the mainframe world for decades. IBM salesmen used to call the customer CIOs who used it routinely hardware hawks, perhaps still do. They were much appreciated, though not much respected. IBM salesmen will be queuing up to sell you an IBM SmartCloud Analytics solution for System z that doesn't actually run on system z. Well, maybe the agent does. It's all going off host these days and that makes sense. You only want to pay $$$ for running your legacy COBOL apps. John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
thought: z/OS structured logging
This is just my mind wandering around loose again. You kind indulgence is appreciated. But I've been thinking about the z/OS syslog for some reason lately. Given what it was originally designed for, review by a human, it is a decent design. But is it really as helpful as it could be in today's z/OS environment? Should z/OS have a more generalized logging facility? I will grant that subsystems have various logs, but they each basically have their own structure. Is there really a need for the z/OS system log anymore? I really don't know. And I will admit that my mind has been corrupted by using Linux too much lately. grin/ So, if such a thing is even needed any more, what might it look like? Should it go to SPOOL? Should it be more like the OPERLOG and go to a LOGGER destination? Or should it go somewhere else? So what would I like? I know most will moan, but I _like_ structured, textual, information. So I would prefer that the output be in something like XML or JSON structure, not column based. And no encoded binary, OK? Now I'm trying to come up with what sort of data should be in the system header type data. These are just some fields that _I_ think would be useful in a good, generic, logging facility. First would be the current date in ISO8601 format, something like 2014-12-04T07:34:03-06:00 which is the date/time as I am typing this near Dallas, TX. This tells us the local time and gives us enough information to determine the UTC for comparison or conversion. I would also like the z/OS sysplex name, the system name, the CPU serial number, LPAR number, z/VM guest name (if applicable), job name (address space name), RACF owner, step name, proc step name, program name in the RB which issued the logging service call, program name in the first RB chained to the JS TCB (which I think should be the EXEC PGM=... name in most cases for batch jobs), ASID number, UNIX process id (==0 if not dubbed because there is no PID of 0 in UNIX, or maybe -1), step number (as used in SMF), substep number (again as defined in some SMF records). Product specific data would be formally encoded as designed by the product. Preferably, if in XML, with a DTD to describe it. And done so that standard XML facilities such as XSLT and XPath can process it. Which I one reason that I like XML a bit better than JSON at this point in time. There are a lot of XML utilities around. And, lastly, I do realize that the above would be very costly. Not necessarily to just implement into z/OS, but to actually change z/OS code to start using it. And that may be the real killer. IMO, one of the biggest obstructions to designing new facility which enhance existing facilities is the cost of implementing them. This combined with the current emphasis on immediate return on investment. I.e. if I invest a million dollars in something, I expect to get back 2 million in 6 months or less. Well, I guess that I've bored you enough with this bit of weirdness. Like many of my ideas, they sound good to me until others point out that they are just silly/stupid/unnecessary. -- The temperature of the aqueous content of an unremittingly ogled culinary vessel will not achieve 100 degrees on the Celsius scale. Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
Hi John, While I am sure this not exactly what you are dreaming about we do run IBM's z/Aware with does consume operlogs and provides an api that uses xml. http://www-01.ibm.com/support/docview.wss?uid=isg24f9114255d7d1f3285257a6a0077c2ca Doug On Thu, 4 Dec 2014 08:06:33 -0600, John McKown john.archie.mck...@gmail.com wrote: This is just my mind wandering around loose again. You kind indulgence is appreciated. But I've been thinking about the z/OS syslog for some reason lately. Given what it was originally designed for, review by a human, it is a decent design. But is it really as helpful as it could be in today's z/OS environment? Should z/OS have a more generalized logging facility? I will grant that subsystems have various logs, but they each basically have their own structure. Is there really a need for the z/OS system log anymore? I really don't know. And I will admit that my mind has been corrupted by using Linux too much lately. grin/ So, if such a thing is even needed any more, what might it look like? Should it go to SPOOL? Should it be more like the OPERLOG and go to a LOGGER destination? Or should it go somewhere else? So what would I like? I know most will moan, but I _like_ structured, textual, information. So I would prefer that the output be in something like XML or JSON structure, not column based. And no encoded binary, OK? Now I'm trying to come up with what sort of data should be in the system header type data. These are just some fields that _I_ think would be useful in a good, generic, logging facility. First would be the current date in ISO8601 format, something like 2014-12-04T07:34:03-06:00 which is the date/time as I am typing this near Dallas, TX. This tells us the local time and gives us enough information to determine the UTC for comparison or conversion. I would also like the z/OS sysplex name, the system name, the CPU serial number, LPAR number, z/VM guest name (if applicable), job name (address space name), RACF owner, step name, proc step name, program name in the RB which issued the logging service call, program name in the first RB chained to the JS TCB (which I think should be the EXEC PGM=... name in most cases for batch jobs), ASID number, UNIX process id (==0 if not dubbed because there is no PID of 0 in UNIX, or maybe -1), step number (as used in SMF), substep number (again as defined in some SMF records). Product specific data would be formally encoded as designed by the product. Preferably, if in XML, with a DTD to describe it. And done so that standard XML facilities such as XSLT and XPath can process it. Which I one reason that I like XML a bit better than JSON at this point in time. There are a lot of XML utilities around. And, lastly, I do realize that the above would be very costly. Not necessarily to just implement into z/OS, but to actually change z/OS code to start using it. And that may be the real killer. IMO, one of the biggest obstructions to designing new facility which enhance existing facilities is the cost of implementing them. This combined with the current emphasis on immediate return on investment. I.e. if I invest a million dollars in something, I expect to get back 2 million in 6 months or less. Well, I guess that I've bored you enough with this bit of weirdness. Like many of my ideas, they sound good to me until others point out that they are just silly/stupid/unnecessary. -- The temperature of the aqueous content of an unremittingly ogled culinary vessel will not achieve 100 degrees on the Celsius scale. Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
John McKown wrote: This is just my mind wandering around loose again. Catch it! Catch it! ;-) ( There is an old Afrikaans song 'Catch it!' which says how difficult is it to catch chicken/pig/sheep/etc. ;-D ) But I've been thinking about the z/OS syslog for some reason lately. Given what it was originally designed for, review by a human, it is a decent design. But is it really as helpful as it could be in today's z/OS environment? Not really because all and every usefull and useless messages are thrown in one h*lluva pool. You need a magician to do your searches... Is there really a need for the z/OS system log anymore? Yes. That is as long there is not a good alternative. And I will admit that my mind has been corrupted by using Linux too much lately. grin/ Please refresh my mind about how is Linux version of log(s) working? So, if such a thing is even needed any more, what might it look like? Should it go to SPOOL? Should it be more like the OPERLOG and go to a LOGGER destination? Or should it go somewhere else? Control-D (yeah, I know you want freebies, but...) can do that. It slurps up any pre-defined/selected output from JES2 and store it in its own archives for easy retrieval. So I would prefer that the output be in something like XML or JSON structure, not column based. And no encoded binary, OK? ... etc ... H. Good idea! And, lastly, I do realize that the above would be very costly. Not necessarily to just implement into z/OS, but to actually change z/OS code to start using it. And that may be the real killer. Backward compatibility... is the murderer for that killing. I.e. if I invest a million dollars in something, I expect to get back 2 million in 6 months or less. Why? I want it all for myself. I'll spare you so about 3 cents or so... ;-D Well, I guess that I've bored you enough with this bit of weirdness. Like many of my ideas, they sound good to me until others point out that they are just silly/stupid/unnecessary. No. As usual you come with good ideas. I believe something good will come out if this and your past good ideas. Keep it up! Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
On Thu, Dec 4, 2014 at 8:44 AM, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: snip And I will admit that my mind has been corrupted by using Linux too much lately. grin/ Please refresh my mind about how is Linux version of log(s) working? Well, the normal UNIX syslogd data is very similar to the z/OS SYSLOG in that it is unstructured. But I ran across RFC 5424, https://tools.ietf.org/html/rfc5424, which does not really seem to be in use. But that RFC inspired me to think about Advanced Message Logging for z/OS (to give it a marketing-type name). One plus of this would be that it would be much simpler to extract data from a structured data stream for automation purposes. And perhaps to store the log information in a DB2 data base using DB2 pureXML, if XML is the structure format of choice. I do like XML. It is wordy, especially compared to JSON, but the tools already available can be quite nice. -- The temperature of the aqueous content of an unremittingly ogled culinary vessel will not achieve 100 degrees on the Celsius scale. Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
While conceptually XML sounds nice, the problem would seem to be the extreme volume of data involved, millions of messages daily for large installations. Uncompressed XML is incredibly inefficient in storage requirements, and compressing/uncompressing XML has processing costs. From my viewpoint I would be much more concerned about the ongoing and continual additional overhead costs an XML Syslog would add to the daily operation of a z/OS shop, including archival and search costs, not just the implementation costs. System automation tools that are common to many z/OS shops already can already incur significant overhead in processing system log records, so one would not want to add to that by significantly increasing the size or processing cost of reading log records. While it would be nice to have an easy way of associating things like Sysplex, System, and LPAR names, etc. with messages, you really need to do this in a way that doesn't replicate what for a given system is essentially constant information on millions of log entry records. At times when I have had to search through system logs, I have given thought about how much easier it might be if they were structured into a relational database, but obviously that would not a practical design for direct recording of SYSLOG as one of the roles of SYSLOG is to track z/OS startup and failures affecting the z/OS relational database servers. There is a basic design conflict inherent with the existing SYSLOG between making a record terse enough for efficient archival and message automation versus creating a human-readable message for consoles and batch job logs. As long as the SYSLOG is also allowed to contain messages generated by local installation code and local application code, all rules on structure of messages become unenforceable by z/OS. Not sure how to get around that. Joel C. Ewing On 12/04/2014 08:06 AM, John McKown wrote: This is just my mind wandering around loose again. You kind indulgence is appreciated. But I've been thinking about the z/OS syslog for some reason lately. Given what it was originally designed for, review by a human, it is a decent design. But is it really as helpful as it could be in today's z/OS environment? Should z/OS have a more generalized logging facility? I will grant that subsystems have various logs, but they each basically have their own structure. Is there really a need for the z/OS system log anymore? I really don't know. And I will admit that my mind has been corrupted by using Linux too much lately. grin/ So, if such a thing is even needed any more, what might it look like? Should it go to SPOOL? Should it be more like the OPERLOG and go to a LOGGER destination? Or should it go somewhere else? So what would I like? I know most will moan, but I _like_ structured, textual, information. So I would prefer that the output be in something like XML or JSON structure, not column based. And no encoded binary, OK? Now I'm trying to come up with what sort of data should be in the system header type data. These are just some fields that _I_ think would be useful in a good, generic, logging facility. First would be the current date in ISO8601 format, something like 2014-12-04T07:34:03-06:00 which is the date/time as I am typing this near Dallas, TX. This tells us the local time and gives us enough information to determine the UTC for comparison or conversion. I would also like the z/OS sysplex name, the system name, the CPU serial number, LPAR number, z/VM guest name (if applicable), job name (address space name), RACF owner, step name, proc step name, program name in the RB which issued the logging service call, program name in the first RB chained to the JS TCB (which I think should be the EXEC PGM=... name in most cases for batch jobs), ASID number, UNIX process id (==0 if not dubbed because there is no PID of 0 in UNIX, or maybe -1), step number (as used in SMF), substep number (again as defined in some SMF records). Product specific data would be formally encoded as designed by the product. Preferably, if in XML, with a DTD to describe it. And done so that standard XML facilities such as XSLT and XPath can process it. Which I one reason that I like XML a bit better than JSON at this point in time. There are a lot of XML utilities around. And, lastly, I do realize that the above would be very costly. Not necessarily to just implement into z/OS, but to actually change z/OS code to start using it. And that may be the real killer. IMO, one of the biggest obstructions to designing new facility which enhance existing facilities is the cost of implementing them. This combined with the current emphasis on immediate return on investment. I.e. if I invest a million dollars in something, I expect to get back 2 million in 6 months or less. Well, I guess that I've bored you enough with this bit of weirdness. Like many of my ideas, they sound good to me until
Re: thought: z/OS structured logging
Commenting just on the issue of Syslog (in the 'nix sense) structure. No, RFC 5424 is not widely observed. Syslog epitomizes the aphorism about God loves standards -- that's why he created so many of them. About the only Syslog standard that everyone almost follows is RFC 3164, and even that is not followed very well. (And it's not even a standard, that's why I put it in quotes. It describes observed behavior.) Bits and pieces of RFC 5424/25/26/27 are in wide use (TCP/IP, longer messages -- but not the header.) There is a standard for XML Syslog (RFC 3195) but so far as I know *no one* uses it. There have been several common attempts to impose some structure on Syslog, most notably the Common Event Expression endeavor pushed by MITRE. I don't think anyone is paying attention. There are two proprietary structures imposed by a particular vendor with some success: CEF is an ArcSight invention. (ArcSight is now owned by HP.) It imposes a rigorous structure on Syslog messages with a rigid header and specific field tags. ArcSight may be the most widely-implement SIEM. (http://en.wikipedia.org/wiki/Security_information_and_event_management if you are not familiar with the term SIEM; it's what your non-Z brethren think of when you say computer security the way you think of RACF. [But no, they are not analogous in function.]) LEEF is a Q1 Labs invention. Q1 labs is now owned by IBM, and I believe IBM is making good headway with the product, QRadar. LEEF is quite similar to CEF, but not quite so rigid. That's probably enough for a mainframe list. Anyone with questions can reply on- or off-line. Charles -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of John McKown Sent: Thursday, December 04, 2014 9:16 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: thought: z/OS structured logging On Thu, Dec 4, 2014 at 8:44 AM, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: snip And I will admit that my mind has been corrupted by using Linux too much lately. grin/ Please refresh my mind about how is Linux version of log(s) working? Well, the normal UNIX syslogd data is very similar to the z/OS SYSLOG in that it is unstructured. But I ran across RFC 5424, https://tools.ietf.org/html/rfc5424, which does not really seem to be in use. But that RFC inspired me to think about Advanced Message Logging for z/OS (to give it a marketing-type name). One plus of this would be that it would be much simpler to extract data from a structured data stream for automation purposes. And perhaps to store the log information in a DB2 data base using DB2 pureXML, if XML is the structure format of choice. I do like XML. It is wordy, especially compared to JSON, but the tools already available can be quite nice. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
My €0.02 The idea of structured logging smells like WIndows Event Log which I hate deeply. JSON ans XML are the format which I like in similar manner like the above. For human and script-powered review we have syslog. Of structured logging we have SMF which can be exported to XML if someone want it. -- Radoslaw Skorupka Lodz, Poland --- Treść tej wiadomości może zawierać informacje prawnie chronione Banku przeznaczone wyłącznie do użytku służbowego adresata. Odbiorcą może być jedynie jej adresat z wyłączeniem dostępu osób trzecich. Jeżeli nie jesteś adresatem niniejszej wiadomości lub pracownikiem upoważnionym do jej przekazania adresatowi, informujemy, że jej rozpowszechnianie, kopiowanie, rozprowadzanie lub inne działanie o podobnym charakterze jest prawnie zabronione i może być karalne. Jeżeli otrzymałeś tę wiadomość omyłkowo, prosimy niezwłocznie zawiadomić nadawcę wysyłając odpowiedź oraz trwale usunąć tę wiadomość włączając w to wszelkie jej kopie wydrukowane lub zapisane na dysku. This e-mail may contain legally privileged information of the Bank and is intended solely for business use of the addressee. This e-mail may only be received by the addressee and may not be disclosed to any third parties. If you are not the intended addressee of this e-mail or the employee authorized to forward it to the addressee, be advised that any dissemination, copying, distribution or any other similar activity is legally prohibited and may be punishable. If you received this e-mail by mistake please advise the sender immediately by using the reply facility in your e-mail software and delete permanently this e-mail including any copies of it either printed or saved to hard drive. mBank S.A. z siedzibą w Warszawie, ul. Senatorska 18, 00-950 Warszawa, www.mBank.pl, e-mail: kont...@mbank.pl Sąd Rejonowy dla m. st. Warszawy XII Wydział Gospodarczy Krajowego Rejestru Sądowego, nr rejestru przedsiębiorców KRS 025237, NIP: 526-021-50-88. Według stanu na dzień 01.01.2014 r. kapitał zakładowy mBanku S.A. (w całości wpłacony) wynosi 168.696.052 złote. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
I really like some of the new centralized logging systems like http://logstash.net/. It can handle loads of different sources and sinks and when you throw in the full power of elasticsearch searching for interesting data is an order of magnitude more powerful then what we currently have on z/OS. You can throw your distributed systems into the mix for a nice holistic view of your entire stack. On 4/12/2014 10:06 PM, John McKown wrote: This is just my mind wandering around loose again. You kind indulgence is appreciated. But I've been thinking about the z/OS syslog for some reason lately. Given what it was originally designed for, review by a human, it is a decent design. But is it really as helpful as it could be in today's z/OS environment? Should z/OS have a more generalized logging facility? I will grant that subsystems have various logs, but they each basically have their own structure. Is there really a need for the z/OS system log anymore? I really don't know. And I will admit that my mind has been corrupted by using Linux too much lately. grin/ So, if such a thing is even needed any more, what might it look like? Should it go to SPOOL? Should it be more like the OPERLOG and go to a LOGGER destination? Or should it go somewhere else? So what would I like? I know most will moan, but I _like_ structured, textual, information. So I would prefer that the output be in something like XML or JSON structure, not column based. And no encoded binary, OK? Now I'm trying to come up with what sort of data should be in the system header type data. These are just some fields that _I_ think would be useful in a good, generic, logging facility. First would be the current date in ISO8601 format, something like 2014-12-04T07:34:03-06:00 which is the date/time as I am typing this near Dallas, TX. This tells us the local time and gives us enough information to determine the UTC for comparison or conversion. I would also like the z/OS sysplex name, the system name, the CPU serial number, LPAR number, z/VM guest name (if applicable), job name (address space name), RACF owner, step name, proc step name, program name in the RB which issued the logging service call, program name in the first RB chained to the JS TCB (which I think should be the EXEC PGM=... name in most cases for batch jobs), ASID number, UNIX process id (==0 if not dubbed because there is no PID of 0 in UNIX, or maybe -1), step number (as used in SMF), substep number (again as defined in some SMF records). Product specific data would be formally encoded as designed by the product. Preferably, if in XML, with a DTD to describe it. And done so that standard XML facilities such as XSLT and XPath can process it. Which I one reason that I like XML a bit better than JSON at this point in time. There are a lot of XML utilities around. And, lastly, I do realize that the above would be very costly. Not necessarily to just implement into z/OS, but to actually change z/OS code to start using it. And that may be the real killer. IMO, one of the biggest obstructions to designing new facility which enhance existing facilities is the cost of implementing them. This combined with the current emphasis on immediate return on investment. I.e. if I invest a million dollars in something, I expect to get back 2 million in 6 months or less. Well, I guess that I've bored you enough with this bit of weirdness. Like many of my ideas, they sound good to me until others point out that they are just silly/stupid/unnecessary. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
On Fri, 5 Dec 2014 11:15:58 +0800, David Crayford dcrayf...@gmail.com wrote: I really like some of the new centralized logging systems like http://logstash.net/. It can handle loads of different sources and sinks and when you throw in the full power of elasticsearch searching for interesting data is an order of magnitude more powerful then what we currently have on z/OS. You can throw your distributed systems into the mix for a nice holistic view of your entire stack. Yeah, but how do you get it there ?. Sysadmins want everything *now*. Not tomorrow after syslog has been archived off. Takes us back to the tail syslog thread last month. Surely the Netview (or whatever) developers must be able to knock up code to push it down UDP port 514 - hipersockets across to zLinux running rsyslog .. Shane ... -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: thought: z/OS structured logging
On 5/12/2014 2:30 PM, Shane Ginnane wrote: Yeah, but how do you get it there ?. Sysadmins want everything*now*. Not tomorrow after syslog has been archived off. Takes us back to the tail syslog thread last month. Surely the Netview (or whatever) developers must be able to knock up code to push it down UDP port 514 - hipersockets across to zLinux running rsyslog .. Plenty of options there. I went to a session on zAware at SHARE which is a PCI-e card that processes syslog data. Nice that it's offloaded to a device. Logstash is open source but IBM have commercial offerings like SCA-LA and BigInsights. For that kind of stuff I would probably stick it on cheap commodity hardware. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
