Resolved: Why is GRS ENQ needed in SMFDUMP program?
Hi to all, In thread 'Why is GRS ENQ needed in SMFDUMP program?' I described the GRS ENQ in SMFDUMP and the resulting START PENDING problem. That was in June 2012 and there was a very interesting discussion lasted a while. This problem is now resolved at all. Thanks to all who replied, all replies were very helpful. I would like to share it in case someone needs it. I'm now using IEFU29 just as it is supplied in SYS1.SAMPLIB with little modifications (WTO white space eliminated and command string 'S SMFDUMP,' changed to 'S something,...' The STC, residing in a single SysPlex-wide Proclib, is using Symbolic names to resolve actual datasets in the JCL. All and every datasets used are unique per LPAR, so absolutely NO ENQ can take place as far as those dumping goes. Since IEE391A is suppressed, no automation software is kicking off SMFDUMP anymore, but on messages IEE366*, IEE985A, etc, the usual SMFDUMP program is started by automation software. This is because the IEFU29 (and the STC submitted) cannot handle situations where more than one MANx is in 'DUMP REQUIRED status. This is fine with me because it is a seldom event. I can now use automation software to issue 'I SMF' on all my dozen plus LPARS simultaneausly. No ENQ, no START PENDING, etc are experienced at all. I do that 'RO *ALL,I SMF' after midnight, 06:00 and 22:00 - 23:00 on all LPARS without causing any delays or problems. Even our nightly backup jobs are running faster. Of course the IEEU29 as supplied by IBM has still that same ENQ macro with the same keywords as used by the SMFDUMP program, but the wall-clock duration of that ENQ is just a second or two. The important difference between IEFU29 and SMFDUMP regarding that ENQ is - IEFU29 ENQ is after the actual 'I SMF', while SMFDUMP is using the ENQ during the whole story of 'I SMF' and finishing off all those MANx datasets and then release that ENQ. I see also the advantage of using a STC to do the dump - no JES2 initiators, higher WLM priority and it seemed to me (or I imagined :-D ) the dumping/emptying of MANx is going somewhat faster. Since I don't have to stagger my SMFDUMP jobs about 30 minutes on all LPARS, all our batch jobs depending on those SMF data can be run earlier in the morning - thus releasing extra CPU cycles to our impatient users. ;-) That is all folks! Many thanks to all who replied to me last year! I'm very grateful of your kind help. Now off to another quest. ;-D Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: SV: Why is GRS ENQ needed in SMFDUMP program?
A nagging doubt since my earlier post compelled me to go look at the UCB mapping. It seems that the bit is actually a one-byte count which makes sense for concurrent RESERVE activity. === Date: Sun, 1 Jul 2012 00:28:04 -0400 From: shmuel+...@patriot.net Subject: Re: SV: Why is GRS ENQ needed in SMFDUMP program? To: IBM-MAIN@LISTSERV.UA.EDU In bay145-w28ad40f15a4f1654e89483a3...@phx.gbl, on 06/25/2012 at 09:25 AM, J R jayare...@hotmail.com said: The RESERVE macro did (still does?) not directly do the hardware reserve. Rather, it set a bit in the UCB to tell the next IO to the unit to prepend a reserve CCW to the channel program. That was the original design, but these days there's an option to issue an I/O with a reserve CCW at the time that the ENQ SVC issued by the RESERVE macro obtains the resource. I don't recall what release added the option. -- Shmuel (Seymour J.) Metz, SysProg and JOAT Atid/2http://patriot.net/~shmuel We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
In 67102dbd-8232-47d1-88be-9b62c0715...@comcast.net, on 06/25/2012 at 03:33 AM, Dale Miller dalelmil...@comcast.net said: You might call the RESERVE macro a special form of ENQ, but the actual reserve was a hardware feature on DASD. There is no the actual reserve; there is a RESERVE macro and a reserve CCW opcode. Neither is a hardware feature on DASD. There were shared DASD options in the OS and on the DASD controller, but neither was called reserve. For the last few decades there's also been the issue of multiple allegiance. -- Shmuel (Seymour J.) Metz, SysProg and JOAT Atid/2http://patriot.net/~shmuel We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: SV: Why is GRS ENQ needed in SMFDUMP program?
In bay145-w28ad40f15a4f1654e89483a3...@phx.gbl, on 06/25/2012 at 09:25 AM, J R jayare...@hotmail.com said: The RESERVE macro did (still does?) not directly do the hardware reserve. Rather, it set a bit in the UCB to tell the next IO to the unit to prepend a reserve CCW to the channel program. That was the original design, but these days there's an option to issue an I/O with a reserve CCW at the time that the ENQ SVC issued by the RESERVE macro obtains the resource. I don't recall what release added the option. -- Shmuel (Seymour J.) Metz, SysProg and JOAT Atid/2http://patriot.net/~shmuel We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
You might call the RESERVE macro a special form of ENQ, but the actual reserve was a hardware feature on DASD. When a reserve was initiated by a processor, I/O's from other processors were delayed until the reserve was released. The use of the ENQ was (IIRC) to provide a finer- grained interlock between processes running on the same processor. Nearly 30 years ago (before LPAR's and SYSPLEXes),I was a systems programmer at one installation where we had lots of 3344 dasd units and were running DOS'es under VM. We did a very successful migration to 2 MVS's with no DOS and no VM. However, it we were plagued with reserve lockouts until we found the problems. The first issue was VM's asinine treatment of reserves - we had a choice of configuring VM to either: (1) pass the reserve on to the disk physically, which provided lockout from I/O by another physical processor but did not distinguish between different guests, or (2) to handle the reserves logically (internal to VM) which successfully interlocked guests on the same processor, but did not do the hardware reserve, so there was no interlock between processors. That issue was resolved when we dumped the DOS'es and VM and ran a test and a production MVS on different machines. However, we still had reserve lockouts from time to time. It turns out that the 3340's were configured (as advertised) as 4 3340's each, and defined in MVS as 4 units. However, it turned out that there was only one reserve register (?) per 3344, so a reserve against one of the logical units effectively reserved all 4, and MVS did not recognize the situation. This was ameliorated by careful dataset placement, but was finally resolved when we went to 3350's. I complained about this to the local IBM staff, but I'm not sure they believed me, and software support was done by FE's, so I'm not sure the situation was ever fixed, or even APAR'ed. Dale Miller -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
In CAE1XxDHCLFzEm2DQ8i_rc3PA9tP=JKa576L=2j0uQ=kp9k6...@mail.gmail.com, on 06/22/2012 at 09:16 AM, John Gilmore jwgli...@gmail.com said: I have reviewed Shmuel's language, and it still seems to me that he was equating a hardware RESERVE with an ENQ macro, Try a remedial reading course. My problem with Shmuel's posts is that they are often contentious for trhe sake of contention, PKB. -- Shmuel (Seymour J.) Metz, SysProg and JOAT Atid/2http://patriot.net/~shmuel We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
Mark Zelden wrote: It's not on my web site. 1) It is IBM copyrighted code. 2) My changes were done for a client. But my comments that I posted told you exactly what I changed from the sample and the IBM changes match exactly what you posted. Thanks. I will look around, reread your kind comments and check up my SMFDUMP programs. Tom Marchant wrote: If it is not SMFDUMP that is issuing the reserve, who is? This is what one of my z/OS guys and gals asked me. I will look into this. Andy Wood wrote: Any GRS exits? No. Barbara Nitz wrote: IIRC, if IOS reports for a system name, that means that it cannot determine which system it is. Which means that it isn't running in the same sysplex. So who else outside that plex have you accessing those DASD? What has that lpar id? I have looked around. No one outside the plex is accessing those DASD. I'm considering moving my SMF datasets (all of them) somewhere else, so they're residing somewhere not busy and NOT being duplicated to a remote site. We are issuing the I SMF via automation at midnight everywhere, and the resulting switch message is automated to start the SMFDUMP program. No reserves here; we hang on the base of the GDG we dump to (as in: One system can have it to roll in a new gdg, the others wait until that dump is done). Good to see what you also do. Each LPAR gets a turn to do the switch after midnight. During the 24 hours as soon as a *IEE391A appears that SMF dataset is dumped immediately, we don't wait for the last one or buffers being filled up. I will see if I can have our automation issue 'I SMF' themselve instead of SMDUMP and then have IFASMFDP read those SMF datasets in. Anyway to all: Many thanks for all your kind comments, I value all of them. For now I have staggered these jobs, so they won't run simultaneausly. I will investigate all comments and see what I can do. Again thanks! This discussions was really fruitful, I'm proud of all of you who commented. :-) Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
In CAE1XxDFG9DZMu=+npbg4tc4pgc2wcydx2gnsrt8vnpke_qu...@mail.gmail.com, on 06/20/2012 at 02:36 PM, John Gilmore jwgli...@gmail.com said: Anciently, under OS/PCP and OS/MFT, the Linkage Editor issued a RESERVE for the DASD volume on which the target PDS for its output load module resided in much the same circumstances. (It could not know in advance how much space it would need.) The ENQ[1] on SYSIEWL had nothing to do with the amount of space; it was there to protect the integrity of SYSLMOD. Why was the ENQ a RESERVE? Because there was no GRS. Since a PDS couldn't span volumes, the RESERVE protected the PDS from concurrent access both from the same system and from other systems. [1] RESERVE is just a special form of ENQ. -- Shmuel (Seymour J.) Metz, SysProg and JOAT Atid/2http://patriot.net/~shmuel We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
I have reviewed Shmuel's language, and it still seems to me that he was equating a hardware RESERVE with an ENQ macro, but the distinction you make is of course a valid one. My problem with Shmuel's posts is that they are often contentious for trhe sake of contention, but this time he may well have meant what you think he meant. John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
-Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@listserv.ua.edu] On Behalf Of John Gilmore Sent: Friday, June 22, 2012 8:17 AM To: IBM-MAIN@listserv.ua.edu Subject: Re: Why is GRS ENQ needed in SMFDUMP program? I have reviewed Shmuel's language, and it still seems to me that he was equating a hardware RESERVE with an ENQ macro, but the distinction you make is of course a valid one. My problem with Shmuel's posts is that they are often contentious for trhe sake of contention, but this time he may well have meant what you think he meant. John Gilmore, Ashland, MA 01721 - USA Oh, I'll definitely agree with that! Also, they are often much too terse. I usually try to include a URL to an IBM manual or other authoritative source, unless it is just my personal opinion. In which case I normally indicate by prefixing with IMO. -- John McKown Systems Engineer IV IT Administrative Services Group HealthMarkets(r) 9151 Boulevard 26 * N. Richland Hills * TX 76010 (817) 255-3225 phone * john.mck...@healthmarkets.com * www.HealthMarkets.com Confidentiality Notice: This e-mail message may contain confidential or proprietary information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. HealthMarkets(r) is the brand name for products underwritten and issued by the insurance subsidiaries of HealthMarkets, Inc. -The Chesapeake Life Insurance Company(r), Mid-West National Life Insurance Company of TennesseeSM and The MEGA Life and Health Insurance Company.SM -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
I don't know the history of SMFDUMP within samplib, but since it is apparently not shipped in supported z/OS releases, I would guess that it is not appropriate to use on said releases. It is possible that the user of the sample is supposed to change SMFRNAME to something like CL44'the_dataset_name'. But for IEEU29 (the IEFU29 sample) it is more likely that the ENQ is to serialize this exit routine with another instance of the same exit routine on the same LPAR.. For IEEU29, it is not at all clear to me what proc DUMPXY is supposed to have in it If this ENQ is having affects on other LPARs then either -- you have this ENQ in your RNL include list to be converted to a reserve -- you are not running with RACF, and your alternate security product does not follow RACF protocols with respect to system-level ENQs. Peter Relson z/OS Core Technology Design -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
In my previous post I mistyped in mentioning alternate security product. I meant instead an alternate serialization product (e.g., MIM) which may have rules different than GRS. Peter Relson z/OS Core Technology Design -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
I attached a text screen print. The lines are the input field lines. I will resend. Doug -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@listserv.ua.edu] On Behalf Of Peter Relson Sent: Friday, June 22, 2012 11:53 AM To: IBM-MAIN@listserv.ua.edu Subject: Re: Why is GRS ENQ needed in SMFDUMP program? In my previous post I mistyped in mentioning alternate security product. I meant instead an alternate serialization product (e.g., MIM) which may have rules different than GRS. Peter Relson z/OS Core Technology Design -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
On 6/22/2012 5:37 AM, McKown, John wrote: I am fairly sure that Shmuel's comment was about the RESERVE macro, not the hardware function. The RESERVE macro is logically equivalent to a SYSTEMS level ENQ plus a hardware reserve (unless the GRSRNL converts it to not do the hardware RESERVE). And, IIRC, the RESERVE macro generally does not immediately result in the hardware reserve being done before returning to the user. It simply marks the UCB somehow so that the hardware reserve is done at the beginning of the next I/O operation on the device. I'm not sure that this is still true or not. Still true unless SYNCHRES(YES) is specified on the GRSDEF statement in GRSCNFxx. -- Edward E Jaffe Phoenix Software International, Inc 831 Parkview Drive North El Segundo, CA 90245 310-338-0400 x318 edja...@phoenixsoftware.com http://www.phoenixsoftware.com/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
Mark Zelden wrote: I'm not sure exactly. Maybe to keep 2 of them start are started at the same time from causing a problem since nothing else uses that qname/rname. This is what I also guessed. Prevent two or more 'I SMF' happening at the same time in a Sysplex. As for the RESERVEs done when the output is on DASD, I commented out that code 15 years ago. I have no clue what purpose it serves to reserve the entire volume when writing an output data set to disk. I also don't use RESERVEs. I forgot to mention that I get a lot of IOS071I ,**,SMS, START PENDING messages during a switch. So, after my nightly jobs, I try to dump/clear all my SMF datasets around 07:00 and thus hopefully avoid a 'I SMF' during production hours and the resulting IOS071I. Thanks Mark for your reply. Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
The reason is that we sometimes we get a deadly embrace where two SMFDUMP issue those ENQs and then while the system is doing its 'I SMF' things, it still scans the DASD despite that we do not record SMF type 19. In the end I need to cancel one of those SMF jobs, let the other run and retry with possible loss of data. You wrote about your SMFDUMP program. What is that? The one provided by z/OS does not use that ENQ. Regardless, the ENQ you show is a SYSTEM level ENQ, not a SYSTEMS level ENQ. Thus reserve is not appropriate and holding of this ENQ on one LPAR will have no effect on another LPAR. It makes no sense to have unique names per LPAR for a system-level ENQ (or any correctly needed ENQ, of course). What exactly is the deadly embrace here? If the two SMFDUMP's both go after the ENQ, one will get it and one will wait. That is expected. Since the system's I SMF does not get this ENQ as far as I know, it cannot be waiting for completion of your SMFDUMP. If you are converting ENQ's to reserves, I could imagine general cases where you would get into deadlocks. But I don't think that would happen in a case where each process gets only one serializing resource (e.g., ENQ). A deadly embrace is typically where process A holds resource RA and goes after a resource RB held by process B, while process B holds resource RB and goes after resource RA held by process A. Obviously it can be far more complex than that. Peter Relson z/OS Core Technology Design -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
Peter Relson wrote: You wrote about your SMFDUMP program. What is that? The one provided by z/OS does not use that ENQ. The one as supplied by IBM. See copyright statement: MODULE NAME = SMFDUMP DESCRIPTIVE NAME = CUSTOM-BUILT IPO SUPPLIED ROUTINE TO DUMP THE SMF DATASETS COPYRIGHT = 5751-CS1 INTERNATIONAL BUSINESS MACHINES CORPORATION, 1983, 1988 and modification history: $MOD=(SMFDUMP) COMP(SMF) HDLD(MVS): $CC= REASONCD,RELEASE#,DATECHGD,CINIT: DESCRIPTION $D1= MSPIPOG,,, CHANGED IN MVS/SP IPO 3.8 G $D2= MSPIPOF,,, CHANGED IN MVS/SP IPO 3.8 F $D3= MSPIPOH,,, ... $D4= MSPIPOI,,, ... $D5= MSEIPO4,,, CHANGED TO SUPPORT MVS/SE2 $D6=CR#I70140,,, INSURE INTEGERITY OF SMFDUMPW $D6= DATASET. $D7= CR24926,9402,04/25/94, FJB SUPPORT 44 CHARACTER DSNAMES $D7= FOR SMF DUMP DATASETS, AND $D7= FOR VERSION 5.1 CHANGES. $D8= PR01623, 9402P,09/20/94, WGL FIX MULTILINE WTO $D9= PR02195, 9601P,01/18/96, JMC FIXED RDS OFFSETS I'm using V5 not V4. it could be that someone modified it before my time... Regardless, the ENQ you show is a SYSTEM level ENQ, not a SYSTEMS level ENQ. Indeed, from source this: ENQ (SMFQNAME,SMFRNAME,E,,SYSTEM) Thus reserve is not appropriate and holding of this ENQ on one LPAR will have no effect on another LPAR. It makes no sense to have unique names per LPAR for a system-level ENQ (or any correctly needed ENQ, of course). Thanks. This is what I also understand after reading MVS Programming: Authorized Assembler Services Reference, Volume 2 (EDTINFO-IXGWRITE). What exactly is the deadly embrace here? If the two SMFDUMP's both go after the ENQ, one will get it and one will wait. That is expected. Of course, I see one SMFDUMP with exclusive ENQ while other SMFDUMP on other LPARS are waiting for it. If one LPAR SMFDUMP finish a 'I SMF', it release the ENQ. Then ANOTHER SMFDUMP on another LPAR which is waiting for the ENQ name, picks up the ENQ and do its own 'I SMF' and so on on all the LPARs. Eventually the first SMFDUMP obtains the ENQ and so on. All the SMFDUMP program are getting a turn to do a 'I SMF'. It goes on until no SMF datasets are remaining in DUMP status. The problem is that overall performance on all the LPARs are suffering due to START PENDING messages. Another problem, the operator and the MVS Team are canceling my jobs thinking there is an error despite my instructions to leave them out. If you are converting ENQ's to reserves, I could imagine general cases where you would get into deadlocks. But I don't think that would happen in a case where each process gets only one serializing resource (e.g., ENQ). I'm not doing any converting. In fact, I asked about a strange name, SYSSMF01, I saw in our GRSRNLxx. I'm very sure I'm not using it or any converting. In any case, thanks for your kind reply. It will help me much! Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
On Thu, 21 Jun 2012 07:54:22 -0400, Peter Relson wrote: You wrote about your SMFDUMP program. What is that? The one provided by z/OS does not use that ENQ. I think the sample programs SMFDUMP and IEFU29 shipped with OS/390 (and earlier) used that ENQ. SMFDUMP is not shipped with z/OS, but SYS1.SAMPLIB(IEEU29) is still (z/OS 1.11) there: ... MVC ENQLIST(LENQLIST),ENQLSTX LOAD IN MODEL PARM LIST ENQ MF=(E,ENQLIST) TEST IF RESOURCE IN USE? LTR R15,R15 WAS THE RESOURCE AVAILABLE? BNZ SKIPDUMPNO, DO NOT START DUMP ... * * DATA AREA * SMFQNAME DCCL8'IPOSMF01' SIPO50 SMFRNAME DCCL7'DATASET' * ... ENQLSTX ENQ (SMFQNAME,SMFRNAME,E,,SYSTEM),RET=TEST,MF=L ... The sample IEFU29 does not start SMFDUMP (S DUMPXY,...) if it's already running. Norbert Friemel -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
On Thu, 21 Jun 2012 08:17:33 -0500, Elardus Engelbrecht wrote: Peter Relson wrote: ... the ENQ you show is a SYSTEM level ENQ, not a SYSTEMS level ENQ. Indeed, from source this: ENQ (SMFQNAME,SMFRNAME,E,,SYSTEM) ... holding of this ENQ on one LPAR will have no effect on another LPAR. Thanks. This is what I also understand after reading MVS Programming: Authorized Assembler Services Reference, Volume 2 (EDTINFO-IXGWRITE). What exactly is the deadly embrace here? If the two SMFDUMP's both go after the ENQ, one will get it and one will wait. That is expected. Of course, I see one SMFDUMP with exclusive ENQ while other SMFDUMP on other LPARS are waiting for it. Not the SYSTEM level ENQ that you have shown, unless you have added IPOSMF01 to the INCLude list in GRSRNKxx. If one LPAR SMFDUMP finish a 'I SMF', it release the ENQ. Then ANOTHER SMFDUMP on another LPAR which is waiting for the ENQ name, picks up the ENQ and do its own 'I SMF' and so on on all the LPARs. Eventually the first SMFDUMP obtains the ENQ and so on. All the SMFDUMP program are getting a turn to do a 'I SMF'. It goes on until no SMF datasets are remaining in DUMP status. So, the problem is not a deadly embrace, but delays in processing. I think you need to determine what is causing the delays. What is the impact of the delay? Is this happening because all of the LPARs are doing this processing at the same time? If, for example, this is happening because the SMF records for the whole day are being dumped on every LPAR at midnight, perhaps you could stagger the dumping somehow. -- Tom Marchant -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
On Thu, 21 Jun 2012 04:25:48 -0500, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: I also don't use RESERVEs. I forgot to mention that I get a lot of IOS071I ,**,SMS, START PENDING messages during a switch. So, after my nightly jobs, I try to dump/clear all my SMF datasets around 07:00 and thus hopefully avoid a 'I SMF' during production hours and the resulting IOS071I. Hi Elardus, This sounds to me like you are collecting SMF type 19 records. If this is the case I would recommend not doing this. Doug -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
Doug Henry wrote: This sounds to me like you are collecting SMF type 19 records. If this is the case I would recommend not doing this. D SMF,O on any LPAR show this: SUBSYS(OMVS,NOTYPE(14,15,19,34:35,40,42,99)) -- PARMLIB SUBSYS(TSO,NOTYPE(14,15,19,40,42,99)) -- SYS SUBSYS(JES2,NOTYPE(14,15,19,40,42,99)) -- SYS SYS(NOTYPE(14,15,19,40,42,99)) -- PARMLIB So, no 19 are involved at all. But, thanks for telling me this, I appreciate it much! Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
The behavior of your systems that you described sounds as if you have a global enqueue that is systems-wide rather than an enqueue that should only affect one system. The source code snippet you posted shows what looks correct for a local enqueue (within one system) rather than a global enqueue. You might also investigate whether your source code matches the load module that is executing. You can do this in at least two ways. The first way is to obtain a superzap dump of the load module, find the enqueue macro within the load module, disassemble the parameter list of the ENQ macro with the aid of the parameter list contents as shown in the MVS Diagnosis: Reference book (GA22-7588) for SVC 56, paying special attention to the bits involved with the SCOPE parameter. If the executable load module shows SCOPE=SYSTEMS, then that would explain your systems' behavior and also signal that your source code is down level. The second way is to run GTF on any one of the systems in! volved and trace only SVC 56 for the jobname of your SMF dump; then find the parameter list in the GTF trace and disassemble it as described above. I am not sure that this second way will work, however, as I do not remember if GTF traces the parameter list for SVC 56. The issuance of multiple START PENDING messages does not automatically mean that there are performance problems anywhere. START PENDING means that some shared DASD data set is either being very heavily used or is RESERVEd on one system while another system is trying to do some kind of I/O to the same device, and the second system might even be trying to access a different data set from the one that the first system is using heavily or has reserved. A START PENDING message is one possible clue as to what is causing a performance problem, but don't go investigating any given START PENDING message unless you already know for some other reason that you are having a performance problem. One possible problem indicator would be that other work on the system with START PENDING messages is stopped because it is has an I/O request in the queue for the problem device. This situation will show up as an abnormally high average IOS queue length on that one device on a system with the! START PENDING messages. There are many other messages or indicators that can be used to diagnose a performance problem, but they do not automatically mean there is a problem. There is no performance problem anywhere unless someone is complaining. Bill Fairchild Programmer Rocket Software 408 Chamberlain Park Lane * Franklin, TN 37069-2526 * USA t: +1.617.614.4503 * e: bfairch...@rocketsoftware.com * w: www.rocketsoftware.com -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@listserv.ua.edu] On Behalf Of Tom Marchant Sent: Thursday, June 21, 2012 9:10 AM To: IBM-MAIN@listserv.ua.edu Subject: Re: Why is GRS ENQ needed in SMFDUMP program? On Thu, 21 Jun 2012 08:17:33 -0500, Elardus Engelbrecht wrote: Peter Relson wrote: ... the ENQ you show is a SYSTEM level ENQ, not a SYSTEMS level ENQ. Indeed, from source this: ENQ (SMFQNAME,SMFRNAME,E,,SYSTEM) ... holding of this ENQ on one LPAR will have no effect on another LPAR. Thanks. This is what I also understand after reading MVS Programming: Authorized Assembler Services Reference, Volume 2 (EDTINFO-IXGWRITE). What exactly is the deadly embrace here? If the two SMFDUMP's both go after the ENQ, one will get it and one will wait. That is expected. Of course, I see one SMFDUMP with exclusive ENQ while other SMFDUMP on other LPARS are waiting for it. Not the SYSTEM level ENQ that you have shown, unless you have added IPOSMF01 to the INCLude list in GRSRNKxx. If one LPAR SMFDUMP finish a 'I SMF', it release the ENQ. Then ANOTHER SMFDUMP on another LPAR which is waiting for the ENQ name, picks up the ENQ and do its own 'I SMF' and so on on all the LPARs. Eventually the first SMFDUMP obtains the ENQ and so on. All the SMFDUMP program are getting a turn to do a 'I SMF'. It goes on until no SMF datasets are remaining in DUMP status. So, the problem is not a deadly embrace, but delays in processing. I think you need to determine what is causing the delays. What is the impact of the delay? Is this happening because all of the LPARs are doing this processing at the same time? If, for example, this is happening because the SMF records for the whole day are being dumped on every LPAR at midnight, perhaps you could stagger the dumping somehow. -- Tom Marchant -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM
Re: Why is GRS ENQ needed in SMFDUMP program?
On Thu, 21 Jun 2012 04:25:48 -0500, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: Mark Zelden wrote: I'm not sure exactly. Maybe to keep 2 of them start are started at the same time from causing a problem since nothing else uses that qname/rname. This is what I also guessed. Prevent two or more 'I SMF' happening at the same time in a Sysplex. As for the RESERVEs done when the output is on DASD, I commented out that code 15 years ago. I have no clue what purpose it serves to reserve the entire volume when writing an output data set to disk. I also don't use RESERVEs. I forgot to mention that I get a lot of IOS071I ,**,SMS, START PENDING messages during a switch. So, after my nightly jobs, I try to dump/clear all my SMF datasets around 07:00 and thus hopefully avoid a 'I SMF' during production hours and the resulting IOS071I. Thanks Mark for your reply. What do you mean you don't use reserves? You have commented them out in the code, or your IODF/HCD isn't coded for shared DASD thus not issuing the reserves? If you are seeing the start pendings, it looks like you do have the reserves in the code. All my LPARs start an STC using the SMFDUMP program that issues the I SMF at the same time and there is no problem. Regards, Mark -- Mark Zelden - Zelden Consulting Services - z/OS, OS/390 and MVS mailto:m...@mzelden.com Mark's MVS Utilities: http://www.mzelden.com/mvsutil.html Systems Programming expert at http://expertanswercenter.techtarget.com/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
Tom Marchant wrote: Not the SYSTEM level ENQ that you have shown, unless you have added IPOSMF01 to the INCLude list in GRSRNKxx. It is not there in GRSRNLxx (or GRSRNKxx which you wrote... ;-D ). So, the problem is not a deadly embrace, but delays in processing. I think you need to determine what is causing the delays. What is the impact of the delay? This why I asked if the usage of that GRS is causing delays or not. Is that SMFDUMP WAD? What is more, I forgot to post this to Peter Relson this: IEF196I IOS071I 6F85,**,*MASTER*, START PENDING IOS071I 6F85,**,*MASTER*, START PENDING 203 IOS431I DEVICE 6F85 RESERVED TO CPU=.,LPAR ID=04 205 SYSTEM= IEA630I OPERATOR SYSIOSRS NOW ACTIVE, SYSTEM= , LU=IOSAS ISG343I 00.32.11 GRS STATUS 713 208 DEVICE:6F85 VOLUME:SMF004 RESERVED BY SYSTEM S=SYSTEMS IPOSMF01 DUMPOUT SYSNAMEJOBNAME ASID TCBADDR EXC/SHR STATUS SMF0048 00AFF048 EXCLUSIVE OWN This, the S=SYSTEMS is NOT the same as in the source which is S=SYSTEM. Majorname is the same in the source, but not the Minor name (RNAME). Is this happening because all of the LPARs are doing this processing at the same time? If, for example, this is happening because the SMF records for the whole day are being dumped on every LPAR at midnight, perhaps you could stagger the dumping somehow. All my SMF datasets are dumped during the 24 hours as soon as a *single* dataset get DUMP marked (IEE391A). Then just after midnight all my SMFDUMP jobs (emptying all SMF datasets) are staggered about 5 - 15 minutes apart. Thanks to all. I'm beginning to wonder if the SMFDUMP program may be corrupt and need to be re-assembled from scratch. Or tryout file 686 from cbttape.org. Or I need to stagger them further apart... Hmmm... anyone have a good idea? Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
On Thu, 21 Jun 2012 07:54:22 -0400, Peter Relson rel...@us.ibm.com wrote: The reason is that we sometimes we get a deadly embrace where two SMFDUMP issue those ENQs and then while the system is doing its 'I SMF' things, it still scans the DASD despite that we do not record SMF type 19. In the end I need to cancel one of those SMF jobs, let the other run and retry with possible loss of data. You wrote about your SMFDUMP program. What is that? The one provided by z/OS does not use that ENQ. It is the old CBIPO SMFDUMP program that has been discussed many times on this list. A lot of shops still use as part of their daily SMF roll up process as it issues an I SMF internally to get the current data plus dumps any / all MANx data sets that need to be dumped (in case any were missed or there is no other process in place to dump them when they fill up). Regards, Mark -- Mark Zelden - Zelden Consulting Services - z/OS, OS/390 and MVS mailto:m...@mzelden.com Mark's MVS Utilities: http://www.mzelden.com/mvsutil.html Systems Programming expert at http://expertanswercenter.techtarget.com/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
IFASMFDP or IFASMFDL SNIP Thanks to all. I'm beginning to wonder if the SMFDUMP program may be corrupt and need to be re-assembled from scratch. Or tryout file 686 from cbttape.org. /SNIP -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
Mark Zelden wrote: What do you mean you don't use reserves? You have commented them out in the code, or your IODF/HCD isn't coded for shared DASD thus not issuing the reserves? Hmmm, interesting. I will have a talk with my HCD folks... If you are seeing the start pendings, it looks like you do have the reserves in the code. All my LPARs start an STC using the SMFDUMP program that issues the I SMF at the same time and there is no problem. Ah, that is a relief. It must be something else or I need to re-assemble/verify my source... Mark, many many many thanks for your kind answers. This will surely help! Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
On Thu, 21 Jun 2012 09:59:13 -0500, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: Mark Zelden wrote: What do you mean you don't use reserves? You have commented them out in the code, or your IODF/HCD isn't coded for shared DASD thus not issuing the reserves? Hmmm, interesting. I will have a talk with my HCD folks... If you are seeing the start pendings, it looks like you do have the reserves in the code. All my LPARs start an STC using the SMFDUMP program that issues the I SMF at the same time and there is no problem. Ah, that is a relief. It must be something else or I need to re-assemble/verify my source... Mark, many many many thanks for your kind answers. This will surely help! You're welcome. As far as I can tell from one of your posts, I am using the same code you do. I do asm/lnk with other usermods / exits for each OS upgrade. These are the only changes I have, and the last change was in 2000 (discussed on this list also IIRC): *** * THIS IS THE SAMPLE SMFDUMP PROGRAM TAKEN FROM CPAC.SAMPLIB * * AS IS, EXCEPT THE RESERVE FOR A DUMPOUT ON DASD HAS BEEN REMOVED. * * SIX LINES OF CODE WERE COMMENTED OUT. FIND 'MSZ' FOR THE LINES.* * * * I ALSO CHANGED A MULTI LINE WTO - BROKE WITH OS/390 2.8* *WTO TEXT=(MSG1,MSG2),MF=(E,MSG007) ISSUE MSG * * WAS CHANGED TO:* *WTO TEXT=((MSG1,),(MSG2,)),MF=(E,MSG007) * * MARK ZELDEN 01/27/2000 * *** Mark -- Mark Zelden - Zelden Consulting Services - z/OS, OS/390 and MVS mailto:m...@mzelden.com Mark's MVS Utilities: http://www.mzelden.com/mvsutil.html Systems Programming expert at http://expertanswercenter.techtarget.com/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
Mark Zelden wrote: As far as I can tell from one of your posts, I am using the same code you do. I do asm/lnk with other usermods / exits for each OS upgrade. These are the only changes I have, and the last change was in 2000 (discussed on this list also IIRC): All the sources I have, all contains those multiline WTO fixes. May I look around in your website to get a copy, please? Thanks again! Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
On Thu, 21 Jun 2012 10:20:56 -0500, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: Mark Zelden wrote: As far as I can tell from one of your posts, I am using the same code you do. I do asm/lnk with other usermods / exits for each OS upgrade. These are the only changes I have, and the last change was in 2000 (discussed on this list also IIRC): All the sources I have, all contains those multiline WTO fixes. May I look around in your website to get a copy, please? It's not on my web site. 1) It is IBM copyrighted code. 2) My changes were done for a client. But my comments that I posted told you exactly what I changed from the sample and the IBM changes match exactly what you posted. Regards, Mark -- Mark Zelden - Zelden Consulting Services - z/OS, OS/390 and MVS mailto:m...@mzelden.com Mark's MVS Utilities: http://www.mzelden.com/mvsutil.html Systems Programming expert at http://expertanswercenter.techtarget.com/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
On Thu, 21 Jun 2012 09:45:05 -0500, Elardus Engelbrecht wrote: IOS431I DEVICE 6F85 RESERVED TO CPU=.,LPAR ID=04 205 SYSTEM= If it is not SMFDUMP that is issuing the reserve, who is? -- Tom Marchant -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
On Thu, 21 Jun 2012 09:45:05 -0500, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: . . . Not the SYSTEM level ENQ that you have shown, unless you have added IPOSMF01 to the INCLude list in GRSRNKxx. It is not there in GRSRNLxx (or GRSRNKxx which you wrote... ;-D ). Any GRS exits? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Why is GRS ENQ needed in SMFDUMP program?
Good day I have reviewed our SMFDUMP program as well the one in CBTTAPE file 686. Why is there a GRS ENQ? Is it needed? APAR? documentation? logic? I see those names: SMFQNAME DCCL8'IPOSMF01' SMFRNAME DCCL7'DATASET' and scope is SYSTEM? What will break if I do not use GRS and then fall through RESERVE which is indeed not recommended? The reason is that we sometimes we get a deadly embrace where two SMFDUMP issue those ENQs and then while the system is doing its 'I SMF' things, it still scans the DASD despite that we do not record SMF type 19. In the end I need to cancel one of those SMF jobs, let the other run and retry with possible loss of data. The alternative is that I need to space out those 'I SMF' across the sysplex over longer elapsed wall clock time. Or making those QNAME / RNAME combination unique per LPAR? Will something break? Or making those wait time in SMFDUMP shorter? I'm asking this because the way how 'I SMF' is partly documented. Other question: In our GRSRNLxx is this line: RNLDEF RNL(EXCL) TYPE(GENERIC) QNAME(SYSSMF01) Anyone knows who is using it? Of course you could point me to any documentations. Thanks very much in advance. Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Why is GRS ENQ needed in SMFDUMP program?
On Wed, 20 Jun 2012 09:44:57 -0500, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: Good day I have reviewed our SMFDUMP program as well the one in CBTTAPE file 686. Why is there a GRS ENQ? Is it needed? APAR? documentation? logic? I see those names: SMFQNAME DCCL8'IPOSMF01' SMFRNAME DCCL7'DATASET' and scope is SYSTEM? What will break if I do not use GRS and then fall through RESERVE which is indeed not recommended? The reason is that we sometimes we get a deadly embrace where two SMFDUMP issue those ENQs and then while the system is doing its 'I SMF' things, it still scans the DASD despite that we do not record SMF type 19. In the end I need to cancel one of those SMF jobs, let the other run and retry with possible loss of data. The alternative is that I need to space out those 'I SMF' across the sysplex over longer elapsed wall clock time. Or making those QNAME / RNAME combination unique per LPAR? Will something break? Or making those wait time in SMFDUMP shorter? I'm asking this because the way how 'I SMF' is partly documented. Other question: In our GRSRNLxx is this line: RNLDEF RNL(EXCL) TYPE(GENERIC) QNAME(SYSSMF01) Anyone knows who is using it? Of course you could point me to any documentations. Thanks very much in advance. I'm not sure exactly. Maybe to keep 2 of them start are started at the same time from causing a problem since nothing else uses that qname/rname. As for the RESERVEs done when the output is on DASD, I commented out that code 15 years ago. I have no clue what purpose it serves to reserve the entire volume when writing an output data set to disk. Regards, Mark -- Mark Zelden - Zelden Consulting Services - z/OS, OS/390 and MVS mailto:m...@mzelden.com Mark's MVS Utilities: http://www.mzelden.com/mvsutil.html Systems Programming expert at http://expertanswercenter.techtarget.com/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
