Looking in my wayback machine I recall an operator who accidentally pushed the IML button on our 3033 console a couple of times. After the second time, we installed a protective device, and the Sheldon-shield was invented.
Mark Jacobs Sent from ProtonMail, Swiss-based encrypted email. GPG Public Key - https://api.protonmail.ch/pks/lookup?op=get&[email protected] ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, March 24th, 2021 at 1:50 PM, Pommier, Rex <[email protected]> wrote: > Hi Radoslaw, > > I knew you meant it as a joke and I took it as such. Hence my smiley face. > The OPERCMDS class has several entries in it but somehow QUIESCE was missed > from way back when for a specific lock down so it was allowed by a more > generic profile. > > I checked the type80 records and there was nothing immediately before the > quiesce command was entered. > > O well, we figured out what happened and put security in place to minimize > the possibility of it happening again, and we now know what to do if it does > happen so we can get the system back up without issue and be able to find and > train the guilty party. > > Rex > > -----Original Message----- > > From: IBM Mainframe Discussion List [email protected] On Behalf Of > Radoslaw Skorupka > > Sent: Wednesday, March 24, 2021 10:26 AM > > To: [email protected] > > Subject: [External] Re: z14 HMC log information > > Obviously the part about killing was only poor joke, but there is some sense > hidden in. > > I mean it is good idea to talk to person who did it. Not to punish, but talk > and explain and hear his/her explanations. > > For RACF admin it is quite obvious the security model should be somehow > checked. Again the person can have good explanation of current state of > protection. > > Regarding traces - It is a little bit hard to test, especially without access > to mainframe ;-) but I guess SMF80 can be written just before system freeze. > Note it is RACF security check - it happens BEFORE the command is interpreted > by the system. Simpler example: when you issue CANCEL CICSABC and you don't > have such started task, you first will be checked by RACF (and maybe > rejected) and then the command is really issued, and you will get answer like > "there is no such started task to cancel". > > BTW: I imagined what would happen after such case on production... > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Radoslaw Skorupka > > (looking for new job) > > Lodz, Poland > > W dniu 24.03.2021 o 14:58, Pommier, Rex pisze: > > > I'm going to agree with most of it. I don't like the part about killing the > > RACF admin. I'm not the one who initially set up the OPERCMDS security but > > I missed the fact the QUIESCE command wasn't set as "don't let anybody > > use". Hari-kari is not on my bucket list. :-) > > > > On to Radoslaw's comment about logging - it is logged, after the fact. > > QUIESCE does exactly that - it stops the LPAR in its tracks. Do not pass > > Go, do not collect $200. No z/OS logging at the time it happens. IBM > > hardware support found and reported the wait state back to us from some > > hardware logs that were forwarded to them from our CE. The z/OS logging > > takes place after the PSW restart from the HMC occurs and yes, it shows the > > console or user that executed the command. However in our case since the > > LPAR stopped in the middle of the day and we had managers breathing down > > our necks to get the system back up we didn't have time to properly > > diagnose until after the fact - which included an IPL which in turn did not > > allow the logging of the quiesce command to take place. > > > > Rex > > > > -----Original Message----- > > > > From: IBM Mainframe Discussion List [email protected] On Behalf Of > > Carmen Vitullo > > > > Sent: Wednesday, March 24, 2021 8:07 AM > > > > To: [email protected] > > > > Subject: [External] Re: z14 HMC log information > > > > agree 100%, when I tested the command on my sandbox system I see my ID in > > the syslog as the culprit :) if done from a console, then the console name > > is shown. > > > > Carmen Vitullo > > > > -----Original Message----- > > > > From: Radoslaw [email protected] > > > > To: IBM-MAIN [email protected] > > > > Date: Wednesday, 24 March 2021 8:01 AM CDT > > > > Subject: Re: z14 HMC log information > > > > IMHO there should be a trace in a syslog. Maybe that part of syslog is > > somehow lost. > > > > And it would be good idea to have SMF record for that command. I don't know > > about console commands, but SMF80 could be cut if you take care about > > AUDIT(ALL(READ)) in advance. I mean RACF profile. > > > > Then find the person and kill him. Don't forget to kill RACF admin who > > forgot to protect the command. Kill'em all. ;-) > > > > Seriously: it is a symptom of bad security model. > > > > -- > > > > Radoslaw Skorupka > > > > (looking for new job) > > > > Lodz, Poland > > > > W dniu 24.03.2021 o 13:54, Pommier, Rex pisze: > > > > > Hi Carmen, > > > > > > We use the quiesce parameter of the reset command as well. I'm thinking > > > that's what happened here as well, but we have no idea "whodunit". We had > > > 1 operator on duty that day and he was off doing something else when it > > > happened so the command made it into the system via an SDSF-like console > > > session. > > > > > > Rex > > > > > > -----Original Message----- > > > > > > From: IBM Mainframe Discussion List [email protected] On > > > > > > Behalf Of Carmen Vitullo > > > > > > Sent: Wednesday, March 24, 2021 7:43 AM > > > > > > To: [email protected] > > > > > > Subject: [External] Re: z14 HMC log information > > > > > > There is a queisce command to put a job to sleep, and I think that's what > > > my operator was thinking he was doing, for a runaway job we do use > > > automation to queisce a job and send us an alert. > > > > > > from SDSF there's a quiesce line command RQ it translates to RESET > > > > > > jobname,QUIESCE > > > > > > Carmen Vitullo > > > > > > -----Original Message----- > > > > > > From: Radoslaw [email protected] > > > > > > To: IBM-MAIN [email protected] > > > > > > Date: Wednesday, 24 March 2021 5:35 AM CDT > > > > > > Subject: Re: z14 HMC log information > > > > > > IMHO not really. > > > > > > By freezing the system you disrupt all the online processing and you > > > loose capability to find out what's wrong. > > > > > > I had to do with CPU hogging tasks and spool 100% filled by some ugly > > > job. In both cases the very first thing to fix it was to logon to the > > > system. > > > > > > -- > > > > > > Radoslaw Skorupka > > > > > > (looking for new job) > > > > > > Lodz, Poland > > > > > > W dniu 24.03.2021 o 03:30, kekronbekron pisze: > > > > > > > It's useful when you want to say "STOP IT!" to run-away CPU or spool > > > > jobs. > > > > > > > > Will buy you some time to investigate. > > > > > > > > - KB > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > > > On Tuesday, March 23, 2021 7:07 PM, Pommier, Rex > > > > [email protected] wrote: > > > > > > > > > Hey Carmen, > > > > > > > > > > At least you had an operator tell you what they did. :-) Once we > > > > > determined what had happened (after a painful middle-of-the-day IPL) > > > > > the only response I got back from anybody was "I didn't do anything > > > > > like that!" But now that I know what that command does and how it > > > > > works I doubt I'll ever forget this one. > > > > > > > > > > Rex > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: IBM Mainframe Discussion List [email protected] On > > > > > > > > > > Behalf Of Carmen Vitullo > > > > > > > > > > Sent: Tuesday, March 23, 2021 7:47 AM > > > > > > > > > > To: [email protected] > > > > > > > > > > Subject: Re: [External] Re: z14 HMC log information > > > > > > > > > > WOW! this takes me back 20 years when an operator wanted to quiesce > > > > > > > > > > a job and issues that command from the console he called my told me > > > > > > > > > > what he did, I researched what that command does, seemed to me it > > > > > > > > > > was like on the old system consoles O2 and O3 IIRC one was a PSW > > > > > > > > > > stop and one a restart, I found the restart just for grins and it > > > > > > > > > > worked, been a long time since I've hear tell someone had that same > > > > > > > > > > experience great find > > > > > > > > > > Carmen Vitullo > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: Rex [email protected] > > > > > > > > > > To: IBM-MAIN [email protected] > > > > > > > > > > Date: Monday, 22 March 2021 5:18 PM CDT > > > > > > > > > > Subject: Re: [External] Re: z14 HMC log information > > > > > > > > > > OK, this is embarrassing. We found the problem - but don't know the > > > > > > > > > > culprit. We discovered the quiesce command wasn't locked down and > > > > > > > > > > somebody somewhere keyed it in. The reason we don't know who/where > > > > > > > > > > is because our SDSF-like product had the capability to also do a > > > > > > > > > > quiesce and since we didn't know that's what caused it, we IPLed the > > > > > > > > > > LPAR. I did some research after the fact to determine just how > > > > > > > > > > quiesce works (been working on these things for 30 years and never > > > > > > > > > > had a reason to even play with quiesce). Now I know - and I know how > > > > > > > > > > to make the machine go again, picking up where it stopped. PSW > > > > > > > > > > Restart from the HMC doesn't do what I consider a "restart" but a > > > > > > > > > > "take the brakes off and let it run". :-) > > > > > > > > > > IBM hardware support found the wait state and informed us of it. > > > > > > > > > > Thanks everybody for the bandwidth and suggestions. > > > > > > > > > > Rex > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: IBM Mainframe Discussion List [email protected] On > > > > > > > > > > Behalf Of kekronbekron > > > > > > > > > > Sent: Wednesday, March 17, 2021 8:50 AM > > > > > > > > > > To: [email protected] > > > > > > > > > > Subject: [External] Re: z14 HMC log information > > > > > > > > > > Please do share when you find out! > > > > > > > > > > I wonder if an LPAR with full BCPii authority over the box can > > > > > silently query/log information from the HMC for monitoring, i.e., > > > > > actions occuring not via BCPii itself, but just accessing HMC logs. > > > > > > > > > > - KB > > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > > > > > On Wednesday, March 17, 2021 6:17 PM, Carmen Vitullo > > > > > [email protected] wrote: > > > > > > > > > > > so the only other thing I can think of is if you are in a sysplex > > > > > > and your SFM policy took the system out of the plex, XCF may have > > > > > > not seen a heartbeat in a long enough time it partitioned the > > > > > > system from the plex, I've had this happen before, you should see > > > > > > some IXC messages if this happened. > > > > > > > > > > > > when this happened to me there was no indication anyone did > > > > > > > > > > > > anything the system was just removed from the plex > > > > > > > > > > > > Carmen Vitullo > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > From: Rex [email protected] > > > > > > > > > > > > To: IBM-MAIN [email protected] > > > > > > > > > > > > Date: Tuesday, 16 March 2021 4:33 PM CDT > > > > > > > > > > > > Subject: Re: z14 HMC log information Nothing out of the ordinary in > > > > > > > > > > > > the HMC logs. > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > From: IBM Mainframe Discussion List [email protected] On > > > > > > > > > > > > Behalf Of Carmen Vitullo > > > > > > > > > > > > Sent: Tuesday, March 16, 2021 1:10 PM > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > Subject: [External] Re: z14 HMC log information so it looks like > > > > > > > > > > > > one place to check is from the HMC, /console actions/View control > > > > > > > > > > > > tasks performed, this may get you what you need another place is > > > > > > > > > > > > from console actions, View console events > > > > > > > > > > > > Carmen Vitullo > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > From: Rex [email protected] > > > > > > > > > > > > To: IBM-MAIN [email protected] > > > > > > > > > > > > Date: Tuesday, 16 March 2021 12:51 PM CDT > > > > > > > > > > > > Subject: Re: z14 HMC log information I found the log - nothing in > > > > > > > > > > > > it. Heading off to IBM. > > > > > > > > > > > > Rex > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > From: IBM Mainframe Discussion List [email protected] On > > > > > > > > > > > > Behalf Of Pommier, Rex > > > > > > > > > > > > Sent: Tuesday, March 16, 2021 12:24 PM > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > Subject: [External] z14 HMC log information Hi all, Probably a > > > > > > > > > > > > simple question but I'm not having any luck looking for it so I'm > > > > > > asking here. > > > > > > > > > > > > Is there a centralized place on a z14 HMC to show activity > > > > > > occurring, LPAR activation/deactivation, pretty much anything that > > > > > > happens on an HMC or on a z. We just took a hit where one of our > > > > > > LPARs and we're not seeing anything. No hardware messages, no > > > > > > dumps, no nothing. The syslog on the LPAR shows everything running > > > > > > as normal then it just stopped, 15 minutes later are the IPL > > > > > > starting messages. It almost looks like somebody hit the > > > > > > "deactivate" button on the HMC so I'm looking for any log info from > > > > > > a hardware point of view. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Rex > > For IBM-MAIN subscribe / signoff / archive access instructions, > > send email to [email protected] with the message: INFO IBM-MAIN > > The information contained in this message is confidential, protected from > disclosure and may be legally privileged. If the reader of this message is > not the intended recipient or an employee or agent responsible for delivering > this message to the intended recipient, you are hereby notified that any > disclosure, distribution, copying, or any action taken or action omitted in > reliance on it, is strictly prohibited and may be unlawful. If you have > received this communication in error, please notify us immediately by > replying to this message and destroy the material in its entirety, whether in > electronic or hard copy format. Thank you. > > > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > For IBM-MAIN subscribe / signoff / archive access instructions, > > send email to [email protected] with the message: INFO IBM-MAIN ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
