yeah - the IMPL and EPO were not placed in a great place, starting with from what I recall the 370/158 to to 30xx series. then they moved the operators consoles to another room or partitioned the console area away from us operators :) I remember an operator who got an award for coming up with the idea of using the 'Sheldon shield' over the EPO over all the DASD and tape controllers also our cleaning folks were accidentally hitting the EPO while sweeping between the rows of DASD controllers Carmen Vitullo
-----Original Message----- From: Mark <[email protected]> To: IBM-MAIN <[email protected]> Date: Wednesday, 24 March 2021 12:58 PM CDT Subject: Re: [External] Re: z14 HMC log information Looking in my wayback machine I recall an operator who accidentally pushed the IML button on our 3033 console a couple of times. After the second time, we installed a protective device, and the Sheldon-shield was invented. Mark Jacobs Sent from ProtonMail, Swiss-based encrypted email. GPG Public Key - https://api.protonmail.ch/pks/lookup?op=get&[email protected] ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, March 24th, 2021 at 1:50 PM, Pommier, Rex <[email protected]> wrote: > Hi Radoslaw, > > I knew you meant it as a joke and I took it as such. Hence my smiley face.. > The OPERCMDS class has several entries in it but somehow QUIESCE was missed > from way back when for a specific lock down so it was allowed by a more > generic profile. > > I checked the type80 records and there was nothing immediately before the > quiesce command was entered. > > O well, we figured out what happened and put security in place to minimize > the possibility of it happening again, and we now know what to do if it does > happen so we can get the system back up without issue and be able to find and > train the guilty party. > > Rex > > -----Original Message----- > > From: IBM Mainframe Discussion List [email protected] On Behalf Of > Radoslaw Skorupka > > Sent: Wednesday, March 24, 2021 10:26 AM > > To: [email protected] > > Subject: [External] Re: z14 HMC log information > > Obviously the part about killing was only poor joke, but there is some sense > hidden in. > > I mean it is good idea to talk to person who did it. Not to punish, but talk > and explain and hear his/her explanations. > > For RACF admin it is quite obvious the security model should be somehow > checked. Again the person can have good explanation of current state of > protection. > > Regarding traces - It is a little bit hard to test, especially without access > to mainframe ;-) but I guess SMF80 can be written just before system freeze. > Note it is RACF security check - it happens BEFORE the command is interpreted > by the system. Simpler example: when you issue CANCEL CICSABC and you don't > have such started task, you first will be checked by RACF (and maybe > rejected) and then the command is really issued, and you will get answer like > "there is no such started task to cancel". > > BTW: I imagined what would happen after such case on production... > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > Radoslaw Skorupka > > (looking for new job) > > Lodz, Poland > > W dniu 24.03.2021 o 14:58, Pommier, Rex pisze: > > > I'm going to agree with most of it. I don't like the part about killing the > > RACF admin. I'm not the one who initially set up the OPERCMDS security but > > I missed the fact the QUIESCE command wasn't set as "don't let anybody > > use". Hari-kari is not on my bucket list. :-) > > > > On to Radoslaw's comment about logging - it is logged, after the fact. > > QUIESCE does exactly that - it stops the LPAR in its tracks. Do not pass > > Go, do not collect $200. No z/OS logging at the time it happens. IBM > > hardware support found and reported the wait state back to us from some > > hardware logs that were forwarded to them from our CE. The z/OS logging > > takes place after the PSW restart from the HMC occurs and yes, it shows the > > console or user that executed the command. However in our case since the > > LPAR stopped in the middle of the day and we had managers breathing down > > our necks to get the system back up we didn't have time to properly > > diagnose until after the fact - which included an IPL which in turn did not > > allow the logging of the quiesce command to take place. > > > > Rex > > > > -----Original Message----- > > > > From: IBM Mainframe Discussion List [email protected] On Behalf Of > > Carmen Vitullo > > > > Sent: Wednesday, March 24, 2021 8:07 AM > > > > To: [email protected] > > > > Subject: [External] Re: z14 HMC log information > > > > agree 100%, when I tested the command on my sandbox system I see my ID in > > the syslog as the culprit :) if done from a console, then the console name > > is shown. > > > > Carmen Vitullo > > > > -----Original Message----- > > > > From: Radoslaw [email protected] > > > > To: IBM-MAIN [email protected] > > > > Date: Wednesday, 24 March 2021 8:01 AM CDT > > > > Subject: Re: z14 HMC log information > > > > IMHO there should be a trace in a syslog. Maybe that part of syslog is > > somehow lost. > > > > And it would be good idea to have SMF record for that command. I don't know > > about console commands, but SMF80 could be cut if you take care about > > AUDIT(ALL(READ)) in advance. I mean RACF profile. > > > > Then find the person and kill him. Don't forget to kill RACF admin who > > forgot to protect the command. Kill'em all. ;-) > > > > Seriously: it is a symptom of bad security model. > > > > -- > > > > Radoslaw Skorupka > > > > (looking for new job) > > > > Lodz, Poland > > > > W dniu 24.03.2021 o 13:54, Pommier, Rex pisze: > > > > > Hi Carmen, > > > > > > We use the quiesce parameter of the reset command as well. I'm thinking > > > that's what happened here as well, but we have no idea "whodunit". We had > > > 1 operator on duty that day and he was off doing something else when it > > > happened so the command made it into the system via an SDSF-like console > > > session. > > > > > > Rex > > > > > > -----Original Message----- > > > > > > From: IBM Mainframe Discussion List [email protected] On > > > > > > Behalf Of Carmen Vitullo > > > > > > Sent: Wednesday, March 24, 2021 7:43 AM > > > > > > To: [email protected] > > > > > > Subject: [External] Re: z14 HMC log information > > > > > > There is a queisce command to put a job to sleep, and I think that's what > > > my operator was thinking he was doing, for a runaway job we do use > > > automation to queisce a job and send us an alert. > > > > > > from SDSF there's a quiesce line command RQ it translates to RESET > > > > > > jobname,QUIESCE > > > > > > Carmen Vitullo > > > > > > -----Original Message----- > > > > > > From: Radoslaw [email protected] > > > > > > To: IBM-MAIN [email protected] > > > > > > Date: Wednesday, 24 March 2021 5:35 AM CDT > > > > > > Subject: Re: z14 HMC log information > > > > > > IMHO not really. > > > > > > By freezing the system you disrupt all the online processing and you > > > loose capability to find out what's wrong. > > > > > > I had to do with CPU hogging tasks and spool 100% filled by some ugly > > > job. In both cases the very first thing to fix it was to logon to the > > > system. > > > > > > -- > > > > > > Radoslaw Skorupka > > > > > > (looking for new job) > > > > > > Lodz, Poland > > > > > > W dniu 24.03.2021 o 03:30, kekronbekron pisze: > > > > > > > It's useful when you want to say "STOP IT!" to run-away CPU or spool > > > > jobs. > > > > > > > > Will buy you some time to investigate. > > > > > > > > - KB > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > > > On Tuesday, March 23, 2021 7:07 PM, Pommier, Rex > > > > [email protected] wrote: > > > > > > > > > Hey Carmen, > > > > > > > > > > At least you had an operator tell you what they did. :-) Once we > > > > > determined what had happened (after a painful middle-of-the-day IPL) > > > > > the only response I got back from anybody was "I didn't do anything > > > > > like that!" But now that I know what that command does and how it > > > > > works I doubt I'll ever forget this one. > > > > > > > > > > Rex > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: IBM Mainframe Discussion List [email protected] On > > > > > > > > > > Behalf Of Carmen Vitullo > > > > > > > > > > Sent: Tuesday, March 23, 2021 7:47 AM > > > > > > > > > > To: [email protected] > > > > > > > > > > Subject: Re: [External] Re: z14 HMC log information > > > > > > > > > > WOW! this takes me back 20 years when an operator wanted to quiesce > > > > > > > > > > a job and issues that command from the console he called my told me > > > > > > > > > > what he did, I researched what that command does, seemed to me it > > > > > > > > > > was like on the old system consoles O2 and O3 IIRC one was a PSW > > > > > > > > > > stop and one a restart, I found the restart just for grins and it > > > > > > > > > > worked, been a long time since I've hear tell someone had that same > > > > > > > > > > experience great find > > > > > > > > > > Carmen Vitullo > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: Rex [email protected] > > > > > > > > > > To: IBM-MAIN [email protected] > > > > > > > > > > Date: Monday, 22 March 2021 5:18 PM CDT > > > > > > > > > > Subject: Re: [External] Re: z14 HMC log information > > > > > > > > > > OK, this is embarrassing. We found the problem - but don't know the > > > > > > > > > > culprit. We discovered the quiesce command wasn't locked down and > > > > > > > > > > somebody somewhere keyed it in. The reason we don't know who/where > > > > > > > > > > is because our SDSF-like product had the capability to also do a > > > > > > > > > > quiesce and since we didn't know that's what caused it, we IPLed the > > > > > > > > > > LPAR. I did some research after the fact to determine just how > > > > > > > > > > quiesce works (been working on these things for 30 years and never > > > > > > > > > > had a reason to even play with quiesce). Now I know - and I know how > > > > > > > > > > to make the machine go again, picking up where it stopped. PSW > > > > > > > > > > Restart from the HMC doesn't do what I consider a "restart" but a > > > > > > > > > > "take the brakes off and let it run". :-) > > > > > > > > > > IBM hardware support found the wait state and informed us of it. > > > > > > > > > > Thanks everybody for the bandwidth and suggestions. > > > > > > > > > > Rex > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: IBM Mainframe Discussion List [email protected] On > > > > > > > > > > Behalf Of kekronbekron > > > > > > > > > > Sent: Wednesday, March 17, 2021 8:50 AM > > > > > > > > > > To: [email protected] > > > > > > > > > > Subject: [External] Re: z14 HMC log information > > > > > > > > > > Please do share when you find out! > > > > > > > > > > I wonder if an LPAR with full BCPii authority over the box can > > > > > silently query/log information from the HMC for monitoring, i.e., > > > > > actions occuring not via BCPii itself, but just accessing HMC logs. > > > > > > > > > > - KB > > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > > > > > On Wednesday, March 17, 2021 6:17 PM, Carmen Vitullo > > > > > [email protected] wrote: > > > > > > > > > > > so the only other thing I can think of is if you are in a sysplex > > > > > > and your SFM policy took the system out of the plex, XCF may have > > > > > > not seen a heartbeat in a long enough time it partitioned the > > > > > > system from the plex, I've had this happen before, you should see > > > > > > some IXC messages if this happened. > > > > > > > > > > > > when this happened to me there was no indication anyone did > > > > > > > > > > > > anything the system was just removed from the plex > > > > > > > > > > > > Carmen Vitullo > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > From: Rex [email protected] > > > > > > > > > > > > To: IBM-MAIN [email protected] > > > > > > > > > > > > Date: Tuesday, 16 March 2021 4:33 PM CDT > > > > > > > > > > > > Subject: Re: z14 HMC log information Nothing out of the ordinary in > > > > > > > > > > > > the HMC logs. > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > From: IBM Mainframe Discussion List [email protected] On > > > > > > > > > > > > Behalf Of Carmen Vitullo > > > > > > > > > > > > Sent: Tuesday, March 16, 2021 1:10 PM > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > Subject: [External] Re: z14 HMC log information so it looks like > > > > > > > > > > > > one place to check is from the HMC, /console actions/View control > > > > > > > > > > > > tasks performed, this may get you what you need another place is > > > > > > > > > > > > from console actions, View console events > > > > > > > > > > > > Carmen Vitullo > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > From: Rex [email protected] > > > > > > > > > > > > To: IBM-MAIN [email protected] > > > > > > > > > > > > Date: Tuesday, 16 March 2021 12:51 PM CDT > > > > > > > > > > > > Subject: Re: z14 HMC log information I found the log - nothing in > > > > > > > > > > > > it. Heading off to IBM. > > > > > > > > > > > > Rex > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > From: IBM Mainframe Discussion List [email protected] On > > > > > > > > > > > > Behalf Of Pommier, Rex > > > > > > > > > > > > Sent: Tuesday, March 16, 2021 12:24 PM > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > Subject: [External] z14 HMC log information Hi all, Probably a > > > > > > > > > > > > simple question but I'm not having any luck looking for it so I'm > > > > > > asking here. > > > > > > > > > > > > Is there a centralized place on a z14 HMC to show activity > > > > > > occurring, LPAR activation/deactivation, pretty much anything that > > > > > > happens on an HMC or on a z. We just took a hit where one of our > > > > > > LPARs and we're not seeing anything. No hardware messages, no > > > > > > dumps, no nothing. The syslog on the LPAR shows everything running > > > > > > as normal then it just stopped, 15 minutes later are the IPL > > > > > > starting messages. It almost looks like somebody hit the > > > > > > "deactivate" button on the HMC so I'm looking for any log info from > > > > > > a hardware point of view. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Rex > > For IBM-MAIN subscribe / signoff / archive access instructions, > > send email to [email protected] with the message: INFO IBM-MAIN > > The information contained in this message is confidential, protected from > disclosure and may be legally privileged. If the reader of this message is > not the intended recipient or an employee or agent responsible for delivering > this message to the intended recipient, you are hereby notified that any > disclosure, distribution, copying, or any action taken or action omitted in > reliance on it, is strictly prohibited and may be unlawful. If you have > received this communication in error, please notify us immediately by > replying to this message and destroy the material in its entirety, whether in > electronic or hard copy format. Thank you. > > > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > For IBM-MAIN subscribe / signoff / archive access instructions, > > send email to [email protected] with the message: INFO IBM-MAIN ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
