I do much better work after a liquid lunch... On May 19, 2014 7:34 AM, "Steven M. Caesare" <[email protected]> wrote:
> Understanding _WHERE_ a person wasn't paying attention is important in > doing a reoot cause analysis that will in turn determine of process > refinement is needed. > > Folks here have jumped to the latter without having any idea if the issue > was a fat-finger, accidentally targeting a wrong collection, not adhering > to existing process, the wrong button on a KVM switch, inadvertently > working on the prod rather than test infrastructure, action after a long > liquid lunch, etc... > > -sc > > > > -----Original Message----- > From: [email protected] on behalf of David McSpadden > Sent: Mon 5/19/2014 10:22 AM > To: '[email protected]' > Subject: RE: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > I would think we all understand that somebody had glazed over eyes and > just wasn't paying attention. > Sounds like they have a good process in place. It's just that those > damned Human Interface Devices were approved to Interface with the system. > Had those been on the unapproved to make changes list we wouldn't even be > talking about their oooops (or Oh Shit) even today. > :-) > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of Steven M. Caesare > Sent: Monday, May 19, 2014 10:18 AM > To: [email protected] > Subject: RE: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > Indeed, but that's a result, not a root cause. > > -sc > > > > -----Original Message----- > From: [email protected] on behalf of Melvin Backus > Sent: Mon 5/19/2014 10:05 AM > To: [email protected] > Subject: RE: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > I think it can safely be assumed (yes I know) that a deployment server > should never overwrite itself as part of a deployment. > > -- > There are 10 kinds of people in the world... > those who understand binary and those who don't. > > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of Steven M. Caesare > Sent: Monday, May 19, 2014 9:15 AM > To: [email protected] > Subject: RE: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > How does one know if process refinement is necessary when you don't know > the root cause of the issue (yet)? > > -sc > > > > -----Original Message----- > From: [email protected] on behalf of Ken Schaefer > Sent: Sun 5/18/2014 10:17 PM > To: [email protected] > Subject: RE: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > Personally, I don't think any of these things will help. > > When creating a change record, the exact steps to be followed are > documented. If someone either: > > a) Creates the wrong documentation, and it's approved by CAB > > b) Creates the right documentation, but someone either fat fingers or > doesn't read the doco > Then creating these extra steps is just process inflation. > > I don't think adding more steps or manual checks to process is the right > answer. Especially in a world where business is clamouring for more agility > and speed, rather than more bureaucracy in the name of risk management. > > If you look at the stuff coming out of CEB or Gartner, we need things like > leaner processes, cross-skilled teams better able to understand > implications across multiple towers, orchestration/automation tied to > process and bunch of other things I don't remember off the top-of-my-head. > > Cheers > Ken > > From: [email protected] [mailto: > [email protected]] On Behalf Of CESAR.ABREG0 > Sent: Monday, 19 May 2014 12:07 PM > To: [email protected] > Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > Verify the number of clients in collection before using to deploy a TS to > it? Verify that the dynamic collection being use contains the intended > clients? Verify that the 'all system' collection is not a target? > There could be more but a couple of that I can think of. > > Most this situations happen by human errors and inexperienced as well. I > think HP consulting did it at a bank a couple of years ago and some that > colleagues have shared with me that happened in a USA government branch. > I've been doing imaging over 10 years and I never do mandatory deployments > to populated collections, only to empty ones and I add clients manually or > have a process to do so. > > This got me thinking of steps that can be taken or be part of a TS to > prevent this type of situation up to an extend, can't never be prevented > completely. > > 1. Put a step that verify DCs and other critical infrastructure systems > and have human click yes before moving forward or fail if no response. > 2. Creat web service/orchestrator to send email or a type of notification > to a group before continuing. Automated. > 3. What I've used in the past. Create an empty collection, deploy TS to it > as mandatory, add required systems manually or by script from a list. Limit > who can add systems and the type of client, like no DCs or SCCM systems. > > Cesar A. > Meaning is NOT in words, but inside people! Dr. Myles Munroe My iPad takes > half the blame for misspells. > > On May 18, 2014, at 5:31 PM, Ken Schaefer <[email protected]<mailto: > [email protected]>> wrote: > I'm assuming someone clicked the wrong button (i.e. "Finished", when they > should've clicked "Cancel"). How does "process verification" (how do you > define this?) help? > > Cheers > Ken > > From: [email protected]<mailto:[email protected]> > [mailto:[email protected]] On Behalf Of Rankin, James R > Sent: Monday, 19 May 2014 2:59 AM > To: [email protected]<mailto:[email protected]> > Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > I think I may use this as an example in an article about the importance of > process verification. > Sent from my (new!) BlackBerry, which may make me an antiques dealer, but > it's reliable as hell for email delivery :-) > ________________________________ > From: "Andrew S. Baker" <[email protected]<mailto:[email protected]>> > Sender: [email protected]<mailto: > [email protected]> > Date: Sun, 18 May 2014 12:55:37 -0400 > To: ntsysadm<[email protected]<mailto: > [email protected]>> > ReplyTo: [email protected]<mailto:[email protected]> > Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > Automation leads to relaxation... > ...unless something goes horribly wrong. > > > > > > > ASB > http://XeeMe.com/AndrewBaker<http://xeeme.com/AndrewBaker> > Providing Virtual CIO Services (IT Operations & Information Security) for > the SMB market. > > > > > On Fri, May 16, 2014 at 10:40 PM, Richard Stovall <[email protected] > <mailto:[email protected]>> wrote: > > Wowzers. That's just incredible. > On May 16, 2014 8:14 PM, "Kennedy, Jim" <[email protected] > <mailto:[email protected]>> wrote: > So SCCM sent win 7 to everything, including servers. > > http://it.emory.edu/windows7-incident/ > > > > > > > > This e-mail and any files transmitted with it are property of Indiana > Members Credit Union, are confidential, and are intended solely for the use > of the individual or entity to whom this e-mail is addressed. If you are > not one of the named recipient(s) or otherwise have reason to believe that > you have received this message in error, please notify the sender and > delete this message immediately from your computer. Any other use, > retention, dissemination, forwarding, printing, or copying of this email is > strictly prohibited. > > Please consider the environment before printing this email. > > > > > >
