The link doesn't work for me, but.... Seems to me a simple breaking up of delegation of servers/desktops would have prevented this. Only in a very small environment (SMB) would I expect one console to be configured to be able to push *anything* to both servers and workstations at the same time. WSUS ***MAAAYBE***, but even in a bigger environment I'd have servers talking to one WSUS server and desktops to another, just to avoid this kind of thing.
That's how I've seen it at my last two %dayjobs% and it makes sense to me. Dave > That's what I would call "more doco" (process needs to be defined, > exception process needs to be defined etc.), not to mention additional > bureaucratic overhead. Eventually it becomes meaningless because the > supervisors have so much to do (my guess is that they are already busy) > then they just start signing things without properly checking each one. > > If every single major (but unlikely) risk was managed like this, nothing > would get done in a timely manner. At least for the environment I work in, > where it takes months to put something into the environment, the business > is screaming for more agility. They would argue "why can't I get this > service from this external provider - they can turn it around in a few > days, and they'll manage the risk of something going wrong" > > So, I can see two solutions; > > - build or buy a product that can stop this happening. However, if > not properly implemented, all you've done is move the risk to another > step. This is both the product, and elucidation of the correct risks to be > eliminated. Develop team members who are across multiple towers, so they > can prepare and sign off on changes that impact multiple towers > > - Outsource to someone who can do it properly > > As a general rule, people here don't seem to like "the cloud", or > outsource providers. But on the other side of the coin, the business > doesn't like IT - it has lumpy expenditures, it's slow and it can't > deliver (or delivers clusterf*cks like this). So, how do we fix this? > Checklists are not the answer. > > Cheers > Ken > > From: [email protected] > [mailto:[email protected]] On Behalf Of Micheal Espinola Jr > Sent: Monday, 19 May 2014 4:31 PM > To: ntsysadm > Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > Perhaps something you seem to less of in IT nowadays: Procedural > checklists? Supervisory/coworker sign-offs/verifications? > > -- > Espi > > > On Sun, May 18, 2014 at 10:49 PM, Ken Schaefer > <[email protected]<mailto:[email protected]>> wrote: > I would say that most of us are guilty of not following or reading docs > after we feel comfortable doing something, had success with it or have > plenty of experience in other areas and with other tools. > > How do you intend to inform people of these verification steps? More doco > I'm guessing > And how are you going to implement the verification step? Someone checking > a checkbox or clicking a button? > And how are you going to record the verification step was done? More post > implementation doco? > > Regards > Ken > > > From: > [email protected]<mailto:[email protected]> > [mailto:[email protected]<mailto:[email protected]>] > On Behalf Of CESAR.ABREG0 > Sent: Monday, 19 May 2014 2:27 PM > > To: [email protected]<mailto:[email protected]> > Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > Would I agree that most or any of this things may not work but when some > one tends to ignore documentation or processes nothing would work. I would > say that most of us are guilty of not following or reading docs after we > feel comfortable doing something,had success with it or have plenty of > experience in other areas and with other tools. (specially when those docs > are written by a person that does not understand product. ) reason that > when we buy a new TV we don bother to read the provided paperwork and > don't use it to the fullest potential. > > An example. We would hand out a page of step/guide to our docs department, > they would turn it into 10 pages that hid the relevant items/steps. > > I'm not a good writer or deep reader and would say over 50% of IT > personnel are not either, based on my experience. I truly believe that > most writer write to impress people and not to teach or guide and most > companies have documentation to fill a requirement and not for the value > that could provide. Just based on my personal access and experience. > > I've worked with SCCM for a while and many people do not understand how > powerful the tool is, therefore not putting enough thought on > implementation. If you think about it, this tool can bypass or circumvent > almost any security tools you have in place. > > Based on this example and past experiences that have been seen, putting > extra validation/steps that execute at runtime are the most ideal to me. > Docs may work up to an extend when followed. > > Cesar A. > Meaning is NOT in words, but inside people! Dr. Myles Munroe > My iPad takes half the blame for misspells. > > On May 18, 2014, at 7:17 PM, Ken Schaefer > <[email protected]<mailto:[email protected]>> wrote: > Personally, I don't think any of these things will help. > > When creating a change record, the exact steps to be followed are > documented. If someone either: > > a) Creates the wrong documentation, and it's approved by CAB > > b) Creates the right documentation, but someone either fat fingers or > doesn't read the doco > Then creating these extra steps is just process inflation. > > I don't think adding more steps or manual checks to process is the right > answer. Especially in a world where business is clamouring for more > agility and speed, rather than more bureaucracy in the name of risk > management. > > If you look at the stuff coming out of CEB or Gartner, we need things like > leaner processes, cross-skilled teams better able to understand > implications across multiple towers, orchestration/automation tied to > process and bunch of other things I don't remember off the top-of-my-head. > > Cheers > Ken > > From: > [email protected]<mailto:[email protected]> > [mailto:[email protected]] On Behalf Of CESAR.ABREG0 > Sent: Monday, 19 May 2014 12:07 PM > To: [email protected]<mailto:[email protected]> > Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > Verify the number of clients in collection before using to deploy a TS to > it? Verify that the dynamic collection being use contains the intended > clients? Verify that the 'all system' collection is not a target? > There could be more but a couple of that I can think of. > > Most this situations happen by human errors and inexperienced as well. I > think HP consulting did it at a bank a couple of years ago and some that > colleagues have shared with me that happened in a USA government branch. > I've been doing imaging over 10 years and I never do mandatory deployments > to populated collections, only to empty ones and I add clients manually or > have a process to do so. > > This got me thinking of steps that can be taken or be part of a TS to > prevent this type of situation up to an extend, can't never be prevented > completely. > > 1. Put a step that verify DCs and other critical infrastructure systems > and have human click yes before moving forward or fail if no response. > 2. Creat web service/orchestrator to send email or a type of notification > to a group before continuing. Automated. > 3. What I've used in the past. Create an empty collection, deploy TS to it > as mandatory, add required systems manually or by script from a list. > Limit who can add systems and the type of client, like no DCs or SCCM > systems. > > Cesar A. > Meaning is NOT in words, but inside people! Dr. Myles Munroe > My iPad takes half the blame for misspells. > > On May 18, 2014, at 5:31 PM, Ken Schaefer > <[email protected]<mailto:[email protected]>> wrote: > I'm assuming someone clicked the wrong button (i.e. "Finished", when they > should've clicked "Cancel"). How does "process verification" (how do you > define this?) help? > > Cheers > Ken > > From: > [email protected]<mailto:[email protected]> > [mailto:[email protected]] On Behalf Of Rankin, James R > Sent: Monday, 19 May 2014 2:59 AM > To: [email protected]<mailto:[email protected]> > Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > I think I may use this as an example in an article about the importance of > process verification. > Sent from my (new!) BlackBerry, which may make me an antiques dealer, but > it's reliable as hell for email delivery :-) > ________________________________ > From: "Andrew S. Baker" <[email protected]<mailto:[email protected]>> > Sender: > [email protected]<mailto:[email protected]> > Date: Sun, 18 May 2014 12:55:37 -0400 > To: > ntsysadm<[email protected]<mailto:[email protected]>> > ReplyTo: [email protected]<mailto:[email protected]> > Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. > > Automation leads to relaxation... > ...unless something goes horribly wrong. > > > > > > > ASB > http://XeeMe.com/AndrewBaker<http://xeeme.com/AndrewBaker> > Providing Virtual CIO Services (IT Operations & Information Security) for > the SMB market... > > > > > On Fri, May 16, 2014 at 10:40 PM, Richard Stovall > <[email protected]<mailto:[email protected]>> wrote: > > Wowzers. That's just incredible. > On May 16, 2014 8:14 PM, "Kennedy, Jim" > <[email protected]<mailto:[email protected]>> wrote: > So SCCM sent win 7 to everything, including servers. > > http://it.emory.edu/windows7-incident/ > > > > >
