That's what I would call "more doco" (process needs to be defined, exception process needs to be defined etc.), not to mention additional bureaucratic overhead. Eventually it becomes meaningless because the supervisors have so much to do (my guess is that they are already busy) then they just start signing things without properly checking each one.
If every single major (but unlikely) risk was managed like this, nothing would get done in a timely manner. At least for the environment I work in, where it takes months to put something into the environment, the business is screaming for more agility. They would argue "why can't I get this service from this external provider - they can turn it around in a few days, and they'll manage the risk of something going wrong" So, I can see two solutions; - build or buy a product that can stop this happening. However, if not properly implemented, all you've done is move the risk to another step. This is both the product, and elucidation of the correct risks to be eliminated. Develop team members who are across multiple towers, so they can prepare and sign off on changes that impact multiple towers - Outsource to someone who can do it properly As a general rule, people here don't seem to like "the cloud", or outsource providers. But on the other side of the coin, the business doesn't like IT - it has lumpy expenditures, it's slow and it can't deliver (or delivers clusterf*cks like this). So, how do we fix this? Checklists are not the answer. Cheers Ken From: [email protected] [mailto:[email protected]] On Behalf Of Micheal Espinola Jr Sent: Monday, 19 May 2014 4:31 PM To: ntsysadm Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. Perhaps something you seem to less of in IT nowadays: Procedural checklists? Supervisory/coworker sign-offs/verifications? -- Espi On Sun, May 18, 2014 at 10:49 PM, Ken Schaefer <[email protected]<mailto:[email protected]>> wrote: I would say that most of us are guilty of not following or reading docs after we feel comfortable doing something, had success with it or have plenty of experience in other areas and with other tools. How do you intend to inform people of these verification steps? More doco I'm guessing And how are you going to implement the verification step? Someone checking a checkbox or clicking a button? And how are you going to record the verification step was done? More post implementation doco? Regards Ken From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of CESAR.ABREG0 Sent: Monday, 19 May 2014 2:27 PM To: [email protected]<mailto:[email protected]> Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. Would I agree that most or any of this things may not work but when some one tends to ignore documentation or processes nothing would work. I would say that most of us are guilty of not following or reading docs after we feel comfortable doing something,had success with it or have plenty of experience in other areas and with other tools. (specially when those docs are written by a person that does not understand product. ) reason that when we buy a new TV we don bother to read the provided paperwork and don't use it to the fullest potential. An example. We would hand out a page of step/guide to our docs department, they would turn it into 10 pages that hid the relevant items/steps. I'm not a good writer or deep reader and would say over 50% of IT personnel are not either, based on my experience. I truly believe that most writer write to impress people and not to teach or guide and most companies have documentation to fill a requirement and not for the value that could provide. Just based on my personal access and experience. I've worked with SCCM for a while and many people do not understand how powerful the tool is, therefore not putting enough thought on implementation. If you think about it, this tool can bypass or circumvent almost any security tools you have in place. Based on this example and past experiences that have been seen, putting extra validation/steps that execute at runtime are the most ideal to me. Docs may work up to an extend when followed. Cesar A. Meaning is NOT in words, but inside people! Dr. Myles Munroe My iPad takes half the blame for misspells. On May 18, 2014, at 7:17 PM, Ken Schaefer <[email protected]<mailto:[email protected]>> wrote: Personally, I don't think any of these things will help. When creating a change record, the exact steps to be followed are documented. If someone either: a) Creates the wrong documentation, and it's approved by CAB b) Creates the right documentation, but someone either fat fingers or doesn't read the doco Then creating these extra steps is just process inflation. I don't think adding more steps or manual checks to process is the right answer. Especially in a world where business is clamouring for more agility and speed, rather than more bureaucracy in the name of risk management. If you look at the stuff coming out of CEB or Gartner, we need things like leaner processes, cross-skilled teams better able to understand implications across multiple towers, orchestration/automation tied to process and bunch of other things I don't remember off the top-of-my-head. Cheers Ken From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of CESAR.ABREG0 Sent: Monday, 19 May 2014 12:07 PM To: [email protected]<mailto:[email protected]> Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. Verify the number of clients in collection before using to deploy a TS to it? Verify that the dynamic collection being use contains the intended clients? Verify that the 'all system' collection is not a target? There could be more but a couple of that I can think of. Most this situations happen by human errors and inexperienced as well. I think HP consulting did it at a bank a couple of years ago and some that colleagues have shared with me that happened in a USA government branch. I've been doing imaging over 10 years and I never do mandatory deployments to populated collections, only to empty ones and I add clients manually or have a process to do so. This got me thinking of steps that can be taken or be part of a TS to prevent this type of situation up to an extend, can't never be prevented completely. 1. Put a step that verify DCs and other critical infrastructure systems and have human click yes before moving forward or fail if no response. 2. Creat web service/orchestrator to send email or a type of notification to a group before continuing. Automated. 3. What I've used in the past. Create an empty collection, deploy TS to it as mandatory, add required systems manually or by script from a list. Limit who can add systems and the type of client, like no DCs or SCCM systems. Cesar A. Meaning is NOT in words, but inside people! Dr. Myles Munroe My iPad takes half the blame for misspells. On May 18, 2014, at 5:31 PM, Ken Schaefer <[email protected]<mailto:[email protected]>> wrote: I'm assuming someone clicked the wrong button (i.e. "Finished", when they should've clicked "Cancel"). How does "process verification" (how do you define this?) help? Cheers Ken From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Rankin, James R Sent: Monday, 19 May 2014 2:59 AM To: [email protected]<mailto:[email protected]> Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. I think I may use this as an example in an article about the importance of process verification. Sent from my (new!) BlackBerry, which may make me an antiques dealer, but it's reliable as hell for email delivery :-) ________________________________ From: "Andrew S. Baker" <[email protected]<mailto:[email protected]>> Sender: [email protected]<mailto:[email protected]> Date: Sun, 18 May 2014 12:55:37 -0400 To: ntsysadm<[email protected]<mailto:[email protected]>> ReplyTo: [email protected]<mailto:[email protected]> Subject: Re: [NTSysADM] A Windows 7 image was deployed to EVERYTHING. Automation leads to relaxation... ...unless something goes horribly wrong. ASB http://XeeMe.com/AndrewBaker<http://xeeme.com/AndrewBaker> Providing Virtual CIO Services (IT Operations & Information Security) for the SMB market... On Fri, May 16, 2014 at 10:40 PM, Richard Stovall <[email protected]<mailto:[email protected]>> wrote: Wowzers. That's just incredible. On May 16, 2014 8:14 PM, "Kennedy, Jim" <[email protected]<mailto:[email protected]>> wrote: So SCCM sent win 7 to everything, including servers. http://it.emory.edu/windows7-incident/
