Thanks Daniel. We're just a bunch of SCCM "slobs". Been running SCCM 2012 for just over a month and it looks like a frat house in here. :)
From: [email protected] [mailto:[email protected]] On Behalf Of Daniel Ratliff Sent: Monday, January 19, 2015 10:04 AM To: [email protected] Subject: [mssms] RE: CM12 - No Task Sequences Available - Finally Solved Ah yeah that makes sense. Despite our large environment, we only have about 50 task sequence deployments total, and don't use TS for app installs, besides pre-caching to the Nomad masters. Great troubleshooting! Daniel Ratliff From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Marable, Mike Sent: Monday, January 19, 2015 9:54 AM To: [email protected]<mailto:[email protected]> Subject: [mssms] RE: CM12 - No Task Sequences Available - Finally Solved We did it to ourselves really. We have a process by which machines can be scheduled to rebuild overnight. Essentially users can select 1 of (I believe) 4 different time slots and a choice of 3 different USMT options. Each timeslot + USMT option combo is a separate deployment of our production build. Every night we have at least 12 of those deployments. Well, we hadn't been real good about cleaning up after ourselves and nobody was removing the old, outdated deployments and left them lingering around. That added maybe a few hundred or so deployments to the mix. We also use task sequences for things other than builds. We have a nightly maintenance sequence that runs at the end of business on all of our machines. There are maybe a dozen packages that are shared between the build and maintenance sequences. We have deployments for this maintenance sequence for all the various EOB times for our different locations, 7 days a week. That adds a couple hundred more deployments to our mix. It was this vast number of deployments that brought the house down. It was like a series of nested FOREACH statements.... FOREACH package in the task sequence FIND all other task sequences that reference the package FOREACH of those task sequences FIND each deployment for that task sequence FOREACH deployment Update policy That's a lot of multipliers there. That's what ultimately killed us, the large number of deployments (~500) we had lingering around, most of which were out of date. Had we been better about cleaning up old, out of date deployments I don't think we would have ever had an issue. Mike From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Daniel Ratliff Sent: Monday, January 19, 2015 8:37 AM To: [email protected]<mailto:[email protected]> Subject: [mssms] RE: CM12 - No Task Sequences Available - Finally Solved Mike, that is pretty interesting. We use Nomad as well, modify task sequences with over 100 package references and have never had policy churn like this before. I wonder if perhaps you have some resource constraints on your MPs? Daniel Ratliff From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Marable, Mike Sent: Friday, January 16, 2015 2:12 PM To: '[email protected]' Subject: [mssms] CM12 - No Task Sequences Available - Finally Solved We finally got this one solved. Here is what happened. We use Nomad Branch in our environment. On the properties of a task sequence 1E adds a tab that allows you to set the Nomad settings for all packages directly referenced by that task sequence. At each outage someone (that would be me) was working on new task sequences and made sure that Nomad was enabled through that tab on the task sequence properties. This is what set an avalanche into motion. In the SMS_Distribution_Manager log you will see a flurry of 2300 ("Distribution Manager is beginning to process package..."), 2311 (Distribution Manager successfully created or updated package...") and 2301 ("Distribution Manager successfully processed package...") entries. One for every package referenced by the task sequence. In our case that was about 112 packages. Soon we begin to see in the SMS_Policy_Povider log 5101 messages ("Policy Provider successfully updated the policy and the policy assignment that were created from package..."). One for every package referenced by the task sequence and (here is where things start to get bad) every advertisement that references that task sequence. So, if we have 100 packages in the TS and the TS is deployed to 10 collections that's 1000 policy updates. Wait, there's more. Things are about to go from bad to worse..... If you have a package that is also referenced in another task sequence, all of the deployments of that task sequences get policy updates. So, let's say you have a prior version of the build with those same 100 packages that is also advertised to 10 collections. That's another 1000 policy updates. Now we use task sequences for our nightly maintenance routines. Some packages are shared between our build and maintenance sequences (scripts to install printers for example). Each shared package then triggers a policy update for each and every nightly maintenance advertisement. We run them every half hour at the end of business so that's another 16 deployments * 7 days a week (112) * maybe 10 packages shared is another 1000+ policy updates. So this all quickly spirals out of control as every single package referenced in the first task sequence then goes out and triggers its own policy refresh for every instance of any deployment of a task sequence that that also references it. All told that one little checkbox generated nearly 40,000 5101 policy updates over the course of 5 hours and knocked out builds out of the water. Learn something new every day. Mike Marable Application Programmer/Analyst Lead Enterprise Device Engineering and Management MCTS, MCITP, MCSA, MS [Profile<https://www.mcpvirtualbusinesscard.com/VBCServer/MikeMarable/profile>] [Blog<http://thesystemsmonkey.wordpress.com/>] -------------------------------------------- "The difficult we do at once. The impossible takes a little longer." -US Army Corps of Engineers "It is better to have less thunder in the mouth and more lightning in the hand." -Apache Proverb I will rise when I have fallen. "Unless you try to do something beyond what you have already mastered, you will never grow." -Ralph Waldo Emerson ********************************************************** Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues The information transmitted is intended only for the person or entity to which it is addressed and may contain CONFIDENTIAL material. If you receive this material/information in error, please contact the sender and delete or destroy the material/information. ********************************************************** Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues The information transmitted is intended only for the person or entity to which it is addressed and may contain CONFIDENTIAL material. If you receive this material/information in error, please contact the sender and delete or destroy the material/information. ********************************************************** Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues
