We did it to ourselves really.
We have a process by which machines can be scheduled to rebuild overnight.
Essentially users can select 1 of (I believe) 4 different time slots and a
choice of 3 different USMT options. Each timeslot + USMT option combo is a
separate deployment of our production build. Every night we have at least 12
of those deployments. Well, we hadn't been real good about cleaning up after
ourselves and nobody was removing the old, outdated deployments and left them
lingering around. That added maybe a few hundred or so deployments to the mix.
We also use task sequences for things other than builds. We have a nightly
maintenance sequence that runs at the end of business on all of our machines.
There are maybe a dozen packages that are shared between the build and
maintenance sequences. We have deployments for this maintenance sequence for
all the various EOB times for our different locations, 7 days a week. That
adds a couple hundred more deployments to our mix.
It was this vast number of deployments that brought the house down. It was
like a series of nested FOREACH statements....
FOREACH package in the task sequence
FIND all other task sequences that reference the package
FOREACH of those task sequences
FIND each deployment for that
task sequence
FOREACH
deployment
Update policy
That's a lot of multipliers there. That's what ultimately killed us, the large
number of deployments (~500) we had lingering around, most of which were out of
date.
Had we been better about cleaning up old, out of date deployments I don't think
we would have ever had an issue.
Mike
From: [email protected] [mailto:[email protected]] On
Behalf Of Daniel Ratliff
Sent: Monday, January 19, 2015 8:37 AM
To: [email protected]
Subject: [mssms] RE: CM12 - No Task Sequences Available - Finally Solved
Mike, that is pretty interesting. We use Nomad as well, modify task sequences
with over 100 package references and have never had policy churn like this
before. I wonder if perhaps you have some resource constraints on your MPs?
Daniel Ratliff
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Marable, Mike
Sent: Friday, January 16, 2015 2:12 PM
To: '[email protected]'
Subject: [mssms] CM12 - No Task Sequences Available - Finally Solved
We finally got this one solved. Here is what happened.
We use Nomad Branch in our environment. On the properties of a task sequence
1E adds a tab that allows you to set the Nomad settings for all packages
directly referenced by that task sequence. At each outage someone (that would
be me) was working on new task sequences and made sure that Nomad was enabled
through that tab on the task sequence properties.
This is what set an avalanche into motion.
In the SMS_Distribution_Manager log you will see a flurry of 2300
("Distribution Manager is beginning to process package..."), 2311 (Distribution
Manager successfully created or updated package...") and 2301 ("Distribution
Manager successfully processed package...") entries. One for every package
referenced by the task sequence. In our case that was about 112 packages.
Soon we begin to see in the SMS_Policy_Povider log 5101 messages ("Policy
Provider successfully updated the policy and the policy assignment that were
created from package..."). One for every package referenced by the task
sequence and (here is where things start to get bad) every advertisement that
references that task sequence. So, if we have 100 packages in the TS and the
TS is deployed to 10 collections that's 1000 policy updates.
Wait, there's more. Things are about to go from bad to worse.....
If you have a package that is also referenced in another task sequence, all of
the deployments of that task sequences get policy updates. So, let's say you
have a prior version of the build with those same 100 packages that is also
advertised to 10 collections. That's another 1000 policy updates.
Now we use task sequences for our nightly maintenance routines. Some packages
are shared between our build and maintenance sequences (scripts to install
printers for example). Each shared package then triggers a policy update for
each and every nightly maintenance advertisement. We run them every half hour
at the end of business so that's another 16 deployments * 7 days a week (112) *
maybe 10 packages shared is another 1000+ policy updates.
So this all quickly spirals out of control as every single package referenced
in the first task sequence then goes out and triggers its own policy refresh
for every instance of any deployment of a task sequence that that also
references it.
All told that one little checkbox generated nearly 40,000 5101 policy updates
over the course of 5 hours and knocked out builds out of the water.
Learn something new every day.
Mike Marable
Application Programmer/Analyst Lead
Enterprise Device Engineering and Management
MCTS, MCITP, MCSA, MS
[Profile<https://www.mcpvirtualbusinesscard.com/VBCServer/MikeMarable/profile>]
[Blog<http://thesystemsmonkey.wordpress.com/>]
--------------------------------------------
"The difficult we do at once. The impossible takes a little longer."
-US Army Corps of Engineers
"It is better to have less thunder in the mouth and more lightning in the hand."
-Apache Proverb
I will rise when I have fallen.
"Unless you try to do something beyond what you have already mastered, you will
never grow."
-Ralph Waldo Emerson
**********************************************************
Electronic Mail is not secure, may not be read every day, and should not be
used for urgent or sensitive issues
The information transmitted is intended only for the person or entity to which
it is addressed
and may contain CONFIDENTIAL material. If you receive this material/information
in error,
please contact the sender and delete or destroy the material/information.
**********************************************************
Electronic Mail is not secure, may not be read every day, and should not be
used for urgent or sensitive issues
