Jorge beat me to it, but I'd pretty much do the same: pick one server (FSMO role holder makes sense - ensure it holds all the roles prior to starting) and then flatten the others. Since you know they've been broken for so long, I'd additionally do the following:
1) prior to starting make a full backup of each DC and store that somewhere safe - gives a warm fuzzy feeling to the people that own it
2) Be sure to write down the configuration of each network as it should be. Since it's been broken, consider these to be new servers that need to be properly configured.
3) Setup a monitoring infrastructure that this company can work with.
4) Setup an administrative model that makes sense for this company. Keep it as minimal as possible. They've proven that they have issues, so try to limit the impact they can have on it.
5) Training for support personnel. Even if this is just a two day, Mark Parris led event with a few beers thrown in, it would be a good start to getting their minds right.
6) Make sure you have the method of doing this well tested and the fastet way to get it done practiced. IFM might be useful to you in restoring service as quickly as possible.
7) alert the helpdesk to the event and let them know there may be a few phone calls. Changes that were made in those times might be lost and as you remove DC's, you may receive some phone calls.
8) Check DHCP reservations if using AD-integrated DNS. Same for WINS records to be sure you're aware of the ip addrs and whether or not you need to change those etc.
9) go ahead and flatten the other 8 DC's. I recommend doing this during off-hours.
10) Clean the remaining "good" DC and ensure it's pristine. The focus here is to have a clean seed without losing all of their forest data.
11) add back new DC's, sites, etc.
I highly suggest using SP1 if possible. It's much easier to use ntdsutil in that version for this process plus some of the fixes around AD-integrated DNS might be of use to you as will several other bug fixes. Integrated deployments might be of use.
Keep in mind that's a lot for this company to bite off. You'll want to fully communicate it to the helpdesk and the support personnel. You're basically building from scratch, so get in the mind-set that you have one DC and it's time for a build-out. Never mind that you need to remove several "worthless" DC's first and you plan to re-use same hardware. That's a different world because once you've flattened them (I mean all the way to bare metal - you cannot trust them) then they never really existed (once you clean the metadata out). I also suggest that you get them to spring for an additional machine to act as the replacement of the "pristine DC" at ground zero. It could be a really powerful desktop for as long as you end up using it, but I suggest this because to be complete you should assume that the DC is compromised given how neglected replication was. I personally wouldn't trust them in that kind of environment and would be concerned about rogue processes etc. You could rebuild on a new machine, clean the old out of there, and then rebuild it. Remove the new machine (if using temporary hardware) and make sure all is well.
My additional $0.04 anyway (all contributions in USD)
Al
On 3/16/06, Almeida Pinto, Jorge de <[EMAIL PROTECTED]> wrote:
Damn...demoting all DCs (except one) and promoting again... auch
if I needed to demote DCs because they have not replicated for more than the TSL I would do the following:
* Investigate which DCs DO replicate...
Before starting the promotion of new DCs while the bad ones are running, stop and disable the NETLOGON service on the BAD DCs (this is to prevent that if you start promoting new DCs they will talk with the BAD DCs) (when force demoting a bad DC set the NETLOGON service to manual but don't start it)
If ALL DCs are shouting the other DCs have been disconnected for too long.... I would choose the DC with the FSMO roles as the DC NOT to demote (I would also hope the other FSMO are on that DC) (that's why I like having all FSMOs one DC) and demote ALL others.
As replication will not work you need to force demote them and clean their metadata (http://support.microsoft.com/?id=216498) on one of the remaining healthy DCs. If you have more than one remaining DC don't forget to let the metadata cleanup replicate to all remaining DCs before promoting again
* If needed seize FSMO roles
* Promote demoted DCs again to DC and when applicable also make it a GC (if single domain forest domain, make all DCs a GC)
this should do it
Met vriendelijke groeten / Kind regards,
Ing. Jorge de Almeida Pinto
Senior Infrastructure Consultant
MVP Windows Server - Directory Services
LogicaCMG Nederland B.V. (BU RTINC Eindhoven)
( Tel : +31-(0)40-29.57.777
( Mobile : +31-(0)6- 26.26.62.80
* E-mail : <see sender address>
________________________________
From: [EMAIL PROTECTED] on behalf of Mark Parris
Sent: Thu 2006-03-16 21:53
To: ActiveDir.org
Subject: [ActiveDir] AD Recovery after tombstone hits all DC's
All,
As per my email from Monday - "not a line from a song...." I have managed to persuade the company to do a DC demotion and promotion. Now I have a question do I need to demote all DC's bar one then promote them all back up or can i do it on a site by site basis?
Will new DC's introduced prior to the demotion excercise be exposed to the tombstoned data too?
and finally can you manually delete tombstoned objects - as it is these objects that are failing replication in the AD. I have ran the clean up of lingering objects multiple times with no joy.
I manually deleted the corrupt machine accounts via adsi edit and the ad then skipped onto the next set of failing attributes.
I then tried via LDP to delete the tombstoned objects and it will not let me.
Any additional guidance would be appreciated.
Sent from my blackberry - so sorry for no spell check.
Regards,
Mark
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
