Al, I think it's appropriate to explain a little more,
to avoid further confusion, as accidentally deleting + recoverying object
and loosing group memberships are NOT separate problems (especially in
multi-domain forests or even in Windows 2000 single-domain forests).
The issues are indeed very much related to each
other:
-
tracking membership of a group to be able to undo a
change "in case one of it's members gets whacked" is generally a good
idea, no matter if a user has been deleted or if simply an administrator
made a mistake while editing group-memberships. When tracked (e.g.
via daily reports or dumps of the group-memberships - or by having a good
group-concept where all owners "know" the members), the owner of a group
should be able to get a group back to the state it should
be.
-
however, when you delete an object (e.g. a user,
computer, contact or a group itself), these objects naturally replicate as
tombstones to other DCs and GCs in the forest. When this happens, the
memberships of these objects in any group in the forest
is "cleaned" automatically - not only in the same domain where the
objects reside, but also in all of the other domains in the forest. I.e.
the objects are also removed from Universal (UG) and Domain Local Groups
(DLG) of any domain in the forest.
So what's the big deal?
Well, if you restore a DC from a system-state backup (on tape or file) and
then authoritatively restore the objects in their domain or even if you
restore the whole domain authoritatively (which not recommended anyways,
unless you really have to), the objects will never "repopulate" into the
UGs and DLGs of the other domains in the forest.
Good to know: if you
restore a GC, it will at least know of the UGs of the other domains incl.
their memberships (as these are a still stored in the AD database file
saved at the time of taking the system-state backup), which you could
leverage to repopulate the UGs in the respective Domains. However,
if you've not previously dumped your DLGs in the other domains, how will
you be able to recover their memberships? They are not stored on the GC
you've recoverd, and they were "cleaned" when the tombstone was able to
replicate to the other DCs/GCs in forest... And don't forget,
that depending on your group-modell, you could also have various nested
groups which are nothing else but members of other groups - these nestings
will also get lost if a group gets deleted. More about these issues
(and others) is described in the afforementioned whitepaper - incl.
details on the differences between 2000 and 2003 rgd. the
recovery challenge.
Here are some ideas to master the recovery
challenge and be on the safe side (besides relying on the group-owners to
recover memberships themselves):
-
hot-site approach: as mentioned in
another part of this thread, you could use DCs in a hot-site (we call it
LAG-site, as it's replication will be set to "lag" behind the other DCs).
These will hopefully not have replicated the tombstones at the time you
notice the deletion of the objects - you could then first use the
appropriate DCs the hot-site to perform the authoritative restore of the
objects without requiring you to perform a system restore (still need to
boot the DC in to DSRM mode). And secondly, you could analyse the groups
on the other DCs in the hotsite and then re-popluate the groups on a
DC outside of the hotsite. Best to script these restore
activities... The biggest challenge with this approach: you must hat
one DC of EVERY domain in the forest in your hotsite, which is fine if you
only have a few, but which can be nasty if you have many domains in your
forest (should really consider using virtual hw solutions for this
approach).
-
database appoach: basically, whith
this approach, you would ensure that you save all the un-recoverable
links (e.g. group-memberships, but also the manager/directReports or
managedBy/managedObjects links which get "cleaned" just like the group
memberships...) into a separate repository. This could be a flat file,
AD/AM, MSDE or "full" SQL or whatever you preferr. When scheduled
with your normal backup cycle of your DCs, the stored data can ensure that
you are able to "fix" the missing object-links (which is what the
group-memberships and the other examples mentioned really are) by
leveraging your "link-repository". Obviously, it would be good if you
would include versioning into your link-storage, to be able to recover
group-memberships an the other links as they were at a specific point in
time. Using this approach will also allow you to easily visualize the
memberships of any object in the forest, as you will have all the
objectlinks accessible in the database (i.e. you can see which DLGs or
UGs a user is a member of, no matter which domain the groups belong
to). This is the approach we took and it's almost getting to a point of
being finished (I've been promising the availability of this solution on
the DEC conferences for a while now... - sorry for the major delay - you
know how things go...;-) The challenge here obviously lies in
collecting and storing the links appropriately, so you have to have some
smart mechanism especially for really large AD environments like our
own. But for forests with many domains, this may be more feasable
than the hot-site approach. However, it can even be well combined with the
hot-site approach, as you could leverage the hot-site DCs for your
link-backups, and after a restore use the database to recover the links
(instead of querying each of the hotsite DCs).
-
backup-tool approach:backup tools,
which concentrate on online recovery of objects in AD (now supported in
2003 with the new API for tombstone reanimation) will also be very useful
in performing the full recovery of objects, as they will have stored the
object links (e.g. groups a user is a member of) withing their backup.
However, only few tools really do the job of link-recovery really well, as
most vendors don't care to query the "remote" domains for the memberships
of objects in DLGs - they will be able to recover UGs, if the object's
backup was taken from a GC, but the DLGs will be lost. I only know
of one tool, which is currently able to also fully recover the DLGs in any
domain of the forest, when restoring objects in AD. This isn't a
place for advertising, but you may want to look at the aforementioned
whitepaper ;-)
Realize, that whichever approach you take, restoring
objects in an AD forest is not a domain-admin's task => it's the task of
an enterprise admin, as only members of this group have the ability to
recover the links outside of the domain that the restored objects belong to
(unless you grant these permissions to other groups, which would then have
similar permissions as an EA).
Cheers,
Guido
I think there's two approaches here but correct me if I
misunderstood to flow.
One concept is to restore the actual object in case of
accidental deletion, intentional deletion, corruption, etc. The other
is to track the membership in case one of it's members gets whacked.
That about what you're saying?
To me, these are two very important, but separate
scenarios. One solution already in place is a tracking mechanism that
exports group information on a daily basis. That's the custom version
of what I have now, but it's nowhere near as efficient as a SQL/AD/AM
solution would be in a multi-domain environment. It only allows us to put
the group membership back, but has nothing to do with the group object
itself. If we lost that, we lost the sID etc that would make it useful
outside of a restore. In case of administrative error, we can look
back at the reference (keep a week's worth for now) and put it back the way
it should be without having to go to tape.
If you're going to the trouble of creating this
homegrown system wouldn't it make sense to make it part of the lifecycle
management system? There's certainly a market for that in the states
with the current round of laws about data and process.
Just a thought, but having a system that audits (for
lack of a better term) user/group lifecycles and resource allocation would
be an interesting thing to have. Kind of another tier in the management of
the system (a meta-directory type solution or other?)
Al
ha, I knew that would be your answer
;-))
and I can partly understand your strategy => the
owner of the group should know what's in it, so if there is a problem with
the memberships it's his and not yours. But this is really only
acceptable for a small issue, where you loose a couple of memberships - not
when you use a couple of hundred of users incl. their
memberships.
Sure losing a DLG membership has the same result in
losing resource access than a GG does - however, DLGs in multi-forest
environments are simply harder to recover the "native" way (i.e. be
authoritatively restoring your accidentally deleted users from one domain),
as that restored DC doesn't know of the DLG memberships outside of it's own
domain, which will then be lost for good (much easier to recover memberships
ins GGs and UGs as the DC/GC will "know" of the memberships after the
recovery). Your DLG memberships won't come back until re-added by
your group-owners, who will be happy to manage re-adding hundreds of users
into various groups via the UI they use... :-(
Obviously your impact will be less than for other
companies, as you have a really cool group-structure however .But no matter
how careful you are, Murphy is watching you. And things will happen. And you
have to be prepared...
The AD/AM idea isn't bad, but I'm just implementing the
same based on SQL and it's almost done - a nice tool that gives you exactly
what you're describing. And will help to recover those lost group-memberhips
and it will allow you to see which group your users or other objects are in
within the forest - in any domain. Stay tuned. However, it will still
require a normal authoritative restore of the actual objects that were
deleted - thus it's not as powerful as some of the online-recovery methods
available out there. So I encurage anyone responsible for back-up of their
AD also to look at these tools.
/Guido
Using the DLG's doesn't kill us any more than if we
used GG's. Same loss of resource access.
As for the accidents, the guys with the big guns don't
use the GUI for most anything, they use very targeted scripts that do very
specific things. We don't, for instance have any mass delete anything
scripts. All one off delete.
The groups are supposed to have well known membership
to the admins running them, they are supposed to be auditing the groups on a
very regular basis as to who should be in them. So loss of a group should
simply be recreate the group, reassign to the proper ACE in the proper file
structure (we don't do one group secures a zillion different things or at
least heavily discourage it), readd the correct people.
I do have some ideas floating in the back of my mind
about pulling all groups, computers, users off into a single AD/AM instance
so we can track things there. Don't sync the deletes other than marking a
field in AD/AM when the delete or occurred. This is more for being able to
do quick checks for things in the directory (everything would be tuple
indexed) but could also help if someone smoked a group that they shouldn't
have as we would have the last known membership for sure. I would also like
to get some form of change log management in there as well but that project
is way pie in the sky at the moment. Trying to get K3 deployed at the moment
and the final pieces of E2K deployed.
-------------
actually, you need to consider this issue more than
others Joe, as you're building all group-memberships on Domain Local Groups
(in a multi-domain environment) which will kill you, if you do accidentally
delete the wrong objects. Obviously you could still restore all
domains - but that's pretty nasty.
And accidents don't only happen to lower privileged
admins - it could be one of you three...
/Guido
Yes, excellent point. We haven't started worrying about
that granularity yet. If something is deleted, we figured the person with
the power to delete it intended it. Have a nice day. There are only
three people who can really do any huge mass deletes across the board and we
all sit within smacking distance of each other so we are careful as we have
sensitive ears and don't want to be cuffed. I do think we need some sort of
solution for this eventually though. But it is more to reduce nuisance
factor for silly OU admins than anything else.
Right now mostly still just worrying about the old
South East Michigan was swallowed by a volcano that came out of nowhere...
How do we make sure we can recover.
-------------
will only be good for restoring the DC hardware, but
depending on your setup won't be sufficient to fully recover accidentally
deleted objects.
I've worked with Aelita on this whitepaper to discuss
the potential issues:
/Guido
1. Multiple DCs in diseparate
locations.
2. Virtual DC for each domain that is shut down nightly
and the disk file for each is copied to some other location.
-------------
What is the best way to backup your domain
controller so you can restore it in a disaster situation.
