Thanks Brett.
Further.... LOL "My home page is under construction. I'll update it once I learn how to program in HTML." -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Thursday, May 05, 2005 4:12 PM To: [email protected] Subject: RE: [ActiveDir] best practice? I should note, that I wrote this section under the pretend context that I had coded a naive/bad restore routine that didn't change the invocation ID at restore. But obviously this can happen at any image based restore. http://brett.shirley.name/ImageBackupsBad.ppt And the slides aren't really about all backup/restore, b/c I've removed about 100 slides from the deck ... Cheers, BrettSh [msft] On Thu, 5 May 2005, joe wrote: > Brett, I think you would like Halo... I can visualize you getting into > shooting other people. Plus it isn't just shooting, when playing MP, > there is a lot of strategizing and thinking going on. :o) > > On the hair issue. It makes you stick out in a crowd - literally. Keep > it up > - again literally. > > > > As for the rest, great post. > > I like the way you specified the distributed system part. I knew what > I wanted to say, basically that a DC doesn't exist in a vacuum; > especially if it is in a forest with other DCs. You can't just pluck > one out and roll it back without helping the other DCs understand that > that has occurred. I couldn't think of a good way to say it. I think > your method of stating it as a distributed system consistency issue > will help folks have an understanding of it as well. > > > The RID issue I hadn't even thought of in this context... Obviously a > great concern to point out. > > > > > [BrettSh] I thought I was being nice ... wow, it's going to suck, > > when > someone actually annoys me. ;) > > I know, that is why I said you were being extremely nice on this list... > > > > How's that blog coming..... This would be good info on it... As I have > oft said to ~Eric - There is so much bad and mediocre info out there > because so few of the people who really know the stuff aren't telling > anyone about it... To that end, I have no problem saying stupid things > in this forum and others if it incites/instigates you and other folks > in the know to write it up and correct me; we all benefit and I don't > mind looking stupid as much as like learning how things work. Barring > that... People can just listen to my guesses and go with them[1]. :o) > > > Again, great post. Thanks. > > > > joe > > > > > [1] My guesses get better and better every day.... Trust me.... > > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Thursday, May 05, 2005 2:19 PM > To: [email protected] > Subject: RE: [ActiveDir] best practice? > > > I don't really have serious time to answer this right now ... so for > now, you're going to have to trust me, it's not just a little bad you > can recover from it with X, it is _really_ bad to do an image based > restore, and hard to restore normality afterwards ... > > I'll prop a portion of a slide deck later on, where I show to the > backup vendors how the inconsistency is introduced ... but I don't > know if it will make sense w/o my delivery. It is also a bit > simplified. joe is close below, some comments inline, in joe's mail, > as it's the closest so far to understanding why this is bad ... > > BTW, clean and dirty AD DB have _nothing_ to do with this. > clean/dirty is an ESE / JET Blue level concept, this is an entirely AD Logical issue. > Nothing prevents an ESE database from being imaged. The AD has a > design decision that prevents image based restores. > > I don't play XBox or any computer games really. I know that sounds > weird, that a computer geek would not play video games, but I met a > girl at a party the other day who is a huge FPS player, so I think the > world somehow balances out in that respect. How could that compare to > the relaxing sense of accomplishment of working out paticularly > cunning methods of compressing replication metadata ... I mean really? > Same goes for hair maintanence tasks. > > On Thu, 5 May 2005, joe wrote: > > > I am actually waiting for Brett or ~Eric to respond to your post as > > well. I am positive they could give you a bulleted list of things > > that you as well as the rest of us are completely unaware of that > > will go pear shaped both because they have seen things like that or > > just know it from familiarity with the code paths involved. > > > > AD will not do a complete reload of the DB on its own, that was an > > NT4 thing that occurred if the change log rolled. All gone now. > > > > Do some searching on DSA IDs/GUIDs and Invocation IDs/GUIDS. A DSA > > ID is the GUID for the DC itself[1], it doesn't change for the life > > of the DC from my understanding. The invocation GUID[2] changes on > > restores, again to flag, hey new DB, > > [BrettSh] It's not a new DB so much, as a new logical stream of > changes to the distributed system ... > > > ... you don't know what my state is, so it can be brought into a > > consistent state. > > [BrettSh] Don't like the term "consistent state" here. I also don't > like how we're talking about the DB ... I know all the AD repl docs, > talked about it as a new database GUID, but that was poor taste ... > there is a subtle but key difference between > > [local] database consistency, and > distributed system consistency. > > It's the later we're worried about. +The later requires multiple > nodes / DCs to have followed all the rules.+ Most of the rules are > coded into the way AD behaves, when possible. Thou shalt not image > restore, is unfortunately not coded, and hard to be defensible against > ... well, without sacraficing availability ... but lets not get into that trade-off right now. > > > You should find hits on invocation id with topics of replication > > consistency, usn polling, AD restores, etc as it is key to all of > > them though it has been awhile since I went searching for that stuff. > > Something I have read on a couple of occasions but can't say I agree > > with is that allegedly the DSA ID and invocation id are identical > > unless a restore has occurred. I don't think I have EVER seen them > > identical so I don't know where that info came from. I am noting it > > simply because I recall seeing documentation to that effect in the past. > > [BrettSh] They should've been the same until the first restore ... > there is a bug somewhere, that no one bothered to iron out. > > BTW, we also change the InvocationID when we _re_-host an Application > Directory Partition ... I'll leave the discussion of why to your > imagination. > > Oh and since IFM is like throwing AD Restore and dcpromo into a > blender for 30 seconds, IFM based dcpromo sort of changes the InvocationID. > You'll notice the invocationID of the DC you took the original backup > from in the retired DSA signature of the newly dcpromo'd DC. > > > > > Really try to find detailed info on how replication works. High USN > > is just the tip of the iceberg, there is a lot of underlying details > > but I understand where the misconceptions can come in, a lot of the > > documentation out there in the public realm simplifies the crap out > > of this stuff with analogies and very high level details without > > ever indicating that it is really quite more involved than that. > > This can burn you when you start making decisions based on those > > simplified > examples. > > > > If you really want to get into it, start fishing through the > > platform sdk > > Ds* API calls. I would especially recommend the > > DsGetDCInfo/DsGetDcInfo2 functions and out of those the ones > > concerning DS_REPL_NEIGHBOR structures which gives a feeling of how > > much info there is involved with replication and consistency. > > > > While it may be possible to force the invocationid to change after > > the image restore, I am not aware of a method other than doing a > > proper DB restore. It could be as simple as tapping that attribute > > in the nTDSDSA object but I certainly would NOT be willing to test > > that in production even if it worked great in the lab. > > [BrettSh] > > Plausible Proposal #1: (please see big warning below) > _Technically_, yes if you trigger an Invocation ID change after > you lay down the image, _AND THIS IS THE KEY_ ... before the DC > talks to any other DCs, and takes any new changes to the database. > > This is one of those rules that all the nodes must follow, > and if you use an AD based backup/restore program, the > appropriate logic will be triggered, and the rules for > distributed consistency upheld. > > _Even_ booting the DC, may institute a change, that causes > distributed system inconsistency. Obviously, tapping the object > from LDAP is not an option, you have to do it from DSRM. > Unfortunately, I've forgotten to tell you how you can trigger a > invocation ID change from DSRM ... > > In short don't go there. These are not the droids you're looking for. > > > > > Certainly, do not image DCs and use that as a recovery mechanism. > > The one way to do that, IMO, would involve snap shooting and rolling > > back all DCs in a forest at the same time. I don't see how this > > could effectively be done in the real world on real hardware. I > > visualize possibilities with virtualization software, but that would > > require a lot of testing and work to get there and some how > > guarantee that the snapshot was done at the exact time for all images. > > [BrettSh] > > Plausible Proposal #2: (please see the big warning below) > _Technically_, this will work too. Requires all DCs to be off at > the same time when you take the image based back ups (I > think). Requires all the existing DCs to be turned off before > you restart the first restored image. I think that is all that > is required ... but I'm not sure ... I don't care enough to try > to give anyone > > Plausible Proposal #3: (please see the big warning below) > Of course a single DC forest can be image based restored as well, > though ... you're more likely to get SIDs reissued, and have old > wacky ACLs in this case, b/c IIRC we invalidate the present RID > pool on restore. This can be mitigated by booting the DC, and > before creating any security principals, booting the next rid up, > can't remember how that is done off the top of my head though ... > > > > > If you have done this in production already, I would recommend going > > back to what Brett said and doing a verification of your DB on all > > of your > DCs. > > [BrettSh] Jeez, I really hope no one is in this state, it can be quite > disturbing to iron out. > > > Again, Brett is someone who knows about the AD DB. Don't let his > > sometimes grouchy demeanor throw you off. He may get difficult at > > times but he is almost always trying to help, he just has > > interesting ways of expressing it on occasion. He has actually been > > extremely nice on this list compared to some other notes I have seen from him. > > [BrettSh] I thought I was being nice ... wow, it's going to suck, when > someone actually annoys me. ;) > > > Basically I say the same about him > > that I have often said about myself; don't mistake the quality of > > the delivery for the quality of the information. :o) > > [BrettSh] > > So first let me divulge, that I am not in fact the Garage Door > Operator for building 7, in fact I am a developer/programmer (we're > call Software Development Engineers at Microsoft) in Windows, ON Active Directory. > Before my recent move to the ESE development, I worked on the AD > Replication development for ~5.5 years, spending time working on AD > Replication, AD backup/restore, a small bit in AD Schema/Database > stuff, AD tools, and even dabbling in DcPromo off and on when required > for those years. Quite frankly I'm the one who has dealt with almost > all the areas affected by a bad image based backup/restore, and the > parts that make a good backup/restore possible. I'm uniquely qualified to say: > Image based backup/restores are not supported for AD. > > So we had this customer who wanted to use SAN based hot split on Win2k > AD (which is even more unsupported, as they didn't shutdown all the > DCs, like Plausible Proposal #2 above), after explaining that they'd > have to shutdown all DCs, and them agreeing (though I doubted they'd > actually do that, it's amazing what customers will do when they think > they understand better than > you) and then they agreed for restore, they'd take ALL the DCs back to > the same backup time, at the same time, and working out this > complicated set of steps they would need, I pointed out this: > > --- begin quote --- > I can't confirm if you will fail ..., but that set of steps if > correctly followed will not cause forest corruption due to USN > rollback. Honestly, it isn.t worrying about this once PSS guided > transition that worries me, following those types of steps once isn't > hard . it is someone not understanding why each of the parts of the > technique were required, and later trying it again, and not getting it > right. In general customers may not truly understand the system's > requirements, EVEN after they say they do (b/c they believe they do, > no one intentionally hoses their domain, but somehow it happens) so > it's just easier to say "no mirror splits on unsupported SANs" > --- end quote --- > > So .... > > Warning! Warning! Danger Will Robinson! Danger! > > So the same goes for all 3 proposals above ... while technically you > could work out the exact set of steps required, it is likely to be an > error prone manual process ... will the next guy who maintains the > corp infrastructure understand it all ... will you miss a step ... if > you have lots of DCs in branches, how do you know one won't be missed > ... you're playing with fire ... and the slightest tweaks can change > the answer substantially, for instance auth restore for proposal #1 > must be done after triggering the invocation ID to change, which would > require a reboot ... even me with all my knowledge, wouldn't implement > such a mechanism in a live corporate deployment ... it's subtle, and it is not worth the risk. > > Friends don't let friends use image based backups of AD. > > Cheers, > -Brett [msft] > I'm just kidding, I just made all the above up, I really am just the > Building 7 Garage Door Operator ... > > > > > > joe > > > > > > > > [1] It is the objectGUID attribute of the ntdsdsa object(aka NTDS > > Settings object). > > [2] It is the invocationID attribute of the ntdsdsa object. > > > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Bahta > > Nathaniel V Contr NASIC/SCNA > > Sent: Thursday, May 05, 2005 10:22 AM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > Joe, > > > > I appreciate you indulging me in detail. I was just curious on what > > the consequences may be of imaging and restoring DC's. We are > > always evaluating and re-evaluating DR methods and techniques, and > > this was the latest hot topic. I thought AD pushed changes up to a > > pre-determined amount and then it would just replicate the whole > > database if the number of changes were too great. I am not sure of > > the in-depth implications of restoring imaged DC's but I know the > > difference between a clean and dirty AD DB and it sounds as though > > the metadata cleanup and synchronization is not meant to happen with > > an AD unaware application such as ghost. Perhaps an application > > that could stabilize an old DC with the new AD DB would be something > > that would have to be looked at. Or maybe an image of a member > > server and a dcpromo is the easiest way to recover a DC. I have > > intentions on working > smarter, not harder, but that does not forgo my lust for understanding > right from wrong. > > > > Thanks again for the rebuttal. It always helps to hear things from > > all perspectives to get a better look at the big picture. > > > > Nathaniel > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > Sent: Wednesday, May 04, 2005 2:36 PM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > I'm not Brett[1] but wanted to just say something really quick here. > > > > Well a couple of things actually. > > > > 1. When it comes to AD Database consistency and replication. Brett > > is someone I would tend to listen to very carefully. I may not > > understand what he is trying to say but I will try like heck to understand it. > > Rough around the edges though he may be, he knows a lot about the > > guts of the AD DB and Replication. Keep in mind he wrote some of the > > most "brilliant" parts of repadmin[2]. > > > > 2. When you image and recover the image you are bypassing any and > > all logic associated with a directory DB recovery. I.E. You aren't > > restoring the database through the very specific DS Backup/Restore > > API so you don't get the cool things that it does like renaming the > > Database GUID aka invocation ID which effectively tells all of the > > other > partners there is a "different" > > database out here that needs to be fully updated. > > > > I haven't fully thought out the implications of that but one thing > > right off the bat is the thought that all DCs maintain high water > > vectors for all databases so they know where they are at for > > replication. This isn't just kept on the DC in question, this is > > kept all over so I could see serious possibilities of issues there. > > Additionally think of a change that mastered on that database and > > replicated out. How do you get it back if the DB is rolled back and > > all of the other DCs already think that DB has that info since it > > was > mastered there? > > > > You get ~Eric, Dean, and Brett thinking about it and I expect you > > could find all sorts of horrible things that this can do to you. > > > > I think the idea that a DC can be restored from an image like that > > because it is "sort" of like restoring the DB is flawed at the very > > best. You don't have a full comprehension of what is being done in > > the backend to support that restore. If it were that simple, why do > > you need a > backup api at all? > > Mirror the DIT and zip it and there is your backup... It doesn't > > work that way. > > > > As Brett indicated... Bad mojo... Heck I will go further, positively evil. > > You could damage your AD in ways that you (and it) has no clue about > > and only later run into it when you are trying to figure out > > niggling consistency issues in applications that act odd some of the time. > > > > > > joe > > > > > > > > [1] And I couldn't play him on TV either, Brett stores a good > > portion of his height in his hair and I store mine in my legs. > > > > [2] His words when I met him in person at an MVP summit. He was > > quite excited to talk about that portion of the code... > > > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Bahta > > Nathaniel V Contr NASIC/SCNA > > Sent: Wednesday, May 04, 2005 1:59 PM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > Brett, > > > > What is your basis for not being able to restore a DC from a image? > > If the DC has an old copy of the directory data, it will check its > > USN's and update its copy. What could cause havok if anything? We > > are about to institute this very same concept here to turn DR into a > > 10 minute process when it comes to operating system recovery. We > > will image the servers monthly and restore from said image whenever > > one crashes. What could cause a problem by restoring a DC, it will > > be timestamped to be old and AD will synchronize it with the rest of > > the > domain. > > > > Please elaborate on your basis for comment. > > > > Nathaniel Bahta > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Brett > > Shirley > > Sent: Wednesday, May 04, 2005 11:47 AM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > jlc, > > > > You can't restore a single DC via an image based backup, either. It > > is not supported, it is not allowed ... it is bad mojo. > > > > Well, it wouldn't cause issues if the forest had ONLY that one DC > > (seems unlikely the case), or for a multi-DC forest, you'd have to > > shutdown all the DCs in the forest at the same time, when you took > > your > backup images. > > And then on restore, restore them all at the same time. Basically a > > pretty infeasible suggestion. > > > > Cheers, > > -Brett Shirley [msft] > > > > This posting is provided "AS IS" with no warranties, and confers no > rights. > > > > > > On Wed, 4 May 2005, Joseph L. Casale wrote: > > > > > Exactly, I do it for DR purposes, the old one dies - I reimage it > > > and put it back out there. > > > No poblem... > > > jlc > > > > > > -----Original Message----- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Phil > > > Renouf > > > Sent: Wednesday, May 04, 2005 7:01 AM > > > To: [email protected] > > > Subject: Re: [ActiveDir] best practice? > > > > > > On 5/4/05, John Shukovsky Jr <[EMAIL PROTECTED]> wrote: > > > > BUT....as for DC's. I do "image" dc's using Symantec Livestate > > > > Recovery ( formerly PowerQuest V2i ). It works wonderfully. I > > > > primarily use for backups. I have not had to recover a server in > > > > production ( and hope I do not have to ) but I have in lab 10+ > > > > times > > > and servers are as clean as ever. > > > > You should take a look. > > > > > > When Brett mentioned imaging DCs being a bad idea and to never > > > ever do it I believe that he was meaning don't Image a DC and try > > > to use that Image to build other new DCs and just trying to change > > > the SID like you would for a desktop. Bad idea! > > > > > > Phil > > > List info : http://www.activedir.org/List.aspx > > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > > List archive: > > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > List info : http://www.activedir.org/List.aspx > > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > > List archive: > > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > > > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > List info : http://www.activedir.org/List.aspx > List FAQ : http://www.activedir.org/ListFAQ.aspx > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx > List FAQ : http://www.activedir.org/ListFAQ.aspx > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
