I should note, that I wrote this section under the pretend context that I had coded a naive/bad restore routine that didn't change the invocation ID at restore. But obviously this can happen at any image based restore.
http://brett.shirley.name/ImageBackupsBad.ppt And the slides aren't really about all backup/restore, b/c I've removed about 100 slides from the deck ... Cheers, BrettSh [msft] On Thu, 5 May 2005, joe wrote: > Brett, I think you would like Halo... I can visualize you getting into > shooting other people. Plus it isn't just shooting, when playing MP, there > is a lot of strategizing and thinking going on. :o) > > On the hair issue. It makes you stick out in a crowd - literally. Keep it up > - again literally. > > > > As for the rest, great post. > > I like the way you specified the distributed system part. I knew what I > wanted to say, basically that a DC doesn't exist in a vacuum; especially if > it is in a forest with other DCs. You can't just pluck one out and roll it > back without helping the other DCs understand that that has occurred. I > couldn't think of a good way to say it. I think your method of stating it as > a distributed system consistency issue will help folks have an understanding > of it as well. > > > The RID issue I hadn't even thought of in this context... Obviously a great > concern to point out. > > > > > [BrettSh] I thought I was being nice ... wow, it's going to suck, when > someone actually annoys me. ;) > > I know, that is why I said you were being extremely nice on this list... > > > > How's that blog coming..... This would be good info on it... As I have oft > said to ~Eric - There is so much bad and mediocre info out there because so > few of the people who really know the stuff aren't telling anyone about > it... To that end, I have no problem saying stupid things in this forum and > others if it incites/instigates you and other folks in the know to write it > up and correct me; we all benefit and I don't mind looking stupid as much as > like learning how things work. Barring that... People can just listen to my > guesses and go with them[1]. :o) > > > Again, great post. Thanks. > > > > joe > > > > > [1] My guesses get better and better every day.... Trust me.... > > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Thursday, May 05, 2005 2:19 PM > To: [email protected] > Subject: RE: [ActiveDir] best practice? > > > I don't really have serious time to answer this right now ... so for now, > you're going to have to trust me, it's not just a little bad you can recover > from it with X, it is _really_ bad to do an image based restore, and hard to > restore normality afterwards ... > > I'll prop a portion of a slide deck later on, where I show to the backup > vendors how the inconsistency is introduced ... but I don't know if it will > make sense w/o my delivery. It is also a bit simplified. joe is close > below, some comments inline, in joe's mail, as it's the closest so far to > understanding why this is bad ... > > BTW, clean and dirty AD DB have _nothing_ to do with this. clean/dirty is > an ESE / JET Blue level concept, this is an entirely AD Logical issue. > Nothing prevents an ESE database from being imaged. The AD has a design > decision that prevents image based restores. > > I don't play XBox or any computer games really. I know that sounds weird, > that a computer geek would not play video games, but I met a girl at a party > the other day who is a huge FPS player, so I think the world somehow > balances out in that respect. How could that compare to the relaxing sense > of accomplishment of working out paticularly cunning methods of compressing > replication metadata ... I mean really? Same goes for hair maintanence > tasks. > > On Thu, 5 May 2005, joe wrote: > > > I am actually waiting for Brett or ~Eric to respond to your post as > > well. I am positive they could give you a bulleted list of things that > > you as well as the rest of us are completely unaware of that will go > > pear shaped both because they have seen things like that or just know > > it from familiarity with the code paths involved. > > > > AD will not do a complete reload of the DB on its own, that was an NT4 > > thing that occurred if the change log rolled. All gone now. > > > > Do some searching on DSA IDs/GUIDs and Invocation IDs/GUIDS. A DSA ID > > is the GUID for the DC itself[1], it doesn't change for the life of > > the DC from my understanding. The invocation GUID[2] changes on > > restores, again to flag, hey new DB, > > [BrettSh] It's not a new DB so much, as a new logical stream of changes to > the distributed system ... > > > ... you don't know what my state is, so it can be brought into a > > consistent state. > > [BrettSh] Don't like the term "consistent state" here. I also don't like > how we're talking about the DB ... I know all the AD repl docs, talked about > it as a new database GUID, but that was poor taste ... there is a subtle but > key difference between > > [local] database consistency, and > distributed system consistency. > > It's the later we're worried about. +The later requires multiple nodes / > DCs to have followed all the rules.+ Most of the rules are coded into the > way AD behaves, when possible. Thou shalt not image restore, is > unfortunately not coded, and hard to be defensible against ... well, without > sacraficing availability ... but lets not get into that trade-off right now. > > > You should find hits on invocation id with topics of replication > > consistency, usn polling, AD restores, etc as it is key to all of them > > though it has been awhile since I went searching for that stuff. > > Something I have read on a couple of occasions but can't say I agree > > with is that allegedly the DSA ID and invocation id are identical > > unless a restore has occurred. I don't think I have EVER seen them > > identical so I don't know where that info came from. I am noting it > > simply because I recall seeing documentation to that effect in the past. > > [BrettSh] They should've been the same until the first restore ... there is > a bug somewhere, that no one bothered to iron out. > > BTW, we also change the InvocationID when we _re_-host an Application > Directory Partition ... I'll leave the discussion of why to your > imagination. > > Oh and since IFM is like throwing AD Restore and dcpromo into a blender for > 30 seconds, IFM based dcpromo sort of changes the InvocationID. > You'll notice the invocationID of the DC you took the original backup from > in the retired DSA signature of the newly dcpromo'd DC. > > > > > Really try to find detailed info on how replication works. High USN is > > just the tip of the iceberg, there is a lot of underlying details but > > I understand where the misconceptions can come in, a lot of the > > documentation out there in the public realm simplifies the crap out of > > this stuff with analogies and very high level details without ever > > indicating that it is really quite more involved than that. This can > > burn you when you start making decisions based on those simplified > examples. > > > > If you really want to get into it, start fishing through the platform > > sdk > > Ds* API calls. I would especially recommend the > > DsGetDCInfo/DsGetDcInfo2 functions and out of those the ones > > concerning DS_REPL_NEIGHBOR structures which gives a feeling of how > > much info there is involved with replication and consistency. > > > > While it may be possible to force the invocationid to change after the > > image restore, I am not aware of a method other than doing a proper DB > > restore. It could be as simple as tapping that attribute in the > > nTDSDSA object but I certainly would NOT be willing to test that in > > production even if it worked great in the lab. > > [BrettSh] > > Plausible Proposal #1: (please see big warning below) > _Technically_, yes if you trigger an Invocation ID change after > you lay down the image, _AND THIS IS THE KEY_ ... before the DC > talks to any other DCs, and takes any new changes to the database. > > This is one of those rules that all the nodes must follow, > and if you use an AD based backup/restore program, the > appropriate logic will be triggered, and the rules for > distributed consistency upheld. > > _Even_ booting the DC, may institute a change, that causes > distributed system inconsistency. Obviously, tapping the object > from LDAP is not an option, you have to do it from DSRM. > Unfortunately, I've forgotten to tell you how you can trigger a > invocation ID change from DSRM ... > > In short don't go there. These are not the droids you're looking for. > > > > > Certainly, do not image DCs and use that as a recovery mechanism. The > > one way to do that, IMO, would involve snap shooting and rolling back > > all DCs in a forest at the same time. I don't see how this could > > effectively be done in the real world on real hardware. I visualize > > possibilities with virtualization software, but that would require a > > lot of testing and work to get there and some how guarantee that the > > snapshot was done at the exact time for all images. > > [BrettSh] > > Plausible Proposal #2: (please see the big warning below) > _Technically_, this will work too. Requires all DCs to be off at > the same time when you take the image based back ups (I > think). Requires all the existing DCs to be turned off before > you restart the first restored image. I think that is all that > is required ... but I'm not sure ... I don't care enough to try > to give anyone > > Plausible Proposal #3: (please see the big warning below) > Of course a single DC forest can be image based restored as well, > though ... you're more likely to get SIDs reissued, and have old > wacky ACLs in this case, b/c IIRC we invalidate the present RID > pool on restore. This can be mitigated by booting the DC, and > before creating any security principals, booting the next rid up, > can't remember how that is done off the top of my head though ... > > > > > If you have done this in production already, I would recommend going > > back to what Brett said and doing a verification of your DB on all of your > DCs. > > [BrettSh] Jeez, I really hope no one is in this state, it can be quite > disturbing to iron out. > > > Again, Brett is someone who knows about the AD DB. Don't let his > > sometimes grouchy demeanor throw you off. He may get difficult at > > times but he is almost always trying to help, he just has interesting > > ways of expressing it on occasion. He has actually been extremely nice > > on this list compared to some other notes I have seen from him. > > [BrettSh] I thought I was being nice ... wow, it's going to suck, when > someone actually annoys me. ;) > > > Basically I say the same about him > > that I have often said about myself; don't mistake the quality of the > > delivery for the quality of the information. :o) > > [BrettSh] > > So first let me divulge, that I am not in fact the Garage Door Operator for > building 7, in fact I am a developer/programmer (we're call Software > Development Engineers at Microsoft) in Windows, ON Active Directory. > Before my recent move to the ESE development, I worked on the AD Replication > development for ~5.5 years, spending time working on AD Replication, AD > backup/restore, a small bit in AD Schema/Database stuff, AD tools, and even > dabbling in DcPromo off and on when required for those years. Quite frankly > I'm the one who has dealt with almost all the areas affected by a bad image > based backup/restore, and the parts that make a good backup/restore > possible. I'm uniquely qualified to say: > Image based backup/restores are not supported for AD. > > So we had this customer who wanted to use SAN based hot split on Win2k AD > (which is even more unsupported, as they didn't shutdown all the DCs, like > Plausible Proposal #2 above), after explaining that they'd have to shutdown > all DCs, and them agreeing (though I doubted they'd actually do that, it's > amazing what customers will do when they think they understand better than > you) and then they agreed for restore, they'd take ALL the DCs back to the > same backup time, at the same time, and working out this complicated set of > steps they would need, I pointed out this: > > --- begin quote --- > I can't confirm if you will fail ..., but that set of steps if correctly > followed will not cause forest corruption due to USN rollback. Honestly, it > isn.t worrying about this once PSS guided transition that worries me, > following those types of steps once isn't hard . it is someone not > understanding why each of the parts of the technique were required, and > later trying it again, and not getting it right. In general customers may > not truly understand the system's requirements, EVEN after they say they do > (b/c they believe they do, no one intentionally hoses their domain, but > somehow it happens) so it's just easier to say "no mirror splits on > unsupported SANs" > --- end quote --- > > So .... > > Warning! Warning! Danger Will Robinson! Danger! > > So the same goes for all 3 proposals above ... while technically you could > work out the exact set of steps required, it is likely to be an error prone > manual process ... will the next guy who maintains the corp infrastructure > understand it all ... will you miss a step ... if you have lots of DCs in > branches, how do you know one won't be missed ... you're playing with fire > ... and the slightest tweaks can change the answer substantially, for > instance auth restore for proposal #1 must be done after triggering the > invocation ID to change, which would require a reboot ... even me with all > my knowledge, wouldn't implement such a mechanism in a live corporate > deployment ... it's subtle, and it is not worth the risk. > > Friends don't let friends use image based backups of AD. > > Cheers, > -Brett [msft] > I'm just kidding, I just made all the above up, I really am just the > Building 7 Garage Door Operator ... > > > > > > joe > > > > > > > > [1] It is the objectGUID attribute of the ntdsdsa object(aka NTDS > > Settings object). > > [2] It is the invocationID attribute of the ntdsdsa object. > > > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Bahta > > Nathaniel V Contr NASIC/SCNA > > Sent: Thursday, May 05, 2005 10:22 AM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > Joe, > > > > I appreciate you indulging me in detail. I was just curious on what > > the consequences may be of imaging and restoring DC's. We are always > > evaluating and re-evaluating DR methods and techniques, and this was > > the latest hot topic. I thought AD pushed changes up to a > > pre-determined amount and then it would just replicate the whole > > database if the number of changes were too great. I am not sure of > > the in-depth implications of restoring imaged DC's but I know the > > difference between a clean and dirty AD DB and it sounds as though the > > metadata cleanup and synchronization is not meant to happen with an AD > > unaware application such as ghost. Perhaps an application that could > > stabilize an old DC with the new AD DB would be something that would > > have to be looked at. Or maybe an image of a member server and a > > dcpromo is the easiest way to recover a DC. I have intentions on working > smarter, not harder, but that does not forgo my lust for understanding right > from wrong. > > > > Thanks again for the rebuttal. It always helps to hear things from > > all perspectives to get a better look at the big picture. > > > > Nathaniel > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > Sent: Wednesday, May 04, 2005 2:36 PM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > I'm not Brett[1] but wanted to just say something really quick here. > > > > Well a couple of things actually. > > > > 1. When it comes to AD Database consistency and replication. Brett is > > someone I would tend to listen to very carefully. I may not understand > > what he is trying to say but I will try like heck to understand it. > > Rough around the edges though he may be, he knows a lot about the guts > > of the AD DB and Replication. Keep in mind he wrote some of the most > > "brilliant" parts of repadmin[2]. > > > > 2. When you image and recover the image you are bypassing any and all > > logic associated with a directory DB recovery. I.E. You aren't > > restoring the database through the very specific DS Backup/Restore API > > so you don't get the cool things that it does like renaming the > > Database GUID aka invocation ID which effectively tells all of the other > partners there is a "different" > > database out here that needs to be fully updated. > > > > I haven't fully thought out the implications of that but one thing > > right off the bat is the thought that all DCs maintain high water > > vectors for all databases so they know where they are at for > > replication. This isn't just kept on the DC in question, this is kept > > all over so I could see serious possibilities of issues there. > > Additionally think of a change that mastered on that database and > > replicated out. How do you get it back if the DB is rolled back and > > all of the other DCs already think that DB has that info since it was > mastered there? > > > > You get ~Eric, Dean, and Brett thinking about it and I expect you > > could find all sorts of horrible things that this can do to you. > > > > I think the idea that a DC can be restored from an image like that > > because it is "sort" of like restoring the DB is flawed at the very > > best. You don't have a full comprehension of what is being done in the > > backend to support that restore. If it were that simple, why do you need a > backup api at all? > > Mirror the DIT and zip it and there is your backup... It doesn't work > > that way. > > > > As Brett indicated... Bad mojo... Heck I will go further, positively evil. > > You could damage your AD in ways that you (and it) has no clue about > > and only later run into it when you are trying to figure out niggling > > consistency issues in applications that act odd some of the time. > > > > > > joe > > > > > > > > [1] And I couldn't play him on TV either, Brett stores a good portion > > of his height in his hair and I store mine in my legs. > > > > [2] His words when I met him in person at an MVP summit. He was quite > > excited to talk about that portion of the code... > > > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Bahta > > Nathaniel V Contr NASIC/SCNA > > Sent: Wednesday, May 04, 2005 1:59 PM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > Brett, > > > > What is your basis for not being able to restore a DC from a image? > > If the DC has an old copy of the directory data, it will check its > > USN's and update its copy. What could cause havok if anything? We > > are about to institute this very same concept here to turn DR into a > > 10 minute process when it comes to operating system recovery. We will > > image the servers monthly and restore from said image whenever one > > crashes. What could cause a problem by restoring a DC, it will be > > timestamped to be old and AD will synchronize it with the rest of the > domain. > > > > Please elaborate on your basis for comment. > > > > Nathaniel Bahta > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > > Sent: Wednesday, May 04, 2005 11:47 AM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > jlc, > > > > You can't restore a single DC via an image based backup, either. It > > is not supported, it is not allowed ... it is bad mojo. > > > > Well, it wouldn't cause issues if the forest had ONLY that one DC > > (seems unlikely the case), or for a multi-DC forest, you'd have to > > shutdown all the DCs in the forest at the same time, when you took your > backup images. > > And then on restore, restore them all at the same time. Basically a > > pretty infeasible suggestion. > > > > Cheers, > > -Brett Shirley [msft] > > > > This posting is provided "AS IS" with no warranties, and confers no > rights. > > > > > > On Wed, 4 May 2005, Joseph L. Casale wrote: > > > > > Exactly, I do it for DR purposes, the old one dies - I reimage it > > > and put it back out there. > > > No poblem... > > > jlc > > > > > > -----Original Message----- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Phil Renouf > > > Sent: Wednesday, May 04, 2005 7:01 AM > > > To: [email protected] > > > Subject: Re: [ActiveDir] best practice? > > > > > > On 5/4/05, John Shukovsky Jr <[EMAIL PROTECTED]> wrote: > > > > BUT....as for DC's. I do "image" dc's using Symantec Livestate > > > > Recovery ( formerly PowerQuest V2i ). It works wonderfully. I > > > > primarily use for backups. I have not had to recover a server in > > > > production ( and hope I do not have to ) but I have in lab 10+ > > > > times > > > and servers are as clean as ever. > > > > You should take a look. > > > > > > When Brett mentioned imaging DCs being a bad idea and to never ever > > > do it I believe that he was meaning don't Image a DC and try to use > > > that Image to build other new DCs and just trying to change the SID > > > like you would for a desktop. Bad idea! > > > > > > Phil > > > List info : http://www.activedir.org/List.aspx > > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > > List archive: > > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > List info : http://www.activedir.org/List.aspx > > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > > List archive: > > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > > > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > List info : http://www.activedir.org/List.aspx > List FAQ : http://www.activedir.org/ListFAQ.aspx > List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx > List FAQ : http://www.activedir.org/ListFAQ.aspx > List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ > List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
