Since no one referenced them during this thread... For a bit more detail on the subject, check these out.
How to detect and recover from a USN rollback in Windows Server 2003 http://support.microsoft.com/?kbid=875495 How to detect and recover from a USN rollback in Windows 2000 Server http://support.microsoft.com/?kbid=885875 > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Thursday, May 05, 2005 13:19 > To: [email protected] > Subject: RE: [ActiveDir] best practice? > > > I don't really have serious time to answer this right now ... > so for now, you're going to have to trust me, it's not just > a little bad you can recover from it with X, it is _really_ > bad to do an image based restore, and hard to restore > normality afterwards ... > > I'll prop a portion of a slide deck later on, where I show to > the backup vendors how the inconsistency is introduced ... > but I don't know if it will make sense w/o my delivery. It > is also a bit simplified. joe is close below, some comments > inline, in joe's mail, as it's the closest so far to > understanding why this is bad ... > > BTW, clean and dirty AD DB have _nothing_ to do with this. > clean/dirty is an ESE / JET Blue level concept, this is an > entirely AD Logical issue. > Nothing prevents an ESE database from being imaged. The AD > has a design decision that prevents image based restores. > > I don't play XBox or any computer games really. I know that > sounds weird, that a computer geek would not play video > games, but I met a girl at a party the other day who is a > huge FPS player, so I think the world somehow balances out in > that respect. How could that compare to the relaxing sense > of accomplishment of working out paticularly cunning methods > of compressing replication metadata ... I mean really? Same > goes for hair maintanence tasks. > > On Thu, 5 May 2005, joe wrote: > > > I am actually waiting for Brett or ~Eric to respond to your post as > > well. I am positive they could give you a bulleted list of > things that > > you as well as the rest of us are completely unaware of > that will go > > pear shaped both because they have seen things like that or > just know > > it from familiarity with the code paths involved. > > > > AD will not do a complete reload of the DB on its own, that > was an NT4 > > thing that occurred if the change log rolled. All gone now. > > > > Do some searching on DSA IDs/GUIDs and Invocation > IDs/GUIDS. A DSA ID > > is the GUID for the DC itself[1], it doesn't change for the life of > > the DC from my understanding. The invocation GUID[2] changes on > > restores, again to flag, hey new DB, > > [BrettSh] It's not a new DB so much, as a new logical stream > of changes to the distributed system ... > > > ... you don't know what my state is, so it can be brought into a > > consistent state. > > [BrettSh] Don't like the term "consistent state" here. I > also don't like how we're talking about the DB ... I know all > the AD repl docs, talked about it as a new database GUID, but > that was poor taste ... there is a subtle but key difference between > > [local] database consistency, and > distributed system consistency. > > It's the later we're worried about. +The later requires > multiple nodes / DCs to have followed all the rules.+ Most > of the rules are coded into the way AD behaves, when > possible. Thou shalt not image restore, is unfortunately not > coded, and hard to be defensible against ... well, without > sacraficing availability ... but lets not get into that > trade-off right now. > > > You should find hits on invocation id with topics of replication > > consistency, usn polling, AD restores, etc as it is key to > all of them > > though it has been awhile since I went searching for that stuff. > > Something I have read on a couple of occasions but can't > say I agree > > with is that allegedly the DSA ID and invocation id are identical > > unless a restore has occurred. I don't think I have EVER seen them > > identical so I don't know where that info came from. I am noting it > > simply because I recall seeing documentation to that effect > in the past. > > [BrettSh] They should've been the same until the first > restore ... there is a bug somewhere, that no one bothered to > iron out. > > BTW, we also change the InvocationID when we _re_-host an > Application Directory Partition ... I'll leave the discussion > of why to your imagination. > > Oh and since IFM is like throwing AD Restore and dcpromo into > a blender for 30 seconds, IFM based dcpromo sort of changes > the InvocationID. > You'll notice the invocationID of the DC you took the > original backup from in the retired DSA signature of the > newly dcpromo'd DC. > > > > > Really try to find detailed info on how replication works. > High USN is > > just the tip of the iceberg, there is a lot of underlying > details but > > I understand where the misconceptions can come in, a lot of the > > documentation out there in the public realm simplifies the > crap out of > > this stuff with analogies and very high level details without ever > > indicating that it is really quite more involved than that. > This can > > burn you when you start making decisions based on those > simplified examples. > > > > If you really want to get into it, start fishing through > the platform > > sdk > > Ds* API calls. I would especially recommend the > > DsGetDCInfo/DsGetDcInfo2 functions and out of those the ones > > concerning DS_REPL_NEIGHBOR structures which gives a feeling of how > > much info there is involved with replication and consistency. > > > > While it may be possible to force the invocationid to > change after the > > image restore, I am not aware of a method other than doing > a proper DB > > restore. It could be as simple as tapping that attribute in the > > nTDSDSA object but I certainly would NOT be willing to test that in > > production even if it worked great in the lab. > > [BrettSh] > > Plausible Proposal #1: (please see big warning below) > _Technically_, yes if you trigger an Invocation ID change after > you lay down the image, _AND THIS IS THE KEY_ ... before the DC > talks to any other DCs, and takes any new changes to > the database. > > This is one of those rules that all the nodes > must follow, > and if you use an AD based backup/restore program, the > appropriate logic will be triggered, and the rules for > distributed consistency upheld. > > _Even_ booting the DC, may institute a change, that causes > distributed system inconsistency. Obviously, tapping the object > from LDAP is not an option, you have to do it from DSRM. > Unfortunately, I've forgotten to tell you how you can trigger a > invocation ID change from DSRM ... > > In short don't go there. These are not the droids you're looking for. > > > > > Certainly, do not image DCs and use that as a recovery > mechanism. The > > one way to do that, IMO, would involve snap shooting and > rolling back > > all DCs in a forest at the same time. I don't see how this could > > effectively be done in the real world on real hardware. I visualize > > possibilities with virtualization software, but that would > require a > > lot of testing and work to get there and some how guarantee > that the > > snapshot was done at the exact time for all images. > > [BrettSh] > > Plausible Proposal #2: (please see the big warning below) > _Technically_, this will work too. Requires all DCs to > be off at > the same time when you take the image based back ups (I > think). Requires all the existing DCs to be turned off before > you restart the first restored image. I think that is all that > is required ... but I'm not sure ... I don't care enough to try > to give anyone > > Plausible Proposal #3: (please see the big warning below) > Of course a single DC forest can be image based > restored as well, > though ... you're more likely to get SIDs reissued, and have old > wacky ACLs in this case, b/c IIRC we invalidate the present RID > pool on restore. This can be mitigated by booting the DC, and > before creating any security principals, booting the > next rid up, > can't remember how that is done off the top of my head > though ... > > > > > If you have done this in production already, I would > recommend going > > back to what Brett said and doing a verification of your DB > on all of your DCs. > > [BrettSh] Jeez, I really hope no one is in this state, it can > be quite disturbing to iron out. > > > Again, Brett is someone who knows about the AD DB. Don't let his > > sometimes grouchy demeanor throw you off. He may get difficult at > > times but he is almost always trying to help, he just has > interesting > > ways of expressing it on occasion. He has actually been > extremely nice > > on this list compared to some other notes I have seen from him. > > [BrettSh] I thought I was being nice ... wow, it's going to > suck, when someone actually annoys me. ;) > > > Basically I say the same about him > > that I have often said about myself; don't mistake the > quality of the > > delivery for the quality of the information. :o) > > [BrettSh] > > So first let me divulge, that I am not in fact the Garage > Door Operator for building 7, in fact I am a > developer/programmer (we're call Software Development > Engineers at Microsoft) in Windows, ON Active Directory. > Before my recent move to the ESE development, I worked on the > AD Replication development for ~5.5 years, spending time > working on AD Replication, AD backup/restore, a small bit in > AD Schema/Database stuff, AD tools, and even dabbling in > DcPromo off and on when required for those years. Quite > frankly I'm the one who has dealt with almost all the areas > affected by a bad image based backup/restore, and the parts > that make a good backup/restore possible. I'm uniquely > qualified to say: > Image based backup/restores are not supported for AD. > > So we had this customer who wanted to use SAN based hot split > on Win2k AD (which is even more unsupported, as they didn't > shutdown all the DCs, like Plausible Proposal #2 above), > after explaining that they'd have to shutdown all DCs, and > them agreeing (though I doubted they'd actually do that, it's > amazing what customers will do when they think they > understand better than you) and then they agreed for restore, > they'd take ALL the DCs back to the same backup time, at the > same time, and working out this complicated set of steps they > would need, I pointed out this: > > --- begin quote --- > I can't confirm if you will fail ..., but that set of steps > if correctly followed will not cause forest corruption due to > USN rollback. Honestly, it isn.t worrying about this once > PSS guided transition that worries me, following those types > of steps once isn't hard . it is someone not understanding > why each of the parts of the technique were required, and > later trying it again, and not getting it right. In general > customers may not truly understand the system's requirements, > EVEN after they say they do (b/c they believe they do, no one > intentionally hoses their domain, but somehow it happens) so > it's just easier to say "no mirror splits on unsupported SANs" > --- end quote --- > > So .... > > Warning! Warning! Danger Will Robinson! Danger! > > So the same goes for all 3 proposals above ... while > technically you could work out the exact set of steps > required, it is likely to be an error prone manual process > ... will the next guy who maintains the corp infrastructure > understand it all ... will you miss a step ... if you have > lots of DCs in branches, how do you know one won't be missed > ... you're playing with fire ... and the slightest tweaks can > change the answer substantially, for instance auth restore > for proposal #1 must be done after triggering the invocation > ID to change, which would require a reboot ... even me with > all my knowledge, wouldn't implement such a mechanism in a > live corporate deployment ... it's subtle, and it is not > worth the risk. > > Friends don't let friends use image based backups of AD. > > Cheers, > -Brett [msft] > I'm just kidding, I just made all the above up, I really am > just the Building 7 Garage Door Operator ... > > > > > > joe > > > > > > > > [1] It is the objectGUID attribute of the ntdsdsa object(aka NTDS > > Settings object). > > [2] It is the invocationID attribute of the ntdsdsa object. > > > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Bahta > > Nathaniel V Contr NASIC/SCNA > > Sent: Thursday, May 05, 2005 10:22 AM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > Joe, > > > > I appreciate you indulging me in detail. I was just > curious on what > > the consequences may be of imaging and restoring DC's. We > are always > > evaluating and re-evaluating DR methods and techniques, and > this was > > the latest hot topic. I thought AD pushed changes up to a > > pre-determined amount and then it would just replicate the whole > > database if the number of changes were too great. I am not sure of > > the in-depth implications of restoring imaged DC's but I know the > > difference between a clean and dirty AD DB and it sounds as > though the > > metadata cleanup and synchronization is not meant to happen > with an AD > > unaware application such as ghost. Perhaps an application > that could > > stabilize an old DC with the new AD DB would be something > that would > > have to be looked at. Or maybe an image of a member server and a > > dcpromo is the easiest way to recover a DC. I have > intentions on working smarter, not harder, but that does not > forgo my lust for understanding right from wrong. > > > > Thanks again for the rebuttal. It always helps to hear things from > > all perspectives to get a better look at the big picture. > > > > Nathaniel > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > Sent: Wednesday, May 04, 2005 2:36 PM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > I'm not Brett[1] but wanted to just say something really > quick here. > > > > Well a couple of things actually. > > > > 1. When it comes to AD Database consistency and > replication. Brett is > > someone I would tend to listen to very carefully. I may not > understand > > what he is trying to say but I will try like heck to understand it. > > Rough around the edges though he may be, he knows a lot > about the guts > > of the AD DB and Replication. Keep in mind he wrote some of > the most > > "brilliant" parts of repadmin[2]. > > > > 2. When you image and recover the image you are bypassing > any and all > > logic associated with a directory DB recovery. I.E. You aren't > > restoring the database through the very specific DS > Backup/Restore API > > so you don't get the cool things that it does like renaming the > > Database GUID aka invocation ID which effectively tells all > of the other partners there is a "different" > > database out here that needs to be fully updated. > > > > I haven't fully thought out the implications of that but one thing > > right off the bat is the thought that all DCs maintain high water > > vectors for all databases so they know where they are at for > > replication. This isn't just kept on the DC in question, > this is kept > > all over so I could see serious possibilities of issues there. > > Additionally think of a change that mastered on that database and > > replicated out. How do you get it back if the DB is rolled back and > > all of the other DCs already think that DB has that info > since it was mastered there? > > > > You get ~Eric, Dean, and Brett thinking about it and I expect you > > could find all sorts of horrible things that this can do to you. > > > > I think the idea that a DC can be restored from an image like that > > because it is "sort" of like restoring the DB is flawed at the very > > best. You don't have a full comprehension of what is being > done in the > > backend to support that restore. If it were that simple, > why do you need a backup api at all? > > Mirror the DIT and zip it and there is your backup... It > doesn't work > > that way. > > > > As Brett indicated... Bad mojo... Heck I will go further, > positively evil. > > You could damage your AD in ways that you (and it) has no > clue about > > and only later run into it when you are trying to figure > out niggling > > consistency issues in applications that act odd some of the time. > > > > > > joe > > > > > > > > [1] And I couldn't play him on TV either, Brett stores a > good portion > > of his height in his hair and I store mine in my legs. > > > > [2] His words when I met him in person at an MVP summit. He > was quite > > excited to talk about that portion of the code... > > > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Bahta > > Nathaniel V Contr NASIC/SCNA > > Sent: Wednesday, May 04, 2005 1:59 PM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > Brett, > > > > What is your basis for not being able to restore a DC from > a image? > > If the DC has an old copy of the directory data, it will check its > > USN's and update its copy. What could cause havok if anything? We > > are about to institute this very same concept here to turn > DR into a > > 10 minute process when it comes to operating system > recovery. We will > > image the servers monthly and restore from said image whenever one > > crashes. What could cause a problem by restoring a DC, it will be > > timestamped to be old and AD will synchronize it with the > rest of the domain. > > > > Please elaborate on your basis for comment. > > > > Nathaniel Bahta > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of > Brett Shirley > > Sent: Wednesday, May 04, 2005 11:47 AM > > To: [email protected] > > Subject: RE: [ActiveDir] best practice? > > > > jlc, > > > > You can't restore a single DC via an image based backup, > either. It > > is not supported, it is not allowed ... it is bad mojo. > > > > Well, it wouldn't cause issues if the forest had ONLY that one DC > > (seems unlikely the case), or for a multi-DC forest, you'd have to > > shutdown all the DCs in the forest at the same time, when > you took your backup images. > > And then on restore, restore them all at the same time. > Basically a > > pretty infeasible suggestion. > > > > Cheers, > > -Brett Shirley [msft] > > > > This posting is provided "AS IS" with no warranties, and > confers no rights. > > > > > > On Wed, 4 May 2005, Joseph L. Casale wrote: > > > > > Exactly, I do it for DR purposes, the old one dies - I reimage it > > > and put it back out there. > > > No poblem... > > > jlc > > > > > > -----Original Message----- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of > Phil Renouf > > > Sent: Wednesday, May 04, 2005 7:01 AM > > > To: [email protected] > > > Subject: Re: [ActiveDir] best practice? > > > > > > On 5/4/05, John Shukovsky Jr > <[EMAIL PROTECTED]> wrote: > > > > BUT....as for DC's. I do "image" dc's using Symantec Livestate > > > > Recovery ( formerly PowerQuest V2i ). It works wonderfully. I > > > > primarily use for backups. I have not had to recover a > server in > > > > production ( and hope I do not have to ) but I have in lab 10+ > > > > times > > > and servers are as clean as ever. > > > > You should take a look. > > > > > > When Brett mentioned imaging DCs being a bad idea and to > never ever > > > do it I believe that he was meaning don't Image a DC and > try to use > > > that Image to build other new DCs and just trying to > change the SID > > > like you would for a desktop. Bad idea! > > > > > > Phil > > > List info : http://www.activedir.org/List.aspx > > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > > List archive: > > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > List info : http://www.activedir.org/List.aspx > > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > > List archive: > > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > > > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: > > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > > > List info : http://www.activedir.org/List.aspx > List FAQ : http://www.activedir.org/ListFAQ.aspx > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
