Hey Brett... I've seen your blog, how about you tell ~Eric the story and he can blog it. :o)
<evilgrin> -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Thursday, April 28, 2005 8:32 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? The dev who put it in, is what I like to call "my boss" ... he has no child, I can guarantee it had nothing to do with that ... Email me directly the Exch product manager's name, and I'll try to light a fire under them ... if they don't product something, I'll produce something on my blog (when it is up) and send it around ... Cheers, BrettSh On Thu, 28 Apr 2005, Michael B. Smith wrote: > One of the Exchange Product Managers said today that she is preparing > a blog on Squeaky Lobster, including a picture of the original > Squeaky. I also asked about the KB and was told, simply, that "it > isn't currently publicly available". > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of joe > Sent: Thursday, April 28, 2005 7:38 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Try - http://www.realcooltoys.com/squeakylobster.html > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > ESE counters. It first came to being, I believe, with Exchange 5.5 > where I heard two different stories, the first being that the dev guy > who put it in had a kid who had a squeaky lobster toy (or he had it) > and the other is that it was thought up after lunch. I would tend to > go with the first explanation myself... Anyway, it was carried through > and is available on AD, or at least it was on 2K AD which is the last > time I used it a couple of years ago. > > There used to be a KB out there that talked about what it made > available but I don't see it anywhere which sucks because if I need it > again I will have to go dig through 8 GB of PSTs and notepad docs. :o) > > I want to say that I think I heard they changed (or were changing) the > name of this reg entry to something like "show advanced counters" or > something like that but I don't think I can point at any references > for that. > > As far as I know, this key wasn't supposed to be hidden or secret, > though it appears it might have gone underground. I don't think I will > post any more on it and let ~Eric or Brett put out in the public > whatever they think should be available. > > > joe > > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, > Joseph > Sent: Thursday, April 28, 2005 1:31 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > This has been a great thread. I've really enjoyed reading it. > > This question is going to illustrate my extreme ignorance; however, > the answer is worth it. What is "Squeaky Lobster"? > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Wednesday, April 27, 2005 3:42 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > >>From ESE's advanced perf counters exist, that tell you on a > >non-per-search > basis: > - Database Pages Transferred/sec > - Database Page Latches/sec > > IIRC, the first is rate of pages being transferred from disk, and the > 2nd is the rate at wich you are "making a read of something on a page > in the cache" > (that will include the read right after a page is transferred, BTW). > It doesn't give you the per query stats you were discussing, but it > does give you an idea of how much disk the DC is requiring ... > > If you were to isolate a DC from load, except your query, it could > give a _rough_ idea for a paticular query, but remember latches aren't > unique references, so if a single query internally has to read a page > several times, that will be several latch counts. > > ... > > Cheers, > -BrettSh > > On Wed, 27 Apr 2005, joe wrote: > > > I waffled on posting that at all. I am not sure I can properly > > illustrate why I think it would be good for educational info. Maybe > > just to see from the outside the deltas in speeds of the same query > > when things are in cache versus not, etc. Overall it is just another > > stat to help understand how your directory is performing. > > > > joe > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > > Fleischman > > Sent: Wednesday, April 27, 2005 2:14 AM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > Correcting myself inline (full of that today aren't I?)..... > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > > Fleischman > > Sent: Tuesday, April 26, 2005 10:41 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > I think it would be kind of interesting if the STATS control could > > > tell you what % of the result set came from cache or something > > > like that > > > > Actually, that's not really what you want. If I may, let me change > > your ask in to what I think you really would like.... > > What you really want is the % of pages touched to service the query > > that were in the cache. It doesn't matter if those pages are > > returned or not, it only matters that you needed the pages to > > effective service > > > the search. As that's what defines the amt of time it takes to > > service > it. > > [Efleis] - I shouldn't say this, it isn't quite true. What I meant > > was, this defines the amt of time that we would spend on I/O, should > > those pages not be in memory. Other things might necessitate more > > time > spent on the search. > > > > That said, assuming you got what you really want, I'm not totally > > sold > > > of the value. What will you learn? > > 1) More db cache -> inefficient searches are faster > > 2) Better search filter optimization -> better index selection -> > > faster searches with less cache needed and less I/O needed > > > > Searches that hit infrequently used indexes will have a lower % of > > pages in memory, but still be faster than inefficient ones that hit > > many pages in memory. And the avg IT admin will wonder why. :) > > > > Inefficient searches are still inefficient, and are still going to > > require a large db cache to service them in any sort of timely manner. > > How much cache? As much as you have dataset that need be traversed > > for > > > the inefficient search in question. Whatever that dataset might be. > > > > Sell me on the learning opportunity here? Sorry, I'm just not seeing > it. > > I like the idea on paper, and would be more than happy to file the > bug. > > I'm just not seeing what you think you can do better with this data > > point than you can today. > > > > ~Eric > > > > > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > Sent: Tuesday, April 26, 2005 9:11 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > Thanks ~Eric. I think it would be kind of interesting if the STATS > > control could tell you what % of the result set came from cache or > > something like that. How feasible would something like that be? > > Possibly the results of that would only be for educational reasons > > but > > > I, at least, would find that info interesting. > > > > > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > > Fleischman > > Sent: Tuesday, April 26, 2005 8:01 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > You beat me to the reply, thanks Brett. > > > > A better way to think of this Joe is that a subset of the DIT is in > > RAM, as much as we can fit, assuming 1) we don't run out of memory > > to use 2) we don't have pressure to back off. And we try and pick > > the best pages to cache ("best" definition omitted for now). > > > > The one thing we can't do today is that we can't proactively cache > > something. Though I've thought a lot about whether or not it is > > something that I should personally be pushing Brett's team to work on. > > There's good and bad, but the bottom line today is that you can "warm" > > the cache. In the absence of memory pressure, this warming technique > > will help get things in the first time. But there are some things it > > doesn't do > > 1) It doesn't let you tell buffer manager to keep something in the > > cache no matter what, if you think you're smarter than the buffer > > manager. I would point out, almost never are you smarter than buffer > > manager, even when you think you are. But that doesn't mean you > > won't complain that we don't have a mechanism for it. > > 2) You can't really guarantee that something is in the cache with > > these sorts of warming techniques. You can get close, but you can't > > (for > > example) say "please prefetch this index". But warming the cache can > > do the big stuff, like walking ancestry and pulling in the mass of > > the > data table. > > > > ~Eric > > > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Brett > > Shirley > > Sent: Tuesday, April 26, 2005 4:46 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > Joe, > > > > When you say > > > the actual DIT isn't cached in RAM, the tables, indexes, and such > > > are cached. > > I'd take issue with that ... that isn't a good way to explain what > > is really happening. > > > > The DIT is most definately cached in RAM, it is cached directly 1 or > > more pages at a time. Where a page is an 8k chunk for Active > > Directory. We do not extrude the tables and indexes from those > > pages, > > > they stay in the pages, and we "take a latch" on that page's memory > > when we want to update the page ... then later we write that 8k > > chunk directly from that memory to the offest (based on it's pgno) > > of the > DIT file it belongs at. > > > > Now, it is true, not all of the DIT may be cached, we'll only cache > > what we need, and it will not pull in free space pages into memory > > (at > > > least in most circumstances ...? I'm thinking of prefetching might ... > but lets ignore). > > > > I _think_ _online_ defrag (I know we're talking offline defrag > > below, but mentioning online defrag is important, it is what makes > > offline defrag unnecessay ... online defrag is frequently abbreviated OLD ... > > which of course would be the acronym of offline defrag if it had > > one, trust me OLD is online defrag (at least as far as the ESE devs > > are > concerned) ... > > poor > > taste for a TLA in my opinion ... that was a long aside), actually > > logs an event on how much free space there is in the database ... > > I'm 57% sure that "the DIT size" - "that free size", is the > > approximate size of the non-empty data pages (i.e. pages with data) in the DIT ... > > > due to underflow of a record size on a page, the actual data size is > > almost assuredly even less than that ... I just made that up w/o > > looking at the code, so I may take that back later ... > > > > You can see exactly how many bytes of the DIT file + Temp DB* are in > > RAM with perfmon, counters, by using perfmon ... first set the > "Squeaky Lobster" > > registry key to get the advanced ESE performance counter, then use > > the > > > "Database" performance object the "Database Cache Size" counter. > > > > Also look at the "Database Cache % Clean", b/c you should multiply > > those by each other to get real data pages currently in memory. > > > > * Temp DB ... so the database cache is global, so any temporary > > sorts we needed to do, during LDAP queries may be taking up some of > > the database cache ... I think it's like tmp.edb next to the > > ntds.dit file. There'd be no technical way to subtract one from the > > other, but > > > maybe just subtract the whole tmp database size, because that gives > > you a lower bound on what is definately ntds.dit. > > > > ( watch for usage of offline and online here ... ) I agree you > > shouldn't worry about offline defrag, but you should make sure that > > online defrag is completing every now and then or the space wastage > > will grow towards (I'll make a number range here) 3-5x what it could > > be. Online defrag ensures that useful data is collected onto the > > same > > > page when it can be, such that the number of non-empty data pages is > > really quite close to what you'd get if you did an offline defrag. > > THOUGH, you'd have free pages in the database in the online defrag > > case, that offline defrag would give you back in the form of a > > smaller > DIT file. > > So for memory purposes, joe is right, don't worry about offline > > defrag, unless there are disk space issues ... but do look for the > > successful online defrag event. > > Note: There was an issue where online defrag was never > completing. > > > > Both online defrag and offline defrag basically scrunch all the data > > closer to where it belongs (on a per table, per index, etc basis), > > with the online version leaving white space in between "places" ... > > BUT all that said, there is technically one difference between > > online defrag and offline defrag data layout ... the offline defrag > > will reorder burst long values, into a order that matches the rows > > in the database ... I don't feel lik delving into that yet ... > > > > That's off the top of my head, I'll check facts, and try to write > > more > > > later ... > > > > Cheers, > > Brett Shirley [msft] > > > > > > posting is as is, but ... > > > > On Tue, 26 Apr 2005, joe wrote: > > > > > Possibly Eric will see my response to this and come on and smack > > > me > > but I > > > think your PSS guy may be less than accurate. It is entirely my > > opinion > > > though. > > > > > > Reducing the physical size of the DIT I don't believe will > > > increase > > the perf > > > of your queries. As Carlos mentioned, the actual DIT isn't cached > > > in > > RAM, > > > the tables, indexes, and such are cached. The empty spaces in the > > > DIT physical file should have little if any impact on those tables > > > in > > memory > > > unless you start looking at things like how long does it take the > > > head > > to > > > get from the physical location on the spindle of one entry of the > > table to > > > the next which again, once in memory, shouldn't come into play. > > > > > > The big bene of offline defrag that I am aware of is simply to > > > reduce > > DIT > > > bloat and bring it down to a smaller size. You can accomplish the > > > same > > with > > > a dcpromo demote and repromote and you can automate that with an > > unattended > > > script. :o) But honestly, unless you are having disk space > > > issues, I > > don't > > > know many people who worry overly much about doing offline defrags. > > > > > > Even once you enable the counters, I am not sure if you will know > > whether or > > > not the whole DB is cached or not simply because the DIT size may > > > not accurately reflect how much data you really have due to free > > > space in > > the > > > DIT. > > > > > > I saw go out and buy a 64 bit machine, load 64 bit Windows Server > > > 2003 > > on it > > > and buy RAM = 4GB+2xDIT size and you can be pretty sure your > > > entire DB > > is > > > cached. ;o) > > > > > > >>From the numbers Wook posted on his slide deck between poems and > > haiku's at > > > the most recent DEC you should see a remarkable increase in perf. > > > > > > joe > > > > > > > > > > > > > > > -----Original Message----- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of > > > Fugleberg, > > David A > > > Sent: Monday, April 18, 2005 11:36 AM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > The reason I asked was out of curiosity, not because of any problem. > > A MS > > > engineer told us that if the DIT is small enough in relation to > > > the > > amount > > > of RAM in the DC, the entire DIT would be cached, increasing > > > directory > > query > > > performance. I was just curious if there was a way to objectively > > measure > > > this. It's always interesting to measure things to see how > > > changes > > affect > > > performance. For example, if I delete a large number of objects > > > and > > wait > > > for the tombstones to age out, I know I could shrink the DIT with > > > an > > offline > > > defrag. Would doing so have any measurable effect on perfomance ? > > > I > > don't > > > know, but it would be interesting to do some before and after > > measurements > > > to find out. > > > > > > By the way, the context of the conversation was that the engineer > > > was recommending offline defrags after removing a large number of > > > objects > > (and > > > waiting the requisite time for garbage collection). I have no > > argument with > > > that, but it's nice to be able to measure what if anything it's > > > buying > > > > > (besides a smaller DIT file). Some of us are just funny that way, > > > I guess.... > > > > > > Dave > > > > > > -----Original Message----- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Carlos > > Magalhaes > > > Sent: Friday, April 15, 2005 4:27 AM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > > > > Well none of the actually DIT is cached (into the RAM), IMO. The > > engine > > > might cache regular/common lookups, indexes etc but none to the > > actually > > > DC's RAM. But then again you have to define but what you mean by > > > "into > > RAM". > > > > > > Nathan is quite right with "Checking the working set size of LSASS > > > is > > not > > > reliable." There are many more processes that the LSASS is taking > > > care > > of. > > > You could dump the LSASS process and take a look and then > > > determine > > from > > > there what is happening. > > > > > > But now I am curious why you asking :P Do you have a hungry LSASS > > process? > > > If you do what Patch/Service Pack level do you have on that box? > > > > > > Carlos Magalhaes > > > > > > -----Original Message----- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Nathan > > > Muggli > > > Sent: 15 April 2005 06:10 AM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > Checking the working set size of LSASS is not reliable. There's > > process > > > overhead for things like lsa session handles and other stuff > > > related > > to the > > > security sub system. > > > > > > The most accurate method is to enable the ESE Database performance > > counters > > > and look at "Cache Size". To enable the DB counters, install > > > Server Performance Advisor, or check out > > > > > http://www.microsoft.com/resources/documentation/Windows/2000/server > > /r > > es > > > > > kit/en-us/Default.asp?url=/resources/documentation/Windows/2000/serv > > er > > /r > > > eskit/en-us/distrib/dsbm_mon_pzgc.asp > > > > > > > > > > > > -----Original Message----- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Roger > > Seielstad > > > Sent: Thursday, April 14, 2005 8:45 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > By checking the working set size of by LSASS? > > > > > > -------- > > > Roger Seielstad > > > E-mail Geek > > > > > > > -----Original Message----- > > > > From: [EMAIL PROTECTED] > > > > [mailto:[EMAIL PROTECTED] On Behalf Of > > > > Fugleberg, David A > > > > Sent: Thursday, April 14, 2005 2:22 PM > > > > To: activedir@mail.activedir.org > > > > Subject: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > > > How can I determine how much of the DIT is being cached in RAM > > > > on a given DC ? > > > > > > > > Dave > > List info : http://www.activedir.org/List.aspx > List FAQ : http://www.activedir.org/ListFAQ.aspx > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > List info : http://www.activedir.org/List.aspx > List FAQ : http://www.activedir.org/ListFAQ.aspx > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
