RE: [ActiveDir] How much of the DIT is cached in RAM ?
lol -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, Joseph Sent: Thursday, June 15, 2006 3:04 To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Awesome! I completely forgot about this. I did; however, thoroughly document the process so that my team can squeak the lobster whenever necessary. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Thursday, June 15, 2006 2:53 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Following up: http://msexchangeteam.com/archive/2006/06/15/427966.aspx Cheers, BrettSh On Thu, 28 Apr 2005, joe wrote: > > > Hey Brett... I've seen your blog, how about you tell ~Eric the story > > and he can blog it. :o) > > > > > > > > > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > > Sent: Thursday, April 28, 2005 8:32 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > The dev who put it in, is what I like to call "my boss" ... he has no > > child, I can guarantee it had nothing to do with that ... > > > > Email me directly the Exch product manager's name, and I'll try to > > light a fire under them ... if they don't product something, I'll > > produce something on my blog (when it is up) and send it around ... > > > > Cheers, > > BrettSh > > > > > > On Thu, 28 Apr 2005, Michael B. Smith wrote: > > > > > One of the Exchange Product Managers said today that she is > > > preparing a blog on Squeaky Lobster, including a picture of the > > > original Squeaky. I also asked about the KB and was told, simply, > > > that "it isn't currently publicly available". > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > > Sent: Thursday, April 28, 2005 7:38 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > Try - http://www.realcooltoys.com/squeakylobster.html > > > > > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > > > ESE counters. It first came to being, I believe, with Exchange 5.5 > > > where I heard two different stories, the first being that the dev > > > guy who put it in had a kid who had a squeaky lobster toy (or he had > > > it) and the other is that it was thought up after lunch. I would > > > tend to go with the first explanation myself... Anyway, it was > > > carried through and is available on AD, or at least it was on 2K AD > > > which is the last time I used it a couple of years ago. > > > > > > There used to be a KB out there that talked about what it made > > > available but I don't see it anywhere which sucks because if I need > > > it again I will have to go dig through 8 GB of PSTs and notepad > > > docs. :o) > > > > > > I want to say that I think I heard they changed (or were changing) > > > the name of this reg entry to something like "show advanced > > > counters" or something like that but I don't think I can point at > > > any references for that. > > > > > > As far as I know, this key wasn't supposed to be hidden or secret, > > > though it appears it might have gone underground. I don't think I > > > will post any more on it and let ~Eric or Brett put out in the > > > public whatever they think should be available. > > > > > > > > > joe > > > > > > > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, > > > Joseph > > > Sent: Thursday, April 28, 2005 1:31 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > This has been a great thread. I've really enjoyed reading it. > > > > > > This question is going to illustrate my extreme ignorance; however, > > > the answer is worth it. What is "Squeaky Lobster"? > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > >
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Awesome! I completely forgot about this. I did; however, thoroughly document the process so that my team can squeak the lobster whenever necessary. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Thursday, June 15, 2006 2:53 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Following up: http://msexchangeteam.com/archive/2006/06/15/427966.aspx Cheers, BrettSh On Thu, 28 Apr 2005, joe wrote: > > > Hey Brett... I've seen your blog, how about you tell ~Eric the story > > and he can blog it. :o) > > > > > > > > > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > > Sent: Thursday, April 28, 2005 8:32 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > The dev who put it in, is what I like to call "my boss" ... he has no > > child, I can guarantee it had nothing to do with that ... > > > > Email me directly the Exch product manager's name, and I'll try to > > light a fire under them ... if they don't product something, I'll > > produce something on my blog (when it is up) and send it around ... > > > > Cheers, > > BrettSh > > > > > > On Thu, 28 Apr 2005, Michael B. Smith wrote: > > > > > One of the Exchange Product Managers said today that she is > > > preparing a blog on Squeaky Lobster, including a picture of the > > > original Squeaky. I also asked about the KB and was told, simply, > > > that "it isn't currently publicly available". > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > > Sent: Thursday, April 28, 2005 7:38 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > Try - http://www.realcooltoys.com/squeakylobster.html > > > > > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > > > ESE counters. It first came to being, I believe, with Exchange 5.5 > > > where I heard two different stories, the first being that the dev > > > guy who put it in had a kid who had a squeaky lobster toy (or he had > > > it) and the other is that it was thought up after lunch. I would > > > tend to go with the first explanation myself... Anyway, it was > > > carried through and is available on AD, or at least it was on 2K AD > > > which is the last time I used it a couple of years ago. > > > > > > There used to be a KB out there that talked about what it made > > > available but I don't see it anywhere which sucks because if I need > > > it again I will have to go dig through 8 GB of PSTs and notepad > > > docs. :o) > > > > > > I want to say that I think I heard they changed (or were changing) > > > the name of this reg entry to something like "show advanced > > > counters" or something like that but I don't think I can point at > > > any references for that. > > > > > > As far as I know, this key wasn't supposed to be hidden or secret, > > > though it appears it might have gone underground. I don't think I > > > will post any more on it and let ~Eric or Brett put out in the > > > public whatever they think should be available. > > > > > > > > > joe > > > > > > > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, > > > Joseph > > > Sent: Thursday, April 28, 2005 1:31 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > This has been a great thread. I've really enjoyed reading it. > > > > > > This question is going to illustrate my extreme ignorance; however, > > > the answer is worth it. What is "Squeaky Lobster"? > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Brett > > > Shirley > > > Sent: Wednesday, April 27, 2005 3:42 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Following up: http://msexchangeteam.com/archive/2006/06/15/427966.aspx Cheers, BrettSh On Thu, 28 Apr 2005, joe wrote: > > > Hey Brett... I've seen your blog, how about you tell ~Eric the story > > and he can blog it. :o) > > > > > > > > > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > > Sent: Thursday, April 28, 2005 8:32 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > The dev who put it in, is what I like to call "my boss" ... he has no > > child, I can guarantee it had nothing to do with that ... > > > > Email me directly the Exch product manager's name, and I'll try to > > light a fire under them ... if they don't product something, I'll > > produce something on my blog (when it is up) and send it around ... > > > > Cheers, > > BrettSh > > > > > > On Thu, 28 Apr 2005, Michael B. Smith wrote: > > > > > One of the Exchange Product Managers said today that she is > > > preparing a blog on Squeaky Lobster, including a picture of the > > > original Squeaky. I also asked about the KB and was told, simply, > > > that "it isn't currently publicly available". > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > > Sent: Thursday, April 28, 2005 7:38 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > Try - http://www.realcooltoys.com/squeakylobster.html > > > > > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > > > ESE counters. It first came to being, I believe, with Exchange 5.5 > > > where I heard two different stories, the first being that the dev > > > guy who put it in had a kid who had a squeaky lobster toy (or he had > > > it) and the other is that it was thought up after lunch. I would > > > tend to go with the first explanation myself... Anyway, it was > > > carried through and is available on AD, or at least it was on 2K AD > > > which is the last time I used it a couple of years ago. > > > > > > There used to be a KB out there that talked about what it made > > > available but I don't see it anywhere which sucks because if I need > > > it again I will have to go dig through 8 GB of PSTs and notepad > > > docs. :o) > > > > > > I want to say that I think I heard they changed (or were changing) > > > the name of this reg entry to something like "show advanced > > > counters" or something like that but I don't think I can point at > > > any references for that. > > > > > > As far as I know, this key wasn't supposed to be hidden or secret, > > > though it appears it might have gone underground. I don't think I > > > will post any more on it and let ~Eric or Brett put out in the > > > public whatever they think should be available. > > > > > > > > > joe > > > > > > > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, > > > Joseph > > > Sent: Thursday, April 28, 2005 1:31 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > This has been a great thread. I've really enjoyed reading it. > > > > > > This question is going to illustrate my extreme ignorance; however, > > > the answer is worth it. What is "Squeaky Lobster"? > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Brett > > > Shirley > > > Sent: Wednesday, April 27, 2005 3:42 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > > > > >>From ESE's advanced perf counters exist, that tell you on a > > > >non-per-search > > > basis: > > > - Database Pages Transferred/sec > > > - Database Page Latches/sec > > > > > > IIRC, the first is rate of pages being transferred from disk, and > > >
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Weird point of truncation ... here's it broken in parts, attache them all together in the address part ... http://www.microsoft.com/resources/documentation/Windows/2000/ser ver/reskit/en-us/Default.asp?url=/resources/documentation/Wind ows/2000/server/reskit/en-us/distrib/dsbm_mon_pzgc.asp Cheers, BrettSh [msft] posting "as is", confers no rights, in this context I gues that means you can sue me if ridiculously long URLs corrupt your browswer ... what was ms.com thinking w/ URLs like that anyway, it's stupid. On Fri, 29 Apr 2005 [EMAIL PROTECTED] wrote: > >>> So this link gets you started enabling Windows ESE perf counters ... > >>> > http://www.microsoft.com/resources/documentation/Windows/2000/server/reskit/e > n-us/Default.asp?url=/resources/docume$ > >>> BUT ... > > That link got me this: > > We're sorry, but there is no Microsoft.com Web page that matches your entry. > It is possible you typed the address incorrectly, or the page may no longer > exist. You may wish to try another entry or choose from the links below, > which we hope will help you find what you're looking for. > > deji > List info : http://www.activedir.org/List.aspx > List FAQ: http://www.activedir.org/ListFAQ.aspx > List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ > List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
RE: [ActiveDir] How much of the DIT is cached in RAM ?
>>> So this link gets you started enabling Windows ESE perf counters ... >>> http://www.microsoft.com/resources/documentation/Windows/2000/server/reskit/e n-us/Default.asp?url=/resources/docume$ >>> BUT ... That link got me this: We're sorry, but there is no Microsoft.com Web page that matches your entry. It is possible you typed the address incorrectly, or the page may no longer exist. You may wish to try another entry or choose from the links below, which we hope will help you find what you're looking for. deji List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
RE: [ActiveDir] How much of the DIT is cached in RAM ?
I thought about that for a little while... I'd like to ask the question a little differently -- if I were to look at any two counters that was going to indicate to me that I MIGHT need to do an offline defrag -- which counters would those be and what would they tell me? :-) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Thursday, April 28, 2005 10:44 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? That would take too long ... I'll tell you want, YOU can ask about any two of them, that have a similar name prefix (i.e. start w/ the same word) ... and I'll try to answer today or tomorrow ... Cheers, -BrettSh [msft] On Thu, 28 Apr 2005, Michael B. Smith wrote: > Heh. Since YOU said that, for Exchange: > > [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESE\Performance] > "Squeaky Lobster"=dword:0001 > > (Given that the name of the feature was "Squeaky Lobster" and that it > was message-store related -- the deduction was fairly obvious.) > > Now - a description of those values - many of which have VERY > interesting names, would rock. > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Thursday, April 28, 2005 9:51 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Ignoring Joe's cruel comment about my categorically challenged blog ... > lets get back to something useful ... > > So this link gets you started enabling Windows ESE perf counters ... > > http://www.microsoft.com/resources/documentation/Windows/2000/server/r > es kit/en-us/Default.asp?url=/resources/docume$ > BUT ... > > Enabling Advanced Perf Counters: > During step 3, also add this registry value in that same > ...\Services\ESENT\Performance regkey: > Squeaky Lobster = REG_DWORD 0001 Now, technically this reg > value also works, and is the professional equivalent of the above value: > Show Advanced Counters = REG_DWORD 0001 But Eric and I try > to promote Squeaky Lobster usage whenever possible to ensure that ESE > will honor the reg value forever. > > > Then you can get started on the looking at all the fascinating > database perf counters ... oooh, I just saw one of our DC's pop to > 405k latches / second, that's cool. > > > Setting up adv perf counters is I think similar for Exchange, but for > a slight different registry key. > > > BTW, does anyone know why does that web page say copy esentprf.dll to > a seperate directory? I've never done that, seems to work fine. Huh. > > Cheers, > BrettSh [msft] > > posting "as is", don't blame me if you hose your DCs. ;) > > > > On Thu, 28 Apr 2005, joe wrote: > > > Hey Brett... I've seen your blog, how about you tell ~Eric the story > > and he can blog it. :o) > > > > > > > > > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Brett > > Shirley > > Sent: Thursday, April 28, 2005 8:32 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > The dev who put it in, is what I like to call "my boss" ... he has > > no child, I can guarantee it had nothing to do with that ... > > > > Email me directly the Exch product manager's name, and I'll try to > > light a fire under them ... if they don't product something, I'll > > produce something on my blog (when it is up) and send it around ... > > > > Cheers, > > BrettSh > > > > > > On Thu, 28 Apr 2005, Michael B. Smith wrote: > > > > > One of the Exchange Product Managers said today that she is > > > preparing a blog on Squeaky Lobster, including a picture of the > > > original Squeaky. I also asked about the KB and was told, simply, > > > that "it isn't currently publicly available". > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > > Sent: Thursday, April 28, 2005 7:38 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > Try - http://www.realcooltoys.com/squeakylobster.html > > > > > > Squeaky Lobster is a magic reg key to enable special "Squeaky > Lobster" > > > ESE co
RE: [ActiveDir] How much of the DIT is cached in RAM ?
I'll post about this (and other such things) soon if I have some time, but I would point out that tonight over dinner, an issue was raised about a topic that will probably be ok to blog about in ~20 years. It was decided that there is no suitable category for this post when the time is right. With that, it's clear that further category refinement is required, so don't hold your breath waiting for that first post. Posting without a 20 year plan would be premature, and just downright irresponsible. ~Eric -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of joe Sent: Thursday, April 28, 2005 8:19 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? :o) Hey I am watching for your blog to take off, I think it will be quite an interesting read... Once there is something to read other than categories. ;o) I am especially interested in the corporate base jumping category. I've think I have done that a few times but I always seem to forget to use a parachute... Or as Dean had someone email me today asking... "Been fired lately?". That is totally undeserved, I have only been fired twice in the last 10 years, the other time I quit. :), Seriously, good to see this info out in a place that will exist for a long time. Thanks for posting it for everyone. I have also wondered about that file copy. Never made sense to me, it is almost like they were thinking the file would get eaten in the system32 folder or something. Plus they don't have you copy all of the esentprf.* files, just the dll... I can't say I have ever done that copy. So now, about the story behind the naming of the key? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Thursday, April 28, 2005 9:51 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Ignoring Joe's cruel comment about my categorically challenged blog ... lets get back to something useful ... So this link gets you started enabling Windows ESE perf counters ... http://www.microsoft.com/resources/documentation/Windows/2000/server/res kit/ en-us/Default.asp?url=/resources/docume$ BUT ... Enabling Advanced Perf Counters: During step 3, also add this registry value in that same ...\Services\ESENT\Performance regkey: Squeaky Lobster = REG_DWORD 0001 Now, technically this reg value also works, and is the professional equivalent of the above value: Show Advanced Counters = REG_DWORD 0001 But Eric and I try to promote Squeaky Lobster usage whenever possible to ensure that ESE will honor the reg value forever. Then you can get started on the looking at all the fascinating database perf counters ... oooh, I just saw one of our DC's pop to 405k latches / second, that's cool. Setting up adv perf counters is I think similar for Exchange, but for a slight different registry key. BTW, does anyone know why does that web page say copy esentprf.dll to a seperate directory? I've never done that, seems to work fine. Huh. Cheers, BrettSh [msft] posting "as is", don't blame me if you hose your DCs. ;) On Thu, 28 Apr 2005, joe wrote: > Hey Brett... I've seen your blog, how about you tell ~Eric the story > and he can blog it. :o) > > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Thursday, April 28, 2005 8:32 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > The dev who put it in, is what I like to call "my boss" ... he has no > child, I can guarantee it had nothing to do with that ... > > Email me directly the Exch product manager's name, and I'll try to > light a fire under them ... if they don't product something, I'll > produce something on my blog (when it is up) and send it around ... > > Cheers, > BrettSh > > > On Thu, 28 Apr 2005, Michael B. Smith wrote: > > > One of the Exchange Product Managers said today that she is > > preparing a blog on Squeaky Lobster, including a picture of the > > original Squeaky. I also asked about the KB and was told, simply, > > that "it isn't currently publicly available". > > > > -----Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > Sent: Thursday, April 28, 2005 7:38 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > Try - http://www.realcooltoys.com/squeakylobster.html > > > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > >
RE: [ActiveDir] How much of the DIT is cached in RAM ?
:o) Hey I am watching for your blog to take off, I think it will be quite an interesting read... Once there is something to read other than categories. ;o) I am especially interested in the corporate base jumping category. I've think I have done that a few times but I always seem to forget to use a parachute... Or as Dean had someone email me today asking... "Been fired lately?". That is totally undeserved, I have only been fired twice in the last 10 years, the other time I quit. :), Seriously, good to see this info out in a place that will exist for a long time. Thanks for posting it for everyone. I have also wondered about that file copy. Never made sense to me, it is almost like they were thinking the file would get eaten in the system32 folder or something. Plus they don't have you copy all of the esentprf.* files, just the dll... I can't say I have ever done that copy. So now, about the story behind the naming of the key? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Thursday, April 28, 2005 9:51 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Ignoring Joe's cruel comment about my categorically challenged blog ... lets get back to something useful ... So this link gets you started enabling Windows ESE perf counters ... http://www.microsoft.com/resources/documentation/Windows/2000/server/reskit/ en-us/Default.asp?url=/resources/docume$ BUT ... Enabling Advanced Perf Counters: During step 3, also add this registry value in that same ...\Services\ESENT\Performance regkey: Squeaky Lobster = REG_DWORD 0001 Now, technically this reg value also works, and is the professional equivalent of the above value: Show Advanced Counters = REG_DWORD 0001 But Eric and I try to promote Squeaky Lobster usage whenever possible to ensure that ESE will honor the reg value forever. Then you can get started on the looking at all the fascinating database perf counters ... oooh, I just saw one of our DC's pop to 405k latches / second, that's cool. Setting up adv perf counters is I think similar for Exchange, but for a slight different registry key. BTW, does anyone know why does that web page say copy esentprf.dll to a seperate directory? I've never done that, seems to work fine. Huh. Cheers, BrettSh [msft] posting "as is", don't blame me if you hose your DCs. ;) On Thu, 28 Apr 2005, joe wrote: > Hey Brett... I've seen your blog, how about you tell ~Eric the story > and he can blog it. :o) > > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Thursday, April 28, 2005 8:32 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > The dev who put it in, is what I like to call "my boss" ... he has no > child, I can guarantee it had nothing to do with that ... > > Email me directly the Exch product manager's name, and I'll try to > light a fire under them ... if they don't product something, I'll > produce something on my blog (when it is up) and send it around ... > > Cheers, > BrettSh > > > On Thu, 28 Apr 2005, Michael B. Smith wrote: > > > One of the Exchange Product Managers said today that she is > > preparing a blog on Squeaky Lobster, including a picture of the > > original Squeaky. I also asked about the KB and was told, simply, > > that "it isn't currently publicly available". > > > > -----Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > Sent: Thursday, April 28, 2005 7:38 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > Try - http://www.realcooltoys.com/squeakylobster.html > > > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > > ESE counters. It first came to being, I believe, with Exchange 5.5 > > where I heard two different stories, the first being that the dev > > guy who put it in had a kid who had a squeaky lobster toy (or he had > > it) and the other is that it was thought up after lunch. I would > > tend to go with the first explanation myself... Anyway, it was > > carried through and is available on AD, or at least it was on 2K AD > > which is the last time I used it a couple of years ago. > > > > There used to be a KB out there that talked about what it made > > available but I don't see it anywhere which sucks because if I need > > it again I will have to go dig through 8 GB of PSTs and n
RE: [ActiveDir] How much of the DIT is cached in RAM ?
That would take too long ... I'll tell you want, YOU can ask about any two of them, that have a similar name prefix (i.e. start w/ the same word) ... and I'll try to answer today or tomorrow ... Cheers, -BrettSh [msft] On Thu, 28 Apr 2005, Michael B. Smith wrote: > Heh. Since YOU said that, for Exchange: > > [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESE\Performance] > "Squeaky Lobster"=dword:0001 > > (Given that the name of the feature was "Squeaky Lobster" and that it > was message-store related -- the deduction was fairly obvious.) > > Now - a description of those values - many of which have VERY > interesting names, would rock. > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Thursday, April 28, 2005 9:51 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Ignoring Joe's cruel comment about my categorically challenged blog ... > lets get back to something useful ... > > So this link gets you started enabling Windows ESE perf counters ... > > http://www.microsoft.com/resources/documentation/Windows/2000/server/res > kit/en-us/Default.asp?url=/resources/docume$ > BUT ... > > Enabling Advanced Perf Counters: > During step 3, also add this registry value in that same > ...\Services\ESENT\Performance regkey: > Squeaky Lobster = REG_DWORD 0001 Now, technically this reg > value also works, and is the professional equivalent of the above value: > Show Advanced Counters = REG_DWORD 0001 But Eric and I try > to promote Squeaky Lobster usage whenever possible to ensure that ESE > will honor the reg value forever. > > > Then you can get started on the looking at all the fascinating database > perf counters ... oooh, I just saw one of our DC's pop to 405k latches > / second, that's cool. > > > Setting up adv perf counters is I think similar for Exchange, but for a > slight different registry key. > > > BTW, does anyone know why does that web page say copy esentprf.dll to a > seperate directory? I've never done that, seems to work fine. Huh. > > Cheers, > BrettSh [msft] > > posting "as is", don't blame me if you hose your DCs. ;) > > > > On Thu, 28 Apr 2005, joe wrote: > > > Hey Brett... I've seen your blog, how about you tell ~Eric the story > > and he can blog it. :o) > > > > > > > > > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > > Sent: Thursday, April 28, 2005 8:32 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > The dev who put it in, is what I like to call "my boss" ... he has no > > child, I can guarantee it had nothing to do with that ... > > > > Email me directly the Exch product manager's name, and I'll try to > > light a fire under them ... if they don't product something, I'll > > produce something on my blog (when it is up) and send it around ... > > > > Cheers, > > BrettSh > > > > > > On Thu, 28 Apr 2005, Michael B. Smith wrote: > > > > > One of the Exchange Product Managers said today that she is > > > preparing a blog on Squeaky Lobster, including a picture of the > > > original Squeaky. I also asked about the KB and was told, simply, > > > that "it isn't currently publicly available". > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > > Sent: Thursday, April 28, 2005 7:38 PM > > > To: ActiveDir@mail.activedir.org > > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > > > Try - http://www.realcooltoys.com/squeakylobster.html > > > > > > Squeaky Lobster is a magic reg key to enable special "Squeaky > Lobster" > > > ESE counters. It first came to being, I believe, with Exchange 5.5 > > > where I heard two different stories, the first being that the dev > > > guy who put it in had a kid who had a squeaky lobster toy (or he had > > > > it) and the other is that it was thought up after lunch. I would > > > tend to go with the first explanation myself... Anyway, it was > > > carried through and is available on AD, or at least it was on 2K AD > > > which is the last time I used it a couple of years
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Thanks guys. Brett, don't worry I won't blame you for hosing my DC. It'll be worth it anyway :-) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Thursday, April 28, 2005 6:51 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Ignoring Joe's cruel comment about my categorically challenged blog ... lets get back to something useful ... So this link gets you started enabling Windows ESE perf counters ... http://www.microsoft.com/resources/documentation/Windows/2000/server/res kit/en-us/Default.asp?url=/resources/docume$ BUT ... Enabling Advanced Perf Counters: During step 3, also add this registry value in that same ...\Services\ESENT\Performance regkey: Squeaky Lobster = REG_DWORD 0001 Now, technically this reg value also works, and is the professional equivalent of the above value: Show Advanced Counters = REG_DWORD 0001 But Eric and I try to promote Squeaky Lobster usage whenever possible to ensure that ESE will honor the reg value forever. Then you can get started on the looking at all the fascinating database perf counters ... oooh, I just saw one of our DC's pop to 405k latches / second, that's cool. Setting up adv perf counters is I think similar for Exchange, but for a slight different registry key. BTW, does anyone know why does that web page say copy esentprf.dll to a seperate directory? I've never done that, seems to work fine. Huh. Cheers, BrettSh [msft] posting "as is", don't blame me if you hose your DCs. ;) On Thu, 28 Apr 2005, joe wrote: > Hey Brett... I've seen your blog, how about you tell ~Eric the story > and he can blog it. :o) > > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Thursday, April 28, 2005 8:32 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > The dev who put it in, is what I like to call "my boss" ... he has no > child, I can guarantee it had nothing to do with that ... > > Email me directly the Exch product manager's name, and I'll try to > light a fire under them ... if they don't product something, I'll > produce something on my blog (when it is up) and send it around ... > > Cheers, > BrettSh > > > On Thu, 28 Apr 2005, Michael B. Smith wrote: > > > One of the Exchange Product Managers said today that she is > > preparing a blog on Squeaky Lobster, including a picture of the > > original Squeaky. I also asked about the KB and was told, simply, > > that "it isn't currently publicly available". > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > Sent: Thursday, April 28, 2005 7:38 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > Try - http://www.realcooltoys.com/squeakylobster.html > > > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > > ESE counters. It first came to being, I believe, with Exchange 5.5 > > where I heard two different stories, the first being that the dev > > guy who put it in had a kid who had a squeaky lobster toy (or he had > > it) and the other is that it was thought up after lunch. I would > > tend to go with the first explanation myself... Anyway, it was > > carried through and is available on AD, or at least it was on 2K AD > > which is the last time I used it a couple of years ago. > > > > There used to be a KB out there that talked about what it made > > available but I don't see it anywhere which sucks because if I need > > it again I will have to go dig through 8 GB of PSTs and notepad > > docs. :o) > > > > I want to say that I think I heard they changed (or were changing) > > the name of this reg entry to something like "show advanced > > counters" or something like that but I don't think I can point at > > any references for that. > > > > As far as I know, this key wasn't supposed to be hidden or secret, > > though it appears it might have gone underground. I don't think I > > will post any more on it and let ~Eric or Brett put out in the > > public whatever they think should be available. > > > > > > joe > > > > > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, > > Joseph > > Sent: Thur
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Heh. Since YOU said that, for Exchange: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESE\Performance] "Squeaky Lobster"=dword:0001 (Given that the name of the feature was "Squeaky Lobster" and that it was message-store related -- the deduction was fairly obvious.) Now - a description of those values - many of which have VERY interesting names, would rock. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Thursday, April 28, 2005 9:51 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Ignoring Joe's cruel comment about my categorically challenged blog ... lets get back to something useful ... So this link gets you started enabling Windows ESE perf counters ... http://www.microsoft.com/resources/documentation/Windows/2000/server/res kit/en-us/Default.asp?url=/resources/docume$ BUT ... Enabling Advanced Perf Counters: During step 3, also add this registry value in that same ...\Services\ESENT\Performance regkey: Squeaky Lobster = REG_DWORD 0001 Now, technically this reg value also works, and is the professional equivalent of the above value: Show Advanced Counters = REG_DWORD 0001 But Eric and I try to promote Squeaky Lobster usage whenever possible to ensure that ESE will honor the reg value forever. Then you can get started on the looking at all the fascinating database perf counters ... oooh, I just saw one of our DC's pop to 405k latches / second, that's cool. Setting up adv perf counters is I think similar for Exchange, but for a slight different registry key. BTW, does anyone know why does that web page say copy esentprf.dll to a seperate directory? I've never done that, seems to work fine. Huh. Cheers, BrettSh [msft] posting "as is", don't blame me if you hose your DCs. ;) On Thu, 28 Apr 2005, joe wrote: > Hey Brett... I've seen your blog, how about you tell ~Eric the story > and he can blog it. :o) > > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Thursday, April 28, 2005 8:32 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > The dev who put it in, is what I like to call "my boss" ... he has no > child, I can guarantee it had nothing to do with that ... > > Email me directly the Exch product manager's name, and I'll try to > light a fire under them ... if they don't product something, I'll > produce something on my blog (when it is up) and send it around ... > > Cheers, > BrettSh > > > On Thu, 28 Apr 2005, Michael B. Smith wrote: > > > One of the Exchange Product Managers said today that she is > > preparing a blog on Squeaky Lobster, including a picture of the > > original Squeaky. I also asked about the KB and was told, simply, > > that "it isn't currently publicly available". > > > > -----Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > Sent: Thursday, April 28, 2005 7:38 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > Try - http://www.realcooltoys.com/squeakylobster.html > > > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > > ESE counters. It first came to being, I believe, with Exchange 5.5 > > where I heard two different stories, the first being that the dev > > guy who put it in had a kid who had a squeaky lobster toy (or he had > > it) and the other is that it was thought up after lunch. I would > > tend to go with the first explanation myself... Anyway, it was > > carried through and is available on AD, or at least it was on 2K AD > > which is the last time I used it a couple of years ago. > > > > There used to be a KB out there that talked about what it made > > available but I don't see it anywhere which sucks because if I need > > it again I will have to go dig through 8 GB of PSTs and notepad > > docs. :o) > > > > I want to say that I think I heard they changed (or were changing) > > the name of this reg entry to something like "show advanced > > counters" or something like that but I don't think I can point at > > any references for that. > > > > As far as I know, this key wasn't supposed to be hidden or secret, > > though it appears it might have gone underground. I don't think I > > will post any more on it and let ~Eric or Brett put out in the > > public what
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Ignoring Joe's cruel comment about my categorically challenged blog ... lets get back to something useful ... So this link gets you started enabling Windows ESE perf counters ... http://www.microsoft.com/resources/documentation/Windows/2000/server/reskit/en-us/Default.asp?url=/resources/docume$ BUT ... Enabling Advanced Perf Counters: During step 3, also add this registry value in that same ...\Services\ESENT\Performance regkey: Squeaky Lobster = REG_DWORD 0001 Now, technically this reg value also works, and is the professional equivalent of the above value: Show Advanced Counters = REG_DWORD 0001 But Eric and I try to promote Squeaky Lobster usage whenever possible to ensure that ESE will honor the reg value forever. Then you can get started on the looking at all the fascinating database perf counters ... oooh, I just saw one of our DC's pop to 405k latches / second, that's cool. Setting up adv perf counters is I think similar for Exchange, but for a slight different registry key. BTW, does anyone know why does that web page say copy esentprf.dll to a seperate directory? I've never done that, seems to work fine. Huh. Cheers, BrettSh [msft] posting "as is", don't blame me if you hose your DCs. ;) On Thu, 28 Apr 2005, joe wrote: > Hey Brett... I've seen your blog, how about you tell ~Eric the story and he > can blog it. :o) > > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Thursday, April 28, 2005 8:32 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > The dev who put it in, is what I like to call "my boss" ... he has no child, > I can guarantee it had nothing to do with that ... > > Email me directly the Exch product manager's name, and I'll try to light a > fire under them ... if they don't product something, I'll produce something > on my blog (when it is up) and send it around ... > > Cheers, > BrettSh > > > On Thu, 28 Apr 2005, Michael B. Smith wrote: > > > One of the Exchange Product Managers said today that she is preparing > > a blog on Squeaky Lobster, including a picture of the original > > Squeaky. I also asked about the KB and was told, simply, that "it > > isn't currently publicly available". > > > > -Original Message----- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of joe > > Sent: Thursday, April 28, 2005 7:38 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > Try - http://www.realcooltoys.com/squeakylobster.html > > > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > > ESE counters. It first came to being, I believe, with Exchange 5.5 > > where I heard two different stories, the first being that the dev guy > > who put it in had a kid who had a squeaky lobster toy (or he had it) > > and the other is that it was thought up after lunch. I would tend to > > go with the first explanation myself... Anyway, it was carried through > > and is available on AD, or at least it was on 2K AD which is the last > > time I used it a couple of years ago. > > > > There used to be a KB out there that talked about what it made > > available but I don't see it anywhere which sucks because if I need it > > again I will have to go dig through 8 GB of PSTs and notepad docs. :o) > > > > I want to say that I think I heard they changed (or were changing) the > > name of this reg entry to something like "show advanced counters" or > > something like that but I don't think I can point at any references > > for that. > > > > As far as I know, this key wasn't supposed to be hidden or secret, > > though it appears it might have gone underground. I don't think I will > > post any more on it and let ~Eric or Brett put out in the public > > whatever they think should be available. > > > > > > joe > > > > > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, > > Joseph > > Sent: Thursday, April 28, 2005 1:31 PM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > This has been a great thread. I've really enjoyed reading it. > > > > This question is going to illustrate my extreme ignorance; however, > > the answer is worth it. What is &
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Hey Brett... I've seen your blog, how about you tell ~Eric the story and he can blog it. :o) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Thursday, April 28, 2005 8:32 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? The dev who put it in, is what I like to call "my boss" ... he has no child, I can guarantee it had nothing to do with that ... Email me directly the Exch product manager's name, and I'll try to light a fire under them ... if they don't product something, I'll produce something on my blog (when it is up) and send it around ... Cheers, BrettSh On Thu, 28 Apr 2005, Michael B. Smith wrote: > One of the Exchange Product Managers said today that she is preparing > a blog on Squeaky Lobster, including a picture of the original > Squeaky. I also asked about the KB and was told, simply, that "it > isn't currently publicly available". > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of joe > Sent: Thursday, April 28, 2005 7:38 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Try - http://www.realcooltoys.com/squeakylobster.html > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > ESE counters. It first came to being, I believe, with Exchange 5.5 > where I heard two different stories, the first being that the dev guy > who put it in had a kid who had a squeaky lobster toy (or he had it) > and the other is that it was thought up after lunch. I would tend to > go with the first explanation myself... Anyway, it was carried through > and is available on AD, or at least it was on 2K AD which is the last > time I used it a couple of years ago. > > There used to be a KB out there that talked about what it made > available but I don't see it anywhere which sucks because if I need it > again I will have to go dig through 8 GB of PSTs and notepad docs. :o) > > I want to say that I think I heard they changed (or were changing) the > name of this reg entry to something like "show advanced counters" or > something like that but I don't think I can point at any references > for that. > > As far as I know, this key wasn't supposed to be hidden or secret, > though it appears it might have gone underground. I don't think I will > post any more on it and let ~Eric or Brett put out in the public > whatever they think should be available. > > > joe > > > > -----Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, > Joseph > Sent: Thursday, April 28, 2005 1:31 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > This has been a great thread. I've really enjoyed reading it. > > This question is going to illustrate my extreme ignorance; however, > the answer is worth it. What is "Squeaky Lobster"? > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Wednesday, April 27, 2005 3:42 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > >>From ESE's advanced perf counters exist, that tell you on a > >non-per-search > basis: > - Database Pages Transferred/sec > - Database Page Latches/sec > > IIRC, the first is rate of pages being transferred from disk, and the > 2nd is the rate at wich you are "making a read of something on a page > in the cache" > (that will include the read right after a page is transferred, BTW). > It doesn't give you the per query stats you were discussing, but it > does give you an idea of how much disk the DC is requiring ... > > If you were to isolate a DC from load, except your query, it could > give a _rough_ idea for a paticular query, but remember latches aren't > unique references, so if a single query internally has to read a page > several times, that will be several latch counts. > > ... > > Cheers, > -BrettSh > > On Wed, 27 Apr 2005, joe wrote: > > > I waffled on posting that at all. I am not sure I can properly > > illustrate why I think it would be good for educational info. Maybe > > just to see from the outside the deltas in speeds of the same query > > when things are in cache versus not, etc. Overall it is just another > > stat to help understand how your directory is performing. > > > >joe > > &g
RE: [ActiveDir] How much of the DIT is cached in RAM ?
The dev who put it in, is what I like to call "my boss" ... he has no child, I can guarantee it had nothing to do with that ... Email me directly the Exch product manager's name, and I'll try to light a fire under them ... if they don't product something, I'll produce something on my blog (when it is up) and send it around ... Cheers, BrettSh On Thu, 28 Apr 2005, Michael B. Smith wrote: > One of the Exchange Product Managers said today that she is preparing a > blog on Squeaky Lobster, including a picture of the original Squeaky. I > also asked about the KB and was told, simply, that "it isn't currently > publicly available". > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of joe > Sent: Thursday, April 28, 2005 7:38 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Try - http://www.realcooltoys.com/squeakylobster.html > > Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" > ESE counters. It first came to being, I believe, with Exchange 5.5 where > I heard two different stories, the first being that the dev guy who put > it in had a kid who had a squeaky lobster toy (or he had it) and the > other is that it was thought up after lunch. I would tend to go with the > first explanation myself... Anyway, it was carried through and is > available on AD, or at least it was on 2K AD which is the last time I > used it a couple of years ago. > > There used to be a KB out there that talked about what it made available > but I don't see it anywhere which sucks because if I need it again I > will have to go dig through 8 GB of PSTs and notepad docs. :o) > > I want to say that I think I heard they changed (or were changing) the > name of this reg entry to something like "show advanced counters" or > something like that but I don't think I can point at any references for > that. > > As far as I know, this key wasn't supposed to be hidden or secret, > though it appears it might have gone underground. I don't think I will > post any more on it and let ~Eric or Brett put out in the public > whatever they think should be available. > > > joe > > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, > Joseph > Sent: Thursday, April 28, 2005 1:31 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > This has been a great thread. I've really enjoyed reading it. > > This question is going to illustrate my extreme ignorance; however, the > answer is worth it. What is "Squeaky Lobster"? > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Wednesday, April 27, 2005 3:42 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > >>From ESE's advanced perf counters exist, that tell you on a > >non-per-search > basis: > - Database Pages Transferred/sec > - Database Page Latches/sec > > IIRC, the first is rate of pages being transferred from disk, and the > 2nd is the rate at wich you are "making a read of something on a page in > the cache" > (that will include the read right after a page is transferred, BTW). It > doesn't give you the per query stats you were discussing, but it does > give you an idea of how much disk the DC is requiring ... > > If you were to isolate a DC from load, except your query, it could give > a _rough_ idea for a paticular query, but remember latches aren't unique > references, so if a single query internally has to read a page several > times, that will be several latch counts. > > ... > > Cheers, > -BrettSh > > On Wed, 27 Apr 2005, joe wrote: > > > I waffled on posting that at all. I am not sure I can properly > > illustrate why I think it would be good for educational info. Maybe > > just to see from the outside the deltas in speeds of the same query > > when things are in cache versus not, etc. Overall it is just another > > stat to help understand how your directory is performing. > > > >joe > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > > Fleischman > > Sent: Wednesday, April 27, 2005 2:14 AM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > > Correcting myself inline (full of that today ar
RE: [ActiveDir] How much of the DIT is cached in RAM ?
One of the Exchange Product Managers said today that she is preparing a blog on Squeaky Lobster, including a picture of the original Squeaky. I also asked about the KB and was told, simply, that "it isn't currently publicly available". -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of joe Sent: Thursday, April 28, 2005 7:38 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Try - http://www.realcooltoys.com/squeakylobster.html Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" ESE counters. It first came to being, I believe, with Exchange 5.5 where I heard two different stories, the first being that the dev guy who put it in had a kid who had a squeaky lobster toy (or he had it) and the other is that it was thought up after lunch. I would tend to go with the first explanation myself... Anyway, it was carried through and is available on AD, or at least it was on 2K AD which is the last time I used it a couple of years ago. There used to be a KB out there that talked about what it made available but I don't see it anywhere which sucks because if I need it again I will have to go dig through 8 GB of PSTs and notepad docs. :o) I want to say that I think I heard they changed (or were changing) the name of this reg entry to something like "show advanced counters" or something like that but I don't think I can point at any references for that. As far as I know, this key wasn't supposed to be hidden or secret, though it appears it might have gone underground. I don't think I will post any more on it and let ~Eric or Brett put out in the public whatever they think should be available. joe -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, Joseph Sent: Thursday, April 28, 2005 1:31 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? This has been a great thread. I've really enjoyed reading it. This question is going to illustrate my extreme ignorance; however, the answer is worth it. What is "Squeaky Lobster"? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Wednesday, April 27, 2005 3:42 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? >From ESE's advanced perf counters exist, that tell you on a >non-per-search basis: - Database Pages Transferred/sec - Database Page Latches/sec IIRC, the first is rate of pages being transferred from disk, and the 2nd is the rate at wich you are "making a read of something on a page in the cache" (that will include the read right after a page is transferred, BTW). It doesn't give you the per query stats you were discussing, but it does give you an idea of how much disk the DC is requiring ... If you were to isolate a DC from load, except your query, it could give a _rough_ idea for a paticular query, but remember latches aren't unique references, so if a single query internally has to read a page several times, that will be several latch counts. ... Cheers, -BrettSh On Wed, 27 Apr 2005, joe wrote: > I waffled on posting that at all. I am not sure I can properly > illustrate why I think it would be good for educational info. Maybe > just to see from the outside the deltas in speeds of the same query > when things are in cache versus not, etc. Overall it is just another > stat to help understand how your directory is performing. > >joe > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Wednesday, April 27, 2005 2:14 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Correcting myself inline (full of that today aren't I?). > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Tuesday, April 26, 2005 10:41 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > I think it would be kind of interesting if the STATS control could > > tell you what % of the result set came from cache or something like > > that > > Actually, that's not really what you want. If I may, let me change > your ask in to what I think you really would like > What you really want is the % of pages touched to service the query > that were in the cache. It doesn't matter if those pages are returned > or not, it only matters that you needed the pages to effective service > the search. As that's what defines the amt of time it takes to service it. > [Efleis
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Try - http://www.realcooltoys.com/squeakylobster.html Squeaky Lobster is a magic reg key to enable special "Squeaky Lobster" ESE counters. It first came to being, I believe, with Exchange 5.5 where I heard two different stories, the first being that the dev guy who put it in had a kid who had a squeaky lobster toy (or he had it) and the other is that it was thought up after lunch. I would tend to go with the first explanation myself... Anyway, it was carried through and is available on AD, or at least it was on 2K AD which is the last time I used it a couple of years ago. There used to be a KB out there that talked about what it made available but I don't see it anywhere which sucks because if I need it again I will have to go dig through 8 GB of PSTs and notepad docs. :o) I want to say that I think I heard they changed (or were changing) the name of this reg entry to something like "show advanced counters" or something like that but I don't think I can point at any references for that. As far as I know, this key wasn't supposed to be hidden or secret, though it appears it might have gone underground. I don't think I will post any more on it and let ~Eric or Brett put out in the public whatever they think should be available. joe -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, Joseph Sent: Thursday, April 28, 2005 1:31 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? This has been a great thread. I've really enjoyed reading it. This question is going to illustrate my extreme ignorance; however, the answer is worth it. What is "Squeaky Lobster"? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Wednesday, April 27, 2005 3:42 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? >From ESE's advanced perf counters exist, that tell you on a >non-per-search basis: - Database Pages Transferred/sec - Database Page Latches/sec IIRC, the first is rate of pages being transferred from disk, and the 2nd is the rate at wich you are "making a read of something on a page in the cache" (that will include the read right after a page is transferred, BTW). It doesn't give you the per query stats you were discussing, but it does give you an idea of how much disk the DC is requiring ... If you were to isolate a DC from load, except your query, it could give a _rough_ idea for a paticular query, but remember latches aren't unique references, so if a single query internally has to read a page several times, that will be several latch counts. ... Cheers, -BrettSh On Wed, 27 Apr 2005, joe wrote: > I waffled on posting that at all. I am not sure I can properly > illustrate why I think it would be good for educational info. Maybe > just to see from the outside the deltas in speeds of the same query > when things are in cache versus not, etc. Overall it is just another > stat to help understand how your directory is performing. > >joe > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Wednesday, April 27, 2005 2:14 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Correcting myself inline (full of that today aren't I?). > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Tuesday, April 26, 2005 10:41 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > I think it would be kind of interesting if the STATS control could > > tell you what % of the result set came from cache or something like > > that > > Actually, that's not really what you want. If I may, let me change > your ask in to what I think you really would like > What you really want is the % of pages touched to service the query > that were in the cache. It doesn't matter if those pages are returned > or not, it only matters that you needed the pages to effective service > the search. As that's what defines the amt of time it takes to service it. > [Efleis] - I shouldn't say this, it isn't quite true. What I meant > was, this defines the amt of time that we would spend on I/O, should > those pages not be in memory. Other things might necessitate more time spent on the search. > > That said, assuming you got what you really want, I'm not totally sold > of the value. What will you learn? > 1) More db cache -> inefficient searches are faster > 2) Better search filter optimization -> better i
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Hey, I had to do some checking for that "squeaky Lobster" also. I *think* this is what you're looking for but I haven't had the time to test it yet. http://www.windowsitpro.com/Windows/Article/ArticleID/21879/21879.html -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, Joseph Sent: Thursday, April 28, 2005 10:31 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? This has been a great thread. I've really enjoyed reading it. This question is going to illustrate my extreme ignorance; however, the answer is worth it. What is "Squeaky Lobster"? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Wednesday, April 27, 2005 3:42 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? >From ESE's advanced perf counters exist, that tell you on a >non-per-search basis: - Database Pages Transferred/sec - Database Page Latches/sec IIRC, the first is rate of pages being transferred from disk, and the 2nd is the rate at wich you are "making a read of something on a page in the cache" (that will include the read right after a page is transferred, BTW). It doesn't give you the per query stats you were discussing, but it does give you an idea of how much disk the DC is requiring ... If you were to isolate a DC from load, except your query, it could give a _rough_ idea for a paticular query, but remember latches aren't unique references, so if a single query internally has to read a page several times, that will be several latch counts. ... Cheers, -BrettSh On Wed, 27 Apr 2005, joe wrote: > I waffled on posting that at all. I am not sure I can properly > illustrate why I think it would be good for educational info. Maybe > just to see from the outside the deltas in speeds of the same query > when things are in cache versus not, etc. Overall it is just another > stat to help understand how your directory is performing. > >joe > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Wednesday, April 27, 2005 2:14 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Correcting myself inline (full of that today aren't I?). > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Tuesday, April 26, 2005 10:41 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > I think it would be kind of interesting if the STATS control could > > tell you what % of the result set came from cache or something like > > that > > Actually, that's not really what you want. If I may, let me change > your ask in to what I think you really would like > What you really want is the % of pages touched to service the query > that were in the cache. It doesn't matter if those pages are returned > or not, it only matters that you needed the pages to effective service > the search. As that's what defines the amt of time it takes to service it. > [Efleis] - I shouldn't say this, it isn't quite true. What I meant > was, this defines the amt of time that we would spend on I/O, should > those pages not be in memory. Other things might necessitate more time spent on the search. > > That said, assuming you got what you really want, I'm not totally sold > of the value. What will you learn? > 1) More db cache -> inefficient searches are faster > 2) Better search filter optimization -> better index selection -> > faster searches with less cache needed and less I/O needed > > Searches that hit infrequently used indexes will have a lower % of > pages in memory, but still be faster than inefficient ones that hit > many pages in memory. And the avg IT admin will wonder why. :) > > Inefficient searches are still inefficient, and are still going to > require a large db cache to service them in any sort of timely manner. > How much cache? As much as you have dataset that need be traversed for > the inefficient search in question. Whatever that dataset might be. > > Sell me on the learning opportunity here? Sorry, I'm just not seeing it. > I like the idea on paper, and would be more than happy to file the bug. > I'm just not seeing what you think you can do better with this data > point than you can today. > > ~Eric > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of joe > Sent: Tues
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Thanks Al, I went to the page you suggested; however, I did not see any information that explains how to turn "Squeaky Lobster" on. Is it a reg setting? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Al Mulnick Sent: Thursday, April 28, 2005 11:09 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? http://support.microsoft.com/default.aspx?scid=%2Fservicedesks%2Fwebcast s%2Fen%2Ftranscripts%2Fwct050603.asp Not sure where it's referenced for Active Directory. Maybe somebody has a reference for that handy? al -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, Joseph Sent: Thursday, April 28, 2005 1:31 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? This has been a great thread. I've really enjoyed reading it. This question is going to illustrate my extreme ignorance; however, the answer is worth it. What is "Squeaky Lobster"? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Wednesday, April 27, 2005 3:42 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? >From ESE's advanced perf counters exist, that tell you on a >non-per-search basis: - Database Pages Transferred/sec - Database Page Latches/sec IIRC, the first is rate of pages being transferred from disk, and the 2nd is the rate at wich you are "making a read of something on a page in the cache" (that will include the read right after a page is transferred, BTW). It doesn't give you the per query stats you were discussing, but it does give you an idea of how much disk the DC is requiring ... If you were to isolate a DC from load, except your query, it could give a _rough_ idea for a paticular query, but remember latches aren't unique references, so if a single query internally has to read a page several times, that will be several latch counts. ... Cheers, -BrettSh On Wed, 27 Apr 2005, joe wrote: > I waffled on posting that at all. I am not sure I can properly > illustrate why I think it would be good for educational info. Maybe > just to see from the outside the deltas in speeds of the same query > when things are in cache versus not, etc. Overall it is just another > stat to help understand how your directory is performing. > >joe > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Wednesday, April 27, 2005 2:14 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Correcting myself inline (full of that today aren't I?). > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Tuesday, April 26, 2005 10:41 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > I think it would be kind of interesting if the STATS control could > > tell you what % of the result set came from cache or something like > > that > > Actually, that's not really what you want. If I may, let me change > your ask in to what I think you really would like > What you really want is the % of pages touched to service the query > that were in the cache. It doesn't matter if those pages are returned > or not, it only matters that you needed the pages to effective service > the search. As that's what defines the amt of time it takes to service it. > [Efleis] - I shouldn't say this, it isn't quite true. What I meant > was, this defines the amt of time that we would spend on I/O, should > those pages not be in memory. Other things might necessitate more time spent on the search. > > That said, assuming you got what you really want, I'm not totally sold > of the value. What will you learn? > 1) More db cache -> inefficient searches are faster > 2) Better search filter optimization -> better index selection -> > faster searches with less cache needed and less I/O needed > > Searches that hit infrequently used indexes will have a lower % of > pages in memory, but still be faster than inefficient ones that hit > many pages in memory. And the avg IT admin will wonder why. :) > > Inefficient searches are still inefficient, and are still going to > require a large db cache to service them in any sort of timely manner. > How much cache? As much as you have dataset that need be traversed for > the inefficient search in question. Whatever that dataset might be. > > Sell me on the learning opportunity here? Sorry, I&
RE: [ActiveDir] How much of the DIT is cached in RAM ?
' This VBScript code enables logging of DIT whitespace information in the
event log.
' ---
' From the book "Active Directory Cookbook" by Robbie Allen
' Publisher: O'Reilly and Associates
' ISBN: 0-596-00466-4
' Book web site: http://rallenhome.com/books/adcookbook/code.html
' ---
' -- SCRIPT CONFIGURATION --
strDCName = ""
' -- END CONFIGURATION -
const HKLM = &H8002
strNTDSReg = "SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics"
set objReg = GetObject("winmgmts:\\" & strDCName &
"\root\default:StdRegProv")
objReg.SetDWORDValue HKLM, strNTDSReg, "6 Garbage Collection", 1
WScript.Echo "Garbage Collection logging set to 1"
Michael Eubank
-
"PLEASE NOTE: The preceding information may be confidential or privileged.
It only should be used or disseminated for the purpose of conducting
business with Parker. If you are not an intended recipient, please notify
the sender by replying to this message and then delete the information from
your system. Thank you for your cooperation."
List info : http://www.activedir.org/List.aspx
List FAQ: http://www.activedir.org/ListFAQ.aspx
List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
RE: [ActiveDir] How much of the DIT is cached in RAM ?
http://support.microsoft.com/default.aspx?scid=%2Fservicedesks%2Fwebcast s%2Fen%2Ftranscripts%2Fwct050603.asp Not sure where it's referenced for Active Directory. Maybe somebody has a reference for that handy? al -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, Joseph Sent: Thursday, April 28, 2005 1:31 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? This has been a great thread. I've really enjoyed reading it. This question is going to illustrate my extreme ignorance; however, the answer is worth it. What is "Squeaky Lobster"? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Wednesday, April 27, 2005 3:42 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? >From ESE's advanced perf counters exist, that tell you on a >non-per-search basis: - Database Pages Transferred/sec - Database Page Latches/sec IIRC, the first is rate of pages being transferred from disk, and the 2nd is the rate at wich you are "making a read of something on a page in the cache" (that will include the read right after a page is transferred, BTW). It doesn't give you the per query stats you were discussing, but it does give you an idea of how much disk the DC is requiring ... If you were to isolate a DC from load, except your query, it could give a _rough_ idea for a paticular query, but remember latches aren't unique references, so if a single query internally has to read a page several times, that will be several latch counts. ... Cheers, -BrettSh On Wed, 27 Apr 2005, joe wrote: > I waffled on posting that at all. I am not sure I can properly > illustrate why I think it would be good for educational info. Maybe > just to see from the outside the deltas in speeds of the same query > when things are in cache versus not, etc. Overall it is just another > stat to help understand how your directory is performing. > >joe > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Wednesday, April 27, 2005 2:14 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Correcting myself inline (full of that today aren't I?). > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Tuesday, April 26, 2005 10:41 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > I think it would be kind of interesting if the STATS control could > > tell you what % of the result set came from cache or something like > > that > > Actually, that's not really what you want. If I may, let me change > your ask in to what I think you really would like > What you really want is the % of pages touched to service the query > that were in the cache. It doesn't matter if those pages are returned > or not, it only matters that you needed the pages to effective service > the search. As that's what defines the amt of time it takes to service it. > [Efleis] - I shouldn't say this, it isn't quite true. What I meant > was, this defines the amt of time that we would spend on I/O, should > those pages not be in memory. Other things might necessitate more time spent on the search. > > That said, assuming you got what you really want, I'm not totally sold > of the value. What will you learn? > 1) More db cache -> inefficient searches are faster > 2) Better search filter optimization -> better index selection -> > faster searches with less cache needed and less I/O needed > > Searches that hit infrequently used indexes will have a lower % of > pages in memory, but still be faster than inefficient ones that hit > many pages in memory. And the avg IT admin will wonder why. :) > > Inefficient searches are still inefficient, and are still going to > require a large db cache to service them in any sort of timely manner. > How much cache? As much as you have dataset that need be traversed for > the inefficient search in question. Whatever that dataset might be. > > Sell me on the learning opportunity here? Sorry, I'm just not seeing it. > I like the idea on paper, and would be more than happy to file the bug. > I'm just not seeing what you think you can do better with this data > point than you can today. > > ~Eric > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of joe > Sent: Tuesday, April 26, 2005 9:11 PM
RE: [ActiveDir] How much of the DIT is cached in RAM ?
This has been a great thread. I've really enjoyed reading it. This question is going to illustrate my extreme ignorance; however, the answer is worth it. What is "Squeaky Lobster"? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Wednesday, April 27, 2005 3:42 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? >From ESE's advanced perf counters exist, that tell you on a >non-per-search basis: - Database Pages Transferred/sec - Database Page Latches/sec IIRC, the first is rate of pages being transferred from disk, and the 2nd is the rate at wich you are "making a read of something on a page in the cache" (that will include the read right after a page is transferred, BTW). It doesn't give you the per query stats you were discussing, but it does give you an idea of how much disk the DC is requiring ... If you were to isolate a DC from load, except your query, it could give a _rough_ idea for a paticular query, but remember latches aren't unique references, so if a single query internally has to read a page several times, that will be several latch counts. ... Cheers, -BrettSh On Wed, 27 Apr 2005, joe wrote: > I waffled on posting that at all. I am not sure I can properly > illustrate why I think it would be good for educational info. Maybe > just to see from the outside the deltas in speeds of the same query > when things are in cache versus not, etc. Overall it is just another > stat to help understand how your directory is performing. > >joe > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Wednesday, April 27, 2005 2:14 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Correcting myself inline (full of that today aren't I?). > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric > Fleischman > Sent: Tuesday, April 26, 2005 10:41 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > I think it would be kind of interesting if the STATS control could > > tell you what % of the result set came from cache or something like > > that > > Actually, that's not really what you want. If I may, let me change > your ask in to what I think you really would like > What you really want is the % of pages touched to service the query > that were in the cache. It doesn't matter if those pages are returned > or not, it only matters that you needed the pages to effective service > the search. As that's what defines the amt of time it takes to service it. > [Efleis] - I shouldn't say this, it isn't quite true. What I meant > was, this defines the amt of time that we would spend on I/O, should > those pages not be in memory. Other things might necessitate more time spent on the search. > > That said, assuming you got what you really want, I'm not totally sold > of the value. What will you learn? > 1) More db cache -> inefficient searches are faster > 2) Better search filter optimization -> better index selection -> > faster searches with less cache needed and less I/O needed > > Searches that hit infrequently used indexes will have a lower % of > pages in memory, but still be faster than inefficient ones that hit > many pages in memory. And the avg IT admin will wonder why. :) > > Inefficient searches are still inefficient, and are still going to > require a large db cache to service them in any sort of timely manner. > How much cache? As much as you have dataset that need be traversed for > the inefficient search in question. Whatever that dataset might be. > > Sell me on the learning opportunity here? Sorry, I'm just not seeing it. > I like the idea on paper, and would be more than happy to file the bug. > I'm just not seeing what you think you can do better with this data > point than you can today. > > ~Eric > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of joe > Sent: Tuesday, April 26, 2005 9:11 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Thanks ~Eric. I think it would be kind of interesting if the STATS > control could tell you what % of the result set came from cache or > something like that. How feasible would something like that be? > Possibly the results of that would only be for educational reasons but > I, at least, would find that info interesting. > &g
RE: [ActiveDir] How much of the DIT is cached in RAM ?
>From ESE's advanced perf counters exist, that tell you on a non-per-search basis: - Database Pages Transferred/sec - Database Page Latches/sec IIRC, the first is rate of pages being transferred from disk, and the 2nd is the rate at wich you are "making a read of something on a page in the cache" (that will include the read right after a page is transferred, BTW). It doesn't give you the per query stats you were discussing, but it does give you an idea of how much disk the DC is requiring ... If you were to isolate a DC from load, except your query, it could give a _rough_ idea for a paticular query, but remember latches aren't unique references, so if a single query internally has to read a page several times, that will be several latch counts. ... Cheers, -BrettSh On Wed, 27 Apr 2005, joe wrote: > I waffled on posting that at all. I am not sure I can properly illustrate > why I think it would be good for educational info. Maybe just to see from > the outside the deltas in speeds of the same query when things are in cache > versus not, etc. Overall it is just another stat to help understand how your > directory is performing. > >joe > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman > Sent: Wednesday, April 27, 2005 2:14 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Correcting myself inline (full of that today aren't I?). > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman > Sent: Tuesday, April 26, 2005 10:41 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > I think it would be kind of interesting if the STATS control could > > tell you what % of the result set came from cache or something like > > that > > Actually, that's not really what you want. If I may, let me change your ask > in to what I think you really would like > What you really want is the % of pages touched to service the query that > were in the cache. It doesn't matter if those pages are returned or not, it > only matters that you needed the pages to effective service the search. As > that's what defines the amt of time it takes to service it. > [Efleis] - I shouldn't say this, it isn't quite true. What I meant was, this > defines the amt of time that we would spend on I/O, should those pages not > be in memory. Other things might necessitate more time spent on the search. > > That said, assuming you got what you really want, I'm not totally sold of > the value. What will you learn? > 1) More db cache -> inefficient searches are faster > 2) Better search filter optimization -> better index selection -> faster > searches with less cache needed and less I/O needed > > Searches that hit infrequently used indexes will have a lower % of pages in > memory, but still be faster than inefficient ones that hit many pages in > memory. And the avg IT admin will wonder why. :) > > Inefficient searches are still inefficient, and are still going to require a > large db cache to service them in any sort of timely manner. > How much cache? As much as you have dataset that need be traversed for the > inefficient search in question. Whatever that dataset might be. > > Sell me on the learning opportunity here? Sorry, I'm just not seeing it. > I like the idea on paper, and would be more than happy to file the bug. > I'm just not seeing what you think you can do better with this data point > than you can today. > > ~Eric > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of joe > Sent: Tuesday, April 26, 2005 9:11 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Thanks ~Eric. I think it would be kind of interesting if the STATS control > could tell you what % of the result set came from cache or something like > that. How feasible would something like that be? Possibly the results of > that would only be for educational reasons but I, at least, would find that > info interesting. > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman > Sent: Tuesday, April 26, 2005 8:01 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > You beat me to the reply, thanks Brett. > > A better way to think of this Joe is that a subset of the DIT is in RAM, as > much as we can fit, assuming 1) we don
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Thanks Brett, a B+-tree does make sense. I didn't intend to mean linked list as the actual data structure, but instead as the type of data recovery scheme, one node points to the next, etc. I.E. It isn't consecutive memory that can be iterated through with simple memory pointer INCRs, instead requiring more involved (or complex if you prefer) iterator type functions. As for reading up on this, I had my fill of understanding the implementation and use of B-tree and other advanced data structures 15-18 years ago when I had to deal with it regularly. I don't think I will go reaquaint myself with them now to keep this line of questioning going. It was never an area that I found a huge amount of fun in. :o) I do appreciate the time you took to work through the questions. I think I have an overall better grasp of what is going. Thanks again, joe -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Wednesday, April 27, 2005 2:13 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? No, the pages will not get loaded into consecutive pages in memory, nor do we use a linked list scheme* for the index entries ... * at least we're not using a linked list, like I think you mean it below. Our indexes are B+ trees ... which is a very standard data structure for databases (and many file systems as well). A B+tree is not related to a binary tree. Often we drop the +, and just say b-tree, though technically a B-tree is a very similar precursor to the B+ tree. A real B-tree is not a binary tree either. Hiding a significant amount of details, "the data on a page is arranged in a way that allows us to see a sorted array of node keys (node is kind of like a record ...)". This way a simple bsearch (in this case the b does stand for binary) within a page finds the next hop down the b-tree, or if on a leaf page, the row/index entry we want. Please read up on B/B+ Trees and re-ask your question ... Cheers, -BrettSh [msft] On Wed, 27 Apr 2005, joe wrote: > Excellent post Brett. This is good info that generally doesn't seem to > make it out of the corridors of msft. I appreciate you taking the time > to write this up. > > Initially your explanation bothered me about loading DIT pages as it > seems it would be more efficient to load the tables and indexes up > versus chasing from page to page for the info... However, thinking > more about it, the mechanism I am visualizing wouldn't scale with any > memory pressure, you could and probably would get into a situation > where you couldn't load an entire table or index and where would you be then? > > I am probably going to show even more ignorance on how the backend > works, but say you have an index that is spread across several pages. > Lets say those pages aren't in consecutive pages on disk, will they > get loaded into consecutive pages in memory so you can tear through it > sort of like a single structure or will it rely on some sort of a > linked list type of scheme where you jump around memory chasing the > index rows. I expect the latter and I also would expect this issue > would be minimized with the successful online defrags as you mentioned > since the indexes/tables will be collected together. > > >joe > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Tuesday, April 26, 2005 7:46 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Joe, > > When you say > > the actual DIT isn't cached in RAM, the tables, indexes, and such > > are cached. > I'd take issue with that ... that isn't a good way to explain what is > really happening. > > The DIT is most definately cached in RAM, it is cached directly 1 or > more pages at a time. Where a page is an 8k chunk for Active > Directory. We do not extrude the tables and indexes from those pages, > they stay in the pages, and we "take a latch" on that page's memory > when we want to update the page ... then later we write that 8k chunk > directly from that memory to the offest (based on it's pgno) of the DIT file it belongs at. > > Now, it is true, not all of the DIT may be cached, we'll only cache > what we need, and it will not pull in free space pages into memory (at > least in most circumstances ...? I'm thinking of prefetching might ... but lets ignore). > > I _think_ _online_ defrag (I know we're talking offline defrag below, > but mentioning online defrag is important, it is what makes offline > defrag unnecessay ... online defrag is frequently abbreviated OLD ..
RE: [ActiveDir] How much of the DIT is cached in RAM ?
I waffled on posting that at all. I am not sure I can properly illustrate
why I think it would be good for educational info. Maybe just to see from
the outside the deltas in speeds of the same query when things are in cache
versus not, etc. Overall it is just another stat to help understand how your
directory is performing.
joe
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman
Sent: Wednesday, April 27, 2005 2:14 AM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Correcting myself inline (full of that today aren't I?).
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman
Sent: Tuesday, April 26, 2005 10:41 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
> I think it would be kind of interesting if the STATS control could
> tell you what % of the result set came from cache or something like
> that
Actually, that's not really what you want. If I may, let me change your ask
in to what I think you really would like
What you really want is the % of pages touched to service the query that
were in the cache. It doesn't matter if those pages are returned or not, it
only matters that you needed the pages to effective service the search. As
that's what defines the amt of time it takes to service it.
[Efleis] - I shouldn't say this, it isn't quite true. What I meant was, this
defines the amt of time that we would spend on I/O, should those pages not
be in memory. Other things might necessitate more time spent on the search.
That said, assuming you got what you really want, I'm not totally sold of
the value. What will you learn?
1) More db cache -> inefficient searches are faster
2) Better search filter optimization -> better index selection -> faster
searches with less cache needed and less I/O needed
Searches that hit infrequently used indexes will have a lower % of pages in
memory, but still be faster than inefficient ones that hit many pages in
memory. And the avg IT admin will wonder why. :)
Inefficient searches are still inefficient, and are still going to require a
large db cache to service them in any sort of timely manner.
How much cache? As much as you have dataset that need be traversed for the
inefficient search in question. Whatever that dataset might be.
Sell me on the learning opportunity here? Sorry, I'm just not seeing it.
I like the idea on paper, and would be more than happy to file the bug.
I'm just not seeing what you think you can do better with this data point
than you can today.
~Eric
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of joe
Sent: Tuesday, April 26, 2005 9:11 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Thanks ~Eric. I think it would be kind of interesting if the STATS control
could tell you what % of the result set came from cache or something like
that. How feasible would something like that be? Possibly the results of
that would only be for educational reasons but I, at least, would find that
info interesting.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman
Sent: Tuesday, April 26, 2005 8:01 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
You beat me to the reply, thanks Brett.
A better way to think of this Joe is that a subset of the DIT is in RAM, as
much as we can fit, assuming 1) we don't run out of memory to use 2) we
don't have pressure to back off. And we try and pick the best pages to cache
("best" definition omitted for now).
The one thing we can't do today is that we can't proactively cache
something. Though I've thought a lot about whether or not it is something
that I should personally be pushing Brett's team to work on.
There's good and bad, but the bottom line today is that you can "warm"
the cache. In the absence of memory pressure, this warming technique will
help get things in the first time. But there are some things it doesn't do
1) It doesn't let you tell buffer manager to keep something in the cache no
matter what, if you think you're smarter than the buffer manager. I would
point out, almost never are you smarter than buffer manager, even when you
think you are. But that doesn't mean you won't complain that we don't have a
mechanism for it.
2) You can't really guarantee that something is in the cache with these
sorts of warming techniques. You can get close, but you can't (for
example) say "please prefetch this index". But warming the cache can do the
big stuff, like walking ancestry and pulling in the mass of the data table.
~Eric
-O
RE: [ActiveDir] How much of the DIT is cached in RAM ?
No, the pages will not get loaded into consecutive pages in memory, nor do we use a linked list scheme* for the index entries ... * at least we're not using a linked list, like I think you mean it below. Our indexes are B+ trees ... which is a very standard data structure for databases (and many file systems as well). A B+tree is not related to a binary tree. Often we drop the +, and just say b-tree, though technically a B-tree is a very similar precursor to the B+ tree. A real B-tree is not a binary tree either. Hiding a significant amount of details, "the data on a page is arranged in a way that allows us to see a sorted array of node keys (node is kind of like a record ...)". This way a simple bsearch (in this case the b does stand for binary) within a page finds the next hop down the b-tree, or if on a leaf page, the row/index entry we want. Please read up on B/B+ Trees and re-ask your question ... Cheers, -BrettSh [msft] On Wed, 27 Apr 2005, joe wrote: > Excellent post Brett. This is good info that generally doesn't seem to make > it out of the corridors of msft. I appreciate you taking the time to write > this up. > > Initially your explanation bothered me about loading DIT pages as it seems > it would be more efficient to load the tables and indexes up versus chasing > from page to page for the info... However, thinking more about it, the > mechanism I am visualizing wouldn't scale with any memory pressure, you > could and probably would get into a situation where you couldn't load an > entire table or index and where would you be then? > > I am probably going to show even more ignorance on how the backend works, > but say you have an index that is spread across several pages. Lets say > those pages aren't in consecutive pages on disk, will they get loaded into > consecutive pages in memory so you can tear through it sort of like a single > structure or will it rely on some sort of a linked list type of scheme where > you jump around memory chasing the index rows. I expect the latter and I > also would expect this issue would be minimized with the successful online > defrags as you mentioned since the indexes/tables will be collected > together. > > >joe > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley > Sent: Tuesday, April 26, 2005 7:46 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Joe, > > When you say > > the actual DIT isn't cached in RAM, the tables, indexes, and such > > are cached. > I'd take issue with that ... that isn't a good way to explain what is really > happening. > > The DIT is most definately cached in RAM, it is cached directly 1 or more > pages at a time. Where a page is an 8k chunk for Active Directory. We do > not extrude the tables and indexes from those pages, they stay in the pages, > and we "take a latch" on that page's memory when we want to update the page > ... then later we write that 8k chunk directly from that memory to the > offest (based on it's pgno) of the DIT file it belongs at. > > Now, it is true, not all of the DIT may be cached, we'll only cache what we > need, and it will not pull in free space pages into memory (at least in most > circumstances ...? I'm thinking of prefetching might ... but lets ignore). > > I _think_ _online_ defrag (I know we're talking offline defrag below, but > mentioning online defrag is important, it is what makes offline defrag > unnecessay ... online defrag is frequently abbreviated OLD ... which of > course would be the acronym of offline defrag if it had one, trust me OLD is > online defrag (at least as far as the ESE devs are concerned) ... poor taste > for a TLA in my opinion ... that was a long aside), actually logs an event > on how much free space there is in the database ... I'm 57% sure that "the > DIT size" - "that free size", is the approximate size of the non-empty data > pages (i.e. pages with data) in the DIT ... due to underflow of a record > size on a page, the actual data size is almost assuredly even less than that > ... I just made that up w/o looking at the code, so I may take that back > later ... > > You can see exactly how many bytes of the DIT file + Temp DB* are in RAM > with perfmon, counters, by using perfmon ... first set the "Squeaky Lobster" > registry key to get the advanced ESE performance counter, then use the > "Database" performance object the "Database Cache Size" counter. > Also look at the "Database Cache % Clean", b/c you should multiply those by > each other to get
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Correcting myself inline (full of that today aren't I?).
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman
Sent: Tuesday, April 26, 2005 10:41 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
> I think it would be kind of interesting if the STATS control
> could tell you what % of the result set came from cache or something
> like that
Actually, that's not really what you want. If I may, let me change your
ask in to what I think you really would like
What you really want is the % of pages touched to service the query that
were in the cache. It doesn't matter if those pages are returned or not,
it only matters that you needed the pages to effective service the
search. As that's what defines the amt of time it takes to service it.
[Efleis] - I shouldn't say this, it isn't quite true. What I meant was,
this defines the amt of time that we would spend on I/O, should those
pages not be in memory. Other things might necessitate more time spent
on the search.
That said, assuming you got what you really want, I'm not totally sold
of the value. What will you learn?
1) More db cache -> inefficient searches are faster
2) Better search filter optimization -> better index selection -> faster
searches with less cache needed and less I/O needed
Searches that hit infrequently used indexes will have a lower % of pages
in memory, but still be faster than inefficient ones that hit many pages
in memory. And the avg IT admin will wonder why. :)
Inefficient searches are still inefficient, and are still going to
require a large db cache to service them in any sort of timely manner.
How much cache? As much as you have dataset that need be traversed for
the inefficient search in question. Whatever that dataset might be.
Sell me on the learning opportunity here? Sorry, I'm just not seeing it.
I like the idea on paper, and would be more than happy to file the bug.
I'm just not seeing what you think you can do better with this data
point than you can today.
~Eric
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of joe
Sent: Tuesday, April 26, 2005 9:11 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Thanks ~Eric. I think it would be kind of interesting if the STATS
control
could tell you what % of the result set came from cache or something
like
that. How feasible would something like that be? Possibly the results of
that would only be for educational reasons but I, at least, would find
that
info interesting.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman
Sent: Tuesday, April 26, 2005 8:01 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
You beat me to the reply, thanks Brett.
A better way to think of this Joe is that a subset of the DIT is in RAM,
as
much as we can fit, assuming 1) we don't run out of memory to use 2) we
don't have pressure to back off. And we try and pick the best pages to
cache
("best" definition omitted for now).
The one thing we can't do today is that we can't proactively cache
something. Though I've thought a lot about whether or not it is
something
that I should personally be pushing Brett's team to work on.
There's good and bad, but the bottom line today is that you can "warm"
the cache. In the absence of memory pressure, this warming technique
will
help get things in the first time. But there are some things it doesn't
do
1) It doesn't let you tell buffer manager to keep something in the cache
no
matter what, if you think you're smarter than the buffer manager. I
would
point out, almost never are you smarter than buffer manager, even when
you
think you are. But that doesn't mean you won't complain that we don't
have a
mechanism for it.
2) You can't really guarantee that something is in the cache with these
sorts of warming techniques. You can get close, but you can't (for
example) say "please prefetch this index". But warming the cache can do
the
big stuff, like walking ancestry and pulling in the mass of the data
table.
~Eric
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley
Sent: Tuesday, April 26, 2005 4:46 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Joe,
When you say
> the actual DIT isn't cached in RAM, the tables, indexes, and such
> are cached.
I'd take issue with that ... that isn't a good way to explain what is
really
happening.
The DIT is most definately cached in RAM, it is cached directly 1 or
more
pages at a time. Where a page is an 8k chunk for Active Direc
RE: [ActiveDir] How much of the DIT is cached in RAM ?
> I think it would be kind of interesting if the STATS control
> could tell you what % of the result set came from cache or something
> like that
Actually, that's not really what you want. If I may, let me change your
ask in to what I think you really would like
What you really want is the % of pages touched to service the query that
were in the cache. It doesn't matter if those pages are returned or not,
it only matters that you needed the pages to effective service the
search. As that's what defines the amt of time it takes to service it.
That said, assuming you got what you really want, I'm not totally sold
of the value. What will you learn?
1) More db cache -> inefficient searches are faster
2) Better search filter optimization -> better index selection -> faster
searches with less cache needed and less I/O needed
Searches that hit infrequently used indexes will have a lower % of pages
in memory, but still be faster than inefficient ones that hit many pages
in memory. And the avg IT admin will wonder why. :)
Inefficient searches are still inefficient, and are still going to
require a large db cache to service them in any sort of timely manner.
How much cache? As much as you have dataset that need be traversed for
the inefficient search in question. Whatever that dataset might be.
Sell me on the learning opportunity here? Sorry, I'm just not seeing it.
I like the idea on paper, and would be more than happy to file the bug.
I'm just not seeing what you think you can do better with this data
point than you can today.
~Eric
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of joe
Sent: Tuesday, April 26, 2005 9:11 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Thanks ~Eric. I think it would be kind of interesting if the STATS
control
could tell you what % of the result set came from cache or something
like
that. How feasible would something like that be? Possibly the results of
that would only be for educational reasons but I, at least, would find
that
info interesting.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman
Sent: Tuesday, April 26, 2005 8:01 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
You beat me to the reply, thanks Brett.
A better way to think of this Joe is that a subset of the DIT is in RAM,
as
much as we can fit, assuming 1) we don't run out of memory to use 2) we
don't have pressure to back off. And we try and pick the best pages to
cache
("best" definition omitted for now).
The one thing we can't do today is that we can't proactively cache
something. Though I've thought a lot about whether or not it is
something
that I should personally be pushing Brett's team to work on.
There's good and bad, but the bottom line today is that you can "warm"
the cache. In the absence of memory pressure, this warming technique
will
help get things in the first time. But there are some things it doesn't
do
1) It doesn't let you tell buffer manager to keep something in the cache
no
matter what, if you think you're smarter than the buffer manager. I
would
point out, almost never are you smarter than buffer manager, even when
you
think you are. But that doesn't mean you won't complain that we don't
have a
mechanism for it.
2) You can't really guarantee that something is in the cache with these
sorts of warming techniques. You can get close, but you can't (for
example) say "please prefetch this index". But warming the cache can do
the
big stuff, like walking ancestry and pulling in the mass of the data
table.
~Eric
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley
Sent: Tuesday, April 26, 2005 4:46 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Joe,
When you say
> the actual DIT isn't cached in RAM, the tables, indexes, and such
> are cached.
I'd take issue with that ... that isn't a good way to explain what is
really
happening.
The DIT is most definately cached in RAM, it is cached directly 1 or
more
pages at a time. Where a page is an 8k chunk for Active Directory. We
do
not extrude the tables and indexes from those pages, they stay in the
pages,
and we "take a latch" on that page's memory when we want to update the
page
... then later we write that 8k chunk directly from that memory to the
offest (based on it's pgno) of the DIT file it belongs at.
Now, it is true, not all of the DIT may be cached, we'll only cache what
we
need, and it will not pull in free space pages into memory (at least in
most
circumstances ...? I'm thinking of prefetching might ... bu
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Thanks ~Eric. I think it would be kind of interesting if the STATS control
could tell you what % of the result set came from cache or something like
that. How feasible would something like that be? Possibly the results of
that would only be for educational reasons but I, at least, would find that
info interesting.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman
Sent: Tuesday, April 26, 2005 8:01 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
You beat me to the reply, thanks Brett.
A better way to think of this Joe is that a subset of the DIT is in RAM, as
much as we can fit, assuming 1) we don't run out of memory to use 2) we
don't have pressure to back off. And we try and pick the best pages to cache
("best" definition omitted for now).
The one thing we can't do today is that we can't proactively cache
something. Though I've thought a lot about whether or not it is something
that I should personally be pushing Brett's team to work on.
There's good and bad, but the bottom line today is that you can "warm"
the cache. In the absence of memory pressure, this warming technique will
help get things in the first time. But there are some things it doesn't do
1) It doesn't let you tell buffer manager to keep something in the cache no
matter what, if you think you're smarter than the buffer manager. I would
point out, almost never are you smarter than buffer manager, even when you
think you are. But that doesn't mean you won't complain that we don't have a
mechanism for it.
2) You can't really guarantee that something is in the cache with these
sorts of warming techniques. You can get close, but you can't (for
example) say "please prefetch this index". But warming the cache can do the
big stuff, like walking ancestry and pulling in the mass of the data table.
~Eric
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley
Sent: Tuesday, April 26, 2005 4:46 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Joe,
When you say
> the actual DIT isn't cached in RAM, the tables, indexes, and such
> are cached.
I'd take issue with that ... that isn't a good way to explain what is really
happening.
The DIT is most definately cached in RAM, it is cached directly 1 or more
pages at a time. Where a page is an 8k chunk for Active Directory. We do
not extrude the tables and indexes from those pages, they stay in the pages,
and we "take a latch" on that page's memory when we want to update the page
... then later we write that 8k chunk directly from that memory to the
offest (based on it's pgno) of the DIT file it belongs at.
Now, it is true, not all of the DIT may be cached, we'll only cache what we
need, and it will not pull in free space pages into memory (at least in most
circumstances ...? I'm thinking of prefetching might ... but lets ignore).
I _think_ _online_ defrag (I know we're talking offline defrag below, but
mentioning online defrag is important, it is what makes offline defrag
unnecessay ... online defrag is frequently abbreviated OLD ... which of
course would be the acronym of offline defrag if it had one, trust me OLD is
online defrag (at least as far as the ESE devs are concerned) ...
poor
taste for a TLA in my opinion ... that was a long aside), actually logs an
event on how much free space there is in the database ... I'm 57% sure that
"the DIT size" - "that free size", is the approximate size of the non-empty
data pages (i.e. pages with data) in the DIT ... due to underflow of a
record size on a page, the actual data size is almost assuredly even less
than that ... I just made that up w/o looking at the code, so I may take
that back later ...
You can see exactly how many bytes of the DIT file + Temp DB* are in RAM
with perfmon, counters, by using perfmon ... first set the "Squeaky Lobster"
registry key to get the advanced ESE performance counter, then use the
"Database" performance object the "Database Cache Size" counter.
Also look at the "Database Cache % Clean", b/c you should multiply those by
each other to get real data pages currently in memory.
* Temp DB ... so the database cache is global, so any temporary sorts we
needed to do, during LDAP queries may be taking up some of the database
cache ... I think it's like tmp.edb next to the ntds.dit file. There'd be
no technical way to subtract one from the other, but maybe just subtract the
whole tmp database size, because that gives you a lower bound on what is
definately ntds.dit.
( watch for usage of offline and online here ... ) I agree you shouldn't
worry about offline defrag, b
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Excellent post Brett. This is good info that generally doesn't seem to make it out of the corridors of msft. I appreciate you taking the time to write this up. Initially your explanation bothered me about loading DIT pages as it seems it would be more efficient to load the tables and indexes up versus chasing from page to page for the info... However, thinking more about it, the mechanism I am visualizing wouldn't scale with any memory pressure, you could and probably would get into a situation where you couldn't load an entire table or index and where would you be then? I am probably going to show even more ignorance on how the backend works, but say you have an index that is spread across several pages. Lets say those pages aren't in consecutive pages on disk, will they get loaded into consecutive pages in memory so you can tear through it sort of like a single structure or will it rely on some sort of a linked list type of scheme where you jump around memory chasing the index rows. I expect the latter and I also would expect this issue would be minimized with the successful online defrags as you mentioned since the indexes/tables will be collected together. joe -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Tuesday, April 26, 2005 7:46 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Joe, When you say > the actual DIT isn't cached in RAM, the tables, indexes, and such > are cached. I'd take issue with that ... that isn't a good way to explain what is really happening. The DIT is most definately cached in RAM, it is cached directly 1 or more pages at a time. Where a page is an 8k chunk for Active Directory. We do not extrude the tables and indexes from those pages, they stay in the pages, and we "take a latch" on that page's memory when we want to update the page ... then later we write that 8k chunk directly from that memory to the offest (based on it's pgno) of the DIT file it belongs at. Now, it is true, not all of the DIT may be cached, we'll only cache what we need, and it will not pull in free space pages into memory (at least in most circumstances ...? I'm thinking of prefetching might ... but lets ignore). I _think_ _online_ defrag (I know we're talking offline defrag below, but mentioning online defrag is important, it is what makes offline defrag unnecessay ... online defrag is frequently abbreviated OLD ... which of course would be the acronym of offline defrag if it had one, trust me OLD is online defrag (at least as far as the ESE devs are concerned) ... poor taste for a TLA in my opinion ... that was a long aside), actually logs an event on how much free space there is in the database ... I'm 57% sure that "the DIT size" - "that free size", is the approximate size of the non-empty data pages (i.e. pages with data) in the DIT ... due to underflow of a record size on a page, the actual data size is almost assuredly even less than that ... I just made that up w/o looking at the code, so I may take that back later ... You can see exactly how many bytes of the DIT file + Temp DB* are in RAM with perfmon, counters, by using perfmon ... first set the "Squeaky Lobster" registry key to get the advanced ESE performance counter, then use the "Database" performance object the "Database Cache Size" counter. Also look at the "Database Cache % Clean", b/c you should multiply those by each other to get real data pages currently in memory. * Temp DB ... so the database cache is global, so any temporary sorts we needed to do, during LDAP queries may be taking up some of the database cache ... I think it's like tmp.edb next to the ntds.dit file. There'd be no technical way to subtract one from the other, but maybe just subtract the whole tmp database size, because that gives you a lower bound on what is definately ntds.dit. ( watch for usage of offline and online here ... ) I agree you shouldn't worry about offline defrag, but you should make sure that online defrag is completing every now and then or the space wastage will grow towards (I'll make a number range here) 3-5x what it could be. Online defrag ensures that useful data is collected onto the same page when it can be, such that the number of non-empty data pages is really quite close to what you'd get if you did an offline defrag. THOUGH, you'd have free pages in the database in the online defrag case, that offline defrag would give you back in the form of a smaller DIT file. So for memory purposes, joe is right, don't worry about offline defrag, unless there are disk space issues ... but do look for the successful online defrag event. Note: There was an issue where online defrag was never completing. Both online de
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Sorry I keep forgetting things.
Brett mentioned:
> Note: There was an issue where online defrag was never completing.
This was an issue on 2k. You might want to know how you would know if
you are hitting this.it shows itself with a series of even 602's in
the event logs. If you see this, holler, and we can provide steps to
clear this. It's a trivial fix.
~Eric
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman
Sent: Tuesday, April 26, 2005 5:10 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Sorry should have said:
> I _think_ _online_ defrag actually logs an event on how much
> free space there is in the database
Yes, it should. It might require turning up GC logging (to 1?) but
either way, yes it does.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman
Sent: Tuesday, April 26, 2005 5:01 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
You beat me to the reply, thanks Brett.
A better way to think of this Joe is that a subset of the DIT is in RAM,
as much as we can fit, assuming 1) we don't run out of memory to use 2)
we don't have pressure to back off. And we try and pick the best pages
to cache ("best" definition omitted for now).
The one thing we can't do today is that we can't proactively cache
something. Though I've thought a lot about whether or not it is
something that I should personally be pushing Brett's team to work on.
There's good and bad, but the bottom line today is that you can "warm"
the cache. In the absence of memory pressure, this warming technique
will help get things in the first time. But there are some things it
doesn't do
1) It doesn't let you tell buffer manager to keep something in the cache
no matter what, if you think you're smarter than the buffer manager. I
would point out, almost never are you smarter than buffer manager, even
when you think you are. But that doesn't mean you won't complain that we
don't have a mechanism for it.
2) You can't really guarantee that something is in the cache with these
sorts of warming techniques. You can get close, but you can't (for
example) say "please prefetch this index". But warming the cache can do
the big stuff, like walking ancestry and pulling in the mass of the data
table.
~Eric
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley
Sent: Tuesday, April 26, 2005 4:46 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Joe,
When you say
> the actual DIT isn't cached in RAM, the tables, indexes, and such
> are cached.
I'd take issue with that ... that isn't a good way to explain what is
really happening.
The DIT is most definately cached in RAM, it is cached directly 1 or
more
pages at a time. Where a page is an 8k chunk for Active Directory. We
do
not extrude the tables and indexes from those pages, they stay in the
pages, and we "take a latch" on that page's memory when we want to
update
the page ... then later we write that 8k chunk directly from that memory
to the offest (based on it's pgno) of the DIT file it belongs at.
Now, it is true, not all of the DIT may be cached, we'll only cache what
we need, and it will not pull in free space pages into memory (at least
in
most circumstances ...? I'm thinking of prefetching might ... but lets
ignore).
I _think_ _online_ defrag (I know we're talking offline defrag below,
but
mentioning online defrag is important, it is what makes offline defrag
unnecessay ... online defrag is frequently abbreviated OLD ... which of
course would be the acronym of offline defrag if it had one, trust me
OLD
is online defrag (at least as far as the ESE devs are concerned) ...
poor
taste for a TLA in my opinion ... that was a long aside), actually logs
an
event on how much free space there is in the database ... I'm 57% sure
that "the DIT size" - "that free size", is the approximate size of the
non-empty data pages (i.e. pages with data) in the DIT ... due to
underflow of a record size on a page, the actual data size is almost
assuredly even less than that ... I just made that up w/o looking at
the
code, so I may take that back later ...
You can see exactly how many bytes of the DIT file + Temp DB* are in RAM
with perfmon, counters, by using perfmon ... first set the "Squeaky
Lobster" registry key to get the advanced ESE performance counter, then
use the "Database" performance object the "Database Cache Size" counter.
Also look at the "Database Cache % Clean", b/c you should multiply those
by each other to get real data pages
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Sorry should have said:
> I _think_ _online_ defrag actually logs an event on how much
> free space there is in the database
Yes, it should. It might require turning up GC logging (to 1?) but
either way, yes it does.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Eric Fleischman
Sent: Tuesday, April 26, 2005 5:01 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
You beat me to the reply, thanks Brett.
A better way to think of this Joe is that a subset of the DIT is in RAM,
as much as we can fit, assuming 1) we don't run out of memory to use 2)
we don't have pressure to back off. And we try and pick the best pages
to cache ("best" definition omitted for now).
The one thing we can't do today is that we can't proactively cache
something. Though I've thought a lot about whether or not it is
something that I should personally be pushing Brett's team to work on.
There's good and bad, but the bottom line today is that you can "warm"
the cache. In the absence of memory pressure, this warming technique
will help get things in the first time. But there are some things it
doesn't do
1) It doesn't let you tell buffer manager to keep something in the cache
no matter what, if you think you're smarter than the buffer manager. I
would point out, almost never are you smarter than buffer manager, even
when you think you are. But that doesn't mean you won't complain that we
don't have a mechanism for it.
2) You can't really guarantee that something is in the cache with these
sorts of warming techniques. You can get close, but you can't (for
example) say "please prefetch this index". But warming the cache can do
the big stuff, like walking ancestry and pulling in the mass of the data
table.
~Eric
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley
Sent: Tuesday, April 26, 2005 4:46 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Joe,
When you say
> the actual DIT isn't cached in RAM, the tables, indexes, and such
> are cached.
I'd take issue with that ... that isn't a good way to explain what is
really happening.
The DIT is most definately cached in RAM, it is cached directly 1 or
more
pages at a time. Where a page is an 8k chunk for Active Directory. We
do
not extrude the tables and indexes from those pages, they stay in the
pages, and we "take a latch" on that page's memory when we want to
update
the page ... then later we write that 8k chunk directly from that memory
to the offest (based on it's pgno) of the DIT file it belongs at.
Now, it is true, not all of the DIT may be cached, we'll only cache what
we need, and it will not pull in free space pages into memory (at least
in
most circumstances ...? I'm thinking of prefetching might ... but lets
ignore).
I _think_ _online_ defrag (I know we're talking offline defrag below,
but
mentioning online defrag is important, it is what makes offline defrag
unnecessay ... online defrag is frequently abbreviated OLD ... which of
course would be the acronym of offline defrag if it had one, trust me
OLD
is online defrag (at least as far as the ESE devs are concerned) ...
poor
taste for a TLA in my opinion ... that was a long aside), actually logs
an
event on how much free space there is in the database ... I'm 57% sure
that "the DIT size" - "that free size", is the approximate size of the
non-empty data pages (i.e. pages with data) in the DIT ... due to
underflow of a record size on a page, the actual data size is almost
assuredly even less than that ... I just made that up w/o looking at
the
code, so I may take that back later ...
You can see exactly how many bytes of the DIT file + Temp DB* are in RAM
with perfmon, counters, by using perfmon ... first set the "Squeaky
Lobster" registry key to get the advanced ESE performance counter, then
use the "Database" performance object the "Database Cache Size" counter.
Also look at the "Database Cache % Clean", b/c you should multiply those
by each other to get real data pages currently in memory.
* Temp DB ... so the database cache is global, so any temporary sorts we
needed to do, during LDAP queries may be taking up some of the database
cache ... I think it's like tmp.edb next to the ntds.dit file. There'd
be
no technical way to subtract one from the other, but maybe just subtract
the whole tmp database size, because that gives you a lower bound on
what
is definately ntds.dit.
( watch for usage of offline and online here ... )
I agree you shouldn't worry about offline defrag, but you should make
sure that online defrag is completing every now and then or the space
wastag
RE: [ActiveDir] How much of the DIT is cached in RAM ?
You beat me to the reply, thanks Brett.
A better way to think of this Joe is that a subset of the DIT is in RAM,
as much as we can fit, assuming 1) we don't run out of memory to use 2)
we don't have pressure to back off. And we try and pick the best pages
to cache ("best" definition omitted for now).
The one thing we can't do today is that we can't proactively cache
something. Though I've thought a lot about whether or not it is
something that I should personally be pushing Brett's team to work on.
There's good and bad, but the bottom line today is that you can "warm"
the cache. In the absence of memory pressure, this warming technique
will help get things in the first time. But there are some things it
doesn't do
1) It doesn't let you tell buffer manager to keep something in the cache
no matter what, if you think you're smarter than the buffer manager. I
would point out, almost never are you smarter than buffer manager, even
when you think you are. But that doesn't mean you won't complain that we
don't have a mechanism for it.
2) You can't really guarantee that something is in the cache with these
sorts of warming techniques. You can get close, but you can't (for
example) say "please prefetch this index". But warming the cache can do
the big stuff, like walking ancestry and pulling in the mass of the data
table.
~Eric
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley
Sent: Tuesday, April 26, 2005 4:46 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
Joe,
When you say
> the actual DIT isn't cached in RAM, the tables, indexes, and such
> are cached.
I'd take issue with that ... that isn't a good way to explain what is
really happening.
The DIT is most definately cached in RAM, it is cached directly 1 or
more
pages at a time. Where a page is an 8k chunk for Active Directory. We
do
not extrude the tables and indexes from those pages, they stay in the
pages, and we "take a latch" on that page's memory when we want to
update
the page ... then later we write that 8k chunk directly from that memory
to the offest (based on it's pgno) of the DIT file it belongs at.
Now, it is true, not all of the DIT may be cached, we'll only cache what
we need, and it will not pull in free space pages into memory (at least
in
most circumstances ...? I'm thinking of prefetching might ... but lets
ignore).
I _think_ _online_ defrag (I know we're talking offline defrag below,
but
mentioning online defrag is important, it is what makes offline defrag
unnecessay ... online defrag is frequently abbreviated OLD ... which of
course would be the acronym of offline defrag if it had one, trust me
OLD
is online defrag (at least as far as the ESE devs are concerned) ...
poor
taste for a TLA in my opinion ... that was a long aside), actually logs
an
event on how much free space there is in the database ... I'm 57% sure
that "the DIT size" - "that free size", is the approximate size of the
non-empty data pages (i.e. pages with data) in the DIT ... due to
underflow of a record size on a page, the actual data size is almost
assuredly even less than that ... I just made that up w/o looking at
the
code, so I may take that back later ...
You can see exactly how many bytes of the DIT file + Temp DB* are in RAM
with perfmon, counters, by using perfmon ... first set the "Squeaky
Lobster" registry key to get the advanced ESE performance counter, then
use the "Database" performance object the "Database Cache Size" counter.
Also look at the "Database Cache % Clean", b/c you should multiply those
by each other to get real data pages currently in memory.
* Temp DB ... so the database cache is global, so any temporary sorts we
needed to do, during LDAP queries may be taking up some of the database
cache ... I think it's like tmp.edb next to the ntds.dit file. There'd
be
no technical way to subtract one from the other, but maybe just subtract
the whole tmp database size, because that gives you a lower bound on
what
is definately ntds.dit.
( watch for usage of offline and online here ... )
I agree you shouldn't worry about offline defrag, but you should make
sure that online defrag is completing every now and then or the space
wastage will grow towards (I'll make a number range here) 3-5x what it
could be. Online defrag ensures that useful data is collected onto the
same page when it can be, such that the number of non-empty data pages
is
really quite close to what you'd get if you did an offline defrag.
THOUGH, you'd have free pages in the database in the online defrag case,
that offline defrag would give you back in the form of a smaller DIT
file.
So for memory purpos
RE: [ActiveDir] How much of the DIT is cached in RAM ?
reduce DIT > bloat and bring it down to a smaller size. You can accomplish the same with > a dcpromo demote and repromote and you can automate that with an unattended > script. :o) But honestly, unless you are having disk space issues, I don't > know many people who worry overly much about doing offline defrags. > > Even once you enable the counters, I am not sure if you will know whether or > not the whole DB is cached or not simply because the DIT size may not > accurately reflect how much data you really have due to free space in the > DIT. > > I saw go out and buy a 64 bit machine, load 64 bit Windows Server 2003 on it > and buy RAM = 4GB+2xDIT size and you can be pretty sure your entire DB is > cached. ;o) > > >>From the numbers Wook posted on his slide deck between poems and haiku's at > the most recent DEC you should see a remarkable increase in perf. > > joe > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Fugleberg, David A > Sent: Monday, April 18, 2005 11:36 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > The reason I asked was out of curiosity, not because of any problem. A MS > engineer told us that if the DIT is small enough in relation to the amount > of RAM in the DC, the entire DIT would be cached, increasing directory query > performance. I was just curious if there was a way to objectively measure > this. It's always interesting to measure things to see how changes affect > performance. For example, if I delete a large number of objects and wait > for the tombstones to age out, I know I could shrink the DIT with an offline > defrag. Would doing so have any measurable effect on perfomance ? I don't > know, but it would be interesting to do some before and after measurements > to find out. > > By the way, the context of the conversation was that the engineer was > recommending offline defrags after removing a large number of objects (and > waiting the requisite time for garbage collection). I have no argument with > that, but it's nice to be able to measure what if anything it's buying > (besides a smaller DIT file). Some of us are just funny that way, I > guess > > Dave > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Carlos Magalhaes > Sent: Friday, April 15, 2005 4:27 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > > Well none of the actually DIT is cached (into the RAM), IMO. The engine > might cache regular/common lookups, indexes etc but none to the actually > DC's RAM. But then again you have to define but what you mean by "into RAM". > > Nathan is quite right with "Checking the working set size of LSASS is not > reliable." There are many more processes that the LSASS is taking care of. > You could dump the LSASS process and take a look and then determine from > there what is happening. > > But now I am curious why you asking :P Do you have a hungry LSASS process? > If you do what Patch/Service Pack level do you have on that box? > > Carlos Magalhaes > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Nathan Muggli > Sent: 15 April 2005 06:10 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > Checking the working set size of LSASS is not reliable. There's process > overhead for things like lsa session handles and other stuff related to the > security sub system. > > The most accurate method is to enable the ESE Database performance counters > and look at "Cache Size". To enable the DB counters, install Server > Performance Advisor, or check out > http://www.microsoft.com/resources/documentation/Windows/2000/server/res > kit/en-us/Default.asp?url=/resources/documentation/Windows/2000/server/r > eskit/en-us/distrib/dsbm_mon_pzgc.asp > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Roger Seielstad > Sent: Thursday, April 14, 2005 8:45 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? > > By checking the working set size of by LSASS? > > > Roger Seielstad > E-mail Geek > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Fugleberg, > > David A > > Sent: Thursday, April 14, 2005 2:22 PM > > To: activedir@mail.activedir.org > > Subject:
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Possibly Eric will see my response to this and come on and smack me but I think your PSS guy may be less than accurate. It is entirely my opinion though. Reducing the physical size of the DIT I don't believe will increase the perf of your queries. As Carlos mentioned, the actual DIT isn't cached in RAM, the tables, indexes, and such are cached. The empty spaces in the DIT physical file should have little if any impact on those tables in memory unless you start looking at things like how long does it take the head to get from the physical location on the spindle of one entry of the table to the next which again, once in memory, shouldn't come into play. The big bene of offline defrag that I am aware of is simply to reduce DIT bloat and bring it down to a smaller size. You can accomplish the same with a dcpromo demote and repromote and you can automate that with an unattended script. :o) But honestly, unless you are having disk space issues, I don't know many people who worry overly much about doing offline defrags. Even once you enable the counters, I am not sure if you will know whether or not the whole DB is cached or not simply because the DIT size may not accurately reflect how much data you really have due to free space in the DIT. I saw go out and buy a 64 bit machine, load 64 bit Windows Server 2003 on it and buy RAM = 4GB+2xDIT size and you can be pretty sure your entire DB is cached. ;o) >From the numbers Wook posted on his slide deck between poems and haiku's at the most recent DEC you should see a remarkable increase in perf. joe -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Fugleberg, David A Sent: Monday, April 18, 2005 11:36 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? The reason I asked was out of curiosity, not because of any problem. A MS engineer told us that if the DIT is small enough in relation to the amount of RAM in the DC, the entire DIT would be cached, increasing directory query performance. I was just curious if there was a way to objectively measure this. It's always interesting to measure things to see how changes affect performance. For example, if I delete a large number of objects and wait for the tombstones to age out, I know I could shrink the DIT with an offline defrag. Would doing so have any measurable effect on perfomance ? I don't know, but it would be interesting to do some before and after measurements to find out. By the way, the context of the conversation was that the engineer was recommending offline defrags after removing a large number of objects (and waiting the requisite time for garbage collection). I have no argument with that, but it's nice to be able to measure what if anything it's buying (besides a smaller DIT file). Some of us are just funny that way, I guess Dave -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Carlos Magalhaes Sent: Friday, April 15, 2005 4:27 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Well none of the actually DIT is cached (into the RAM), IMO. The engine might cache regular/common lookups, indexes etc but none to the actually DC's RAM. But then again you have to define but what you mean by "into RAM". Nathan is quite right with "Checking the working set size of LSASS is not reliable." There are many more processes that the LSASS is taking care of. You could dump the LSASS process and take a look and then determine from there what is happening. But now I am curious why you asking :P Do you have a hungry LSASS process? If you do what Patch/Service Pack level do you have on that box? Carlos Magalhaes -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Nathan Muggli Sent: 15 April 2005 06:10 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Checking the working set size of LSASS is not reliable. There's process overhead for things like lsa session handles and other stuff related to the security sub system. The most accurate method is to enable the ESE Database performance counters and look at "Cache Size". To enable the DB counters, install Server Performance Advisor, or check out http://www.microsoft.com/resources/documentation/Windows/2000/server/res kit/en-us/Default.asp?url=/resources/documentation/Windows/2000/server/r eskit/en-us/distrib/dsbm_mon_pzgc.asp -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Roger Seielstad Sent: Thursday, April 14, 2005 8:45 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? By checking the working set size of by LSASS? Roger Seielstad E-mail Geek > -Original Message- >
RE: [ActiveDir] How much of the DIT is cached in RAM ?
The reason I asked was out of curiosity, not because of any problem. A MS engineer told us that if the DIT is small enough in relation to the amount of RAM in the DC, the entire DIT would be cached, increasing directory query performance. I was just curious if there was a way to objectively measure this. It's always interesting to measure things to see how changes affect performance. For example, if I delete a large number of objects and wait for the tombstones to age out, I know I could shrink the DIT with an offline defrag. Would doing so have any measurable effect on perfomance ? I don't know, but it would be interesting to do some before and after measurements to find out. By the way, the context of the conversation was that the engineer was recommending offline defrags after removing a large number of objects (and waiting the requisite time for garbage collection). I have no argument with that, but it's nice to be able to measure what if anything it's buying (besides a smaller DIT file). Some of us are just funny that way, I guess Dave -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Carlos Magalhaes Sent: Friday, April 15, 2005 4:27 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Well none of the actually DIT is cached (into the RAM), IMO. The engine might cache regular/common lookups, indexes etc but none to the actually DC's RAM. But then again you have to define but what you mean by "into RAM". Nathan is quite right with "Checking the working set size of LSASS is not reliable." There are many more processes that the LSASS is taking care of. You could dump the LSASS process and take a look and then determine from there what is happening. But now I am curious why you asking :P Do you have a hungry LSASS process? If you do what Patch/Service Pack level do you have on that box? Carlos Magalhaes -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Nathan Muggli Sent: 15 April 2005 06:10 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Checking the working set size of LSASS is not reliable. There's process overhead for things like lsa session handles and other stuff related to the security sub system. The most accurate method is to enable the ESE Database performance counters and look at "Cache Size". To enable the DB counters, install Server Performance Advisor, or check out http://www.microsoft.com/resources/documentation/Windows/2000/server/res kit/en-us/Default.asp?url=/resources/documentation/Windows/2000/server/r eskit/en-us/distrib/dsbm_mon_pzgc.asp -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Roger Seielstad Sent: Thursday, April 14, 2005 8:45 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? By checking the working set size of by LSASS? Roger Seielstad E-mail Geek > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Fugleberg, David A > Sent: Thursday, April 14, 2005 2:22 PM > To: activedir@mail.activedir.org > Subject: [ActiveDir] How much of the DIT is cached in RAM ? > > How can I determine how much of the DIT is being cached in > RAM on a given DC ? > > Dave > List info : http://www.activedir.org/List.aspx > List FAQ: http://www.activedir.org/ListFAQ.aspx > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Well none of the actually DIT is cached (into the RAM), IMO. The engine might cache regular/common lookups, indexes etc but none to the actually DC's RAM. But then again you have to define but what you mean by "into RAM". Nathan is quite right with "Checking the working set size of LSASS is not reliable." There are many more processes that the LSASS is taking care of. You could dump the LSASS process and take a look and then determine from there what is happening. But now I am curious why you asking :P Do you have a hungry LSASS process? If you do what Patch/Service Pack level do you have on that box? Carlos Magalhaes -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Nathan Muggli Sent: 15 April 2005 06:10 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? Checking the working set size of LSASS is not reliable. There's process overhead for things like lsa session handles and other stuff related to the security sub system. The most accurate method is to enable the ESE Database performance counters and look at "Cache Size". To enable the DB counters, install Server Performance Advisor, or check out http://www.microsoft.com/resources/documentation/Windows/2000/server/res kit/en-us/Default.asp?url=/resources/documentation/Windows/2000/server/r eskit/en-us/distrib/dsbm_mon_pzgc.asp -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Roger Seielstad Sent: Thursday, April 14, 2005 8:45 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? By checking the working set size of by LSASS? Roger Seielstad E-mail Geek > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Fugleberg, David A > Sent: Thursday, April 14, 2005 2:22 PM > To: activedir@mail.activedir.org > Subject: [ActiveDir] How much of the DIT is cached in RAM ? > > How can I determine how much of the DIT is being cached in > RAM on a given DC ? > > Dave > List info : http://www.activedir.org/List.aspx > List FAQ: http://www.activedir.org/ListFAQ.aspx > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
RE: [ActiveDir] How much of the DIT is cached in RAM ?
Checking the working set size of LSASS is not reliable. There's process overhead for things like lsa session handles and other stuff related to the security sub system. The most accurate method is to enable the ESE Database performance counters and look at "Cache Size". To enable the DB counters, install Server Performance Advisor, or check out http://www.microsoft.com/resources/documentation/Windows/2000/server/res kit/en-us/Default.asp?url=/resources/documentation/Windows/2000/server/r eskit/en-us/distrib/dsbm_mon_pzgc.asp -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Roger Seielstad Sent: Thursday, April 14, 2005 8:45 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ? By checking the working set size of by LSASS? Roger Seielstad E-mail Geek > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Fugleberg, David A > Sent: Thursday, April 14, 2005 2:22 PM > To: activedir@mail.activedir.org > Subject: [ActiveDir] How much of the DIT is cached in RAM ? > > How can I determine how much of the DIT is being cached in > RAM on a given DC ? > > Dave > List info : http://www.activedir.org/List.aspx > List FAQ: http://www.activedir.org/ListFAQ.aspx > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
RE: [ActiveDir] How much of the DIT is cached in RAM ?
By checking the working set size of by LSASS? Roger Seielstad E-mail Geek > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Fugleberg, David A > Sent: Thursday, April 14, 2005 2:22 PM > To: activedir@mail.activedir.org > Subject: [ActiveDir] How much of the DIT is cached in RAM ? > > How can I determine how much of the DIT is being cached in > RAM on a given DC ? > > Dave > List info : http://www.activedir.org/List.aspx > List FAQ: http://www.activedir.org/ListFAQ.aspx > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx List FAQ: http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
