Joe,
When you say
> the actual DIT isn't cached in RAM, the tables, indexes, and such
> are cached.
I'd take issue with that ... that isn't a good way to explain what is
really happening.
The DIT is most definately cached in RAM, it is cached directly 1 or more
pages at a time. Where a page is an 8k chunk for Active Directory. We do
not extrude the tables and indexes from those pages, they stay in the
pages, and we "take a latch" on that page's memory when we want to update
the page ... then later we write that 8k chunk directly from that memory
to the offest (based on it's pgno) of the DIT file it belongs at.
Now, it is true, not all of the DIT may be cached, we'll only cache what
we need, and it will not pull in free space pages into memory (at least in
most circumstances ...? I'm thinking of prefetching might ... but lets
ignore).
I _think_ _online_ defrag (I know we're talking offline defrag below, but
mentioning online defrag is important, it is what makes offline defrag
unnecessay ... online defrag is frequently abbreviated OLD ... which of
course would be the acronym of offline defrag if it had one, trust me OLD
is online defrag (at least as far as the ESE devs are concerned) ... poor
taste for a TLA in my opinion ... that was a long aside), actually logs an
event on how much free space there is in the database ... I'm 57% sure
that "the DIT size" - "that free size", is the approximate size of the
non-empty data pages (i.e. pages with data) in the DIT ... due to
underflow of a record size on a page, the actual data size is almost
assuredly even less than that ... I just made that up w/o looking at the
code, so I may take that back later ...
You can see exactly how many bytes of the DIT file + Temp DB* are in RAM
with perfmon, counters, by using perfmon ... first set the "Squeaky
Lobster" registry key to get the advanced ESE performance counter, then
use the "Database" performance object the "Database Cache Size" counter.
Also look at the "Database Cache % Clean", b/c you should multiply those
by each other to get real data pages currently in memory.
* Temp DB ... so the database cache is global, so any temporary sorts we
needed to do, during LDAP queries may be taking up some of the database
cache ... I think it's like tmp.edb next to the ntds.dit file. There'd be
no technical way to subtract one from the other, but maybe just subtract
the whole tmp database size, because that gives you a lower bound on what
is definately ntds.dit.
( watch for usage of offline and online here ... )
I agree you shouldn't worry about offline defrag, but you should make
sure that online defrag is completing every now and then or the space
wastage will grow towards (I'll make a number range here) 3-5x what it
could be. Online defrag ensures that useful data is collected onto the
same page when it can be, such that the number of non-empty data pages is
really quite close to what you'd get if you did an offline defrag.
THOUGH, you'd have free pages in the database in the online defrag case,
that offline defrag would give you back in the form of a smaller DIT file.
So for memory purposes, joe is right, don't worry about offline defrag,
unless there are disk space issues ... but do look for the successful
online defrag event.
Note: There was an issue where online defrag was never completing.
Both online defrag and offline defrag basically scrunch all the data
closer to where it belongs (on a per table, per index, etc basis), with
the online version leaving white space in between "places" ... BUT all
that said, there is technically one difference between online defrag and
offline defrag data layout ... the offline defrag will reorder burst long
values, into a order that matches the rows in the database ... I don't
feel lik delving into that yet ...
That's off the top of my head, I'll check facts, and try to write more
later ...
Cheers,
Brett Shirley [msft]
posting is as is, but ...
On Tue, 26 Apr 2005, joe wrote:
> Possibly Eric will see my response to this and come on and smack me but I
> think your PSS guy may be less than accurate. It is entirely my opinion
> though.
>
> Reducing the physical size of the DIT I don't believe will increase the perf
> of your queries. As Carlos mentioned, the actual DIT isn't cached in RAM,
> the tables, indexes, and such are cached. The empty spaces in the DIT
> physical file should have little if any impact on those tables in memory
> unless you start looking at things like how long does it take the head to
> get from the physical location on the spindle of one entry of the table to
> the next which again, once in memory, shouldn't come into play.
>
> The big bene of offline defrag that I am aware of is simply to reduce DIT
> bloat and bring it down to a smaller size. You can accomplish the same with
> a dcpromo demote and repromote and you can automate that with an unattended
> script. :o) But honestly, unless you are having disk space issues, I don't
> know many people who worry overly much about doing offline defrags.
>
> Even once you enable the counters, I am not sure if you will know whether or
> not the whole DB is cached or not simply because the DIT size may not
> accurately reflect how much data you really have due to free space in the
> DIT.
>
> I saw go out and buy a 64 bit machine, load 64 bit Windows Server 2003 on it
> and buy RAM = 4GB+2xDIT size and you can be pretty sure your entire DB is
> cached. ;o)
>
> >>From the numbers Wook posted on his slide deck between poems and haiku's at
> the most recent DEC you should see a remarkable increase in perf.
>
> joe
>
>
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Fugleberg, David A
> Sent: Monday, April 18, 2005 11:36 AM
> To: [email protected]
> Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
>
> The reason I asked was out of curiosity, not because of any problem. A MS
> engineer told us that if the DIT is small enough in relation to the amount
> of RAM in the DC, the entire DIT would be cached, increasing directory query
> performance. I was just curious if there was a way to objectively measure
> this. It's always interesting to measure things to see how changes affect
> performance. For example, if I delete a large number of objects and wait
> for the tombstones to age out, I know I could shrink the DIT with an offline
> defrag. Would doing so have any measurable effect on perfomance ? I don't
> know, but it would be interesting to do some before and after measurements
> to find out.
>
> By the way, the context of the conversation was that the engineer was
> recommending offline defrags after removing a large number of objects (and
> waiting the requisite time for garbage collection). I have no argument with
> that, but it's nice to be able to measure what if anything it's buying
> (besides a smaller DIT file). Some of us are just funny that way, I
> guess....
>
> Dave
>
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Carlos Magalhaes
> Sent: Friday, April 15, 2005 4:27 AM
> To: [email protected]
> Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
>
>
> Well none of the actually DIT is cached (into the RAM), IMO. The engine
> might cache regular/common lookups, indexes etc but none to the actually
> DC's RAM. But then again you have to define but what you mean by "into RAM".
>
> Nathan is quite right with "Checking the working set size of LSASS is not
> reliable." There are many more processes that the LSASS is taking care of.
> You could dump the LSASS process and take a look and then determine from
> there what is happening.
>
> But now I am curious why you asking :P Do you have a hungry LSASS process?
> If you do what Patch/Service Pack level do you have on that box?
>
> Carlos Magalhaes
>
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Nathan Muggli
> Sent: 15 April 2005 06:10 AM
> To: [email protected]
> Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
>
> Checking the working set size of LSASS is not reliable. There's process
> overhead for things like lsa session handles and other stuff related to the
> security sub system.
>
> The most accurate method is to enable the ESE Database performance counters
> and look at "Cache Size". To enable the DB counters, install Server
> Performance Advisor, or check out
> http://www.microsoft.com/resources/documentation/Windows/2000/server/res
> kit/en-us/Default.asp?url=/resources/documentation/Windows/2000/server/r
> eskit/en-us/distrib/dsbm_mon_pzgc.asp
>
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Roger Seielstad
> Sent: Thursday, April 14, 2005 8:45 PM
> To: [email protected]
> Subject: RE: [ActiveDir] How much of the DIT is cached in RAM ?
>
> By checking the working set size of by LSASS?
>
> --------
> Roger Seielstad
> E-mail Geek
>
> > -----Original Message-----
> > From: [EMAIL PROTECTED]
> > [mailto:[EMAIL PROTECTED] On Behalf Of Fugleberg,
> > David A
> > Sent: Thursday, April 14, 2005 2:22 PM
> > To: [email protected]
> > Subject: [ActiveDir] How much of the DIT is cached in RAM ?
> >
> > How can I determine how much of the DIT is being cached in RAM on a
> > given DC ?
> >
> > Dave
> > List info : http://www.activedir.org/List.aspx
> > List FAQ : http://www.activedir.org/ListFAQ.aspx
> > List archive:
> > http://www.mail-archive.com/activedir%40mail.activedir.org/
> >
> >
>
> List info : http://www.activedir.org/List.aspx
> List FAQ : http://www.activedir.org/ListFAQ.aspx
> List archive:
> http://www.mail-archive.com/activedir%40mail.activedir.org/
> List info : http://www.activedir.org/List.aspx
> List FAQ : http://www.activedir.org/ListFAQ.aspx
> List archive:
> http://www.mail-archive.com/activedir%40mail.activedir.org/
> List info : http://www.activedir.org/List.aspx
> List FAQ : http://www.activedir.org/ListFAQ.aspx
> List archive:
> http://www.mail-archive.com/activedir%40mail.activedir.org/
> List info : http://www.activedir.org/List.aspx
> List FAQ : http://www.activedir.org/ListFAQ.aspx
> List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
>
> List info : http://www.activedir.org/List.aspx
> List FAQ : http://www.activedir.org/ListFAQ.aspx
> List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
>
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
