Eric Fleischman
Mon, 06 Dec 2004 12:50:41 -0800
Tremendous amt of churn on this thread. Let me see if I can pull it all together. One of the things we do internally on the ESE level is caching of pages of the DIT from disk. The perf benefit is clear, and measurable. In 2003 on 32bit hardware, the /3gb switch begins to make sense when your dit is in the neighborhood of 2gb and you have >2GB of physical memory. At that point we might hit the max cache size, and to grow beyond that /3gb will help. Max cache size is in the neighborhood of 2.6 or 2.7gb when /3gb is used. On 64bit, our max cache size is 2^48bytes if memory serves me correctly. If you have that much ram on a 64bit box, call me. I want to see your box. :) I should note that /3gb does not come w/o a cost. I would be careful in using this setting this value on machines which are not just DCs, as it does have a perf impact on your system more generally. Without going too far off topic, I'll say it will yield a scenario where you have fewer resources for kernel data structures, like non-paged pool and system PTEs. If you are interested in the details, this is a question best fielded by a book like Inside Windows 2000 I'd think. There was discussion around the amt of benefit (I think someone tossed out a phrase like "a factor of 5"). The reality is that the benefit depends greatly upon your workload. If you have a workload which can be optimized through server-side indexes, to accurately measure the benefit of 64bit you probably want to compare a 32bit box with heavy indexes, custom tailored to your environment, vs. 64bit with either comparable or no indexes (your choice) and a _warm_ cache. I say it in this way as really, you want to compare max perf you can get on 32bit with max you can get on 64bit. That might mean enabling some indexes, as that can help with perf even w/o loading everything in memory (probably intuitive, but wanted to draw special attention to it). Note my usage of the word "warm" to describe the cache. I say warm cache as out of the box, we won't preload your DIT in to memory, even if you have the physical memory for it (32bit or 64bit). Rather, we cache things as they are fetched. So if you issue a query which need traverse a series of pages not yet cached, we still take the same I/O hit. It is when they are in memory and you try to use them a second time that you get the benefit, as we don't need to fetch them again. This yields the fact that some customers that run 64bit write a little script to "walk" their database. They do this to warm the cache and get most everything preloaded in to memory. There was also discussion around how large the cache is. In essence, like most software which caches stuff, we have algorithms for it. :) Joe eluded to it, but basically we have a series of elements we look at to help decide what movements in cache size should be done. I won't go in to the details of such things for the sake of brevity. Hope that helps. ~Eric -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of joe Sent: Monday, December 06, 2004 1:24 PM To: [EMAIL PROTECTED] Subject: RE: [ActiveDir] Stress testing and performance analysis of domain controllers Brett is fun. :o) He isn't so much the type that will fish for you or even teach you to fish, he will throw you a fishing line and let you figure it all out. This can be troublesome though if you lived in a desert and had never even seen water let alone a fish. Understanding Tech involved with the AD code specifically, he is very strong. Understanding Tech as it has to be used in some sites and locations and operational support concerns, not always so strong. He is very much like the rest of the Dev team guys which is mostly working in the Dev this is what you should shoot for world versus the ditches and puddles that many people end up having to work in. Overall a good guy though. Extremely entertaining guy to talk to. On the beefy DC side of things. I don't know, I don't really consider 2GB to be exceptionally beefy or even 4GB. Not when we have workstations now coming from the factory with 1GB and 2GB and options to do 4GB. Exceeding 4GB RAM gets a little unusual and you truly get beefy based on proc Architectures (say multiproc opteron versus athlon or on the intel side the Xeon versus the non-Xeon's, etc) and disk subsystems with heavy duty hardware RAID solutions with oodles of cache and RAID type offerings. We need to go to 64 bit for no better reason than the cost of memory is consistently dropping and we need good easy ways of dealing with more than 4GB of RAM that doesn't depend on goofy paging mechanisms. Finally, I don't recommend /3gb unless you truly need it and all of the software on the machine properly supports it. It has been long while (years) but I have seen some odd /3GB failures with apps that didn't properly implement that functionality due to memory addressing issues. Also obviously you can't use /3GB with 2K standard, that could cause some fun things to happen as well, collectively termed as undefined results. No reason to force the kernel to live in 1GB unless it is required for some other reason which if I recall can impact some video drivers and other kernel apps that may need to grab a chunk of address space for some reason. joe -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Renouf, Phil Sent: Monday, December 06, 2004 1:59 PM To: [EMAIL PROTECTED] Subject: RE: [ActiveDir] Stress testing and performance analysis of domain controllers > > Gotcha, then yeah the /3gb switch would help with performance. > > I've learned something new, thanks :) > > Maybe. It depends on the DIT size as well as what else needs memory. > From what I understand based on old conversations, the DIT caching > routines are sensitive to memory pressure and will not page DIT cache, > it will release memory instead. > Again if you have a DIT of 200MB, you can use /3gb and most likely > wouldn't see a benefit. You might not see a benefit with a small DIT size, but then again why go with such a beefed up DC if your DIT size is that small (unless you are planning for it to grow substantially). Adding the /3GB switch shouldn't cause any issues even if the DIT is small enough to not get much benefit from it, unless the OS is effected by being reduced to 1GB of virtual address space. > Hopefully ~Eric will pop along shortly with some info as I know he > loves this stuff. In the meanwhile, you can be pretty sure BrettSh > generally knows what he is talking about with AD. Not saying he can't > be wrong, but all things being equal concerning a bet on AD internals, > I would bet with Brett. > Unless he was betting against Will, Dmitri, ~Eric, Dean or some of > those guys and then I would simply put my wallet away, pull out some > popcorn, and watch the show. I'm definitely interested to see what they have to say :) I certainly wasn't implying Brett didn't know what he was talking about, but showing me the size of a DIT really didn't tell me much without the information that LSASS is large address aware. Now it makes sense ;) Anyway, looking forward to some more information on this and its effect on performance. Phil List info : http://www.activedir.org/mail_list.htm List FAQ : http://www.activedir.org/list_faq.htm List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/mail_list.htm List FAQ : http://www.activedir.org/list_faq.htm List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/mail_list.htm List FAQ : http://www.activedir.org/list_faq.htm List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/