Thanks for the confirmation. Kurt
On Tue, Feb 14, 2012 at 12:13, Brian Desmond <[email protected]> wrote: > Doesn’t look out of the orindary offhand to m. > > Thanks, > Brian Desmond > [email protected] > > w – 312.625.1438 | c – 312.731.3132 > > From: Kurt Buff [mailto:[email protected]] > Sent: Tuesday, February 14, 2012 1:38 PM > To: NT System Admin Issues > Subject: Re: Picking up file server tuning again > > I could be wrong, but poolmon doesn't seem to be showing any issues with > tokens - it's not very high on the list when I do: > 'poolmon -p -b -n ppool.txt %% findstr /iv nonp ppool.txt > out.txt && > notepad out.txt' > > I show the output up to Toke below: > > Tag Type Allocs Frees Diff Bytes Per Alloc > > MmSt Paged 24917295 24883438 33857 127428056 3763 > NtfF Paged 6130096 6116549 13547 12679992 936 > IoNm Paged -1909637662 -1909656779 19117 4802256 251 > UlHT Paged 1 0 1 4198400 4198400 > Ntfo Paged 36021123 36008224 12899 2237648 173 > CM35 Paged 1382 1322 60 2134016 35566 > MmSm Paged 24147039 24118979 28060 1795840 64 > FSim Paged 20383134 20369623 13511 1729408 128 > Gh45 Paged 11602992 11602606 386 1415808 3667 > NtFB Paged 28187 27737 450 1299880 2888 > Gh08 Paged 9051 7900 1151 1285344 1116 > Ttfd Paged 443136 442570 566 1034080 1826 > Toke Paged 50941423 50940719 704 881056 1251 > > > I expect this output should be monitored over time, as this snapshot isn't > definitive - but the full backups for this machine are still in process, and > it's under normal daily load, so it shouldn't vary too much over time. > > Kurt > > On Mon, Feb 13, 2012 at 22:03, Brian Desmond > <[email protected]<mailto:[email protected]>> wrote: >> Yes. Security tokens are stored in Paged Pool. When you get the token bloat >> issue (well if you start approaching it), you will start seeing issues on >> x86 application servers where they are running out of paged pool. If you >> look at a report of paged pool consumers, you'll find the Toke tag at the >> top. >> >> # of spindles is going to directly correlate to disk queue lengths and >> latency. If you have 2 spindles which can do 100 IOPS each, and you are >> throwing 225 IOPS at them, you will have a problem. If you add a third >> spindle, now you have 75 IOPS head room. >> >> Thanks, >> Brian Desmond >> [email protected]<mailto:[email protected]> >> >> w – 312.625.1438 | c – 312.731.3132 >> >> >> -----Original Message----- >> From: Kurt Buff [mailto:[email protected]<mailto:[email protected]>] >> Sent: Monday, February 13, 2012 11:13 PM >> To: NT System Admin Issues >> Subject: Re: Picking up file server tuning again >> >> PSTs on file shares - it's been a while since I looked at that issue. >> >> Crappy drivers are a small possibility - it is a P2V of an old machine. >> >> I'm not sure that the number of spindles has anything to do with it, and in >> any case there isn't anything I can do about that for a while. >> >> Can you explain what you mean by "large tokens"? Is that related to token >> bloat in AD, or is it something else? >> >> Thanks, >> >> Kurt >> >> On Mon, Feb 13, 2012 at 19:25, Brian Desmond >> <[email protected]<mailto:[email protected]>> wrote: >>> Well, the % Interrupts/DPC Time/Kernel Mode CPU time isn't necessarily >>> going to be fixed by x64. It may very well mean you've got some crappy >>> drivers in play. >>> >>> The disk stuff indicates the disk is not fast enough to keep up with >>> demand. You can solve that with more spindles or faster spindles. >>> >>> Page Pool utilization will be resolved by x64 (or even x86 on 2008). That's >>> indicative of crappy drivers, large tokens, and/or people doing things like >>> using PSTs off file shares. >>> >>> Thanks, >>> Brian Desmond >>> [email protected]<mailto:[email protected]> >>> >>> w – 312.625.1438 | c – 312.731.3132 >>> >>> >>> -----Original Message----- >>> From: Michael B. Smith >>> [mailto:[email protected]<mailto:[email protected]>] >>> Sent: Monday, February 13, 2012 6:18 PM >>> To: NT System Admin Issues >>> Subject: RE: Picking up file server tuning again >>> >>> Well, the kernel mode, paged pool, and interrupt time are items that will >>> be specifically reduced with an x64 OS. >>> >>> The I/O situation is indicative of disk queuing which is "hypervisor >>> related". Dunno how you optimize that in VMware, there are a number of >>> potentials in Hyper-V. >>> >>> Regards, >>> >>> Michael B. Smith >>> Consultant and Exchange MVP >>> http://TheEssentialExchange.com >>> >>> >>> -----Original Message----- >>> From: Kurt Buff [mailto:[email protected]<mailto:[email protected]>] >>> Sent: Monday, February 13, 2012 5:33 PM >>> To: NT System Admin Issues >>> Subject: Re: Picking up file server tuning again >>> >>> It *is* a busy box, and migrating the iSCSI LUNs to a 64bit server is >>> something I've definitely considered. I have a Dell R310 with 16gb RAM >>> that I could use, but it's already got 9 active VMs, although they're >>> not heavy hitters. AFAICT, probably the highest-use machines on the >>> ESXi 4.1 box are the secondary DC (no FSMO roles, but does do DNS and >>> WINS) and the issuing CA box. >>> >>> It's currently a VM on what I believe to be an underpowered ESX 3.5 box - I >>> think it's possible that it's simply starved for resources on that ESX box. >>> >>> I'm sure there's something out there like perfmon for VMware that I can use >>> to capture performance over time - I'd like to measure and analyze the >>> performance of the ESX 3.5 box while the backups are happening against the >>> file server. >>> >>> I'm also considering moving the Win2k3 file server VM to the ESX box and >>> seeing if the situation improves. >>> >>> Kurt >>> >>> On Mon, Feb 13, 2012 at 12:08, Michael B. Smith >>> <[email protected]<mailto:[email protected]>> wrote: >>>> That's a busy box. I'd suggest moving to a 64-bit OS. >>>> >>>> Regards, >>>> >>>> Michael B. Smith >>>> Consultant and Exchange MVP >>>> http://TheEssentialExchange.com >>>> >>>> -----Original Message----- >>>> From: Kurt Buff [mailto:[email protected]<mailto:[email protected]>] >>>> Sent: Monday, February 13, 2012 3:00 PM >>>> To: NT System Admin Issues >>>> Subject: Re: Picking up file server tuning again >>>> >>>> Ran PAL against the log. >>>> >>>> Um, wow. It's a freaking christmas tree - red and yellow all over the >>>> place in CPU and disk. >>>> >>>> Who should I be talking with to analyze this? >>>> >>>> A sample of the issues shown - all of which show up in more than one >>>> time slice - some in every or almost every slice: >>>> o- More than 50% Processor Utilization >>>> o- More than 30% privileged (kernel) mode CPU usage >>>> o- More than 2 packets are waiting in the output queue >>>> o- Greater than 25ms physical disk READ response times >>>> o- Greater than 25ms physical disk WRITE response times >>>> o- More than 80% of Pool Paged Kernel Memory Used >>>> o- More than 2 I/O's are waiting on the physical disk >>>> o- 20 (Processor(_Total)\DPC Rate) >>>> o- More than 30% Interrupt Time >>>> o- Greater than 1000 page inputs per second (Memory\Pages Input/sec) >>>> >>>> Some things that showed no alerts: >>>> o- Memory\Available MBytes >>>> o- Memory\Free System Page Table Entrie >>>> o- Memory\Pages/sec >>>> o- Memory\System Cache Resident Bytes >>>> o- Memory\Cache Bytes >>>> o- Memory\% Committed Bytes In Use >>>> o- Network Interface(*)\% Network Utilization >>>> MS TCP Loopback interface >>>> VMware Accelerated AMD PCNet Adapter >>>> VMware Accelerated AMD PCNet Adapter#1 >>>> o- Network Interface(*)\Packets Outbound Errors >>>> MS TCP Loopback interface >>>> VMware Accelerated AMD PCNet Adapter >>>> VMware Accelerated AMD PCNet Adapter#1 >>>> >>>> >>>> Kurt >>>> >>>> On Fri, Feb 10, 2012 at 16:04, Brian Desmond >>>> <[email protected]<mailto:[email protected]>> wrote: >>>>> Rather than trying to do this yourself, check out PAL - >>>>> http://pal.codeplex.com/. It will setup all the right counters for you >>>>> and crunch the data. >>>>> >>>>> Thanks, >>>>> Brian Desmond >>>>> [email protected]<mailto:[email protected]> >>>>> >>>>> w – 312.625.1438 | c – 312.731.3132 >>>>> >>>>> -----Original Message----- >>>>> From: Kurt Buff [mailto:[email protected]<mailto:[email protected]>] >>>>> Sent: Friday, February 10, 2012 4:43 PM >>>>> To: NT System Admin Issues >>>>> Subject: Picking up file server tuning again >>>>> >>>>> I'm getting back to monitoring my situation with the file server again, >>>>> and just finished a perfmon session covering the 3rd through the 7th of >>>>> this month. Simultaneously, I set up perfmon on the same workstation to >>>>> monitor the backup server. >>>>> >>>>> If anyone cares to help, I'd be deeply appreciative. >>>>> >>>>> I set up perfmon on a Win7 VM on an ESXi 4.1 host to take measurements at >>>>> 60 second intervals of a whole bunch of counters, many of them probably >>>>> just noise. >>>>> >>>>> I'll describe the history of the configuration first, however: >>>>> >>>>> The file server is a Win2k3 R2 VM running on a ESX 3.5 host with 16g of >>>>> RAM - it's one of 10 VMs, and is definitely the heaviest hitter in terms >>>>> of disk I/O. About 2.5-3 months ago we noticed that the time to >>>>> completion for the weekly full backups spiked dramatically. >>>>> >>>>> Prior to that time, the fulls would start around 7pm on a Friday, and >>>>> finish by about 7pm on Sunday. >>>>> >>>>> Now they take until Thursday or Friday to complete. >>>>> >>>>> This coincided with some changes to the environment: I had to move >>>>> the VM to a new host (it was a manual copy - we don't have vmotion >>>>> licensed and configured for these hosts) and at about that time I >>>>> also had to expand 2 of the 4 LUNS. Finally, the OS drive for the >>>>> VM on the old host was on a LUN on our Lefthand unit - I had to >>>>> migrate it to the local disk storage on the new home for the VM. The >>>>> 4 data drives for this VM are attached via the MSFT iSCSI client >>>>> running on the VM, not through VMWare's iSCSI client. So, at that >>>>> point, all of the LUNS were on the Lefthand SAN, which is a 3-node >>>>> cluster, and we use 2-way replication for all LUNS. The 2 LUNS that >>>>> were expanded went to 2tb or slightly beyond. The Lefthand has two >>>>> NSM 2060s and a P4300G2, with 6 and 8 disks each, respectively - a >>>>> total of 20 disks >>>>> >>>>> Since that time, I've also added in our EMC VNXe 3100 with 6 disks in it >>>>> in a RAID6 array. I mention this because this means that all of the file >>>>> systems on the VNXe are clean and defragged. >>>>> >>>>> Currently, I've migrated 3 of the 4 data LUNs for the VM to the EMC. I >>>>> made sure to align the partitions on the EMC to a megabyte boundary. >>>>> >>>>> So, to make this simpler to visualize, a little table: >>>>> >>>>> c: - local disk on ESX 3.5, 40gb, 23.6gb free >>>>> j: - iSCSI LUN on Lefthand, 2.5tb, 900gb free >>>>> k: - iSCSI LUN on VNXe, 1.98tb, 336gb free >>>>> l: - iSCSI LUN on VNXe, 1tb, 79gb free >>>>> m: - iSCSI LUN on VNXe 750gb, 425gb free >>>>> >>>>> I tried to capture separate disk queue stats for each LUN, but in spite >>>>> of selecting and adding each drive letter separately in the perfmon >>>>> interface, all I got was _Total. >>>>> >>>>> Selected stats are as follows: >>>>> >>>>> PhysicalDisk counters >>>>> Current disk queue length - average 0.483, maximum 33.000 Average >>>>> disk read queue length - 0.037, maximum 1.294 %disk time - average >>>>> 34.068, maximum 153.877 Average disk write queue length - average >>>>> 0.645, maximum 2.828 Average disk queue length - average 0.681, >>>>> maximum 3.078 >>>>> >>>>> I have more data on PhysicalDisk, and data on other objects, including >>>>> Memory, NetworkInterface, Paging File, Processor and Server Work Queues. >>>>> >>>>> If anyone has thoughts, I'd surely like to hear them. >>>>> >>>>> Thanks, >>>>> >>>>> Kurt >>>>> >>>>> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >>>>> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >>>>> >>>>> --- >>>>> To manage subscriptions click here: >>>>> http://lyris.sunbelt-software.com/read/my_forums/ >>>>> or send an email to >>>>> [email protected]<mailto:[email protected]> >>>>> with the body: unsubscribe ntsysadmin >>>>> >>>>> >>>>> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >>>>> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >>>>> >>>>> --- >>>>> To manage subscriptions click here: >>>>> http://lyris.sunbelt-software.com/read/my_forums/ >>>>> or send an email to >>>>> [email protected]<mailto:[email protected]> >>>>> with the body: unsubscribe ntsysadmin >>>> >>>> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >>>> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >>>> >>>> --- >>>> To manage subscriptions click here: >>>> http://lyris.sunbelt-software.com/read/my_forums/ >>>> or send an email to >>>> [email protected]<mailto:[email protected]> >>>> with the body: unsubscribe ntsysadmin >>>> >>>> >>>> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >>>> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >>>> >>>> --- >>>> To manage subscriptions click here: >>>> http://lyris.sunbelt-software.com/read/my_forums/ >>>> or send an email to >>>> [email protected]<mailto:[email protected]> >>>> with the body: unsubscribe ntsysadmin >>> >>> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >>> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >>> >>> --- >>> To manage subscriptions click here: >>> http://lyris.sunbelt-software.com/read/my_forums/ >>> or send an email to >>> [email protected]<mailto:[email protected]> >>> with the body: unsubscribe ntsysadmin >>> >>> >>> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >>> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >>> >>> --- >>> To manage subscriptions click here: >>> http://lyris.sunbelt-software.com/read/my_forums/ >>> or send an email to >>> [email protected]<mailto:[email protected]> >>> with the body: unsubscribe ntsysadmin >>> >>> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >>> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >>> >>> --- >>> To manage subscriptions click here: >>> http://lyris.sunbelt-software.com/read/my_forums/ >>> or send an email to >>> [email protected]<mailto:[email protected]> >>> with the body: unsubscribe ntsysadmin >> >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >> >> --- >> To manage subscriptions click here: >> http://lyris.sunbelt-software.com/read/my_forums/ >> or send an email to >> [email protected]<mailto:[email protected]> >> with the body: unsubscribe ntsysadmin >> >> >> >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ >> ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >> >> --- >> To manage subscriptions click here: >> http://lyris.sunbelt-software.com/read/my_forums/ >> or send an email to >> [email protected]<mailto:[email protected]> >> with the body: unsubscribe ntsysadmin > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to > [email protected]<mailto:[email protected]> > with the body: unsubscribe ntsysadmin > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to [email protected] > with the body: unsubscribe ntsysadmin > --_000_040311EACB6A5641B59395C8639A36CF1CB75718CH1PRD0106MB172_ > Content-Type: text/html; > charset="utf-8" > Content-Transfer-Encoding: base64 > > <html xmlns:v="urn:schemas-microsoft-com:vml" > xmlns:o="urn:schemas-microsoft-com:office:office" > xmlns:w="urn:schemas-microsoft-com:office:word" > xmlns:m="http://schemas.microsoft.com/office/2004/12/omml"; > xmlns="http://www.w3.org/TR/REC-html40";> > <head> > <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> > <meta name="Generator" content="Microsoft Word 14 (filtered medium)"> > <style><!-- > /* Font Definitions */ > @font-face > {font-family:"Cambria Math"; > panose-1:2 4 5 3 5 4 6 3 2 4;} > @font-face > {font-family:Calibri; > panose-1:2 15 5 2 2 2 4 3 2 4;} > @font-face > {font-family:Tahoma; > panose-1:2 11 6 4 3 5 4 4 2 4;} > /* Style Definitions */ > p.MsoNormal, li.MsoNormal, div.MsoNormal > {margin:0in; > margin-bottom:.0001pt; > font-size:12.0pt; > font-family:"Times New Roman","serif";} > a:link, span.MsoHyperlink > {mso-style-priority:99; > color:blue; > text-decoration:underline;} > a:visited, span.MsoHyperlinkFollowed > {mso-style-priority:99; > color:purple; > text-decoration:underline;} > p > {mso-style-priority:99; > mso-margin-top-alt:auto; > margin-right:0in; > mso-margin-bottom-alt:auto; > margin-left:0in; > font-size:12.0pt; > font-family:"Times New Roman","serif";} > span.EmailStyle18 > {mso-style-type:personal-reply; > font-family:"Calibri","sans-serif"; > color:#002060; > font-weight:bold;} > .MsoChpDefault > {mso-style-type:export-only; > font-family:"Calibri","sans-serif";} > @page WordSection1 > {size:8.5in 11.0in; > margin:1.0in 1.0in 1.0in 1.0in;} > div.WordSection1 > {page:WordSection1;} > --></style><!--[if gte mso 9]><xml> > <o:shapedefaults v:ext="edit" spidmax="1026" /> > </xml><![endif]--><!--[if gte mso 9]><xml> > <o:shapelayout v:ext="edit"> > <o:idmap v:ext="edit" data="1" /> > </o:shapelayout></xml><![endif]--> > </head> > <body lang="EN-US" link="blue" vlink="purple"> > <div class="WordSection1"> > <p class="MsoNormal"><a name="_MailEndCompose"><b><span > style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#002060">Doesn’t > look out of the orindary offhand to m. > <o:p></o:p></span></b></a></p> > <p class="MsoNormal"><b><span > style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#002060"><o:p> </o:p></span></b></p> > <p class="MsoNormal"><b><span > style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#002060">Thanks,<o:p></o:p></span></b></p> > <p class="MsoNormal"><b><span > style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#002060">Brian > Desmond<o:p></o:p></span></b></p> > <p class="MsoNormal"><b><span > style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#002060">[email protected]<o:p></o:p></span></b></p> > <p class="MsoNormal"><b><span > style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#002060"><o:p> </o:p></span></b></p> > <p class="MsoNormal"><b><span > style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#002060">w > – 312.625.1438 | c – 312.731.3132<o:p></o:p></span></b></p> > <p class="MsoNormal"><b><span > style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#002060"><o:p> </o:p></span></b></p> > <p class="MsoNormal"><b><span > style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span > > style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> > Kurt Buff [mailto:[email protected]] > <br> > <b>Sent:</b> Tuesday, February 14, 2012 1:38 PM<br> > <b>To:</b> NT System Admin Issues<br> > <b>Subject:</b> Re: Picking up file server tuning again<o:p></o:p></span></p> > <p class="MsoNormal"><o:p> </o:p></p> > <p class="MsoNormal" style="margin-bottom:12.0pt">I could be wrong, but > poolmon doesn't seem to be showing any issues with tokens - it's not very > high on the list when I do:<br> > 'poolmon -p -b -n ppool.txt %% findstr /iv nonp ppool.txt > > out.txt && notepad out.txt'<br> > <br> > I show the output up to Toke below:<br> > <br> > <span style="font-family:"Courier New"">Tag > Type > Allocs > Frees Diff Bytes Per Alloc<br> > <br> > MmSt Paged 24917295 24883438 33857 > 127428056 > 3763 <br> > NtfF Paged 6130096 > 6116549 13547 > 12679992 > 936 <br> > IoNm Paged -1909637662 -1909656779 19117 > 4802256 > 251 <br> > UlHT Paged > 1 > 0 1 > 4198400 > 4198400 <br> > Ntfo Paged 36021123 36008224 12899 > 2237648 > 173 <br> > CM35 Paged > 1382 > 1322 60 > 2134016 > 35566 <br> > MmSm Paged 24147039 24118979 28060 > 1795840 > 64 <br> > FSim Paged 20383134 20369623 13511 > 1729408 > 128 <br> > Gh45 Paged 11602992 > 11602606 386 > 1415808 > 3667 <br> > NtFB Paged 28187 > 27737 450 > 1299880 > 2888 <br> > Gh08 Paged > 9051 7900 1151 > 1285344 > 1116 <br> > Ttfd Paged 443136 > 442570 566 > 1034080 > 1826 <br> > Toke Paged 50941423 > 50940719 704 > 881056 > 1251 <br> > </span><br> > <br> > I expect this output should be monitored over time, as this snapshot isn't > definitive - but the full backups for this machine are still in process, and > it's under normal daily load, so it shouldn't vary too much over time.<br> > <br> > Kurt<br> > <br> > On Mon, Feb 13, 2012 at 22:03, Brian Desmond <<a > href="mailto:[email protected]";>[email protected]</a>> wrote:<br> > > Yes. Security tokens are stored in Paged Pool. When you get the token > bloat issue (well if you start approaching it), you will start seeing issues > on x86 application servers where they are running out of paged pool. If you > look at a report of paged pool consumers, > you'll find the Toke tag at the top.<br> > ><br> > > # of spindles is going to directly correlate to disk queue lengths and > latency. If you have 2 spindles which can do 100 IOPS each, and you are > throwing 225 IOPS at them, you will have a problem. If you add a third > spindle, now you have 75 IOPS head room.<br> > ><br> > > Thanks,<br> > > Brian Desmond<br> > > <a href="mailto:[email protected]";>[email protected]</a><br> > ><br> > > w – 312.625.1438 | c – 312.731.3132<br> > ><br> > ><br> > > -----Original Message-----<br> > > From: Kurt Buff [mailto:<a > href="mailto:[email protected]";>[email protected]</a>]<br> > > Sent: Monday, February 13, 2012 11:13 PM<br> > > To: NT System Admin Issues<br> > > Subject: Re: Picking up file server tuning again<br> > ><br> > > PSTs on file shares - it's been a while since I looked at that issue.<br> > ><br> > > Crappy drivers are a small possibility - it is a P2V of an old > machine.<br> > ><br> > > I'm not sure that the number of spindles has anything to do with it, and > in any case there isn't anything I can do about that for a while.<br> > ><br> > > Can you explain what you mean by "large tokens"? Is that > related to token bloat in AD, or is it something else?<br> > ><br> > > Thanks,<br> > ><br> > > Kurt<br> > ><br> > > On Mon, Feb 13, 2012 at 19:25, Brian Desmond <<a > href="mailto:[email protected]";>[email protected]</a>> wrote:<br> > >> Well, the % Interrupts/DPC Time/Kernel Mode CPU time isn't > necessarily going to be fixed by x64. It may very well mean you've got some > crappy drivers in play.<br> > >><br> > >> The disk stuff indicates the disk is not fast enough to keep up with > demand. You can solve that with more spindles or faster spindles.<br> > >><br> > >> Page Pool utilization will be resolved by x64 (or even x86 on 2008). > That's indicative of crappy drivers, large tokens, and/or people doing things > like using PSTs off file shares.<br> > >><br> > >> Thanks,<br> > >> Brian Desmond<br> > >> <a > href="mailto:[email protected]";>[email protected]</a><br> > >><br> > >> w – 312.625.1438 | c – 312.731.3132<br> > >><br> > >><br> > >> -----Original Message-----<br> > >> From: Michael B. Smith [mailto:<a > href="mailto:[email protected]";>[email protected]</a>]<br> > >> Sent: Monday, February 13, 2012 6:18 PM<br> > >> To: NT System Admin Issues<br> > >> Subject: RE: Picking up file server tuning again<br> > >><br> > >> Well, the kernel mode, paged pool, and interrupt time are items that > will be specifically reduced with an x64 OS.<br> > >><br> > >> The I/O situation is indicative of disk queuing which is > "hypervisor related". Dunno how you optimize that in VMware, there > are a number of potentials in Hyper-V.<br> > >><br> > >> Regards,<br> > >><br> > >> Michael B. Smith<br> > >> Consultant and Exchange MVP<br> > >> <a > href="http://TheEssentialExchange.com";>http://TheEssentialExchange.com</a><br> > >><br> > >><br> > >> -----Original Message-----<br> > >> From: Kurt Buff [mailto:<a > href="mailto:[email protected]";>[email protected]</a>]<br> > >> Sent: Monday, February 13, 2012 5:33 PM<br> > >> To: NT System Admin Issues<br> > >> Subject: Re: Picking up file server tuning again<br> > >><br> > >> It *is* a busy box, and migrating the iSCSI LUNs to a 64bit server > is<br> > >> something I've definitely considered. I have a Dell R310 with 16gb > RAM<br> > >> that I could use, but it's already got 9 active VMs, although > they're<br> > >> not heavy hitters. AFAICT, probably the highest-use machines on > the<br> > >> ESXi 4.1 box are the secondary DC (no FSMO roles, but does do DNS > and<br> > >> WINS) and the issuing CA box.<br> > >><br> > >> It's currently a VM on what I believe to be an underpowered ESX 3.5 > box - I think it's possible that it's simply starved for resources on that > ESX box.<br> > >><br> > >> I'm sure there's something out there like perfmon for VMware that I > can use to capture performance over time - I'd like to measure and analyze > the performance of the ESX 3.5 box while the backups are happening against > the file server.<br> > >><br> > >> I'm also considering moving the Win2k3 file server VM to the ESX box > and seeing if the situation improves.<br> > >><br> > >> Kurt<br> > >><br> > >> On Mon, Feb 13, 2012 at 12:08, Michael B. Smith <<a > href="mailto:[email protected]";>[email protected]</a>> wrote:<br> > >>> That's a busy box. I'd suggest moving to a 64-bit OS.<br> > >>><br> > >>> Regards,<br> > >>><br> > >>> Michael B. Smith<br> > >>> Consultant and Exchange MVP<br> > >>> <a > href="http://TheEssentialExchange.com";>http://TheEssentialExchange.com</a><br> > >>><br> > >>> -----Original Message-----<br> > >>> From: Kurt Buff [mailto:<a > href="mailto:[email protected]";>[email protected]</a>]<br> > >>> Sent: Monday, February 13, 2012 3:00 PM<br> > >>> To: NT System Admin Issues<br> > >>> Subject: Re: Picking up file server tuning again<br> > >>><br> > >>> Ran PAL against the log.<br> > >>><br> > >>> Um, wow. It's a freaking christmas tree - red and yellow all > over the<br> > >>> place in CPU and disk.<br> > >>><br> > >>> Who should I be talking with to analyze this?<br> > >>><br> > >>> A sample of the issues shown - all of which show up in more than > one<br> > >>> time slice - some in every or almost every slice:<br> > >>> o- More than 50% Processor Utilization<br> > >>> o- More than 30% privileged (kernel) mode CPU usage<br> > >>> o- More than 2 packets are waiting in the output queue<br> > >>> o- Greater than 25ms physical disk READ response times<br> > >>> o- Greater than 25ms physical disk WRITE response times<br> > >>> o- More than 80% of Pool Paged Kernel Memory Used<br> > >>> o- More than 2 I/O's are waiting on the physical disk<br> > >>> o- 20 (Processor(_Total)\DPC Rate)<br> > >>> o- More than 30% Interrupt Time<br> > >>> o- Greater than 1000 page inputs per second (Memory\Pages > Input/sec)<br> > >>><br> > >>> Some things that showed no alerts:<br> > >>> o- Memory\Available MBytes<br> > >>> o- Memory\Free System Page Table Entrie<br> > >>> o- Memory\Pages/sec<br> > >>> o- Memory\System Cache Resident Bytes<br> > >>> o- Memory\Cache Bytes<br> > >>> o- Memory\% Committed Bytes In Use<br> > >>> o- Network Interface(*)\% Network Utilization<br> > >>> MS TCP Loopback interface<br> > >>> VMware Accelerated AMD PCNet Adapter<br> > >>> VMware Accelerated AMD PCNet Adapter#1<br> > >>> o- Network Interface(*)\Packets Outbound Errors<br> > >>> MS TCP Loopback interface<br> > >>> VMware Accelerated AMD PCNet Adapter<br> > >>> VMware Accelerated AMD PCNet Adapter#1<br> > >>><br> > >>><br> > >>> Kurt<br> > >>><br> > >>> On Fri, Feb 10, 2012 at 16:04, Brian Desmond <<a > href="mailto:[email protected]";>[email protected]</a>> wrote:<br> > >>>> Rather than trying to do this yourself, check out PAL - <a > href="http://pal.codeplex.com/";> > http://pal.codeplex.com/</a>. It will setup all the right counters for you > and crunch the data.<br> > >>>><br> > >>>> Thanks,<br> > >>>> Brian Desmond<br> > >>>> <a > href="mailto:[email protected]";>[email protected]</a><br> > >>>><br> > >>>> w – 312.625.1438 | c – 312.731.3132<br> > >>>><br> > >>>> -----Original Message-----<br> > >>>> From: Kurt Buff [mailto:<a > href="mailto:[email protected]";>[email protected]</a>]<br> > >>>> Sent: Friday, February 10, 2012 4:43 PM<br> > >>>> To: NT System Admin Issues<br> > >>>> Subject: Picking up file server tuning again<br> > >>>><br> > >>>> I'm getting back to monitoring my situation with the file > server again, and just finished a perfmon session covering the 3rd through > the 7th of this month. Simultaneously, I set up perfmon on the same > workstation to monitor the backup server.<br> > >>>><br> > >>>> If anyone cares to help, I'd be deeply appreciative.<br> > >>>><br> > >>>> I set up perfmon on a Win7 VM on an ESXi 4.1 host to take > measurements at 60 second intervals of a whole bunch of counters, many of > them probably just noise.<br> > >>>><br> > >>>> I'll describe the history of the configuration first, > however:<br> > >>>><br> > >>>> The file server is a Win2k3 R2 VM running on a ESX 3.5 host > with 16g of RAM - it's one of 10 VMs, and is definitely the heaviest hitter > in terms of disk I/O. About 2.5-3 months ago we noticed that the time to > completion for the weekly full backups spiked > dramatically.<br> > >>>><br> > >>>> Prior to that time, the fulls would start around 7pm on a > Friday, and finish by about 7pm on Sunday.<br> > >>>><br> > >>>> Now they take until Thursday or Friday to complete.<br> > >>>><br> > >>>> This coincided with some changes to the environment: I had > to move<br> > >>>> the VM to a new host (it was a manual copy - we don't have > vmotion<br> > >>>> licensed and configured for these hosts) and at about that > time I<br> > >>>> also had to expand 2 of the 4 LUNS. Finally, the OS > drive for the<br> > >>>> VM on the old host was on a LUN on our Lefthand unit - I had > to<br> > >>>> migrate it to the local disk storage on the new home for the > VM. The<br> > >>>> 4 data drives for this VM are attached via the MSFT iSCSI > client<br> > >>>> running on the VM, not through VMWare's iSCSI client. So, at > that<br> > >>>> point, all of the LUNS were on the Lefthand SAN, which is a > 3-node<br> > >>>> cluster, and we use 2-way replication for all LUNS. The 2 > LUNS that<br> > >>>> were expanded went to 2tb or slightly beyond. The Lefthand > has two<br> > >>>> NSM 2060s and a P4300G2, with 6 and 8 disks each, > respectively - a<br> > >>>> total of 20 disks<br> > >>>><br> > >>>> Since that time, I've also added in our EMC VNXe 3100 with 6 > disks in it in a RAID6 array. I mention this because this means that all of > the file systems on the VNXe are clean and defragged.<br> > >>>><br> > >>>> Currently, I've migrated 3 of the 4 data LUNs for the VM to > the EMC. I made sure to align the partitions on the EMC to a megabyte > boundary.<br> > >>>><br> > >>>> So, to make this simpler to visualize, a little table:<br> > >>>><br> > >>>> c: - local disk on ESX 3.5, 40gb, 23.6gb free<br> > >>>> j: - iSCSI LUN on Lefthand, 2.5tb, 900gb free<br> > >>>> k: - iSCSI LUN on VNXe, 1.98tb, 336gb free<br> > >>>> l: - iSCSI LUN on VNXe, 1tb, 79gb free<br> > >>>> m: - iSCSI LUN on VNXe 750gb, 425gb free<br> > >>>><br> > >>>> I tried to capture separate disk queue stats for each LUN, > but in spite of selecting and adding each drive letter separately in the > perfmon interface, all I got was _Total.<br> > >>>><br> > >>>> Selected stats are as follows:<br> > >>>><br> > >>>> PhysicalDisk counters<br> > >>>> Current disk queue length - average 0.483, maximum 33.000 > Average<br> > >>>> disk read queue length - 0.037, maximum 1.294 %disk time - > average<br> > >>>> 34.068, maximum 153.877 Average disk write queue length - > average<br> > >>>> 0.645, maximum 2.828 Average disk queue length - average > 0.681,<br> > >>>> maximum 3.078<br> > >>>><br> > >>>> I have more data on PhysicalDisk, and data on other objects, > including Memory, NetworkInterface, Paging File, Processor and Server > Work Queues.<br> > >>>><br> > >>>> If anyone has thoughts, I'd surely like to hear them.<br> > >>>><br> > >>>> Thanks,<br> > >>>><br> > >>>> Kurt<br> > >>>><br> > >>>> ~ Finally, powerful endpoint security that ISN'T a resource > hog! ~ ~<br> > >>>> <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>> > ~<br> > >>>><br> > >>>> ---<br> > >>>> To manage subscriptions click here:<br> > >>>> <a > href="http://lyris.sunbelt-software.com/read/my_forums/";>http://lyris.sunbelt-software.com/read/my_forums/</a><br> > >>>> or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br> > >>>> with the body: unsubscribe ntsysadmin<br> > >>>><br> > >>>><br> > >>>> ~ Finally, powerful endpoint security that ISN'T a resource > hog! ~ ~<br> > >>>> <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>> > ~<br> > >>>><br> > >>>> ---<br> > >>>> To manage subscriptions click here:<br> > >>>> <a > href="http://lyris.sunbelt-software.com/read/my_forums/";>http://lyris.sunbelt-software.com/read/my_forums/</a><br> > >>>> or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br> > >>>> with the body: unsubscribe ntsysadmin<br> > >>><br> > >>> ~ Finally, powerful endpoint security that ISN'T a resource hog! > ~ ~<br> > >>> <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>> > ~<br> > >>><br> > >>> ---<br> > >>> To manage subscriptions click here:<br> > >>> <a > href="http://lyris.sunbelt-software.com/read/my_forums/";>http://lyris.sunbelt-software.com/read/my_forums/</a><br> > >>> or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br> > >>> with the body: unsubscribe ntsysadmin<br> > >>><br> > >>><br> > >>> ~ Finally, powerful endpoint security that ISN'T a resource hog! > ~ ~<br> > >>> <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>> > ~<br> > >>><br> > >>> ---<br> > >>> To manage subscriptions click here:<br> > >>> <a > href="http://lyris.sunbelt-software.com/read/my_forums/";>http://lyris.sunbelt-software.com/read/my_forums/</a><br> > >>> or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br> > >>> with the body: unsubscribe ntsysadmin<br> > >><br> > >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~<br> > >> <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>> > ~<br> > >><br> > >> ---<br> > >> To manage subscriptions click here:<br> > >> <a > href="http://lyris.sunbelt-software.com/read/my_forums/";>http://lyris.sunbelt-software.com/read/my_forums/</a><br> > >> or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br> > >> with the body: unsubscribe ntsysadmin<br> > >><br> > >><br> > >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~<br> > >> <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>> > ~<br> > >><br> > >> ---<br> > >> To manage subscriptions click here:<br> > >> <a > href="http://lyris.sunbelt-software.com/read/my_forums/";>http://lyris.sunbelt-software.com/read/my_forums/</a><br> > >> or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br> > >> with the body: unsubscribe ntsysadmin<br> > >><br> > >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~<br> > >> <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>> > ~<br> > >><br> > >> ---<br> > >> To manage subscriptions click here:<br> > >> <a > href="http://lyris.sunbelt-software.com/read/my_forums/";>http://lyris.sunbelt-software.com/read/my_forums/</a><br> > >> or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br> > >> with the body: unsubscribe ntsysadmin<br> > ><br> > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>> > ~<br> > ><br> > > ---<br> > > To manage subscriptions click here: <a > href="http://lyris.sunbelt-software.com/read/my_forums/";> > http://lyris.sunbelt-software.com/read/my_forums/</a><br> > > or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br> > > with the body: unsubscribe ntsysadmin<br> > ><br> > ><br> > ><br> > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~<br> > > ~ <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>> > ~<br> > ><br> > > ---<br> > > To manage subscriptions click here: <a > href="http://lyris.sunbelt-software.com/read/my_forums/";> > http://lyris.sunbelt-software.com/read/my_forums/</a><br> > > or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br> > > with the body: unsubscribe ntsysadmin<o:p></o:p></p> > <p>~ Finally, powerful endpoint security that ISN'T a resource hog! ~<br> > ~ <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>> > ~<br> > <br> > ---<br> > To manage subscriptions click here: <a > href="http://lyris.sunbelt-software.com/read/my_forums/";> > http://lyris.sunbelt-software.com/read/my_forums/</a><br> > or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br> > with the body: unsubscribe ntsysadmin<o:p></o:p></p> > </div> > > <p>~ Finally, powerful endpoint security that ISN'T a resource hog! ~<br /> > ~ <<a > href="http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/";>http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/</a>>  > ~<br /> > <br /> > ---<br /> > To manage subscriptions click here: <a > href="http://lyris.sunbelt-software.com/read/my_forums/";>http://lyris.sunbelt-software.com/read/my_forums/</a><br > /> > or send an email to <a > href="mailto:[email protected]";>[email protected]</a><br > /> > with the body: unsubscribe ntsysadmin</p> > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ <BR> ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ <BR> <BR> --- <BR> To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ <BR> or send an email to [email protected] <BR> with the body: unsubscribe ntsysadmin </BODY> > </html> > > --
