Late to the game (and I didn't see anything to indicate solved), but have you checked your NIC speeds? Reason I ask is that my backups suddenly went from 4 hours to 12 in a day. It drove me absolutely mental (there was a lot of developer work going on and they had made some *innocent changes*) trying to track down what happened. Turns out...for whatever reason, the NIC has spontaneously changed it's port speed from gig/full to 100/full
Just a thought... On Tue, Feb 14, 2012 at 11:04 PM, Crawford, Scott <[email protected]>wrote: > Works great for us > > > Sent from my Windows Phone > ------------------------------ > From: Kurt Buff > Sent: 2/14/2012 7:48 PM > > To: NT System Admin Issues > Subject: Re: Picking up file server tuning again > > I will be looking at that, for sure. > > On Tue, Feb 14, 2012 at 16:26, Crawford, Scott <[email protected]> > wrote: > > Or the "disable PST" GPO. > > > > Sent from my Windows Phone > > ________________________________ > > From: Kurt Buff > > Sent: 2/14/2012 5:04 PM > > > > To: NT System Admin Issues > > Subject: Re: Picking up file server tuning again > > > > I've been beating on my users for years not to use PST files - they > > don't listen. > > > > I think it's time for another round of admonitions... > > > > Kurt > > > > On Tue, Feb 14, 2012 at 12:18, Paul Hutchings <[email protected] > > > > wrote: > >> If you're going to open PST files off a server, you at least want a x64 > >> version of Windows. > >> > >> One of my biggest frustrations with the FSRM on Windows 2008 is that you > >> can't specify that a particular type of file can be stored on a sever, > but > >> not opened from the server. > >> ________________________________ > >> From: Kurt Buff [[email protected]] > >> Sent: 14 February 2012 7:54 PM > > > >> > >> To: NT System Admin Issues > >> Subject: Re: Picking up file server tuning again > >> > >> RE: PST files. > >> > >> This might well be part of the issue: > >> > >> #net file | findstr /i pst > >> 21310605 J:\Home\...\Archive PSTs\archive.pst USER1 0 > >> 21310606 J:\Home\USER1\Archive PSTs\xxxxx.PST USER1 0 > >> 21359101 J:\Home\...\xxxx Folders.pst USER2 2 > >> 21375086 J:\Home\...\Outlook\xxxxxxxx.pst USER3 1 > >> 21375089 J:\Home\...\Outlook\~xxxxxxxx.pst.tmp USER3 0 > >> 21375091 J:\Home\...\Outlook\Jokes.pst USER3 1 > >> 21375094 J:\Home\...\Outlook\~Jokes.pst.tmp USER3 0 > >> 21386255 J:\Home\USER4\Private\USER4.pst USER4 1 > >> > >> Kurt > >> > >> On Mon, Feb 13, 2012 at 19:25, Brian Desmond <[email protected]> > >> wrote: > >>> > >>> Well, the % Interrupts/DPC Time/Kernel Mode CPU time isn't necessarily > >>> going to be fixed by x64. It may very well mean you've got some crappy > >>> drivers in play. > >>> > >>> The disk stuff indicates the disk is not fast enough to keep up with > >>> demand. You can solve that with more spindles or faster spindles. > >>> > >>> Page Pool utilization will be resolved by x64 (or even x86 on 2008). > >>> That's indicative of crappy drivers, large tokens, and/or people doing > >>> things like using PSTs off file shares. > >>> > >>> Thanks, > >>> Brian Desmond > >>> [email protected] > >>> > >>> w – 312.625.1438 | c – 312.731.3132 > >>> > >>> > >>> -----Original Message----- > >>> From: Michael B. Smith > >>> [mailto:[email protected]<[email protected]> > ] > >>> Sent: Monday, February 13, 2012 6:18 PM > >>> To: NT System Admin Issues > >>> Subject: RE: Picking up file server tuning again > >>> > >>> Well, the kernel mode, paged pool, and interrupt time are items that > will > >>> be specifically reduced with an x64 OS. > >>> > >>> The I/O situation is indicative of disk queuing which is "hypervisor > >>> related". Dunno how you optimize that in VMware, there are a number of > >>> potentials in Hyper-V. > >>> > >>> Regards, > >>> > >>> Michael B. Smith > >>> Consultant and Exchange MVP > >>> http://TheEssentialExchange.com <http://theessentialexchange.com/> > >>> > >>> > >>> -----Original Message----- > >>> From: Kurt Buff [mailto:[email protected] <[email protected]>] > >>> Sent: Monday, February 13, 2012 5:33 PM > >>> To: NT System Admin Issues > >>> Subject: Re: Picking up file server tuning again > >>> > >>> It *is* a busy box, and migrating the iSCSI LUNs to a 64bit server is > >>> something I've definitely considered. I have a Dell R310 with 16gb RAM > >>> that > >>> I could use, but it's already got 9 active VMs, although they're not > >>> heavy > >>> hitters. AFAICT, probably the highest-use machines on the ESXi 4.1 box > >>> are > >>> the secondary DC (no FSMO roles, but does do DNS and > >>> WINS) and the issuing CA box. > >>> > >>> It's currently a VM on what I believe to be an underpowered ESX 3.5 > box - > >>> I think it's possible that it's simply starved for resources on that > ESX > >>> box. > >>> > >>> I'm sure there's something out there like perfmon for VMware that I can > >>> use to capture performance over time - I'd like to measure and analyze > >>> the > >>> performance of the ESX 3.5 box while the backups are happening against > >>> the > >>> file server. > >>> > >>> I'm also considering moving the Win2k3 file server VM to the ESX box > and > >>> seeing if the situation improves. > >>> > >>> Kurt > >>> > >>> On Mon, Feb 13, 2012 at 12:08, Michael B. Smith <[email protected] > > > >>> wrote: > >>> > That's a busy box. I'd suggest moving to a 64-bit OS. > >>> > > >>> > Regards, > >>> > > >>> > Michael B. Smith > >>> > Consultant and Exchange MVP > >>> > http://TheEssentialExchange.com <http://theessentialexchange.com/> > >>> > > >>> > -----Original Message----- > >>> > From: Kurt Buff [mailto:[email protected] <[email protected]>] > >>> > Sent: Monday, February 13, 2012 3:00 PM > >>> > To: NT System Admin Issues > >>> > Subject: Re: Picking up file server tuning again > >>> > > >>> > Ran PAL against the log. > >>> > > >>> > Um, wow. It's a freaking christmas tree - red and yellow all over the > >>> > place in CPU and disk. > >>> > > >>> > Who should I be talking with to analyze this? > >>> > > >>> > A sample of the issues shown - all of which show up in more than one > >>> > time slice - some in every or almost every slice: > >>> > o- More than 50% Processor Utilization > >>> > o- More than 30% privileged (kernel) mode CPU usage > >>> > o- More than 2 packets are waiting in the output queue > >>> > o- Greater than 25ms physical disk READ response times > >>> > o- Greater than 25ms physical disk WRITE response times > >>> > o- More than 80% of Pool Paged Kernel Memory Used > >>> > o- More than 2 I/O's are waiting on the physical disk > >>> > o- 20 (Processor(_Total)\DPC Rate) > >>> > o- More than 30% Interrupt Time > >>> > o- Greater than 1000 page inputs per second (Memory\Pages Input/sec) > >>> > > >>> > Some things that showed no alerts: > >>> > o- Memory\Available MBytes > >>> > o- Memory\Free System Page Table Entrie > >>> > o- Memory\Pages/sec > >>> > o- Memory\System Cache Resident Bytes > >>> > o- Memory\Cache Bytes > >>> > o- Memory\% Committed Bytes In Use > >>> > o- Network Interface(*)\% Network Utilization > >>> > MS TCP Loopback interface > >>> > VMware Accelerated AMD PCNet Adapter > >>> > VMware Accelerated AMD PCNet Adapter#1 > >>> > o- Network Interface(*)\Packets Outbound Errors > >>> > MS TCP Loopback interface > >>> > VMware Accelerated AMD PCNet Adapter > >>> > VMware Accelerated AMD PCNet Adapter#1 > >>> > > >>> > > >>> > Kurt > >>> > > >>> > On Fri, Feb 10, 2012 at 16:04, Brian Desmond <[email protected] > > > >>> > wrote: > >>> >> Rather than trying to do this yourself, check out PAL - > >>> >> http://pal.codeplex.com/. It will setup all the right counters for > you > >>> >> and > >>> >> crunch the data. > >>> >> > >>> >> Thanks, > >>> >> Brian Desmond > >>> >> [email protected] > >>> >> > >>> >> w – 312.625.1438 | c – 312.731.3132 > >>> >> > >>> >> -----Original Message----- > >>> >> From: Kurt Buff [mailto:[email protected] <[email protected]>] > >>> >> Sent: Friday, February 10, 2012 4:43 PM > >>> >> To: NT System Admin Issues > >>> >> Subject: Picking up file server tuning again > >>> >> > >>> >> I'm getting back to monitoring my situation with the file server > >>> >> again, > >>> >> and just finished a perfmon session covering the 3rd through the 7th > >>> >> of this > >>> >> month. Simultaneously, I set up perfmon on the same workstation to > >>> >> monitor > >>> >> the backup server. > >>> >> > >>> >> If anyone cares to help, I'd be deeply appreciative. > >>> >> > >>> >> I set up perfmon on a Win7 VM on an ESXi 4.1 host to take > measurements > >>> >> at 60 second intervals of a whole bunch of counters, many of them > >>> >> probably > >>> >> just noise. > >>> >> > >>> >> I'll describe the history of the configuration first, however: > >>> >> > >>> >> The file server is a Win2k3 R2 VM running on a ESX 3.5 host with 16g > >>> >> of > >>> >> RAM - it's one of 10 VMs, and is definitely the heaviest hitter in > >>> >> terms of > >>> >> disk I/O. About 2.5-3 months ago we noticed that the time to > >>> >> completion for > >>> >> the weekly full backups spiked dramatically. > >>> >> > >>> >> Prior to that time, the fulls would start around 7pm on a Friday, > and > >>> >> finish by about 7pm on Sunday. > >>> >> > >>> >> Now they take until Thursday or Friday to complete. > >>> >> > >>> >> This coincided with some changes to the environment: I had to move > >>> >> the VM to a new host (it was a manual copy - we don't have vmotion > >>> >> licensed and configured for these hosts) and at about that time I > >>> >> also had to expand 2 of the 4 LUNS. Finally, the OS drive for the > VM > >>> >> on the old host was on a LUN on our Lefthand unit - I had to migrate > >>> >> it to the local disk storage on the new home for the VM. The 4 data > >>> >> drives for this VM are attached via the MSFT iSCSI client running on > >>> >> the VM, not through VMWare's iSCSI client. So, at that point, all of > >>> >> the LUNS were on the Lefthand SAN, which is a 3-node cluster, and we > >>> >> use 2-way replication for all LUNS. The 2 LUNS that were expanded > >>> >> went to 2tb or slightly beyond. The Lefthand has two NSM 2060s and a > >>> >> P4300G2, with 6 and 8 disks each, respectively - a total of 20 disks > >>> >> > >>> >> Since that time, I've also added in our EMC VNXe 3100 with 6 disks > in > >>> >> it in a RAID6 array. I mention this because this means that all of > the > >>> >> file > >>> >> systems on the VNXe are clean and defragged. > >>> >> > >>> >> Currently, I've migrated 3 of the 4 data LUNs for the VM to the > EMC. I > >>> >> made sure to align the partitions on the EMC to a megabyte boundary. > >>> >> > >>> >> So, to make this simpler to visualize, a little table: > >>> >> > >>> >> c: - local disk on ESX 3.5, 40gb, 23.6gb free > >>> >> j: - iSCSI LUN on Lefthand, 2.5tb, 900gb free > >>> >> k: - iSCSI LUN on VNXe, 1.98tb, 336gb free > >>> >> l: - iSCSI LUN on VNXe, 1tb, 79gb free > >>> >> m: - iSCSI LUN on VNXe 750gb, 425gb free > >>> >> > >>> >> I tried to capture separate disk queue stats for each LUN, but in > >>> >> spite > >>> >> of selecting and adding each drive letter separately in the perfmon > >>> >> interface, all I got was _Total. > >>> >> > >>> >> Selected stats are as follows: > >>> >> > >>> >> PhysicalDisk counters > >>> >> Current disk queue length - average 0.483, maximum 33.000 Average > >>> >> disk read queue length - 0.037, maximum 1.294 %disk time - average > >>> >> 34.068, maximum 153.877 Average disk write queue length - average > >>> >> 0.645, maximum 2.828 Average disk queue length - average 0.681, > >>> >> maximum 3.078 > >>> >> > >>> >> I have more data on PhysicalDisk, and data on other objects, > including > >>> >> Memory, NetworkInterface, Paging File, Processor and Server Work > >>> >> Queues. > >>> >> > >>> >> If anyone has thoughts, I'd surely like to hear them. > >>> >> > >>> >> Thanks, > >>> >> > >>> >> Kurt > >>> >> > >>> >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > >>> >> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > >>> >> > >>> >> --- > >>> >> To manage subscriptions click here: > >>> >> http://lyris.sunbelt-software.com/read/my_forums/ > >>> >> or send an email to [email protected] > >>> >> with the body: unsubscribe ntsysadmin > >>> >> > >>> >> > >>> >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > >>> >> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > >>> >> > >>> >> --- > >>> >> To manage subscriptions click here: > >>> >> http://lyris.sunbelt-software.com/read/my_forums/ > >>> >> or send an email to [email protected] > >>> >> with the body: unsubscribe ntsysadmin > >>> > > >>> > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > >>> > <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > >>> > > >>> > --- > >>> > To manage subscriptions click here: > >>> > http://lyris.sunbelt-software.com/read/my_forums/ > >>> > or send an email to [email protected] > >>> > with the body: unsubscribe ntsysadmin > >>> > > >>> > > >>> > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > >>> > <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > >>> > > >>> > --- > >>> > To manage subscriptions click here: > >>> > http://lyris.sunbelt-software.com/read/my_forums/ > >>> > or send an email to [email protected] > >>> > with the body: unsubscribe ntsysadmin > >>> > >>> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > >>> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > >>> > >>> --- > >>> To manage subscriptions click here: > >>> http://lyris.sunbelt-software.com/read/my_forums/ > >>> or send an email to [email protected] > >>> with the body: unsubscribe ntsysadmin > >>> > >>> > >>> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > >>> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > >>> > >>> --- > >>> To manage subscriptions click here: > >>> http://lyris.sunbelt-software.com/read/my_forums/ > >>> or send an email to [email protected] > >>> with the body: unsubscribe ntsysadmin > >>> > >>> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > >>> ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > >>> > >>> --- > >>> To manage subscriptions click here: > >>> http://lyris.sunbelt-software.com/read/my_forums/ > >>> or send an email to [email protected] > >>> with the body: unsubscribe ntsysadmin > >> > >> > >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > >> ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > >> > >> --- > >> To manage subscriptions click here: > >> http://lyris.sunbelt-software.com/read/my_forums/ > >> or send an email to [email protected] > >> with the body: unsubscribe ntsysadmin > >> > >> ________________________________ > >> MIRA Ltd > >> > >> Watling Street, Nuneaton, Warwickshire, CV10 0TU, England > >> Registered in England and Wales No. 402570 > >> VAT Registration GB 100 1464 84 > >> > >> The contents of this e-mail are confidential and are solely for the use > of > >> the intended recipient. If you receive this e-mail in error, please > >> delete > >> it and notify us either by e-mail, telephone or fax. You should not > copy, > >> forward or otherwise disclose the content of the e-mail as this is > >> prohibited. > >> > >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > >> > >> > >> ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > >> > >> --- > >> To manage subscriptions click here: > >> http://lyris.sunbelt-software.com/read/my_forums/ > >> or send an email to [email protected] > >> with the body: unsubscribe ntsysadmin > > > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > > > --- > > To manage subscriptions click here: > > http://lyris.sunbelt-software.com/read/my_forums/ > > or send an email to [email protected] > > with the body: unsubscribe ntsysadmin > > > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > > > --- > > To manage subscriptions click here: > > http://lyris.sunbelt-software.com/read/my_forums/ > > or send an email to [email protected] > > with the body: unsubscribe ntsysadmin > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to [email protected] > with the body: unsubscribe ntsysadmin > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to [email protected] > with the body: unsubscribe ntsysadmin > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected] with the body: unsubscribe ntsysadmin
