Just replied offline to you directly - let me know if you get it ? Thanks, Brian Desmond [email protected]
c - 312.731.3132 -----Original Message----- From: Kurt Buff [mailto:[email protected]] Sent: Tuesday, March 31, 2009 4:35 PM To: NT System Admin Issues Subject: Re: Pool consumption (was: Re: Virtualized server issue...) BTW - I think I've found the culprit, and will be testing this evening. I have a job that runs every night to check permissions, using fileacl.exe. Normally its report is a few kilobytes. Today's was over 10 megabytes - this happens occasionally when a certain department of ours needs to rearrange the directories allocated to it, and signals that thousands of files have been moved. I investigated, and found that this person was indeed performing this task around the time of the server having difficulties, both yesterday and today. This leads me to believe that it's indeed VIPRE and its Active Protection settings - I've disabled that for now, and will be testing it this evening by doing a mass copy from one directory to another on the file server, both with and without the Active Protection setting turned on. This should help pinpoint the issue.. Kurt On Tue, Mar 31, 2009 at 13:58, Brian Desmond <[email protected]> wrote: > Kurt- > > > > Is pooltag.txt in that archive I sent you offline? > > > > If not, install the Debug Tools for Windows (free download) and go in the > triage folder�������� pooltag.txt has a dictionary so to speak of all the MS > tags > > > > ����� �� - nt!����� ��������������������� - General security allocations > > > > Allocs is��������t really terribly interesting, what is is the number of bytes > allocated which in this case is ~232MB. That is not normal. I have no idea > what this one is ������� i��������s not somethi�������������ve seen before > an�������������s not > normal. I would be inclined to tell you to call PSS and let them work on it. > I know how to get the stacks for the allocations with a live debug but > that�������s fairly involved as things go. > > > > MmSt is associated with memory tracking shared files (in a nutshel������������ > looks ok in your snapshot. > > > > Thanks, > > Brian Desmond > > [email protected] > > > > c - 312.731.3132 > > > > Active Directory, 4th Ed - http://www.briandesmond.com/ad4/ > > Microsoft MVP - https://mvp.support.microsoft.com/profile/Brian > > > > From: Kurt Buff [mailto:[email protected]] > Sent: Tuesday, March 31, 2009 1:46 PM > To: NT System Admin Issues > Subject: Pool consumption (was: Re: Virtualized server issue...) > > > > Have added the reg entry, and am examining poolmon output. > > The server needed to be rebooted again today, so the reg entry should now be > active. I note that in perfmon Pool Paged Allocs for this machine had > slammed to the ceiling, although nothing else seemed to be > > Poolmon is proving problematic - I've examined the top two memory eaters, > and am either getting too many or no hits. > > At 10:12 this morning, I took this reading, and a few minutes later had to > reboot the box. > > Memory: 1047456K Avail: 220432K PageFlts: 47 InRam Krnl: 2912K > P:281220K > Commit: 570560K Limit:3053000K Peak: 580612K Pool N:35804K > P:282140K > System pool information > Tag Type Allocs Frees Diff Bytes Per > Alloc > > Se Paged 13914251 ( 0) 1770996 ( 0) 12143255 243309424 ( 0) > 20 > MmSt Paged 755490 ( 9) 753741 ( 9) 1749 15544008 ( 0) > 8887 > > Memory: 1047456K Avail: 229772K PageFlts: 66 InRam Krnl: 2912K > P:281000K > Commit: 570396K Limit:3053000K Peak: 580612K Pool N:35776K > P:281920K > System pool information > Tag Type Allocs Frees Diff Bytes Per > Alloc > > MmCm Nonp 1090 ( 0) 15 ( 0) 1075 7864864 ( 0) > 7316 > LSwi Nonp 1 ( 0) 0 ( 0) 1 2576384 ( 0) > 2576384 > > As the winner and new champeen, Se is at least an order of magnitude larger > than its closes competitor. However, the Se tag is in lots of files, so I > prefixed it in the findstr command with 'h' per the KB article. That > narrowed it considerably, but what's left pretty much looks to be MSFT > files, and there are still 21 files. > > MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in > srv.sys - I note that the Se tag is also found in the latter. > > Later today, perhaps after standard business hours, I'll reenable Active > Protection for VIPRE on this machine, and copy a lot of files to it. If > that's the issue, it should start spiking pretty quickly. > > On Mon, Mar 30, 2009 at 15:29, Brian Desmond <[email protected]> wrote: >> Kurt- >> >> Can you add the http://support.microsoft.com/kb/244139 CrashOnCtrlScroll >> registry value and reboot? This will allow you to generate a dump next time >> this happens (the hang, specifically) by pressing the /right/ Ctrl key and >> Scroll Lock twice. >> >> Also, Poolmon can help tremendously here too for logging. >> >> Thanks, >> Brian Desmond >> [email protected] >> >> c - 312.731.3132 >> >> >> -----Original Message----- >> From: Kurt Buff [mailto:[email protected]] >> Sent: Monday, March 30, 2009 5:00 PM >> To: NT System Admin Issues >> Subject: Virtualized server issue... >> >> All, >> >> Over the weekend we virtualized our file/print server, and it seemed >> to go well. Host is a Dell machine running ESX 3.5 update 2. >> >> The physical machine has an Intel HT processor and 1gbyte of RAM. I >> gave the VM 2 procs and 2gbytes of RAM, just for good measure. >> >> Both machines were talking to our LeftHand SAN, on a separate physical >> LAN, but today I had to reboot the VM, then a couple of hours later >> shut it down and revert to the physical machine after it stopped >> responding. >> >> The logs were indicating lack of server memory - specifically, these >> were being emitted to my syslog server: >> >> 2009-03-30 14:05:12 User.Notice home-01 Mar 30 14:05:12 >> home-01 MSWinEventLog 1 System 13892 Mon Mar 30 14:05:08 2009 >> 2020 Srv Unknown User N/A Error HOME-01 None >> 0000: 00 00 04 00 01 00 54 00 ....... 0008: 00 00 00 00 e4 07 00 c0 >> ........ 0010: 00 00 00 00 9a 00 00 c0 ........ 0018: 00 00 00 >> 00 00 00 00 00 ........ 0020: 00 00 00 00 00 00 00 00 ........ >> 0028: ae 04 00 00 d0 02 70 00 ....... The server was unable to >> allocate from the system paged pool because the pool was empty. >> >> Then this, as I tried to log in to shut it down: >> >> 2009-03-30 14:09:39 User.Notice zet-home-01 Mar 30 >> 14:09:39 zet-home-01 MSWinEventLog 1 Application 13935 Mon >> Mar 30 14:09:39 2009 1512 Userenv SYSTEM User >> Error ZET-HOME-01 None Windows cannot unload your >> registry file. The memory used by the registry has not been freed. >> This is often caused by services running as a user account, try >> configuring the services to run in either the LocalService or >> NetworkService account. If this problem persists, contact your >> administrator. DETAIL - Insufficient system resources exist to >> complete the requested service. 30 >> >> >> and couldn't log in - I had to use psshutdown to make it go. >> >> I was starting to troubleshoot the paged pool issue, but didn't get >> far enough into it before it required kick, and I reverted to the >> physical box. >> >> Anyone have any ideas what might have been the problem, or where I can >> start to look for clues? >> >> >> Kurt >> >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ >> ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >> >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ >> ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > > > > > > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~
