Status Update, It appears adjusting that 3rd sharedsection entry to 2048 did the trick. I did however discover that the root cause of the issue was large blast of targeted email from a realtor company list serve to one of our users was producing this behavior. When I was logged in, you could see where I would have a steady 45 to 50 processes running, then it would climb to 500 in a matter of seconds. With the new entry, memory would go from 400MB on average and spiked at 1.2GB. Within min. it would drop back to 400MB. I checked the logs and found the email blast. In doing discovery, I found Microsoft has released a tool to monitor the heap. Loaded it, but haven't tried it out, however, here it is:
http://www.microsoft.com/downloads/details.aspx?familyid=5cfc9b74-97aa-4510-b4b9-b2dc98c8ed8b&displaylang=en Keith _____ From: Message Sniffer Community on behalf of Matt Sent: Sun 6/10/2007 6:29 PM To: Message Sniffer Community Subject: [sniffer] Re: Error Messages since WeightGate Here's a better page from someone at Microsoft all about the desktop heap. This one suggests that you can change the limit from 48 MB to a value as much as 450 MB. You will probably normally not need more than the total number of processes that Declude can use times the amount of memory allocated per session, so if you have 512 MB/session, and have 100 processes defined in Declude, you would need about 50 MB, but adding something like weightgate to an app that has latency could very well increase the needs even more. http://blogs.msdn.com/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx Matt Matt wrote: Keith, When I looked at this several years ago, this is what I came up with: Windows allows a total of 48 MB in the heap, and each service started process uses the third setting in the chain, or 512 KB by default, and there is about 10 MB that gets used for other things. Based on what Scott Perry wrote concerning this in a obscure page on the Declude site about Declude Queue, there can only be a total of 77 service started processes before having issues, and you can assume that there will be one for Declude up to my limit of 40, and also often times another process whether it is a virus scanner or external filter application in JunkMail. Windows apparently starts to barf when the limit is reached, and applications can go into a bad state, only partially launching and becoming corrupted. This has a high association with load, but the true association seems to be the number of processes, which typically correspond to load but not necessarily. This is probably also what has caused McAfee to barf on occasion on my server with similar errors. McAfee has a decent amount of latency compared to most other things that Declude launches except of course for Eradispam due to the timeout issues. There are two camps on what to do with the mystery heap, aka desktop heap. Some have indicated on the IMail and Declude lists in the past that setting it to 2048 would resolve some issues with IMail's SMTP and also the 16 bit version of F-Prot when run from Declude which is awfully slow and CPU intensive. That change however would reduce the number of service started processes that were possible by a factor of 4. Scott suggests that reducing it to maybe 256 would help in high traffic servers, though this is a limit that you wouldn't want to pass because it could cause instability. FYI, the error messages will contribute to heap usage, so these must be cleared, and when you have a bunch of these, it will limit what you can run, and in fact make the problem worse. If you are using Declude as a service, that certainly takes one process off the top that used to count towards the heap, but it's likely what is running concurrently that is causing the issue along with error dialogs. Weightgate certainly adds to this issue, as well as other plugins and virus scanners. The best solution for a high volume server that wants to do weight skipping would be for either Sniffer or Declude to skip based on both a high and a low weight within the config. I have been asking for over three years for this and have even recently documented a solution for Declude that would be backwards compatible with current configs should they opt to do this. Here's a quote from the old Declude site authored by Scott: Flaw #1 - Server crashing: Microsoft's Mystery Heap Fortunately, not many people experience this problem. However, it is listed first because it is more serious than the other flaw. This one can back up mail for hours/days, and crash the server. The problem here is that each process that is started by a service uses a certain (unknown) amount of an undocumented type of memory that Windows allocates. Without knowing how much of the mystery heap is used, or how much is left, or how much is available when the system starts, it's impossible to know when you will run out. When you DO run out, Windows does a *terrible* job in handling it. Instead of preventing the program from loading and recording an error to the event log, Windows will keep the program half-loaded (the error almost always occurs while loading .DLLs) and pop up an error message saying that it can't start the program. When this happens, unless you happen to be at the server, you won't have a chance to close the box. So, another one will soon pop up as another SMTP process is started. By the time you find out, there could be hundreds or thousands of the pop-up boxes. Since Microsoft doesn't clear them automatically, when the original 30 SMTP processes end, there still isn't enough of this mystery heap left, because Microsoft is using it to display these error messages. So until you click "OK" to all the hundreds of pop-up boxes, or you reboot the computer, new mail will not be delivered. Eventually, the server may crash completely. Additional Help Most versions of Windows NT/2000 will apparently by default allocate 512KB of the "Mystery Heap" to each service-started process. There is also apparently a total of 48MB of the "Mystery Heap" available. That means you can have a maximum of about 77 service-started processes (48Megs minus (3Meg * 3 default desktops) minus (1Meg system-wide) divided by 512). Changing it to 256KB should approximately double the amount of service-started processes that can run before the mystery heap is depleted. However, some people have reported better results by raising the value to 2048KB -- that's one of the problems with undocumented resources (there's no way to know for sure which value is better or why). We recommend going to http://support.microsoft.com/default.aspx?scid=kb;EN-US;q142676 and changing the registry entry to use a value of "256" or "2048" (NOTE: Microsoft recommends 512 in that article; if you use 512, make sure not to have IMail's MaxQueProc registry entry set to more than 30). Matt Keith Johnson wrote: Darrell, Did you alter your heap size 3rd entry? If so, did you go to 1024 or other. I found this article by crossing a Declude page, appears to be what I need to go after. http://support.microsoft.com/default.aspx?scid=kb;EN-US;q142676 -Keith _____ From: Message Sniffer Community on behalf of Darrell ([EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>) Sent: Sun 6/10/2007 2:31 PM To: Message Sniffer Community Subject: [sniffer] Re: Error Messages since WeightGate After looking into it I am on board with what Pete said about the "heap" issue. It makes sense to me that its the heap issue since were launching weight gate -> SNF. Effectively doubling the amount of processes being launched. Darrell ------------------------------------------- Check out http://www.invariantsystems.com for utilities for Declude, Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. Keith Johnson wrote: Darrell, You are right, a reboot will take care of it for a season, then it comes back out of the blue. Very strange indeed. Keith _____ From: Message Sniffer Community on behalf of Darrell ([EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>) Sent: Sat 6/9/2007 9:36 PM To: Message Sniffer Community Subject: [sniffer] Re: Error Messages since WeightGate Keith, I was having the same problems last week. Just came out of the blue and was across several of our servers as well. Same error verbatim. FWIW - I also use weightgate. I rebooted the servers I was seeing this issue on and the problem has not returned. Very odd you mentioned that as I thought this was isolated to just me. Darrell ------------------------------------------- Check out http://www.invariantsystems.com for utilities for Declude, Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. Keith Johnson wrote: It appears since installing WeightGate we have been receiving a lot of the below Application PopUps indicating an error: The application failed to initialize properly 0xc0000142. Click on OK to terminate the application The application entry is our Sniffer .exe. Today alone I saw over 300. I thought it was an isolated issue. However, it is happening across all our servers. We are running the latest Sniffer in Persistent mode. We never saw these prior to WeightGate. Has anyone seen this before? Below is the actual entry in Event Log. -Keith Event Type: Information Event Source: Application Popup Event Category: None Event ID: 26 Date: 6/9/2007 Time: 12:12:35 AM User: N/A Computer: NAIMAIL2 Description: Application popup: rrctp2ez.exe - Application Error : The application failed to initialize properly (0xc0000142). Click on OK to terminate the application. ############################################################# This message is sent to you because you are subscribed to the mailing list <[email protected]><mailto:[email protected]>. To unsubscribe, E-mail to: <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> -- ------------------------------------------- Check out http://www.invariantsystems.com for utilities for Declude, Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. ############################################################# This message is sent to you because you are subscribed to the mailing list <[email protected]><mailto:[email protected]>. To unsubscribe, E-mail to: <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> ############################################################# This message is sent to you because you are subscribed to the mailing list <[email protected]><mailto:[email protected]>. To unsubscribe, E-mail to: <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> ############################################################# This message is sent to you because you are subscribed to the mailing list <[email protected]><mailto:[email protected]>. To unsubscribe, E-mail to: <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> ############################################################# This message is sent to you because you are subscribed to the mailing list <[email protected]><mailto:[email protected]>. To unsubscribe, E-mail to: <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]> ############################################################# This message is sent to you because you are subscribed to the mailing list <[email protected]>. To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
