Keith,
When I looked at this several years ago, this is what I came up with:
Windows allows a total of 48 MB in the heap, and each service
started process uses the third setting in the chain, or 512 KB by
default, and there is about 10 MB that gets used for other things.
Based on what Scott Perry wrote concerning this in a obscure page on
the Declude site about Declude Queue, there can only be a total of
77 service started processes before having issues, and you can
assume that there will be one for Declude up to my limit of 40, and
also often times another process whether it is a virus scanner or
external filter application in JunkMail.
Windows apparently starts to barf when the limit is reached, and
applications can go into a bad state, only partially launching and
becoming corrupted. This has a high association with load, but the
true association seems to be the number of processes, which
typically correspond to load but not necessarily. This is probably
also what has caused McAfee to barf on occasion on my server with
similar errors. McAfee has a decent amount of latency compared to
most other things that Declude launches except of course for
Eradispam due to the timeout issues.
There are two camps on what to do with the mystery heap, aka desktop
heap. Some have indicated on the IMail and Declude lists in the
past that setting it to 2048 would resolve some issues with IMail's
SMTP and also the 16 bit version of F-Prot when run from Declude
which is awfully slow and CPU intensive. That change however would
reduce the number of service started processes that were possible by
a factor of 4. Scott suggests that reducing it to maybe 256 would
help in high traffic servers, though this is a limit that you
wouldn't want to pass because it could cause instability.
FYI, the error messages will contribute to heap usage, so these must be
cleared, and when you have a bunch of these, it will limit what you can
run, and in fact make the problem worse.
If you are using Declude as a service, that certainly takes one process
off the top that used to count towards the heap, but it's likely what is
running concurrently that is causing the issue along with error
dialogs. Weightgate certainly adds to this issue, as well as other
plugins and virus scanners. The best solution for a high volume server
that wants to do weight skipping would be for either Sniffer or Declude
to skip based on both a high and a low weight within the config. I have
been asking for over three years for this and have even recently
documented a solution for Declude that would be backwards compatible
with current configs should they opt to do this.
Here's a quote from the old Declude site authored by Scott:
*Flaw #1 - Server crashing: Microsoft's Mystery Heap*
Fortunately, not many people experience this problem. However, it is
listed first because it is more serious than the other flaw. This
one can back up mail for hours/days, and crash the server.
The problem here is that each process that is started by a service
uses a certain (unknown) amount of an undocumented type of memory
that Windows allocates. Without knowing how much of the mystery heap
is used, or how much is left, or how much is available when the
system starts, it's impossible to know when you will run out.
When you DO run out, Windows does a *terrible* job in handling it.
Instead of preventing the program from loading and recording an
error to the event log, Windows will keep the program half-loaded
(the error almost always occurs while loading .DLLs) and pop up an
error message saying that it can't start the program.
When this happens, unless you happen to be at the server, you won't
have a chance to close the box. So, another one will soon pop up as
another SMTP process is started. By the time you find out, there
could be hundreds or thousands of the pop-up boxes. Since Microsoft
doesn't clear them automatically, when the original 30 SMTP
processes end, there still isn't enough of this mystery heap left,
because Microsoft is using it to display these error messages. So
until you click "OK" to all the hundreds of pop-up boxes, or you
reboot the computer, new mail will not be delivered. Eventually, the
server may crash completely.
*Additional Help*
Most versions of Windows NT/2000 will apparently by default allocate
512KB of the "Mystery Heap" to each service-started process. There
is also apparently a total of 48MB of the "Mystery Heap" available.
That means you can have a maximum of about 77 service-started
processes (48Megs minus (3Meg * 3 default desktops) minus (1Meg
system-wide) divided by 512). Changing it to 256KB should
approximately double the amount of service-started processes that
can run before the mystery heap is depleted. However, some people
have reported better results by raising the value to 2048KB --
that's one of the problems with undocumented resources (there's no
way to know for sure which value is better or why).
We recommend going to
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q142676 and
changing the registry entry to use a value of "256" or "2048" (NOTE:
Microsoft recommends 512 in that article; if you use 512, make sure
not to have IMail's MaxQueProc registry entry set to more than 30).
Matt
Keith Johnson wrote:
Darrell,
Did you alter your heap size 3rd entry? If so, did you go to 1024 or other. I
found this article by crossing a Declude page, appears to be what I need to go
after.
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q142676
-Keith
_____
From: Message Sniffer Community on behalf of Darrell ([EMAIL PROTECTED])
Sent: Sun 6/10/2007 2:31 PM
To: Message Sniffer Community
Subject: [sniffer] Re: Error Messages since WeightGate
After looking into it I am on board with what Pete said about the "heap"
issue. It makes sense to me that its the heap issue since were
launching weight gate -> SNF. Effectively doubling the amount of
processes being launched.
Darrell
-------------------------------------------
Check out http://www.invariantsystems.com for utilities for Declude,
Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring,
SURBL/URI integration, MRTG Integration, and Log Parsers.
Keith Johnson wrote:
Darrell,
You are right, a reboot will take care of it for a season, then it comes back
out of the blue. Very strange indeed.
Keith
_____
From: Message Sniffer Community on behalf of Darrell ([EMAIL PROTECTED])
Sent: Sat 6/9/2007 9:36 PM
To: Message Sniffer Community
Subject: [sniffer] Re: Error Messages since WeightGate
Keith,
I was having the same problems last week. Just came out of the blue and
was across several of our servers as well. Same error verbatim. FWIW -
I also use weightgate. I rebooted the servers I was seeing this issue
on and the problem has not returned.
Very odd you mentioned that as I thought this was isolated to just me.
Darrell
-------------------------------------------
Check out http://www.invariantsystems.com for utilities for Declude,
Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring,
SURBL/URI integration, MRTG Integration, and Log Parsers.
Keith Johnson wrote:
It appears since installing WeightGate we have been receiving a lot of the
below Application PopUps indicating an error:
The application failed to initialize properly 0xc0000142. Click on OK to
terminate the application
The application entry is our Sniffer .exe. Today alone I saw over 300. I
thought it was an isolated issue. However, it is happening across all our
servers. We are running the latest Sniffer in Persistent mode. We never saw
these prior to WeightGate. Has anyone seen this before? Below is the actual
entry in Event Log.
-Keith
Event Type: Information
Event Source: Application Popup
Event Category: None
Event ID: 26
Date: 6/9/2007
Time: 12:12:35 AM
User: N/A
Computer: NAIMAIL2
Description:
Application popup: rrctp2ez.exe - Application Error : The application failed to
initialize properly (0xc0000142). Click on OK to terminate the application.
#############################################################
This message is sent to you because you are subscribed to
the mailing list <sniffer@sortmonster.com>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
--
-------------------------------------------
Check out http://www.invariantsystems.com for utilities for Declude, Imail,
mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI
integration, MRTG Integration, and Log Parsers.
#############################################################
This message is sent to you because you are subscribed to
the mailing list <sniffer@sortmonster.com>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
#############################################################
This message is sent to you because you are subscribed to
the mailing list <sniffer@sortmonster.com>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
#############################################################
This message is sent to you because you are subscribed to
the mailing list <sniffer@sortmonster.com>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
#############################################################
This message is sent to you because you are subscribed to
the mailing list <sniffer@sortmonster.com>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
