Keith,
When I looked at this several years ago, this is what I came up with:
Windows allows a total of 48 MB in the heap, and each service
started process uses the third setting in the chain, or 512 KB by
default, and there is about 10 MB that gets used for other
things. Based on what Scott Perry wrote concerning this in a
obscure page on the Declude site about Declude Queue, there can
only be a total of 77 service started processes before having
issues, and you can assume that there will be one for Declude up
to my limit of 40, and also often times another process whether it
is a virus scanner or external filter application in JunkMail.
Windows apparently starts to barf when the limit is reached, and
applications can go into a bad state, only partially launching and
becoming corrupted. This has a high association with load, but
the true association seems to be the number of processes, which
typically correspond to load but not necessarily. This is
probably also what has caused McAfee to barf on occasion on my
server with similar errors. McAfee has a decent amount of latency
compared to most other things that Declude launches except of
course for Eradispam due to the timeout issues.
There are two camps on what to do with the mystery heap, aka
desktop heap. Some have indicated on the IMail and Declude lists
in the past that setting it to 2048 would resolve some issues with
IMail's SMTP and also the 16 bit version of F-Prot when run from
Declude which is awfully slow and CPU intensive. That change
however would reduce the number of service started processes that
were possible by a factor of 4. Scott suggests that reducing it
to maybe 256 would help in high traffic servers, though this is a
limit that you wouldn't want to pass because it could cause
instability.
FYI, the error messages will contribute to heap usage, so these must
be cleared, and when you have a bunch of these, it will limit what you
can run, and in fact make the problem worse.
If you are using Declude as a service, that certainly takes one
process off the top that used to count towards the heap, but it's
likely what is running concurrently that is causing the issue along
with error dialogs. Weightgate certainly adds to this issue, as well
as other plugins and virus scanners. The best solution for a high
volume server that wants to do weight skipping would be for either
Sniffer or Declude to skip based on both a high and a low weight
within the config. I have been asking for over three years for this
and have even recently documented a solution for Declude that would be
backwards compatible with current configs should they opt to do this.
Here's a quote from the old Declude site authored by Scott:
*Flaw #1 - Server crashing: Microsoft's Mystery Heap*
Fortunately, not many people experience this problem. However, it
is listed first because it is more serious than the other flaw.
This one can back up mail for hours/days, and crash the server.
The problem here is that each process that is started by a service
uses a certain (unknown) amount of an undocumented type of memory
that Windows allocates. Without knowing how much of the mystery
heap is used, or how much is left, or how much is available when
the system starts, it's impossible to know when you will run out.
When you DO run out, Windows does a *terrible* job in handling it.
Instead of preventing the program from loading and recording an
error to the event log, Windows will keep the program half-loaded
(the error almost always occurs while loading .DLLs) and pop up an
error message saying that it can't start the program.
When this happens, unless you happen to be at the server, you
won't have a chance to close the box. So, another one will soon
pop up as another SMTP process is started. By the time you find
out, there could be hundreds or thousands of the pop-up boxes.
Since Microsoft doesn't clear them automatically, when the
original 30 SMTP processes end, there still isn't enough of this
mystery heap left, because Microsoft is using it to display these
error messages. So until you click "OK" to all the hundreds of
pop-up boxes, or you reboot the computer, new mail will not be
delivered. Eventually, the server may crash completely.
*Additional Help*
Most versions of Windows NT/2000 will apparently by default
allocate 512KB of the "Mystery Heap" to each service-started
process. There is also apparently a total of 48MB of the "Mystery
Heap" available. That means you can have a maximum of about 77
service-started processes (48Megs minus (3Meg * 3 default
desktops) minus (1Meg system-wide) divided by 512). Changing it to
256KB should approximately double the amount of service-started
processes that can run before the mystery heap is depleted.
However, some people have reported better results by raising the
value to 2048KB -- that's one of the problems with undocumented
resources (there's no way to know for sure which value is better
or why).
We recommend going to
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q142676
and changing the registry entry to use a value of "256" or "2048"
(NOTE: Microsoft recommends 512 in that article; if you use 512,
make sure not to have IMail's MaxQueProc registry entry set to
more than 30).
Matt
Keith Johnson wrote:
Darrell,
Did you alter your heap size 3rd entry? If so, did you go to 1024 or other. I
found this article by crossing a Declude page, appears to be what I need to go
after.
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q142676
-Keith
_____
From: Message Sniffer Community on behalf of Darrell ([EMAIL PROTECTED])
Sent: Sun 6/10/2007 2:31 PM
To: Message Sniffer Community
Subject: [sniffer] Re: Error Messages since WeightGate
After looking into it I am on board with what Pete said about the "heap"
issue. It makes sense to me that its the heap issue since were
launching weight gate -> SNF. Effectively doubling the amount of
processes being launched.
Darrell
-------------------------------------------
Check out http://www.invariantsystems.com for utilities for Declude,
Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring,
SURBL/URI integration, MRTG Integration, and Log Parsers.
Keith Johnson wrote:
Darrell,
You are right, a reboot will take care of it for a season, then it comes back
out of the blue. Very strange indeed.
Keith
_____
From: Message Sniffer Community on behalf of Darrell ([EMAIL PROTECTED])
Sent: Sat 6/9/2007 9:36 PM
To: Message Sniffer Community
Subject: [sniffer] Re: Error Messages since WeightGate
Keith,
I was having the same problems last week. Just came out of the blue and
was across several of our servers as well. Same error verbatim. FWIW -
I also use weightgate. I rebooted the servers I was seeing this issue
on and the problem has not returned.
Very odd you mentioned that as I thought this was isolated to just me.
Darrell
-------------------------------------------
Check out http://www.invariantsystems.com for utilities for Declude,
Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring,
SURBL/URI integration, MRTG Integration, and Log Parsers.
Keith Johnson wrote:
It appears since installing WeightGate we have been receiving a lot of the
below Application PopUps indicating an error:
The application failed to initialize properly 0xc0000142. Click on OK to
terminate the application
The application entry is our Sniffer .exe. Today alone I saw over 300. I
thought it was an isolated issue. However, it is happening across all our
servers. We are running the latest Sniffer in Persistent mode. We never saw
these prior to WeightGate. Has anyone seen this before? Below is the actual
entry in Event Log.
-Keith
Event Type: Information
Event Source: Application Popup
Event Category: None
Event ID: 26
Date: 6/9/2007
Time: 12:12:35 AM
User: N/A
Computer: NAIMAIL2
Description:
Application popup: rrctp2ez.exe - Application Error : The application failed to
initialize properly (0xc0000142). Click on OK to terminate the application.
#############################################################
This message is sent to you because you are subscribed to
the mailing list <sniffer@sortmonster.com>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
--
-------------------------------------------
Check out http://www.invariantsystems.com for utilities for Declude, Imail,
mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI
integration, MRTG Integration, and Log Parsers.
#############################################################
This message is sent to you because you are subscribed to
the mailing list <sniffer@sortmonster.com>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
#############################################################
This message is sent to you because you are subscribed to
the mailing list <sniffer@sortmonster.com>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
#############################################################
This message is sent to you because you are subscribed to
the mailing list <sniffer@sortmonster.com>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
#############################################################
This message is sent to you because you are subscribed to
the mailing list <sniffer@sortmonster.com>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>