https://issues.apache.org/bugzilla/show_bug.cgi?id=55708
Bug ID: 55708
Summary: Access violation in msvcrt.dll when calling
getenv("PATH") under load running on Windows Server
Product: Apache httpd-2
Version: 2.2.22
Hardware: PC
Status: NEW
Severity: normal
Priority: P2
Component: Core
Assignee: [email protected]
Reporter: [email protected]
Created attachment 30972
--> https://issues.apache.org/bugzilla/attachment.cgi?id=30972&action=edit
Screen shot of Visual Studio debugging the crash dump
Environment:
-Windows Server 2003 R2 SP2
-Running in a VMWare VM with 4 Intel Xeon CPUs @2.00Ghz with 3.8 GB of RAM
-See attached manifest.txt and appcompat.txt from one of the crash dumps for a
listing of the applications and libraries running at the time of the crash.
Error as seen in the Windows Application Event Viewer:
Faulting application httpd.exe, version 2.2.22.0, faulting module msvcrt.dll,
version 7.0.3790.3959, fault address 0x00027d70.
Problem description and analysis:
Under load, we are experiencing access violation errors in msvcrt.dll running
under the Apache httpd.exe process. Analysis of two heap dumps that have been
collected from two different servers running the same application shows that
the access violation happens in calls to the getenv() function on line 184 of
util_script.c. These are rare occurrences in the order of one in many
thousands of calls. In both the heap dumps that we have collected, the call is
exactly the same.
getenv(“PATH”).
Attached are two screen shots of Visual Studio debugging each of the crash
dumps we have analyzed. The line of code highlighted is the last line of httpd
code before it falls into the msvcrt library. This is where the call to
getenv(“PATH”) is made.
The access violation happens two calls deeper inside msvcrt.dll in
__getenv_lk() as seen in the call stack in the bottom right of the screen
shots.
What we see leads us to believe that there must be some kind of race condition
or something that is somehow not thread safe in that function that results in
an access violation in some rare cases. We have reproduced the error in a test
environment. It happened once in a 1.5 hour test at a sustained rate of close
to 1000 hits per second on the Apache httpd server. It has also happened in
our production environment at a lower load of around 100 hits per second.
Under that load, it can run for several days before we see an occurrence of it.
In both cases, the request were quite different from one another. They were
both requests that Apache passes through to back end servers via mod_jk. In
one case the backend server is a Tomcat 7 instance and in the other case it is
an entirely other application running in JBoss 4.3. And both those requests
are successful most of the time, so it doesn't look like the crash is specific
to any particular type of request. But I wouldn't rule out that mod_jk is
somehow involved since it does appear in the call stack of the crash dump.
Some thoughts:
Is the getenv() function being called supposed to be thread safe?
Should the code calling it be locking around it if it isn't thread safe?
I can provide the crash dump files for analysis privately.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
