Here is a problem I have, let me know of any suggestions and or other people I can send this question to.
I have a module that is 15,000 lines of c code.
It uses global apache tables and structures and it uses apache pools. Sometimes the apache tables are holding several thousand key-value pairs. A single value may contain a 5 or 10Kbyte or larger page and or template.
Initially the modules start out as 3 to 5 MB size children and grow to about 15 or 20 MB over hours or days. Then suddenly during some particular request it explodes. Takes all of physical ram and virtual ram. The server then requires a reboot.
When this is running on redhat with apache 1.3.18-1.3.29 a couple of times a week it will runaway. There will be a request and it will take all of ram and bring down the server.
Every .html or .htm page is parsed by the module. The module extends the functionality of the html language and adds 10 or 20 more tags that allow data base access and unique functionality. Some tags allow 'post' or 'get' to run arbitrary cgi scripts or just post to other pages in the same web site. It does session accounting and keeps track of thousands of key-value pairs across an arbitrary number of pages within a web site.
Have tried to debug with electric fence and other approaches.
It has been tested also on FreeBSD and Mac-OSX and Windows.
Question is as follows:
a) Is this just a memory leak? b) Is this because of excessive apache pool reuse? c) Is this because of over use of global structures which use the stack and not the heap? d) Does anyone else have any ideas or suggestions.
Also,
The module has been rewritten and is being redeveloped and tested in windows XP under apache 1.3.29 using the MS .NET development environment. It is now only 8,000 lines of code which is better and more manageable.
We are no longer using apache pools but using malloc, realloc, calloc and free. We are no longer using apache tables and have written our own hash tables. The hash tables are global and may contain hundreds to thousands of key-value pairs. A given value may be 5Kbyte 10Kbyte or more page and or template. It can post or get to itself and or cgi-scripts and or php on the same server.
In windows apache does not fork() it uses multi threaded environment. This shares the globals.
Its cool we can attach to the running process and step through live thread of the module and set breakpoints.
The module parses any page that uses a .htm or .html extension.
Problem now is that when we load a frameset page the module breaks, segfaults. It appears that it is reusing the space and running over itself.
Question is: a) If we go back to using apache pools will this alleviate the problem of running over the same memory location? b) Is this because we are using global structures and tables as in the previous module? c) If we go back to using apache tables will this relieve the problem of running over the same memory location? d) Do you have any other ideas or thoughts or do you know who I could contact to discuss this with?
Thanks
-Mark-
