Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Filip Sielimowicz]

> The reason is that httpd memory usage patterns significantly differ from
> your simple example. E.g., there might be some memory block allocated at
> the end of the heap, which is not freed at the end of the request - and it
> will prevent all the rest of memory from being returned to the system.

Don't blame "httpd memory usage patterns" ! Of course they differ from
my simple example, but...

I've changed "a bit" files Zend/zend_alloc.c and ...h files.
I've just replaced the definitions of _emalloc, _efree... functions
with simple #defines, linking them back to standard malloc, free... and
so on functions.
The compilation - went OK (I suppose, debug can not be set).
The apache restart - OK.
Testing php -  -  OK !

And the memory holding problem disapeared ! Memory is beeing returned
to system as the script finishes.

So...

It is not a good solution, of course, because Zend memory optimaliser is
turned off. But now we know, where should be put some afford to fix
the problem. The optimaliser propably leaves only some memory blocks
unfreed, but they make it unposible to return rest of memory back to
system. 

Am I not right ? May be not...

I send the changed files, but I suggest not to use them...
For example replacement of function _persist_alloc with value 1
is very ... hmm... controversial ? It should return the size of
block (as I suppose), but it was difficult to do that by simple means.



/*
   +--+
   | Zend Engine  |
   +--+
   | Copyright (c) 1998-2000 Zend Technologies Ltd. (http://www.zend.com) |
   +--+
   | This source file is subject to version 0.92 of the Zend license, |
   | that is bundled with this package in the file LICENSE, and is| 
   | available at through the world-wide-web at   |
   | http://www.zend.com/license/0_92.txt.|
   | If you did not receive a copy of the Zend license and are unable to  |
   | obtain it through the world-wide-web, please send a note to  |
   | [EMAIL PROTECTED] so we can mail you a copy immediately.  |
   +--+
   | Authors: Andi Gutmans <[EMAIL PROTECTED]>|
   |  Zeev Suraski <[EMAIL PROTECTED]>|
   +--+
*/


#include 

#include "zend.h"
#include "zend_alloc.h"
#include "zend_globals.h"
#include "zend_fast_cache.h"
#ifdef HAVE_SIGNAL_H
# include 
#endif
#ifdef HAVE_UNISTD_H
# include 
#endif

#ifndef ZTS
ZEND_API zend_alloc_globals alloc_globals;
#endif


#define ZEND_DISABLE_MEMORY_CACHE 0

#if ZEND_DEBUG
# define END_MAGIC_SIZE sizeof(long)
# define END_ALIGNMENT(size) 
(((size)%PLATFORM_ALIGNMENT)?(PLATFORM_ALIGNMENT-((size)%PLATFORM_ALIGNMENT)):0)
#else
# define END_MAGIC_SIZE 0
# define END_ALIGNMENT(size) 0
#endif


# if MEMORY_LIMIT
#  if ZEND_DEBUG
#define CHECK_MEMORY_LIMIT(s, rs) _CHECK_MEMORY_LIMIT(s, rs, __zend_filename, 
__zend_lineno)
#  else
#define CHECK_MEMORY_LIMIT(s, rs)   _CHECK_MEMORY_LIMIT(s, rs, NULL,0)
#  endif

#define _CHECK_MEMORY_LIMIT(s, rs, file, lineno) { AG(allocated_memory) += rs;\
if 
(AG(memory_limit)persistent && p==AG(head)) {\
AG(head) = p->pNext;\
} else if (p->persistent && p==AG(phead)) { \
AG(phead) = p->pNext;   \
} else {   
 \
p->pLast->pNext = p->pNext; \
}  
 \
if (p->pNext) { \
p->pNext->pLast = p->pLast; \
}

#define ADD_POINTER_TO_LIST(p)  \
if (p->persistent) {\
p->pNext = AG(phead);   \
if (AG(phead)) {\
AG(phead)->pLast = p;   \
}   \
AG(phead) = p;  \
} else {\
p->pNext = AG(head);\
if (AG(head)) { \
AG(head)->pLast = p;\
}

Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Stanislav Malyshev]

FS>> And here is the basic difference between 'mall' and 'httpd'
FS>> processes: the last are on the top ALL THE TIME.

The reason is that httpd memory usage patterns significantly differ from
your simple example. E.g., there might be some memory block allocated at
the end of the heap, which is not freed at the end of the request - and it
will prevent all the rest of memory from being returned to the system.

-- 
Stanislav Malyshev, Zend Products Engineer
[EMAIL PROTECTED]  http://www.zend.com/ +972-3-6139665 ext.115



-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Filip Sielimowicz]

The answer is really not so easy.
I have RedHat Linux 6.2 (with standard kernel, I suppose 2.1).
I tried such program in c++:

//File mall.cc

#include 
#include 
#include 
#include 

int main() {
void* v[1000];
for (int i=0;i<1000;i++) {
v[i]=malloc(10);
//printf("%i. %p; ",i,v[i]);
memset(v[i],i,1);
}
printf("\n\nMemory allocated. Wait 20 s.\n");
sleep(20);
for (int i=0;i<1000;i++) {
free(v[i]);
//printf("%i. ^; ",i,v[i]);
}
printf("\nMemory freed. Wait 20 s.\n");
sleep(20);

}

Then I examine it with 'top' command (then shift-M to sort by memory
usage).

In first 20 seconds process 'mall' is on the top with 13M, then
in next 20 s it is NOT.

And here is the basic difference between 'mall' and 'httpd' processes: the
last are on the top ALL THE TIME.

So ?


-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Stanislav Malyshev]

SV>> How does Zend allocate memory then? I guess I could try to
SV>> figure it out myself, but it's a bit complex. Are there many
SV>> small chunks? Seems like emalloc() is called a lot creating
SV>> small chunks, but each emalloc() doesn't result in a
SV>> corresponding malloc() perhaps? I guess with glibc on Linux it

emalloc caches small memory blocks (certain amount of them), which are
pre-allocated at the init  time. I.e., actual malloc() doesn't get called
every time emalloc() is called. But this cache is mostly useful for Zend's
own memory allocations, it won't help if you allocate a lot of space in
your scripts.

SV>> would be nice if malloc() allocated say 128k chunks, and
SV>> emalloc() used memory inside those.

That would require us to implement own memory management, which is not too
simple.
-- 
Stanislav Malyshev, Zend Products Engineer
[EMAIL PROTECTED]  http://www.zend.com/ +972-3-6139665 ext.115




-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Stig Venaas]

On Wed, Jan 10, 2001 at 09:23:43PM +0200, Stanislav Malyshev wrote:
> SS>> Thanks for the correction!  I have never seen that in action
> SS>> though.  Usually, the C library will keep freed memory in
> SS>> free lists and does not decrease the size of the data
> SS>> segment using sbrk.
> 
> Actually, from glibc source there's a hook that is supposed to do this in
> some circumstances. Never could actually make it work in real life. Also,
> IIRC glibc on Linux allocates really large (8k+?  16k+?) blocks as mmap's
> of /dev/zero and those are really given back to system, since on free they
> are just munmap'ed.

Checked the glibc 2.2 sources, looks like 128k+ blocks are mmap'ed and
given back to the system.

How does Zend allocate memory then? I guess I could try to figure it
out myself, but it's a bit complex. Are there many small chunks? Seems
like emalloc() is called a lot creating small chunks, but each emalloc()
doesn't result in a corresponding malloc() perhaps? I guess with glibc
on Linux it would be nice if malloc() allocated say 128k chunks, and
emalloc() used memory inside those.

Stig

-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Stanislav Malyshev]

SS>> Thanks for the correction!  I have never seen that in action
SS>> though.  Usually, the C library will keep freed memory in
SS>> free lists and does not decrease the size of the data
SS>> segment using sbrk.

Actually, from glibc source there's a hook that is supposed to do this in
some circumstances. Never could actually make it work in real life. Also,
IIRC glibc on Linux allocates really large (8k+?  16k+?) blocks as mmap's
of /dev/zero and those are really given back to system, since on free they
are just munmap'ed.
-- 
Stanislav Malyshev, Zend Products Engineer
[EMAIL PROTECTED]  http://www.zend.com/ +972-3-6139665 ext.115



-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Stanislav Malyshev]

BS>> the number of bytes contained in the Increment parameter and
BS>> changes the amount of available space accordingly. The Increment
BS>> parameter can be a negative number, in which case the amount of
BS>> available space is decreased."

Theoretically, you are right. Practically, you need first to clean that
memory (i.e., not have malloc-ed blocks there), otherwise it would just
drop dead. Given that standard malloc/free combo doesn't give you _any_
control on where blocks are allocated, you'll have hard time freeing top
of your heep unless you free totally everything. If you really need that,
use custom allocator.

-- 
Stanislav Malyshev, Zend Products Engineer
[EMAIL PROTECTED]  http://www.zend.com/ +972-3-6139665 ext.115



-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Sascha Schumann]

> >From AIX docco...
> "The sbrk subroutine adds to the break value the number of bytes contained in
> the Increment parameter and changes the amount of available space accordingly.
> The Increment parameter can be a negative number, in which case the amount of
> available space is decreased."

Thanks for the correction!  I have never seen that in action
though.  Usually, the C library will keep freed memory in
free lists and does not decrease the size of the data
segment using sbrk.

- Sascha


-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Stig Venaas]

On Wed, Jan 10, 2001 at 02:04:43PM -0500, Bill Stoddard wrote:
> 
> Really?
> 
> >From AIX docco...
> "The sbrk subroutine adds to the break value the number of bytes contained in
> the Increment parameter and changes the amount of available space accordingly.
> The Increment parameter can be a negative number, in which case the amount of
> available space is decreased."

Right, but I don't know of any malloc() implementations that returns
freed memory to the system.

Stig

-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Bill Stoddard]

> > Any ideas ?
>
> Welcome to Unix.  You can only increase the heap size using
> sbrk(), you cannot shrink it.  So, if you have allocated n
> bytes, you can only reuse those bytes, but never give it back
> to the system (without terminating the process).
>
> - Sascha
>
Really?

>From AIX docco...
"The sbrk subroutine adds to the break value the number of bytes contained in
the Increment parameter and changes the amount of available space accordingly.
The Increment parameter can be a negative number, in which case the amount of
available space is decreased."

Bill





-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

Re: [PHP-DEV] PHP4: php_module holds memory to long !

[Sascha Schumann]

> Any ideas ?

Welcome to Unix.  You can only increase the heap size using
sbrk(), you cannot shrink it.  So, if you have allocated n
bytes, you can only reuse those bytes, but never give it back
to the system (without terminating the process).

- Sascha


-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

[PHP-DEV] PHP4: php_module holds memory to long !

[Filip Sielimowicz]

Hi !
Look at such example:


The apache server configuration:

Timeout 100 
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 3
MaxSpareServers 40
StartServers 3
MaxClients 40   <-- This is important !
MaxRequestsPerChild 40  <-- This is important !

Let us assume, that these 40 httpd processes handle
many requests to maintain some not small www portal.
Many php scripts are beeing executed. Most of them need
a small amount of memory (1-2 M). But there are a few scripts,
which use, mmm...let it be 30 M. And execucution of such "BIG"
script happens statistically 1 per 39 executions of "SMALL" scripts.
Let us assume, that times of execution of all scripts are similiar.
And all httpd processes are busy in almost 100% of time.

So...

In simple model the situation is not bad: in every moment
there should be about 1 (let it be 5 maximally)  "BIG" processes
and the rest are "SMALL".
The used memory is 5*30M + something - maximum 200 M.

But...
It is not in real life. 

The php_module DOES NOT release allocated memory (it is my opinion)
, until the httpd process is finishing (after 40 RequestPerChild from the
mentioned before apache configuration). It holds it ALL in order to use
again when next request comes. So the

Effect is...

After a short time we have about 20 httpd processes (mmm.. 15? when we
are lucky...), where EACH holds 30 M (with hope, that next script will
use it...).
This comes from such computation: every httpd process during its lifetime
executes a "BIG" script once (statistically), in random moment, so
statistically half of time is "before BIG execution", half is "after
BIG execution". So in every moment half of all httpd processes
are after "the BIG execution" and each of them holds 30 M memory.

20*30 M = 600 M =:O

oops... It is not so good.


Solutions:

1. Decrease the number of MaxRequestsPerChild

hmmm... but what is the optimal value ? It is
a problem to compute this in real life (where 
scripts are not just BIG and SMALL).
And the time for forks - performance lowers down.
:(

2. Decrese the value of MaxClients

...it is even worse then 1. The performance of 
our web server ... :(

3. Set the limit of memory

... BIG scripts can not be executed at all... :(

4. Force the httpd processes (exactly: force the 
php module as I suppose) to release the memory
BEFORE next request. Or better: set some limit
to which memory should be released at least. 

Yeah ... :))

That is it !

But...

How to do that ?
Any ideas ?


I've found out some interesting code in 
Zend/zend_alloc.h Zend/zend_alloc.c.
But setting CACHE_MEMORY_DISABLED to 1 does
not help.

Also functions shutdown_memory_manager()
and set_memory_manager are not simply useable.
I suppose it is not Zend's memory cache, that is
responsible for described effect.

Filip Sielimowicz


-- 
PHP Development Mailing List 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]