Re: Failures in TestRun.pm with A::T 1.09
William McKee wrote: [...] OK, I was up all night last night and am stopping here for the night. I'll rebuild mod_perl tomorrow and try again. I don't suspect this will make any difference since these messages started coming up when I was running 5.8.2. It's worth a try though. OK, please let us know once you get a clean build and if it's still failing we will take it from there. Thanks. __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: apache 1.3.29 apache 2.0.X pool problems and analysis
On Fri, 2004-03-12 at 04:21, Bojan Smojver wrote:
Quoting Mark Rowe [EMAIL PROTECTED]:
FYI, I'll only speak for APR/Apache 2.0. 1.3 has a somewhat different
implementation.
Check out this further pool test,
[...]
apr_pool_create_ex(subp1, p, fun, NULL);
apr_pool_create_ex(subp2, p, fun, NULL);
for (i = 0; i 1; i++) {
apr_pool_clear(subp1);
s = apr_pstrcat(subp1, s, tenbyte_string, NULL);
apr_pool_clear(subp2);
s = apr_pstrdup(subp2, s);
The the above duplicates s, so now we have 220,000 raw storage requirement,
given there are two pools in play. At least I think.
Let's disect this shall we? :)
apr_pool_create_ex(subp1, p, fun, NULL);
Create a subpool of p (which happens to be the request pool, but that doesn't
really matter here), sharing the allocator of p.
apr_pool_create_ex(subp2, p, fun, NULL);
Ditto.
for (i = 0; i 1; i++) {
apr_pool_clear(subp1);
Clear the subpool, returning all of it's memory, save 8k, to the allocator.
s = apr_pstrcat(subp1, s, tenbyte_string, NULL);
Allocate a new string out of subp1, of length s + tenbyte_string. Note
that once len(s) passes the 8k mark, we are always having to get memory
from the allocator in this case. In the allocator we look for a block
of the suitable size (always multiples of 4k), and return that. Note
that the smaller blocks are just sitting there, waiting to be reused.
apr_pool_clear(subp2);
Return all the memory of subp2 to the allocator, save 8k.
s = apr_pstrdup(subp2, s);
And allocate a new string out of subp2 of length s. Note that this
will get another block from the allocator, the same size as allocated
a few lines above for subp1. These blocks are in use at the same time,
so no opportunity for reuse.
}
apr_pool_destroy(subp1);
apr_pool_destroy(subp2);
And destroy both subpools, returning all their memory to the allocator
for reuse.
Isn't there some sort of algorithm in play that says:
- allocate contiguousregion of memory
- fit stuff into that region
- if it doesn't fit, allocate twice the size (or something)
- fit stuff into that other region if first doesn't have enough space
- and so on
That would be the algorithm employed by apr_psprintf and friends.
This might explain why the memory usage is soaring as you keep increasing the
string size.
The memory usage is soaring because no memory is being reused.
I'm not sure why memory isn't returned to the system on apr_pool_destroy().
Maybe it simply gets pasted to the parent pool in case that one needs more
memory in the future?
The memory is returned to the pools allocator. All pools sharing an
allocator can reuse memory from there. The allocator will return memory
it is holding, over a certain threshold, to the system. In the Apache
HTTP Server 2.0, the MaxMemFree directive can be used to influence the
threshold. The default is unlimited (except on NetWare IIRC).
HTH,
Sander
Re: cvs commit: httpd-2.0/modules/ssl ssl_engine_log.c
[EMAIL PROTECTED] wrote: jorton 2004/03/10 13:54:17 Modified:modules/ssl ssl_engine_log.c Log: * modules/ssl/ssl_engine_log.c (ssl_log_annotate, ssl_log_annotation, ssl_log_ssl_error): const-ify annotation strings and simplify ssl_log_annotation. -static char *ssl_log_annotation(char *error) +static const char *ssl_log_annotation(char *error) Shouldn't error be const, too? Cheers, Ben. -- http://www.apache-ssl.org/ben.html http://www.thebunker.net/ There is no limit to what a man can do or how far he can go if he doesn't mind who gets the credit. - Robert Woodruff
mod_ssl vulnerability in apache 2 - does it affect 1.3?
Greetings, A DoS vulnerability has been reported in mod_ssl in apache 2 (see http://secunia.com/advisories/11092/). Can anyone comment whether this vulnerability is present in mod_ssl in apache 1.3 or is it simply that mod_ssl, prior to apache 2, is not the responsibility of ASF? Rgds, Owen Boyle Disclaimer: Any disclaimer attached to this message may be ignored. This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please notify the sender urgently and then immediately delete the message and any copies of it from your system. Please also immediately destroy any hardcopies of the message. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. The sender's company reserves the right to monitor all e-mail communications through their networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to state them to be the views of the sender's company.
Re: mod_ssl vulnerability in apache 2 - does it affect 1.3?
On Fri, Mar 12, 2004 at 01:17:03PM +0100, Boyle Owen wrote: A DoS vulnerability has been reported in mod_ssl in apache 2 (see http://secunia.com/advisories/11092/). Can anyone comment whether this vulnerability is present in mod_ssl in apache 1.3 There has already been a couple of responses to the mod_ssl mailinglist (where this question belongs) that it doesn't affect 1.3 or is it simply that mod_ssl, prior to apache 2, is not the responsibility of ASF? It isn't an ASF project. Mod_ssl is a project run by Ralf Engelschall. vh Mads Toftum -- `Darn it, who spiked my coffee with water?!' - lwall
Re: functions in SSI
That definitely sounds useful. I think you can get the same effect, though, by using the existing 2.0/2.1 mod_include hook to add new directives... something like this: !--#set_random var=blah min=$min max=$max -- That's not quite as syntactically elegant, but it has the advantage of not requiring any new parsing code. What do you think? Brian On Mar 11, 2004, at 6:46 AM, Andre Breiler wrote: Hi, I wonder what your opinion is on having functions in addition to variables in SSIs. An example would be --#set var=blah value=random($min,$max) -- . My idea is to make ap_ssi_parse_string aware of functions and call a function which does the SSI function parsing/calling. In addition I'd export a function so that other modules can register SSI functions as well. Andre -- Andre' Breiler | Tel: +44 (0) 1628 40 BBC Internet Services | URL: http://support.bbc.co.uk Maiden House, Vanwell Road | Maidenhead, SL6 4UB | Mail me if possible. And use a Subject line.
apache written in c versus c++
c is smaller than c++ STL,
Using apache 2.0.48
I defined my own string struct
and my own 'mystrcat' function
and tenbyte_string as 0123456789
which is ten bytes.
Did this iteration 30,000 times as shown
below. It ran fast and small.
It built a string with a strlen() of 300,000
The child never grew beyond about 300 KB
of the resting child process. All children
dropped back down below 2 MB when
they were done.
I hammered it! I held down the 'F5' key on
this page on a high bandwidth connection
for 2 minutes or more.
Apache was fine very fast spawned many
children. All children were small and returned
to below 2 MB when I let up on 'F5' key.
---
#define tenbyte_string 0123456789
typedef struct mystr {
char *p;
int len;
} String;
void mystrcat(String *s, char *str) {
char *tmp_str = s-p;
int str_len = strlen(str);
s-p = realloc(tmp_str, s-len + str_len + 1 );
strcpy(s-p + s-len, str);
s-len += str_len;
}
void fun(void) {
fprintf(stderr, cannot create pool\n);
}
static int x_handler(request_rec *r)
{
int i;
String s = {0};
if (strcmp(r-handler, example-handler)) {
return DECLINED;
}
ap_set_content_type(r, text/html);
if (r-header_only) {
return OK;
}
ap_rputs(DOCTYPE_HTML_3_2, r);
ap_rputs(HTML\n, r);
ap_rputs( HEAD\n, r);
ap_rputs( TITLEmod_example Module Content-Handler Output\n, r);
ap_rputs( /TITLE\n, r);
ap_rputs( /HEAD\n, r);
ap_rputs( BODY\n, r);
ap_rputs( H1SAMPmod_example/SAMP Module Content-Handler
Output\n, r);
ap_rputs( /H1\n, r);
ap_rputs( P\n, r);
ap_rprintf(r, Apache HTTP Server version: \%s\\n,
ap_get_server_version());
ap_rputs( BR\n, r);
ap_rprintf(r, Server built: \%s\\n, ap_get_server_built());
ap_rputs(h1hello 30,000 world/h1br\n, r);
for (i = 0; i 3; i++) {
mystrcat(s, tenbyte_string);
}
ap_rprintf(r, strlen(s.p)=%d\n, strlen(s2.p));
free(s.p);
ap_rputs( /BODY\n, r);
ap_rputs(/HTML\n, r);
return OK;
}
---
Then I put the above code in to a single
command line program set iterations to
1,000,000 to make a 10,000,000 byte
long string. The program took 1.33 seconds
and it took 10,002,432 max mmap bytes
according to malloc_stats();
int i = 0;
String s2 = {0};
for (i = 0; i 1e6; ++i) {
mystrcat(s2, tenbyte_string);
}
Then I wrote the same equivalent code
in c++ using STL it took 1.25 seconds
and it took 25,174,016 bytes according
to malloc_stats();
int i = 0;
string s = ;
for(i=0; i 1e6; ++i) {
s += tenbyte_string;
}
So the c program took 1.33/1.25 = 1.06
times longer than the c++ program .
The c++ program took about 2.5 times
as much RAM as the c program.
c++ is marginally faster but takes
over twice as much RAM as c.
So if Apache is written well enough then
it will be very nearly as fast as if written in c++
and run in less than half the size in RAM from
this preliminary study.
My tests were run using gcc version
2.96 running on Redhat version 7.2
on an old Dell PowerApp 110 with
128 MB or RAM, 265 MB swap
and 600 MHz 80526 cpu.
glibc was GNU C Library stable release version 2.2.4
--- Mark R. Rowe, MSEE
Re: functions in SSI
On Mar 12, 2004, at 1:40 PM, Patrick Welche wrote: .. and if we are talking wish-list, I have often wanted to do something like !--#set var=dir value=/long/and/complicated/directory/path/name/ -- !--#include file=!--#echo var=dir --filename -- How about !--#include virtual=$dir/filename.html -- :) BTW, while we're discussing mod_include, you folks might be interested in a patch I posted a while ago for sticking query string values in mod_include vars. I don't know if it's still current to the newer mod_include code, but here it is if anyone's still interested. http://marc.theaimsgroup.com/?l=apache-httpd-devm=106352327003791w=2
Re: functions in SSI
Hi, On Fri, 12 Mar 2004, Brian Pane wrote: That definitely sounds useful. I think you can get the same effect, though, by using the existing 2.0/2.1 mod_include hook to add new directives... something like this: !--#set_random var=blah min=$min max=$max -- That's not quite as syntactically elegant, but it has the advantage of not requiring any new parsing code. What do you think? Yes that was one of my other ideas as well :) The reason I asked for the below first was that it makes it a little more easier for the html author and that other modules which might use ap_ssi_parse_string won't get the benefit of functions. In addition I was thinking about a function eval feeding the content of a variable into ap_ssi_parse_string again. The last minor thought was that I wouldn't have to replicate most of the #set code again. The third idea (which we use atm.) is: !--#set var=blah function=random value=$min value=$max . I really don't like that one as it means that I have to modify #set without getting a benefit. What speaks for your idea is that here wouldn't be any code changes to existing code needed and that it prevent dump users even getting near recursions. Let me know what you think if I should go for the more flexible (imho) or the safer way. Happy weekend, Andre' Brian On Mar 11, 2004, at 6:46 AM, Andre Breiler wrote: Hi, I wonder what your opinion is on having functions in addition to variables in SSIs. An example would be --#set var=blah value=random($min,$max) -- . My idea is to make ap_ssi_parse_string aware of functions and call a function which does the SSI function parsing/calling. In addition I'd export a function so that other modules can register SSI functions as well. Andre -- Andre' Breiler | Tel: +44 (0) 1628 40 BBC Internet Services | URL: http://support.bbc.co.uk Maiden House, Vanwell Road | Maidenhead, SL6 4UB | Mail me if possible. And use a Subject line. -- Andre' Breiler | Tel: +44 (0) 1628 40 BBC Internet Services | URL: http://support.bbc.co.uk Maiden House, Vanwell Road | Maidenhead, SL6 4UB | Mail me if possible. And use a Subject line.
Re: apache written in c versus c++
Memory allocation varies widely with the platform and runtime
libraries. It's said that the new[] operator is preferred over malloc
because it lets the OS handle memory management where it belongs. The
concept being (I gather) that an implementation of malloc doesn't
necessarily understand the hardware it's running on, while the OS does.
That's not true for MSVCRT6 - new and malloc call the same function,
and in either Borland or Watcom I've heard that new actually gets
thunked a bit before ultimately calling malloc anyway (so new is the
same efficiency for management, but a hair slower).
In MSVCRT7 multithreaded, memory allocations (and releases) are
preceded by a mutex lock which requires a context switch to the kernel
(and back). Allocation under linux (at least under RedHat 9, and likely
under all linux) is many times faster than under Windows due to this.
It sounds like the apache allocator works directly with pages though?
(or is the 4k just an optimal number?) In the case of the apache web
server I'd think that the common case is that you'd be repeatedly
allocating and releasing similar sized chunks of memory, and the example
of an ever-growing string is a little unusual. If you optimize for the
former, then when a block of memory is released you want to keep it
around in its current size because it's very likely another call will
request the same size block again momentarily. If you optimize for the
later then you want to combine smaller blocks into bigger blocks
agressively, or if that's not possible then ditch the smaller blocks
because it's likely that the smaller blocks will never be reused.
These two situations are at odds with each other. The former solution
handles the later situation pretty poorly. The later solution would
handle the former situation no worse than any other condition, but not
as good as the former solution.
You can find a worst case for most any allocation scheme, but is it a
real situation that you need to solve, or is this just an example of
finding that worst case situation?
Andrew
Mark Rowe wrote:
c is smaller than c++ STL,
Using apache 2.0.48
I defined my own string struct
and my own 'mystrcat' function
and tenbyte_string as 0123456789
which is ten bytes.
Did this iteration 30,000 times as shown
below. It ran fast and small.
It built a string with a strlen() of 300,000
The child never grew beyond about 300 KB
of the resting child process. All children
dropped back down below 2 MB when
they were done.
I hammered it! I held down the 'F5' key on
this page on a high bandwidth connection
for 2 minutes or more.
Apache was fine very fast spawned many
children. All children were small and returned
to below 2 MB when I let up on 'F5' key.
---
#define tenbyte_string 0123456789
typedef struct mystr {
char *p;
int len;
} String;
void mystrcat(String *s, char *str) {
char *tmp_str = s-p;
int str_len = strlen(str);
s-p = realloc(tmp_str, s-len + str_len + 1 );
strcpy(s-p + s-len, str);
s-len += str_len;
}
void fun(void) {
fprintf(stderr, cannot create pool\n);
}
static int x_handler(request_rec *r)
{
int i;
String s = {0};
if (strcmp(r-handler, example-handler)) {
return DECLINED;
}
ap_set_content_type(r, text/html);
if (r-header_only) {
return OK;
}
ap_rputs(DOCTYPE_HTML_3_2, r);
ap_rputs(HTML\n, r);
ap_rputs( HEAD\n, r);
ap_rputs( TITLEmod_example Module Content-Handler Output\n, r);
ap_rputs( /TITLE\n, r);
ap_rputs( /HEAD\n, r);
ap_rputs( BODY\n, r);
ap_rputs( H1SAMPmod_example/SAMP Module Content-Handler
Output\n, r);
ap_rputs( /H1\n, r);
ap_rputs( P\n, r);
ap_rprintf(r, Apache HTTP Server version: \%s\\n,
ap_get_server_version());
ap_rputs( BR\n, r);
ap_rprintf(r, Server built: \%s\\n, ap_get_server_built());
ap_rputs(h1hello 30,000 world/h1br\n, r);
for (i = 0; i 3; i++) {
mystrcat(s, tenbyte_string);
}
ap_rprintf(r, strlen(s.p)=%d\n, strlen(s2.p));
free(s.p);
ap_rputs( /BODY\n, r);
ap_rputs(/HTML\n, r);
return OK;
}
---
Then I put the above code in to a single
command line program set iterations to
1,000,000 to make a 10,000,000 byte
long string. The program took 1.33 seconds
and it took 10,002,432 max mmap bytes
according to malloc_stats();
int i = 0;
String s2 = {0};
for (i = 0; i 1e6; ++i) {
mystrcat(s2, tenbyte_string);
}
Then I wrote the same equivalent code
in c++ using STL it took 1.25 seconds
and it took 25,174,016 bytes according
to malloc_stats();
int i = 0;
string s = ;
for(i=0;
Re: functions in SSI
On Fri, Mar 12, 2004 at 01:53:20PM -0500, Mike Friedman wrote: On Mar 12, 2004, at 1:40 PM, Patrick Welche wrote: .. and if we are talking wish-list, I have often wanted to do something like !--#set var=dir value=/long/and/complicated/directory/path/name/ -- !--#include file=!--#echo var=dir --filename -- How about !--#include virtual=$dir/filename.html -- :) OK, you can pick yourself of the floor and stop laughing :-) Given the above, there can't be a contrived example - must revisit the problem.. Thanks, Patrick
Re: 2.0.49 (rc1) tarballs available for testing
On Tue, Mar 09, 2004 at 06:02:03PM +0100, Sander Striker wrote: Hi, There are 2.0.49-rc1 tarballs available for testing at: http://httpd.apache.org/dev/dist/ Please report your results to [EMAIL PROTECTED] Thanks in advance, Sander Fails in BSD/OS 5.1 sh-2.02# make install Making install in srclib Making install in apr Making all in strings Making all in passwd Making all in tables Making all in file_io/unix Making all in network_io/unix Making all in threadproc/unix Making all in misc/unix Making all in locks/unix Making all in time/unix Making all in mmap/unix Making all in shmem/unix Making all in user/unix Making all in memory/unix Making all in atomic/unix Making all in poll/unix Making all in support/unix Making all in dso/unix if [ ! -d /usr/include/apache ]; then /usr/source/httpd-2.0.49-rc1/srclib/apr/build/mkdir.sh /usr/include/apache; fi; cp -p /usr/source/httpd-2.0.49-rc1/srclib/apr/include/*.h /usr/include/apache; if test -n /usr/source/httpd-2.0.49-rc1/srclib/apr; then cp -p /usr/source/httpd-2.0.49-rc1/srclib/apr/include/*.h /usr/include/apache; fi; if [ ! -d /usr/lib ]; then /usr/source/httpd-2.0.49-rc1/srclib/apr/build/mkdir.sh /usr/lib; fi; /bin/bash /usr/source/httpd-2.0.49-rc1/srclib/apr/libtool --mode=install cp libapr-0.la /usr/lib cp .libs/libapr-0.lai /usr/lib/libapr-0.la cp .libs/libapr-0.a /usr/lib/libapr-0.a ranlib /usr/lib/libapr-0.a chmod 644 /usr/lib/libapr-0.a -- Libraries have been installed in: /usr/lib If you ever happen to want to link against installed libraries in a given directory, LIBDIR, you must either use libtool, and specify the full pathname of the library, or use the `-LLIBDIR' flag during linking and do at least one of the following: - add LIBDIR to the `LD_RUN_PATH' environment variable during linking - use the `-Wl,--rpath -Wl,LIBDIR' linker flag - have your system administrator add LIBDIR to `/etc/ld.so.conf' See any operating system documentation about shared libraries for more information, such as the ld(1) and ld.so(8) manual pages. -- /bin/bash /usr/source/httpd-2.0.49-rc1/srclib/apr/libtool --mode=install cp apr.exp /usr/lib cp apr.exp /usr/lib/apr.exp if [ ! -d /var/www/build ]; then /usr/source/httpd-2.0.49-rc1/srclib/apr/build/mkdir.sh /var/www/build; fi; if [ -f libtool ]; then /bin/bash /usr/source/httpd-2.0.49-rc1/srclib/apr/libtool --mode=install cp libtool /var/www/build; fi; cp libtool /var/www/build/libtool if [ -f shlibtool ]; then /bin/bash /usr/source/httpd-2.0.49-rc1/srclib/apr/libtool --mode=install cp shlibtool /var/www/build; fi; if [ -f build/apr_rules.mk ]; then cp build/apr_rules.mk /var/www/build; fi; if [ ! -d /usr/bin ]; then /usr/source/httpd-2.0.49-rc1/srclib/apr/build/mkdir.sh /usr/bin; fi; /bin/bash /usr/source/httpd-2.0.49-rc1/srclib/apr/libtool --mode=install cp apr-config /usr/bin cp apr-config /usr/bin/apr-config chmod 755 /usr/bin/apr-config Making install in apr-util Making all in buckets Making all in crypto Making all in dbm Making all in sdbm Making all in . Making all in encoding Making all in hooks Making all in ldap Making all in uri Making all in xml Making all in expat Making all in misc Making all in strmatch Making all in xlate if [ ! -d /usr/include/apache ]; then /usr/source/httpd-2.0.49-rc1/srclib/apr/build/mkdir.sh /usr/include/apache; fi; cp -p /usr/source/httpd-2.0.49-rc1/srclib/apr-util/include/*.h /usr/include/apache if [ -n /usr/source/httpd-2.0.49-rc1/srclib/apr-util ]; then cp -p /usr/source/httpd-2.0.49-rc1/srclib/apr-util/include/*.h /usr/include/apache; fi; if [ ! -d /usr/lib ]; then /usr/source/httpd-2.0.49-rc1/srclib/apr/build/mkdir.sh /usr/lib; fi; list='xml/expat'; for i in $list; do ( cd $i ; make DESTDIR= install ); done /bin/bash /usr/source/httpd-2.0.49-rc1/srclib/apr-util/xml/expat/conftools/mkinstalldirs /usr/lib /usr/include/apache /bin/bash ../libtool --mode=install /usr/bin/install -c libexpat.la /usr/lib/libexpat.la /usr/bin/install -c .libs/libexpat.lai /usr/lib/libexpat.la /usr/bin/install -c .libs/libexpat.a /usr/lib/libexpat.a ranlib /usr/lib/libexpat.a chmod 644 /usr/lib/libexpat.a -- Libraries have been installed in: /usr/lib If you ever happen to want to link against installed libraries in a given directory, LIBDIR, you must either use libtool, and specify the full pathname of the library, or use the `-LLIBDIR' flag during linking and do at least one of the following: - add LIBDIR to the `LD_RUN_PATH' environment variable during linking - use the `-Wl,--rpath -Wl,LIBDIR' linker flag - have your system administrator add LIBDIR to `/etc/ld.so.conf' See any operating system documentation about shared libraries for more information, such as the ld(1) and ld.so(8) manual pages.
Re: 2.0.49 (rc1) tarballs available for testing
The Doctor wrote: On Tue, Mar 09, 2004 at 06:02:03PM +0100, Sander Striker wrote: Hi, There are 2.0.49-rc1 tarballs available for testing at: http://httpd.apache.org/dev/dist/ Please report your results to [EMAIL PROTECTED] Thanks in advance, Sander Fails in BSD/OS 5.1 sh-2.02# apace hectl start Syntax error on line 232 of /var/www/conf/httpd.conf: Cannot load /usr/libexec/apache/mod_access.so into server: Cannot open /usr/libexec/apache/mod_access.so sh-2.02# exit exit Script done on Wed Mar 10 10:53:17 2004 What got installed into /usr/libexec/apache by make install? Did the DSOs get installed with unexpected names? Second, does 2.0.48 built in exactly the same way fail in a similar manner?
