DO NOT REPLY [Bug 36083] - can't configure nested applications
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG· RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://issues.apache.org/bugzilla/show_bug.cgi?id=36083. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND· INSERTED IN THE BUG DATABASE. http://issues.apache.org/bugzilla/show_bug.cgi?id=36083 --- Additional Comments From [EMAIL PROTECTED] 2005-08-19 00:49 --- First. Each app have it's own web.config. Second. Inverting order does not affect the result. Omiting sub-mounts does not change the result. It's not an apache error. ASP.NET informs that some web.config instructions must have been declared at application level, same on IIS when directories are not configured as applications. Pet. -- Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the assignee for the bug, or are watching the assignee.
DO NOT REPLY [Bug 36083] - can't configure nested applications
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG· RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://issues.apache.org/bugzilla/show_bug.cgi?id=36083. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND· INSERTED IN THE BUG DATABASE. http://issues.apache.org/bugzilla/show_bug.cgi?id=36083 --- Additional Comments From [EMAIL PROTECTED] 2005-08-19 01:00 --- Interesting. I'll inspect the code to ensure we aren't reordering things internally. Does the error change if you mount /foo then /foo/this/that, rather than in the order /foo/this/that and then /foo? -- Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the assignee for the bug, or are watching the assignee.
Re: mod_python 3.2.0-BETA available for testing
A few comments: 1. If you have an older version of flex than that expected, it gives message: checking flex version... configure: WARNING: Flex version 2.5.31 or greater is required. The one you have seems to be 2.5.4. Use --with-flex to specify another. There is nothing in the README describing why such a requirement exists. The only veiled comment is in the configure script itself where it says: # check for correct flex version # requires flex 2.5.31 for reentrant support A more prominent explanation is required in the README with configure outputing with the message above a pointer to consult the README for further details. What also isn't at all clear, is whether the flex warning is relevant anyway if pre-generated code for the lexical analyser is provided in the tar ball and that is used. Thus, what is the real story, am I going to have a problem or not if I have an older version of flex? 2. Publisher still doesn't check special extensions supplied by the AddHandler or PythonHandler directives. Ie., http://issues.apache.org/jira/browse/MODPYTHON-72 This may mean that code that previously worked no longer works. Ie., this feature was in there before. Only caveat is that the feature may not have worked properly before anyway, so existing code may not be affected after all. See: http://issues.apache.org/jira/browse/MODPYTHON-22 Bit unsure on this one. 3. Only other issue at the moment I can think of is whether or not anything prominent has been put in main documentation about changed behaviour of module loading in mod_python.publisher. In the main the changes are for the better and things will actually work better. There is one case though where existing code can stop working. This is where code makes an assumption that global data within a publisher module is preserved across module reloads. I am not saying this change is wrong, but it perhaps may need to be clearly explained in documentation else we will get lots of questions about it no doubt, with the common answer probably being that they shouldn't be storing such persistent module data in a module that can be reloaded anyway. Thats all for now until I can get some real time to look at it. :-) Graham On 18/08/2005, at 4:03 AM, Jim Gallacher wrote: Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim
Re: mod_python 3.2.0-BETA available for testing
Hello, I ran into a problem with the loader on MacOSX. MaxOSX 1.4.2 python 2.4.1 apache 2.0.54 The loader seems to not like the -undefined suppress arguments in the final load. I modified dist/setup.py by removing the two -undefined suppress and ran configure and make again and the new mod_python does build and seems to work ok. test.py can not find the apache log file after it has shut down apache, after all of the other tests pass! This may be cause by my installation of apache and may not be a mod_python problem, but I'll need to look further. cheers, Ron Ron Reisor [EMAIL PROTECTED] (RWR3) University of Delaware Information Technologies/Network and Systems Services Computing Center/192 South Chapel Street/Newark DE, 19716 pgp finger print: 0D 73 06 6F D3 6A 99 D3 F5 D5 6E FF 3B B9 7C 2Cchecking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ANSI C... none needed checking for ar... ar checking for a BSD-compatible install... /usr/bin/install -c checking whether make sets $(MAKE)... yes checking for main in -lm... yes checking for an ANSI C-conforming const... yes checking your blood pressure... a bit high, but we can proceed configure: checking whether apxs is available... checking for --with-apxs... /usr/local/bin/apxs executable, good checking Apache version... 2.0.54 checking for Apache libexec directory... /usr/local/modules checking for Apache include directory... -I/usr/local/include checking for --with-python... no checking for python... /usr/local/bin/python checking Python version... 2.4 checking Python install prefix... /Library/Frameworks/Python.framework/Versions/2.4 checking what libraries Python was linked with... -framework Python-ldl checking linker flags used to link Python... checking where Python include files are... -I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 checking for --with-python-src... no checking for --with-flex... no checking for flex... /usr/bin/flex found /usr/bin/flex, we'll use this. Use --with-flex to specify another. checking flex version... configure: WARNING: Flex version 2.5.31 or greater is required. The one you have seems to be 2.5.4. Use --with-flex to specify another. checking for --with-max-locks... no Using 8 MAX_LOCKS. configure: creating ./config.status config.status: creating Makefile config.status: creating src/Makefile config.status: creating Doc/Makefile config.status: creating src/include/mod_python.h config.status: creating test/testconf.py config.status: creating dist/setup.py config.status: creating dist/Makefile log.make Description: Binary data
Re: mod_python 3.2.0-BETA available for testing
+1 on Win32 with Python 2.4. Here is how I tested it : 1) Switched to the 3.2.0-BETA tag 2) Update 3) cd dist build_installer.bat 4) Got this installer, that you can also download : http://nicolas.lehuen.com/download/mod_python/mod_python-3.2.0-BETA.win32-py2.4.exe 5) Ran the installer, everything installed correctly 6) cd test python test.py, all unit test pass 7) Used the test handler to check various version information 8) Made a few test this my own applications, everything seems OK (which is not a big surprise since I used the dev version before). Regards, Nicolas2005/8/18, Jim Gallacher [EMAIL PROTECTED]: Graham Dumpleton wrote: A few comments: 1. If you have an older version of flex than that expected, it gives message: checking flex version... configure: WARNING: Flex version 2.5.31 or greater is required.The one you have seems to be 2.5.4. Use --with-flex to specify another. There is nothing in the README describing why such a requirement exists. The only veiled comment is in the configure script itself where it says: # check for correct flex version # requires flex 2.5.31 for reentrant support A more prominent explanation is required in the README with configure outputing with the message above a pointer to consult the README for further details. What also isn't at all clear, is whether the flex warning is relevant anyway if pre-generated code for the lexical analyser is provided in the tar ball and that is used.This is explained in the html docs, but didn't make it into the README.I wasn't sure if the WARNING was the right idea,See doc-html/inst-configure.html: Attempts to locate flex and determine its version. If flex cannotbe found in your PATH configurewill fail. If the wrong version isfound configure will generate a warning. You can generally ignore this warning unless you need to re-create src/psp_parser.c.The parser used by psp (See 4.9) is written in C generated using flex.This requires a reentrant version of flex which at this time is 2.5.31.Most platforms however ship with version 2.5.4 which is not suitable, soa pre-generated copy of psp_parser.c is included with the source. If youdo need to compile src/psp_parser.c you must get the correct flex version. If the first flex binary in the path is not suitable or not the one desired you can specify an alternative location with the --with-flexoption, e.g: $ ./configure --with-flex=/usr/local/bin/flexI'll copy this into the README as well. In 3.1.x the path to flex was hard coded to /usr/local/bin, which causedcompilation to fail and required the user to edit the Makefilemanually.Other than the detection of the flex path, the behaviour is the same as 3.1.x. The previous warning about needing the correct flexversion was just a comment in the Makefile. Any suggestions on how tomake this clearer are welcome. Thus, what is the real story, am I going to have a problem or not if I have an older version of flex?No, see html docs snippet above.No time to look at the rest. Will comment tonight.Jim 2. Publisher still doesn't check special extensions supplied by the AddHandler or PythonHandler directives. Ie., http://issues.apache.org/jira/browse/MODPYTHON-72 This may mean that code that previously worked no longer works. Ie., this feature was in there before. Only caveat is that the feature may not have worked properly before anyway, so existing code may not be affected after all. See: http://issues.apache.org/jira/browse/MODPYTHON-22 Bit unsure on this one. 3. Only other issue at the moment I can think of is whether or not anything prominent has been put in main documentation about changed behaviour of module loading in mod_python.publisher. In the main the changes are for the better and things will actually work better. There is one case though where existing code can stop working. This is where code makes an assumption that global data within a publisher module is preserved across module reloads. I am not saying this change is wrong, but it perhaps may need to be clearly explained in documentation else we will get lots of questions about it no doubt, with the common answer probably being that they shouldn't be storing such persistent module data in a module that can be reloaded anyway. Thats all for now until I can get some real time to look at it. :-) Graham On 18/08/2005, at 4:03 AM, Jim Gallacher wrote: Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the
Re: flex [was mod_python 3.2.0-BETA available for testing]
Gregory (Grisha) Trubetskoy wrote: OK, here is the flex scoop - as the the docs point out, anything before 2.5.31 is not reentrant and I think even uses a slightly different interface so older flex won't even process the psp_parser.l file correctly. Looking at Fedora Core 4, it still has flex 2.5.4a. (Note that 2.5.31 2.5.4 because 31 4 - I for a while had trouble seeing that for some reason), so the new flex is still not commonplace. This is true. I took a quick survey when I was adding the --with-flex option, and 2.5.4 was the most common. Oh, and the version numbering confused me for a while as well, but the configure script does take it into account when deciding if it has the correct flex. So until reentrant flex becomes commonplace, the solution was to include a pre-parsed psp_parser.c so that you woudln't need flex at all to compile mod_python. Looks like this still should be the case. The ./configure should just print a warning that if flex is not found or too old, should you need to rebuild psp_parser.c you will need to get the right version of flex. When I added the flex detection I had not considered the case where there was *no* flex on the system... just tested for the correct version. :-( Also, I wasn't sure how verbose to make the configure message. I just figured that people would see the WARNING and then go to the README to see what the deal was, which would have been the perfect plan... if I had actually updated the README accordingly. I'll make the fixes to configure and the README. Is the explanation in doc-html/inst-configure.html clear or does it need to be revised as well? Jim See doc-html/inst-configure.html: Attempts to locate flex and determine its version. If flex cannot be found in your PATH configure will fail. If the wrong version is found configure will generate a warning. You can generally ignore this warning unless you need to re-create src/psp_parser.c. The parser used by psp (See 4.9) is written in C generated using flex. This requires a reentrant version of flex which at this time is 2.5.31. Most platforms however ship with version 2.5.4 which is not suitable, so a pre-generated copy of psp_parser.c is included with the source. If you do need to compile src/psp_parser.c you must get the correct flex version. If the first flex binary in the path is not suitable or not the one desired you can specify an alternative location with the --with-flex option, e.g: $ ./configure --with-flex=/usr/local/bin/flex I'll copy this into the README as well. In 3.1.x the path to flex was hard coded to /usr/local/bin, which caused compilation to fail and required the user to edit the Makefile manually. Other than the detection of the flex path, the behaviour is the same as 3.1.x. The previous warning about needing the correct flex version was just a comment in the Makefile. Any suggestions on how to make this clearer are welcome.
Re: mod_python 3.2.0-BETA available for testing
On 19/08/2005, at 2:59 AM, Jim Gallacher wrote: Ron Reisor wrote: Hello, I ran into a problem with the loader on MacOSX. MaxOSX 1.4.2 python 2.4.1 apache 2.0.54 The loader seems to not like the -undefined suppress arguments in the final load. I modified dist/setup.py by removing the two -undefined suppress and ran configure and make again and the new mod_python does build and seems to work ok. The bit of code which includes -undefined suppress came from the patch Graham submitted for MODPYTHON-65. See http://issues.apache.org/jira/browse/MODPYTHON-65 for details. Not being a Mac person I can't comment further. Feel free to talk among yourselves. ;) Yep, my fault. It will not build on Mac OS X 10.3.9 without that option if you are using the standard version of GCC shipped with the operating system. I'll get onto my new Tiger laptop and try it there (haven't yet), but can you tell me which GCC you are using? I know the GCC version shipped on the box with Tiger, but are you using that, or are you using one supplied with Fink? Was worried that the change might not apply to the Fink version of GCC. Graham
Re: [PATCH] mod_disk_cache deterministic tempfiles
On Wed, Aug 17, 2005 at 03:17:41PM -0700, Justin Erenkrantz wrote:
Content definitely should not be served from the cache after it has
expired imo. However I think an approach like;
if((now + interval) expired) {
if(!stat(tmpfile)) {
update_cache_from_backend();
}
}
ie revalidate the cache content after N-seconds before it is due to be
expired would have the same effect, but avoid serving stale content.
Does that make sense?
Um, isn't that what we already do? -- justin
No :-) Right now, once the cache entity stops being considered fresh,
each worker that gets a request will fetch it, store it to their own
tmpfile and race to be first to rename it.
With the above approach, there could be some kind of user-configurable
interval before expiry, say 10 minutes in this example. So, the first
request that happens within that 10 minute interval would re-validate
the cache entity.
If it doesn't need to be fetched and gets a not-modified, we just
touch the cache entity and go on our merry way.
If it does need to be fetched then a deterministic tmpfile ensures that
no other worker bothers to fetch it at the same time, they can serve
from the cache in the meantime - which still hasn't expired.
If after the 10 minutes the content still isn't there, the user should
have set a better value for interval, and we've no choice but to start
fetching the content per-request until something writes a cache file.
--
Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Re: Rolling 2.1.7 On Friday
On Wed, Aug 17, 2005 at 10:43:01PM -0700, Paul Querna wrote: Just a heads up, I am planning to RM and tag 2.1.7 (and re-branch from trunk the 2.2.x branch) on Friday or Saturday this week. I intend to include APR and APR-Util 1.2.1 with this release. Sounds great. Can you exclude the mod_setenvif OID() changes? I don't think they are ready for release. mod_setenvif.c @ rev 153384 was the last good one. joe
Re: [PATCH] mod_disk_cache deterministic tempfiles
Jim Jagielski wrote: Would be interesting to profile that to determine how expensive those stats could be... Should be much less expensive than the case now -- the thundering herd when an object expires. Of course, the behavior would be configurable. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: [PATCH] mod_cache. Allow override of some vary headers
[EMAIL PROTECTED] wrote: What sort of testing did you do in those few hours? That's why I posted it, to get some more input. If I could play devil's advocate for a moment... my concern would be that you haven't considered certain scenarios that won't work with this patch. Most of the cache code has corner cases where it falls apart. Did you test a scenario whereby the COS ( Content Origin Server ) is actually trying to Vary: the content based on User-Agent: regardless of the Accept-Encoding: value? Then you shouldn't override User-Agent. I assume that the admin knows alot about the COS. If not, then these options should not be used. I am targeting the reverse cache where the admin know alot about the origin. COS has sent 2 totally different responses based on whether 'User-Agent' was MSIE or NETSCAPE. Both of those responses were compressed ( Content-Encoding: gzip ) and they should have been stored as 2 different ( compressed ) 'variants' for the same response based on ( non-overriden ) actual value of User-Agent field. In this case, you could use SetEnvIf to set User-Agent to NETSCAPE or MSIE based on the actual value. If you know this much about the content, then its trivial. If not, don't override anything. It's like CacheIgnoreHeaders. It breaks RFC's, but there are cases where this is A Good Thing. In the general case, as you pointed out, it is not. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: [PATCH] mod_cache. Allow override of some vary headers
Here's a new patch that changes the option name to CacheVaryOverride and
does some of the stuff Justin recommened.
--
Brian Akins
Lead Systems Engineer
CNN Internet Technologies
diff -ru httpd-trunk.orig/modules/cache/cache_storage.c
httpd-trunk.new/modules/cache/cache_storage.c
--- httpd-trunk.orig/modules/cache/cache_storage.c 2005-07-13
15:23:03.892378000 -0400
+++ httpd-trunk.new/modules/cache/cache_storage.c 2005-08-18
08:13:30.098771299 -0400
@@ -230,7 +230,7 @@
* is this header in the request and the header in the cached
* request identical? If not, we give up and do a straight get
*/
-h1 = apr_table_get(r-headers_in, name);
+h1 = ap_cache_override_header(r, r-headers_in, name);
h2 = apr_table_get(h-req_hdrs, name);
if (h1 == h2) {
/* both headers NULL, so a match - do nothing */
diff -ru httpd-trunk.orig/modules/cache/cache_util.c
httpd-trunk.new/modules/cache/cache_util.c
--- httpd-trunk.orig/modules/cache/cache_util.c 2005-07-13 15:23:03.869381000
-0400
+++ httpd-trunk.new/modules/cache/cache_util.c 2005-08-18 08:15:05.349865494
-0400
@@ -534,3 +534,49 @@
}
return headers_out;
}
+
+CACHE_DECLARE(const char* )ap_cache_override_header(request_rec *r,
+apr_table_t *t,
+const char* key) {
+const char *o;
+cache_server_conf *conf = ap_get_module_config(r-server-module_config,
+ cache_module);
+const char *header = NULL;
+
+if((o = apr_table_get(conf-override_headers, key)) != NULL) {
+if((header = apr_table_get(r-subprocess_env, o)) == NULL) {
+header = -;
+}
+}
+
+if(!header) {
+header = apr_table_get(t, key);
+}
+
+return header;
+}
+
+/* replace headers based on environment overrides
+ * modifies t in place
+ */
+CACHE_DECLARE(apr_status_t )ap_cache_override_hdrs(request_rec *r,
+ apr_table_t *t)
+{
+int i;
+apr_table_entry_t *elts;
+cache_server_conf *conf = ap_get_module_config(r-server-module_config,
+ cache_module);
+
+/* replace headers with environment overrides*/
+elts = (apr_table_entry_t *) apr_table_elts(conf-override_headers)-elts;
+
+for (i = 0; i apr_table_elts(conf-override_headers)-nelts; ++i) {
+const char *val = NULL;
+
+val = ap_cache_override_header(r, t, elts[i].key);
+apr_table_set(t, elts[i].key, val);
+
+}
+return APR_SUCCESS;
+}
+
diff -ru httpd-trunk.orig/modules/cache/mod_cache.c
httpd-trunk.new/modules/cache/mod_cache.c
--- httpd-trunk.orig/modules/cache/mod_cache.c 2005-08-09 11:51:09.471251000
-0400
+++ httpd-trunk.new/modules/cache/mod_cache.c 2005-08-18 08:14:39.755139238
-0400
@@ -745,6 +745,7 @@
/* array of headers that should not be stored in cache */
ps-ignore_headers = apr_array_make(p, 10, sizeof(char *));
ps-ignore_headers_set = CACHE_IGNORE_HEADERS_UNSET;
+ps-override_headers = apr_table_make(p, 10);
return ps;
}
@@ -790,6 +791,9 @@
(overrides-ignore_headers_set == CACHE_IGNORE_HEADERS_UNSET)
? base-ignore_headers
: overrides-ignore_headers;
+ps-override_headers = apr_table_copy(p, base-override_headers);
+apr_table_overlap(ps-override_headers, overrides-override_headers, 0);
+
return ps;
}
static const char *set_cache_ignore_no_last_mod(cmd_parms *parms, void *dummy,
@@ -873,6 +877,19 @@
return NULL;
}
+static const char *add_override_header(cmd_parms *parms, void *dummy,
+ const char *header, const char* env)
+{
+cache_server_conf *conf;
+conf =
+(cache_server_conf *)ap_get_module_config(parms-server-module_config,
+ cache_module);
+
+apr_table_set(conf-override_headers, header, env);
+
+return NULL;
+}
+
static const char *add_cache_enable(cmd_parms *parms, void *dummy,
const char *type,
const char *url)
@@ -1002,6 +1019,9 @@
AP_INIT_ITERATE(CacheIgnoreHeaders, add_ignore_header, NULL, RSRC_CONF,
A space separated list of headers that should not be
stored by the cache),
+AP_INIT_TAKE2(CacheVaryOverride, add_override_header, NULL, RSRC_CONF,
+A header that should be replaced by the value of
+ the given environment variable),
AP_INIT_TAKE1(CacheLastModifiedFactor, set_cache_factor, NULL, RSRC_CONF,
The factor used to estimate Expires date from
LastModified date),
diff -ru
[PATCH] mod_disk_cache: speed up read_table
This one's kinda ugly. Rather than rely on apr_file_gets, this stores the total length of the tables, then the serialized table in store_table. In read_table, it reads this length, allocs that amount, and reads the headers into the buffer. Then it just uses memchr to parse it into a table. I didn't mess with the EBCDIC stuff, so alot of the patch is just where I commented all that out. I also commented out where the headers file is opened buffered. It is faster on my tests boxen (Linux 2.6) to use unbuffered (probably because at lease pagesize if being buffered anyway). YMMV on this one. Anyway, I got a nice 5-8% increase in all my benchmarks by using this change. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: mod_python 3.2.0-BETA available for testing
Graham Dumpleton wrote: A few comments: 1. If you have an older version of flex than that expected, it gives message: checking flex version... configure: WARNING: Flex version 2.5.31 or greater is required. The one you have seems to be 2.5.4. Use --with-flex to specify another. There is nothing in the README describing why such a requirement exists. The only veiled comment is in the configure script itself where it says: # check for correct flex version # requires flex 2.5.31 for reentrant support A more prominent explanation is required in the README with configure outputing with the message above a pointer to consult the README for further details. What also isn't at all clear, is whether the flex warning is relevant anyway if pre-generated code for the lexical analyser is provided in the tar ball and that is used. This is explained in the html docs, but didn't make it into the README. I wasn't sure if the WARNING was the right idea, See doc-html/inst-configure.html: Attempts to locate flex and determine its version. If flex cannot be found in your PATH configure will fail. If the wrong version is found configure will generate a warning. You can generally ignore this warning unless you need to re-create src/psp_parser.c. The parser used by psp (See 4.9) is written in C generated using flex. This requires a reentrant version of flex which at this time is 2.5.31. Most platforms however ship with version 2.5.4 which is not suitable, so a pre-generated copy of psp_parser.c is included with the source. If you do need to compile src/psp_parser.c you must get the correct flex version. If the first flex binary in the path is not suitable or not the one desired you can specify an alternative location with the --with-flex option, e.g: $ ./configure --with-flex=/usr/local/bin/flex I'll copy this into the README as well. In 3.1.x the path to flex was hard coded to /usr/local/bin, which caused compilation to fail and required the user to edit the Makefile manually. Other than the detection of the flex path, the behaviour is the same as 3.1.x. The previous warning about needing the correct flex version was just a comment in the Makefile. Any suggestions on how to make this clearer are welcome. Thus, what is the real story, am I going to have a problem or not if I have an older version of flex? No, see html docs snippet above. No time to look at the rest. Will comment tonight. Jim 2. Publisher still doesn't check special extensions supplied by the AddHandler or PythonHandler directives. Ie., http://issues.apache.org/jira/browse/MODPYTHON-72 This may mean that code that previously worked no longer works. Ie., this feature was in there before. Only caveat is that the feature may not have worked properly before anyway, so existing code may not be affected after all. See: http://issues.apache.org/jira/browse/MODPYTHON-22 Bit unsure on this one. 3. Only other issue at the moment I can think of is whether or not anything prominent has been put in main documentation about changed behaviour of module loading in mod_python.publisher. In the main the changes are for the better and things will actually work better. There is one case though where existing code can stop working. This is where code makes an assumption that global data within a publisher module is preserved across module reloads. I am not saying this change is wrong, but it perhaps may need to be clearly explained in documentation else we will get lots of questions about it no doubt, with the common answer probably being that they shouldn't be storing such persistent module data in a module that can be reloaded anyway. Thats all for now until I can get some real time to look at it. :-) Graham On 18/08/2005, at 4:03 AM, Jim Gallacher wrote: Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim
[PATCH] mod_disk_cache, only cache req_hdrs if vary
Some more low hanging fruit.
This moves varray into disk_cache_object_t and only writes out/reads in
the request headers if the response actually varied.
An easy 4-7% performance boost in my tests for responses with no vary.
--
Brian Akins
Lead Systems Engineer
CNN Internet Technologies
diff -ru httpd-trunk.orig/modules/cache/mod_disk_cache.c
httpd-trunk.new/modules/cache/mod_disk_cache.c
--- httpd-trunk.orig/modules/cache/mod_disk_cache.c 2005-08-09
11:51:09.473251000 -0400
+++ httpd-trunk.new/modules/cache/mod_disk_cache.c 2005-08-18
09:23:34.389106747 -0400
@@ -75,6 +75,7 @@
*/
typedef struct disk_cache_object {
const char *root;/* the location of the cache directory */
+apr_array_header_t *varray;
char *tempfile;/* temp file tohold the content */
const char *prefix;
const char *datafile;/* name of file where the data will go */
@@ -425,7 +426,6 @@
apr_file_read_full(dobj-hfd, format, len, len);
if (format == VARY_FORMAT_VERSION) {
-apr_array_header_t* varray;
apr_time_t expire;
len = sizeof(expire);
@@ -435,8 +435,8 @@
return DECLINED;
}
-varray = apr_array_make(r-pool, 5, sizeof(char*));
-rc = read_array(r, varray, dobj-hfd);
+dobj-varray = apr_array_make(r-pool, 5, sizeof(char*));
+rc = read_array(r, dobj-varray, dobj-hfd);
if (rc != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_ERR, rc, r-server,
disk_cache: Cannot parse vary header file: %s,
@@ -445,7 +445,7 @@
}
apr_file_close(dobj-hfd);
-nkey = regen_key(r-pool, r-headers_in, varray, key);
+nkey = regen_key(r-pool, r-headers_in, dobj-varray, key);
dobj-hashfile = NULL;
dobj-prefix = dobj-hdrsfile;
@@ -690,7 +690,9 @@
/* Call routine to read the header lines/status line */
read_table(h, r, h-resp_hdrs, dobj-hfd);
-read_table(h, r, h-req_hdrs, dobj-hfd);
+if(dobj-varray dobj-varray-nelts) {
+read_table(h, r, h-req_hdrs, dobj-hfd);
+}
apr_file_close(dobj-hfd);
@@ -767,7 +769,6 @@
tmp = apr_table_get(r-headers_out, Vary);
if (tmp) {
-apr_array_header_t* varray;
apr_uint32_t format = VARY_FORMAT_VERSION;
mkdir_structure(conf, dobj-hdrsfile, r-pool);
@@ -786,10 +787,10 @@
amt = sizeof(info-expire);
apr_file_write(dobj-tfd, info-expire, amt);
-varray = apr_array_make(r-pool, 6, sizeof(char*));
-tokens_to_array(r-pool, tmp, varray);
+dobj-varray = apr_array_make(r-pool, 6, sizeof(char*));
+tokens_to_array(r-pool, tmp, dobj-varray);
-store_array(dobj-tfd, varray);
+store_array(dobj-tfd, dobj-varray);
apr_file_close(dobj-tfd);
@@ -804,7 +805,7 @@
}
dobj-tempfile = apr_pstrcat(r-pool, conf-cache_root,
AP_TEMPFILE, NULL);
-tmp = regen_key(r-pool, r-headers_in, varray, dobj-name);
+tmp = regen_key(r-pool, r-headers_in, dobj-varray, dobj-name);
dobj-prefix = dobj-hdrsfile;
dobj-hashfile = NULL;
dobj-datafile = data_file(r-pool, conf, dobj, tmp);
@@ -866,13 +867,18 @@
/* Parse the vary header and dump those fields from the headers_in. */
/* FIXME: Make call to the same thing cache_select_url calls to crack
Vary. */
if (r-headers_in) {
-apr_table_t *headers_in;
+/*we only need to write these out if we will vary*/
+/*TODO: only cache the headers contained in varray*/
+if(dobj-varray dobj-varray-nelts) {
+apr_table_t *headers_in;
+
+headers_in = ap_cache_cacheable_hdrs_out(r-pool, r-headers_in,
+ r-server);
-headers_in = ap_cache_cacheable_hdrs_out(r-pool, r-headers_in,
- r-server);
-rv = store_table(dobj-hfd, headers_in);
-if (rv != APR_SUCCESS) {
-return rv;
+rv = store_table(dobj-hfd, headers_in);
+if (rv != APR_SUCCESS) {
+return rv;
+}
}
}
Re: flex [was mod_python 3.2.0-BETA available for testing]
OK, here is the flex scoop - as the the docs point out, anything before 2.5.31 is not reentrant and I think even uses a slightly different interface so older flex won't even process the psp_parser.l file correctly. Looking at Fedora Core 4, it still has flex 2.5.4a. (Note that 2.5.31 2.5.4 because 31 4 - I for a while had trouble seeing that for some reason), so the new flex is still not commonplace. So until reentrant flex becomes commonplace, the solution was to include a pre-parsed psp_parser.c so that you woudln't need flex at all to compile mod_python. Looks like this still should be the case. The ./configure should just print a warning that if flex is not found or too old, should you need to rebuild psp_parser.c you will need to get the right version of flex. Grisha On Thu, 18 Aug 2005, Jim Gallacher wrote: Graham Dumpleton wrote: A few comments: 1. If you have an older version of flex than that expected, it gives message: checking flex version... configure: WARNING: Flex version 2.5.31 or greater is required. The one you have seems to be 2.5.4. Use --with-flex to specify another. There is nothing in the README describing why such a requirement exists. The only veiled comment is in the configure script itself where it says: # check for correct flex version # requires flex 2.5.31 for reentrant support A more prominent explanation is required in the README with configure outputing with the message above a pointer to consult the README for further details. What also isn't at all clear, is whether the flex warning is relevant anyway if pre-generated code for the lexical analyser is provided in the tar ball and that is used. This is explained in the html docs, but didn't make it into the README. I wasn't sure if the WARNING was the right idea, See doc-html/inst-configure.html: Attempts to locate flex and determine its version. If flex cannot be found in your PATH configure will fail. If the wrong version is found configure will generate a warning. You can generally ignore this warning unless you need to re-create src/psp_parser.c. The parser used by psp (See 4.9) is written in C generated using flex. This requires a reentrant version of flex which at this time is 2.5.31. Most platforms however ship with version 2.5.4 which is not suitable, so a pre-generated copy of psp_parser.c is included with the source. If you do need to compile src/psp_parser.c you must get the correct flex version. If the first flex binary in the path is not suitable or not the one desired you can specify an alternative location with the --with-flex option, e.g: $ ./configure --with-flex=/usr/local/bin/flex I'll copy this into the README as well. In 3.1.x the path to flex was hard coded to /usr/local/bin, which caused compilation to fail and required the user to edit the Makefile manually. Other than the detection of the flex path, the behaviour is the same as 3.1.x. The previous warning about needing the correct flex version was just a comment in the Makefile. Any suggestions on how to make this clearer are welcome. Thus, what is the real story, am I going to have a problem or not if I have an older version of flex? No, see html docs snippet above. No time to look at the rest. Will comment tonight. Jim 2. Publisher still doesn't check special extensions supplied by the AddHandler or PythonHandler directives. Ie., http://issues.apache.org/jira/browse/MODPYTHON-72 This may mean that code that previously worked no longer works. Ie., this feature was in there before. Only caveat is that the feature may not have worked properly before anyway, so existing code may not be affected after all. See: http://issues.apache.org/jira/browse/MODPYTHON-22 Bit unsure on this one. 3. Only other issue at the moment I can think of is whether or not anything prominent has been put in main documentation about changed behaviour of module loading in mod_python.publisher. In the main the changes are for the better and things will actually work better. There is one case though where existing code can stop working. This is where code makes an assumption that global data within a publisher module is preserved across module reloads. I am not saying this change is wrong, but it perhaps may need to be clearly explained in documentation else we will get lots of questions about it no doubt, with the common answer probably being that they shouldn't be storing such persistent module data in a module that can be reloaded anyway. Thats all for now until I can get some real time to look at it. :-) Graham On 18/08/2005, at 4:03 AM, Jim Gallacher wrote: Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me
Re: [PATCH] mod_disk_cache: speed up read_table
On Thu, Aug 18, 2005 at 09:22:13AM -0400, Brian Akins wrote:
Thanks to all who reminded me what a dumb-a## I am this morning...
I forgot the patch. Here it is.
Cool.
diff -ru httpd-trunk.orig/modules/cache/mod_disk_cache.c
httpd-trunk.new2/modules/cache/mod_disk_cache.c
--- httpd-trunk.orig/modules/cache/mod_disk_cache.c 2005-08-09
11:51:09.473251000 -0400
+++ httpd-trunk.new2/modules/cache/mod_disk_cache.c 2005-08-18
08:52:33.346214476 -0400
@@ -51,7 +51,7 @@
*/
#define VARY_FORMAT_VERSION 3
-#define DISK_FORMAT_VERSION 4
+#define DISK_FORMAT_VERSION 5
There should be a mod_disk_cache.h, which htcacheclean can include. It
only need contain these define's and the disk_cache_info_t structure,
but it prevents the silly out-of-sync problems that keep cropping up :)
@@ -414,7 +414,7 @@
dobj-prefix = NULL;
dobj-hdrsfile = header_file(r-pool, conf, dobj, key);
-flags = APR_READ|APR_BINARY|APR_BUFFERED;
+flags = APR_READ|APR_BINARY/*|APR_BUFFERED*/;
I can understand why this is faster, I'm guessing that you've enough RAM
that you're retrieving files from the vmcache and that the higher-layer
buffering is just overhead. This is probably going to be the majority
case, but would do with testing.
+/*read length first*/
+len = sizeof(length);
+
+if((rv = apr_file_read_full(file, length, len, len)) != APR_SUCCESS) {
+return rv;
+}
+
+buffer = apr_palloc(r-pool, length);
Embedding more integers like this in the file doesn't make much sense to
me, especially when using unbuffered IO, the read() seems like
unneccessary pain.
It's not nearly as neat a patch, but including the lengths of the tables
in the cache_info_t object is doable, and in fact would allow for
decreasing the number of read()'s to just 3. One for the format, one
more for the cache_info_t, and one more the tables. This would probably
produce another similar speed improvement.
I'll add that to what I'm tinkering on.
-#if APR_CHARSET_EBCDIC
Well that won't fly ;-)
--
Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Re: [PATCH] mod_disk_cache: speed up read_table
Colm MacCarthaigh wrote: I can understand why this is faster, I'm guessing that you've enough RAM that you're retrieving files from the vmcache and that the higher-layer buffering is just overhead. This is probably going to be the majority case, but would do with testing. Not to heap more configuration into it, but this could also be configurable. It's not nearly as neat a patch, but including the lengths of the tables in the cache_info_t object is doable, and in fact would allow for decreasing the number of read()'s to just 3. One for the format, one more for the cache_info_t, and one more the tables. This would probably produce another similar speed improvement. That's what I wanted to do, but it was messy. I would like to have the length of resp_hdrs, req_hdrs, and the data (would save a stat...) in the disk_cache_info_t, but that is a much more major thing from what I can tell. Basically would have to rearrange store_headers completely... -#if APR_CHARSET_EBCDIC Well that won't fly ;-) I know, I know :) I just wanted to get the idea out there without having to muck with that. Obviously the final patch should consider EBDCID as well. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: [PATCH] mod_disk_cache: speed up read_table
On Thu, Aug 18, 2005 at 10:43:41AM -0400, Brian Akins wrote: I can understand why this is faster, I'm guessing that you've enough RAM that you're retrieving files from the vmcache and that the higher-layer buffering is just overhead. This is probably going to be the majority case, but would do with testing. Not to heap more configuration into it, but this could also be configurable. Thinking about it, everything is going into memory anyway, so why not just stat() the file and read it all in in one go? We know it's never going to be that big anyway. This completely minimises the number of read()'s, which is a big deal and if memory management is an issue, we can use a subpool for the buffer. If we try really hard, a table_serialize() and table_unserialize() could just cut up the strings and user pointer-manipulation to escape even the need for a subpool destory. It's not nearly as neat a patch, but including the lengths of the tables in the cache_info_t object is doable, and in fact would allow for decreasing the number of read()'s to just 3. One for the format, one more for the cache_info_t, and one more the tables. This would probably produce another similar speed improvement. That's what I wanted to do, but it was messy. I would like to have the length of resp_hdrs, req_hdrs, and the data (would save a stat...) in the disk_cache_info_t, but that is a much more major thing from what I can tell. Basically would have to rearrange store_headers completely... Yep, but there are definite speed-up's to be had. I'm going to canibalise your patch and try some of the above anyway. -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Apache 2.1 on Win32 nightmare
Hi all, as we discussed more than once here on the list the latest sources require that one has LDAP 3.0 support in the platform SDK. I was not able to get that ldap 3.0 piece to fit into the platform sdk for MSVC6, so I'm for longer no more able to build 2.1 or later self. So I asked another one for that who kindly build a version for me Now when I wanted to start that beast on W2K SP4 it complains about an error in the w32ldap.dll! So seems even W2K SP4 is not sufficient enough to run latest Apache 2.x! Ok, I went to my XP box, and wanted to give it a try there, but hey! the next dependency which I didnt get on the W2K box cause there I installed the .NET stuff for testing: MSVCR71.dll missing! The last 2.1 build (before LDAP went into APR) I compiled did just start, and now I have to update and patch my box! I think this will certainly be the next thing which will the users hold up from updating to 2.2 when it comes out I'm now a bit in doubt if it was such a good idea to have the LDAP stuff moved into APR; at least it would be better if we could have two additional targets in Win32: - build with the CLDAP SDK from Novell (freely available) - build without LDAP support at all another option is probably to support LDAP 2.0 again so that it builds with MSVC6. thanks, Guenter.
Re: [PATCH] mod_disk_cache: speed up read_table
Colm MacCarthaigh wrote: Thinking about it, everything is going into memory anyway, so why not just stat() the file and read it all in in one go? Yes! Brilliant. We need all that data anyway. so: open file. length = get_info read_full(file, length). manipulate pointers for table (apr_table_addn) If we try really hard, a table_serialize() and table_unserialize() could just cut up the strings and user pointer-manipulation to escape even the need for a subpool destory. That's basically what I was doing. Yep, but there are definite speed-up's to be had. I'm going to canibalise your patch and try some of the above anyway. Cool. I'd be willing to help. On a side note, maybe interesting: what if the whole file was just mmaped? (use '\0' for delimiter in tables). This would allow every worker referencing an object to reference the same memory. I did some testing with our caching module and it was not very efficient, but it is structure differently. Should be fairly easy to try: replace the alloc and read with a mmap. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: [PATCH] mod_disk_cache: speed up read_table
On Thu, Aug 18, 2005 at 12:11:56PM -0400, Brian Akins wrote: Yep, but there are definite speed-up's to be had. I'm going to canibalise your patch and try some of the above anyway. Cool. I'd be willing to help. I'll be putting on-line all of my TODO's and patches-in-progress shortly, I have some reading and non-code writing to do first, but then back to being more productive ;-) On a side note, maybe interesting: what if the whole file was just mmaped? (use '\0' for delimiter in tables). This would allow every worker referencing an object to reference the same memory. Do you mean as in using MAP_SHARED? I could see that being a race-condition nightmare. For example cache_info_t's pointers to the header tables would have to be local to the process, but the memory for the cache_info_t itself wouldn't be. Definitely to be avoided! Regular MMap, as long as it honoured EnableMMap would be fine though. -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Re: [PATCH] mod_disk_cache: speed up read_table
Colm MacCarthaigh wrote: Do you mean as in using MAP_SHARED? I could see that being a race-condition nightmare. For example cache_info_t's pointers to the header tables would have to be local to the process, but the memory for the cache_info_t itself wouldn't be. Definitely to be avoided! True. your comment about regular mmap are also true -- it would be interesting to test. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: [PATCH] mod_disk_cache: speed up read_table
On 8/18/05, Brian Akins [EMAIL PROTECTED] wrote: I didn't mess with the EBCDIC stuff, so alot of the patch is just where I commented all that out. The EBCDIC stuff came here: http://svn.apache.org/viewcvs.cgi?rev=105236view=rev Apparently Justin needed logic similar to ap_scan_script_header_err() and extracted a lot of that code to add here. ap_scan_script_header_err() on an EBCDIC platform is set up to allow a CGI script the capability to write a response header in ASCII (giant shrug). Is there a way to get response header written to the cache in ASCII? If not, the code can be scrapped. Maybe the folks who tweaked the code later to actually compile on EBCDIC considered that, maybe not. It just seems really odd to me. (Caveat: cache-ignorant; never used mod_*_cache on EBCDIC)
Re: [PATCH] mod_disk_cache: speed up read_table
--On August 18, 2005 3:59:05 PM +0100 Colm MacCarthaigh [EMAIL PROTECTED] wrote: Thinking about it, everything is going into memory anyway, so why not just stat() the file and read it all in in one go? We know it's never going to be that big anyway. This completely minimises the number of read()'s, which is a big deal and if memory management is an issue, we can use a subpool for the buffer. But, that's why we do the buffered read. It almost always ends up being one read() for me as the header file is always less than 4k. I don't believe we should necessarily optimize based on the results of one platform when we know its going to kill us everywhere else. -- justin
Re: mod_python 3.2.0-BETA available for testing
Ron Reisor wrote: Hello, I ran into a problem with the loader on MacOSX. MaxOSX 1.4.2 python 2.4.1 apache 2.0.54 The loader seems to not like the -undefined suppress arguments in the final load. I modified dist/setup.py by removing the two -undefined suppress and ran configure and make again and the new mod_python does build and seems to work ok. The bit of code which includes -undefined suppress came from the patch Graham submitted for MODPYTHON-65. See http://issues.apache.org/jira/browse/MODPYTHON-65 for details. Not being a Mac person I can't comment further. Feel free to talk among yourselves. ;) Jim
Re: [PATCH] mod_disk_cache deterministic tempfiles
--On August 18, 2005 9:24:57 AM +0100 Colm MacCarthaigh [EMAIL PROTECTED]
wrote:
On Wed, Aug 17, 2005 at 03:17:41PM -0700, Justin Erenkrantz wrote:
Content definitely should not be served from the cache after it has
expired imo. However I think an approach like;
if((now + interval) expired) {
if(!stat(tmpfile)) {
update_cache_from_backend();
}
}
ie revalidate the cache content after N-seconds before it is due to be
expired would have the same effect, but avoid serving stale content.
Does that make sense?
Um, isn't that what we already do? -- justin
No :-) Right now, once the cache entity stops being considered fresh,
each worker that gets a request will fetch it, store it to their own
tmpfile and race to be first to rename it.
With the above approach, there could be some kind of user-configurable
interval before expiry, say 10 minutes in this example. So, the first
request that happens within that 10 minute interval would re-validate
the cache entity.
If it doesn't need to be fetched and gets a not-modified, we just
touch the cache entity and go on our merry way.
If it does need to be fetched then a deterministic tmpfile ensures that
no other worker bothers to fetch it at the same time, they can serve
from the cache in the meantime - which still hasn't expired.
If after the 10 minutes the content still isn't there, the user should
have set a better value for interval, and we've no choice but to start
fetching the content per-request until something writes a cache file.
Okay. I see what you mean now.
If the interval is configurable via a directive, then yes that makes sense.
This could be done independently of deterministic tempfiles. (I hope this
means you volunteer to write the patch!)
However, using deterministic tempfiles means that there's a possibility of a
'deadlock' - in that a response will never be updated because of a stuck or
dead process. I've not seen a feasible strategy to resolve this issue.
In the case of fetching before expiration, the 'lock' can be 'advisory' -
after expiration, it can fall back to the current case. -- justin
Re: mod_cache recall_body
--On August 18, 2005 9:55:57 AM -0400 Brian Akins [EMAIL PROTECTED] wrote: Why do we pass request_rec to recall_headers, but not to recall_body. We also always pass r-pool. How horrible would it be if the prototype was changed to: apr_status_t (*recall_body) (cache_handle_t *h, request_rec *r,apr_bucket_brigade *bb); I wanted to get r in recall_body so mod_disk_cache could honor EnableMMap. open_entity can already be modified to honor EnableSendfile fairly easily. I'd be fine with it matching store_body's prototype. (Which is what you have.) -- justin
Re: [PATCH] mod_disk_cache deterministic tempfiles
On Thu, Aug 18, 2005 at 10:07:29AM -0700, Justin Erenkrantz wrote: Okay. I see what you mean now. If the interval is configurable via a directive, then yes that makes sense. This could be done independently of deterministic tempfiles. It could, though it would require using a seperate locking mechanism. (I hope this means you volunteer to write the patch!) Unless someone beats me too it. However, using deterministic tempfiles means that there's a possibility of a 'deadlock' - in that a response will never be updated because of a stuck or dead process. I've not seen a feasible strategy to resolve this issue. Dead process is solveable with IPC, the existing locking schemes should be enough for this. The hard problem I think is when a backend has stalled. Can't think of an easy fix for that case. -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Re: [PATCH] mod_disk_cache: speed up read_table
Justin Erenkrantz wrote: But, that's why we do the buffered read. It almost always ends up being one read() for me as the header file is always less than 4k. True. I don't believe we should necessarily optimize based on the results of one platform when we know its going to kill us everywhere else. -- justin The optimizations that Colm is talking about should be helpful everywhere. Also, that's why we test to make sure it does kill us. Also, I don't think we know it will kill us. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: [PATCH] mod_disk_cache: speed up read_table
--On August 18, 2005 1:34:56 PM -0400 Brian Akins [EMAIL PROTECTED] wrote: The optimizations that Colm is talking about should be helpful everywhere. Also, that's why we test to make sure it does kill us. Also, I don't think we know it will kill us. None of the code in mod_disk_cache used buffering before I turned it on. It gave significant speedups in my performance tests by reducing the syscall overhead. I also had identified and fixed some bugs in apr's buffering code to go along with these speedups that went out in APR 1.0.1. So, unless the profiles have changed (unlikely), removing buffering is a bad idea. -- justin
[PATCH] Re: mod_cache recall_body
Justin Erenkrantz wrote:
I'd be fine with it matching store_body's prototype. (Which is what you
have.) -- justin
Here's a patch that does this.
--
Brian Akins
Lead Systems Engineer
CNN Internet Technologies
diff -ru httpd-trunk.orig/modules/cache/mod_cache.c
httpd-trunk.new2/modules/cache/mod_cache.c
--- httpd-trunk.orig/modules/cache/mod_cache.c 2005-08-09 11:51:09.471251000
-0400
+++ httpd-trunk.new2/modules/cache/mod_cache.c 2005-08-18 13:41:31.414487623
-0400
@@ -206,7 +206,7 @@
r-status = cache-handle-cache_obj-info.status;
/* recall_headers() was called in cache_select_url() */
-cache-provider-recall_body(cache-handle, r-pool, bb);
+cache-provider-recall_body(cache-handle, r, bb);
/* This filter is done once it has served up its content */
ap_remove_output_filter(f);
@@ -695,7 +695,7 @@
APR_BRIGADE_INSERT_TAIL(bb, bkt);
}
else {
-cache-provider-recall_body(cache-handle, r-pool, bb);
+cache-provider-recall_body(cache-handle, r, bb);
}
cache-block_response = 1;
diff -ru httpd-trunk.orig/modules/cache/mod_cache.h
httpd-trunk.new2/modules/cache/mod_cache.h
--- httpd-trunk.orig/modules/cache/mod_cache.h 2005-07-13 15:23:03.882379000
-0400
+++ httpd-trunk.new2/modules/cache/mod_cache.h 2005-08-18 13:41:04.332966989
-0400
@@ -192,7 +192,7 @@
apr_status_t (*store_headers)(cache_handle_t *h, request_rec *r,
cache_info *i);
apr_status_t (*store_body)(cache_handle_t *h, request_rec *r,
apr_bucket_brigade *b);
apr_status_t (*recall_headers) (cache_handle_t *h, request_rec *r);
-apr_status_t (*recall_body) (cache_handle_t *h, apr_pool_t *p,
apr_bucket_brigade *bb);
+apr_status_t (*recall_body)(cache_handle_t *h, request_rec *r,
apr_bucket_brigade *b);
int (*create_entity) (cache_handle_t *h, request_rec *r,
const char *urlkey, apr_off_t len);
int (*open_entity) (cache_handle_t *h, request_rec *r,
diff -ru httpd-trunk.orig/modules/cache/mod_disk_cache.c
httpd-trunk.new2/modules/cache/mod_disk_cache.c
--- httpd-trunk.orig/modules/cache/mod_disk_cache.c 2005-08-09
11:51:09.473251000 -0400
+++ httpd-trunk.new2/modules/cache/mod_disk_cache.c 2005-08-18
13:42:57.209296497 -0400
@@ -116,7 +116,7 @@
static apr_status_t store_headers(cache_handle_t *h, request_rec *r,
cache_info *i);
static apr_status_t store_body(cache_handle_t *h, request_rec *r,
apr_bucket_brigade *b);
static apr_status_t recall_headers(cache_handle_t *h, request_rec *r);
-static apr_status_t recall_body(cache_handle_t *h, apr_pool_t *p,
apr_bucket_brigade *bb);
+static apr_status_t recall_body(cache_handle_t *h, request_rec *r,
apr_bucket_brigade *bb);
static apr_status_t read_array(request_rec *r, apr_array_header_t* arr,
apr_file_t *file);
@@ -699,12 +699,12 @@
return APR_SUCCESS;
}
-static apr_status_t recall_body(cache_handle_t *h, apr_pool_t *p,
apr_bucket_brigade *bb)
+static apr_status_t recall_body(cache_handle_t *h, request_rec *r,
apr_bucket_brigade *bb)
{
apr_bucket *e;
disk_cache_object_t *dobj = (disk_cache_object_t*) h-cache_obj-vobj;
-e = apr_bucket_file_create(dobj-fd, 0, (apr_size_t) dobj-file_size, p,
+e = apr_bucket_file_create(dobj-fd, 0, (apr_size_t) dobj-file_size,
r-pool,
bb-bucket_alloc);
APR_BRIGADE_INSERT_HEAD(bb, e);
e = apr_bucket_eos_create(bb-bucket_alloc);
diff -ru httpd-trunk.orig/modules/cache/mod_mem_cache.c
httpd-trunk.new2/modules/cache/mod_mem_cache.c
--- httpd-trunk.orig/modules/cache/mod_mem_cache.c 2005-07-13
15:23:03.871381000 -0400
+++ httpd-trunk.new2/modules/cache/mod_mem_cache.c 2005-08-18
13:42:11.626836265 -0400
@@ -642,7 +642,7 @@
return rc;
}
-static apr_status_t recall_body(cache_handle_t *h, apr_pool_t *p,
apr_bucket_brigade *bb)
+static apr_status_t recall_body(cache_handle_t *h, request_rec *r,
apr_bucket_brigade *bb)
{
apr_bucket *b;
mem_cache_object_t *mobj = (mem_cache_object_t*) h-cache_obj-vobj;
@@ -650,8 +650,8 @@
if (mobj-type == CACHE_TYPE_FILE) {
/* CACHE_TYPE_FILE */
apr_file_t *file;
-apr_os_file_put(file, mobj-fd, mobj-flags, p);
-b = apr_bucket_file_create(file, 0, mobj-m_len, p, bb-bucket_alloc);
+apr_os_file_put(file, mobj-fd, mobj-flags, r-pool);
+b = apr_bucket_file_create(file, 0, mobj-m_len, r-pool,
bb-bucket_alloc);
}
else {
/* CACHE_TYPE_HEAP */
Re: [PATCH] mod_disk_cache: speed up read_table
Justin Erenkrantz wrote: None of the code in mod_disk_cache used buffering before I turned it on. It gave significant speedups in my performance tests by reducing the syscall overhead. I also had identified and fixed some bugs in apr's buffering code to go along with these speedups that went out in APR 1.0.1. So, unless the profiles have changed (unlikely), removing buffering is a bad idea. -- justin Colm's idea would eliminate the need for the buffering as it would read the entire headers file in at one time and manipulate the memory in place. In this case buffering would actually hurt performance as it would have lots of extra memcpy's. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: [PATCH] mod_disk_cache: speed up read_table
On Thu, Aug 18, 2005 at 10:43:45AM -0700, Justin Erenkrantz wrote: None of the code in mod_disk_cache used buffering before I turned it on. It gave significant speedups in my performance tests by reducing the syscall overhead. I also had identified and fixed some bugs in apr's buffering code to go along with these speedups that went out in APR 1.0.1. So, unless the profiles have changed (unlikely), removing buffering is a bad idea. -- justin The optimisations wouldn't be removing buffering, they'd be using a different kind of buffering :-) If MMap is not available/enabled we can fail back to a buffered APR read. -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Re: [PATCH] mod_disk_cache: speed up read_table
Colm MacCarthaigh wrote: The optimisations wouldn't be removing buffering, they'd be using a different kind of buffering :-) If MMap is not available/enabled we can fail back to a buffered APR read. So it's evolved even further... I would be interested to see in a variety of cases which one is actually faster: -normal buffered apr read -alloc entire buffer, read_full into it. -mmap file. Colm, are you already writing code? If not, I may give it a swing... -- Brian Akins Lead Systems Engineer CNN Internet Technologies
input filter problems
Hi there, I've spent the last few days trying to get an input filter to work in mod_python, in the context of requests that are being reverse proxied to an app-server by mod_proxy. I've also tried mod_rewrite. The input filter works fine if mod_proxy or mod_rewrite are *not* in play and things are just posted into the void. When posting to an appserver, the symptoms for mod_proxy and mod_rewrite appear to be very similar. My apologies in advance for the information dump... I hope someone can at least give me any clue as to where this problem may lay or what further actions to pursue. My 'fix' to mod_python might also be worth consideration, though I have no idea why it works. I ran into a host of problems that I think have something to do with the underlying apache, but may also have something to do with mod_python itself. Symptoms on Apache 2.0.54, using mod_python 3.1.3 - * input filters appear to be unreliable in the face of changing data. (writing out something else than goes in) Apache hangs or the filter gets called infinite times in a single request (subrequests appear to be not in play though, at least I can't detect them using req.main). * even when data is not changed, filtering sometimes hangs. * the infinite calling thing can be worked around by disabling the filter after one is done. * I can suppress the infinite call behavior using .disable(), but then the system hangs instead. Apache 2.0.54, latest svn mod_python * same story as before. Latest apache 2.0.x svn as well as svn mod_python - * Infinite calls do occur, but.. * Hangs still occur, but.. * Hangs and infinite calls disappear and everything works as expected (except for message in error log) if an exception is raised inside the filter code! Tracking this down to lib/python/mod_python/apache.py, in FilterDispatch() there's the following section: if object: # call the object if config.has_key(PythonEnablePdb): pdb.runcall(object, filter) else: object(filter) # always flush the filter. without a FLUSH or EOS bucket, # the content is never written to the network. # XXX an alternative is to tell the user to flush() always filter.flush() The hang/calling behavior seems to be triggered when filter.flush() is called. If instead I put in a line: return OK before 'filter.flush()' is ever reached (as is the case when the exception is raised), everything appears to work. Unfortunately the same trick doesn't work on Apache 2.0.54... (even if I use the disable() trick). Does this mean that filter.flush() is buggy when mod_proxy or mod_rewrite are in effect? I don't know, but I thought I'd report it here. Perhaps it's also a problem to do with input filters in particular? The comment talks about needing to flush to make sure things are sent to the network, but that comment makes more sense for output filters than input filters (even though mod_proxy in turn sends stuff to the network again). Apache change hunt -- Trying to figure out what in Apache itself might've changed, I found that in the bleeding-edge Apache 2.0 trunk there is a patch that seems to have to do something with this. More on this apache patch is here: http://svn.apache.org/viewcvs.cgi/httpd/httpd/branches/2.0.x/CHANGES?rev=233302view=markup in particular: *) proxy HTTP: Rework the handling of request bodies to handle chunked input and input filters which modify content length, and avoid spooling arbitrary-sized request bodies in memory. PR 15859. [Jeff Trawick] Though it also seems to work when mod_rewrite is used instead of mod_proxy, so perhaps this fix isn't it and it's something deeper inside Apache that changed... ... Feel free to ask me more questions; I can do more testing if you like. I can also post the test code if people are interested. Of course ideally I'd make all of this work on a released version of Apache with a released version of mod_python, but I'll take any hint I can get. :) Regards, Martijn
Re: [PATCH] mod_disk_cache deterministic tempfiles
On Thu, Aug 18, 2005 at 01:39:18PM -0400, Brian Akins wrote: Dead process is solveable with IPC, the existing locking schemes should be enough for this. The hard problem I think is when a backend has stalled. Can't think of an easy fix for that case. When you stat the temp file, if its older than x seconds, delete it. x is configurable. It's that simple. I don't think that it is. The write()'s to the actual file arn't going to occur very often because both reading from the CGI/proxy-backend is buffered as is writing the file to disk. (and they should be, or the general case would become much slower). So mtime not being recent is no-indication of death, it could easily be a trickling download. And then the same thing is going to happen to the current request, and so on and so on and you're going to end up not caching the very requests that would benefit the most from being cached. -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Re: [PATCH] mod_disk_cache deterministic tempfiles
Colm MacCarthaigh wrote: So mtime not being recent is no-indication of death, it could easily be a trickling download. True. But, if the files mtime has not changed in 120 seconds (for example) the download is probably hung? -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: [PATCH] mod_disk_cache deterministic tempfiles
On Thu, Aug 18, 2005 at 02:00:52PM -0400, Brian Akins wrote: Colm MacCarthaigh wrote: So mtime not being recent is no-indication of death, it could easily be a trickling download. True. But, if the files mtime has not changed in 120 seconds (for example) the download is probably hung? 120 second stalls arn't uncommon, and there are plenty of overloaded servers and terrible CGI's that have those kind of response times. A major use of mod_cache to solve just that problem, but any approach - no matter what number of seconds you pick - will always introduce inefficiency. If you pick a value that is too low, you'll never cache slow-to-serve content. If you pick a value that is too high, you'll end up sending a lot of requests to the (already slow) backend. There might be a solution in using the scoreboard. -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
canonical port
We have servers that listen on a ports other than 80 which our load balancers communicate with. The client hits port 80 on the load balancer. I hacked up a quick module to try to emulate the old Port config directive so that redirects go to the canonical port. It does this by changing r-parsed_uri.port, based on what ap_get_server_port does This normally works. Now we have a server with a large number of vhosts and it appears to work for some and not for others. That is my current problem. Surely I'm not the only one who has a similar setup. Is there a config thing I'm just missing or do we need to bring back Port? -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: canonical port
Brian,
On Aug 18, 2005, at 11:45 AM, Brian Akins wrote:
We have servers that listen on a ports other than 80 which our load
balancers communicate with. The client hits port 80 on the load
balancer. I hacked up a quick module to try to emulate the old
Port config directive so that redirects go to the canonical port.
It does this by changing r-parsed_uri.port, based on what
ap_get_server_port does
I'm in much the same boat. My module substitutes the outside
listening port in the post_config handler:
static int bn_post_config(apr_pool_t *pconf, apr_pool_t *plog,
apr_pool_t *ptemp, server_rec *s)
{
bn_cfg *cfg;
server_rec *swalk;
for (swalk = s; swalk; swalk = swalk-next) {
cfg = BNSrvConfig(swalk);
if (cfg-enabled) {
swalk-port = cfg-listenport;
}
}
return OK;
}
This has been lightly tested and is based on ap_get_server_port
looking in that particular place first. So far no horrible side
effects have been observed.
Do you do more/different things?
S.
--
[EMAIL PROTECTED] http://www.temme.net/sander/
PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF
Re: canonical port
Sander Temme wrote:
Do you do more/different things?
This is what I do in translate_name:
if((conf = ap_get_module_config(r-server-module_config,
port_module)) != NULL) {
if(conf-port != NOCONFIG) {
/*looking at server/core.c, ap_get_server_port This looks
to be the best place
* UseCanonicalName MUST be off, or this is ignored*/
r-parsed_uri.port = conf-port;
r-parsed_uri.port_str = conf-port_str;
}
}
This is a fixed version which appears to have fixed my original breakage.
--
Brian Akins
Lead Systems Engineer
CNN Internet Technologies
[STATUS] (httpd-2.0) Wed Aug 17 23:45:26 2005
APACHE 2.0 STATUS: -*-text-*- Last modified at [$Date: 2005-08-11 05:57:40 -0400 (Thu, 11 Aug 2005) $] The current version of this file can be found at: * http://svn.apache.org/repos/asf/httpd/httpd/branches/2.0.x/STATUS Documentation status is maintained seperately and can be found at: * docs/STATUS in this source tree, or * http://svn.apache.org/repos/asf/httpd/httpd/branches/2.0.x/docs/STATUS Consult the following STATUS files for information on related projects: * http://svn.apache.org/repos/asf/apr/apr/branches/0.9.x/STATUS * http://svn.apache.org/repos/asf/apr/apr-util/branches/0.9.x/STATUS Consult the trunk/ for all new development and documentation efforts: * http://svn.apache.org/repos/asf/httpd/httpd/trunk/STATUS * http://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/STATUS Release history: 2.0.55 : in development 2.0.54 : released April 17, 2005 as GA. 2.0.53 : released February 7, 2005 as GA. 2.0.52 : released September 28, 2004 as GA. 2.0.51 : released September 15, 2004 as GA. 2.0.50 : released June 30, 2004 as GA. 2.0.49 : released March 19, 2004 as GA. 2.0.48 : released October 29, 2003 as GA. 2.0.47 : released July 09, 2003 as GA. 2.0.46 : released May 28, 2003 as GA. 2.0.45 : released April 1, 2003 as GA. 2.0.44 : released January 20, 2003 as GA. 2.0.43 : released October 3, 2002 as GA. 2.0.42 : released September 24, 2002 as GA. 2.0.41 : rolled September 16, 2002. not released. 2.0.40 : released August 9, 2002 as GA. 2.0.39 : released June 17, 2002 as GA. 2.0.38 : rolled June 16, 2002. not released. 2.0.37 : rolled June 11, 2002. not released. 2.0.36 : released May 6, 2002 as GA. 2.0.35 : released April 5, 2002 as GA. 2.0.34 : tagged March 26, 2002. 2.0.33 : tagged March 6, 2002. not released. 2.0.32 : released Feburary 16, 2002 as beta. 2.0.31 : rolled Feburary 1, 2002. not released. 2.0.30 : tagged January 8, 2002. not rolled. 2.0.29 : tagged November 27, 2001. not rolled. 2.0.28 : released November 13, 2001 as beta. 2.0.27 : rolled November 6, 2001 2.0.26 : tagged October 16, 2001. not rolled. 2.0.25 : rolled August 29, 2001 2.0.24 : rolled August 18, 2001 2.0.23 : rolled August 9, 2001 2.0.22 : rolled July 29, 2001 2.0.21 : rolled July 20, 2001 2.0.20 : rolled July 8, 2001 2.0.19 : rolled June 27, 2001 2.0.18 : rolled May 18, 2001 2.0.17 : rolled April 17, 2001 2.0.16 : rolled April 4, 2001 2.0.15 : rolled March 21, 2001 2.0.14 : rolled March 7, 2001 2.0a9 : released December 12, 2000 2.0a8 : released November 20, 2000 2.0a7 : released October 8, 2000 2.0a6 : released August 18, 2000 2.0a5 : released August 4, 2000 2.0a4 : released June 7, 2000 2.0a3 : released April 28, 2000 2.0a2 : released March 31, 2000 2.0a1 : released March 10, 2000 Contributors looking for a mission: * Just do an egrep on TODO or XXX in the source. * Review the bug database at: http://issues.apache.org/bugzilla/ * Review the PatchAvailable bugs in the bug database: http://issues.apache.org/bugzilla/buglist.cgi?bug_status=NEWbug_status=ASSIGNEDbug_status=REOPENEDproduct=Apache+httpd-2.0keywords=PatchAvailable After testing, you can append a comment saying Reviewed and tested. * Open bugs in the bug database. CURRENT RELEASE NOTES: * Forward binary compatibility is expected of Apache 2.0.x releases, such that no MMN major number changes will occur. Such changes can only be made in the trunk. * All commits to branches/2.0.x must be reflected in SVN trunk, as well, if they apply. Logical progression is commit to trunk, get feedback and votes in STATUS, and then merge into branches/2.0.x. RELEASE SHOWSTOPPERS: * Copy the backport branch of all of the mod_proxy_http.c's request body handling security, protocol and bug fixes; by svn copy'ing the file httpd/httpd/branches/proxy-reqbody-2.0.x/modules/proxy/proxy_http.c back to httpd/branches/2.0.x/... preserving the detail of all of the individually backported changes. +1: wrowe, jim -1: For a complete history of individual unit changes, see r230703 - r230744 in http://svn.apache.org/viewcvs.cgi/httpd/httpd/branches/proxy-reqbody-2.0.x/ [...] modules/proxy/proxy_http.c?view=log Cite the specific patch with justification for each specific objection. Suggested; revert r219061 to thoroughly test this patch, as r219061 masks some underlying bugs (although it is a -good- patch in and of itself and provides additional protection to other content-handling modules). * TRACE must not have a request body per RFC2616; see the -trace.patch below for one of
[STATUS] (httpd-2.1) Wed Aug 17 23:45:30 2005
APACHE 2.1 STATUS: -*-text-*- Last modified at [$Date: 2005-06-30 16:42:43 -0400 (Thu, 30 Jun 2005) $] The current version of this file can be found at: * http://svn.apache.org/repos/asf/httpd/httpd/trunk/STATUS Documentation status is maintained seperately and can be found at: * docs/STATUS in this source tree, or * http://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/STATUS Consult the following STATUS files for information on related projects: * http://svn.apache.org/repos/asf/apr/apr/trunk/STATUS * http://svn.apache.org/repos/asf/apr/apr-util/trunk/STATUS Patches considered for backport are noted in their branches' STATUS: * http://svn.apache.org/repos/asf/httpd/httpd/branches/1.3.x/STATUS * http://svn.apache.org/repos/asf/httpd/httpd/branches/2.0.x/STATUS Release history: [NOTE that only Alpha/Beta releases occur in 2.1 development] 2.1.7 : in development 2.1.6 : Released on 6/27/2005 as alpha. 2.1.5 : Tagged on 6/17/2005. 2.1.4 : not released. 2.1.3 : Released on 2/22/2005 as alpha. 2.1.2 : Released on 12/08/2004 as alpha. 2.1.1 : Released on 11/19/2004 as alpha. 2.1.0 : not released. Contributors looking for a mission: * Just do an egrep on TODO or XXX in the source. * Review the bug database at: http://issues.apache.org/bugzilla/ * Review the PatchAvailable bugs in the bug database: http://issues.apache.org/bugzilla/buglist.cgi?bug_status=NEWbug_status=ASSIGNEDbug_status=REOPENEDproduct=Apache+httpd-2.0keywords=PatchAvailable After testing, you can append a comment saying Reviewed and tested. * Open bugs in the bug database. CURRENT RELEASE NOTES: RELEASE SHOWSTOPPERS: * Handling of non-trailing / config by non-default handler is broken http://marc.theaimsgroup.com/?l=apache-httpd-devm=105451701628081w=2 jerenkrantz asks: Why should this block a release? wsanchez agrees: this may be a change in behavior, but isn't clearly wrong, and even if so, it doesn't seem like a showstopper. * the edge connection filter cannot be removed http://marc.theaimsgroup.com/?l=apache-httpd-devm=105366252619530w=2 jerenkrantz asks: Why should this block a release? stas replies: because it requires a rewrite of the filters stack implementation (you have suggested that) and once 2.2 is released you can't do that anymore. CURRENT VOTES: * httpd-std.conf and friends a) httpd-std.conf should be tailored by install (from src or binbuild) even if user has existing httpd.conf +1: trawick, slive, gregames, ianh, Ken, wrowe, jwoolley, jim, nd, erikabele wrowe - prefer httpd.default.conf to avoid ambiguity with cvs b) tailored httpd-std.conf should be copied by install to sysconfdir/examples -0: striker c) tailored httpd-std.conf should be installed to sysconfdir/examples or manualdir/exampleconf/ +1: slive, trawick, Ken, nd (prefer the latter), erikabele +1: wsanchez (propose sysconfdir/examples/version for diffiness) d) Installing a set of default config files when upgrading a server doesn't make ANY sense at all. +1: ianh - medium/big sites don't use 'standard config' anyway, as it usually needs major customizations -1: Ken, wrowe, jwoolley, jim, nd, erikabele wrowe - diff is wonderful when comparing old/new default configs, even for customized sites that ianh mentions jim - ... assuming that the default configs have been updated with the required inline docs to explain the changes * If the parent process dies, should the remaining child processes gracefully self-terminate. Or maybe we should make it a runtime option, or have a concept of 2 parent processes (one being a hot spare). See: Message-ID: [EMAIL PROTECTED] Self-destruct: Ken, Martin, Lars Not self-destruct: BrianP, Ian, Cliff, BillS Make it runtime configurable: Aaron, jim, Justin, wrowe, rederpj, nd /* The below was a concept on *how* to handle the problem */ Have 2 parents: +1: jim -1: Justin, wrowe, rederpj, nd +0: Lars, Martin (while standing by, could it do something useful?) * Make the worker MPM the default MPM for threaded Unix boxes. +1: Justin, Ian, Cliff, BillS, striker, wrowe, nd +0: BrianP, Aaron (mutex contention is looking better with the latest code, let's continue tuning and testing), rederpj, jim -0: Lars pquerna: Do we want to change this for 2.2? RELEASE NON-SHOWSTOPPERS BUT WOULD BE REAL NICE TO WRAP THESE UP: * Patches submitted to
[STATUS] (httpd-test: flood) Wed Aug 17 23:45:43 2005
flood STATUS: -*-text-*- Last modified at [$Date: 2004-11-24 19:36:41 -0500 (Wed, 24 Nov 2004) $] Release: 1.0: Released July 23, 2002 milestone-03: Tagged January 16, 2002 ASF-transfer: Released July 17, 2001 milestone-02: Tagged August 13, 2001 milestone-01: Tagged July 11, 2001 (tag lost during transfer) RELEASE SHOWSTOPPERS: * Everything needs to work perfectly Other bugs that need fixing: * I get a SIGBUS on Darwin with our examples/round-robin-ssl.xml config, on the second URL. I'm using OpenSSL 0.9.6c 21 dec 2001. * iPlanet sends Content-length - there is a hack in there now to recognize it. However, all HTTP headers need to be normalized before checking their values. This isn't easy to do. Grr. * OpenSSL 0.9.6 Segfaults under high load. Upgrade to OpenSSL 0.9.6b. Aaron says: I just found a big bug that might have been causing this all along (we weren't closing ssl sockets). How can I reproduce the problem you were seeing to verify if this was the fix? * SEGVs when /tmp/.rnd doesn't exist are bad. Make it configurable and at least bomb with a good error message. (See Doug's patch.) Status: This is fixed, no? * If APR has disabled threads, flood should as well. We might want to have an enable/disable parameter that does this also, providing an error if threads are desired but not available. * flood needs to clear pools more often. With a long running test it can chew up memory very quickly. We should just bite the bullet and create/destroy/clear pools for each level of our model: farm, farmer, profile, url/request-cycle, etc. * APR needs to have a unified interface for ephemeral port exhaustion, but aparently Solaris and Linux return different errors at the moment. Fix this in APR then take advantage of it in flood. * The examples/analyze-relative scripts fail when there are less than 5 unique URLs. Other features that need writing: * More analysis and graphing scripts are needed * Write robust tool (using tethereal perhaps) to take network dumps and convert them to flood's XML format. Status: Justin volunteers. Aaron had a script somewhere that is a start. Jacek is working on a Mozilla application, codename Flood URL bag (much like Live HTTP Headers) and small HTTP proxy. * Get chunked encoding support working. Status: Justin volunteers. He got sidetracked by the httpd implementation of input filtering and never finished this. This is a stopgap until apr-serf is completed. * Maybe we should make randfile and capath runtime directives that come out of the XML, instead of autoconf parameters. * We are using apr_os_thread_current() and getpid() in some places when what we really want is a GUID. The GUID will be used to correlate raw output data with each farmer. We may wish to print a unique ID for each of farm, farmer, profile, and url to help in postprocessing. * We are using strtol() in some places and strtoll() in others. Pick one (Aaron says strtol(), but he's not sure). * Validation of responses (known C-L, specific strings in response) Status: Justin volunteers * HTTP error codes (ie. teach it about 302s) Justin says: Yeah, this won't be with round_robin as implemented. Need a linked list-based profile where we can insert new URLs into the sequence. * Farmer (Single-thread, multiple profiles) Status: Aaron says: If you have threads, then any Farmer can be run as part of any Farm. If you don't have threads, you can currently only run one Farmer named Joe right now (this will be changed so that if you don't have threads, flood will attempt to run all Farmers in serial under one process). * Collective (Single-host, multiple farms) This is a number of Farms that have been fork()ed into child processes. * Megaconglomerate (Multiple hosts each running a collective) This is a number of Collectives running on a number of hosts, invoked via RSH/SSH or maybe even some proprietary mechanism. * Other types of urllists a) Random / Random-weighted b) Sequenced (useful with cookie propogation) c) Round-robin d) Chaining of the above strategies Status: Round-robin is complete. * Other types of reports Status: Aaron says: simple reports are functional. Justin added a new type that simply prints the approx. timestamp when the test was run, and the result as OK/FAIL; it is called easy reports (see flood_easy_reports.h).
[STATUS] (httpd-test: perl-framework) Wed Aug 17 23:45:47 2005
httpd-test/perl-framework STATUS: -*-text-*- Last modified at [$Date: 2004-11-24 19:36:41 -0500 (Wed, 24 Nov 2004) $] Stuff to do: * finish the t/TEST exit code issue (ORed with 0x2C if framework failed) * change existing tests that frob the DocumentRoot (e.g., t/modules/access.t) to *not* do that; instead, have Makefile.PL prepare appropriate subdirectory configs for them. Why? So t/TEST can be used to test a remote server. * problems with -d perl mode, doesn't work as documented Message-ID: [EMAIL PROTECTED] Date: Sat, 20 Oct 2001 12:58:33 +0800 Subject: Re: perldb Tests to be written: * t/apache - simulations of network failures (incomplete POST bodies, chunked and unchunked; missing POST bodies; slooow client connexions, such as taking 1 minute to send 1KiB; ...) * t/modules/autoindex - something seems possibly broken with inheritance on 2.0 * t/ssl - SSLPassPhraseDialog exec: - SSLRandomSeed exec:
Re: canonical port
Hmm, it seems if useCanonicalName is off and you use Servername like this: ServerName www.domain.com:80 That ap_get_servername will use that port unless the client used a port in the Host: header. My testing seems to confirm this. Is this correct? -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: [STATUS] (httpd-2.1) Wed Aug 17 23:45:30 2005
Rodent of Unusual Size wrote: * mod_cache: Resolve issue of how to cache page fragements (or perhaps -if- we want to cache page fragements). Today, mod_cache/mod_mem_cache will cache #include 'virtual' requests (but not #include 'file' requests). This was accomplished by making CACHE_IN a CONTENT_SET-1 filter to force it to run before the SUBREQ_CORE filter. the svn version of mod_cache I have (r220038 | pquerna | 2005-07-21 07:40:30 -0400 (Thu, 21 Jul 2005)) registers cache save as AP_FTYPE_CONTENT_SET+1, which is where it should be, IMO. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: svn commit: r233369 - /httpd/httpd/trunk/modules/mappers/mod_dir.c
-1.
If what the PR you fixed wanted to avoid mod_cgi, or the core
handler dealing with the request, then the correct answer
was to have those modules reject this request. Now I expect
you've just hijacked the request away from tomcat via mod_jk
among other methods for a module to substitute their own
directory handling.
A handler is an explicit reference, and now you have just
hijacked APR_DIR type files to mod_dir, irrespective of
the configuration.
Please don't make code changes to make up for sloppy
configuration changes.
Perhaps differentiating SetHandler into SetFileHandler
and SetDirHandler would be an appropriate solution.
Bill
At 03:10 PM 8/18/2005, [EMAIL PROTECTED] wrote:
Author: pquerna
Date: Thu Aug 18 13:10:26 2005
New Revision: 233369
URL: http://svn.apache.org/viewcvs?rev=233369view=rev
Log:
Do not check the value of r-handler.
This allows the use of SetHandler for an entire directory, and since we
already check via the stat structure if this is a directory, there is no
reason for this extra check, which causes a regression since 1.3.
PR: 25435
Modified:
httpd/httpd/trunk/modules/mappers/mod_dir.c
Modified: httpd/httpd/trunk/modules/mappers/mod_dir.c
URL:
http://svn.apache.org/viewcvs/httpd/httpd/trunk/modules/mappers/mod_dir.c?rev=233369r1=233368r2=233369view=diff
==
--- httpd/httpd/trunk/modules/mappers/mod_dir.c (original)
+++ httpd/httpd/trunk/modules/mappers/mod_dir.c Thu Aug 18 13:10:26 2005
@@ -151,10 +151,6 @@
return HTTP_MOVED_PERMANENTLY;
}
-if (strcmp(r-handler, DIR_MAGIC_TYPE)) {
-return DECLINED;
-}
-
if (d-index_names) {
names_ptr = (char **)d-index_names-elts;
num_names = d-index_names-nelts;
Re: svn commit: r233369 - /httpd/httpd/trunk/modules/mappers/mod_dir.c
William A. Rowe, Jr. wrote:
-1.
If what the PR you fixed wanted to avoid mod_cgi, or the core
handler dealing with the request, then the correct answer
was to have those modules reject this request. Now I expect
you've just hijacked the request away from tomcat via mod_jk
among other methods for a module to substitute their own
directory handling.
A handler is an explicit reference, and now you have just
hijacked APR_DIR type files to mod_dir, irrespective of
the configuration.
How does mod_]
do this in 1.3, which does not have the check that I just removed?
Please don't make code changes to make up for sloppy
configuration changes.
Excuse me? This worked perfectly in 1.3.
Perhaps differentiating SetHandler into SetFileHandler
and SetDirHandler would be an appropriate solution.
Bill
At 03:10 PM 8/18/2005, [EMAIL PROTECTED] wrote:
Author: pquerna
Date: Thu Aug 18 13:10:26 2005
New Revision: 233369
URL: http://svn.apache.org/viewcvs?rev=233369view=rev
Log:
Do not check the value of r-handler.
This allows the use of SetHandler for an entire directory, and since we
already check via the stat structure if this is a directory, there is no
reason for this extra check, which causes a regression since 1.3.
PR: 25435
Modified:
httpd/httpd/trunk/modules/mappers/mod_dir.c
Modified: httpd/httpd/trunk/modules/mappers/mod_dir.c
URL:
http://svn.apache.org/viewcvs/httpd/httpd/trunk/modules/mappers/mod_dir.c?rev=233369r1=233368r2=233369view=diff
==
--- httpd/httpd/trunk/modules/mappers/mod_dir.c (original)
+++ httpd/httpd/trunk/modules/mappers/mod_dir.c Thu Aug 18 13:10:26 2005
@@ -151,10 +151,6 @@
return HTTP_MOVED_PERMANENTLY;
}
-if (strcmp(r-handler, DIR_MAGIC_TYPE)) {
-return DECLINED;
-}
-
if (d-index_names) {
names_ptr = (char **)d-index_names-elts;
num_names = d-index_names-nelts;
Some RFC 2616 questions
The spec for If-{None-}Match and If-{Un}Modified-Since is driving
me batty.
The biggest item has to do with having to know the response code
for the request without processing the request. Specifically, 14.24
(If-Match) and the others have a requirement like:
If the request would, without the If-Match header field,
result in
anything other than a 2xx status, then the If-Match header
MUST be
ignored.
This implies that we have to attempt the request, check the
status, and act on the If-Match header only if the request resulted
in an non-error status. That seems rather expensive for the server.
For example, if we have a GET request that recomputes uncached data
or something like that.
Furthermore, for many operations, such as PUT, COPY, MOVE and
DELETE, we'd have to back out the successful operation, since I might
not know that PUT will fail without actually trying it first.
Next up, 14.25 (If-Modified-Since) implies in the first paragraph
that it applies to all methods, but then only proceeds to describe
what to do with a GET, and even then, a GET with no Range header. If
I get an IMS for a PUT, is that a valid request? How about a DAV
COPY? Presumably in that case would apply, if valid, to the source
resource, not the destination.
-wsv
smime.p7s
Description: S/MIME cryptographic signature
Supporting RBL in mod_smtpd
Here is my current plan for introducing the RBL support in mod_smtpd, using the existing mod_dnsbl_lookup which I posted earlier. This way of accomplishing the RBL support should not require any code modification to mod_smtpd itself. Nick and Rian, let me know if I should be going about this a different way? I thought the most modular fashion would be to create a mod_smtpd_rbl that registers the following mod_smtpd hooks: smtpd_run_connect (might deny service to connecting IP, per request_rec) smtpd_run_mail (might deny service to this envelope domain, per loc) These would query whitelists and blacklists, whatever is available. I don't mind whipping up this bridging mod_smtpd_rbl module, but if it seems excessive to introduce a new module for this purpose then the other way of doing this would be to add the RBL supporting code into mod_smtpd itself. Either way it's done, RBLs are still not required and mod_dnsbl_lookup does not have to be present for mod_smtpd to function normally. However, adding a new bridging module has the advantage of leaving mod_smtpd code alone and taking advantage of the hooks interface.
Re: Supporting RBL in mod_smtpd
On Aug 18, 2005, at 7:11 PM, Jem Berkes wrote: Here is my current plan for introducing the RBL support in mod_smtpd, using the existing mod_dnsbl_lookup which I posted earlier. This way of accomplishing the RBL support should not require any code modification to mod_smtpd itself. Nick and Rian, let me know if I should be going about this a different way? I thought the most modular fashion would be to create a mod_smtpd_rbl that registers the following mod_smtpd hooks: smtpd_run_connect (might deny service to connecting IP, per request_rec) smtpd_run_mail (might deny service to this envelope domain, per loc) +1 These would query whitelists and blacklists, whatever is available. I don't mind whipping up this bridging mod_smtpd_rbl module, but if it seems excessive to introduce a new module for this purpose then the other way of doing this would be to add the RBL supporting code into mod_smtpd itself. Don't do this just yet, mod_smtpd is changing completely! completely = structures/io. I should commit my changes very soon so you can start working on this. Either way it's done, RBLs are still not required and mod_dnsbl_lookup does not have to be present for mod_smtpd to function normally. However, adding a new bridging module has the advantage of leaving mod_smtpd code alone and taking advantage of the hooks interface. +1 -rian
Re: Rolling 2.1.7 On Friday
At 03:36 AM 8/18/2005, Joe Orton wrote: On Wed, Aug 17, 2005 at 10:43:01PM -0700, Paul Querna wrote: Just a heads up, I am planning to RM and tag 2.1.7 (and re-branch from trunk the 2.2.x branch) on Friday or Saturday this week. I intend to include APR and APR-Util 1.2.1 with this release. Sounds great. Can you exclude the mod_setenvif OID() changes? I don't think they are ready for release. Unless you trip the feature deliberately, this won't interfere with anyone using the current trunk, would it? Yes, the feature might be 'experimental', but I don't think it's harmful. Am I mistaken? Bill
Re: Supporting RBL in mod_smtpd
smtpd_run_connect (might deny service to connecting IP, per request_rec) smtpd_run_mail (might deny service to this envelope domain, per loc) +1 ... Don't do this just yet, mod_smtpd is changing completely! completely = structures/io. I should commit my changes very soon so you can start working on this. OK, I'll watch for the changes. Make sure you keep what I need though :) smtpd_run_connect should somehow pass the address of the peer (currently that's within the request_rec) smtp_run_mail should still pass the MAIL FROM address the peer specifies, it currently comes in the char* loc. As long as this data is still available to me and I can return a code to reject the mail, we should be good to go. Somewhat trivial I hope.
Re: Rolling 2.1.7 On Friday
At 12:43 AM 8/18/2005, Paul Querna wrote: Just a heads up, I am planning to RM and tag 2.1.7 (and re-branch from trunk the 2.2.x branch) on Friday or Saturday this week. I intend to include APR and APR-Util 1.2.1 with this release. Note you can't tag the recent commit to mod_dir, there is a veto on the floor. As long as 2.1.7 seems good, I would like to do a vote on making it a Beta. Our procedure (I forget you are relatively speaking a bit new to this role :-) is to let this ride three days on our own infrastructure, eating our own dogfood, so to speak. If that doesn't keel over I'll support the vote to go beta. This is not quite GA, IMHO, but getting quite close. FYI you can't trash branches willy nilly, which is why I had warned you not to branch. You must now backport each change you wish to introduce to 2.2 from trunk/. Not R-T-C, of course, but C-T-R. Sorry, you dug the hole, now climb out. Trashing a branch means that one of your co-committers with a checkout is going to be very, very borked when they svn up. Just don't do it. Bill
Re: Rolling 2.1.7 On Friday
[EMAIL PROTECTED]'ers... As Paul mentions below, on Friday (probably 1 a.m. ;-) he will tag 2.1.7. This is just a reminder to our most excellent doco team that, once ack'ed - this becomes 2.1 beta, and then... (drum roll please...) 2.2 GA(!) So if you had changes to catch up with 2.1 that perhaps have been sitting in your local checkouts, now is an excellent time to commit the docs improvements. Each release, we state v x.x is the best available version of Apache, and certainly the docs@ team has proven this! When 2.2 final (not beta, not alpha) drops, we certainly all want this to be true not only of the code, but also of the accompanying documentation! And I trust you will make it so. Yours, Bill p.s. it's been a while - have I told you all lately that you rock?!? :) At 12:43 AM 8/18/2005, Paul Querna wrote: Just a heads up, I am planning to RM and tag 2.1.7 (and re-branch from trunk the 2.2.x branch) on Friday or Saturday this week. I intend to include APR and APR-Util 1.2.1 with this release. As long as 2.1.7 seems good, I would like to do a vote on making it a Beta. Thanks, -Paul
Re: Rolling 2.1.7 On Friday
William A. Rowe, Jr. wrote: At 12:43 AM 8/18/2005, Paul Querna wrote: Just a heads up, I am planning to RM and tag 2.1.7 (and re-branch from trunk the 2.2.x branch) on Friday or Saturday this week. I intend to include APR and APR-Util 1.2.1 with this release. Note you can't tag the recent commit to mod_dir, there is a veto on the floor. Yes, I disagree to the said veto, and I hope to have an alternative available before I do this. (If not, I agree completely and will omit that change from the branch, or revert trunk.) As long as 2.1.7 seems good, I would like to do a vote on making it a Beta. Our procedure (I forget you are relatively speaking a bit new to this role :-) is to let this ride three days on our own infrastructure, eating our own dogfood, so to speak. Sure, if someone in Infrastructure is willing and has the time to set it up. I don't have the karma to do it myself. If that doesn't keel over I'll support the vote to go beta. This is not quite GA, IMHO, but getting quite close. FYI you can't trash branches willy nilly, which is why I had warned you not to branch. You must now backport each change you wish to introduce to 2.2 from trunk/. Not R-T-C, of course, but C-T-R. Sorry, you dug the hole, now climb out. Actually, after consulting Karl Fogel(svn-developer) on IRC, he says that re-creating the branch will not cause a problem with people's working copies, as long as both branches have the same origin, which they will. -Paul
