Re: 3.0 - Proposed Goals
Sander Temme wrote: How many Apache 'D' versions do we want to maintain? Popularity of 1.3 is still too high for us to completely ignore, and there is much 2.0 still out there. Any many people taking up 2.2... -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ If you can dodge a wrench, you can dodge a ball.
Re: Board Report is due on Monday....
Paul Querna wrote: Events since November: - mod_ftp graduates from incubator httpd tlp - next generation aka 'amsterdamn' discussions started with sandbox creation. - TSU encryption notifications sent for httpd and flood. - httpd 2.2.4 released - no pmc changes or committer changes - mod_wombat IP clearance done. (import happening RSN) Anything else anyone can come up with? Not really. -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ If you can dodge a wrench, you can dodge a ball.
Re: Using mod_proxy from another module
Yep. No problem. On Feb 21, 2007, at 4:50 AM, Marc Stern wrote: Is it also possible to make a post with this ? This is what I need. Jim Jagielski wrote: How could we use mod_proxy for outgoing connections from another module ? Is there any simple way ? Is there any standard (I mean documented) way, or would it rely on internals only ? In the latter case, we would have to recode everything when mod_proxy changes. You would do something like this: rr = ap_sub_req_method_uri(OPTIONS, *, r, NULL); /* this MUST succeed! */ apr_snprintf(newurl, sizeof(newurl), proxy:%s://%s:%d/%s, protocol, sendToIP, sendToPort, myURL); ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r-server, query constructed: %s, newurl); rr-filename = apr_pstrdup(r-pool, newurl); rr-proxyreq = PROXYREQ_REVERSE; rr-handler = proxy-server; ap_run_sub_req(rr); and then use the response as needed... The above I've used in some where I'm just interested in getting some backend header info...
sed filter module
There have been times when having a simple sed filter in Apache would be useful... I used to use just ext_filter to do this, but this got more and more painful the more I used it. So awhile ago I made mod_sed_filter which I find pretty useful. I've just built and tested in with 2.2 and trunk... Anyone mind if I fold it into trunk and maybe have us consider making it part of 2.2 (even under experimental)? No docs yet but the code is: http://people.apache.org/~jim/code/mod_sed_filter.c and the usage is easy: AddOutputFilterByType SEDFILTER text/html Sed s/foo/bar/in Sed s#monkey(hat)#chimp-$1#i Sed s/works/functions/in note that it uses sed line controls, flexible delims and support regex and simple pattern match (the 'n' flag... no real sed option there ;) )
Re: sed filter module
On Mar 13, 2007, at 1:10 PM, William A. Rowe, Jr. wrote: Is this sed or pcre syntax? I'm a bit confused :) It's a mutant ;) But, of course, we maintain that confusion internally with regex's being pcre... Although it's sed-ish, is it misleading to confuse the user with the phrase sed considering the unsupported constructs? E.g. I presume the more complex sed language features aren't present. I'm wondering if mod_pcre_filter wouldn't be more accurate? 'sed' certainly gets the message across though :) But basically it allows for regex pattern matching and substitution in a very sed-like way. By agreed that docs would help this
Re: sed filter module
On Mar 13, 2007, at 2:08 PM, Nick Kew wrote: AFAICS, this not merely looks like mod_line_edit: the filter *is* mod_line_edit, right down to the bucket manipulation logic used as an example in The Book! It's just missing a couple of minor features, and has a slightly different configuration syntax. The other difference is 15 months out there in widespread use. What logic? Let me know what sections you mean because most of what I based it on is stuff from mod_include and mod_proxy_ftp.c (and other ASF modules). I don't see anything in either module which is new or not done by any other modules out there that need to split out sections from buckets. Bill told me about mod_line_edit maybe 3-4 days ago. I had known about mod_proxy_html, which is also something we've pointed clients to, so maybe that's where the confusion comes from.
Re: sed filter module
On Mar 13, 2007, at 3:34 PM, William A. Rowe, Jr. wrote: Jim Jagielski wrote: On Mar 13, 2007, at 1:10 PM, William A. Rowe, Jr. wrote: Is this sed or pcre syntax? I'm a bit confused :) It's a mutant ;) But, of course, we maintain that confusion internally with regex's being pcre... Of course :) But it appears to be a tiny fraction of the sed language... Although it's sed-ish, is it misleading to confuse the user with the phrase sed considering the unsupported constructs? E.g. I presume the more complex sed language features aren't present. I'm wondering if mod_pcre_filter wouldn't be more accurate? 'sed' certainly gets the message across though :) But basically it allows for regex pattern matching and substitution in a very sed-like way. since it is only a pattern substitution subset, I'd prefer to see some RewriteBody directive or similar. As I'm looking at the module, I'm more convinced that Sed foo should be reserved for at least a basic sed implementation that implemented (at least!) the pre-GNU language subset. :) Well, like I said, the main issue was avoiding the overhead of having mod_ext_filter do simple in-line replacements by calling sed to do 's/foo/bar/'... So yeah, it's closer to what a Perl guy would think than a Unix sed-head :)
Re: sed filter module
On Mar 14, 2007, at 5:07 AM, Frank wrote:
RewriteBodyLine 'http://(.*?)/(.*)/(.*)' 'http://${LOWERCASE:$1}/$
{MD5:$2}/$3'
Yeah, that would be useful... Of course, the main issue is
that whereas mod_rewrite can afford to be dog slow, because,
after all, the URLs aren't *that* big, in-place rewriting
of content can't be. The more complex the functionality,
the slower it will be... :/
Re: sed filter module
On Mar 14, 2007, at 11:01 AM, Nick Kew wrote: Oh, I guess you mean the copying to get a null-terminated string when applying a regexp? And I see it's repeated for every regexp (ouch)! mod_line_edit uses a local pool which is cleared at the end of each brigade, and avoids multiple copies of the same buffer. Hmmm... I'm confused. The way I do it is: loop over sed scripts loop over buckets read bucket make copy of bucket data for regex comparison so everytime we read in bucket data, I have to make a null-termed string. It changes with each bucket. So I don't understand the issue with it being repeated for every regexp. How can that be avoided? I reuse allocated space (I don't just simply keep making strdups)... so yeah, there will be a chunk of allocated spool still hanging around. So maybe making that a subpool and then clearing/destroying it would be best.
Re: sed filter module
As a rough proof of concept, I refactored the design, allowing for the pattern matching and substitution to be done as soon as we have a line. Also is some rough ability to pass the data to the next filter after we get more than ~AP_MIN_BYTES_TO_WRITE bytes. Doesn't alleviate all the problems, but it allows for us to pass data quicker (we still have the issue where we need to fully read in the bb though...) It's rough but passes superficial testing... More work needs to be done, but more people could work on it if I just commit to trunk :) Same URL, different version: http://people.apache.org/~jim/code/mod_sed_filter.c
sed and offline
I'll be offline most of tomorrow and pretty much the whole weekend. Unless I hear vetos, I'll commit the latest mod_sed_filter.c to trunk. If we change the name, which is fine with me, well... that's the joy of svn move :)
Re: mod_proxy patches
On Mar 16, 2007, at 8:52 AM, Mathias Herberts wrote: I agree that reusing the backend connections can be a good thing, but there are times when this is just not a very good idea. I agree that there are times when having a single-shot connection is better than having a pool. It's certainly not a normal situation but for those people affected by this, it's a deal-breaker.
Re: [PATCH] ProxyAddXHeaders for mod_proxy
On Feb 28, 2007, at 11:59 AM, Dziugas Baltrunas wrote: Hi, list, attaching same patch with small correction to the one submitted previously (was unnecessary double check for PROXYREQ_REVERSE). Patch is for httpd-2.2.x branch. Patches should be against trunk. Once approved and proven in the trunk environ, then they may be proposed for backporting to 2.2.x
Re: Bug 41897 / Session-Stickiness with mod_proxy_balancer
On Apr 4, 2007, at 8:46 AM, Plüm, Rüdiger, VF-Group wrote: But this means that we break the ABI here. If struct proxy_balancer is part of a public API (I am not sure about this) this would require a major bump and would prevent backporting. Since the whole idea of loadable lb method via providers implies an external API to do this, I consider it a public API. But r-subprocess_env is a table and apr_table_get is never case sensitive. So the case parameter does not make sense for environment variables. To be honest I do not understand why we need the case parameter at all (also for cookie and path). I did not understand the hint you gave regarding PHP and ASP on Jean-Frederic's question on the same matter. FWIW, neither did I.
Re: Bug 41897 / Session-Stickiness with mod_proxy_balancer
On Apr 4, 2007, at 9:38 AM, Mladen Turk wrote: Plüm wrote: Yes, although MAX_ENTRIES would be 3 for now (cookie, path and env) But this means that we break the ABI here. If struct proxy_balancer is part of a public API (I am not sure about this) this would require a major bump and would prevent backporting. The other solution is to keep that in balancer-sticky separated by / (think %2f is unacceptable char anyhow) and do a strtok inside find_session_route. Hmmm... I like this idea. Not sure about the use of '/' as the delim (simply because of its other meanings) but that's secondary. Of course, we can't use strtok, since sticky is a const char and we can't be shoving NULLs in there :)
Re: Bug 41897 / Session-Stickiness with mod_proxy_balancer
On Apr 4, 2007, at 10:27 AM, Plüm, Rüdiger, VF-Group wrote: Hmmm... I like this idea. Not sure about the use of '/' as the delim (simply because of its other meanings) but that's secondary. Of course, we can't use strtok, since sticky is a const char and we can't be shoving NULLs in there :) Plus does it make sense to split the sticky string everytime we search for a route? From my performance feeling this split should be done during configuration time. Agreed... It doesn't change. How about adding either a const char * some_name[FIXED_VALUE] or an apr_array_header_t * to the end of struct proxy_balancer to store the results of the split and setting sticky = some_name[last_non_null_entry] for backward compatibility? We already have an existing context we could overload or simply append another element to the struct. Of course, doing that really confuses the whole issue, since we're doing all these weird things to get around changing sticky from a const char* to a apr_array_header_t*... Sooo... how about in trunk making it right (sticky - apr_array_header_t*) and in 2.2 appending sticky_table (apr_array_header_t*)...
Re: Bug 41897 / Session-Stickiness with mod_proxy_balancer
On Apr 4, 2007, at 11:34 AM, Georg von Zezschwitz wrote: Jim Jagielski schrieb: Rüdiger's point is that we would be doing the scanning for each and every request, which is wasteful since they aren't changing. Even in the above the strlen() is counting chars that don't change between requests. the performance aspect is why I would also prefer the arbitrary sequence of stickysession-attributes with an optional Cookie/Path-prefix. Currently, the URL path search is performed before the Cookie search. I do not know why this sequence was chosen (most users will rather support cookie-based session management, so the URL approach is less likely to match). If we state that the evaluation takes place in the occurence of stickysession attributes and suggest stickysession=Cookie:JSESSIONID stickysession=Path:;jsessionid to the user it will perform faster in average. As I promised to write the patch, I would do it the way Jim suggested it for backward compatibility, however I would also add a table sticky_type containing the Cookie vs. Path vs. Env-prefix (in a parsed way) to the balancer struct. Is this OK for everybody? Patch for trunk...
Re: mod_ftp enhancements - group directories
On Apr 5, 2007, at 7:16 PM, Filip Hanik - Dev Lists wrote: I would like to propose two enhancements to mod_ftp. The same way we have FTPJailUser and CreateHomeDir directives, we would need FTPJailGroup and CreateGroupDirs directives. This would allow us manage FTP files based on groups, rather than single individual users. Of course, this would only make sense for those auth mechanisms that support the concept of groups Thoughts? Anyone interested? (too far down the java trench to be capable of providing a patch) Filip
Re: mod_proxy buffering small chunks
On Apr 11, 2007, at 3:34 PM, Ruediger Pluem wrote: On 04/11/2007 06:42 PM, Filipe wrote: I'm trying to use apache to proxy a push application, using chunked transfer encoding. The problem is that the mod_proxy buffers the server response internally and only sends the data to the client when the buffer is filled or the connection with the server is terminated. This behavior is not adequate for push applications, where the response must be redirected instantly to the http client. This problem was discussed a year ago: http://mail-archives.apache.org/mod_mbox/httpd-dev/200602.mbox/raw/ [EMAIL PROTECTED]/ This is only fixed in trunk so far. See http://issues.apache.org/bugzilla/show_bug.cgi?id=41056 and http://svn.apache.org/viewvc?view=revrevision=480135 Hmmm. Looks like a backport candidate... I'll likely do some testing and propose if it works :)
Re: ProxyErrorOverride and redirects (PR 39245)
On Apr 12, 2007, at 10:16 AM, Joe Orton wrote: On Thu, Apr 12, 2007 at 10:05:06AM -0400, Jeff Trawick wrote: I wonder why Error in ProxyErrorOverride doesn't match the meaning of ap_is_HTTP_ERROR(), as in the attached patch (with doc). Great, +1 1xx isn't something the user should see/react to either. Forwarding 1xx responses is actually a 2616 MUST, but they don't have bodies so are handled separately in this code anyway (though are not in fact forwarded IIRC). Agreed.
Re: mod_proxy buffering small chunks
On Apr 13, 2007, at 5:20 AM, Mladen Turk wrote: Anyhow, I'll commit a patch to the trunk for http, cause its configurable by flushwait and skipped otherwise. Looking forward to seeing it... recall that if we flush at every chunk with HTTP, we will be dead slow and filters will not be happy :)
Re: mod_proxy buffering small chunks
On Apr 13, 2007, at 5:20 AM, Mladen Turk wrote: Plüm wrote: +1 I will try to check it once you have proposed it and give it a quick vote. I have another one that fixes this issues for non-chunked content. I haven't tried yet, but IMHO it should already work for non-chunked content. Is this not the case? Not according to my tests. The simple server push still buffers the data. Hmmm a followup commit has: URL: http://svn.apache.org/viewvc?view=revrev=504559 Log: Further refinement for PR41056 / PR 19954 (mostly-fixed in r480135.) We assume that a successful read but an empty brigade is NOT cause for EAGAIN. Testing may or may not confirm this assumption, in which case that test may be required as well. so that may be exactly the case...
Re: mod_proxy buffering small chunks
On Apr 13, 2007, at 11:34 AM, Jim Jagielski wrote: On Apr 13, 2007, at 5:20 AM, Mladen Turk wrote: Plüm wrote: +1 I will try to check it once you have proposed it and give it a quick vote. I have another one that fixes this issues for non-chunked content. I haven't tried yet, but IMHO it should already work for non-chunked content. Is this not the case? Not according to my tests. The simple server push still buffers the data. Hmmm a followup commit has: URL: http://svn.apache.org/viewvc?view=revrev=504559 Log: Further refinement for PR41056 / PR 19954 (mostly-fixed in r480135.) We assume that a successful read but an empty brigade is NOT cause for EAGAIN. Testing may or may not confirm this assumption, in which case that test may be required as well. so that may be exactly the case... I refer people to http://mail-archives.apache.org/mod_mbox/httpd-dev/200702.mbox/% [EMAIL PROTECTED] ... so maybe a successful read of an empty line in AP_MODE_GET_LINE does *not* return an empty brigade... We currently assuming it does. Anyone able to test if Rüdiger's version addresses this?
Re: mod_proxy buffering small chunks
On Apr 13, 2007, at 12:03 PM, Mladen Turk wrote: Jim Jagielski wrote: Not according to my tests. The simple server push still buffers the data. Hmmm a followup commit has: URL: http://svn.apache.org/viewvc?view=revrev=504559 so that may be exactly the case... Huh, looks like it works now. Although I didn't test is with mod_deflate, but I suppose it should work as well. Can we get those back trough 2.0? Def for 2.2 (maybe you were testing 2.2 and not trunk??)... I'll see how much can actually go to 2.0 :)
Re: RFE -- external overload procedure
It would be best, I think, if the patches actually used normal httpd coding standards... The whole MAX_OVLP_LINE stuff is very out of place. On Apr 25, 2007, at 10:02 AM, Juerg Umhang wrote: its now in bugzilla. patches submitted http://issues.apache.org/bugzilla/show_bug.cgi?id=42216 -- juerg On 4/20/07, Juerg Umhang [EMAIL PROTECTED] wrote: hello please consider this posting as a request for enhancement httpd knows about his overload situation. [error] server reached MaxClients setting, consider raising the MaxClients setting this overload is easily created by an external attacker. in case of an attack you have to react. best done on a lower osi-layer (iptables, pf, ...). realtime log analysis has his own odds and twists. we would prefer a call to an 'external helper procedure'. in this context we have some questions: -- do you think it makes sense to implement this feature ? -- could it be done in a module (without the overhead of going through the scoreboard for each pre_connection call) ? It is reasonable to me for httpd to provide a module interface (hook) so that a third-party module can take action when httpd reaches the MaxClients (Unix) or ThreadsPerChild (Windows) condition. (Maybe the hook just provides some basic statistics, and the module can determine whether the absolute limit has been reached or its own configurable threshhold has been reached.) A way that a module can do something reasonable without modifying the server is to create a separate child process that monitors the scoreboard at its own interval, and takes whatever action is appropriate. That check can be infrequent enough that the performance overhead is negligible. -- can we expect this enhancement in a future release ? Some other committer can speak for themselves, but I wouldn't expect it without a patch submitted. btw: we hope to see separately configurable timeouts ( http://httpd.apache.org/docs/2.2/mod/core.html#timeout ) very soon. I don't recall anyone here interested in fulfilling the goal expressed in that comment.
Re: mod_ftp, status and progress?
On Apr 18, 2007, at 1:22 PM, Guenter Knauf wrote: Hi, the current code fails to build for Win32 target. This is because ftp_glob.c seems not APR-ised yet; I'm actually looking at removing the whole glob stuff and emulating it as regexes...
Re: Store data accessible by all process and threads.
On Apr 19, 2007, at 6:25 PM, moh bad wrote: Hi, within a module, what is the best way to store data, who need to be accessible to all threads and process ? 1/ using shared memory, with apr_shm_* #1
Re: Store data accessible by all process and threads.
On Apr 21, 2007, at 12:28 AM, Saju Pillai wrote: Read-Write data ? shm is ok. Any standard IPC should work. apr provides this.
Re: bug with Apache 1.3 NetWare build system
On Apr 23, 2007, at 1:20 PM, Brad Nicholes wrote: +1, go ahead and commit it. Brad +1
Re: Patch for implementing ap_document_root as a hook
On Apr 23, 2007, at 10:46 AM, Brian J. France wrote: On Apr 23, 2007, at 10:32 AM, Jakob Goldbach wrote: -1 on the face of things. The map_to_storage hook was added to accomplish what you desire. I thought map_to_storage was made to do per-dir configuration. Not path-translation. The problem is not really doing the translation. I can always provide my own translate handler in my module. But in the current API I cannot to set my env. variables at will. They will be overwritten by ap_add_common_vars which returns carved-in-stone docroot from ap_document_root. We need this same functionality (would like to back port to 2.2 if possible). I'm +1 on the concept but -0 on the provided method of doing it... We currently hack the doc root in the post read hook in 1.3, would like to be able to do it with out hacking the core and screwing around with internal structs at runtime. VERY doubtful that 1.3 will be updated to do this.
Re: mod_ftp, status and progress?
On Apr 26, 2007, at 8:55 AM, Niklas Edmundsson wrote: On Thu, 26 Apr 2007, Jim Jagielski wrote: On Apr 18, 2007, at 1:22 PM, Guenter Knauf wrote: Hi, the current code fails to build for Win32 target. This is because ftp_glob.c seems not APR-ised yet; I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility...
Re: mod_ftp, status and progress?
On Apr 26, 2007, at 9:22 AM, Jim Jagielski wrote: On Apr 26, 2007, at 8:55 AM, Niklas Edmundsson wrote: On Thu, 26 Apr 2007, Jim Jagielski wrote: On Apr 18, 2007, at 1:22 PM, Guenter Knauf wrote: Hi, the current code fails to build for Win32 target. This is because ftp_glob.c seems not APR-ised yet; I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility... which, of course, implies apr_fnmatch and not apr_match_glob...
Re: DO NOT REPLY [Bug 39710] - mod_cgi truncates error replies
On Apr 26, 2007, at 11:58 AM, Nick Kew wrote: On Thu, 26 Apr 2007 07:21:27 -0700 (PDT) [EMAIL PROTECTED] wrote: --- Additional Comments From [EMAIL PROTECTED] 2007-04-26 07:21 --- (In reply to comment #7) Anything I can do to help get this bug fixed? Is testing the patch all that stands in the way of getting it folded into the trunk? Hello? Anybody home? What's next? He has a good point: this bug is a regression. I'm not entirely clear on the problem that was being fixed in the patch that introduced it. But it looks like confusion between different CGI error paths. If noone has the round tuits to work through a fix, can we just apply the patch (which reverses r231167)? Can we confirm that we don't still suffer from the original problem?
Re: mod_ftp, status and progress?
On Apr 26, 2007, at 5:39 PM, Niklas Edmundsson wrote: On Thu, 26 Apr 2007, Jim Jagielski wrote: I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility... Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot of people still running 1.3 and 2.0, but that doesn't mean that we have to make it run on all of them... Why? Really, it's no big deal to ensure it runs on both.
Re: Releases around ApacheCon EU?
On Apr 26, 2007, at 5:52 PM, Sander Temme wrote: Community, As ApacheCon draws near, perhaps we should plan some httpd releases during the week? Between beer and bitterballen there must be time to bat some patches around(1). Going in with the intention of, say, a TR on Wednesday after the Tuesday Hackathon should get us in the right frame of mind to get something out by the end of the week. I'd like to try my hand at the TR process so if no one objects I volunteer to play RM. -- 1.3.x, any patches that justify a release? -- 2.0.x There are some backports in need of votes. There seems to be only one patch to APR 0.9.x since the last release, do we need one? -- 2.2.x IIRC the initial intent was to roll 2.2.5 fairly quickly after 2.2.4, but that never happened. Many changes to APR 1.2.x, we'll likely need a new release of that. I think this is a good idea... have people review STATUS and start voting. Yeah, my hope was to get a 2.2.5 out semi-quickly, I think with the various proxy changes, it's time, but I was holding off because there are quite a few backports missing just 1 vote.
Re: SatisfyOne
Are you sure that there are no other conflicting ACLs? On Apr 27, 2007, at 1:30 PM, Patrick Welche wrote: Basically, bug or configuration error? Using httpd trunk 529626, of Apr 19 2007, I tried a FAQ configuration with the new authentication framework: Directory /usr/local/share/httpd/htdocs/learn AuthType basic AuthName raven test AuthBasicProvider file AuthUserFile /usr/local/etc/pass.txt SatisfyOne Require host quartz.itdept.newn.cam.ac.uk Require ip 192.168.200.180 Require valid-user /SatisfyOne /Directory quartz% hostname quartz.itdept.newn.cam.ac.uk quartz% lynx http://test.itdept.newn.cam.ac.uk/learn Alert!: Access without authorization denied -- retrying Username for 'raven test' at server 'test.itdept.newn.cam.ac.uk': I expected not to be prompted to login by the above configuration. (also tried AuthBasicAuthoritative Off, and have read the fine manual..) Cheers, Patrick
Re: mod_ftp, status and progress?
On Apr 27, 2007, at 5:18 PM, Niklas Edmundsson wrote: On Fri, 27 Apr 2007, Jim Jagielski wrote: I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility... Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot of people still running 1.3 and 2.0, but that doesn't mean that we have to make it run on all of them... Why? Really, it's no big deal to ensure it runs on both. I'm not against keeping compatibility. However I feel that the right way to do it would be to design stuff for current httpd and then add glue for the backward compat stuff (and not doing it the #ifdef-mess-way). So, going for regexes just because apr_match_glob() doesn't exist in 2.0.x seems a bit sub-optimal... The idea was either to redo ftp_glob via regexes or simply use apr_fnmatch... The emulate was to allow for extensions to simple globing, which could make the FTP module more useful, but, of course, client support would be lacking. But really, again, I repeat that it's no big deal to ensure it runs on both. Even if we simply just create ftp_match_glob() from apr_match_glob(). And, fwiw, mod_ftp was originally written *for* 2.0.x, not 2.2.x. So most of the work had been in getting it to work with the current httpd. But if anyone does anything to prevent mod_ftp from building on 2.0.x, expect a big -1.
Re: mod_ftp, status and progress?
On Apr 28, 2007, at 1:34 PM, Jim Jagielski wrote: On Apr 27, 2007, at 5:18 PM, Niklas Edmundsson wrote: On Fri, 27 Apr 2007, Jim Jagielski wrote: I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility... Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot of people still running 1.3 and 2.0, but that doesn't mean that we have to make it run on all of them... Why? Really, it's no big deal to ensure it runs on both. I'm not against keeping compatibility. However I feel that the right way to do it would be to design stuff for current httpd and then add glue for the backward compat stuff (and not doing it the #ifdef-mess-way). So, going for regexes just because apr_match_glob() doesn't exist in 2.0.x seems a bit sub-optimal... The idea was either to redo ftp_glob via regexes or simply use apr_fnmatch... The emulate was to allow for extensions to simple globing, which could make the FTP module more useful, but, of course, client support would be lacking. But really, again, I repeat that it's no big deal to ensure it runs on both. Even if we simply just create ftp_match_glob() from apr_match_glob(). And, fwiw, mod_ftp was originally written *for* 2.0.x, not 2.2.x. So most of the work had been in getting it to work with the current httpd. But if anyone does anything to prevent mod_ftp from building on 2.0.x, expect a big -1. In fact, to be honest, it would be easier still to just update ftp_direntry_get() to use apr_fnmatch(), since we always want to support globing. ftp_direntry_get already does most of what makes apr_match_glob attractive in the 1st place.
Re: mod_ftp, status and progress?
On Apr 28, 2007, at 1:40 PM, Jim Jagielski wrote: In fact, to be honest, it would be easier still to just update ftp_direntry_get() to use apr_fnmatch(), since we always want to support globing. ftp_direntry_get already does most of what makes apr_match_glob attractive in the 1st place. Should have a patch to commit later on tomorrow, after some more tests :)
Re: mod_ftp, status and progress?
On May 3, 2007, at 4:37 AM, William A. Rowe, Jr. wrote: Niklas Edmundsson wrote: On Wed, 2 May 2007, Jim Jagielski wrote: In fact, to be honest, it would be easier still to just update ftp_direntry_get() to use apr_fnmatch(), since we always want to support globing. ftp_direntry_get already does most of what makes apr_match_glob attractive in the 1st place. Should have a patch to commit later on tomorrow, after some more tests :) I suspect that you're fixing the large file issues while you're at it? Another thing I noticed when we started to look at mod_ftp (looking at strace/truss-output trying to figure out why things didn't work) was that it stats all entries in a directory twice, first explicitly and then via the subreq. Wouldn't the subreq be enough? It's no biggie for now, but it would be nice to get rid of unneccessary stats as a bonus ;) This is a separate issue; we need to refactor out 90% of the subrequests and treat these at top level requests. Agreed... Some of them, of course, we can't get rid of, and the current design was basically for ease of development.
Re: mod proxy disabling workers after a single error
On May 4, 2007, at 11:37 AM, Brian Hayward wrote: I have 2 questions: 1) What are the negative implications of disabling this? 2) Is there a cleaner way to accomplish this? So you just want to setup Apache so that even if it thinks there's an error, to just ignore it?
Re: mod proxy disabling workers after a single error
Well, I doubt completely bypassing setting works to being in the error state is something you want to do lightly. :) PROXY_WORKER_IGNORE_ERRORS is used when setting up the generic forward and reverse workers, since they are shared for all requests and not specific to a balancer/url. I guess allowing someone to set that as an option might be a potential enhancement... On May 4, 2007, at 2:16 PM, Brian Hayward wrote: Yea, as it currently stands, one timeout is causing us to lose up to 10 more transactions during the next second (with retry=1) Thanks, Brian Hayward On 5/4/07, Jim Jagielski [EMAIL PROTECTED] wrote: On May 4, 2007, at 11:37 AM, Brian Hayward wrote: I have 2 questions: 1) What are the negative implications of disabling this? 2) Is there a cleaner way to accomplish this? So you just want to setup Apache so that even if it thinks there's an error, to just ignore it?
CHANGES file for 1.3 and 2.x
Seems to me that the more we work on the various 2.x trees (2.0.x, 2.2.x and trunk), the harder it becomes to get the various correct CHANGES entries in sync... For example, the CHANGES for 2.2 and trunk just refer to changes up to 2.0.56... What's the best way of syncing these? Should we stop having a single CHANGES that tries to merge all development together? Why not have something like: o Apache 1.3.x: CHANGES stay as is o Apache 2.0.x: CHANGES just incorporates 2.0.x work and we can either URL refer to older changes at the bottom of the file or, when we release, merge them. o Apache 2.2.x: CHANGES just notes 2.2.x work and we either refer to 2.0.x CHANGES and 1.3.x CHANGES or auto-merge when we release. o : Same pattern... Comments? Ideas?
Re: svn commit: r536291 - in /httpd/httpd/branches/2.2.x: STATUS modules/proxy/proxy_util.c
On May 9, 2007, at 8:18 AM, Roy T. Fielding wrote: On May 8, 2007, at 11:25 AM, [EMAIL PROTECTED] wrote: +#define USE_ALTERNATE_IS_CONNECTED 1 + +#if !defined(APR_MSG_PEEK) defined(MSG_PEEK) +#define APR_MSG_PEEK MSG_PEEK +#endif + +#if USE_ALTERNATE_IS_CONNECTED defined(APR_MSG_PEEK) Huh? Why are we polluting macro space with useless defines? The USE_ALTERNATE_IS_CONNECTED is to allow that section to be bypassed (although it should be protected by a ifndef USE_ALTERNATE_IS_CONNECTED to allow compile time changes I'm guessing). The APR_MSG_PEEK is so when APR is updated to reflect the existence of APR_MSG_PEEK, we don't need to touch this code. Finally, the older version assumed that MSG_PEEK was defined and the compile would barf if not; this avoid that.
Re: svn commit: r536291 - in /httpd/httpd/branches/2.2.x: STATUS modules/proxy/proxy_util.c
On May 9, 2007, at 8:55 AM, Roy T. Fielding wrote: On May 9, 2007, at 5:32 AM, Jim Jagielski wrote: On May 9, 2007, at 8:18 AM, Roy T. Fielding wrote: On May 8, 2007, at 11:25 AM, [EMAIL PROTECTED] wrote: +#define USE_ALTERNATE_IS_CONNECTED 1 + +#if !defined(APR_MSG_PEEK) defined(MSG_PEEK) +#define APR_MSG_PEEK MSG_PEEK +#endif + +#if USE_ALTERNATE_IS_CONNECTED defined(APR_MSG_PEEK) Huh? Why are we polluting macro space with useless defines? The USE_ALTERNATE_IS_CONNECTED is to allow that section to be bypassed (although it should be protected by a ifndef USE_ALTERNATE_IS_CONNECTED to allow compile time changes I'm guessing). Well, that would at least give it some reason to exist. If we know that new function works, then remove the defines and the old function. If it only works when APR_MSG_PEEK is known by APR, then that is the only #if needed (and we should not need to define our own). If we don't know if it works, then -1 on the backport. It's used in mod_jk (a variant, at least)... I've seen no issues on my testing... I agree that making it conditional isn't 100% correct, but it's how it's done in trunk so far and the backport proposed the exact diffs. The APR_MSG_PEEK is so when APR is updated to reflect the existence of APR_MSG_PEEK, we don't need to touch this code. Finally, the older version assumed that MSG_PEEK was defined and the compile would barf if not; this avoid that. Well, the code won't work anyway if APR doesn't know about MSG_PEEK. +status = apr_socket_recvfrom(unused, socket, APR_MSG_PEEK, + buf[0], len); presumably means something to APR, since otherwise we will lose data. The previous version passed just MSG_PEEK, which worked, but was cumbersome. Again, the idea is that apr_socket_recvfrom() really should know about some basic recvfrom() flags, but right now it doesn't; well, know about the same way that apr_poll() knows about POLLIN/APR_POLLIN for example. So right now, since apr_socket_recvfrom() does not, we simply set APR_MSG_PEEK to MSG_PEEK and be done with it. I'm fine with saying until apr_socket_recvfrom() is updated to support MSG_PEEK we leave this out. I'm even more fine in saying until then we leave this in but don't make it the default (or use some configure magic to determine the default). But it seemed like a cool feature in trunk, and a good candidate for backport
Re: ProxyPass globbing
On May 11, 2007, at 5:23 PM, Mladen Turk wrote: Jim Jagielski wrote: 1. ProxyPass /foo/* balancer://bar should silently rewrite itself to ProxyPass /foo/ balancer://bar In other words, we already assume a prefix glob. But should we? In other words, there is no difference between /foo/ and /foo/*. If you are trying to mimic the mod_jk behavior Actually, I'm not since it would break existing ProxyPass usage... This should be of course the balancer://bar/i/j/jim.gif Once again, if you wish to mimic the JkMount. Yes, that's my thoughts... However I'm not sure how would ProxyPassReverse fit in there. And of course once you introduce wild char maps there must be some sort of exclusion rules. ProxyPass already has that.
Re: ProxyPass globbing
On May 11, 2007, at 5:39 PM, Nick Kew wrote: On Fri, 11 May 2007 15:19:58 -0400 Jim Jagielski [EMAIL PROTECTED] wrote: I've been looking at extending ProxyPass to accept and use globbing patterns (basically, to make it easier for those migrating from mod_jk and JkMount to Apache 2.2), and it's almost trivially easy, but there are some gotchas: Two half-thoughts: Is there a strong reason to prefer globs over regexps? Regex's are almost as trivially easy to add but 1. slightly more complex structure 2. minor bump to proxy_alias struct A regexp doesn't have to be more complex than a glob, but gives the user who wants it that bit more hack-fu, and enables things people will want like ProxyPass \.(gif|jpe?g|png)$ http://backend/ Second, putting ProxyPass in a Directory context accomplishes the same thing more simply by enabling LocationMatch on it. Except, of course, that this does not work :)
Re: ProxyPass globbing
On May 11, 2007, at 5:56 PM, Ruediger Pluem wrote: On 05/11/2007 11:23 PM, Mladen Turk wrote: And of course once you introduce wild char maps there must be some sort of exclusion rules. ProxyPass /*/foo/* balancer://bar then for example by extending ProxyBlock ProxyBlock /*/foo/*.gif balancer://bar or by having an ! in front of path ProxyPass !/*/foo/*.gif balancer://bar Do we really need this? As we know from Rich's lightning talk at the ApacheCon every directive can be replaced by a rewrite rule :-). Except that mod_rewrite is not balancer aware... so using rewrite rules uses the default reverse proxy work, which, I think, diminishes its usefullness. Seriously, I wouldn't put too much effort in this. If we can get some globbing easily and ease some of the migrations from mod_jk to mod_proxy then this is cool. But IMHO if someone has more complex stuff to solve he should use mod_rewrite. It seems to me like the natural tool for this. Maybe we should remove something of the 'voodoo aura' of mod_rewrite. Not only for mod_jk people but for anyone... Imagine wanting to proxy all *.gif files to a proxy balancer; You have a cluster of image servers. mod_rewrite doesn't allow you to efficient use the balancer and the normal Directory|Location|Files regex containers don't work with ProxyPass... The existing ProxyPass logic expects the URL space (either as the 1st argument to ProxyPass or as obtained from the Directory container) to be a simple URL prefix, not any sort of glob/regex... This seems a major disadvantage to me :)
Re: ProxyPass globbing
On May 12, 2007, at 3:46 AM, Ruediger Pluem wrote: Do we really need this? As we know from Rich's lightning talk at the ApacheCon every directive can be replaced by a rewrite rule :-). Except that mod_rewrite is not balancer aware... so using rewrite rules uses the default reverse proxy work, which, I think, diminishes its usefullness. Sorry, but this is not true. You can do RewriteRule ^/somewhere(.*) balancer://mycluster/somewhereelse$1 [P] You're right I misspoke regarding the lameness of mod_rewrite :) Since mod_rewrite just tucks a 'proxy:' in front, mod_proxy does the right thing... My mistake. Still, just because mod_rewrite allows it doesn't mean we shouldn't allow ProxyPass to do what's right. ;)
Re: ProxyPass globbing
On May 12, 2007, at 10:09 AM, Jim Jagielski wrote: Still, just because mod_rewrite allows it doesn't mean we shouldn't allow ProxyPass to do what's right. ;) in any case, added to trunk... next step is to also make the Directory container method work...
Re: svn commit: r537429 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h
On May 12, 2007, at 12:45 PM, Joshua Slive wrote: On 05/12/2007 04:12 PM, [EMAIL PROTECTED] wrote: Author: jim Date: Sat May 12 07:12:24 2007 New Revision: 537429 URL: http://svn.apache.org/viewvc?view=revrev=537429 Log: Add regex pattern matching to ProxyPass, allowing, for example: ProxyPass ~ \.gif balancer://imagecluster On the configuration consistency side, the ~ thing is deprecated everywhere and the standard way to do this is to have a ProxyPassMatch directive (see RedirectMatch, DirectoryMatch, etc). Yep. The ProxyPassMatch stuff is currently being worked (basically, some function name changes ala add_proxy_regex) but I was hoping to work the support for DirectoryMatch .. into that patch as well before committing... but most likely I'll just add in ProxyPassMatch first.
Re: ProxyPass globbing
On May 12, 2007, at 11:19 AM, Ruediger Pluem wrote: On 05/12/2007 04:15 PM, Jim Jagielski wrote: On May 12, 2007, at 10:09 AM, Jim Jagielski wrote: Still, just because mod_rewrite allows it doesn't mean we shouldn't allow ProxyPass to do what's right. ;) in any case, added to trunk... next step is to also make the Directory container method work... While we are at it: It would be useful if the backreferences from the fake regex could be used on the real side. But of course then we are very close to mod_rewrite :-). Agreed on both... But then, wouldn't it require that the simple case be made more complex. ie: ProxyPass (/.*\.gif)$ balancer://foo$1 Instead of just: ProxyPass \.gif$ balancer://foo Otherwise the code would need to be a bit more complex to see if the substitution worked and, if not, assume that there was no requested substitution, if you follow me... And that seemed controversial enough to me to want to avoid it (for fear of the whole concept of adding globbing/regex support to ProxyPass being vigorously debated :) ). I think starting simple and then expanding it is safer if you actually want an enhancement to be folded in. Lessons learned from 11 years of httpd development ;)
Re: ProxyPass globbing
On May 13, 2007, at 1:01 AM, Paul Querna wrote: Well... I think we should consider a more generic way to do this. What I would like to see is something like AddHandler gif proxy=balancer://bar If mod_proxy knew about a handler syntax like this, you could do lots of creative things, using SetHandler DirectoryMatch /foo/bar*cats/ Set proxy=balancer://bar /DirectoryMatch Which to me, seems like an easier thing to explain. There is one way to set handlers.. and thats all everything uses. Thoughts? I like it...
Re: svn commit: r537429 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h
On May 12, 2007, at 11:10 AM, Ruediger Pluem wrote: Sorry for being picky here: One space too much. ... Again I am picky here: Too many spaces. No worries. Good catches.
Re: svn commit: r537429 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h
On May 13, 2007, at 3:07 PM, Ruediger Pluem wrote:
On 05/12/2007 05:10 PM, Ruediger Pluem wrote:
On 05/12/2007 04:12 PM, [EMAIL PROTECTED] wrote:
Author: jim
Date: Sat May 12 07:12:24 2007
New Revision: 537429
URL: http://svn.apache.org/viewvc?view=revrev=537429
Log:
Add regex pattern matching to ProxyPass, allowing,
for example:
ProxyPass ~ \.gif balancer://imagecluster
URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/proxy/
mod_proxy.h?view=diffrev=537429r1=537428r2=537429
==
--- httpd/httpd/trunk/modules/proxy/mod_proxy.h (original)
+++ httpd/httpd/trunk/modules/proxy/mod_proxy.h Sat May 12
07:12:24 2007
@@ -109,6 +109,7 @@
struct proxy_alias {
const char *real;
const char *fake;
+ap_regex_t *regex;
};
Doesn't this require a minor bump?
What about the minor bump? Is it not needed because there is no
typedef
for struct proxy_alias and thus struct proxy_alias is regarded as
private?
Yep... at least I don't think we need one since it's not
a publicly used struct.
Re: ftp glob/limits?
On May 14, 2007, at 4:18 PM, William A. Rowe, Jr. wrote:
What would folks think about changing
if (ap_strchr_c(arg, '*') != NULL) {
/* Prevent DOS attacks, only allow one segment to have a
wildcard */
int found = 0; /* The number of segments with a
wildcard */
to permit multiple wildcards, but to restrict the number of matches
returned (configurable with a directive, of course)?
Over a small pattern space, uploads/*/* is often very useful.
What would be the sane default? 1,000 entries?
In my mind, that's the problem. If you allow multiple wildcards
then you shouldn't limit the returned entries, because how
does the client have any idea that you've done so...
In other words, how does it know that foo.java doesn't
exist because it really doesn't exist or rather it
would have been the 1001st entry :)
Re: ProxyTimeout does not work as documented
On May 19, 2007, at 3:22 PM, Ruediger Pluem wrote: On 05/19/2007 04:07 PM, Eric Covener wrote: On 5/18/07, Ruediger Pluem [EMAIL PROTECTED] wrote: Currently ProxyTimeout does not work as documented as the default value is not 300 secs, but the Timeout setting of the server. The question to me is now: What should be fixed? - Documentation (such that it matches the code) - Code (such that it matches the documentation) Acting like a connection timeout only for me proxying HTTP on 2.2.4. I think I've read about similiar befuddlements on assorted PRs. It is raised a secondary issue here: http://issues.apache.org/bugzilla/show_bug.cgi?id=11540 I know :-). This is the next issue I want to address once the issue above is solved. In 2.2.x / trunk ProxyTimeout is ignored almost completely (it is only used for CONNECT). Workers either use their own timeout set via the worker timeout parameter or they use the server timeout as default, if no worker timeout is set. Although this (nearly) works as documented I plan to change this to let the workers use the ProxyTimeout setting as a default value in the case that they do not have their own timeout set via a parameter. This sounds a lot more sane to me instead of using the server timeout here as a default value. The logic should be: 1. If a per-worker value is set, use that. 2. If not, then if a ProxyTimeout value is set, use that. 3. Otherwise, use Timeout +1 on fixing that :)
Re: Any progress on PR41230 (HEAD issues on cached items)?
On May 18, 2007, at 5:26 PM, Justin Erenkrantz wrote:
On 5/18/07, Ruediger Pluem [EMAIL PROTECTED] wrote:
Well, because rv == !OK, wouldn't the CACHE_REMOVE_URL filter hit?
That should do the dirty deed, no? -- justin
No, as the CACHE_REMOVE_URL filter will only work if there is a
cache-handle or a cache-stale_handle. We have neither, as cache-
stale_handle
is set to NULL in the case that the cached entity is *really*
stale and we do
not create a new entity (and thus a cache-handle) in the HEAD case.
Well, let's not clear cache-stale_handle then. How does the
following look? -- justin
+1 !
Index: modules/cache/mod_cache.c
===
--- modules/cache/mod_cache.c (revision 539607)
+++ modules/cache/mod_cache.c (working copy)
@@ -477,8 +477,10 @@
reason = No Last-Modified, Etag, or Expires headers;
}
else if (r-header_only) {
-/* HEAD requests */
-reason = HTTP HEAD request;
+/* Forbid HEAD requests unless we have it cached already */
+if (!cache-stale_handle) {
+reason = HTTP HEAD request;
+}
}
else if (!conf-store_nostore
ap_cache_liststr(NULL, cc_out, no-store, NULL)) {
@@ -596,7 +598,12 @@
* the headers).
*/
-/* Did we have a stale cache entry that really is stale? */
+/* Did we have a stale cache entry that really is stale?
+ *
+ * Note that for HEAD requests, we won't get the body, so for
a stale
+ * HEAD request, we don't remove the entity - instead we let the
+ * CACHE_REMOVE_URL filter remove the stale item from the cache.
+ */
if (cache-stale_handle) {
if (r-status == HTTP_NOT_MODIFIED) {
/* Oh, hey. It isn't that stale! Yay! */
@@ -604,7 +611,7 @@
info = cache-handle-cache_obj-info;
rv = OK;
}
-else {
+else if (!r-header_only) {
/* Oh, well. Toss it. */
cache-provider-remove_entity(cache-stale_handle);
/* Treat the request as if it wasn't conditional. */
@@ -612,8 +619,8 @@
}
}
-/* no cache handle, create a new entity */
-if (!cache-handle) {
+/* no cache handle, create a new entity only for non-HEAD
requests */
+if (!cache-handle !r-header_only) {
rv = cache_create_entity(r, size);
info = apr_pcalloc(r-pool, sizeof(cache_info));
/* We only set info-status upon the initial creation. */
Re: [Issue] External links @ the wiki, aka pagechange wars
On May 24, 2007, at 8:50 AM, Colm MacCarthaigh wrote: On Thu, May 24, 2007 at 08:05:30AM -0400, Joshua Slive wrote: External links are encouraged where they add substantial value, but you may not link to your own pages or otherwise seek private benefits from external links. I like the elegance of this rule, because if it's your page and you words, well you can just add the content to the wiki anyway. But at the same time it may invite even more awkward and inappropriate behaviour, e.g. paying someone else to add the links on your behalf. I think this problem is always going to fall into the We know abuse when we see it category, it requires a vague kind of rule-making which only humans can apply. I'm in favour of banning these links in this instance, though not all external links. +1
Re: [vote] Piped loggers and APR_SHELLCMD_ENV
On May 24, 2007, at 4:04 AM, Sander Temme wrote: On May 23, 2007, at 4:39 PM, William A. Rowe, Jr. wrote: [ ] Revert to |foo to invoke foo, and add |$foo syntax to launch foo via sh I like this one the best, since it consumes fewest resources in the default case. [ ] Retain |foo to invoke foo through sh, and add ||foo syntax to directly launch foo These were the best solutions I could come up with, others are welcome. However, the indirection through the shell is now the default case and has been for quite a while... I figure dropping it now might violate the principle of least astonishment. I prefer the first one, but given the circumstances see myself tending towards the second alternative. +1...
Re: [Fwd: Apache httpd vulenrabilities]
On May 29, 2007, at 5:28 PM, William A. Rowe, Jr. wrote: Essentially, PID tables need to move from the score to a local process list only in the parent, and unshared. That would solve the 80/20 of this entire class of issues. Yes... Of course, it doesn't even need to be that extensive. If the parent process simply keeps in local storage a list of PIDs and before it does anything to the child process (send signal), it checks that the PID in the scoreboard is also in its own list, and only continues if it is This means that the scoreboard stays as is and the parent process just needs a small list of pid's added to it, plus some minor logic. The next coupla days I'll be mostly offline since my oldest son Jon is graduating and so there's the graduation, and planning, and party, etc...
Re: httpd 1.3 / 2.0 / 2.2 tags this weekend?
On May 30, 2007, at 1:56 AM, William A. Rowe, Jr. wrote: I'd like to see new tarballs rolled soonish, given the single significant bug that was disclosed earlier today. Obviously most mass-vhosters are capable of compiling their own binary, so providing the seperate-pid-table patch (whoever gets around to writing one) resolves any immediate urgency. But people get skittish when they see httpd anywhere near vulnerability, so I'll roll apr 0.9/1.2 in 36 hours which means midday Sunday it's likely to be released and ready to drop into 2.0 / 2.2. 1.3 could be rolled/released whenever it's been patched, but I'd personally rather see *one* announcement, again, like we did about a year back, so we don't have silly people scrambling to install 1.3 in place of 2.2 :) I volunteer to roll 1.3 when it's ready, since Sander offered to roll 2.2 (and perhaps 2.0?) Sounds good. I can likely take a look at adding the parent has local pid table and verifies scoreboard with it patch to 1.3 maybe over the weekend (the 2.0 and 2.2 will likely follow the same concept), unless someone beats me to it :)
Re: [Fwd: Apache httpd vulenrabilities]
On May 30, 2007, at 2:57 PM, Ruediger Pluem wrote: On 05/30/2007 08:10 PM, Jim Jagielski wrote: On May 29, 2007, at 5:28 PM, William A. Rowe, Jr. wrote: Essentially, PID tables need to move from the score to a local process list only in the parent, and unshared. That would solve the 80/20 of this entire class of issues. Yes... Of course, it doesn't even need to be that extensive. If the parent process simply keeps in local storage a list of PIDs and before it does anything to the child process (send signal), it checks that the PID in the scoreboard is also in its own list, and only continues if it is This means that the scoreboard stays as is and the parent process just needs a small list of pid's added to it, plus some minor logic. This is also my thought on this. Maybe the logic could be extended somewhat so that the parent cross checks / sanity checks the number of pids in its local storage and the number of active pids (a.k.a non empty slots) in whatever state they are every time it decides that it needs to start additional childs / kill childs. This should avoid / reduce issues #1 / #4. :) My thoughts exactly... The only issue with pulling parent out of scoreboard is, of course, the backwards compatibility with modules that may be interested in it...
Re: httpd 1.3 / 2.0 / 2.2 tags this weekend?
On May 30, 2007, at 2:41 PM, Sander Temme wrote: On May 29, 2007, at 10:56 PM, William A. Rowe, Jr. wrote: I volunteer to roll 1.3 when it's ready, since Sander offered to roll 2.2 (and perhaps 2.0?) I'll be happy to RM both. I'd like to, but my time will be sporadic enough the next few days as to make it unwise to depend on me to do this :)
Re: [Fwd: Apache httpd vulenrabilities]
Ruediger Pluem wrote: On 05/30/2007 09:45 PM, Jim Jagielski wrote: On May 30, 2007, at 2:57 PM, Ruediger Pluem wrote: On 05/30/2007 08:10 PM, Jim Jagielski wrote: On May 29, 2007, at 5:28 PM, William A. Rowe, Jr. wrote: Essentially, PID tables need to move from the score to a local process list only in the parent, and unshared. That would solve the 80/20 of this entire class of issues. Yes... Of course, it doesn't even need to be that extensive. If the parent process simply keeps in local storage a list of PIDs and before it does anything to the child process (send signal), it checks that the PID in the scoreboard is also in its own list, and only continues if it is This means that the scoreboard stays as is and the parent process just needs a small list of pid's added to it, plus some minor logic. This is also my thought on this. Maybe the logic could be extended somewhat so that the parent cross checks / sanity checks the number of pids in its local storage and the number of active pids (a.k.a non empty slots) in whatever state they are every time it decides that it needs to start additional childs / kill childs. This should avoid / reduce issues #1 / #4. :) My thoughts exactly... The only issue with pulling parent out of scoreboard is, of course, the backwards compatibility with modules that may be interested in it... Sorry for being confused now. I thought the idea was to have an additional local pid list in the parent for cross / sanity checking. The scoreboard remains as it is (including the pids). So what could modules possible miss after the patch? Not sure where the confusion is... the idea is to keep the scoreboard exactly as-is (as noted above, pulling out the parent parts (pid), would be a major issue for backwards compatibility), and just add pid-checking logic (and a local store of pids) in the parent process. Thus, it's completely transparent and no changes to the scoreboard or any entities that use it (except for the mentioned parent-side checking, 'natch) :) The only issue... refers to the problems if we try to restructure the scoreboard instead, which is good for 2.4/3.0 but not for 2.2, 2.0 and 1.3... Any patches that tried to address the issue using that method would be problematic. Hence my thoughts to just have local storage for checking and keeping scoreboard as-is. -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ If you can dodge a wrench, you can dodge a ball.
Re: [Fwd: Apache httpd vulenrabilities]
For 1.3, I'm looking at something like this...
Similar approach for 2.x...
Comments and feedback appreciated before I work on
porting to the 2.x trees:
Index: main/http_main.c
===
--- main/http_main.c(revision 543486)
+++ main/http_main.c(working copy)
@@ -354,9 +354,17 @@
char tpf_mutex_key[TPF_MUTEX_KEY_SIZE];
#endif /* TPF */
+/*
+ * Shared memory scoreboard
+ */
scoreboard *ap_scoreboard_image = NULL;
/*
+ * Parent process local storage of child pids
+ */
+static table *pid_table;
+
+/*
* Pieces for managing the contents of the Server response header
* field.
*/
@@ -372,6 +380,26 @@
API_VAR_EXPORT int ap_change_shmem_uid = 0;
/*
+ * Check the pid table to see if the actual pid exists
+ */
+static int in_pid_table(int pid) {
+char apid[64];
+const char *spid;
+snprintf(apid, sizeof(apid), %d, pid);
+spid = ap_table_get(pid_table, apid);
+if (spid spid[0] == '1' spid[1] == '\0')
+return 1;
+else
+return 0;
+}
+
+static void set_pid_table(int pid) {
+char apid[64];
+snprintf(apid, sizeof(apid), %d, pid);
+ap_table_set(pid_table, apid, 1);
+}
+
+/*
* This routine is called when the pconf pool is vacuumed. It
resets the
* server version string to a known value and [re]enables
modifications
* (which are disabled by configuration completion).
@@ -2787,6 +2815,11 @@
if (pid == my_pid || pid == 0)
continue;
+if (!in_pid_table(pid)) {
+ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR,
server_conf,
+ Bad pid (%d) in scoreboard slot %d,
pid, i);
+continue;
+}
waitret = waitpid(pid, status, WNOHANG);
if (waitret == pid || waitret == -1) {
ap_scoreboard_image-parent[i].pid = 0;
@@ -2887,13 +2920,21 @@
for (n = 0; n max_daemons_limit; ++n) {
ap_sync_scoreboard_image();
- if (ap_scoreboard_image-servers[n].status != SERVER_DEAD
- kill((pid = ap_scoreboard_image-parent[n].pid), 0) == -1) {
- ap_update_child_status(n, SERVER_DEAD, NULL);
- /* just mark it as having a successful exit status */
- bzero((char *) status, sizeof(ap_wait_t));
- return(pid);
- }
+pid = ap_scoreboard_image-parent[n].pid;
+if (ap_scoreboard_image-servers[n].status != SERVER_DEAD) {
+if (in_pid_table(pid)) {
+if (kill(pid, 0) == -1) {
+ap_update_child_status(n, SERVER_DEAD, NULL);
+/* just mark it as having a successful exit
status */
+bzero((char *) status, sizeof(ap_wait_t));
+return(pid);
+}
+}
+else {
+ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR,
server_conf,
+Bad pid (%d) in scoreboard slot %d,
pid, n);
+}
+}
}
return 0;
}
@@ -2916,15 +2957,21 @@
#define MAXWAITOBJ MAXIMUM_WAIT_OBJECTS
HANDLE h[MAXWAITOBJ];
int e[MAXWAITOBJ];
-int round, pi, hi, rv, err;
+int round, pi, hi, rv, err, pid;
for (round = 0; round = (HARD_SERVER_LIMIT - 1) / MAXWAITOBJ +
1; round++) {
hi = 0;
for (pi = round * MAXWAITOBJ;
(pi (round + 1) * MAXWAITOBJ) (pi HARD_SERVER_LIMIT);
pi++) {
if (ap_scoreboard_image-servers[pi].status != SERVER_DEAD) {
- e[hi] = pi;
- h[hi++] = (HANDLE) ap_scoreboard_image-parent[pi].pid;
+e[hi] = pi;
+pid = ap_scoreboard_image-parent[pi].pid;
+if (in_pid_table(pid))
+h[hi++] = (HANDLE) pid;
+else {
+ap_log_error(APLOG_MARK, APLOG_NOERRNO|
APLOG_ERR, server_conf,
+ Bad pid (%d) in scoreboard slot %
d, pid, pi);
+}
}
}
@@ -4339,6 +4386,7 @@
ap_server_pre_read_config = ap_make_array(pcommands, 1, sizeof
(char *));
ap_server_post_read_config = ap_make_array(pcommands, 1, sizeof
(char *));
ap_server_config_defines = ap_make_array(pcommands, 1, sizeof
(char *));
+pid_table = ap_make_table(pglobal, 100);
}
#ifndef MULTITHREAD
@@ -4987,6 +5035,7 @@
ap_scoreboard_image-parent[slot].last_rtime = now;
#endif
ap_scoreboard_image-parent[slot].pid = pid;
+set_pid_table(pid);
#ifdef SCOREBOARD_FILE
lseek(scoreboard_fd, XtOffsetOf(scoreboard, parent[slot]), 0);
force_write(scoreboard_fd, ap_scoreboard_image-parent[slot],
@@ -5049,6 +5098,7 @@
int i;
int to_kill;
int idle_count;
+int pid;
short_score *ss;
time_t now = time(NULL);
int free_length;
@@ -5113,8 +5163,15 @@
else if (ps-last_rtime + ss-timeout_len now) {
Re: [Fwd: Apache httpd vulenrabilities]
On Jun 1, 2007, at 10:19 AM, Colm MacCarthaigh wrote:
On Fri, Jun 01, 2007 at 10:05:26AM -0400, Jim Jagielski wrote:
- if (ap_scoreboard_image-servers[n].status != SERVER_DEAD
- kill((pid = ap_scoreboard_image-parent[n].pid), 0) == -1) {
- ap_update_child_status(n, SERVER_DEAD, NULL);
- /* just mark it as having a successful exit status */
- bzero((char *) status, sizeof(ap_wait_t));
- return(pid);
- }
+pid = ap_scoreboard_image-parent[n].pid;
+if (ap_scoreboard_image-servers[n].status != SERVER_DEAD) {
+if (in_pid_table(pid)) {
+if (kill(pid, 0) == -1) {
+ap_update_child_status(n, SERVER_DEAD, NULL);
+/* just mark it as having a successful exit
status */
+bzero((char *) status, sizeof(ap_wait_t));
+return(pid);
+}
Should we get rid of it from the table here? Can we get away without
removing stale pids in general? What if they are recycled by the OS
for something else?
No, that's a good point. We should likely remove the
pid from our table once the child goes away.
Re: [Fwd: Apache httpd vulenrabilities]
On Jun 1, 2007, at 10:45 AM, Colm MacCarthaigh wrote: On Fri, Jun 01, 2007 at 10:50:09AM -0400, Jim Jagielski wrote: Should we get rid of it from the table here? Can we get away without removing stale pids in general? What if they are recycled by the OS for something else? No, that's a good point. We should likely remove the pid from our table once the child goes away. I think we need to do it with an actual table unset call too, not just set the entry to 0 or whatever, so that we don't exhaust our table. Yes, agreed. On that point, instead of initialising with ap_make_table(pglobal, 100) , it should probably use HARD_SERVER_LIMIT (if that's not in scope, it may even justify making it in scope) instead of 100 too, to avoid potential immediate exhaustion. Nice simple fix though, using a simple table seems like a way better approach than trying to replicate a paralell scoreboard. I will likely just commit the updated patch, and we can fine-tune via commits rather than having rounds of patches posted :)
Re: [Fwd: Apache httpd vulenrabilities]
On Jun 1, 2007, at 11:39 AM, Jim Jagielski wrote: I will likely just commit the updated patch, and we can fine-tune via commits rather than having rounds of patches posted :) I just started on the trunk patches, not sure when they will be done... anyway, I was think that in addition to the checks, it would also be useful if, instead of just logging the issue, maybe the code should also set the pid to 0 and maybe a different status (maybe SERVER_UNKNOWN ?) Haven't thought this all the way through yet. As one would expect the 2.x patches are much larger, since things aren't as centralized...
Re: [Fwd: Apache httpd vulenrabilities]
On Jun 1, 2007, at 3:35 PM, Jim Jagielski wrote: FWIW, I've created a branch of the work in progress, so people can follow along and provide comments and patches :) http://svn.apache.org/repos/asf/httpd/httpd/branches/httpd-pid- table this is based off of trunk, so once we have this where we want it, we'll backport to 2.2 and 2.0
Re: svn commit: r543511 - /httpd/httpd/branches/1.3.x/src/main/http_main.c
Ruediger Pluem wrote: For my understanding (and a bit of devils advocate here :-)): Why do we use a table here and not a fixed size array (HARD_SERVER_LIMIT) of ints (apr_array of pid_t in the 2.x case). If we keep the pids at the same index as in the scoreboard the checks would be somewhat faster and simpler to do. Of course we waste more memory. We can afford the time taken I think, to save space and to be more efficient... I really dislike all that wasted space :) -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ If you can dodge a wrench, you can dodge a ball.
Re: svn commit: r543667 - /httpd/httpd/branches/httpd-pid-table/server/mpm/mpmt_os2/mpmt_os2.c
On Jun 2, 2007, at 3:57 AM, Ruediger Pluem wrote:
On 06/02/2007 01:44 AM, [EMAIL PROTECTED] wrote:
Author: jim
Date: Fri Jun 1 16:44:36 2007
New Revision: 543667
URL: http://svn.apache.org/viewvc?view=revrev=543667
Log:
Minor nit... be consistent and unset even now :)
Modified:
httpd/httpd/branches/httpd-pid-table/server/mpm/mpmt_os2/
mpmt_os2.c
Modified: httpd/httpd/branches/httpd-pid-table/server/mpm/mpmt_os2/
mpmt_os2.c
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/httpd-pid-
table/server/mpm/mpmt_os2/mpmt_os2.c?
view=diffrev=543667r1=543666r2=543667
=
=
--- httpd/httpd/branches/httpd-pid-table/server/mpm/mpmt_os2/
mpmt_os2.c (original)
+++ httpd/httpd/branches/httpd-pid-table/server/mpm/mpmt_os2/
mpmt_os2.c Fri Jun 1 16:44:36 2007
@@ -337,6 +337,7 @@
pid = ap_scoreboard_image-parent[n].pid;
if (ap_in_pid_table(pid)) {
kill(pid, is_graceful ? SIGHUP : SIGTERM);
+ap_unset_pid_table(pid);
Good catch. But is this also correct in the graceful / SIGHUP case?
Couldn't it happen that we want to sent a SIGTERM later?
Could be... It's been a LONG time since I looked at this
MPM :)
Re: svn commit: r543511 - /httpd/httpd/branches/1.3.x/src/main/http_main.c
On Jun 4, 2007, at 2:35 AM, David McCreedy wrote: On 06/01/2007 05:42 PM, [EMAIL PROTECTED] wrote: Author: jim Date: Fri Jun 1 08:42:57 2007 New Revision: 543511 URL: http://svn.apache.org/viewvc?view=revrev=543511 Log: Add in parent process PID table, to provide for a check against the pid values located in the scoreboard. Modified: httpd/httpd/branches/1.3.x/src/main/http_main.c I tried out the code in apache-1.3_20070603161656.tar.gz on TPF because I thought the pid_table stuff might affect the platform. Hopefully this is enough information to recreate the issues. If I left something out please let me know. Thanks for the testing... I'm unable to recreate the problems you're seeing, but at least I know what areas might be causing them... Thanks again!
Re: svn commit: r543511 - /httpd/httpd/branches/1.3.x/src/main/http_main.c
On Jun 4, 2007, at 2:35 AM, David McCreedy wrote: On 06/01/2007 05:42 PM, [EMAIL PROTECTED] wrote: Author: jim Date: Fri Jun 1 08:42:57 2007 New Revision: 543511 URL: http://svn.apache.org/viewvc?view=revrev=543511 Log: Add in parent process PID table, to provide for a check against the pid values located in the scoreboard. Modified: httpd/httpd/branches/1.3.x/src/main/http_main.c I tried out the code in apache-1.3_20070603161656.tar.gz on TPF because I thought the pid_table stuff might affect the platform. I think I squashed those. Could you check out trunk and try another test? Thanks!
Re: svn commit: r543511 - /httpd/httpd/branches/1.3.x/src/main/http_main.c
On Jun 4, 2007, at 10:29 PM, David McCreedy wrote: June 04, 2007 5:51 PM David McCreedy wrote: On 06/01/2007 05:42 PM, [EMAIL PROTECTED] wrote: I think I squashed those. Could you check out trunk and try another test? Thanks! It fixes the Bad pid error but I'm not sure all is well... On TPF we're not calling unset_pid_table on SIG_IDLE_KILLs. I'll have to track down why. I've figured out why some pids aren't being unset and I think it could affect other platforms besides TPF. They're hitting the else part of the if (child_slot = 0) statement in http_main.c's standalone_main function. I think the unset_pid_table call should be moved before this if. If we're in this section of code, don't we want to remove the pid from the table regardless of whether the slot is found in the scoreboard? Yes, in fact Rüdiger just committed a patch that does that.
pid_table (Was: Re: svn commit: r543511 - /httpd/httpd/branches/1.3.x/src/main/http_main.c)
It looks the the 1.3 pid_table impl is pretty much on target. I've been testing the trunk (2.3.x) version with no issues that I've been able to see, but was wondering how many others are testing... In the meantime, should I create a 2.2 branch for the 2.2-version of the pid_table code and backport the changes to that? Unless I hear otherwise, I'll likely do that since the backport from 2.2 to 2.0 shouldn't be that involved.
Re: [PATCH] 'clogging' input filters and the event MPM
+1 (concept)
On Jun 10, 2007, at 9:13 PM, Paul Querna wrote:
Attached is a patch that should let people run mod_ssl under the
Event MPM.
Previously, the event mpm would put a socket into its event thread to
wait for input, but due to issues with how mod_ssl might be buffering
data (or the entire request) in it, this would cause SSL'ized requests
to stall and fail with the event mpm.
The attach patch adds a new field, clogging_input_filters, to the
conn_rec. When this field is true, the event mpm will run a connection
just like the worker mpm is -- that is a single thread will run the
entire connection, including keepalives.
Thoughts on committing this to trunk?
Thanks,
-Paul
Index: server/mpm/experimental/event/event.c
===
--- server/mpm/experimental/event/event.c (revision 545614)
+++ server/mpm/experimental/event/event.c (working copy)
@@ -620,6 +620,16 @@
pt = cs-pfd.client_data;
}
+if (c-clogging_input_filters !c-aborted) {
+/* Since we have an input filter which 'cloggs' the input
stream,
+ * like mod_ssl, lets just do the normal read from input
filters,
+ * like the Worker MPM does.
+ */
+ap_run_process_connection(c);
+ap_lingering_close(c);
+return 0;
+}
+
read_request:
if (cs-state == CONN_STATE_READ_REQUEST_LINE) {
if (!c-aborted) {
Index: server/core.c
===
--- server/core.c (revision 545614)
+++ server/core.c (working copy)
@@ -3803,6 +3803,7 @@
c-cs-c = c;
c-cs-p = ptrans;
c-cs-bucket_alloc = alloc;
+c-clogging_input_filters = 0;
return c;
}
Index: modules/http/http_core.c
===
--- modules/http/http_core.c(revision 545614)
+++ modules/http/http_core.c(working copy)
@@ -119,11 +119,17 @@
return DEFAULT_HTTP_PORT;
}
+static int ap_process_http_connection(conn_rec *c);
+
static int ap_process_http_async_connection(conn_rec *c)
{
request_rec *r;
conn_state_t *cs = c-cs;
+if (c-clogging_input_filters) {
+return ap_process_http_connection(c);
+}
+
AP_DEBUG_ASSERT(cs-state == CONN_STATE_READ_REQUEST_LINE);
while (cs-state == CONN_STATE_READ_REQUEST_LINE) {
Index: modules/ssl/ssl_engine_io.c
===
--- modules/ssl/ssl_engine_io.c (revision 545614)
+++ modules/ssl/ssl_engine_io.c (working copy)
@@ -1665,6 +1665,9 @@
filter_ctx-pbioWrite = BIO_new(bio_filter_out_method);
filter_ctx-pbioWrite-ptr = (void *)bio_filter_out_ctx_new
(filter_ctx, c);
+/* We insert a clogging input filter. Let the core know. */
+c-clogging_input_filters = 1;
+
ssl_io_input_add_filter(filter_ctx, c, ssl);
SSL_set_bio(ssl, filter_ctx-pbioRead, filter_ctx-pbioWrite);
Index: include/httpd.h
===
--- include/httpd.h (revision 545614)
+++ include/httpd.h (working copy)
@@ -1081,6 +1081,11 @@
int data_in_input_filters;
/** Is there data pending in the output filters? */
int data_in_output_filters;
+
+/** Are there any filters that clogg/buffer the input stream,
breaking
+ * the event mpm.
+ */
+int clogging_input_filters;
};
/**
Re: pid_table (Was: Re: svn commit: r543511 - /httpd/httpd/branches/1.3.x/src/main/http_main.c)
On Jun 6, 2007, at 9:13 AM, Jim Jagielski wrote: In the meantime, should I create a 2.2 branch for the 2.2-version of the pid_table code and backport the changes to that? Unless I hear otherwise, I'll likely do that since the backport from 2.2 to 2.0 shouldn't be that involved. Done and done: passes tests. http://svn.apache.org/repos/asf/httpd/httpd/branches/httpd-2.2- pid-table Next to come is 2.0, but, IMO, we should start folding these in, ala 1.3
Re: pid_table (Was: Re: svn commit: r543511 - /httpd/httpd/branches/1.3.x/src/main/http_main.c)
Status Update: The pid-table code is: o Applied to 1.3 branch o In httpd-2.0-pid-table branch (branches/2.0.x fork) o In httpd-2.2-pid-table branch (branches/2.2.x fork) o In httpd-pid-table branch (trunk fork) Passes httpd-tests, as well as 'ab' with *very* small MaxRequestsPerChild and Max/MinSpare* settings to exercise the child fork/reap logic... so far, so good. Proposal is to fold them into their respective branches (or trunk) in anticipation of additional testing and release. Comments?
Re: PID table changes (was Re: svn commit: r547987 - in /httpd/httpd/trunk)
On Jun 21, 2007, at 1:18 PM, Colm MacCarthaigh wrote: On Thu, Jun 21, 2007 at 05:51:34PM +0100, Joe Orton wrote: On Sat, Jun 16, 2007 at 09:29:25PM -, Jim Jagielski wrote: Secondly: I think this approach is unnecessarily complex. I think it's sufficient to simply check whether the target process is in the right process group before sending a signal, i.e. getpgid(pid) == getpgrp (). This ensures that the parent will only kill things it created. I actually thought avoiding this was a design goal, to prevent someone from killing piped loggers and cgi processes ? Yes, that's the case.
Re: PID table changes (was Re: svn commit: r547987 - in /httpd/httpd/trunk)
On Jun 21, 2007, at 12:51 PM, Joe Orton wrote: Firstly my sincere apologies to Jim for bringing this up after such considerable work was put in already - I've had a busy month with a week out for a holiday :( Secondly: I think this approach is unnecessarily complex. I think it's sufficient to simply check whether the target process is in the right process group before sending a signal, i.e. getpgid(pid) == getpgrp(). This ensures that the parent will only kill things it created. If: 1. The required getpgid/getpgrp functions are available on all current systems, and... 2. it provides as much protection as the more... exacting... check, then I have no real push one way or another...
Re: PID table changes (was Re: svn commit: r547987 - in /httpd/httpd/trunk)
On Jun 21, 2007, at 6:20 PM, William A. Rowe, Jr. wrote: Joe Orton wrote: On Thu, Jun 21, 2007 at 06:18:59PM +0100, Colm MacCarthaigh wrote: On Thu, Jun 21, 2007 at 05:51:34PM +0100, Joe Orton wrote: On Sat, Jun 16, 2007 at 09:29:25PM -, Jim Jagielski wrote: Secondly: I think this approach is unnecessarily complex. I think it's sufficient to simply check whether the target process is in the right process group before sending a signal, i.e. getpgid(pid) == getpgrp(). This ensures that the parent will only kill things it created. I actually thought avoiding this was a design goal, to prevent someone from killing piped loggers and cgi processes ? What's the security threat there? Given that the attacker can arrange for arbitrary execution of code in any unprivileged child, preventing logging or CGI script execution is possible anyway. Two different cases. It will be trivial to kill the CGI scripts launched from the process (not from cgid) for other children in, say, a worker config. The cgid and logging processes spawned by the parents aren't vulnerable as such (and the logging processes are spawned with the httpd's launching uid anyways, not nobody, IIRC.) We only care about constraining the fallout of a malicious in process script to that specific process, and the loggers, if we go beyond testing the pid group, won't have that issue from the child process. So I agree with testing the pid groups, I also believe it is worthwhile to sanity test any critical data from the shared-scoreboard against the parent processes' private reference table. So what's the word... should we back out all the pid table stuff (from both 1.3 and 2.x) and wait for Joe to provide his pgrp changes (including any required configure magic to detect function existance) or what? It's kind of embarrassing that this is taking so long :)
Re: svn commit: r551171 - /httpd/httpd/branches/2.2.x/STATUS
On Jun 27, 2007, at 11:08 AM, Nick Kew wrote: On Wed, 27 Jun 2007 14:17:36 - [EMAIL PROTECTED] wrote: +* mod_proxy: Arrange the timeout handling. + Trunk version of patch: +http://svn.apache.org/viewvc?view=revrevision=550514 +http://svn.apache.org/viewvc?view=revrevision=546128 + +1: jfclere Looks reasonable, but is there a reference to the problem it solves? + +* mod_proxy: Improve traces in ap_proxy_http_process_response() + to investigate PR37770. + Trunk version of patch: +http://svn.apache.org/viewvc?view=revrev=549420 + +1: jfclere Hmmm. This is designed to improve an error message? +tmp_bb = apr_brigade_create(r-pool, r-connection- bucket_alloc); +rv = ap_rgetline(tmp_s, n, len, r, fold, tmp_bb); +apr_brigade_destroy(tmp_bb); Isn't a whole new brigade an unnecessarily overhead (and potentially large if the function gets used more in future)? What problem does it solve? Yeah... all this just so we can see the return val of ap_rgetline()??
Re: svn commit: r551171 - /httpd/httpd/branches/2.2.x/STATUS
On Jun 27, 2007, at 12:20 PM, Ruediger Pluem wrote: On 06/27/2007 05:51 PM, Jim Jagielski wrote: On Jun 27, 2007, at 11:08 AM, Nick Kew wrote: On Wed, 27 Jun 2007 14:17:36 - [EMAIL PROTECTED] wrote: +* mod_proxy: Arrange the timeout handling. + Trunk version of patch: +http://svn.apache.org/viewvc?view=revrevision=550514 +http://svn.apache.org/viewvc?view=revrevision=546128 + +1: jfclere Looks reasonable, but is there a reference to the problem it solves? + +* mod_proxy: Improve traces in ap_proxy_http_process_response() + to investigate PR37770. + Trunk version of patch: +http://svn.apache.org/viewvc?view=revrev=549420 + +1: jfclere Hmmm. This is designed to improve an error message? +tmp_bb = apr_brigade_create(r-pool, r-connection- bucket_alloc); +rv = ap_rgetline(tmp_s, n, len, r, fold, tmp_bb); +apr_brigade_destroy(tmp_bb); Isn't a whole new brigade an unnecessarily overhead (and potentially large if the function gets used more in future)? What problem does it solve? Yeah... all this just so we can see the return val of ap_rgetline()?? Yes, but have a look at ap_getline in protocol.c which was used before instead of ap_proxygetline. It does exactly the same thing regarding the brigade. I wasn't concerned about the brigade, but rather the extra layer of complexity just so we see a return value... Is it worth it?
Re: PID table changes (was Re: svn commit: r547987 - in /httpd/httpd/trunk)
On Jun 27, 2007, at 12:40 PM, Joe Orton wrote: Right: it adds overhead without benefit, and there is also a risk of regressions. e.g. in the trunk code it looks like children from ap_register_extra_mpm_process() don't go in the pid table, so the reclaim_one_pid() path to kill them won't work. IIRC, these are already in the pid table at creation time. ap_register_extra_mpm_process worries about existing processes that are in the slot we want to use, but since they are existing, their pid had already been stored.
Re: [PATCH] pid safety checks for 2.2.x
On Jun 27, 2007, at 3:38 PM, Ruediger Pluem wrote: Hm. Wouldn't it make sense to log this in the case waitret != APR_CHILD_DONE as in the PID table patches? This could give the admin a hint that something is rotten on his box. +1 on the logging... Looking forward to seeing the 1.3 patch...
Re: [PATCH] pid safety checks for 2.2.x
On Jun 27, 2007, at 1:52 PM, Joe Orton wrote: Here's the updated (and simpler) version of my patch which uses apr_proc_wait() to determine whether a pid is a valid child. Simplifies the MPM logic a bit since the pid != 0 check is moved into ap_mpm_safe_kill(). Tested for both prefork and worker (on Linux) to fix the vulnerability using mod_scribble: I might be missing this (just did a quick scan) but what about ap_reclaim_child_processes/reclaim_one_pid()? Here we trust the pid in the scoreboard and send signals.
Re: [PATCH] pid safety checks for 2.2.x
On Jun 28, 2007, at 7:56 AM, Joe Orton wrote: So, final comments on this? If there's consensus that this is the approach to take I'll revert the pidtable stuff out of trunk, commit this there, and propose the backport. Don't forget the 1.3 branch...
Re: [PATCH] pid safety checks for 2.2.x
On Jul 4, 2007, at 12:52 PM, Joe Orton wrote: On Thu, Jun 28, 2007 at 12:50:37PM -0400, Jim Jagielski wrote: On Jun 28, 2007, at 7:56 AM, Joe Orton wrote: So, final comments on this? If there's consensus that this is the approach to take I'll revert the pidtable stuff out of trunk, commit this there, and propose the backport. Don't forget the 1.3 branch... I've been trying to get a patch together for 1.3.x but the portability stuff keeps biting me - accurately detecting presence of getpgid is harder with the 1.3 build system even on Linux (it's hidden without _GNU_SOURCE defined, but helpers/TestCompile still detects it... etc). So I'd say stick with the existing pid-table stuff for 1.3 - I tested it with mod_scribble and it prevents the exploits tested there. I agree... I also tried porting the pgid stuff back and it was not as easy or straightforward as I would have hoped. With 1.3, having a parent table is certainly more self-contained than possible with 2.x and doesn't involve any portability issues, afaikt.
Re: Async write completion for 2.2.x?
On Jul 5, 2007, at 9:20 PM, Paul Querna wrote: Jim Jagielski wrote: Any interest in seeing if the Async write completion code in trunk would be suitable for backporting to 2.2? The biggest problem is that async write completion in trunk got tied up in doing async reads, and some of those patches got reverted because they broke too much stuff Brian Panee was the one spearheading this in trunk, an I don't know where it was really left off. TBH, I'm not sure the async write stuff even works perfectly in trunk, so I am hesisant to backport it to 2.2, when I know the event MPM works pretty well for its limmited use cases there (just KeepAlive). My preference would be to get a workable solution for async reads and write in turnk, and then look at backporting that as a whole. (or releasing 2.4.x with it) Well, that's the big reason I proposed this... I fear that the async stuff will not be worked on/polished unless there is some indication that it'll be in a released version of Apache. My proposing a backport I'm hoping to generate interest in that, so we can either *really* backport it to 2.2 or encourage a pre-amsterdam 2.4 release.
Re: svn commit: r556931 - /httpd/httpd/trunk/modules/proxy/mod_proxy_balancer.c
On Jul 17, 2007, at 10:31 AM, [EMAIL PROTECTED] wrote:
+if (*(workers-s-route) strcmp(workers-s-route,
route) == 0) {
Is that right? I'm guessing the 1st check was to make sure
that workers-s-route wasn't NULL (and therefore the strcmp
didn't dump), but instead you're checking that the 1st
character isn't '\0'... or was that the intent?
Re: svn commit: r556931 - /httpd/httpd/trunk/modules/proxy/mod_proxy_balancer.c
On Jul 17, 2007, at 12:48 PM, jean-frederic clere wrote:
Jim Jagielski wrote:
On Jul 17, 2007, at 10:31 AM, [EMAIL PROTECTED] wrote:
+if (*(workers-s-route) strcmp(workers-s-
route, route) == 0) {
Is that right? I'm guessing the 1st check was to make sure
that workers-s-route wasn't NULL (and therefore the strcmp
didn't dump), but instead you're checking that the 1st
character isn't '\0'... or was that the intent?
It is checking for '\0' see modules/proxy/mod_proxy.h (typedef
struct { ... } proxy_worker_stat.
Ahh... I had forgotten that route in proxy_worker_stat
was an char [] and not a *char
Re: 2.2.5?
On Jul 19, 2007, at 10:49 AM, Sander Temme wrote: On Jul 19, 2007, at 3:22 AM, Ruediger Pluem wrote: Now that the security related patches have been backported to 2.2.x is there anything that prevents us from releasing 2.2.5? Sander Temme volunteered to be the RM back in May. Is this still valid? Absolutely. I was going to propose a Tag Roll later this week, but then all hell broke loose and I found myself on the road. I can do a TR either this Friday (the 20th) or by the end of the weekend if that will get more people testing. I'll probably need one or more folks on irc to walk me through the process as this is my first. Can I have a hum on the timing? I'm +1 on a TR soon; lots of good fixes improvements as well as the security stuff. As noted on this and other lists, I'll be offline (on vacation) the next week w/ very limited cell and Net access (that's the way it is at some remote beaches) otherwise I'd offer to RM if you couldn't. But the earliest I could would be like the 30th...
Re: svn commit: r559837 - /httpd/httpd/trunk/modules/filters/mod_filter.c
On Jul 26, 2007, at 3:46 PM, Ruediger Pluem wrote: This is correct (because provider-match is a union and provider- match.string and provider-match.regex are the same thing), but confusing. I would prefer checking provider-match.regex instead. Seems to me that avoiding unions altogether, if possible, is a Good Thing.
Re: TR Procedure for 1.3?
http://httpd.apache.org/dev/how-to-release.html is
kind of dated and yes is in need of an update, but
I've never bothered doing so (except for the last
1.3 release, I've been RM for 1.3 for the last several
years)
The svn copy location however is
https://svn.apache.org/repos/asf/httpd/httpd/tags/1.3/APACHE_1_3_MM
On Jul 25, 2007, at 10:30 PM, Sander Temme wrote:
Folks,
Before I perform my first Tag Roll, I'm trying to familiarize
myself with the exact process. Maybe I'm not looking in the right
places, or is the documentation actually a bit thin on the nuts and
bolts of the process? Here's what I think should happen, gleaned
from the various pages and scripts:
branches/1.3.x to tag version 1.3.MM:
1) Edit STATUS and src/include/httpd.h:
In STATUS, change version number 1.3.MM line to say Tagged mm/
dd/
In httpd,h, remove -dev from SERVER_BASEREVISION define
Commit as revision rXX
2) svn copy -rXX https://svn.apache.org/repos/asf/httpd/httpd/
branches/1.3.x \
https://svn.apache.org/repos/asf/httpd/httpd/tags/1.3.MM
This will be committed as rYY (likely rXX + 1 but that
is not certain)
3) Edit STATUS and src/include/httpd.h:
In STATUS, add line 1.3.(MM + 1): In development
In httpd.h, change SERVER_BASEREVISION define to say 1.3.(MM +
1)-dev
Also bump the APACHE_RELEASE define
Commit
The previous I've inferred from the svn logs. Now I'm going by
http://httpd.apache.org/dev/how-to-release.html which points to
release.sh, but it looks like that only works for 2.x. What of the
below still applies?
4) Export the source:
umask 022
svn export http://svn.apache.org/repos/asf/httpd/httpd/tags/
1.3.MM \
apache_1.3.MM
5) curl http://httpd.apache.org/docs/1.3/misc/FAQ.html \
htdocs/manual/misc/FAQ.html
6) cp src/Configuration.tmpl src/Configuration
7) Remove various dev-only files:
rm -f STATUS RULES.CVS src/INDENT htdocs/manual/misc./FAQ-*.html
find . -name .cvsignore -exec rm {} \;
find . -type d -name test -exec rm -Rf {} \;
find . -type d -name .svn -exec rm -Rf {} \;
8) Expand the Server-Side-Include directives in the manual:
cd htdocs/manual
./expand.pl
9) Roll the tarballs:
tar -czf apache_1.3.MM.tar.gz apache_1.3.MM
tar -cf apache_1.3.MM.tar apache_1.3.MM
compress apache_1.3.MM.tar
10) Test the tarballs for untarrability
tar -tzf apache_1.3.MM.tar.gz
unzompress apache_1.3.MM.tar.Z | tar -tf -
12) PGP sign the tarballs
gpg -sba apache_1.3.MM.tar.gz
gpg -sba apache_1.3.MM.tar.Z
13) Test the signatures
gpg --verify apache_1.3.MM.tar.gz.asc
gpg --verify apache_1.3.MM.tar.Z.asc
14) Pull out the CHANGES file for upload to the web
cp apache_1.3.MM/src/CHANGES ./CHANGES_1.3
15) Upload the tarballs to the website
scp apache_1.3.MM.tar.gz apache_1.3.MM.tar.Z \
apache_1.3.MM.tar.gz.asc apache_1.3.MM.tar.Z.asc \
CHANGES_1.3 people.apache.org:
16) Move in place:
ssh people.apache.org
umask 002
mv apache_1.3.MM.tar.gz apache_1.3.MM.tar.Z \
apache_1.3.MM.tar.gz.asc apache_1.3.MM.tar.Z.asc \
CHANGES_1.3 /www/httpd.apache.org/dev/dist
They should show up group-writable. If not, you umask was off,
do correct.
If the mv doesn't work, you're not in the right unix group.
Have infra
correct.
17) Wait patiently for rsync, avoid bitching to infra at all costs.
18) Mail [EMAIL PROTECTED], [EMAIL PROTECTED] with a [VOTE] e-mail
copied from an
earlier release effort lovingly archived on mail-archives.a.o/
mod_mbox.
NOTE it'd be highly desirable if all branches' tarballs were
announced for
testing in one such e-mail
19) Sit back and watch the votes roll in
Comments? Critique? I'll be happy to update how-to-release.html
accordingly.
S.
--
[EMAIL PROTECTED] http://www.temme.net/sander/
PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF
1.3 bugs
I went through and cleared out maybe 200 or so bugzilla bugs for 1.3... Will start on the 2.x ones tomorrow and try to clear out most of the crud there... We have some real old ones :)
Re: 1.3 bugs
Nick Kew wrote: On Thu, 2 Aug 2007 16:30:25 -0400 Jim Jagielski [EMAIL PROTECTED] wrote: I went through and cleared out maybe 200 or so bugzilla bugs for 1.3... Will start on the 2.x ones tomorrow and try to clear out most of the crud there... We have some real old ones :) Nice going! As for 2.x bugs, there are quite a few which are going to be harder to deal with. Perhaps we want a new Archived status, for PRs which have merit but which aren't going to get 'fixed'. Particularly those with PatchAvailable. Or maybe FutureFix ? -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ If you can dodge a wrench, you can dodge a ball.
Re: 1.3 bugs
It's easy to be brave when being heartless :) Lots of WONTFIX :) Tony Stevenson wrote: Impressive feat, takes a brave man to take on that many bugs! Jim Jagielski wrote: I went through and cleared out maybe 200 or so bugzilla bugs for 1.3... Will start on the 2.x ones tomorrow and try to clear out most of the crud there... We have some real old ones :) -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ If you can dodge a wrench, you can dodge a ball.
PR 9727
This PR (correctly) states that we aren't tagging double quote as a tspecial/separator. Adding it in causes no regressions, but I'm surprised that we haven't had more issues about its absence, which makes me nervous about changing it now... Comments?
TR schedule
Lets shoot for a TR of 1.3, 2.0 and 2.2 on Aug 10th... That means a possible release on the 13th. That way, admins aren't compelled to upgrade before/during the weekend (other- wise, TR on the 8th and release on the 10th would make sense) I volunteer to RM 1.3 and even 2.2. afaict, 1.3 and 2.0 are ready to go; we just have some outstanding potentials still in line for 2.2
