Re: [for beginners] How to have a patch validated?
Bruno; You did everything right. I have committed this to trunk in r1648201 and proposed for 2.4 backport in STATUS. Thanks for the patch. -- Daniel Ruggeri On 12/28/2014 6:28 AM, Bruno Raoult wrote: > Hi, > > I am really sorry for this stupid question. > > I did send a bug report, with a patch, bug number is 57227 > (https://issues.apache.org/bugzilla/show_bug.cgi?id=57227). > > However, I believe this bug could stay forever if the patch (or > better) is not applied. > Did I do something wrong when I did submit the bug and patch together? > > Thanks, > > Bruno. > > > -- > 2 + 2 = 5, for very large values of 2.
Re: [APACHECON] Proposed httpd (and related) track
Hi, Rich; I dig it. I'm all for presenting and helping to make ApacheCon great but I won't be able to make it on day 1 since I'll probably be somewhere in the air over the Gulf of Mexico mid-day. If you are struggling for content, I can be convinced to take on the topic that I brought up earlier on the list (http://marc.info/?l=apache-httpd-dev&m=141753066016122&w=2) or to jump in to help Bill with what he is working on (FWIW, I can't see http://events.linuxfoundation.org/cfp/proposals/4346, so I don't know what Bill has up his sleeve other than what was brought up in the context of this discussion). Also, don't hesitate to reach out if I can help out with any of the regular or "extracurricular" activities during/after/around the conference. -- Daniel Ruggeri On 2/10/2015 1:36 PM, Rich Bowen wrote: > Here's my proposed httpd (and related) track. If anyone has any > objections, changes, suggestions, whatever, please speak up. Thanks. > > > Day 1: > > * Panel: The Apache Group greybeards: If I'd only known then (Don’t > yet have confirmation of who’s actually going to be there. Brian has > declined to speak on such a panel.) Note if that these folks don't > show up, we'll need to find something to fill this slot with. Brian > has confirmed that he'll attend, but so far I don't have an absolute > confirmation from anybody else from that era. > > * What's New In Apache HTTPD 2.4 - jimjag - > http://events.linuxfoundation.org/cfp/proposals/4014 > > * mod_rewrite and friends - rbowen - > http://events.linuxfoundation.org/cfp/proposals/1528/4013 > > * The State of TLS on Apache HTTP Server - wrowe - > http://events.linuxfoundation.org/cfp/proposals/4346 > > * Reverse Proxy with Apache HTTPD 2.4: The hidden gem - jimjag - > http://events.linuxfoundation.org/cfp/proposals/4015 > > * The mod_proxy cookbook - Daniel Ruggeri - > http://events.linuxfoundation.org/cfp/proposals/3816 > > > Day 2: > > > * Apache HTTP Configuration API for Developers - wrowe - > http://events.linuxfoundation.org/cfp/proposals/4347 > > * Begone mod_php! - Jim Riggs - > http://events.linuxfoundation.org/cfp/proposals/4208 > > * A Peek at PHP 7 - John Coggeshall - > http://events.linuxfoundation.org/cfp/proposals/4226 > > * Using Apache Traffic Server to cache "Live TV" - Mark Torluemke - > http://events.linuxfoundation.org/cfp/proposals/4180 > > * Traffic Server on the Edge - Alan Carroll - > http://events.linuxfoundation.org/cfp/proposals/4133 > > * Replacing Squid with Apache Traffic Server for Yahoo - Shu Kit Chan > - http://events.linuxfoundation.org/cfp/proposals/4118 > > Day 3 - TOMCAT > > > * Intro to Load-Balancing Tomcat with httpd and mod_jk - Christopher > Schultz - http://events.linuxfoundation.org/cfp/proposals/4206 > > * Tomcat clustering: Part 1 - Reverse Proxies - Mark Thomas - > http://events.linuxfoundation.org/cfp/proposals/4053 > > * Tomcat clustering: Part 2 - Load-balancing - Mark Thomas - > http://events.linuxfoundation.org/cfp/proposals/4054 > > * Tomcat clustering: Part 3 - Session Replication - Mark Thomas - > http://events.linuxfoundation.org/cfp/proposals/4055 > > * Monitoring Apache Tomcat - Christopher Schultz - > http://events.linuxfoundation.org/cfp/proposals/3847 > > * Choosing tomcat connectors: internals and performances - > Jean-Frederic Clere - > http://events.linuxfoundation.org/cfp/proposals/3839 > > >
Re: Balancer manager
+1 There are also some neat-o features I added in my notes during ACNA to stick into the balancer manager, too... I plan to get around to them in days. -- Daniel Ruggeri On 4/24/2015 7:52 AM, Jim Jagielski wrote: > Right now, the balancer manager allows for a member to be > disabled/stopped, but it cannot *remove* that member... > Seems to me that that would be good, especially since > we could always re-use that slot. > > Comments?
Re: *Match, RewriteRule POLA violation?
+1 By unbreaking configurations we are indeed changing behavior. This could be an unexpected change for an admin during a minor upgrade but I weigh that against the fact that directives enclosed by these matches may be intended to add security/authorization/authentication which a badly written link could circumvent if an admin isn't using the appropriate regex. -- Daniel Ruggeri On 4/30/2015 8:16 AM, Yann Ylavic wrote: > On Thu, Apr 30, 2015 at 2:57 PM, Jim Riggs wrote: >> Thanks, Yann. I remember looking at this code before. The question remains, >> though: Is it currently "wrong"? >> Does it need to be "fixed", or was this distinction made intentionally? >> Is there a specific use case that requires the regex-matching directives to >> not get slash-normalized URIs? > I would like it to be fixed, non leading "/+" is equivalent to "/", > this would break very few (if any) cases IMHO, and may even unbreak > more ones .
Re: Proposal/RFC: "informed" load balancing
; response data. For this, I have created a module tentatively named
> mod_proxy_backend_info that does nothing except insert an output filter to
> populate the response header with version, provider, workers-*, request,
> uptime, and load-* values when the request header is present. Here is an
> example request-response:
>
> % curl -v -H 'X-Backend-Info: version=1.0' http://localhost/
> * Trying 127.0.0.1...
> * Connected to localhost (127.0.0.1) port 80 (#0)
>> GET / HTTP/1.1
>> User-Agent: curl/7.41.0
>> Host: localhost
>> Accept: */*
>> X-Backend-Info: version=1.0
>>
> < HTTP/1.1 200 OK
> < Date: Thu, 30 Apr 2015 04:32:08 GMT
> < Server: Apache/2.4.9 (Unix) PHP/5.5.14
> < Last-Modified: Wed, 15 Apr 2015 14:04:54 GMT
> < ETag: "2d-513c3d4d78d80"
> < Accept-Ranges: bytes
> < Content-Length: 45
> < X-Backend-Info: version=1.0, provider="mod_proxy_backend_info [Apache/2.4.9
> (Unix) PHP/5.5.14]", workers-max=256, workers-busy=1, workers-ready=4,
> workers-free=255, uptime=1448, requests=3, load-current=1.737305,
> load-5=1.733887, load-15=1.668457
> < Connection: X-Backend-Info
> < Content-Type: text/html
> <
> It works!
>
>
> The second piece is when httpd is used as the load balancer. For this, I have
> created a module tentatively named mod_lbmethod_bybackendinfo that will:
>
> 1. Periodically (based on elapsed time, number of requests, or both since
> last update) insert the X-Backend-Info request header into a proxied request.
>
> 2. Parse and remove the X-Backend-Info response header.
>
> 3. Calculate the member's "informed" load factor based on a formula specified
> by the user/admin in the configuration. I hope to just use the existing
> lbfactor field to store this calculated value. Then we can use existing logic
> to balance based on lbset and lbfactor for subsequent requests.
>
> 4. Store the current time and request count in the member's data structure so
> the lbmethod knows when it needs to be updated again.
>
>
> What I need from all of you:
>
> - Input/commentary on the proposed idea, approach, and implementation.
> Renaming things, additional fields that might be useful, etc., are all up for
> discussion.
>
> - Help with handling the configuration formula mentioned in #3 above. Can we
> just add some math operators to the expression parser to handle this? What
> all operations/functions might we need (+-*/? max? min? ternary if-then-else?
> ...)? A simple-ish example (something like this maybe?):
>
>
> BalancerMember ...
> ...
> ProxySet \
> lbmethod=bybackendinfo \
> backendupdateseconds=30 \
> backendupdaterequests=100 \
> backendformula="%{BACKEND:uptime} -lt 120 ? 1 : %{BACKEND:workers-free} /
> %{BACKEND:workers-max} * 100"
>
Since the spec above states that one or more X-Backend-Info headers is
acceptable, there should be a way for the admin to declare upon which
header the calculation should be performed. A provider flag or option
(FIRST|LAST|"provider-name") ought to do the trick.
Side note: Is there a generic place for maths based on strings in httpd
at all? Creating our own mini-language makes me nervous since we have to
consider that backends can and will do stupid things that could result
in httpd attempting an operation that could cause a crash if we aren't
very careful about validating inputs and making sure we aren't doing
silly things like div/0
>
> - [Near-long-term] Help adding X-Backend-Info backend support and
> documentation to various projects. Tomcat, php-fpm, others(?) should be
> fairly easy to implement and submit patches. This work does us no good if
> none of our backends support it.
I will be happy to provide a ServletFilter that can be used in a J2EE
app or a Tomcat Valve (good for the many Tomcat variations and JBoss)
once we settle on details.
>
> - [Long-term] Help adding X-Backend-Info frontend support and documentation
> to various projects to help this become an "accepted ad-hoc standard"...or
> something like that. Nginx, haproxy, and many others would be targets.
>
>
> Warn out from writing all of this and hopeful that someone other than me
> actually cares, I wish you all well today/tonight!
>
> - Jim
>
All in all, man, this is solid. I like what you've done here.
--
Daniel Ruggeri
Re: Balancer manager
(oops - saw this sitting int he outbox for the past week - sorry for slow reply) These were the notes I took. I was going to start biting them off after I wrapped up splitting/editing the recordings from the ACNA talks: *Ensuring all stats showed up on the page (I don't recall if any stuck out that were missing) *Add ability to reset the stats captured *Set or adjust min/max for the connection pooling *Send what httpd thinks the worker status is (useful for backends that would like to know about drain, etc) to the backend in a header -- Daniel Ruggeri On 4/27/2015 9:43 AM, Jim Jagielski wrote: > Can you list 'em here? I'd like to help w/ adding them :) > >> On Apr 25, 2015, at 12:28 PM, Daniel Ruggeri wrote: >> >> +1 >> >> There are also some neat-o features I added in my notes during ACNA to >> stick into the balancer manager, too... I plan to get around to them in >> days. >> >> -- >> Daniel Ruggeri >> >> On 4/24/2015 7:52 AM, Jim Jagielski wrote: >>> Right now, the balancer manager allows for a member to be >>> disabled/stopped, but it cannot *remove* that member... >>> Seems to me that that would be good, especially since >>> we could always re-use that slot. >>> >>> Comments?
Re: cPanel & Apache 2.4
Nice! -- Daniel Ruggeri Original Message From: Jacob Perkins Sent: May 15, 2015 10:18:08 AM CDT To: dev@httpd.apache.org Subject: cPanel & Apache 2.4 Good afternoon, As some of you may be aware, cPanel is a leader in the hosting industry as we provide software that allows end users to easily run a server and associated LAMP stack. Yesterday, May 14th, we changed the default installation target to Apache 2.4 over Apache 2.2. Currently, around 80% of our user base uses Apache 2.2. I’m hoping to have those customers using Apache 2.4 in the next year. I know that you guys like to keep track of 2.2 vs 2.4 usage, so hopefully we’ll start noticing the 2.4 usage increase gradually over the next year or so. Thanks for your time! — Jacob Perkins Product Owner cPanel Inc. jacob.perk...@cpanel.net <mailto:jacob.perk...@cpanel.net> Office: 713-529-0800 x 4046 Cell: 713-560-8655
Style checker?
All; I still develop in what a lot of folks would consider a fairly "primitive" environment (vi) that doesn't do anything for style checking things like line width/spacing before and after control statements/indentation/variable declaration/etc. I know of the indent tool available on most unix-like systems, but was wondering if you folks use any other tools to help along that path? -- Daniel Ruggeri
Re: silly ab patch for SNI and OCSP stapling
+1, but I would also propose a command line flag to override the SNI host name
supplied in case one is testing directly by IP address.
--
Daniel Ruggeri
Original Message
From: Jeff Trawick
Sent: May 12, 2015 2:31:37 PM CDT
To: Apache HTTP Server Development List
Subject: silly ab patch for SNI and OCSP stapling
... where "OCSP stapling" means "get the server to do the related work
but don't care what you get back".
Perhaps this doesn't save any time for anybody that would want to test
such a thing, but who knows?
Index: support/ab.c
===
--- support/ab.c(revision 1679028)
+++ support/ab.c(working copy)
@@ -1287,6 +1287,8 @@
bio = BIO_new_socket(fd, BIO_NOCLOSE);
SSL_set_bio(c->ssl, bio, bio);
SSL_set_connect_state(c->ssl);
+SSL_set_tlsext_host_name(c->ssl, hostname);
+SSL_set_tlsext_status_type(c->ssl, TLSEXT_STATUSTYPE_ocsp);
if (verbosity >= 4) {
BIO_set_callback(bio, ssl_print_cb);
BIO_set_callback_arg(bio, (void *)bio_err);
The lack of SNI is a pretty big hole now; it probably doesn't need much
extra in the way of #if/if to do the right thing.
Re: silly ab patch for SNI and OCSP stapling
Yep, my mistake. I thought there was a command line switch to change the host header. You're correct - it wouldn't make much sense to override one and not the other. -- Daniel Ruggeri On 5/16/2015 11:25 AM, Jeff Trawick wrote: > in that case shouldn't you also be overriding Host:, so the SNI host > name can use the same override? I think this may lead the user into a > more helpful scenario, if indeed they don't already know when to > override Host:, and I don't know how useful it is to have different > values for Host: and SNI.
Re: 2.2 and 2.4 and 2.6/3.0
On 5/28/2015 2:54 PM, Jim Riggs wrote: > Having to expend effort (e.g. re-design/update config and deployment) to switch/update/upgrade to a new paradigm does not, IMO, mean that it's not a solution for everyone. Anyone can take the time to implement and automate the switch. Once that effort has been spent, you should have nearly the same (maybe better) setup, with hopefully better security and resource utilization in many cases. All of those php_admin_* directives end up in a php-fpm conf file that can easily be automatically updated and deployed. Thinking about this more, what are the things preventing people from an _easy_ upgrade path configuration-wise? A lot of this conversation surrounded users and the impact of an upgrade to them. The interface for the users' to the server is the configuration file. Maybe if we can tackle that we can greatly reduce a barrier to upgrade (or maybe I'm too optimistic)? For the majority of my configs, it was the changes to the authorization directives - it takes brainpower to figure out what AdminXYZ was trying to do years ago and reflect that with the new directives. However, this is deterministic... a perl script could do this work for me if I'd just write it. At $dayjob, this is the stuff I focus on. Tweaks to an existing script I put together years ago to upgrade from 2.0 to 2.2 (or as Rich eloquently put, "poop out" new configs) only required an hour's worth of work or so to support upgrades from 2.2 to 2.4 minus the aforementioned authz directives. -- Daniel Ruggeri
Re: 2.2 and 2.4 and 2.6/3.0
On 5/30/2015 1:47 PM, Daniel Ruggeri wrote:> Thinking about this more, what are the things preventing people from an > _easy_ upgrade path configuration-wise? A lot of this conversation > surrounded users and the impact of an upgrade to them. The interface for > the users' to the server is the configuration file. Maybe if we can > tackle that we can greatly reduce a barrier to upgrade (or maybe I'm too > optimistic)? > > For the majority of my configs, it was the changes to the authorization > directives - it takes brainpower to figure out what AdminXYZ was trying > to do years ago and reflect that with the new directives. However, this > is deterministic... a perl script could do this work for me if I'd just > write it. > > At $dayjob, this is the stuff I focus on. Tweaks to an existing script I > put together years ago to upgrade from 2.0 to 2.2 (or as Rich eloquently > put, "poop out" new configs) only required an hour's worth of work or so > to support upgrades from 2.2 to 2.4 minus the aforementioned authz > directives. > > -- Daniel Ruggeri aand on the original topic at hand, I'll share that none of us are really *forced* to focus on any particular branch or any particular area of the code base. As an army of volunteers, we scratch our own itch. I, too, have noticed that 2.2 hasn't had a whole lot going on lately. I don't know if that means we ought to get ready to drop the axe on it, but I also don't know if that means we shouldn't be thinking about it, either. I think Jim's email served it's purpose of at least vocalizing a thought that's probably in the back of all of our minds: 2.2 isn't getting a ton of love (or at least as much love as 2.4) lately. What that means to each of us is clearly different... for me, I noticed that the effort to backport some of the work I've done in 2.4 doesn't pay off so I don't do it. I haven't put much more thought into 2.2 than that. For others it might be a push to backport more stuff. At the same time, Bill points out a reality we're faced with. As the most widely deployed HTTP server, we have some sort of responsibility (whether real or imagined) to the users not to "cut them off" if we can avoid it. The point about distros taking their sweet time to update are well taken - it's one of the reasons I always build from latest source. Maybe there's a reason other than inertia for the slow adoption. Their own lack of volunteer cycles? Configuration differences? Performance differences? What we have works, so why change it? Do we know those reasons enough to try to keep ahead of them? P.S. I'm not a Member or PMC... do I have access to the report that spurred the conversation? P.P.S. Wow... I don't read my email for a few days... -- Daniel Ruggeri
Re: PMC Reporting [Was: Re: 2.2 and 2.4 and 2.6/3.0]
On 5/30/2015 9:03 PM, William A Rowe Jr wrote: > So I'll let Eric share what he submitted for May on our behalf, but here > is the submitted/accepted/recorded report of Feb '15 - it's awfully high > level, so I'm not sure that updating dev@ regularly with the contents > offers a whole lot of benefit. Thoughts? Yeah - The information seems great to share with the community behind httpd, IMO, so I think it would make sense to have on the dev@ list... and it's not like this is a particularly low volume mailing list that such emails would be considered inbox pollution. I guess it's just as easy to pull up the minutes each month by hand, too. -- Daniel Ruggeri
Re: [VOTE] Release Apache httpd 2.4.13 as GA
On 6/4/2015 8:43 PM, Noel Butler wrote: > I disagree with this removal, it will cause more problems then benefits > IMHO, make it a recommendation to use SSLCertificateFile, but not an > essential requirement, perhaps log it, and log it only once, not per > every domain, not throw them to std out. +1 FWIW, I think Kaspar had a driving technical reason for its deprecation, but I can't seem to find the original email talking about it. -- Daniel Ruggeri
Re: Additional LB providers
Additional providers is cool... but what do you mean by "fold in?" Add them as additional modules? (Sorry for top-post... mobile email client) -- Daniel Ruggeri Original Message From: Jim Jagielski Sent: June 18, 2015 11:52:12 AM CDT To: httpd Subject: Additional LB providers I'm playing around w/ a newish LB provider that balances based on latency. I'd like to fold it in after I clean it up a bit but it seems to me that we could also fold in the round-robin provider in example (after a suitable cleanup) as well as add in a simply by_random as well. Comments?
Re: Proxy balancer providers and aging
On 6/24/2015 3:12 PM, Jim Jagielski wrote: > All LBmethods have an "age" function which is designed to appropriately > "age" the data so that things even out after awhile. Of course, right > now, none actually *uses* that. > I had never realized this function exists... what triggers it? ... or are you saying that today nothing triggers it because of the next sentence? > But I think the reason is because there is no real good way, > currently, in mod_proxy to do that... > > Well, in the LBmethod I was hacking away on, it really DOES need > its age function, and currently it's doing so via mod_watchdog. > But the more I think of it, ideally, mod_proxy *itself* should > create the watchdog thread and just handle the age itself, rather > than having a LBmethod provider having to be responsible for > creating that (it also kind of destroys the abstraction that pure > providers should have, imo)... > Yup - if there is some sort of contract between mod_proxy such that each balancer ought to be able to "age" its data, mod_proxy should provide the facilities to call that age function on some interval. However, looking at the age function signature in the existing balancers, I'm seeing a conspicuous lack of a time delta since the last call to age(). It seems to me that if mod_proxy is responsible for the 'tick' it should also be responsible for providing a time delta so the lbmethod can age the data proportionally to the time since last tick. (or, maybe I'm missing it altogether and the module should be managing this itself which kinda puts us back to the beginning of which module should be responsible for this stuff) > Anyway... anyone opposed to me adding this to mod_proxy in > trunk with hope towards it backporting to 2.4? It does mean > that mod_proxy will now have a dependency on mod_watchdog. I'm conflicted. This is useful functionality but would break existing configurations using 2.4 in an unexpected way. I remember dealing with lots of questions about "why is this slotmem thing needed all of a sudden" when moving from 2.2 to 2.4. I think that if I were to be forced to work through the cognitive dissonance I'd be +1 for adding this to trunk but -1 for 2.4 unless we can find a way to avoid the dependency unless the lbmethod really needs it (I don't see how, but please do enlighten me if this is possible). -- Daniel Ruggeri
Re: svn commit: r1696960 - in /httpd/httpd/trunk: CHANGES docs/log-message-tags/next-number modules/proxy/mod_proxy_balancer.c
On 8/25/2015 10:11 AM, Jim Jagielski wrote: > Again, if the slotmem exists and is persisted, the assumption > is that THAT is what the admin wants, and when Apache restarts, > THAT is the running config they desire. If there are changes > to the reverse proxy setup, the assumption must be we are > starting from scratch; There are, iirc, a number of places where > these kinds of checks are done. Trying to somehow "merge" a just-changed > file config and a running config (based on an older file config > with dynamic changes) is nasty and tough to do correctly. This is a common problem with serialization. The persisted config is serialized data of what is in memory. It can only represent the "version" of the conf file that created it. If you change the conf, deserialization should fail because there could be other material changes that might get in the way. I'm +1 on the idea that using bgrowth space to add stuff via conf+graceful restart should be avoided. -- Daniel Ruggeri
Re: Work in progress: mod_proxy Health Check module
++1 on this vein of thought. Implementing the health checking similar to the modularized lb methods makes the most sense to me. I can already think of several methods (file, conf, external script, lua script) of obtaining data on how to probe and react. Just some thoughts worth considering... since you asked :-) *The module should not probe if a probe is already running against that backend (hang or takes too long to respond) *The probes to different backends should be done in parallel rather than serially to avoid pileups due to a slow responder *How will the existing balancer re-enable logic interfere with ERR backends? *Maybe another flag that avoids retry if the monitor marked it down requiring the monitor to bring it back up before any traffic is allowed to go to it? I've followed along with what you described by using watchdog to trigger the probing, but I'm curious where your thoughts are for submitting a request to the backend. Are you planning to simulate one from your module by directly connecting to the backend or is the plan to run the request through the existing proxy infrastructure (the former allows lots of control while the latter permits other directives to come into play that might be handy like disabling based on status code). I haven't seen the code, but your previous email said you were thinking of the former case. P.S. Thanks for taking this on. It's been on my own todo list for a long time. -- Daniel Ruggeri On 1/8/2016 1:09 PM, Jim Jagielski wrote: > Thx! Any other comments/suggestions? > >> On Jan 8, 2016, at 10:53 AM, Eric Covener wrote: >> >> On Fri, Jan 8, 2016 at 10:18 AM, Jim Jagielski wrote: >>> I am thinking about the actual health-check impl a bit more, >>> hence the lack of commits. As noted, initially I was thinking >>> about optional functions, defined in mod_proxy_http, mod_proxy_ajp, >>> etc. Then I started thinking that maybe doing it via providers >>> might be better, allowing for more "custom" health checks via >>> the end user. Then I started thinking about doing it rough-and- >>> ready and just adding 'em in mod_proxy_hcheck... >>> >>> Before I spend too much time on this, any prefs one way or >>> another regarding this? >> I haven't looked at the architecture so far -- but I think basics in >> mod_proxy_hcheck >> and at least one "standalone" provider (as an example) would be a good >> middle ground. >> >> Maybe the standalone one would try to deal with the body whereas the >> built-in ones >> would be TCP up, SSL up, status-code, etc.
Re: [Patch] mod_tcp / mod_proxy_tcp / mod_ssl_tcp
+1 Really nice work -- Daniel Ruggeri On 3/13/2016 10:45 AM, Jim Jagielski wrote: > I've given it a quick look-thru and I. Am. Impressed. > > This is more Super Cool Mojo!
Re: Status for 2.4.20
On 3/23/2016 7:27 AM, Jim Jagielski wrote: > Release Often is hardly a Bad Thing, at least IMHO. When the > time is right for a release, then we release. It seemed a > good time, again IMHO. Kinda late to the party, but shouldn't what's committed to a stable branch _always_ be ready to release? Just my .02 on the side conversation. As a sysadmin, I have no strong preference for release often versus release seldom... just so long as what is released is stable and won't break my stuff. At $dayjob where I have to answer to regulatory and audit authorities, there have been plenty of releases of various software platforms that I have chosen not to pursue because they don't address anything I need to worry about. No harm done. So, my stance as a sysadmin (and probably a fairly common stance, I'd guess) can be summarized as: If there's a bug fix for a problem I'm seeing, I'd really like the release sooner rather than later. On the other hand, if there's a security fix, I need it released immediately. On yet another hand (because it seems I have three), if the release has nothing I care about... then I don't care. But whatever happens, don't break my configs :-) As an even further side note... after following this dev list (community) as long as I have, I'd happily put an httpd 2.4.nightly against just about any other released software. I can't think of a single case where something got *committed* to 2.4 that would break my configs let alone something that got formally released. Indeed, the few thousand httpd servers I run haven't been harmed by a release in all of my time as an admin (1.3, 2.0 and 2.4 alike). Our release process is, indeed, pretty damn good. So whether we release often or release seldom, we can at least hold our head high while we do it. -- Daniel Ruggeri
Re: [users@httpd] Strange with AllowOverrideList Directive
I'm assuming that compiler optimizations would make both patches "six to
one, half dozen to the other" as far as code path followed during the
request cycle... but I agree.
Fixed in trunk in r1737114 and proposed for backport in 2.4 in STATUS.
--
Daniel Ruggeri
On 3/30/2016 8:02 AM, Plüm, Rüdiger, Vodafone Group wrote:
>
> But the attached patch is not correct. IMHO it needs to be
>
>
>
> Index: request.c
>
> ===
>
> --- request.c (revision 1735931)
>
> +++ request.c (working copy)
>
> @@ -1009,7 +1009,9 @@
>
> /* No htaccess in an incomplete root path,
>
> * nor if it's disabled
>
> */
>
> -if (seg < startseg || (!opts.override &&
> opts.override_list == NULL)) {
>
> +if (seg < startseg || (!opts.override
>
> + &&
> apr_is_empty_table(opts.override_list)
>
> + )) {
>
> break;
>
> }
>
>
>
>
>
> Regards
>
>
>
> Rüdiger
>
Re: Allow SSLProxy* config in context?
On 4/13/2016 2:22 PM, Rainer Jung wrote:
>
> We could pass the worker name from mod_proxy to mod_ssl via a
> connection note, similar to currently already passing the SNI name via
> the connection note proxy-request-hostname.
+1 on the connection note idea, but see below about having to inform the
callback function about too much information
On 4/13/2016 10:04 AM, Graham Leggett wrote:
> What I had in mind was a syntax where the certs were named, for example:
>
> SSLProxyCertificate foo /path/to/foo.cert
>
> Followed somewhere else by:
>
>
> SSLProxyEnable foo
>
On one hand, this is an attractive idea because we have already
conditioned users/administrators to think about naming things via config
as you can do with authn providers and aliases.
On another hand, there are two gotchas I see. Giving something a name in
the configuration does not necessarily mean anything to the file or
certificate store because they aren't married in the same way that authn
providers and their aliases are. That is, if a different team/individual
manages the certificate store than the server configuration, the two
would have to keep in sync with additions or removals. This can be a
PITA if you have a short certificate lifecycle and have to renew often.
PLUS, there are cases where there may be many certificates in the file.
Combine that with the previous point and you could have a nightmare.
Perhaps in addition to or instead of aliases via config, something more
"dynamic" could be used where a DN, CN or some other attribute about the
cert that was parsed during startup becomes the representation?
...
#As today
SSLProxyMachineCertificateFile /path/to/file
#Plus new stuff
SSLProxyMachineCertificateCN myIdentity
#or
SSLProxyMachineCertificateDN /C=US/ST=Missouri/L=St.
Louis/O=BitNebula/CN=DanielRuggeri.bitnebula.com
This could be a really clean solution because the proxy can just pass
this name or DN to the ssl module in a note and the ssl module could
make the selection of certs from that data.
Stashing this (completely untested and thrown together) code around line
1778 in modules/ssl/ssl_engine_kernel.c could be all that's needed in
mod_ssl
char *preferred_name;
..
if (preferred_name) {
for (i = 0; i < sk_X509_INFO_num(certs); i++) {
X509_NAME *name;
info = sk_X509_INFO_value(certs, i);
name = X509_get_subject_name(inf->x509);
for(j = 0; j < X509_NAME_entry_count(name); j++) {
const char *this_name = //something
if (strcmp(preferred_name, this_name) == 0) {
modssl_proxy_info_log(c, info, APLOGNO(02279)
"found acceptable cert");
modssl_set_cert_info(info, x509, pkey);
return TRUE;
}
}
}
ap_log_error(APLOG_MARK, APLOG_WARNING, 0, s, APLOGNO(00)
SSLPROXY_CERT_CB_LOG_FMT
"server configuration requested certificate name %s"
"but it was not found in the certificate store",
preferred_name);
}
I haven't cracked open the proxy code to see what it would take to
collapse this down to a per proxy/worker/etc, but it doesn't seem like
terrible endeavor.
--
Daniel Ruggeri
Re: Allow SSLProxy* config in context?
On 4/14/2016 3:08 AM, Rainer Jung wrote: > Your idea to allow selecting a client cert based on CN or DN sounds > attractive to me as well. But since it wouldn't help with other per > backend settings (like different Verify settings) we might even think > about combining both. As long as your only problem would be client > certs, you can select via CN or DN and keep the config vhost global, > but once you need more adjustments per backend, you would start to > configure SSLProxy* inside . > > WDYT? Yep! Agreed - I was focusing on the use case presented, but didn't think about other cases such as different SSLProxyVerify settings and friends such as this. I'd cast a vote for the more comprehensive solution you and Graham shared that makes each option available in a directory context. Adding yet another directive would satisfy this one use case, but the bigger picture would serve users better and be less confusing. Food for thought - I haven't dug into these... * Not sure if it would be a handy target of opportunity or PITA... Since ProxyPass is currently server/vhost/dir configurable, should we think to include this capability as arguments to ProxyPass/BalancerMember, too? * Maybe I missed it, but I don't think the proposed ideas truly provide per-backend configuration. Rather, they provide configuration for all backends in a dir context if you land on a balancer containing many targets. Is that OK, or should we be thinking about ways to make it even more granular? (the use cases that come to mind are rather contrived, so I may be overthinking it) * Would each of these per-dir configs post-merge result in a distinct SSL_CTX that needs to be created in mod_ssl? If so, does that require significant refactoring in mod_ssl or does the proxy module manage these? -- Daniel Ruggeri
Re: End of the road of 2.2.x maintenance?
On 2016-05-11 14:13, Ruediger Pluem wrote: On 05/10/2016 10:38 PM, William A Rowe Jr wrote: . . . Are we ready to start the 12 month countdown as of the next/final bug fix release of 2.2, and highlight this in both the 2.2 and 2.4 announce broadcasts? I'm hoping we conclude some fixes of 2.4 scoreboard regressions and get to the point of releasing 2.4 with mod_proxy_http2 sometime within the next month or so, and that we can reach a consensus about how we will proceed on the 2.2 branch, before we get to that release. Feedback desired, Sounds like a sensible plan. Regards Rüdiger +1 -- Daniel Ruggeri
Re: mod_proxy_hcheck backport
On 5/16/2016 8:19 AM, Jim Jagielski wrote: > THANKS! This feature seemed to cause a lot of buzz @ ApacheCon so > would be I believe I heard and/or used the term "sexy" at least once to describe it ;-) On 5/16/2016 8:19 AM, Jim Jagielski wrote: > Hmmm... The balancer-manager page does display the configured > values for 'hcpasses' and 'hcfails' as well as the current count > of passes/fails. Is that sufficient? Yeah - didn't think about that. It'd be fine for my purposes. It could be a PITA if someone is monitoring for a specific string like "transitive" or "fail", but it's probably not worth monkeying with. -- Daniel Ruggeri
Re: [POLL] Commitment to 2.2.x lifecycle? (Was: End of the road of 2.2.x maintenance?)
*) I intend to help maintain/test 2.2.x releases over the next [_12___] mos *) I intend to backport/review 2.2.x security patches over the next [_18___] mos -- Daniel Ruggeri
Re: 2.6/3.0
On 12/9/2016 8:32 AM, Jim Jagielski wrote: > Instead, maybe we could backport all that stuff to 2.4, in a backwards > compatible fashion. That is, basically backport trunk to 2.4. This > would give us more runway to work on httpd-nextgen. > > Thoughts? Considering a lot of the changes in trunk, I'm not entirely sure it can be backported without creating some code maintenance challenges... I agree about the concern w/ branching so I guess it's really a matter of weighing the benefits - are there *enough* features to be worthy of a release, have we reached API stability to make it worth it and do we lose anything in our "sandbox" by doing this? Pressed for a response, I'd lean towards probably not but it's not a terribly strong opinion. It's also not unprecedented to have three major versions in play, too. It's been announced that 2.2 is on the "EOL" track with most folks interested in maintaining that branch in the 12 to 18 month range (June - Dec 2017ish). So, that said, it wouldn't be a super long time we maintain three branches... a year-ish from now. -- Daniel Ruggeri
Re: T&R of 2.4.24
On 12/12/2016 12:26 AM, William A Rowe Jr wrote: > In spite of 34 registered project committee members, until other > contributors come forward to participate in the security patch review > process, we may simply have to declare all further efforts are currently > on pause. Does one have to be on PMC to review security patches? If not, can you give me a general idea on volume? This would be something I think $dayjob would be OK with me doing as part of keeping a shirt on my back and roof over the childrens' heads ;-) -- Daniel Ruggeri
Confusing behavior w/ buckets in connection filter
Hi, all; I'm hoping just for a quick sanity check here... when consuming buckets from the brigade inside a connection filter, I've seen that the bucket length doesn't *appear* to accurately represent what data has been made available when SSL is used. const char *tmp_buf; apr_size_t nbytes; apr_bucket_read(b, &tmp_buf, &nbytes, APR_BLOCK_READ); The particular scenario is that apr_bucket_read tells me that 11 bytes were read according to nbytes... but a print of the string stashed in the buffer shows many more. In fact, all data that should be available at this time appears to be there (about as I would expect). This seems to only happen when using SSL, FWIW. I consistently see 11 bytes as being read by apr_bucket_read... but all 48 expected bytes are there (strlen as well as any other method of examining the input concurs). I'm sure I've made a goofy assumption or am thinking about this wrong somewhere... but the mistake in my understanding escapes me. Any thoughts? P.S. For context: I'm working on teaching mod_remoteip how to consume PROXY protocol headers. This is a somewhat odd edge case where a connection filter must be inserted before SSL... but that doesn't appear to be the problem since I'm seeing the SSL handshake init (\x16\x03) right after the raw bytes I'm wanting. P.P.S. All the best to everyone during this holiday season! -- Daniel Ruggeri
Re: Confusing behavior w/ buckets in connection filter
On 12/26/2016 11:23 AM, Eric Covener wrote: > No good hint here, but I have found the gdb macros in .gdbinit to be > really helpful to show these details. > > (dump_brigade, dump_bucket IIRC) Hi, Eric; Right - that more or less confirms the confusion. There's only one bucket in the brigade (which seems odd) and it believes it's 11 bytes in length. (gdb) dump_brigade bb dump of brigade 0x7fffd8007630 | type (address)| length | data addr | contents | rc 0 | HEAP (0x7fffd8000928) | 11 | 0x7fffd80009c8 | [PROXY TCP4 ] | 2 end of brigade (gdb) dump_bucket b bucket=HEAP (0x7fffd8000928) length=11 data=0x7fffd80009c8 contents=[PROXY TCP4 ] rc=2 (gdb) p nbytes $13 = 11 (gdb) p tmp_buf $14 = 0x7fffd801a0d8 "PROXY TCP4 192.168.0.103 172.17.0.2 44585 82\r\n\026\003" (gdb) call strlen(tmp_buf) $15 = 48 This output is immediately after apr_bucket_read(b, &tmp_buf, &nbytes, APR_BLOCK_READ);... indeed, nbytes and the length of the bucket agree but there are 48 characters available (in this example) in the buffer. -- Daniel Ruggeri
Re: Confusing behavior w/ buckets in connection filter
On 12/26/2016 1:59 PM, Eric Covener wrote: > or changing the # of bytes when you see yourself get called by > mod_ssl? Not sure. Actually, this is the most sane idea by far. The PROXY protocol spec even helpfully gives you an idea on how many bytes may be sent in such a header and also specifies that they should be sent all at once. No reason to try to deal with multiple invocations of the filtering function, etc. Thanks for the sanity check and pointer! I'll follow up with a patch for review shortly. -- Daniel Ruggeri
Re: Confusing behavior w/ buckets in connection filter
On 12/26/2016 4:40 PM, Eric Covener wrote: > Did you see Jim just imported a third-party module for this? Argh! I didn't until you mentioned this, hahaha! Did I miss a note about it on list? Oh well... learning something is never a loss. Looks to be a more complete patch since it handles the v2 protocol versus what I was writing in mod_remoteip. Just glad to have it in place. -- Daniel Ruggeri
Re: svn commit: r1776076 - in /httpd/httpd/trunk: docs/manual/mod/mod_proxy_protocol.xml modules/filters/mod_proxy_protocol.c
On 12/26/2016 3:42 PM, j...@apache.org wrote: > Author: jim > Date: Mon Dec 26 21:42:26 2016 > New Revision: 1776076 > > URL: http://svn.apache.org/viewvc?rev=1776076&view=rev > Log: > revert back... no conflict w/ name > > Modified: > httpd/httpd/trunk/docs/manual/mod/mod_proxy_protocol.xml > httpd/httpd/trunk/modules/filters/mod_proxy_protocol.c > ... I feel a little dissonance toward this being a separate module versus being folded into mod_remoteip. This is mostly since mod_remoteip already has a bit more of a comprehensive suite of access checks for internal/trusted proxies and is (what I would expect) users would look toward first to enable this kind of functionality. The other part is that it could also help avoid namespace confusion with the many server-side proxy modules since they their module names with mod_proxy_* and their directives with Proxy*. Thoughts? As a side note, this should probably have checks for APR_HAVE_IPV6 scattered here and there during the address parsing stuff. -- Daniel Ruggeri
Re: svn commit: r1776076 - in /httpd/httpd/trunk: docs/manual/mod/mod_proxy_protocol.xml modules/filters/mod_proxy_protocol.c
On 12/28/2016 7:49 AM, Jim Jagielski wrote: > Sounds good... We could simply move the filter aspects over to > mod_remoteip and save us a module :) This may be a naive question, but how would the copyright for such mixed copyright sources be handled in the file for such a case? -- Daniel Ruggeri
Re: svn commit: r1776575 - in /httpd/httpd/trunk: docs/log-message-tags/next-number docs/manual/mod/mod_remoteip.xml modules/metadata/mod_remoteip.c
I went ahead and ported this code to mod_remoteip. I also taught it how
to be optional (process PROXY headers when available, but don't fail
otherwise), checks for IPv6 in APR as well as log numbers. A small bit
of refactoring was included.
A few questions came up during initial code review and the porting...
opinions would be appreciated.
On 12/30/2016 8:20 AM, drugg...@apache.org wrote:
> Author: druggeri
> Date: Fri Dec 30 14:20:48 2016
> New Revision: 1776575
>
> URL: http://svn.apache.org/viewvc?rev=1776575&view=rev
> Log:
> Merge new PROXY protocol code into mod_remoteip
>
> Modified:
> httpd/httpd/trunk/docs/log-message-tags/next-number
> httpd/httpd/trunk/docs/manual/mod/mod_remoteip.xml
> httpd/httpd/trunk/modules/metadata/mod_remoteip.c
. . .
> Modified: httpd/httpd/trunk/modules/metadata/mod_remoteip.c
> URL:
> http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/metadata/mod_remoteip.c?rev=1776575&r1=1776574&r2=1776575&view=diff
> ==
> --- httpd/httpd/trunk/modules/metadata/mod_remoteip.c (original)
> +++ httpd/httpd/trunk/modules/metadata/mod_remoteip.c Fri Dec 30 14:20:48 2016
> @@ -12,15 +12,20 @@
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> + *
> + * The majority of the input filter code for PROXY protocol support is
> + * Copyright 2014 Cloudzilla Inc.
> */
Is this sufficient attribution?
> +/* XXX: Unsure if this is needed if v6 support is not available on
> + this platform */
> +#ifndef INET6_ADDRSTRLEN
> +#define INET6_ADDRSTRLEN 46
> +#endif
I don't have a non-IPv6 capable platform so I'm not sure if this will be
defined anyway. Is this needed?
> +
> +/* add our filter */
> +if (!ap_add_input_filter(remoteip_filter_name, NULL, NULL, c)) {
> +/* XXX: Shouldn't this WARN in log? */
> +return DECLINED;
> +}
This came from the original code... but I assume that if the admin
desired for this to be configured, but we failed to enable it... we
should at least complain. Maybe more?
--
Daniel Ruggeri
Re: svn commit: r1776575 - in /httpd/httpd/trunk: docs/log-message-tags/next-number docs/manual/mod/mod_remoteip.xml modules/metadata/mod_remoteip.c
On 12/30/2016 8:28 AM, Jim Jagielski wrote: > Kewl beans! Indeed - the best beans to have! > Any issue if we rename the directive to just ProxyProtocolEnable? > The "RemoteIP" prefix just seems weird :) I assume we try to keep a "namespace" for the more optional modules like this. It does seem weird and is unnecessarily long but starting with "Proxy" would lead me to think this directive _belongs_ to one of the mod_proxy modules. > Also, just as a head's up, I'm looking on adding PROXY support > to the proxy module itself (that is, we *send* the PROXY line > to backends) as a configurable option. So when that > happens, we may wish to rethink the command again. Yes, I planned the same (not cookie licking! It's all yours if you want it!) which kinda makes me like the RemoteIP prefix on this new directive to further differentiate it from the client-side work of mod_proxy_*. In the case of mod_proxy sending the header, a directive like "ProxySendProxyProtocolHeader On" seems like a viable option (again, long... and kinda redundant... but "clear" in my mind who's doing what). Pending further responses to the followup I sent a moment ago, I'll do another commit for final cleanup and to remove the new module now ported to remoteip and will then work on a backport for 2.4 -- Daniel Ruggeri
Re: The Version Bump fallacy [Was Re: Post 2.4.25]
On 12/28/2016 6:40 PM, Yehuda Katz wrote: > On Wed, Dec 28, 2016 at 12:35 AM, William A Rowe Jr > mailto:wr...@rowe-clan.net>> wrote: > > Our adoption is *broadly* based on the OS distributions > from vendors, not from people picking up our sources. > Yes - some integrate directly from source, and others > use a non-OS distribution. > > > I think a significant number of users of nginx add the official nginx > yum/apt sources and keep up to date that way > (http://nginx.org/en/linux_packages.html#mainline). > This is particularly true because the vendor-supplied version are so > old. You can see this in the w3techs data: nginx 1.10.2 came out in > October and already makes up 75% of all nginx 1.10 users. nginx 1.11.8 > usage has similar trends. > > A possible solution to this would be to start publishing binaries in a > package-manager-accessible format. > I am confident it would see a much higher rate of adoption. > > - Y I feel strongly about this... As a package builder/maintainer at $dayjob, this idea terrifies me. Given the huge variation in distributions and what is current on those platforms, the "best" option I see is to build for the least common denominator (minimum common libc, APR, APR-UTIL, openssl, openldap, etc). Otherwise, the package may only work on sufficiently modern installations. Things like Docker containers for the different distros are nice, but I'm not sure those are guaranteed to be current or accurately represent what an installation will look like. Additionally, vendors set different prefixes or split their configurations up differently meaning we would then have to bite off the work of creating vendor-specific packages (sucks for us) or force a standard installation format (sucks for operators of the servers). A really good illustration of this challenge is the layout differences between Debian and CentOS where even the name of the server binary is changed from "httpd" to "apache2" in the former distro. Or worse... we would have to bundle/vendor a copy of the dependencies inside the httpd package. This becomes a nightmare for the package builders because (as wrowe pointed out recently) it requires us to build these updated libraries and push the new package at some cadence as well as changing library search paths to potentially funky locations. It also becomes a challenge for server operators because a library now exists in two locations on the machine so compliance auditing gets forked (my httpd installation may be using openssl 1.0.2j but my postfix server may be using 0.9.8zh). Also, I'm sure it goes without saying, but we can't reasonably consider either approach without full CI... doing all this manually is unmaintainable (heh - ask me how I know). -- Daniel Ruggeri
Re: svn commit: r1776575 - in /httpd/httpd/trunk: docs/log-message-tags/next-number docs/manual/mod/mod_remoteip.xml modules/metadata/mod_remoteip.c
On 12/30/2016 9:04 AM, Jim Jagielski wrote: > I was thinking something like > > ProxyProtocolEnable Incoming | Outgoing | Both | Optional | Off > > so that we have 1 command for all possible PROXY PROTOCOL usages. > This, I think, is clearer for end-users. I do like the idea of being nice to end users... but this may be a dumb followup question showing a lack of understanding. How would we trigger behavior in two different modules for the same directive unless the directive were added to core? > However, I also see the value in having it as a ProxySet > parameter for outgoing, so I'm fine whichever way ;) Actually, I didn't even consider this as an option but *really* like the idea. It aligns with how we handle flags for outbound behavior already plus aligns with how HAProxy handles it (on your server declaration you simply add the "send-proxy" flag). Not that we have to work the same as another server out there, but as mentioned above, it's nice to be friendly to the operator by being consistent with expectations. Between the options of an all-in-one directive and a directive for incoming with a flag for outgoing, I definitely like the latter option. -- Daniel Ruggeri
Re: svn commit: r1776616 - in /httpd/httpd/trunk/docs/manual/mod: mod_remoteip.html.en mod_remoteip.xml.fr mod_remoteip.xml.meta
On 12/30/2016 12:47 PM, Luca Toscano wrote: > I personally like a lot RemoteIPProxyProtocol (rather than > RemoteIPProxyProtocolEnable that seems a bit heavy to read), but > everything is fine as long as we use a single name, especially in the > logs that admins will read :) > > I haven't checked the code in detail so I might say something > completely irrelevant, just writing the first things that I noticed! Hrm - yes... these are inconsistencies that came up as I was renaming stuff and moving it around. I also like the suggestion to shorten the name since "Enable" is repetitive... I have adjusted the references to be RemoteIPProxyProtocol in docs and code, fixed references to the "PROXY protocol" to align with the case that HAProxy uses as well as removed the mod_proxy_protocol module in r1776624. Thanks for the pointer. -- Daniel Ruggeri
Re: svn commit: r1776575 - in /httpd/httpd/trunk: docs/log-message-tags/next-number docs/manual/mod/mod_remoteip.xml modules/metadata/mod_remoteip.c
Aye - I suspected this would raise eyebrows so I did bring it up a few times [1][2]. I'm sure we're in agreement that attribution is important in the Open Source world so I'd like to be sure it's done appropriately. I'm happy to fix. Currently, though, I'm not sure how best to handle this veto... what would be the preferred path forward? As a first step, I've remove the three lines mentioned here and added to CHANGES in r1776674. The 2.4 backport has also been modified (simply by removing the lines since CHANGES already has attribution to the original authors which seems to be preferred per your message). Does that work or did you have another approach in mind? [1] https://lists.apache.org/thread.html/95173df808992097f565bc7bcd06a01b2129415bff0aae6c608b82fe@%3Cdev.httpd.apache.org%3E [2] https://lists.apache.org/thread.html/28e660f38d945216d9d0bb4cba3e1b4336a4c5051a46f17c8f99a0f0@%3Cdev.httpd.apache.org%3E -- Daniel Ruggeri On 12/30/2016 8:00 PM, William A Rowe Jr wrote: > -1 (yes, veto.) > > In general, as the original author of this particular module, you > might notice I don't claim attribution. We rely on svn provenance and > Changes to describe code legacy, so these make me very uncomfortable, > especially when injected into our sources. That isn't the key issue. > > The statement itself may be presently true. The statement a number of > years from now might be radically false. The presence of this > statement makes it impossible for the future svn hacker to know when > to modify the statement or determine when the sources have changed > such that it becomes untrue. > > It is a relativistic value judgement, and these aren't useful as > legalistic elements, so the conventional 'portions copyright...' sort > of language is used instead.
Re: svn commit: r1776575 - in /httpd/httpd/trunk: docs/log-message-tags/next-number docs/manual/mod/mod_remoteip.xml modules/metadata/mod_remoteip.c
On 12/30/2016 8:37 PM, William A Rowe Jr wrote: > Simply address the issue that caused the -1... 'Code mostly > copyright...' needs to be 'Portions copyright'... A statement which is > unlikely to become entirely false. OK, got it. It wasn't entirely clear to me that was the hangup. On 12/30/2016 8:34 PM, Eric Covener wrote: > Hi Daniel, not to pull you in too many directions, but I would suggest > to revert r1776674 and go back to erring on the side of attribution > until we can arrive at a consensus.The license requires that > copyright notices are preserved in derivative works. Understood. Since it was a lapse in my initial commit that I didn't add to CHANGES already, I'd like to keep r1776674 to cover that mistake, but adjust mod_remoteip.c as Bill mentions. I will do that later this evening or in the morning in the absence of any disagreement. -- Daniel Ruggeri
Re: clang-analyzer?
On 1/9/2017 3:48 AM, Graham Leggett wrote: > Agreed that getting these things to zero would be a good thing to have. +1 here. The error report by itself is much less useful than an error report accompanied by a patch that addresses the benign things. -- Daniel Ruggeri
Re: RemoteIPProxyProtocol
On 1/9/2017 8:49 AM, Jim Jagielski wrote: > Once we backport to 2.4, it will be nigh-impossible to change > the name... > > As we *sure* we want to call it RemoteIPProxyProtocol instead > of just "regular" ProxyProtocol ? > > The latter just sounds and looks "more right" to me. I still like RemoteIPProxyProtocol because I also like the idea of "namespacing" modules' directives. -- Daniel Ruggeri
Re: svn commit: r1776575 - in /httpd/httpd/trunk: docs/log-message-tags/next-number docs/manual/mod/mod_remoteip.xml modules/metadata/mod_remoteip.c
tx->version = 0;
>>>> +ctx->mode = AP_MODE_READBYTES;
>>>> +ctx->bb = apr_brigade_create(f->c->pool, f->c->bucket_alloc);
>>>> +ctx->done = 0;
>>>> +}
>>>> +
>>>> +if (ctx->done) {
>>>> +/* Note: because we're a connection filter we can't remove
>>>> ourselves
>>>> + * when we're done, so we have to stay in the chain and just go
>>>> into
>>>> + * passthrough mode.
>>>> + */
>>>> +return ap_get_brigade(f->next, bb_out, mode, block, readbytes);
>>>> +}
>>>> +
>>>> +conn_conf = ap_get_module_config(f->c->conn_config, &remoteip_module);
>>>> +
>>>> +/* try to read a header's worth of data */
>>>> +while (!ctx->done) {
>>>> +if (APR_BRIGADE_EMPTY(ctx->bb)) {
>>>> +ret = ap_get_brigade(f->next, ctx->bb, ctx->mode, block,
>>>> + ctx->need - ctx->rcvd);
>>>> +if (ret != APR_SUCCESS) {
>>>> +return ret;
>>>> +}
>>>> +}
>>>> +if (APR_BRIGADE_EMPTY(ctx->bb)) {
>>> What about the case of an non blocking read where the upstream filter
>>> returns an empty brigade
>>> and APR_SUCCESS. This is equal to returning EAGAIN.
Also from the original code
>>>> +return APR_EOF;
>>>> +}
>>>> +
>>>> +while (!ctx->done && !APR_BRIGADE_EMPTY(ctx->bb)) {
>>>> +b = APR_BRIGADE_FIRST(ctx->bb);
>>>> +
>>>> +ret = apr_bucket_read(b, &ptr, &len, block);
>>>> +if (APR_STATUS_IS_EAGAIN(ret) && block == APR_NONBLOCK_READ) {
>>>> +return APR_SUCCESS;
>>>> +}
>>>> +if (ret != APR_SUCCESS) {
>>>> +return ret;
>>>> +}
>>>> +
>>>> +memcpy(ctx->header + ctx->rcvd, ptr, len);
>>>> +ctx->rcvd += len;
>>>> +
>>>> +/* Remove instead of delete - we may put this bucket
>>>> + back into bb_out if the header was optional and we
>>>> + pass down the chain */
>>>> +APR_BUCKET_REMOVE(b);
Note: In the original code this was apr_bucket_delete.
>>>> +psts = HDR_NEED_MORE;
>>>> +
>>>> +if (ctx->version == 0) {
>>>> +/* reading initial chunk */
>>>> +if (ctx->rcvd >= MIN_HDR_LEN) {
>>>> +ctx->version = remoteip_determine_version(f->c,
>>>> ctx->header);
>>>> +if (ctx->version < 0) {
>>>> +psts = HDR_MISSING;
>>>> +}
>>>> +else if (ctx->version == 1) {
>>>> +ctx->mode = AP_MODE_GETLINE;
>>>> +ctx->need = sizeof(proxy_v1);
>>>> +}
>>>> +else if (ctx->version == 2) {
>>>> +ctx->need = MIN_V2_HDR_LEN;
>>>> +}
>>>> +}
>>>> +}
>>>> +else if (ctx->version == 1) {
>>>> +psts = remoteip_process_v1_header(f->c, conn_conf,
>>>> +(proxy_header *) ctx->header,
>>>> +ctx->rcvd, &ctx->need);
>>>> +}
>>>> +else if (ctx->version == 2) {
>>>> +if (ctx->rcvd >= MIN_V2_HDR_LEN) {
>>>> +ctx->need = MIN_V2_HDR_LEN +
>>>> +ntohs(((proxy_header *)
>>>> ctx->header)->v2.len);
>>>> +}
>>>> +if (ctx->rcvd >= ctx->need) {
>>>> +psts = remoteip_process_v2_header(f->c, conn_conf,
>>>> +(proxy_header *)
>>>> ctx->header);
>>>> +}
>>>> +}
>>>> +else {
>>>> +ap_log_cerror(APLOG_MARK, APLOG_ERR, 0, f->c,
>>>> APLOGNO(03509)
>>>> + "RemoteIPProxyProtocol: internal error:
>>>> unknown version "
>>>> + "%d", ctx->version);
>>>> +f->c->aborted = 1;
>>>> +apr_bucket_delete(b);
>>>> +apr_brigade_destroy(ctx->bb);
>>>> +return APR_ECONNABORTED;
>>>> +}
>>>> +
>>>> +switch (psts) {
>>>> +case HDR_MISSING:
>>>> +if (conn_conf->proxy_protocol_optional) {
>>>> +/* Same as DONE, but don't delete the bucket.
>>>> Rather, put it
>>>> + back into the brigade and move the request
>>>> along the stack */
>>>> +ctx->done = 1;
>>>> +APR_BRIGADE_INSERT_HEAD(bb_out, b);
>>> See below. We need to restore all buckets. What if the original read was
>>> speculative?
>>>
>>>> +return ap_pass_brigade(f->next, ctx->bb);
>>> Why using ap_pass_brigade on f->next? We are an input filter and f->next is
>>> our upstream input filter.
This is probably my own misunderstanding... what would be the preferred
way to move this down the stack, then?
>>>
>>>> +}
>>>> +else {
>>>> +ap_log_cerror(APLOG_MARK, APLOG_ERR, 0, f->c,
>>>> APLOGNO(03510)
>>>> + "RemoteIPProxyProtocol: no valid
>>>> PROXY header found");
>>>> +/* fall through to error case */
>>>> +}
>>>> +case HDR_ERROR:
>>>> +f->c->aborted = 1;
>>>> +apr_bucket_delete(b);
>>>> +apr_brigade_destroy(ctx->bb);
>>>> +return APR_ECONNABORTED;
>>>> +
>>>> +case HDR_DONE:
>>>> +apr_bucket_delete(b);
>>>> +ctx->done = 1;
>>>> +break;
>>>> +
>>>> +case HDR_NEED_MORE:
>>>> +/* It is safe to delete this bucket if we get here
>>>> since we know
>>>> + that we are definitely processing a header (we've
>>>> read enough to
>>>> + know if the signatures exist on the line) */
>>> No. We might not even received MIN_HDR_LEN data so far and thus did not
>>> check for the signature
>>> so far. We need to safe all the buckets that we might need to restore in a
>>> separate bucket brigade,
>>> that we need to be set aside before doing the next ap_get_brigade to avoid
>>> killing transient buckets.
This is a good point. As a side note, in the original code, this delete
didn't exist on this line, but was done earlier (noted above).
Editorially, while I was testing a patch I was writing to do this same
work with mod_ssl down stack, I was consistently getting only 11 bytes
read and I would have expected that same thing to be an issue here
(since MIN_HDR_LEN is 16)... but never actually observed that. I'm sure
that's because ap_get_brigade on the first round through is called
requesting MIN_HDR_LEN bytes (as assigned to ctx->need). Would
ap_get_brigade ever turn less than what was requested?
>>>> +apr_bucket_delete(b);
>>>> +break;
>>>> +}
>>>> +}
>>>> +}
>>>> +
>>>> +/* we only get here when done == 1 */
>>>> +if (psts == HDR_DONE) {
>>>> +ap_log_cerror(APLOG_MARK, APLOG_DEBUG, 0, f->c, APLOGNO(03511)
>>>> + "RemoteIPProxyProtocol: received valid PROXY
>>>> header: %s:%hu",
>>>> + conn_conf->client_ip, conn_conf->client_addr->port);
>>>> +}
>>>> +else {
>>>> +ap_log_cerror(APLOG_MARK, APLOG_DEBUG, 0, f->c, APLOGNO(03512)
>>>> + "RemoteIPProxyProtocol: PROXY header was missing");
>>>> +}
>>>> +
>>>> +if (ctx->rcvd > ctx->need || !APR_BRIGADE_EMPTY(ctx->bb)) {
>>>> +ap_log_cerror(APLOG_MARK, APLOG_ERR, 0, f->c, APLOGNO(03513)
>>>> + "RemoteIPProxyProtocol: internal error: have data
>>>> left over; "
>>>> + " need=%lu, rcvd=%lu, brigade-empty=%d", ctx->need,
>>>> + ctx->rcvd, APR_BRIGADE_EMPTY(ctx->bb));
>>>> +f->c->aborted = 1;
>>>> +apr_brigade_destroy(ctx->bb);
>>>> +return APR_ECONNABORTED;
>>>> +}
>>>> +
>>>> +/* clean up */
>>>> +apr_brigade_destroy(ctx->bb);
>>>> +ctx->bb = NULL;
>>>> +
>>>> +/* now do the real read for the upper layer */
>>>> +return ap_get_brigade(f->next, bb_out, mode, block, readbytes);
>>>> +}
>>>> +
>>>> static const command_rec remoteip_cmds[] =
>>>> {
>>>> AP_INIT_TAKE1("RemoteIPHeader", header_name_set, NULL, RSRC_CONF,
>>> Regards
>>>
>>> Rüdiger
>>>
--
Daniel Ruggeri
What to call forcing a worker into error state for IO timeout?
Quick use case solved by adding a simple note in the request and checking for it later: Take a backend BalancerMember out of service if a read timeout occurs. This is nearly identical to the failonstatus parameter. I'm thinking "failontimeout" would be the most consistent. Any opposition to that as the parameter name? -- Daniel Ruggeri
Re: svn commit: r1466053 - /httpd/httpd/branches/2.4.x/STATUS
On 4/9/2013 9:25 AM, cove...@apache.org wrote: > Author: covener > Date: Tue Apr 9 14:25:44 2013 > New Revision: 1466053 > > URL: http://svn.apache.org/r1466053 > Log: > remove a superceded comment, but add a new one. > > > Modified: > httpd/httpd/branches/2.4.x/STATUS > > Modified: httpd/httpd/branches/2.4.x/STATUS > URL: > http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/STATUS?rev=1466053&r1=1466052&r2=1466053&view=diff > == > --- httpd/httpd/branches/2.4.x/STATUS (original) > +++ httpd/httpd/branches/2.4.x/STATUS Tue Apr 9 14:25:44 2013 > @@ -127,7 +127,8 @@ PATCHES PROPOSED TO BACKPORT FROM TRUNK: > 2.4.x patch: > http://people.apache.org/~druggeri/patches/AuthLDAPBindPasswordExec-2.4.patch > (20130119 - updated to include minor mmn bump) > +1: druggeri, jim > -covener: minor MMM bump? (Thanks for reminder. This is done. -druggeri) > +covener: wrowe mentioned a thread-safety issue in 2.2 status, what up > with that? > + > >* mod_lua: Sync 2.4 branch with trunk. This includes (but is not limited > to) > Server pools, new apr functions for the request_rec table, > input/ > > Not sure - good question. Can anyone else advise (I pinged Bill on the side, but suspect he's busy)? I guess a "big hammer" approach to enable thread safety would be to wrap the whole thing with a static mutex... but I'm not sure that's the most elegant solution. I'm also not sure I see what is unsafe... -- Daniel Ruggeri
Re: svn commit: r1467433 - /httpd/httpd/branches/2.2.x/STATUS
On 4/12/2013 2:32 PM, wr...@apache.org wrote: > Author: wrowe > Date: Fri Apr 12 19:32:55 2013 > New Revision: 1467433 > > URL: http://svn.apache.org/r1467433 > Log: > Belated explantion for druggeri > > Modified: > httpd/httpd/branches/2.2.x/STATUS > > Modified: httpd/httpd/branches/2.2.x/STATUS > URL: > http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/STATUS?rev=1467433&r1=1467432&r2=1467433&view=diff > == > --- httpd/httpd/branches/2.2.x/STATUS (original) > +++ httpd/httpd/branches/2.2.x/STATUS Fri Apr 12 19:32:55 2013 > @@ -189,7 +189,8 @@ PATCHES PROPOSED TO BACKPORT FROM TRUNK: > 2.2.x patch: > http://people.apache.org/~druggeri/patches/AuthLDAPBindPasswordExec-2.2.patch >(20130119 - updated to include minor mmn bump) > +1: druggeri > - -1: wrowe (switch to +1 once ap_get_exec_line is made thread-safe) > + -1: wrowe (switch to +1 once ap_get_exec_line is made thread-safe, right > + now it uses a static which is verboten in reentrant code.) > kudos for using apr_tokenize_to_argv to allow spaces in args. > >* mod_ssl: Quiet FIPS mode weak keys disabled and FIPS not selected emits Argh... That makes sense now. Missed that as I ripped the code out of mod_ssl. I have updated the patches at: http://people.apache.org/~druggeri/patches/AuthLDAPBindPasswordExec-2.2.patch http://people.apache.org/~druggeri/patches/AuthLDAPBindPasswordExec-2.4.patch ... and corrected trunk + notes in STATUS. Thanks, Bill. If that works for you, I've also proposed this in 2.4 and could use votes there, too :) -- Daniel Ruggeri
Re: svn commit: r1467523 - /httpd/httpd/trunk/server/util.c
On 4/14/2013 7:46 AM, Christophe JAILLET wrote:
> Le 13/04/2013 02:07, drugg...@apache.org a écrit :
>> Author: druggeri
>> Date: Sat Apr 13 00:07:44 2013
>> New Revision: 1467523
>>
>> URL: http://svn.apache.org/r1467523
>> Log:
>> Static var not neccessary here
>>
>> Modified:
>> httpd/httpd/trunk/server/util.c
>>
>> Modified: httpd/httpd/trunk/server/util.c
>> URL:
>> http://svn.apache.org/viewvc/httpd/httpd/trunk/server/util.c?rev=1467523&r1=1467522&r2=1467523&view=diff
>> ==
>>
>> --- httpd/httpd/trunk/server/util.c (original)
>> +++ httpd/httpd/trunk/server/util.c Sat Apr 13 00:07:44 2013
>> @@ -2955,7 +2955,7 @@ AP_DECLARE(char *) ap_get_exec_line(apr_
>> const char *cmd,
>> const char * const * argv)
>> {
>> -static char buf[MAX_STRING_LEN];
>> +char buf[MAX_STRING_LEN];
>> apr_procattr_t *procattr;
>> apr_proc_t *proc;
>> apr_file_t *fp;
> Without the static attribute, gcc gives the following warning:
>
> util.c: In function 'ap_get_exec_line':
> util.c:2997:5: warning: function returns address of local variable
> [enabled by default]
>
>
> Either, you should allocate memory another way or change the prototype
> of the function.
> The best would be IMO to pass a new char* parameter and the length of
> this buffer.
>
> Best regards,
> jailletc36
Sorry, yes... somehow I missed adding this in my commit but not in my
testing directory:
return apr_pstrndup(p, buf, k);
Will return a string allocated out of the pool instead.
--
Daniel Ruggeri
Re: svn commit: r1476688 - /httpd/httpd/branches/2.4.x/STATUS
On 4/27/2013 5:30 PM, minf...@apache.org wrote: > Author: minfrin > Date: Sat Apr 27 22:30:54 2013 > New Revision: 1476688 > > URL: http://svn.apache.org/r1476688 > Log: > Vote, promote. > > Modified: > httpd/httpd/branches/2.4.x/STATUS > > Modified: httpd/httpd/branches/2.4.x/STATUS > URL: > http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/STATUS?rev=1476688&r1=1476687&r2=1476688&view=diff > == > --- httpd/httpd/branches/2.4.x/STATUS (original) > +++ httpd/httpd/branches/2.4.x/STATUS Sat Apr 27 22:30:54 2013 > @@ -90,6 +90,12 @@ RELEASE SHOWSTOPPERS: > PATCHES ACCEPTED TO BACKPORT FROM TRUNK: >[ start all new proposals below, under PATCHES PROPOSED. ] > > + * mod_proxy_balancer: Add failontimeout parameter. Timeout will put worker > + in error state if an IO timeout is detected. > + trunk patch: http://svn.apache.org/viewvc?view=revision&revision=1465839 > + 2.4.x patch: > http://people.apache.org/~druggeri/patches/httpd-2.4.x-failontimeout.patch > + 2.2.x patch: > http://people.apache.org/~druggeri/patches/httpd-2.2.x-failontimeout.patch > + +1: druggeri, jim, minfrin > > > PATCHES PROPOSED TO BACKPORT FROM TRUNK: > @@ -200,13 +206,6 @@ PATCHES PROPOSED TO BACKPORT FROM TRUNK: > 2.4.x patch: http://people.apache.org/~jim/patches/wstunnel.patch > +1: jim, minfrin (with fix to mmn bump) > > - * mod_proxy_balancer: Add failontimeout parameter. Timeout will put worker > - in error state if an IO timeout is detected. > - trunk patch: http://svn.apache.org/viewvc?view=revision&revision=1465839 > - 2.4.x patch: > http://people.apache.org/~druggeri/patches/httpd-2.4.x-failontimeout.patch > - 2.2.x patch: > http://people.apache.org/~druggeri/patches/httpd-2.2.x-failontimeout.patch > - +1: druggeri, jim > - > * skiplist: Add skiplist functionality >trunk patches: https://svn.apache.org/r1411190 > https://svn.apache.org/r1411274 Site note about this patch and the AuthLDAPBindPassword patch that were accepted in 2.4: Both have been proposed for 2.2 if anyone could spare the time to verify/vote there as well. -- Daniel Ruggeri
Re: Improve mod_proxy's error marking of workers
On 5/7/2013 2:00 PM, Jim Jagielski wrote: > Agreed... An "all or nothing" setting will likely create more > trouble than not. > > On May 7, 2013, at 8:08 AM, Eric Covener wrote: > >> On Tue, May 7, 2013 at 6:24 AM, Thomas Eckert >> wrote: >>> Attached patch contains a directive to improve the error marking of workers. >>> Basically, some errors will cause a worker to be marked as "in error" while >>> others don't. I can't see a reason for this so I added a directive to have >>> all errors mark the error correctly - especially useful for automated >>> systems using mod_proxy to check if the system is healthy. >> I think you need to screen out 4xx at least to prevent client errors >> from marking down your backends. >> There is also failonstatus which, granted, could probably be enhanced to accept ranges instead of just a list to accommodate this need. -- Daniel Ruggeri
Re: disable pid file writing?
On 5/8/2013 3:29 PM, Rainer Jung wrote: > Careful: I didn't test it but we delete the pid file during web server > shutdown. That might remove /dev/null then. > > On a quick look through the code I had the impression you can not easily > get rid of the pid file. Agreed - setting to /dev/null under the current code also fails startup anyway with the following error: (20014)Internal error: Error retrieving pid file /dev/null Remove it before continuing if it is corrupted. I haven't looked into it any further than that, though. -- Daniel Ruggeri
RE: T-24:00:00 to 2.0, 2.2 final tags
Nothing to stall the work, but I'd like to campaign once more for a quick look at the 2.2 backport I have outstanding (which was added to 2.4 already). Cheers -- Daniel Ruggeri Original message From: "William A. Rowe Jr." Date: 06/20/2013 4:07 PM (GMT-05:00) To: dev@httpd.apache.org Subject: T-24:00:00 to 2.0, 2.2 final tags Devs, I plan to tag both the 2.0 and 2.2 trees now that they look relatively stable, given that we are a couple weeks beyond the originally discussed 2.0/2.2/2.4 simultaneous release. There is one backport to apply in 2.2 and we had been waiting for Jeff to confirm that the proposed security fix now addresses all of his issues now in the 2.0 tree. I have free time to do this tomorrow late afternoon, so if anyone has their fingers in the trees with last minute work, send me a heads up so I can avoid colliding with your efforts. If there is a 2.4 vote proceeding smoothly we can always defer the announce so that there is a single, clear message we recommend 2.4, that 2.2 is a maintenance release for legacy users, and that 2.0 represents the final release of that software and that it will no longer be maintained by the project.
Re: T-24:00:00 to 2.0, 2.2 final tags
On 6/21/2013 10:19 AM, William A. Rowe Jr. wrote: > On Thu, 20 Jun 2013 17:27:14 -0400 > Daniel Ruggeri wrote: > >> Nothing to stall the work, but I'd like to campaign once more for a >> quick look at the 2.2 backport I have outstanding (which was added to >> 2.4 already). > Which, the one which is approved, or failontimeout? Sorry - I should have been more clear. The failontimeout balancer parameter is the one still up for votes while the AuthLDAPBindPassword patch has been accepted. I see you JUST added a vote, so thanks for that :-) -- Daniel Ruggeri
Re: [VOTE] The 'RM' Baton
On 7/10/2013 7:13 AM, Eric Covener wrote: > So my concern with the proposal -- are there really wiling/able RM's > waiting in the wings in these periods? If they're there -- are they > afraid of stepping on an RM's toes, or of drawing a line in the sane > for the half-approved backports? > > (I have personally never RM'ed. To me it's intimidating and unknown > and with a log of competing work it's hard to step up and tackle. > Maybe we should make it make a condition of PMC membership at least > once per decade?) Sure, I can do this if it's a time thing. The doc [1] seems straight forward enough and I can make the time to roll a release (though I wouldn't mind a practice swing or two first). The guts of the procedure itself seems like it could easily be scripted. To get back on the topic, I don't see a lot of firm "no, you can't release now because I'm not done working" messages. My perception is that it boils down to a matter of time (or lack thereof) for the RM's. There also isn't a guideline set forth that states exactly when to release so that all too familiar this-product-has-to-ship-on-July-8th-or-our-company-is-sunk pressure that "motivates" devs isn't quite there. [1] http://httpd.apache.org/dev/release.html#how-to-do-a-release -- Daniel Ruggeri
Re: [VOTE] Release Apache httpd 2.4.6 as GA
On 7/15/2013 11:48 AM, Jim Jagielski wrote: > The pre-release test tarballs for Apache httpd 2.4.6 can be found > at the usual place: > > http://httpd.apache.org/dev/dist/ > > I'm calling a VOTE on releasing these as Apache httpd 2.4.6 GA. > NOTE: The -deps tarballs are included here *only* to make life > easier for the tester. They will not be, and are not, part > of the official release. > > [ ] +1: Good to go > [ ] +0: meh > [ ] -1: Danger Will Robinson. And why. > > Vote will last the normal 72 hrs. +1 on RHEL 5/gcc/x86, Sol 5.10/SUNWspro/sparc, AIX 6.1/XLC/ppc -- Daniel Ruggeri
Re: UDS and Balancers
On 7/23/2013 11:06 AM, Jim Jagielski wrote: > druggeri notes that the uds support in trunk fails if used for > a balancemember... Are there any reports/backtraces/etc > re: this? It would be a nice thing for me to play around w/ > while on vaca Jim; No, while I was testing UDS against the patches provided I came across it myself. I don't have a server running on a UDS on my test machine but my normal proxy tests always involve a balancer. It seems to only be a matter of parsing. Error message is "BalancerMember must define remote proxy server" when BalancerMember is set to something like http://socket=%2Fhome%2Fwww%2Fsocket I didn't pursue further than that since it doesn't seem like the functionality would be used a lot in balancers and can be ironed out in future patches if that is desired. -- Daniel Ruggeri
Re: reverse proxy bind backend connection to request
On 7/23/2013 4:31 AM, Thomas Eckert wrote: > In a reverse proxy scenario, I want to do the following > > 1) read incoming request A and keep it on hold > 2) set up connection to backend for (new) request B > 3) send request B and read response over that backend connection > 4) "bind" that backend connection to request A so that mod_proxy will > use that connection when sending request A. No other request is to use > that backend connection, so prevent mod_proxy from reusing that > backend connection for any request other then request A (and it's sub > requests) > > I'm done with steps 1 to 3 but the last one is somewhat tricky. My > current plan is to patch mod_proxy_http.c so it will only set up a new > backend connection for a request if it cannot find an existing one in > some table (probably use some module config to get that data across). > But this plan lacks a part of step 4) which is to prevent any other > request from using that backend connection. Is there a 'built-in' way > to do this ? If there is none, what would be the pitfalls I should > watch out for when writing a patch for that specific part of mod_proxy ? > > Cheers, > Thomas Have you tried with the ENV variables proxy-initial-not-pooled and proxy-nokeepalive set? Seems like that combo might do what you are hoping for... -- Daniel Ruggeri
Fixing UDS in trunk/2.4 proposal
So I'm tasked with making httpd hold its own weight better against nginx
as a reverse proxy to a local service. Unfortunately, nginx supports UDS
and we don't quite yet. I've come across a bug that seems easy enough to
fix, but I am wondering if there's a better way. Thoughts are welcome.
With the currently proposed UDS support in mod_proxy, first requests
always seem fine but subsequent requests for the worker fail (attempted
DNS lookup when none should be done). I wouldn't have +1'ed had I
realized this at the time, but I have a proposed fix...
--- httpd-2.4.6-UDS/modules/proxy/proxy_util.c 2013-08-09
15:12:23.0 -0500
+++ httpd-2.4.6/modules/proxy/proxy_util.c 2013-08-09
15:15:33.0 -0500
@@ -2127,7 +2127,7 @@
*/
if (!conn->hostname || !worker->s->is_address_reusable ||
-worker->s->disablereuse || strncmp(conn->hostname, "socket=",
7) == 0) {
+worker->s->disablereuse) {
if (proxyname) {
conn->hostname = apr_pstrdup(conn->pool, proxyname);
conn->port = proxyport;
I haven't spent a ton of time on this so I'm wondering... This seems
simple enough, but isn't there a place we could do this once to avoid
having to execute the same logic (substring and path decode) on all
subsequent requests? I'd also much rather not have to do a string
comparison on all subsequent hits...
If I don't hear anything otherwise, I'll just commit this and add it to
the backport proposal next week or so
P.S.
My simple tests with 100 concurrent users for a total of 1,000,000
requests yielded the following numbers (with the above patch applied).
The backend supports about 20k requests/sec. This seems to be a mighty
compelling case for UDS...
nginx UDS: 16001.28
nginx UDS: 18138.94
nginx UDS: 15499.64
Apache UDS: 16348.70
Apache UDS: 14580.92
Apache UDS: 15211.97
Apache TCP: 11859.46
(only got one in)
--
Daniel Ruggeri
Re: svn commit: r1513508 - /httpd/httpd/trunk/modules/proxy/proxy_util.c
On 8/13/2013 10:01 AM, Jeff Trawick wrote:
>
> --- httpd/httpd/trunk/modules/proxy/proxy_util.c (original)
> +++ httpd/httpd/trunk/modules/proxy/proxy_util.c Tue Aug 13
> 14:35:47 2013
> @@ -2119,7 +2119,7 @@ ap_proxy_determine_connection(apr_pool_t
> * spilling the cached addr from the worker.
> */
> if (!conn->hostname || !worker->s->is_address_reusable ||
> -worker->s->disablereuse) {
> +worker->s->disablereuse || strncmp(conn->hostname,
> "socket=", 7) == 0) {
>
>
> I guess this means FastCGI connections can't be reused regardless of
> the configuration? (That doesn't seem like a good thing; maybe users
> should have to configure disablereuse if reuse doesn't work for them.)
>
> I guess you need to turn off FCGI_KEEP_CONN in send_begin_request() if
> reuse is always disabled?
Jeff;
I'm not sure I follow - can you help me understand how this messes
with fcgi? As I understand, it wouldn't match what this string
comparison is seeking... right?
--
Daniel Ruggeri
Re: uds support
On 8/23/2013 2:50 PM, Jim Jagielski wrote: > I've also looked at (and am hoping it works out) > > sock://var/run/server.sock|http://localhost/foo/bar > > The reason I like using '|' is it's quite Perlish :) > The other advantage is that it keep both "sides" completely > separate and self-contained. I like this one better... it's definitely the "cleanest" representation. I'm assuming that the post-pipe stuff gets parsed as usual and then 'localhost' gets sent as the Host header (unless ProxyPreserveHost is set)? -- Daniel Ruggeri
Re: Planning for 2.4.7 in Oct
On 9/5/2013 8:08 AM, Jim Jagielski wrote: > ... UDS support > for mod_proxy ... votes++ ... speaking of which. Have you decided on a direction to take the config string syntax/tested it out? I have a test environment ready to try out any patches if you'd like another set of eyes. -- Daniel Ruggeri
Re: uds support
On 10/16/2013 6:43 AM, Jim Jagielski wrote: > One thing about "lumping" both the UDS path *and* the > actual URL into the name field is that it really limits > the size of both, such that, long term, I think it will > come back and bite us. Since last night I've been working > on a plan to simply create a new field for the path, > which gives us a lot more breathing room and places > less restrictions on URL and pathname length. Been MIA for the past few days and I'm confused on what the final direction is - can you clarify? FWIW, I rather liked this idea: On Oct 11, 2013, at 12:24 PM, Jim Jagielski wrote: > Committed in r1531340 the above is implemented... kinda. > I instead went with > > http://localhost/foo/bar|sock:/var/run.s.sock > > which worked out just a bit cleaner... ... because as a server administrator, it's fairly clear what's going on. Can you elaborate on what the challenges were with merging the two paths? Why merge them at all? It seems logical that everything after the pipe is used to open the socket and everything prior is treated as it has always been and never shall the two mix... or am I over simplifying things? IMO, the syntax you suggested on the 11th also allows for a bit of "futureproofing" in that "sock:" can be replaced with all kinds of things down the road. Maybe some day I'll have an ethernet cord plugged into my ear and it'll become http://localhost/memory|brain:/pub/wetware.sock :-) Also Per some other discussion, it seems like using localhost as HTTP host is too restrictive. I'd hate to think that a UDS backend can't implement its own concept of name-based vhosts behind Apache because localhost is forced as the Host header. -- Daniel Ruggeri
Exposing more loggable data from the proxy
As I stand up a simple IPv6 test proxy that supports both AF_INET and AF_INET6 addresses, I was looking for a way to log what addr family (and maybe the IP address) mod_proxy settled on for each request in the access_log. I'm not seeing a way to do that (but correct me if I'm missing something) and was poking through the code and got to thinking that there are all kinds of data bits that'd be interesting to have available in the ENV. I'm thinking it'd be worth adding a directive (ProxyAddEnvironment?) that adds these ENV entries to each r->subprocess_env: * Host header sent to backend (useful when dynamic targets are used) * Target DNS name if set * Target IP address * Target Address family * Target port * Target connection protocol * Flag for SSL enabled All of the data is readily available once a connection is acquired in ap_proxy_acquire_connection sans the HTTP Host header. Aside from logging, exporting these as ENV entries to the request allows us to do all sorts of stuff in other modules, too Any thoughts? Is there something I should include or exclude before I begin? -- Daniel Ruggeri
Re: Exposing more loggable data from the proxy
On 10/25/2013 8:16 AM, Jeff Trawick wrote:
> (unrefined, right out of my ... head)
:-)
> useful to have a convention (if not API) for how this info is made
> available for logging, etc., so that other modules can play the same
> game (e.g., mod_jk, FastCGI, whatever)
I think what I'd be proposing is to follow in the footsteps that mod_ssl
set but MAKE it the general convention. A lot of env variables are
available so you can do all sorts of stuff. The *other* convention could
simply be the notes mechanism but I think there's better ways. In any
event, there should be some good documentation to let server admins know
what's available - that's the main reason I even suggested adding a
directive (so the list of ENV entries are exposed like mod_ssl has
documented)
> what about a plugin with optional functions that has APIs for
> recording backend state that is meaningful across variety of "gateway"
> modules? for now maybe it is just for logging, but it could save in
> shared memory for extraction in mod_status or other reports
>
> maybe that solution is a bit farfetched, but I guess the theme is that
> creating a proxy-specific solution can be a wasted effort given the
> same need for any number of other "gateway" modules
The main reason I lean toward env is because a few modules already use
env variables in meaningful ways today (big use-cases for me are log,
rewrite, headers, expr).
So... in that case, I think unrefined kinda works in our favor.
On 10/25/2013 9:10 AM, Yann Ylavic wrote:
> How about setting backend->r to r->backend (when applyable)?
At a cursory look, that'd probably suffice if I wanted to log FROM
mod_proxy. I wasn't clear, but my initial goal was to just add a
%{proxy_addr_family}e chunk in the access log format so the request and
target info is handy on a single line. As mentioned to Jeff, I'd also
like a little more utility from the work involved. Plus, as you say, I
haven't looked to see how well those lifecycles mesh.
On 10/25/2013 9:21 AM, Eric Covener wrote:
> I think the how simple the data is compared to that underlying
> structure, I would rather see it in a flat table that other proxy-like
> things could emulate. Similar to SSL variables.
That was kind of the idea... the stuff I'm looking for is pretty basic
but I wanted to shoot an email along the lines of, "Anyone else want a
beer while I'm in the fridge?" to see if there were some other
attributes that are worth gathering. And yes, SSL led me to the suggestion.
--
Daniel Ruggeri
Re: persisting the slotmem (Was: Re: mod_proxy: maximum hostname length for workers)
On 11/8/2013 12:42 PM, Jim Jagielski wrote: > This has me thinking... we should likely do something to > better error-check the store/restore aspects of slotmem. > Even some sort of quick checksum would be better than > what we have now. :/ > > Gotta mull this over a bit more. +1 to that. I couldn't imagine someone (or thing) would go muck with that file... but stranger things have happened on MY file systems :-) -- Daniel Ruggeri
Re: Intent to T&R 2.4.7
On 11/13/2013 10:39 AM, Jim Jagielski wrote: > Now that APR 1.5 is soon-to-be released, we are good for > a release of 2.4.7. > > I propose a T&R next week (I'll RM) and would request > that people look thru STATUS for some remaining backports. Are you hoping to push for UDS in 2.4.7? Seems like a great feature... (yes, I'm guilty in not testing out the latest trunk patches and providing feedback - I had planned to do that last week... and this week... and probably next week) -- Daniel Ruggeri
Re: [VOTE] Release Apache httpd 2.2.26 as GA
On 11/13/2013 11:03 AM, Jim Jagielski wrote: > [X] +1: Good to go > [ ] +0: meh > [ ] -1: Danger Will Robinson. And why. Verified on Debian 7.0 "wheezy" w/ openssl 0.9.8 and php 5.3 as accompaniments -- Daniel Ruggeri
Re: Intent to T&R 2.4.7
*Finally* ran this through my test cases with three poundings with wrk. Here are the requests/sec: httpd (2.4 + proposed UDS patch) Req/Sec 147.34 28.89 282.00 71.38% Req/Sec 147.48 27.18 250.00 67.75% Req/Sec 147.87 28.17 239.00 70.94% nginx Req/Sec 180.99 27.81 311.00 77.92% Req/Sec 183.46 32.59 369.00 72.44% Req/Sec 176.81 28.18 325.00 71.99% Three samples with 30 second tests is far from scientific, but I think illustrates general timings (and, more importantly, that it works!). -- Daniel Ruggeri On 11/14/2013 9:56 AM, Jim Jagielski wrote: > What the heck. STATUS is updated w/ the backport proposal > and the patch... > > On Nov 14, 2013, at 7:46 AM, Jim Jagielski wrote: > >> I'd like to yes, but I don't want to push 2.4.7 out much >> longer. There are other things in STATUS, like the event patches >> which have been running on ASF infra for quite awhile, that >> I'd like to see in 2.4.7 when we ship. We can save UDS for >> 2.4.8 and make that a(nother) reason for people to upgrade. >> >> On Nov 13, 2013, at 8:27 PM, Daniel Ruggeri wrote: >> >>> On 11/13/2013 10:39 AM, Jim Jagielski wrote: >>>> Now that APR 1.5 is soon-to-be released, we are good for >>>> a release of 2.4.7. >>>> >>>> I propose a T&R next week (I'll RM) and would request >>>> that people look thru STATUS for some remaining backports. >>> Are you hoping to push for UDS in 2.4.7? Seems like a great feature... >>> >>> (yes, I'm guilty in not testing out the latest trunk patches and >>> providing feedback - I had planned to do that last week... and this >>> week... and probably next week) >>> >>> -- >>> Daniel Ruggeri >>>
Re: Intent to T&R 2.4.7
Oops - I copypasta'd the per-thread stats. Total stats for the test follow: httpd: Requests/sec: 4633.17 Requests/sec: 4664.49 Requests/sec: 4657.63 nginx: Requests/sec: 5701.16 Requests/sec: 5798.08 Requests/sec: 5584.60 -- Daniel Ruggeri On 11/18/2013 1:09 PM, Daniel Ruggeri wrote: > httpd (2.4 + proposed UDS patch) > Req/Sec 147.34 28.89 282.00 71.38% > Req/Sec 147.48 27.18 250.00 67.75% > Req/Sec 147.87 28.17 239.00 70.94% > > nginx > Req/Sec 180.99 27.81 311.00 77.92% > Req/Sec 183.46 32.59 369.00 72.44% > Req/Sec 176.81 28.18 325.00 71.99%
Re: Intent to T&R 2.4.7
And... this is a bit discouraging, but as a comparison to the older UDS patch 2.4.6 + original UDS patch: Requests/sec: 5347.17 Requests/sec: 5102.16 Requests/sec: 5074.15 This is a sizable difference... Note that the current 2.4 backport proposal was applied to 2.4.6 since that is what I tested the original patch with (to keep everything apples to apples). I'll jump in to take a look at this when time is available (next week?) but would like to fish for any immediate thoughts in the mean time. -- Daniel Ruggeri On 11/18/2013 1:11 PM, Daniel Ruggeri wrote: > Oops - I copypasta'd the per-thread stats. Total stats for the test follow: > httpd: > Requests/sec: 4633.17 > Requests/sec: 4664.49 > Requests/sec: 4657.63 > > nginx: > Requests/sec: 5701.16 > Requests/sec: 5798.08 > Requests/sec: 5584.60
Re: UDS Patch
On 11/18/2013 3:38 PM, Jim Jagielski wrote:
> Can you retry with this applied:
>
> https://svn.apache.org/viewvc?view=revision&revision=1543174
Definitely. I'll report back tomorrow so long as the universe wills
it... but one last note
I failed to mention in my original notes that there were two hunks that
didn't apply cleanly to 2.4.6 - these appear to be from this change:
https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/proxy/proxy_util.c?r1=1511313&r2=1511312&pathrev=1511313
... which is in the neighborhood of what you adjusted in r1543174... but
doesn't appear to conflict directly.
I'm thinking I should also apply r1511313 to 2.4.6 as a prereq to
r1543174 in order to remove ambiguity... I'm frankly not sure if the
machine was performing DNS lookups during the test or not (and I have
only given this a cursory review), but that would *definitely* account
for a measurable slowdown.
The context of what was rejected:
> --- modules/proxy/proxy_util.c
> +++ modules/proxy/proxy_util.c
> @@ -2228,7 +2324,8 @@
> conn->port = uri->port;
> }
> socket_cleanup(conn);
> -if (!worker->s->is_address_reusable || worker->s->disablereuse) {
> +if (!(*worker->s->uds_path) &&
> +(!worker->s->is_address_reusable ||
> worker->s->disablereuse)) {
> /*
> * Only do a lookup if we should not reuse the backend
> address.
> * Otherwise we will look it up once for the worker.
> @@ -2239,7 +2336,7 @@
> conn->pool);
> }
> }
> -if (worker->s->is_address_reusable && !worker->s->disablereuse) {
> +if (!(*worker->s->uds_path) && worker->s->is_address_reusable &&
> !worker->s->disablereuse) {
> /*
> * Looking up the backend address for the worker only makes
> sense if
> * we can reuse the address.
I'll have to see what the delta with both patches applied turns out to be...
--
Daniel Ruggeri
Re: UDS Patch
Well, I don't have good news to report... doesn't seem to be a
significant change in behavior...
nginx:
Requests/sec: 5082.43
Requests/sec: 5111.94
Requests/sec: 5063.27
2.4.6 - First UDS patch:
Requests/sec: 4733.09
Requests/sec: 4529.49
Requests/sec: 4573.27
2.4.6 - r1511313 + new UDS patch + r1543174:
Requests/sec: 3774.41
Requests/sec: 3878.02
Requests/sec: 3852.34
Will try to look into this next week...
--
Daniel Ruggeri
On 11/18/2013 6:37 PM, Daniel Ruggeri wrote:
> On 11/18/2013 3:38 PM, Jim Jagielski wrote:
>> Can you retry with this applied:
>>
>> https://svn.apache.org/viewvc?view=revision&revision=1543174
> Definitely. I'll report back tomorrow so long as the universe wills
> it... but one last note
>
> I failed to mention in my original notes that there were two hunks that
> didn't apply cleanly to 2.4.6 - these appear to be from this change:
> https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/proxy/proxy_util.c?r1=1511313&r2=1511312&pathrev=1511313
> ... which is in the neighborhood of what you adjusted in r1543174... but
> doesn't appear to conflict directly.
>
> I'm thinking I should also apply r1511313 to 2.4.6 as a prereq to
> r1543174 in order to remove ambiguity... I'm frankly not sure if the
> machine was performing DNS lookups during the test or not (and I have
> only given this a cursory review), but that would *definitely* account
> for a measurable slowdown.
>
> The context of what was rejected:
>> --- modules/proxy/proxy_util.c
>> +++ modules/proxy/proxy_util.c
>> @@ -2228,7 +2324,8 @@
>> conn->port = uri->port;
>> }
>> socket_cleanup(conn);
>> -if (!worker->s->is_address_reusable || worker->s->disablereuse) {
>> +if (!(*worker->s->uds_path) &&
>> +(!worker->s->is_address_reusable ||
>> worker->s->disablereuse)) {
>> /*
>> * Only do a lookup if we should not reuse the backend
>> address.
>> * Otherwise we will look it up once for the worker.
>> @@ -2239,7 +2336,7 @@
>> conn->pool);
>> }
>> }
>> -if (worker->s->is_address_reusable && !worker->s->disablereuse) {
>> +if (!(*worker->s->uds_path) && worker->s->is_address_reusable &&
>> !worker->s->disablereuse) {
>> /*
>> * Looking up the backend address for the worker only makes
>> sense if
>> * we can reuse the address.
> I'll have to see what the delta with both patches applied turns out to be...
>
> --
> Daniel Ruggeri
>
Re: UDS Patch
Yes, agreed. Not sure if I made it clear, but I did apply r1511313 for the tests I did today (but not the one from yesterday). Of the several emails sent, the following have been tested: 2.4.6 w the (several) originally proposed UDS patches applied in order 2.4.6 w proposed backport (the 2 chunks around the DNS changes fail to apply since they do not exist in 2.4.6) 2.4.6 w r1511313 + proposed backport + r1543174 I DID double check that the machine wasn't requesting DNS lookups for the socket name or anything strange against the DNS server - but that was only for the test I ran today. -- Daniel Ruggeri On 11/19/2013 1:43 PM, Jim Jagielski wrote: > OK... the DNS lookup code seems to have changed between 2.4.6 and 2.4.7: > > https://svn.apache.org/viewvc?view=revision&revision=1511313 > > So I'm wondering if there's something there.
Re: UDS Patch
Sorry, I thought the diffs I sent off list were good enough. I'll have to see if I even still have the original build lying around. Effectively, I just took the list of patches in the backport proposal and applied them one at a time to the 2.4.6 sources. If I can't find the build, I'll do the same over and send that instead. -- Daniel Ruggeri On 11/22/2013 10:38 AM, Jim Jagielski wrote: > Any luck with generating the diff yet? > > On Nov 19, 2013, at 3:08 PM, Jim Jagielski wrote: > >> The main thing is that it would be interesting to see >> the diffs between '2.4.6 w the (several) originally proposed UDS patches >> applied in order' >> and '2.4.6 w proposed backport'... >> >> Those diffs should show just the differences between the UDS >> implementations... >> >> On Nov 19, 2013, at 2:51 PM, Daniel Ruggeri wrote: >> >>> Yes, agreed. Not sure if I made it clear, but I did apply r1511313 for >>> the tests I did today (but not the one from yesterday). >>> >>> Of the several emails sent, the following have been tested: >>> 2.4.6 w the (several) originally proposed UDS patches applied in order >>> 2.4.6 w proposed backport (the 2 chunks around the DNS changes fail to >>> apply since they do not exist in 2.4.6) >>> 2.4.6 w r1511313 + proposed backport + r1543174 >>> >>> I DID double check that the machine wasn't requesting DNS lookups for >>> the socket name or anything strange against the DNS server - but that >>> was only for the test I ran today. >>> >>> -- >>> Daniel Ruggeri >>> >>> On 11/19/2013 1:43 PM, Jim Jagielski wrote: >>>> OK... the DNS lookup code seems to have changed between 2.4.6 and 2.4.7: >>>> >>>>https://svn.apache.org/viewvc?view=revision&revision=1511313 >>>> >>>> So I'm wondering if there's something there.
Re: UDS Patch
I had the same inclination as Cristophe but haven't been able to substantiate anything due to lack of time last week wasn't as kind to my free time as I had hoped. This would be very easy to tweak/test. Within the next day or two I should be able to get back in to perform some rebuilds and do more thorough testing and tampering as I squeeze time in between various work-related crises. Most of my testing is automated-ish, so turnaround from patch to test results is fairly quick. Jim, what does your test setup look like to measure performance delta? My setup is fairly simple with httpd on the frontend targeting a small Node.js backend application... I don't suspect the application is skewing the results because of how consistent the results have been, but I may just remove that from the equation to be absolutely sure. -- Daniel Ruggeri On 12/2/2013 8:14 AM, Jim Jagielski wrote: > But from what I see, all of those are during non critical paths. > It's like when workers are being defined, initialized, etc and > that's only done during config or when added via balancer-manager. > > On Dec 2, 2013, at 8:09 AM, Marion et Christophe JAILLET > wrote: > >> Hi, >> >> >> one of my thought was the change from >> >>worker->s->name >> >> to >> >>ap_proxy_worker_name(r->pool, worker) >> >> in logging function. >> >> ap_proxy_worker_name allocates memory in the pool and performs some >> operations on strings (apr_pstrcat). >> >> >> These operations are performed in all cases, even if DEBUG messages are not >> logged. >> >> >> I don't think this should have a real effect on performance. (If I remember >> well when I looked at it, there is no ap_log_error calls in sensitive code) >> >> Just to be sure, you could try to simplify ap_proxy_worker_name in Daniel's >> build to remove the apr_pstrcat and check performance with his build. >> >> >> Should you and Daniel have different logging levels, it could explain why >> you don't measure the same discrepancy. >> >> >> >> Just my 2 cents. >> >> If I have time, I'll give another look tonight. >> >> >> CJ >> >> >> >> >>> Message du 02/12/13 13:46 >>> De : "Jim Jagielski" >>> A : dev@httpd.apache.org >>> Copie à : >>> Objet : Re: UDS Patch >>> >>> OK, I can't by inspection or by test see any performance >>> differences between the 2 implementations (in fact, >>> the older one, in some benchmarks, was slower due to >>> the string operations in the critical path)... >>> >>> Any ideas? >>> >>> On Nov 26, 2013, at 4:23 PM, Jim Jagielski wrote: >>> >>>> Thx... the key is httpd-2.4.6-uds-delta.patch and >>>> that shows nothing, that I can see, which would >>>> result in the "old" being faster than the "new"... >>>> especially in the critical section where we do >>>> the apr_sockaddr_info_get() stuff... >>>> >>>> On Nov 26, 2013, at 3:07 PM, Daniel Ruggeri wrote: >>>> >>>>> I reapplied the patches in order to 2.4.6 before r1531340 was added to >>>>> the proposal. Attached are the three diff's of use: >>>>> httpd-2.4.6-uds-original.patch - Everything in the backport proposal up >>>>> to (but not including) r1531340 sans the stuff that doesn't fit >>>>> httpd-2.4.6-uds-new.patch - The 2.4 patch proposed with r1511313 applied >>>>> first. Note that this doesn't include r1543174 >>>>> httpd-2.4.6-uds-delta.patch - The delta between the two modified trees >>>>> >>>>> -- >>>>> Daniel Ruggeri >>>>> >>>>> On 11/22/2013 5:27 PM, Daniel Ruggeri wrote: >>>>>> Sorry, I thought the diffs I sent off list were good enough. I'll have >>>>>> to see if I even still have the original build lying around. >>>>>> Effectively, I just took the list of patches in the backport proposal >>>>>> and applied them one at a time to the 2.4.6 sources. If I can't find the >>>>>> build, I'll do the same over and send that instead. >>>>>> >>>>>> -- >>>>>> Daniel Ruggeri >>>>> >>>
Re: UDS Patch
Thanks for getting back about that. Two days ago I retried and was able to tease out what appeared to be environmental variance in my numbers . After modifying the configuration to eliminate cruft as well as replacing the app with nothing more than a simple 'hello world' type of responder (over 32 running processes), I was able to get a much more reasonable set of numbers. The results, tested over a few hours were also quite stable: httpd-2.4.6 - w new patches Requests/sec: 35745.11 Requests/sec: 36763.18 Requests/sec: 36568.09 httpd2.4.6 - original UDS patch Requests/sec: 24413.15 Requests/sec: 24015.11 Requests/sec: 24346.76 The nginx server is in use by another application right now so I was unable to test it for an apples to apples comparison but this confirms exactly as you expected, the newer patch set is faster than the original UDS patch. I agree that decoding as well as the string comparison in the critical path is the most likely culprit there... but that's old hat anyway. So, in short, my past test cases were invalid because they included other bottlenecks. Sorry for unnecessary noise! -- Daniel Ruggeri On 12/5/2013 6:54 AM, Jim Jagielski wrote: > My test setup looks pretty much the same as yours: a simple > node.js server listening on the UDS path, but mine serves > just static content. > > On Dec 2, 2013, at 7:04 PM, Daniel Ruggeri wrote: > >> I had the same inclination as Cristophe but haven't been able to >> substantiate anything due to lack of time last week wasn't as kind >> to my free time as I had hoped. This would be very easy to tweak/test. >> Within the next day or two I should be able to get back in to perform >> some rebuilds and do more thorough testing and tampering as I squeeze >> time in between various work-related crises. Most of my testing is >> automated-ish, so turnaround from patch to test results is fairly quick. >> >> Jim, what does your test setup look like to measure performance delta? >> My setup is fairly simple with httpd on the frontend targeting a small >> Node.js backend application... I don't suspect the application is >> skewing the results because of how consistent the results have been, but >> I may just remove that from the equation to be absolutely sure. >> >> -- >> Daniel Ruggeri >> >> On 12/2/2013 8:14 AM, Jim Jagielski wrote: >>> But from what I see, all of those are during non critical paths. >>> It's like when workers are being defined, initialized, etc and >>> that's only done during config or when added via balancer-manager. >>> >>> On Dec 2, 2013, at 8:09 AM, Marion et Christophe JAILLET >>> wrote: >>> >>>> Hi, >>>> >>>> >>>> one of my thought was the change from >>>> >>>> worker->s->name >>>> >>>> to >>>> >>>> ap_proxy_worker_name(r->pool, worker) >>>> >>>> in logging function. >>>> >>>> ap_proxy_worker_name allocates memory in the pool and performs some >>>> operations on strings (apr_pstrcat). >>>> >>>> >>>> These operations are performed in all cases, even if DEBUG messages are >>>> not logged. >>>> >>>> >>>> I don't think this should have a real effect on performance. (If I >>>> remember well when I looked at it, there is no ap_log_error calls in >>>> sensitive code) >>>> >>>> Just to be sure, you could try to simplify ap_proxy_worker_name in >>>> Daniel's build to remove the apr_pstrcat and check performance with his >>>> build. >>>> >>>> >>>> Should you and Daniel have different logging levels, it could explain why >>>> you don't measure the same discrepancy. >>>> >>>> >>>> >>>> Just my 2 cents. >>>> >>>> If I have time, I'll give another look tonight. >>>> >>>> >>>> CJ >>>> >>>> >>>> >>>> >>>>> Message du 02/12/13 13:46 >>>>> De : "Jim Jagielski" >>>>> A : dev@httpd.apache.org >>>>> Copie à : >>>>> Objet : Re: UDS Patch >>>>> >>>>> OK, I can't by inspection or by test see any performance >>>>> differences between the 2 implementations (in fact, >>>>> the older one, in some benchmarks, was slower due to >>>>> the string operation
Re: Deprecating (and eventually removing) encrypted private key support in mod_ssl?
On 11/14/2013 5:54 AM, Joe Orton wrote: > a) people who want the ability to do filesystem backups without exposing > private keys to the set of admins who can read such backups; or e.g. > stick keys on NFS mounts, a similar requirement there. > > b) people who like or are required to follow "security by checklist" or > "security by regulator"; some auditor has "No Plaintext Keys !!!" on the > checklist, and so we must not use plaintext keys, no argument. > > I'm most sceptical of all about (b) as motivation for increasing > software complexity, but (a) I can at least appreciate. c) Like a, the system has lots of users on it that aren't necessarily trusted (application developers who look at logs) that shouldn't have access to the key. d) If any bug/exploit in an application running on the same box or in httpd is found allowing an arbitrary remote file to be read, an attacker could easily locate the key (since most httpd.conf files are in predictable locations) and read it remotely. An even worse case is a server admin accidentally exposing / due to ignorance and/or malice... but I can't really defend those guys much :-). Also worth noting, other SSL implementations protect their keys. Java, for example has a password on the keystore and an optional, additional password on the key object. I'd say that if the implementation supports it, httpd should try to accommodate it... there is no limit to what some people are willing to do in order to increase their apparent security posture. -- Daniel Ruggeri
Re: Deprecating (and eventually removing) encrypted private key support in mod_ssl?
On 12/5/2013 6:17 PM, Daniel Ruggeri wrote: > On 11/14/2013 5:54 AM, Joe Orton wrote: >> a) people who want the ability to do filesystem backups without exposing >> private keys to the set of admins who can read such backups; or e.g. >> stick keys on NFS mounts, a similar requirement there. >> >> b) people who like or are required to follow "security by checklist" or >> "security by regulator"; some auditor has "No Plaintext Keys !!!" on the >> checklist, and so we must not use plaintext keys, no argument. >> >> I'm most sceptical of all about (b) as motivation for increasing >> software complexity, but (a) I can at least appreciate. > c) Like a, the system has lots of users on it that aren't necessarily > trusted (application developers who look at logs) that shouldn't have > access to the key. > > d) If any bug/exploit in an application running on the same box or in > httpd is found allowing an arbitrary remote file to be read, an attacker > could easily locate the key (since most httpd.conf files are in > predictable locations) and read it remotely. An even worse case is a > server admin accidentally exposing / due to ignorance and/or malice... > but I can't really defend those guys much :-). > > Also worth noting, other SSL implementations protect their keys. Java, > for example has a password on the keystore and an optional, additional > password on the key object. I'd say that if the implementation supports > it, httpd should try to accommodate it... there is no limit to what some > people are willing to do in order to increase their apparent security > posture. > > -- > Daniel Ruggeri > I don't mean to say that a password protecting the file is "secure"... but it adds a layer, and (most) layers are good. -- Daniel Ruggeri
Re: Question about syntax
On 12/15/2013 11:37 PM, Christophe JAILLET wrote: > Apparently is equivalent to > So, should we: > 1) just leave it as it is. > 2) remove this behavior, which has never been documented. > 3) update doc. > > > My choice is 2) because > - it has never been documented, so it is likely that no one use it > - apparently it is equivalent to +1 -- Daniel Ruggeri
Re: Looking to T&R 2.4.8 in Feb...
On 1/6/2014 11:58 AM, Jim Jagielski wrote: > nuff said :) One more vote for the UDS patch would be appreciated if anyone could spare a moment to have a look. Happy New Year all, BTW. -- Daniel Ruggeri
Re: mod_ssl: querying any certificate in the chain
On 1/14/2014 12:16 PM, Graham Leggett wrote: > Would a syntax like this make some sense? > > SSL_CLIENT_S_DN_n - Give me the subject DN of the nth certificate in the > chain. > SSL_CLIENT_S_DN_x509_n - Give me the element of the subject DN of the nth > certificate in the chain. I like this. +1 I am assuming SSL_CLIENT_S_DN_n == SSL_CLIENT_S_DN. Would that be the case? ...or are you counting to n from the other direction? -- Daniel Ruggeri
Re: Looking to T&R 2.4.8 in Feb...
FYI, I also floated a few patches here that apply directly to 2.4.6 which includes the (many) proposed 2.4.7 patches. -- Daniel Ruggeri On 1/21/2014 8:27 AM, Yann Ylavic wrote: > Hi, > > please have a look at > https://issues.apache.org/bugzilla/show_bug.cgi?id=54101#c19 where a > patch is available against 2.4.7 (or 2.4.x). > > This is the same as the original > http://people.apache.org/~jim/patches/uds-2.4.patch (proposed but not > integrated into 2.4.7), but this one applies with no error against > current 2.4.7 or 2.4.x sources. > > Regards, > Yann.
Re: UDS support for mod_rewrite
On 1/22/2014 5:48 AM, Juan José Medina Godoy wrote: > Do you think that approach is safe or is it likely to break at some > point? (relaying on the workers being located by url in that way, > without having to provide the socket in the rewrite) Seems safe... and quite clever, actually. -- Daniel Ruggeri
Re: Welcome to Mike Rumph and Yann Ylavic!
On 2/17/2014 11:28 AM, Daniel Gruno wrote: > Welcome aboard, and well deserved!! > > With regards, > Daniel. welcome++ (from another Daniel) See you in Denver
Zombies from rotatelogs
I was taking a look at a server that had a handful of zombies and came to see they are caused by rotatelogs. It seems pretty straight forward why - I am calling gzip post-rotate to compress the log file and child cleanup only happens during the post_rotate function, but before apr_proc_create for this rotation. So, effectively, you will have a zombie process from time_child_dies to next_rotation. I would like to get rid of those zombie processes sooner but am wary to call apr_proc_wait_all_procs very often... I don't know what the expense of doing that would be. So... I'm seeking advice for the best option here: *Call apr_proc_wait_all_procs in the main loop (expensive?) *Spawn a thread on the side to reap the child a la blocking mode *Check an interval in the main loop to call apr_proc_wait_all_procs once per X time *or... your idea Thanks P.S. ApacheCon was a blast - can't wait until next one! -- Daniel Ruggeri
Re: Zombies from rotatelogs
On 4/14/2014 11:41 AM, Joe Orton wrote: > It's free... dunno why I didn't think of this before. > > http://svn.apache.org/viewvc?view=revision&revision=1587255 > > Regards, Joe Awesome - proposed for backport in 2.4. Thanks! -- Daniel Ruggeri
Re: svn commit: r1587650 - /httpd/httpd/branches/2.4.x/STATUS
On 4/15/2014 2:21 PM, Jim Jagielski wrote: > I can't recall... isn't the issue still being worked an > additional aspect of mod_rewrite and UDS; that is, a new > behavior to be added (or handled) rather than a "broken" > behavior. That was my understanding, too -- Daniel Ruggeri
Re: failonstatus only works on backend provided status codes
On 5/12/2014 7:31 AM, Ruediger Pluem wrote: > While trying to use the failonstatus option of a balancer for the first time > I noticed that it only works on status > codes provided by the backend not on status codes only set by the proxy (in > my case a 502 because the backend timed > out). Is this intentional? Hi, Ruediger; Yes, that was the original goal. The use case I was tackling was a case where a backend application server started accepting HTTP requests before the Servlets had all completed init(). In that case, the backend returns a 503. -- Daniel Ruggeri
Re: ApacheCon Austin, httpd track
On 11/30/2014 11:08 AM, Jeff Trawick wrote: > * deploying Python web apps under uWSGI behind mod_proxy_fcgi/scgi > (some material > here: http://emptyhammock.com/projects/info/pyweb/index.html) > * a debugging tricks talk I've given a few times (relatively minor > updates from the last North America AC) > * drastically updated (rewritten) version of an old > capacity-tuning-and-performance talk I gave at a Sun conference in > 2009 > (https://blogs.oracle.com/trawick/resource/DeepDive/WebStackDeepDiveApache.pdf) Similarly, I'm always up for giving my proxy talk if it's welcome (after the first day since I can't make it until Tues). If we think proxy is a big topic, we ought to arrange for a general overview like my proxy talk followed by more specific deep dives such as what Jeff mentions here and a session on new sexiness like WebSockets. Tuning for throughput is also an interesting topic and in line with the conversations lately (Re: commercial support). A side note on SSL/security: I had the idea a few years back that there is probably enough content to do a "here is 5 minutes about how to configure SSL in httpd" and then 50 minutes of other important security topics (What Ciphers should I enable? Should I use SSLv3 any more? How to treat my keys and what the hell is an HSM anyway? Passphrase encrypted keys or not? Should I trust my distro's build?). Thoughts are welcome on that topic... not sure if I'm overly paranoid or if these are things that people actually want to hear? -- Daniel Ruggeri
Re: 2.4.42 soon?
Sure - happy to oblige! -- Daniel Ruggeri On 2020-03-13 07:51, Eric Covener wrote: Looks like STATUS is in good shape. Any cycles Daniel or Jim? -- Eric Covener cove...@gmail.com
[NOTICE] Intenet to T&R 2.4.42 in the next 36 hours.
Hi, all; It looks like we're in a good place to do a release so I will aim to T&R 2.4.42 tomorrow. Please feel free to shout loudly if any issues are found. -- Daniel Ruggeri
[VOTE] Release httpd-2.4.42
Hi, all; Please find below the proposed release tarball and signatures: https://dist.apache.org/repos/dist/dev/httpd/ I would like to call a VOTE over the next few days to release this candidate tarball as 2.4.42: [ ] +1: It's not just good, it's good enough! [ ] +0: Let's have a talk. [ ] -1: There's trouble in paradise. Here's what's wrong. The computed digests of the tarball up for vote are: sha1: 2e4505796dfaebcceab6ba22e5fa221e07ad488e *httpd-2.4.42.tar.gz sha256: cbac6f8211744a74f798db792b74da9f6fb5a4fbee94234cf2b01aeb9ffe57ed *httpd-2.4.42.tar.gz sha512: 09d0f3bd9266907eea91ac9129a3c41658929b9fd88d627c1fccceaf952548d2c3ad62099b9bcd1ae4822402c1dbda90b8bfb9f64cd5eac9f84ed249faffb837 *httpd-2.4.42.tar.gz The SVN tag is '2.4.42' at r1875427. -- Daniel Ruggeri
Re: [VOTE] Release httpd-2.4.42
Hi, all; Per the issues surfaced/fixed, I'll go ahead and declare this release as dead-on-the vine. I'll target another T&R later this week, hopefully after the discussion around OpenSSL versioning plays out. How about Thursday? -- Daniel Ruggeri On 3/19/2020 9:45 AM, Daniel Ruggeri wrote: > Hi, all; > Please find below the proposed release tarball and signatures: > https://dist.apache.org/repos/dist/dev/httpd/ > > I would like to call a VOTE over the next few days to release this > candidate tarball as 2.4.42: > [ ] +1: It's not just good, it's good enough! > [ ] +0: Let's have a talk. > [ ] -1: There's trouble in paradise. Here's what's wrong. > > The computed digests of the tarball up for vote are: > sha1: 2e4505796dfaebcceab6ba22e5fa221e07ad488e *httpd-2.4.42.tar.gz > sha256: > cbac6f8211744a74f798db792b74da9f6fb5a4fbee94234cf2b01aeb9ffe57ed > *httpd-2.4.42.tar.gz > sha512: > 09d0f3bd9266907eea91ac9129a3c41658929b9fd88d627c1fccceaf952548d2c3ad62099b9bcd1ae4822402c1dbda90b8bfb9f64cd5eac9f84ed249faffb837 > *httpd-2.4.42.tar.gz > > The SVN tag is '2.4.42' at r1875427. >
[VOTE] Release httpd-2.4.43
Hi, all; Please find below the proposed release tarball and signatures: https://dist.apache.org/repos/dist/dev/httpd/ I would like to call a VOTE over the next few days to release this candidate tarball as 2.4.43: [ ] +1: It's not just good, it's good enough! [ ] +0: Let's have a talk. [ ] -1: There's trouble in paradise. Here's what's wrong. The computed digests of the tarball up for vote are: sha1: 15d8605b094dfe5e283cd9e90770368dd14e26f2 *httpd-2.4.43.tar.gz sha256: 2624e92d89b20483caeffe514a7c7ba93ab13b650295ae330f01c35d5b50d87f *httpd-2.4.43.tar.gz sha512: d9879b8f8ef7d94dee1024e9c25b56d963a3b072520878a88a044629ad577c109a5456791b39016bf4f6672c04bf4a0e5cfd32381211e9acdc81d4a50b359e5e *httpd-2.4.43.tar.gz The SVN tag is '2.4.43' at r1875715. -- Daniel Ruggeri
Re: [VOTE] Release httpd-2.4.43
On 3/27/2020 12:58 PM, William A Rowe Jr wrote: > On Fri, Mar 27, 2020 at 12:34 PM Steffen <mailto:i...@apachelounge.com>> wrote: > > +1 All fine on Windows. > > > Your's are still the .dsp based builds, right? I can confirm also on > the CMake flavor. Thanks, Bill. Shall this response be considered a +1 for the purposes of the release vote? -- Daniel Ruggeri
Re: [VOTE] Release httpd-2.4.43
On 3/26/2020 9:50 AM, Daniel Ruggeri wrote: > Hi, all; > Please find below the proposed release tarball and signatures: > https://dist.apache.org/repos/dist/dev/httpd/ > > I would like to call a VOTE over the next few days to release this > candidate tarball as 2.4.43: > [ ] +1: It's not just good, it's good enough! > [ ] +0: Let's have a talk. > [ ] -1: There's trouble in paradise. Here's what's wrong. > > The computed digests of the tarball up for vote are: > sha1: 15d8605b094dfe5e283cd9e90770368dd14e26f2 *httpd-2.4.43.tar.gz > sha256: 2624e92d89b20483caeffe514a7c7ba93ab13b650295ae330f01c35d5b50d87f > *httpd-2.4.43.tar.gz > sha512: > d9879b8f8ef7d94dee1024e9c25b56d963a3b072520878a88a044629ad577c109a5456791b39016bf4f6672c04bf4a0e5cfd32381211e9acdc81d4a50b359e5e > *httpd-2.4.43.tar.gz > > The SVN tag is '2.4.43' at r1875715. For my own vote: [X] +1: It's not just good, it's good enough! system: kernel: name: Linux release: 4.9.0-8-amd64 version: #1 SMP Debian 4.9.144-3.1 (2019-02-19) machine: x86_64 libraries: openssl: "1.1.1e" openldap: "2.4.49" apr: "1.7.0" apr-util: "1.6.1" iconv: "1.2.2" brotli: "1.0.7" nghttp2: "1.40.0" zlib: "1.2.11" pcre: "8.44" libxml2: "2.9.9" php: "7.4.4" lua: "5.3.5" curl: "7.69.1" -- Daniel Ruggeri
[RESULT - PASS][VOTE] Release httpd-2.4.43
On 3/26/2020 9:50 AM, Daniel Ruggeri wrote: > Hi, all; > Please find below the proposed release tarball and signatures: > https://dist.apache.org/repos/dist/dev/httpd/ > > I would like to call a VOTE over the next few days to release this > candidate tarball as 2.4.43: > [ ] +1: It's not just good, it's good enough! > [ ] +0: Let's have a talk. > [ ] -1: There's trouble in paradise. Here's what's wrong. > > The computed digests of the tarball up for vote are: > sha1: 15d8605b094dfe5e283cd9e90770368dd14e26f2 *httpd-2.4.43.tar.gz > sha256: 2624e92d89b20483caeffe514a7c7ba93ab13b650295ae330f01c35d5b50d87f > *httpd-2.4.43.tar.gz > sha512: > d9879b8f8ef7d94dee1024e9c25b56d963a3b072520878a88a044629ad577c109a5456791b39016bf4f6672c04bf4a0e5cfd32381211e9acdc81d4a50b359e5e > *httpd-2.4.43.tar.gz > > The SVN tag is '2.4.43' at r1875715. > Hi, all; I am pleased to report that the vote to release httpd-2.4.43 has PASSED with the following +1 votes recorded: PMC jorton, ylavic, icing, jfclere, jim, steffenal, jailletc36, covener, gsmith, druggeri, rjung Community gbechis ... and zero -1 votes. I will begin the process of promoting to the mirror system and will prep for announcement in the next day or two. -- Daniel Ruggeri
Odd vulnerabilities_24.html output
Hi, all; I'm not sure what mechanism is used to generate https://httpd.apache.org/security/vulnerabilities_24.html from https://svn.apache.org/repos/asf/httpd/site/trunk/content/security/vulnerabilities-httpd.xml, but an anomaly has been reported to me in response to the security announcements from last release. For both CVE-2020-1934 and CVE-2020-1927, the source file says "Apache HTTP Server versions 2.4.0 to 2.4.41" in the description, but the rendered result is "Apache HTTP Server versions 2.4.0 to 2.41". If anyone has pointers on how the site build happens, I can look into it further. If it's too complicated a fix, I'm OK with removing that line from the description. The CVE reports must include the version vulnerability info in the description, but it's not really a requirement for the site (I was just keeping them consistent). -- Daniel Ruggeri
Re: RFC: Documenting changes in the CHANGES file
On 6/1/2020 6:23 AM, Yann Ylavic wrote: > On Fri, May 29, 2020 at 9:30 PM Ruediger Pluem wrote: >> Reviewing our backport process I noticed that in many cases a clean merge >> via svn merge fails due to conflicts in CHANGES. While >> these are easy to solve it puts IMHO unnecessary extra work on the backport >> process, both for reviewing and for actually doing the >> backport. How about if we change the way we document changes the following >> way: >> >> 1. We create a changes-fragments directory (name to be determined) at the >> top level. >> 2. For each release we create a subdirectory such that we end up with the >> following structure: >> >>changes-fragments/ >> 2.4.41/ >> 2.4.42/ >> 2.4.43/ >> 2.4.44/ >> >> 3. Each directory contains the changes for each release and each change >> entry is a single file. >> 4. We have a script that builds our current CHANGES file from the content in >> changes-fragments directories with the help of >>a template or at least some sort of header / footer that is static. >> 5. This script can be called either manually and we commit the resulting >> CHANGES file as we like just like the x-forms commits >>for documentation plus this script is called by the release scripts from >> Daniel as part of the preparation of rolling a >>release. > +1 from me, I don't volonteer for the scripts though :) > > Regards; > Yann. Hi, Yann; I'm open to whatever... and don't mind writing or tweaking scripts once we decide on an approach :-) While we are discussing ideas in this neighborhood, one thing to keep in mind is that during release of security fixes, sometimes there are items added to CHANGES and sometimes CHANGES is modified to add CVE information. There have been minor bumps in the road where these patches don't always apply cleanly. So, if possible, it would be great to consider. There may be nothing to do, though, since that happens way after backport. -- Daniel Ruggeri
NOTICE: Intent to T&R late this week
Hi, all; It's been a while since we've rolled a release and gotten fixes/etc in our community's hands. Apologies for not suggesting this sooner. How about a T&R Friday? That will let vote run through the weekend. -- Daniel Ruggeri
