Re: svn commit: r209837 - /httpd/apreq/trunk/include/apreq_version.h
[EMAIL PROTECTED] wrote: Modified: httpd/apreq/trunk/include/apreq_version.h URL: http://svn.apache.org/viewcvs/httpd/apreq/trunk/include/apreq_version.h?rev=209837r1=209836r2=209837view=diff == --- httpd/apreq/trunk/include/apreq_version.h (original) +++ httpd/apreq/trunk/include/apreq_version.h Fri Jul 8 09:33:11 2005 @@ -61,7 +61,7 @@ #define APREQ_MINOR_VERSION 1 /** patch level */ -#define APREQ_PATCH_VERSION 1 +#define APREQ_PATCH_VERSION 2 speaking of which shouldn't the patchlevel be at 4 at this point ? I'm probably not grasping how apreq is using this though. -- END What doesn't kill us can only make us stronger. Nothing is impossible. Philip M. Gollucci ([EMAIL PROTECTED]) 301.254.5198 Consultant / http://p6m7g8.net/Resume/resume.shtml Senior Developer / Liquidity Services, Inc. http://www.liquidityservicesinc.com
Re: [VOTE] libapreq2-2.06-dev-rc4
Philip M. Gollucci [EMAIL PROTECTED] writes: Joe Schaefer wrote: Still, candidates require majority votes from committers (actually pmc members) to be properly released by the ASF. Speaking of which is the PMC for apreq even listed anywhere ? We're a subproject of httpd. Haven't you read the README file yet (just teasing)? All the httpd folks are already committers on this project, and the httpd pmc is formally responsible for overseeing our work on apreq. The oversight mechanism is how the ASF extends its umbrella of protection, should it ever start to rain in these parts. -- Joe Schaefer
Re: [VOTE] libapreq2-2.06-dev-rc4
Randy Kobes [EMAIL PROTECTED] writes: On Tue, 19 Jul 2005, Joe Schaefer wrote: Please test vote on rc4 for tomorrow's release of: http://people.apache.org/~joes/libapreq2-2.06-dev-rc4.tar.gz The pgp signature is also available: http://people.apache.org/~joes/libapreq2-2.06-dev-rc4.tar.gz.asc I don't think the commit I made today for library/t/at.h, changing the printing of AT_skip() from not ok to ok, made it into the above. Or perhaps the above link is to an earlier tarball? Nope- I'm an idiot. Somehow I forgot to svn up after reading your commit message. Now rolling rc5... -- Joe Schaefer
DNSBL filtering for Apache
While I was thinking about Nick's suggestion for mod_rbl (blacklist lookups with mod_smtpd) I happened upon this idea, which is somewhat unrelated to the smtp project. DNSBLs, the dominant form of real time blacklisting, are not specific to SMTP because this is just a way to publish lists of IP addresses. RHSBLs, which look up the address in an SMTP envelope, are specific to SMTP however. Apache -- the HTTP side too -- would benefit from DNSBL support. Or does this already do this? For example, both the CBL and AHBL projects list IP addresses of hosts engaging in activities such as proxy hijacking and spam relaying. This means it would be useful for webmasters to be able to make use of the published DNSBL to deny access to http requests. Because DNSBLs are an efficient way to publish lists, webmasters might start using a DNSBL lookup feature in Apache to limit abuse of say message forums, cgi scripts, proxy gateways. Currently, this has to be done by importing a complete list of IP address (often tens of megabytes) into a firewall script or Apache configuration.
Re: DNSBL filtering for Apache
Apache -- the HTTP side too -- would benefit from DNSBL support. Or does this already do this? For example, both the CBL and AHBL projects list IP addresses of hosts engaging in activities such as proxy hijacking and spam relaying. This means it would be useful for webmasters to be able to make use of the published DNSBL to deny access to http requests. Gosh, it already exists thanks to Blars Blarson http://www.blars.org/mod_access_rbl.html I wonder if the existing module can somehow be used for mod_smtpd as well? I'm still not familiar with enough 2.x style modules to know if that would work somehow.
Re: Dispatching MPM
Did you check Metux MPM? It works by passing socket discriptor to worker process. The project is on beta status but works fine. Please see the following pages. http://www.metux.de/mpm/ http://www.sannes.org/metuxmpm/ If you are interested in extending this MPM, join in the project. I'm sure some of the maintainers are still watching the move. Hideki Noma Hi, I'm going to make a new MPM (I called it Dispatching MPM, or D-MPM) that would do what perchild is supposed to do. Here is a rough proposal of the architecture. My general idea is shown on the following diagram of example state of D-MPM: +---+ +---++---+ +---+ |worker1| |worker2||worker3| |workerN| |---| |---||---| |-- | |user1 | |user2 ||user1 | ... |userX | |group1 | |group2 ||group2 | |groupY | |/home/a| |/home/b||/home/c| |/home/Z| +---+ +---++---+ +---+ /\ /\ /\ /\ /\ || || || || ... || || \/ \/ \/ \/ || ++ ++ || | dispatcher | | master | || || || || | nobody | |root| || | nogroup | === |root| || | /var/empty | |/ | || ++ ++ || /\ || || || || \/ \/ +---+ +---+ |client1| |client2| +---+ +---+ Arrows indicate communication, eg. using sockets. First line of each item contains its name, next lines contain user, group, and root directory it runs with, respectively. System consists of 3 parts: Workers--they are basically the same as in other MPMs, the only difference is that each has some set of permissions (uid, gid, root directory, etc.) of particular group of virtual hosts. Dispatcher, accepting incoming connections, reading requests and redirecting them to one of appropriate workers (i.e. those having set of permissions needed to handle that virtual host). Apache master process, listening to requests from dispatcher, and creating new workers with specified permissions, according to the requests. In presented example, client1 has specified Host: header and has been redirected to worker1, while client2 hasn't specified it yet, so it is still connected to the dispatcher. The diagram is created with assumption that socket passing will be used, not proxying. Rationale for the Architecture Most existing solutions (that separate different virtual hosts) have the architecture where many threads/processes (each running with privilege of some virtual host) accept connections, and redirect them if client wants another virtual host. This may create potential security problem--if there were an exploitable bug in code before redirection, an attacker would have a possibility to gain privilege of user of any virtual host. In D-MPM, an attacker would gain only dispatcher's permissions, which can be very restricted. In existing MPMs number of threads for each virtual host is static, set in configuration file. D-MPM approach fixes that problem--new workers are dynamically created by master process on dispatcher's demand. Of course there might be a security hole in master's code. But communication protocol between dispatcher and creator is going to be rather simple: dispatcher may only request that new worker with specified permissions should be created, creator returns socket descriptor connected to a newly created child or error code. Extensibility Nowadays permissions of process depend not only on process user and group ID, but on many other settings that vary from system to system. The most popular is process' root directory (changable by chroot(2)), but there are many others (eg. POSIX capabilities, jails, other security settings in RSBAC and SELinux). To allow to utilize any of security extensions particular system has, D-MPM will have a loadable module support. Making D-MPM change some non-standard permissions would involve only writing a small shared library, without need for changing D-MPM's source or even recompiling it. Someone creating a module needs to provide only two functions: first, that reads part of virtual host's configuration, and second, that changes process permissions according to that configuration. Implementation issues Dispatcher It would be best to make dispatcher single-threaded, I suppose that
Re: Initial mod_smtpd code.
Nifty! I had some compilation problems involving regex, so in the attached patch I use ap_regex.h and change some defines. Hope this doesn't break anything. that was a good idea, ap_regex.h was implictly getting included for me. The other bug I partially fixed was, strstr in smtp_protocol.c only does exact matches so uppercase commands like MAIL FROM would fail. I added support for the upper case, but this needs to be improved still because mixed case doesn't work. Is there an APR function like stristr? agh i forgot about an upper case FROM: or TO:. process_smtp_connection_internal() already lowercases the actual command, but not the rest of the params. Either MAIL from: or RCPT to: would work, but not FROM or TO. Anyway, stristr can be easily hacked by calling strstr with two lowercased strings. The overall structure and the approach you took is very nice, easy to understand. I would recommend adding a hook immediately upon the client connection, because an external module (maybe for DNSBLs, or some rate limiting control) might not even want us to return a greeting at all -- i.e. close with 554 Service unavailable right away. Hmm. That sounds like a good idea, maybe there already is a hook defined that could deal with this, I'll look into it. But I like what you have, would be happy to keep working around this design. Thanks! Do you now have svn access? After I apply your patches I'll see about checking this in so more of us can deal with this code. It sucks that most are busy with ApacheCon right now. -rian
Apache in a loop during startup
On Freebsd 4.10, Apache/2.0.52 - during starttup when there is a zero byte __db.ssl_cache sitting in /var/run: drwxr-xr-x 4 root wheel 1536 Jul 19 07:47 . drwxr-xr-x 20 root wheel 512 Sep 17 2004 .. -rw-r--r-- 1 root wheel0 Jul 19 07:41 __db.ssl_scache -rw--- 1 root wheel0 Dec 27 2004 accept.lock.51219 ... with config: SSLSessionCache dbm:/var/run/ssl_scache SSLSessionCacheTimeout 300 SSLMutexfile:/var/run/ssl_mutex then apache sits in an endless loop during startup doing: stat(/var/run/ssl_scache,0xbfbff570) ERR#2 'No such file or directory' open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists' open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists' open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists' stat(/var/run/ssl_scache,0xbfbff570) ERR#2 'No such file or directory' open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists' open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists' open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists' repeat... Before I dive into this - does this ring a bell with anyone ? Dw
Re: Hackathon Attendees
Yeah, no kidding. We haven't made any decisions yet. Decisions are what happens when people vote on releases or changes. Did one of those go by and I didn't notice? Nope. Yup - branching 2.1.x - 2.2.x - trunk. No, branching is part of the task of an RM. It is not a decision of the group, nor is it subject to vote, for the same reason we do not allow people to veto releases. As long as the RM is doing it for a good reason (i.e., preparing a tarball, not just dicking around), then we want them to use the version management tools the way that they are intended to be used. Nobody is forced to support the new branch. Nobody is forced to vote to release it. It doesn't exist for the public until we vote to release it, so just keep working on what you are doing and ignore the dude behind the curtains. At least until he asks us to make a decision. Roy
Re: NTLM HTTP Authentication is insecure by design - a new writeup by Amit Klein
On Jul 18, 2005, at 12:30 PM, William A. Rowe, Jr. wrote: NTLM HTTP Authentication (and possibly other connection-oriented HTTP authentication and authorization protocols) is insecure by design Yep, no shit -- that's what the Microsoft fools were told when they introduced it. *) The web server (IIS/6.0) must receive a Via-less request. The Microsoft implementation assumes that the Via header is always sent by a proxy server, and this is indeed mandated by the HTTP/1.1 RFC 2616 (http://www.ietf.org/rfc/rfc2616.txt), section 14.45: The Via general-header field MUST be used by gateways and proxies to indicate the intermediate protocols and recipients between the user agent and the server on requests [...] Yep. However, it seems that not all servers adhere to this standard. For example, Apache 2.0.54 mod_proxy does not generate a Via header by default (see the ProxyVia directive - http://httpd.apache.org/docs-2.0/mod/mod_proxy.html#proxyvia, yet the default httpd.conf file contains a commented-out ProxyVia On directive, so it's possible that many Apache proxy deployments do WTF? That's a bug. *) Proxy vendors do not to share TCP connections to the server among several clients. Yes, it improves performance, but it's also insecure and enables/aids 3 different attacks (the one described here, HTTP Request Smuggling and HTTP Response Splitting). Also, comply to the RFC and send the HTTP Via request header by default (Apache Group - please take note). As reverse proxy is never enabled without intent, the impact of Apache on this vector is very low (note that Amit deliberately introduced this into his reproduction case) - but anyone who has intentionally used Apache as a reverse proxy to protect sensitive IIS servers behind their DMZ using NTLM auth is vulnerable (as are users of various NTLM Apache auth modules sitting behind Apache reverse proxies.) We don't really need to get in between their gun and their foot. My thinking is that rather than 'revealing' the reverse proxy origin server, we should be dumping the auth headers if they are NTLM. I don't think that will work from the browser's perspective. I would just mark the connection as being in use by a single client and remove it when the client closes. Roy
Re: Apache in a loop during startup
On Tue, Jul 19, 2005 at 01:02:56AM -0700, Dirk-Willem van Gulik wrote:
with config:
SSLSessionCache dbm:/var/run/ssl_scache
SSLSessionCacheTimeout 300
SSLMutexfile:/var/run/ssl_mutex
Using s/shmcb/dbm/ and SSLMutex default is IMO the best default (on
modern platforms).
then apache sits in an endless loop during startup doing:
stat(/var/run/ssl_scache,0xbfbff570) ERR#2 'No such file or
directory'
open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists'
open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists'
open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists'
stat(/var/run/ssl_scache,0xbfbff570) ERR#2 'No such file or
directory'
open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists'
open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists'
open(/var/run/__db.ssl_scache,0xa02,0644) ERR#17 'File exists'
repeat...
Before I dive into this - does this ring a bell with anyone ?
No, what's the backtrace when it's doing this? I don't think mod_ssl
would loop, nor APR. What dbm library?
joe
Re: [vote] Revoke R-T-C [was: svn commit: r219520]
--On July 18, 2005 1:19:54 PM -0500 William A. Rowe, Jr. [EMAIL PROTECTED] wrote: Some of us, trawick, orton and myself come to mind, are still up for supporting our current users. As it is, backports aren't reviewed, or committed once they are (I even split STATUS just to call out approved-yet-not-backported patches :-). Well, why haven't you backported them then? Why are you looking for someone else to do the work? So, I call a vote that we drop R-T-C altogether. It's pretty clear to me that those interested in current / near-future / far-future users are almost three distinct groups. It will be up to those small groups to call out and vote on changes within our individual domains. -1. R-T-C is the crux of our stability pact with our users. -- justin
Re: Dispatching MPM
On Mon, 18 Jul 2005, Michal Kosek wrote: [ replying as I read - some points here which you address later ] It would be great to see feedback from the authors of the original perchild here - I'd guess you'll have insights into the difficulties Michal is likely to face. System consists of 3 parts: Workers--they are basically the same as in other MPMs, the only difference is that each has some set of permissions (uid, gid, root directory, etc.) of particular group of virtual hosts. Dispatcher, accepting incoming connections, reading requests and redirecting them to one of appropriate workers (i.e. those having set of permissions needed to handle that virtual host). Apache master process, listening to requests from dispatcher, and creating new workers with specified permissions, according to the requests. Nice start. From this point, you probably want to do the minimum possible work, and re-use existing code wherever possible. In presented example, client1 has specified Host: header and has been redirected to worker1, while client2 hasn't specified it yet, so it is still connected to the dispatcher. The diagram is created with assumption that socket passing will be used, not proxying. That is how 1.x and its successors work, isn't it? Rationale for the Architecture Most existing solutions (that separate different virtual hosts) have the architecture where many threads/processes (each running with privilege of some virtual host) accept connections, and redirect them if client wants another virtual host. This may create potential security problem--if there were an exploitable bug in code before redirection, an attacker would have a possibility to gain privilege of user of any virtual host. I'm not quite sure what you're saying. What existing solution do you have in mind that is open to a transfer-of-privileges attack? Surely not the standard MPMs and suexec? In D-MPM, an attacker would gain only dispatcher's permissions, which can be very restricted. Agreed. A minimal dispatcher sounds great. You should probably aim to have it require chrooting, as well as running as a user with no shell or login (perhaps drop those constraints under #ifdef DEBUG). In existing MPMs number of threads for each virtual host is static, set in configuration file. D-MPM approach fixes that problem--new workers are dynamically created by master process on dispatcher's demand. Are you planning to use reslist to manage your pool of children per-vhost? Extensibility Nowadays permissions of process depend not only on process user and group ID, but on many other settings that vary from system to system. The most popular is process' root directory (changable by chroot(2)), but there are many others (eg. POSIX capabilities, jails, other security settings in RSBAC and SELinux). To allow to utilize any of security extensions particular system has, D-MPM will have a loadable module support. Making D-MPM change some non-standard permissions would involve only writing a small shared library, without need for changing D-MPM's source or even recompiling it. Do you mean it will implement a new optional hook specifically for chroot-like operations? That sound very nice: maybe just make my insist on chroot suggestion the default if no module does something else. Someone creating a module needs to provide only two functions: first, that reads part of virtual host's configuration, and second, that changes process permissions according to that configuration. Right:-) Implementation issues Dispatcher It would be best to make dispatcher single-threaded, I suppose that creating separate thread for each connection would take much more resources. OTOH it may cause some problems. One I am currently aware of is mod_ssl, which AFAIK doesn't currently have non-blocking API. Would it be difficult to create such API? That would certainly be fair at this point. And for a first-pass implementation you might also want to ignore mod_ssl. I want dispatcher to be as simple as possible, and make workers do all the work (filters, handlers, etc.). Of course, as dispatcher must read Host header, it must use connection filters. This headers is needed, because dispatcher has to decide to which worker a connection should be redirected. I think that it would be best if it didn't do full header parsing, but just look for virtual host name. There are two ways of dispatching connection: - passing a socket descriptor (along with data that were already read) to the appropriate worker, - acting as a proxy, redirecting data to the appropriate worker. The tricky bit seems to me to be the conn_rec. Of course a connection _should_ only ever be between a single client and a single vhost, but if you rely on that, bad people will notice ... Is there any way to avoid
Re: [vote] Revoke R-T-C [was: svn commit: r219520]
William A. Rowe, Jr. wrote: At 12:51 PM 7/18/2005, Roy T. Fielding wrote: Or you could simply keep working on trunk like everyone else and let releases be made when a tarball gets three +1s. Version numbers are cheap. Telling the entire group to stop while you work on the next big patch is extremely expensive. Ok, so we are now three levels deep? /trunk C-T-R /branches/2.2.x [misnomer, we don't have a beta yet] R-T-C ? If not now, when? /branches/2.0.x R-T-C There should be natural migration to 2.2.x. 2.0.x should be considered closed for new features, that's what the development line is for. Triple commits to fix one, one-line segfault? Well, this isn't workable. I think lack of progress shows it's not workable. The lack of progress is not due to having to commit to multiple branches. Some of us, trawick, orton and myself come to mind, are still up for supporting our current users. You make it sound like everybody else is dissing our current users. This broad characterization is not exactly productive. As it is, backports aren't reviewed, or committed once they are (I even split STATUS just to call out approved-yet-not-backported patches :-). The person who proposed the backport is expected to perform the backport when it has enough votes. The person casting the 3rd vote sometimes does the backport. And there are cases when a friendly RM clears the slate when it comes to approved proposed backports. Some were working on the stable release of 2.2.x, pquerna comes firstmost to mind. It isn't just Paul who wants to see 2.2.x finally materialize. We've been trying to get 2.2.x out for quite a while. We've come very close a couple of times, and because we like consensus we've not pushed too hard. I for one don't want to sit here again next year and still discuss what needs to be fixed/refactored/whatever before 2.2.x can be released. Let's just move on. 2.2.x is already a lot better than 2.0.x; our users deserve a release. Great progress is afoot, I see that release going beta very soon, the number of issues has dropped quite significantly. [Although we have 29 PatchAvailable issues, not sure how many correspond to 2.2 commits not backported.] And others are yet working on 2.4.x. 2.3.x, leading up to 2.4.x. As of the branch you are one of the people working on that, what's the issue with that? So, I call a vote that we drop R-T-C altogether. It's pretty clear to me that those interested in current / near-future / far-future users are almost three distinct groups. It will be up to those small groups to call out and vote on changes within our individual domains. Define current, near-future, far-future. current == 1.3? near-future == 2.0? far-future == 2.x? As it stands, only trunk should be C-T-R IMO. Stability on the _stable_ branches needs to be ensured. The question is; would we rather be saturated by commits we feel need review, or exhausted waiting for commits to be approved? The latter, but again, it's a broad characterization. This means code might be committed to 2.0.x and never fixed in head, it might mean code is fixed in head and never considered for backport to our current users, but all in all, it beats the situation today. No it doesn't. trunk needs to be the branch that has the latest code, and is most complete. I'm not suggesting that the 2.0.x users would entertain a break to ABI compatibility. But I'm suggesting that parallel development doesn't work when most folks are focused on tangential development. Sander
Re: Initial mod_smtpd code.
Hi Rian, Useful start: you seem to have dealt with the core SMTP stuff:-) I like how the code is done. I am not sure a hook for each smtp command is the good solution. Adding a new command here is very simple and quick. That's my feeling too. But I'm happy with the way Rian has done it - and Jem was IIRC in favour of the same approach. the problem i found when i did my poc is when there is in the command, different destination email. It's difficult here to keep the virtualHost scheme. It would be nice to keep a conf file like virtualHost ServerName mail.bla.com SmtpUserMap mail.bla.com-user.map or SmtpRelay host /virtualHost Yes. I think there is a logical difficulty here. smtpd_create_request is run before process_smtp_connection_internal, and the design doesn't allow for multiple recipients to be processed differently. My own feeling was that multiple recipients require a request each. Maybe we could get that in your design by creating a subrequest for each RCPT TO? We should also deal with spooling. Maybe a protocol-level input filter can do that. I still like the idea of a spool bucket (being a long-life file bucket with crash/recovery capabilities and reference counting), in which case the input filter would simply convert everything to that. That would then run before creating any subrequests. -- Nick Kew
Re: Initial mod_smtpd code.
On Jul 19, 2005, at 6:51 AM, Nick Kew wrote: the problem i found when i did my poc is when there is in the command, different destination email. It's difficult here to keep the virtualHost scheme. It would be nice to keep a conf file like virtualHost ServerName mail.bla.com SmtpUserMap mail.bla.com-user.map or SmtpRelay host /virtualHost Yes. I think there is a logical difficulty here. smtpd_create_request is run before process_smtp_connection_internal, and the design doesn't allow for multiple recipients to be processed differently. My own feeling was that multiple recipients require a request each. Maybe we could get that in your design by creating a subrequest for each RCPT TO? This is possible but I'm not sure what the advantage is. Would you mind setting up a hypothetical situation where this is necessary? I figure that ultimately the handler will be responsible for dealing with each rcpt to differently. Early on I wanted configuration possibility similar to: # Listen 21 VirtualHost *:21 # mod_smtpd conf SmtpProtocol On SetHandler unix-module # mod_smtpd_unix AcceptDomains thelaststop.net www.thelaststop.net Relay On # mod_smtpd_easyfilter Filter # matches against email in MAIL TO: smtp command RegexMailTo /thelaststop.net$/ SetHandler maildir-module # mod_smtpd_maildir conf MaildirBase /usr/local/virtual MailboxLimit 50M /Filter # Simple spam filter using mod_smtpd_easyfilter # Default handler does nothing with mail message Filter RegexHeader Subject /cialis/ SetHandler none /Filter /VirtualHost ## In the case of this httpd.conf embedded filtering mechanism, I figured the handlers could be changed based on the certain RegExs right before ap_run_handler() was called. Maybe ap_run_fixups() could be called, and my hypothetical mod_smtpd_easyfilter would have a fixups hook where it accomplished something similar to this situation. Although after thinking about it I realize now that mod_smtpd_easyfilter couldn't set different handlers for different rcpt tos. Is this what you meant? I think this requires some more thought considering different smtp connections and server requirements. The main drawback to sub- requesting each rcpt to is that we have two different handlers trying to read data from the socket. Is this problem solved by spooling the data, and letting the two separate requests read from the spool bucket? -rian
Re: [vote] Revoke R-T-C [was: svn commit: r219520]
-1
Re: Dispatching MPM
On Tue, Jul 19, 2005 at 11:14:51AM +0100, Nick Kew wrote: In presented example, client1 has specified Host: header and has been redirected to worker1, while client2 hasn't specified it yet, so it is still connected to the dispatcher. The diagram is created with assumption that socket passing will be used, not proxying. That is how 1.x and its successors work, isn't it? No, diagram shows how D-MPM should work (other MPMs don't have anything like dispatcher). But if we decide to make dispatcher act as proxy (and we probably must, if we want connection level filters work), clients won't ever be connected to worker directly, but through dispatcher. Most existing solutions (that separate different virtual hosts) have the architecture where many threads/processes (each running with privilege of some virtual host) accept connections, and redirect them if client wants another virtual host. This may create potential security problem--if there were an exploitable bug in code before redirection, an attacker would have a possibility to gain privilege of user of any virtual host. I'm not quite sure what you're saying. What existing solution do you have in mind that is open to a transfer-of-privileges attack? Surely not the standard MPMs and suexec? I compare design of D-MPM to design of perchild, Metux and peruser, ie. other MPMs that aim for separating privileges of different virtual hosts. With D-MPM, some code will be executed with dispatcher's permissions, and rest with permissions of appropriate worker. Only spawning new children must be done by master apache process (with root privileges), but it would be a small piece of code... In D-MPM, an attacker would gain only dispatcher's permissions, which can be very restricted. Agreed. A minimal dispatcher sounds great. You should probably aim to have it require chrooting, as well as running as a user with no shell or login (perhaps drop those constraints under #ifdef DEBUG). Of course, that's why I put /var/empty as dispatcher's root directory in the example. To allow to utilize any of security extensions particular system has, D-MPM will have a loadable module support. Making D-MPM change some non-standard permissions would involve only writing a small shared library, without need for changing D-MPM's source or even recompiling it. Do you mean it will implement a new optional hook specifically for chroot-like operations? Yes, exactly. That sound very nice: maybe just make my insist on chroot suggestion the default if no module does something else. [...] Workers won't change much, so code from existing MPMs should be somehow reused. Even running different workers for different virtual hosts should be possible. So I'm thinking of possibility of some generic code reuse, that would allow me to use workers from almost any MPM (prefork, worker, event...). Agreed. For the time being, maybe just take the preforked processes. That'll mean you get to satisfy the PHP crowd. Of course if you can harness Worker/Event/Other for no additional work, that would be the icing on the cake:-) Prefork looks like a good place to start, as its the simplest of existing stable MPMs... Logging. Currently every worker has descriptors of every log file. To provide security, after creating a worker and changing its permissions all the unnecessary log file descriptors will be closed. Since a worker process only ever serves one vhost in this architecture, it'll only ever need a single server_rec. For security, it shouldn't even see the server_recs for other vhosts. A single server_rec has a single logfile descriptor open. I actually want to have worker for each set of permissions (there may be more vhosts with the same permissions). It would have descriptors of log files of all vhosts it serves, but as all of them have the same owner, group, etc. it won't cause security problems. Michał
Re: Dispatching MPM
On Tue, Jul 19, 2005 at 04:05:06PM +0900, Hideki Noma wrote: Did you check Metux MPM? It works by passing socket discriptor to worker process. Yes, I looked at Metux MPM, but it also passes descriptors in the same way as perchild, from process handling one vhost to another, and I don't like it. I think that the architecture with low-privileged dispatcher is potentially more secure. Moreover, with this architecture it would be easier to handle eg. SSL. Michał
Re: Initial mod_smtpd code.
Rian Hunter [EMAIL PROTECTED] writes: I think this requires some more thought considering different smtp connections and server requirements. The main drawback to sub- requesting each rcpt to is that we have two different handlers trying to read data from the socket. Is this problem solved by spooling the data, and letting the two separate requests read from the spool bucket? Hmm, what would the smtp return status for DATA be, if only some of the RCPT_TO addresses are handled successfully? I've been assuming the http analog of RCPT_TO: [EMAIL PROTECTED] was POST: /foo\nHost: bar but I now think that's wrong from a resource identifier standpoint. -- Joe Schaefer
Re: AT_skip
On Tue, 19 Jul 2005, Joe Schaefer wrote: Joe Schaefer [EMAIL PROTECTED] writes: Randy Kobes [EMAIL PROTECTED] writes: Should AT_skip() print ok ..., rather than not ok ...? If I change this to do so, then the tests are reported as all pass; however, the message that these tests have been skipped aren't seen. +1 to dropping the not. Please apply that patch soonish, so I can roll rc4 and subject it to a vote. OK - thanks, that's been applied. -- best regards, randy
Re: Initial mod_smtpd code.
Joe Schaefer [EMAIL PROTECTED] writes: Rian Hunter [EMAIL PROTECTED] writes: I think this requires some more thought considering different smtp connections and server requirements. The main drawback to sub- requesting each rcpt to is that we have two different handlers trying to read data from the socket. Is this problem solved by spooling the data, and letting the two separate requests read from the spool bucket? Hmm, what would the smtp return status for DATA be, if only some of the RCPT_TO addresses are handled successfully? I've been assuming the http analog of RCPT_TO: [EMAIL PROTECTED] was POST: /foo\nHost: bar but I now think that's wrong from a resource identifier standpoint. OTOH, maybe we should just return success in this case, and only retry/bounce the failed subrequests later on. -- Joe Schaefer
Re: svn commit: r218978 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c
On Thu, Jul 14, 2005 at 07:43:35AM -0400, Jeff Trawick wrote: I'm so confused while trying to draw the line between alternate RFC-compliant philosophy fixes for actual RFC violations fixes for security issues I think CHANGES should be crystal clear on what change has a security implication. I am also confused and still trying to catch up and understand these changes... Bill, can you please describe *exactly* what security issues you see in the 2.0 proxy after the two already-committed patches (r219061). CAN-2005-2088 MUST NOT be used to refer to anything other than specific issue described in the WatchFire report. When you start bandying this CVE reference around with each new patch it defeats the purpose of having a CVE reference in the first place. If there are wider issues in the proxy then they will need new CVE names assigned. Regards, joe
Re: Initial mod_smtpd code.
Hmm. That sounds like a good idea, maybe there already is a hook defined that could deal with this, I'll look into it. I could also start work on a mod_smtpd_dnsbl if the mentors feel that is worthwhile? This would look up a connecting IP address against a blacklist and return a descriptive string to mod_smtpd if the client should be rejected with an error: 550 5.7.1 Email rejected because 127.0.0.2 is listed by sbl-xbl.spamhaus.org I'd also like to include support for RHSBL, a newer type of listing by domain names from the envelope sender address. That's used by a growing number of projects.
Re: Initial mod_smtpd code.
On Tue, 19 Jul 2005, Jem Berkes wrote: Hmm. That sounds like a good idea, maybe there already is a hook defined that could deal with this, I'll look into it. I could also start work on a mod_smtpd_dnsbl if the mentors feel that is worthwhile? This would look up a connecting IP address against a blacklist and return a descriptive string to mod_smtpd if the client should be rejected with an error: 550 5.7.1 Email rejected because 127.0.0.2 is listed by sbl-xbl.spamhaus.org I'd also like to include support for RHSBL, a newer type of listing by domain names from the envelope sender address. That's used by a growing number of projects. Happy to see you do either/both of those: it's all part of what we want to see. But is anyone dealing with outgoing SMTP via a proxy_smtp in the mod_proxy framework? I think you were discussing that a short while ago, weren't you? I think that might be higher priority. -- Nick Kew
Re: [VOTE] libapreq2-2.06-dev-rc4
Please test vote on rc4 for tomorrow's release of: http://people.apache.org/~joes/libapreq2-2.06-dev-rc4.tar.gz From userland mod_perl 2.01, perl 5.8.6, httpd 2.0.54, linux 2.6.12, all tests pass, +1.
Re: Apache in a loop during startup
On Tue, 19 Jul 2005, Dirk-Willem van Gulik wrote: Before I dive into this - does this ring a bell with anyone ? With help from Paul - seems to be a bug in BerklyDB where it scans through possible backup index files before opening the real file - and then gets confused; and gets its nickers in a twist. Dw
Re: svn commit: r218978 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c
At 10:07 AM 7/19/2005, Joe Orton wrote: On Thu, Jul 14, 2005 at 07:43:35AM -0400, Jeff Trawick wrote: I'm so confused while trying to draw the line between alternate RFC-compliant philosophy Roy spelled it out, it's not in the RFC but if there is -any- way we can use C-L, let's do it. The current 2.0.x code doesn't actually do that because it doesn't check to see if we can read all of the body within our hardcoded limit, before electing a C-L or T-E method. Everything else represents violations. fixes for actual RFC violations The currently applied backport patch 171205 by Jeff... still believes cl_zero, even with T-E. still drops the body altogether for empty bodies in the stream_chunked and spool_cl cases (an empty body is still a body!) still looks for a C-L value before looking for T-E ignored the fact that *any* T-E overrides C-L introduced proxy-sendunchangedcl, something that an administrator (as opposed to a developer) is unlikely to be certain of, and then never tested that the C-L values don't mismatch. provided no protection against proto_input_filters changing the body length In other words, send_request_body is altogether bogus. The problem is exacerbated by the fact that send_request_body is out-of-line with the rest of the header handling, making this harder code to review than a single inline function to evaluate request headers+bodies. fixes for security issues Each RFC violation between proxy and origin server (or next hop) becomes a cache poisioning/splitting/spoofing opportunity, I think anyone who's read Watchfire groks that by now. I think CHANGES should be crystal clear on what change has a security implication. I am also confused and still trying to catch up and understand these changes... Bill, can you please describe *exactly* what security issues you see in the 2.0 proxy after the two already-committed patches (r219061). E.g. the code remains bogus, as it once was, only with more flavors of mishandling the body. CAN-2005-2088 MUST NOT be used to refer to anything other than specific issue described in the WatchFire report. When you start bandying this CVE reference around with each new patch it defeats the purpose of having a CVE reference in the first place. If there are wider issues in the proxy then they will need new CVE names assigned. This patch addresses the fact that 2.0.x today HANGS when passed a C-L and T-E. The 'protocol.c' patch previously committed does mitigate the problem. But Watchfire explicitly called out the mishandling of C-L + T-E headers and the fact that the resulting request body in some cases would have no request headers whatsoever. The code in 2.0.x branch is ---still--- invalid, because the C-L and T-E selection is invalid. The symptoms are trivial; the proxy never processes the body correctly because the wrong request headers are ---still--- passed to the origin server. And that is exactly what Watchfire discussed. I suggested Jeff's approach is *GOOD*, backport the correct body handling code in its entirety. Unfortunately, at that point in time, request body handling was still broken. Since the intent was to backport, yet the elections and edge cases were not thought through, I'm vetoing and that specific backport. It leaves me with a question; is it better to revert [and then, re-credit Jeff in that CHANGES entry if/when we have an approved backport to reapply?] or better to patch over to the current proxy_http body handling code? I don't know which will be more legible, later. We enforce rules up front now with the protocol.c patch, masking how problematic that mod_proxy_http.c still is; always revert http://people.apache.org/~wrowe/httpd-2.0-cl-te-protocol.patch before you test the newest mod_proxy_http patches. Every other module which diddles the headers_in can re-trigger these edge cases, and anyone using the proxy_http.c as an example of anything will be led astray. Yet, even with this wondrous sipmnle patch, the existing proxy_http code in 2.0.x trunk backported from 2.1.x still has issues. In order to test, I'd applied; http://people.apache.org/~wrowe/httpd-2.0-trace.patch which provides trivial echo of the body and its headers, and then used http://people.apache.org/~wrowe/httpd.proxy.conf to set up a pretty wide range of tests from 1.3, 2.0, 2.1 against 1.3, 2.0 and 2.1 back-end servers. You don't have to do this, of course, but TraceEnable extended is simpler than sniffing sometimes. And you can see from using netcat to pipe; http://people.apache.org/~wrowe/chunked20.req through httpd-2.0 how badly this is all, still, mishandled, both according to Roy's points r.e. C-L election, and sans the protocol patch (which doesn't apply to module-mauled C-L and T-E headers). Some examples; ** Using 2.0.54 proxy HEAD /20/ HTTP/1.1 Host: localhost Content-Length: 75 Transfer-Encoding: chunked c Test This! 0 ** HANGS, because this is
Pondering strings in Apache 3.x
Greg and a few others voiced interest in moving from null-term strings to counted strings for a future version of Apache. This was too broad a scope change to make it into 2.0, of course, and was dropped on the floor for the time being. I'm wondering today; what metadata interests us in an ap_string_t prefix header? I have a hunch that a short, 65536, is enough to map most data we want to handle in one chunk; brigades are better for handling large sets of data. Of course we could push that to an int, or size_t, but there would be a small memory penalty. It might be overcome by cpu-specific optimized int or size_t handling behavior, since the assembly code wouldn't need to truncate short values. Perhaps, both bytes allocated/used, in order to play optimized games with string allocation. Perhaps, a refcount? (This doesn't play well with pool allocations, obviously.) But the byte count clearly isn't enough. I'm thinking of; encoding; is this data URI escaped or un-escaped? tainted; is it raw? or has it been untainted with context-specific validity checks? charset; is this native? (e.g. EBCDIC). utf-8? opaque or otherwise a specific set? What else interests us within an 'ap_string_t' header, that would help eliminate bugs within httpd? A random trailing short following the string, in a 'string debug' mode, to detect buffer overflows? Something similar to detect underflows? Open to all ideas. Bill
Re: mod-cache-requestor plan
Parin Shah wrote: you should be using a mix of # requests last access time cost of reproducing the request. Just to double check, we would insert entry into the 'refresh queue' only if the page is requested and the page is soon-to-be-expired. once it is in the queue we would use above parameters to calculate the priority. Is this correct? or let me know If I have mistaken it. yep. thats the idea. refresh the most-popular pages first. see memcache_gdsf_algorithm() in mod_mem_cache.c for an implementation of this, which assumes 'length' of request is related to the cost of reproducing the request. the priority queue implementation is sitting in mod_mem_cache, and could be used to implement the 'refresh' queue I would think. I feel comfortable with mod-cache and mod-mem-cache code now. but we also need to start new thread/process for mod-cache-requester when server starts. I am not too sure how we could implement it. any pointers to the similar piece of code would be really helpful to me. I don't have any code which does this to share with you (others might know of some). Thanks, Parin. --Ian
mod_mbox and generate_index
I trying to build the standalone helper program generate_index that the catchup-archive scriptneeds. This program does not seem to be made by the makefile and the build-dso errors out looking for lucene4c headers. Any ideas?
build-dso?
What (or who) generates the build-dso scripts for modules? What are they used for? If it is used why not use make? I am fighting with mod_mbox in particular but I'm also interested in a more generalized answer if someone has one.
Re: Initial mod_smtpd code.
Wouldn't something like VirtualHost www.apache.org:25 ServerName www.apache.org:81 ServerAlias httpd.apache.org internal_dev ServerAdmin [EMAIL PROTECTED] ErrorLog logs/error.log TransferLog logs/access.log SMTP On # like with mod_dav (dav on) /VirtualHost so reusing existing command, then this main an user of www.apache.org lets call him [EMAIL PROTECTED] that the following addresse point to the same. [EMAIL PROTECTED] [EMAIL PROTECTED] this seems to be more logical to me. btw: i'm not a member, just someone who follow this list closely Jorge
Re: Pondering strings in Apache 3.x
On Jul 19, 2005, at 12:55 PM, William A. Rowe, Jr. wrote: Greg and a few others voiced interest in moving from null-term strings to counted strings for a future version of Apache. This was too broad a scope change to make it into 2.0, of course, and was dropped on the floor for the time being. I'm wondering today; what metadata interests us in an ap_string_t prefix header? I have a hunch that a short, 65536, is enough to map most data we want to handle in one chunk; brigades are better for handling large sets of data. Of course we could push that to an int, or size_t, but there would be a small memory penalty. It might be overcome by cpu-specific optimized int or size_t handling behavior, since the assembly code wouldn't need to truncate short values. Perhaps, both bytes allocated/used, in order to play optimized games with string allocation. Perhaps, a refcount? (This doesn't play well with pool allocations, obviously.) But the byte count clearly isn't enough. I'm thinking of; encoding; is this data URI escaped or un-escaped? tainted; is it raw? or has it been untainted with context-specific validity checks? charset; is this native? (e.g. EBCDIC). utf-8? opaque or otherwise a specific set? What else interests us within an 'ap_string_t' header, that would help eliminate bugs within httpd? A random trailing short following the string, in a 'string debug' mode, to detect buffer overflows? Something similar to detect underflows? Open to all ideas. This may be a bit more radical than you were hoping for, but... I like the idea of using a reference-counted, non-null-terminated string type for 3.x. More generally, it would be great to have overflow detection on all arrays. And although I like the performance benefits of the pool memory allocators, I remember how tricky it was to debug some of the pool and bucket lifetime problems that we encountered during the development of 2.0 (especially in filters). All things considered, I don't think I'd mind the overhead of a garbage collection thread. Thus I can't help but wonder: Would 3.0 be a good time to consider trying a Java-based httpd? Brian
