Re: [RFC] Apache Privilege Separation for WebDAV (now against 2.2.6)
Michael Clark wrote: * A stackable performance statistics vfs module implementation that does timings on filesystem io calls. BTW I did some timings on linux for the wrapped apr_stat unix socket sendmsg/recvmsg ping/pong calls to the privsep process and a OS cached stat was about 22 microseconds vs 4-5 microseconds for a regular direct apr_stat call. At this level (45,000 calls per second), we would most probably be seek IO bound in real life anyway (in our use case the mod_ssl encryption load would by far dominate).
Re: [RFC] Apache Privilege Separation for WebDAV (now against 2.2.6)
Michael Clark wrote: Paul Querna wrote: How you stubbed out the file io seems fine for the lifetime of 2.2.x and maybe 2.4.x, but in the long run, I believe we need to support some kind of VFS layer, to make all IO pluggable. (open file, directory listing, etc etc). Directory listings in particular are interesting because; * mod_proxy_ftp LIST/NLST contents would be a special case that could be handled through the usual mod_dir/mod_autoindex modules. * mod_ftp would prepare it's directory listing through this VFS layer. * virtual (alias, mount) resources could appear in the virtualized index listing.
Re: [RFC] Apache Privilege Separation for WebDAV (now against 2.2.6)
Michael Clark wrote: Hi Folks, I posted a note about my privilege separation patches the other day and received some good private help/feedback, and have now made the patches a considerable amount more portable and they are using apr much more extensively. The patch is now fully modular and allows mod_privsep to be compiled out (although it does add some additional hooks to the core). First off, I want to say, this is a very cool set of patches, and I would like to see it or some derivative go into trunk! Due to the nature of the patch it has to change some core code to use privileged wrapper calls for file io. I can't see any way around this - I have tried to minimise the impact by adding hooks. How you stubbed out the file io seems fine for the lifetime of 2.2.x and maybe 2.4.x, but in the long run, I believe we need to support some kind of VFS layer, to make all IO pluggable. (open file, directory listing, etc etc). Preferably Async IO and pluggable too :-) Dream mode on: Location /upload Dav On Mount privsep:/opt/upload /Location # static content Location /mysite Mount /www/content /Location # proxied content Location /foo Mount balancer://bigcluster /Location Anyways, if all IO was abstracted with a little VFS layer, it would mean all modules would now work with your privilege separation code, rather than just the core, mod_autoindex, and any other modules you write patches for :-)
Re: [RFC] Apache Privilege Separation for WebDAV (now against 2.2.6)
Paul Querna wrote: Michael Clark wrote: Hi Folks, I posted a note about my privilege separation patches the other day and received some good private help/feedback, and have now made the patches a considerable amount more portable and they are using apr much more extensively. The patch is now fully modular and allows mod_privsep to be compiled out (although it does add some additional hooks to the core). First off, I want to say, this is a very cool set of patches, and I would like to see it or some derivative go into trunk! Thanks. BTW - I have filled out and sent in a CLA about a week ago. I'll send an email later about issues that I think should be addressed before a possibility of this or some derivative going in (perhaps just the required vfs hooks). There are also alternative approaches I'd like to mention - such as an mpm with non-privileged accept threads that forward the accepted socket to a dynamic pool of processes running as the desired uid - they could be created on demand (out of band by a privileged parent process) for authenticated users with a minimum pool size of 0 and timed out and killed after so many seconds of inactivity. This would also get rid of BIG_SECURITY_HOLE as there would be no race with the accept thread running as root (as with the other approaches I've seen). The advantage of this approach over the privsep patches' approach would be higher performance as there would be no ping/pong for each stat/open call although I don't think it would be quite as anally secure as my approach (That said, it could still check a signed token in a separated process before forwarding the accepted socket to reduce the potential of a buffer overflow exploit in one of the unprivileged processes allowing access as another uid). To do it this way we would need the ability to forward a request mid-processing (need to see host headers, translated path and authenticate before forwarding it) - so the process that gets forwarded the accepted socket would need to receive the headers out of band (perhaps in the message sending the accepted socket). This approach would be more intrusive to the core than my current approach I think (although it wouldn't need the modules to use hooked io functions). There would still be the issue of needing the stat calls in ap_directory_walk needing to be done with higher privs before the map_to_storage and access hooks have completed (so this approach may still benefit from the privilege separated process - it is also needed for system authentication in the PAM case if using /etc/shadow). Although I think the privsep patch approach is valid given a pluggable vfs io layer which has the request_rec as context - it could then be done unobtrusively. Perhaps I should add some notes to the PrivilegeSeparation wiki page on wiki.apache.org? Due to the nature of the patch it has to change some core code to use privileged wrapper calls for file io. I can't see any way around this - I have tried to minimise the impact by adding hooks. How you stubbed out the file io seems fine for the lifetime of 2.2.x and maybe 2.4.x, but in the long run, I believe we need to support some kind of VFS layer, to make all IO pluggable. (open file, directory listing, etc etc). Yes. It would be nice to get some core changes that would support this patch as an external module until it is ready (and make the filesystem IO layer generally pluggable). Either way, I think having a pluggable IO layer would be a good idea. The core patch could be made more palatable by making the apr IO hook API not privsep specific. So we could change the core patch as follows: * Rename server/privsep.c - server/vfs.c * Rename include/privsep.h - include/vfs.h * Rename layered APR io calls from ap_privsep_foo to ap_vfs_foo * Change context argument on io hooks calls from privsep_token_t* to request_rec* Shall I run up a patch set like this? This could perhaps be a starting point for a vfs (I believe request_rec* would be the correct context for these vfs indirected calls?). There is only one place I need more context information than the request_rec structure - this is for the apr_stat calls in ap_directory_walk before the map_to_storage and authentication hooks have completed. I can work around this somehow (by adding the preauth stat token to the request before doing these stat calls - or perhaps hook ap_directory_walk in this vfs). Later the hook dispatcher in vfs.c (server/privsep.c) could be modified to use the context/config from request_rec to work out where to dispatch the io calls rather than the simple dispatch all calls here approach I have currently. We could then have an ap_vfs_mount API to hook which paths we want to go where. Preferably Async IO and pluggable too :-) The current approach doesn't preclude async IO as the returned file descriptors inside of the apr_file_t could have async io operations performed on them.
Re: [RFC] Apache Privilege Separation for WebDAV (now against 2.2.6)
FYI Michael Clark wrote: http://oss.metaparadigm.com/apache-privsep/2.3.0-dev/mod_privsep-2.3.0-0.2.patch The module proper, implements the privsep privileged operation process that listens on a unix socket for privileged operations and performs them on behalf of the unprivileged apache worker processes. I have since made a fix for a file descriptor leak in the ap_privsep_file_open implementation due to my assumption that apr_os_file_put did the cleanup for me. http://oss.metaparadigm.com/apache-privsep/2.3.0-dev/mod_privsep-2.3.0-0.2.1.patch There is also a newer version of the patch which depends on Iain Wade's recent apr patch proposals (sent to apr-dev list) to add a working version of apr_os_dir_put (called apr_os_dir_put_ex) and a version of apr_os_file_put (named apr_os_file_put_ex) that can register a cleanup. http://oss.metaparadigm.com/apache-privsep/2.3.0-dev/mod_privsep-2.3.0-0.2.2.patch With Iain's patch I am now able to remove any dependencies on internal apr structures and arch headers needed to work around limitations (and a little brokenness *) in apr_os_dir_put and apr_os_file_put * Note: it was not previously possible to get a properly working apr_dir_t using apr_os_dir_put interface, apr_os_dir_put_ex fixes this problem. Michael.
Re: [RFC] Apache Privilege Separation for WebDAV (now against 2.2.6)
Michael Clark wrote: Hi Folks, I posted a note about my privilege separation patches the other day and received some good private help/feedback, and have now made the patches a considerable amount more portable and they are using apr much more extensively. Cool stuff! I'll start reviewing this, and look at mod_privsep to solve the very same issue within mod_ftp. But I won't have cycles till the holidays, so it will be some time. And due to the security implications, I'm sure you are just as anxious to have some serious review of the proposed module before it's widely deployed, so I'd suggest we either hold off a bit on committing it until it's received sufficient eyeballs, or create a sandbox for its review. If you don't have a CLA on file, please file one so your contribution can be considered? http://www.apache.org/licenses/#clas Although your name's on file, there's an email mismatch, and I suspect with 1500 folks, that is possibly a name clash (middle initial might be helpful for the record). Bill
Re: [RFC] Apache Privilege Separation for WebDAV (now against 2.2.6)
William A. Rowe, Jr. wrote: Michael Clark wrote: Hi Folks, I posted a note about my privilege separation patches the other day and received some good private help/feedback, and have now made the patches a considerable amount more portable and they are using apr much more extensively. Cool stuff! I'll start reviewing this, and look at mod_privsep to solve the very same issue within mod_ftp. But I won't have cycles till the holidays, so it will be some time. And due to the security implications, I'm sure you are just as anxious to have some serious review of the proposed module before it's widely deployed, so I'd suggest we either hold off a bit on committing it until it's received sufficient eyeballs, or create a sandbox for its review. Yes. I don't expect it will be ready for prime time for some time. I need to address some scalability issues (although the privsep process is single threaded this is mitigated by the fact that read/write IO is still all done in the workers - it is just the pam auth / stat / open / ... load) - I would however like to look at making it more scalable. We will have a production site doing user testing on this over the holidays and expect that it will have had some production load some time in mid to late January (a few hundred users). Yes, I would really appreciate the review - although there is no hurry. If you don't have a CLA on file, please file one so your contribution can be considered? http://www.apache.org/licenses/#clas Although your name's on file, there's an email mismatch, and I suspect with 1500 folks, that is possibly a name clash (middle initial might be helpful for the record). It is another Michael Clark as I haven't filed one - very common name. I'll fax one through in due course. Thanks much, Michael.
