The trouble with the authentication problem is that the credentials used for authentication are often used for way more than just finding out whether a user has access. That said, this is definitely a very useful addition.
That's exactly the point.
I think of this in terms of "user management", as I find the terms "authentication", "authorization" have too much baggage and often lead people to making the same mistake too many times.
Something like an auth module that can do "form based" auth, in addition to "basic" and "digest" etc would probably be very useful.
Well, on one level I think the API for authentication is basically inadequate. One major problem is that many applications (if they're going to be USABLE rather than to just exist) need "optional" authentication -- Amazon.com's personalization is an obvious case, but a more common "intranet" application is as follows: we have an event calendar which has some events that are open to the public and some of which are intended for staff. We've got an apache auth module that plugs into our campus kerberos system, and in a lot of ways it's pretty good. But a staff person who wants to see what events are coming up needs to log in and then has to click two more times (the fault of our designer) to actually see the private events.
Let's face it -- few users are going to bother to do that, so the event calendar is going to get a small fraction of the usage that it should.
If on the other hand, the system could remember the identity of the user, they could just go to the URL and see the events; that improves the chance that they'll actually look at the calendar on a regular basis.
----
The other trouble with the Apache authentication system is that the whole API between Apache and most programming environments is inflexible. You can do better in mod_perl, but with PHP, JSP and most other environments, you get a lunk of data from Apache at the beginning of the request and you send back a lunk of data, but you don't get to call subroutines on Apache and get results; for instance, you can't ask Apache what the name of the user is, but instead Apache has to pass everything that it's going to pass in notes or environment variables.
(One of the things I'd love, for instance, is lazy evaluation and caching for reverse DNS lookups, something that would mean big changes...)
----
On top of that, the real problem is user management isn't the actual authentication (I've got an "authentication module" that supports cookie-based authentication that I've written again and again in different languages that's about 100-200 lines of code), but rather in a domain that Apache has never concerned itself: user interface.
For instance, public web sites that let anybody join need to verify the e-mail address of people who join. There's basically two ways to do this:
(1) Send a randomly generated password to the user
(2) Let the user pick a password (that they've got a fighting chance to remember) and send them a token that they need to supply to activate their account
Empirical research shows that approach 2 has less than half the failure rate of 1, and helps in turning registrants into active users because their relationship of the site doesn't begin with a password change, or, more likely, a password reset.
Over the course of a few years, we made obserations and found we could cut 2's failure rate in half again by making little improvements in the details.
Similarly, if a site has passwords, there are going to be password resets, and you need a good interface for users to reset their own passwords.
Big sites will spend what it takes to get a good UI, but most web app developers still see user management as an afterthought, and will quickly pound out something that will cause 20-50% of users to drop out before they've participated in the system.
----
And then there's the question of user interface for the administrators: one of the great pains of Apache administration is getting phone calls from non-technical people who need changes made to an .htpasswd file. Make that a public site with 10^3 or 10^6 users, and the problem takes on a whole new dimension.
UNLESS people adopt a user-management framework that's already written, they're just not going to give this problem the resources it deserves, and they're going to suffer with half-assed solutions.
----
I've been thinking about this a lot, and my answer to this problem is here
http://www.honeylocust.com/x/products/tum/
I have to apologize that I haven't made a public release in a while; the version that's there is pretty far behind what I've got running on some production sites. The version that's up there is pretty battle tested -- it's run on about ten sites, one of which had 300,000+ users.
The biggest disadvantage of TUM is that it doesn't integrate with Apache authentication at all so it can't be used to protect static files. (I bet that a mod_tum could be written pretty easily, however.)
Nobody has really cared much for this project, and I think a lot of it is that most web developers (particularly PHP developers) are more interested in applications than frameworks. I'd think that something like this would be infinitely more valuable if it were built into a major web server.
----
The big objection that people will have about TUM is that it's a greenfield system that runs everything out of an RDBMS rather than the kind of modular system that will plug into somebody's LDAP.
My reasoning there is that I couldn't guarantee performance, reliability and security if I was plugging into some back end system. (Just like the Postfix authors won't guarantee it's secure if you plug it into Cyrus-SASL.) I could offer a higher level of service by picking one back-end user database and sticking to it.
That said, having worked with the system for a long time, and having often confronted the devil of pluggin into other people's systems, I see a fault line you could divide a system that like into and still get reasonable performance -- certainly it could log people in against a foreign user database, although providing a good UI for administrators could be tough.
(I think of how Mac OS X comes across as better quality than Fedora because it doesn't try to appease everybody... Rather than having two desktops that almost work, it has one that does.)
Also I think the Apache AAA API attacks the problem at the wrong level. I've got a cookie-based authentication system that can be implemented as
