On Sun, May 30, 2010 at 2:07 AM, Gustavo Narea <m...@gustavonarea.net> wrote: > Hello, > > On May 28, 6:13 pm, Russell Keith-Magee <russ...@keith-magee.com> > wrote: >> This is all very helpful information; thanks for breaking it down like this. >> >> I've talked this over with a few people at the sprints, and we've >> pretty much ended up at the same point -- we're deeply confused. >> Comments inline below. > > Thank you very much for taking time to talk about it. :) > > Sorry, I think I didn't explain some components well and that led to > the confusion. I'll try to do it better this time... > >> I also want to be very clear -- this feedback is *not* us saying "go >> away", it's us saying "we're not clear on what you're proposing and >> why you're proposing it". Please take this criticism in the sprit it >> is intended -- to work out exactly what it is that we *aren't* doing, >> and more importantly, *why* we should be doing it another way. > > Sure, I understand. > >> So - what you seem to be proposing here is that we add Paste as a >> dependency of Django in order to support: >> >> 1) A format of configuration that is different to Django's >> 2) Support for WSGI middlewares in the development server > > That's right, although the proposal is not mainly to add Paste as a > dependency, but support WSGI middleware in a transparent way; for > example, the way Paste Deploy does.
The issue I'm unclear on is what you mean by "transparent". Django provides an interface that implements the WSGI spec. There may well be errors in that implementation or violations of the spec, and if there are, I'm happy to address them. But ultimately, we either implement the spec or we don't. If Django's implementation of the WSGI spec is compliant, but you can't use it with out WSGI components, then to my reading either the other components aren't implementing the spec, or the WSGI spec is missing something big. > The reason why I proposed Paste is because it's widely used and it'd > prevent us from implementing something that is already implemented. Your argument also presupposes that Paste is an inherently better implementation. Well, I don't wan't to brag, but Django also has an implemented WSGI implementation, and it's just as battle hardened as Paste. :-) I won't argue that duplication of code and effort is a virtue. If there is a benefit to be gained in leveraging someone else's work, and we can do so without compromising our own project, then we should. However, this raises the much larger issue of how to manage Django's relationship with external libraries in a way that maintains our 'beginner friendly' history. The 'minimal dependency' philosophy is, in my opinion, one of the reasons that Django has been able to get the traction that it has gained. There have been some initial discussions about this general problem -- specifically, in relation to introducing support for unittest2. However, it's a discussion that is much larger than improving WSGI support, and it's a discussion that we need to have as a community. >> (2) isn't a matter of opinion; however, it's not a use case that I've >> seen a particularly compelling use case for, either. I'm certainly >> willing to be convinced otherwise, though -- now is your opportunity >> to convince us. > > Basically, when you need to integrate a piece of WSGI middleware that > must be present both on development and deployment, you have to get > rid of `manage runserver' and use a development server like the one > from Paste or CherryPy. Sure - if you have a complex deployment (by which I mean "anything other than a single Django project and some static files"), you need to have a complex deployment scheme. I don't see anything unusual in that. Django ships a development server as a "getting started fast" hack; it's not intended to be anything remotely approaching a useful web server, and we're going to resist any attempts to turn it into one. Django isn't a web server, and the development server isn't intended for production use. > There are lots of WSGI middleware out there that you may want/need to > use, and there are even projects like Paste or Repoze whose only goal > is to develop framework-independent WSGI libraries (many if not most > of them are middleware). In the following links you can find some of > the WSGI middleware available and find that most of them are > applicable to both development and deployment: > http://pythonpaste.org/modindex.html (some items aren't middleware, > but WSGI applications) > http://repoze.org/repoze_components.html#middleware > http://wsgi.org/wsgi/Middleware_and_Utilities > > Some of them are alternatives to components offered by Django itself, > which some people might prefer to use. Others are just things that > cannot/shouldn't be done at the framework level. > > I'll give you one real-world example: > > We don't put all the static files under MEDIA_ROOT for the limitations > described on <http://groups.google.com/group/django-developers/ > browse_thread/thread/b333c14f40acd22a>. We have four kind of static > files and we want them to be in different directories: > > - Front-end files, like JS or CSS. > - Files generated by our application, like thumbnails. > - Uploaded files. > - Uploaded files whose downloads are restricted. This is a great example of a use case where you shouldn't be using Django's development server at all. You have a bunch of serving rules for static content. A proper web server can handle those. But as far as developing and testing your Django application is concerned, it shouldn't matter how the static files are served - a static file resource is either available or it isn't. When you move to production, you're going to have to test the integration of the various components inside your production web server; I'm not sure I see hiding the need for this integration testing as a virtue. >> Regardless, as you've demonstrated with twod.wsgi -- support for a >> "rich" development server is also something that can be provided as an >> external resource. Given that the overhead of this change is adding a >> dependency to a project that has made a history of not having >> dependencies, I'm more inclined to modify the docs to say "If you >> want/need to use WSGI middleware as part of your development >> environment, you'll need to test your deployments under a full WSGI >> container -- either a full web server, or a lightweight test server >> such as the one provided by twod.wsgi". > > Django's development server *is* a WSGI compliant gateway, the problem > is that you're limiting it to do simple things only. Neither mod_wsgi > or Paste development servers are more WSGI compliant, AFAIK. I could > run a TurboGears application with your development server if I wanted, > for example. Yes - but the limitation is (at least partially) by design. We don't want anyone to be under the impression that Django's development server is anywhere close to being suitable for production. The more we make the development server behave like a production server, the harder it is to maintain that distinction. > Unlocking the development server so that it could serve a WSGI > application other than django.core.handlers.wsgi:WSGIHandler would be > a huge step forward, and doing it is a trivial task: That server could > take the WSGI application defined in the setting WSGI_APPLICATION. > When that setting is not set, django.core.handlers.wsgi:WSGIHandler > would be used. ... Ok - I'm still not completely sold on the need for this, but I'm at least interested in seeing some code to see how big a change it would be. If it doesn't require a whole lot of complexity, it may be worth making this change. >> > Better request objects >> > ---------------------- >> >> > There are two problems with Django's request objects from a WSGI point >> > of view: >> >> > - It copies the WSGI environment variables. That makes >> > interoperability with WSGI libraries harder or not possible at all, >> > because the request can be changed but the WSGI environ wouldn't be >> > modified, and vice versa, if the WSGI environ is modified the request >> > wouldn't be updated. >> > - It doesn't expose an API to handle most of the properties of a >> > request -- only the most common ones. >> >> > WebOb's request object is a better proxy of the WSGI environ from my >> > point of view, which is why I think Django's request object should >> > extend it. It doesn't have the problems above and it has more >> > features: >> >http://packages.python.org/twod.wsgi/manual/request-objects.html >> >> > I've managed to sub-class both WebOb.Request and Django's HttpRequest >> > without breaking backwards compatibility. >> >> Why do we need to subclass WebOb here? Again, you're proposing the >> introduction of a major dependency to Django; if there are >> bugs/inconsistencies in Django's WSGI implementation, those bugs >> should be fixed. I'm not sure I see why introducing WebOb is the right >> approach here. > > > If I had to implement the changes I'm proposing and I had the > limitation of not introducing a new dependency, I would end up copying > code from the WebOb project. > > WebOb offers "the" framework-independent Pythonic wrappers for WSGI > requests and responses, so it's widely used and known to work well. > I'd prefer to extend it instead of copying it, taking advantage of any > bugfix in the future. > > I believe reusing WebOb would be the best for you the core development > team (less code to write/maintain) and the users, who would get these > new backwards-compatible features thanks to code that has been around > for a while (instead of a less mature implementation specific to > Django). WebOb is not *the* set of Pythonic wrappers - it's *a" set of Pythonic wrappers. Yes, it's the implementation that is used by many web frameworks that aren't Django, but that doesn't automatically make it better. WebOb may be used by more frameworks, but I'd argue Django is used on more actual websites. As for 'less mature' -- Django has been open sourced for 5 years, and was in closed-source development for a couple of years before that. It's used by many high traffic sites. Claiming Django's implementation is immature borders on being troll bait. You'll get a lot more traction for your arguments if you stay away from the epithets. I'm a lot more interested in seeing the *specific* changes that are required in order to make Django's request/response WSGI compliant. >> > Embedded WSGI applications >> > -------------------------- >> >> > At present there's no built-in mechanism to run WSGI applications from >> > Django. Doing so could be extremely powerful and useful in some >> > situations, because it gives you the ability to filter the request >> > another application receives, and the response it returns -- Using >> > request and response objects, respectively. >> >> > And by "WSGI" application, I mean pretty much *any* Web application. >> > Java applications, PHP applications, a Web site likewww.google.com. >> > Anything, thanks to 3rd party WSGI applications that can run CGI >> > scripts and proxies, for example: >> >http://pythonpaste.org/modules/cgiapp.html#paste.cgiapp.CGIApplication >> >http://pythonpaste.org/modules/proxy.html#paste.proxy.Proxy >> >> > So, if you have, say, a Trac instance running on your Web site, you >> > can make it use Django's authentication data for the current user, and >> > authenticate users with your own Django-powered login form, and log >> > users out from Django, and many other things. Check this sample Django >> > application that implements a Single Sign-On mechanism with Trac: >> >http://bitbucket.org/Gustavo/weesgo/ >> >> Isn't this the first point again? If you're using mod_wsgi (or any >> other WSGI container) you can deploy whatever you want; the problem is >> purely the development server. > > No, I think there's a misunderstanding here. > > A WSGI application can be almost anything, as I said above. It could > be a Python function that serves the static files under a root > directory. Or a fully-featured, standalone, database-driven, Python- > based application. It could be a CGI/FastCGI script running PHP under- > the-hood. It could be a proxy to a web site running on a local/remote > server. Or it could be something completely different. ... > It's not pretty common but there are lots of cases for this, specially > for developers working on complex applications which usually iteract > with other applications. And Django is the only mainstream Python > framework that doesn't support this. I believe this should change. Ok - I misunderstood your argument. This sounds like a reasonable suggestion; again -- what specifically is the flaw/bug in Django that makes this impossible at present? What changes are required? >> > URL routing arguments support >> > ----------------------------- >> >> > From the manual <http://packages.python.org/twod.wsgi/manual/routing- >> > args.html>: >> >> > "routing_args is an extension to the WSGI standard which normalises >> > the place to put the arguments found in the URL. This is particularly >> > useful for 3rd party WSGI libraries and the dispatching components in >> > Web frameworks are expected to set it, but Django does not: Therefore >> > we created the RoutingArgsMiddleware Django middleware. >> >> > If you requested the path /blog/posts/hello-world/comments/3, then the >> > arguments hello-world and 3 will be available in the request object. >> > Depending on how you defined your URL patterns, they’ll be a >> > dictionary (if you used named groups) or a tuple (if you didn’t set >> > names for the matching groups). The former are referred to as “named >> > arguments” and the later as “positional arguments” in routing_args >> > specification. >> >> > RoutingArgsMiddleware simply puts the arguments found by the Django >> > URL resolver in the request object. It’s such a simple thing, but it’s >> > key for Django-independent libraries, which may not be run in the >> > context of a Django middleware nor a Django view." >> >> Ok; so here you're proposing that Django support an extension to the >> WSGI specification for which Django provides an alternate >> implementation. Until routing args becomes part of the WSGI spec, this >> strikes me as something that *should* be housed in an external >> project. > > That is an official extension from the Python Web-SIG, the same that > maintains the WSGI specification, and Django doesn't even provide > something similar AFAIK. > > Put simply, the routing_args specification requires dispatchers to put > the arguments found in the URL in the WSGI environment dictionary. > Django doesn't put this information in the WSGI environ (or in the > request object, AFAIR), but it's something absolutely trivial to > solve. > > This is an implementation in the form of Django middleware, but I > believe an appropriate solution would be to do it in the WSGI handler: > http://bitbucket.org/2degrees/twod.wsgi/src/tip/twod/wsgi/middleware.py Ok; I wasn't aware this was an official spec. I'd need to see the spec before I could pass comment on this. Can you provide a reference? I'd also point out that just because a specification exists, doesn't necessarily mean that we have to implement it. Django has a way to parse URL arguments, and it's fundamentally different to the "routing" approach advocated by Rails and Pylons; if there is a way for the two to coexist, I'm happy to look at it, but again, I'm not especially interested in making Django "more like Pylons" just so we can satisfy a spec. For example, CGI is a perfectly functioning spec - that doesn't mean we have to support it. >> We're aware that we haven't been the most responsive citizens when it >> comes to working with the rest of the Python web community, and we >> certainly intend to get more active with the WSGI specification >> process -- especially as it relates to the Python 3 transition. >> However, that doesn't mean we're about to completely change the way >> Django works in order to support aspects of the WSGI spec that the >> Django community has been able to live without, or for which the >> Django community has alternate approaches. We're entirely happy to >> accept changes that make Django more WSGI compliant; we're not going >> to be so responsive to changes that try to make Django into a >> different web framework. > > The changes I am proposing are all backwards compatible. None of them > require a major rewrite of the framework. And more importantly, people > who don't need these advanced features won't notice any difference*. > So, I don't see any of the things I have proposed as a risk of turning > Django into a different framework. Well, you say these changes don't require a major rewrite, but then you propose introducing two major external dependencies. When you factor in the social cost of introducing dependencies, it's not a trivial set of changes that you're proposing. In summary, here's what I'm hearing: You have four proposals: 1) Modifying the development server to provide a configurable WSGI interface 2) Cleaning up request/response objects 3) Allowing embedded WSGI applications 4) Support URL routing arguments. I'm not sold on the need for (1), but if it can be implemented in a simple patch, it's possibly worth doing. (2) and (3) sounds like a legitimate bugs/incompatibilities that are worth fixing. I don't know enough about (4) to be able to judge it. I'd be interested to see patches; Is your django-wsgi branch [1] still a useful resource to look at? From a quick inspection of the twod.wsgi code, it appears to be implemented in the 'use Paste and WebOb' style; do you have any version of the code that is implemented as patches against trunk Django? [1] http://bitbucket.org/Gustavo/django-wsgi/ I'm also interested how this relates to the work that was done on Django's WSGI interface during last year's GSoC. Are you at all familiar with the work on this branch? If so, what is the extent of the overlap between your code and that branch? Yours Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
