OK, here is some of the feedback from the Technical Board, aggregated together:
* There were questions around contextvars and if they might supplant the need for a threading.local override - I clarified why this doesn't work in the DEP. * Several board members queried around how we might distinguish async versions of functions/method from sync ones, and if we could get core Python to implement a better, language-level way of having both under the same name. For now, the conclusion is to pursue just having them as separate names, likely under a sub-object (so, for example, cache.get and cache.async.get), and ensure the documentation is very clear when we must present both * Templating was queried by multiple people, with one board member referencing the streaming template work that happened in the past ( https://github.com/django/django/pull/11157), and another wondering if we should just say that Django templates are sync forever and potentially look at Jinja2 for async support if users desire. My personal takeaway from this is that we should address async templates on its own, with a team who can do some experimentation and come back with a recommendation. * There was a concern over the fact that asgiref is now a core dependency of Django, joining tzdata, and that there had already been some negative user reactions about this in the wild. As part of helping out this problem, I removed asgiref's own dependency on async_timeout, so it's now just a single flat package. * There was concern about users being able to opt out of async mode should it cause a performance hit (like we allow with localisation), and another member mentioned the djangobench benchmark project. I stated that if adding async to the core flow impacted synchronous use by 10% we would find a way to make it strictly opt-in, as well as my personal belief that this may actually make things faster even for synchronous code and projects. I also said that I'd like to revive regular benchmarking and add an async one in to both help stop potential slowdown caused by adding async, as well as illustrating the advantage of async code for certain workloads. * A board member mentioned that the behaviour of transactions is already tricky, and adding this in might make it worse. I agree with them, but I've already started looking at ways to keep transactions and threads tied strictly together and put some explicit safeguards to stop them leaking across threads. That said, transactions and the ORM are my own personal biggest worry about this first phase of the project. * It was pointed out that while Django does not ship a threadlocal for request, this is a very common pattern and we need to make sure it works. I said that asgiref.local.Local is intended to be a drop-in, backwards-compatible replacement for threadlocal for this very reason; most projects will merely need to change a single import to get the correct behaviour. * There was some questioning about how the debug server should run and if it should enable the asyncio debug mode. I think it should, but I'm not sure at what stage we will make an async runserver the default, considering we will likely have to bring in Daphne or Uvicorn as a dependency to do so. * The final query I'm going to pull out here was non-technical - that the Django project has lost many contributors over the years, is essentially in a maintenance mode, and that we likely do not have the people to staff a project like this. I agree with the observation that things have substantially slowed down, but I personally believe that a project like async is exactly what Django needs to get going again. There's now a large amount of fertile ground to change and update things that isn't just fixing five year old bugs. Hopefully this gives you some idea of the conversation we had. In my years on the Board, this is by far the most detailed a vote has ever gotten, and I can only apologise to the incoming board for springing this on them right after an election! If anyone on this list would like to continue to talk about the above, or if a Board member wants to bring their conversation out here, you are all more than welcome. Andrew On Sun, Jul 21, 2019 at 1:02 PM Andrew Godwin <and...@aeracode.org> wrote: > I'll ask permission and then summarise the points raised back out here! > > Andrew > > On Sun, Jul 21, 2019 at 1:01 PM Jacob Kaplan-Moss <ja...@jacobian.org> > wrote: > >> Congratulations, and great news! >> >> I hope the TB will consider sharing details and/or a summary of the "long >> and involved vote"; I'll bet there's a bunch the broader community could >> learn from the specifics. >> >> Jacob >> >> On Sun, Jul 21, 2019 at 3:54 PM Andrew Godwin <and...@aeracode.org> >> wrote: >> >>> Hi everyone, >>> >>> After a long and involved vote, I can announce that the Technical Board >>> has voted in favour of DEP 0009 (Async Django), and so the DEP has been >>> moved to the "accepted" state. >>> >>> As some may have seen, I've started work on adding async support to >>> views ( >>> https://github.com/django/django/compare/master...andrewgodwin:async_views) >>> - this is, as the DEP states, the last "blocking" change before we can open >>> up lots of parallel work on making other parts of Django async, and so now >>> the DEP is approved the next step is to work out funding and organisation >>> for future async work. >>> >>> If you are interested in helping with fundraising, then please get in >>> touch with me directly; I have some ideas about how to structure it, but I >>> could do with some people to help out. Otherwise, stay tuned for more >>> information about how to get involved contributing and what to work on! >>> >>> Andrew >>> >>> On Sat, Jun 8, 2019 at 9:14 AM Andrew Godwin <and...@aeracode.org> >>> wrote: >>> >>>> >>>> >>>> On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon <chambon.pas...@gmail.com> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> There is something a little scary for me, in changing all the core of >>>>> Django to async, when this really helps only, imho, a tiny fraction of >>>>> users : websocket/long polling services, and reddit-like sites with >>>>> thousands+ hits per second. For most webpages and webservices, async >>>>> artillery sounds quite overkill. >>>>> >>>>> Are cpython threads inefficient ? As far as I know they are only >>>>> kernel threads constrained by the Gil, so they shouldnt wake up when they >>>>> are blocked on io syscalls/mutexes (or do they?), and context switches >>>>> remain acceptable compared to the slowness of python itself. >>>>> >>>> >>>> It's fine when you only at 5/10 threads - which, notably, is what most >>>> WSGI servers run at. When you get to the hundreds, though, you start losing >>>> a large proportion of your execution time (tens of percent, in some cases). >>>> >>>> >>>>> >>>>> We used to provide provisioning and automatic authentication for 20 >>>>> million users, with partner webservices tar-pitting us for sometimes 1mn. >>>>> The nightmare scenario. But with 2 machines, 1 process by core, and 800 >>>>> threads by process, it did the job, enough for us to answer millions of >>>>> hits a day. Without even relying on other no-recoding optimizations like >>>>> pypy or gevent. >>>>> >>>>> Async would certainly have been a relevant techno if we had known in >>>>> advance that our partners would be so slow, but avoiding the extra >>>>> complexity burden of this style (where a single buggy dependency can block >>>>> all requests in a process, where all modules have to be recoded for it) >>>>> was >>>>> also a huge benefit. And the limited thread pool also protected our DB >>>>> from >>>>> unbearable loads. >>>>> >>>> >>>> Please remember that even after this change, Django will still expect >>>> you to write synchronously by default, and not impose any of that extra >>>> complexity on you. We will only swap out the "native" implementation of >>>> things if the performance matches (within ~10%) or exceeds that of the >>>> synchronous version when there's a couple of threads going; it's expected >>>> this will largely be the case due to the direct benefits of idling less. >>>> >>>> But - the plan is not to make it more complex by default (you only have >>>> to interact with the async if you want to) or slower. >>>> >>>> >>>>> >>>>> It's very nice if a proper async ecosystem emerges in python, but I >>>>> fear lots of people are currently jumping into it without a need for such >>>>> performance, and at the expense of lots of much more important matters >>>>> like >>>>> robust ess, correctness, compatibility... like it happened for docker and >>>>> microservices, transforming into fragile bloatwares simple intranets, >>>>> which >>>>> just needed a single django codebase deployed in a single container. >>>>> >>>>> A few days ago I audited a well used django module, the current user >>>>> was stored in a global variable (!!!). People might eventually fix that >>>>> ticket, use threadlocals, and then switch to a future django-async without >>>>> realizing that the security issue has come back due to the way async >>>>> works. >>>>> >>>>> Still I hope I'm wrong, that the performance gains will prove worth >>>>> the software fragmentation and complexity brought by asyncio, but I still >>>>> dont understand them for 99% users... Especially as long as key-in-hand >>>>> solutions like greenlets exist for power users. >>>>> >>>>> >>>> I agree with you that there's a chance this is all useless and doesn't >>>> bear fruit, in which case I will be the first person to pull the plug and >>>> say that Python async isn't ready. However, I've been working with it for >>>> the last four years, including on several very large deployments, and there >>>> are some direct benefits that I believe we can get without making things a >>>> lot more complex, even inside Django. >>>> >>>> Andrew >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-developers+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/CAFwN1up%2BRzR3FcA8xBPPSfpY_5%2B5bfyZbJ%3DN9G0Ohy9kOGb53A%40mail.gmail.com >>> <https://groups.google.com/d/msgid/django-developers/CAFwN1up%2BRzR3FcA8xBPPSfpY_5%2B5bfyZbJ%3DN9G0Ohy9kOGb53A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-developers+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/CAK8PqJHBvv-Gd-%2BQR95NB52zr8meGhwgyFTysgqc1KWdvwP9KA%40mail.gmail.com >> <https://groups.google.com/d/msgid/django-developers/CAK8PqJHBvv-Gd-%2BQR95NB52zr8meGhwgyFTysgqc1KWdvwP9KA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upXBo9DFT9U6Sskb1NZnZnnhOQJa9M8bY4EU-5v9NsqJA%40mail.gmail.com.
