I'll ask permission and then summarise the points raised back out here! Andrew
On Sun, Jul 21, 2019 at 1:01 PM Jacob Kaplan-Moss <ja...@jacobian.org> wrote: > Congratulations, and great news! > > I hope the TB will consider sharing details and/or a summary of the "long > and involved vote"; I'll bet there's a bunch the broader community could > learn from the specifics. > > Jacob > > On Sun, Jul 21, 2019 at 3:54 PM Andrew Godwin <and...@aeracode.org> wrote: > >> Hi everyone, >> >> After a long and involved vote, I can announce that the Technical Board >> has voted in favour of DEP 0009 (Async Django), and so the DEP has been >> moved to the "accepted" state. >> >> As some may have seen, I've started work on adding async support to views >> ( >> https://github.com/django/django/compare/master...andrewgodwin:async_views) >> - this is, as the DEP states, the last "blocking" change before we can open >> up lots of parallel work on making other parts of Django async, and so now >> the DEP is approved the next step is to work out funding and organisation >> for future async work. >> >> If you are interested in helping with fundraising, then please get in >> touch with me directly; I have some ideas about how to structure it, but I >> could do with some people to help out. Otherwise, stay tuned for more >> information about how to get involved contributing and what to work on! >> >> Andrew >> >> On Sat, Jun 8, 2019 at 9:14 AM Andrew Godwin <and...@aeracode.org> wrote: >> >>> >>> >>> On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon <chambon.pas...@gmail.com> >>> wrote: >>> >>>> Hello, >>>> >>>> There is something a little scary for me, in changing all the core of >>>> Django to async, when this really helps only, imho, a tiny fraction of >>>> users : websocket/long polling services, and reddit-like sites with >>>> thousands+ hits per second. For most webpages and webservices, async >>>> artillery sounds quite overkill. >>>> >>>> Are cpython threads inefficient ? As far as I know they are only kernel >>>> threads constrained by the Gil, so they shouldnt wake up when they are >>>> blocked on io syscalls/mutexes (or do they?), and context switches remain >>>> acceptable compared to the slowness of python itself. >>>> >>> >>> It's fine when you only at 5/10 threads - which, notably, is what most >>> WSGI servers run at. When you get to the hundreds, though, you start losing >>> a large proportion of your execution time (tens of percent, in some cases). >>> >>> >>>> >>>> We used to provide provisioning and automatic authentication for 20 >>>> million users, with partner webservices tar-pitting us for sometimes 1mn. >>>> The nightmare scenario. But with 2 machines, 1 process by core, and 800 >>>> threads by process, it did the job, enough for us to answer millions of >>>> hits a day. Without even relying on other no-recoding optimizations like >>>> pypy or gevent. >>>> >>>> Async would certainly have been a relevant techno if we had known in >>>> advance that our partners would be so slow, but avoiding the extra >>>> complexity burden of this style (where a single buggy dependency can block >>>> all requests in a process, where all modules have to be recoded for it) was >>>> also a huge benefit. And the limited thread pool also protected our DB from >>>> unbearable loads. >>>> >>> >>> Please remember that even after this change, Django will still expect >>> you to write synchronously by default, and not impose any of that extra >>> complexity on you. We will only swap out the "native" implementation of >>> things if the performance matches (within ~10%) or exceeds that of the >>> synchronous version when there's a couple of threads going; it's expected >>> this will largely be the case due to the direct benefits of idling less. >>> >>> But - the plan is not to make it more complex by default (you only have >>> to interact with the async if you want to) or slower. >>> >>> >>>> >>>> It's very nice if a proper async ecosystem emerges in python, but I >>>> fear lots of people are currently jumping into it without a need for such >>>> performance, and at the expense of lots of much more important matters like >>>> robust ess, correctness, compatibility... like it happened for docker and >>>> microservices, transforming into fragile bloatwares simple intranets, which >>>> just needed a single django codebase deployed in a single container. >>>> >>>> A few days ago I audited a well used django module, the current user >>>> was stored in a global variable (!!!). People might eventually fix that >>>> ticket, use threadlocals, and then switch to a future django-async without >>>> realizing that the security issue has come back due to the way async works. >>>> >>>> Still I hope I'm wrong, that the performance gains will prove worth the >>>> software fragmentation and complexity brought by asyncio, but I still dont >>>> understand them for 99% users... Especially as long as key-in-hand >>>> solutions like greenlets exist for power users. >>>> >>>> >>> I agree with you that there's a chance this is all useless and doesn't >>> bear fruit, in which case I will be the first person to pull the plug and >>> say that Python async isn't ready. However, I've been working with it for >>> the last four years, including on several very large deployments, and there >>> are some direct benefits that I believe we can get without making things a >>> lot more complex, even inside Django. >>> >>> Andrew >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-developers+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/CAFwN1up%2BRzR3FcA8xBPPSfpY_5%2B5bfyZbJ%3DN9G0Ohy9kOGb53A%40mail.gmail.com >> <https://groups.google.com/d/msgid/django-developers/CAFwN1up%2BRzR3FcA8xBPPSfpY_5%2B5bfyZbJ%3DN9G0Ohy9kOGb53A%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/CAK8PqJHBvv-Gd-%2BQR95NB52zr8meGhwgyFTysgqc1KWdvwP9KA%40mail.gmail.com > <https://groups.google.com/d/msgid/django-developers/CAK8PqJHBvv-Gd-%2BQR95NB52zr8meGhwgyFTysgqc1KWdvwP9KA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upqMLyZgtmcqgmYV8whH43SCXJimTCZGJxA4p1D5PRYTA%40mail.gmail.com.
