Re: Django Async DEP

[Andrew Godwin]

OK, here is some of the feedback from the Technical Board, aggregated
together:

* There were questions around contextvars and if they might supplant the
need for a threading.local override - I clarified why this doesn't work in
the DEP.

* Several board members queried around how we might distinguish async
versions of functions/method from sync ones, and if we could get core
Python to implement a better, language-level way of having both under the
same name. For now, the conclusion is to pursue just having them as
separate names, likely under a sub-object (so, for example, cache.get and
cache.async.get), and ensure the documentation is very clear when we must
present both

* Templating was queried by multiple people, with one board member
referencing the streaming template work that happened in the past (
https://github.com/django/django/pull/11157), and another wondering if we
should just say that Django templates are sync forever and potentially look
at Jinja2 for async support if users desire. My personal takeaway from this
is that we should address async templates on its own, with a team who can
do some experimentation and come back with a recommendation.

* There was a concern over the fact that asgiref is now a core dependency
of Django, joining tzdata, and that there had already been some negative
user reactions about this in the wild. As part of helping out this problem,
I removed asgiref's own dependency on async_timeout, so it's now just a
single flat package.

* There was concern about users being able to opt out of async mode should
it cause a performance hit (like we allow with localisation), and another
member mentioned the djangobench benchmark project. I stated that if adding
async to the core flow impacted synchronous use by 10% we would find a way
to make it strictly opt-in, as well as my personal belief that this may
actually make things faster even for synchronous code and projects. I also
said that I'd like to revive regular benchmarking and add an async one in
to both help stop potential slowdown caused by adding async, as well as
illustrating the advantage of async code for certain workloads.

* A board member mentioned that the behaviour of transactions is already
tricky, and adding this in might make it worse. I agree with them, but I've
already started looking at ways to keep transactions and threads tied
strictly together and put some explicit safeguards to stop them leaking
across threads. That said, transactions and the ORM are my own personal
biggest worry about this first phase of the project.

* It was pointed out that while Django does not ship a threadlocal for
request, this is a very common pattern and we need to make sure it works. I
said that asgiref.local.Local is intended to be a drop-in,
backwards-compatible replacement for threadlocal for this very reason; most
projects will merely need to change a single import to get the correct
behaviour.

* There was some questioning about how the debug server should run and if
it should enable the asyncio debug mode. I think it should, but I'm not
sure at what stage we will make an async runserver the default, considering
we will likely have to bring in Daphne or Uvicorn as a dependency to do so.

* The final query I'm going to pull out here was non-technical - that the
Django project has lost many contributors over the years, is essentially in
a maintenance mode, and that we likely do not have the people to staff a
project like this. I agree with the observation that things have
substantially slowed down, but I personally believe that a project like
async is exactly what Django needs to get going again. There's now a large
amount of fertile ground to change and update things that isn't just fixing
five year old bugs.

Hopefully this gives you some idea of the conversation we had. In my years
on the Board, this is by far the most detailed a vote has ever gotten, and
I can only apologise to the incoming board for springing this on them right
after an election!

If anyone on this list would like to continue to talk about the above, or
if a Board member wants to bring their conversation out here, you are all
more than welcome.

Andrew

On Sun, Jul 21, 2019 at 1:02 PM Andrew Godwin  wrote:

> I'll ask permission and then summarise the points raised back out here!
>
> Andrew
>
> On Sun, Jul 21, 2019 at 1:01 PM Jacob Kaplan-Moss 
> wrote:
>
>> Congratulations, and great news!
>>
>> I hope the TB will consider sharing details and/or a summary of the "long
>> and involved vote"; I'll bet there's a bunch the broader community could
>> learn from the specifics.
>>
>> Jacob
>>
>> On Sun, Jul 21, 2019 at 3:54 PM Andrew Godwin 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> After a long and involved vote, I can announce that the Technical Board
>>> has voted in favour of DEP 0009 (Async Django), and so the DEP has been
>>> moved to the "accepted" state.
>>>
>>> As some may have seen, I've started work on adding async support to
>>> views (
>>> 

Re: Django Async DEP

[Andrew Godwin]

On Sun, Jul 21, 2019 at 1:11 PM Ehigie Aito  wrote:

> Django 3.0?
>

Django follows time-based releases; what's in Django 3.0 will depend on
when we can get it landed. At the moment I am optimistic something will
make it in, but I make no promises!

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1ur%2BgWcjavb6%2B-_qH0bE-P2qVOuWULVq%3DuNRq5j-TWkpTA%40mail.gmail.com.

Re: Django Async DEP

[Ehigie Aito]

Django 3.0?

On Sun, Jul 21, 2019 at 9:02 PM Andrew Godwin  wrote:

> I'll ask permission and then summarise the points raised back out here!
>
> Andrew
>
> On Sun, Jul 21, 2019 at 1:01 PM Jacob Kaplan-Moss 
> wrote:
>
>> Congratulations, and great news!
>>
>> I hope the TB will consider sharing details and/or a summary of the "long
>> and involved vote"; I'll bet there's a bunch the broader community could
>> learn from the specifics.
>>
>> Jacob
>>
>> On Sun, Jul 21, 2019 at 3:54 PM Andrew Godwin 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> After a long and involved vote, I can announce that the Technical Board
>>> has voted in favour of DEP 0009 (Async Django), and so the DEP has been
>>> moved to the "accepted" state.
>>>
>>> As some may have seen, I've started work on adding async support to
>>> views (
>>> https://github.com/django/django/compare/master...andrewgodwin:async_views)
>>> - this is, as the DEP states, the last "blocking" change before we can open
>>> up lots of parallel work on making other parts of Django async, and so now
>>> the DEP is approved the next step is to work out funding and organisation
>>> for future async work.
>>>
>>> If you are interested in helping with fundraising, then please get in
>>> touch with me directly; I have some ideas about how to structure it, but I
>>> could do with some people to help out. Otherwise, stay tuned for more
>>> information about how to get involved contributing and what to work on!
>>>
>>> Andrew
>>>
>>> On Sat, Jun 8, 2019 at 9:14 AM Andrew Godwin 
>>> wrote:
>>>


 On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon 
 wrote:

> Hello,
>
> There is something a little scary for me, in changing all the core of
> Django to async, when this really helps only, imho, a tiny fraction of
> users : websocket/long polling services, and reddit-like sites with
> thousands+ hits per second. For most webpages and webservices, async
> artillery sounds quite overkill.
>
> Are cpython threads inefficient ? As far as I know they are only
> kernel threads constrained by the Gil, so they shouldnt wake up when they
> are blocked on io syscalls/mutexes (or do they?), and context switches
> remain acceptable compared to the slowness of python itself.
>

 It's fine when you only at 5/10 threads - which, notably, is what most
 WSGI servers run at. When you get to the hundreds, though, you start losing
 a large proportion of your execution time (tens of percent, in some cases).


>
> We used to provide provisioning and automatic authentication for 20
> million users, with partner webservices tar-pitting us for sometimes 1mn.
> The nightmare scenario. But with 2 machines, 1 process by core, and 800
> threads by process, it did the job, enough for us to answer millions of
> hits a day. Without even relying on other no-recoding optimizations like
> pypy or gevent.
>
> Async would certainly have been a relevant techno if we had known in
> advance that our partners would be so slow, but avoiding the extra
> complexity burden of this style (where a single buggy dependency can block
> all requests in a process, where all modules have to be recoded for it) 
> was
> also a huge benefit. And the limited thread pool also protected our DB 
> from
> unbearable loads.
>

 Please remember that even after this change, Django will still expect
 you to write synchronously by default, and not impose any of that extra
 complexity on you. We will only swap out the "native" implementation of
 things if the performance matches (within ~10%) or exceeds that of the
 synchronous version when there's a couple of threads going; it's expected
 this will largely be the case due to the direct benefits of idling less.

 But - the plan is not to make it more complex by default (you only have
 to interact with the async if you want to) or slower.


>
> It's very nice if a proper async ecosystem emerges in python, but I
> fear lots of people are currently jumping into it without a need for such
> performance, and at the expense of lots of much more important matters 
> like
> robust ess, correctness, compatibility... like it happened for docker and
> microservices, transforming into fragile bloatwares simple intranets, 
> which
> just needed a single django codebase deployed in a single container.
>
> A few days ago I audited a well used django module, the current user
> was stored in a global variable (!!!). People might eventually fix that
> ticket, use threadlocals, and then switch to a future django-async without
> realizing that the security issue has come back due to the way async 
> works.
>
> Still I hope I'm wrong, that the performance gains will prove worth
> the software fragmentation and complexity brought by asyncio, 

Re: Django Async DEP

[Andrew Godwin]

I'll ask permission and then summarise the points raised back out here!

Andrew

On Sun, Jul 21, 2019 at 1:01 PM Jacob Kaplan-Moss 
wrote:

> Congratulations, and great news!
>
> I hope the TB will consider sharing details and/or a summary of the "long
> and involved vote"; I'll bet there's a bunch the broader community could
> learn from the specifics.
>
> Jacob
>
> On Sun, Jul 21, 2019 at 3:54 PM Andrew Godwin  wrote:
>
>> Hi everyone,
>>
>> After a long and involved vote, I can announce that the Technical Board
>> has voted in favour of DEP 0009 (Async Django), and so the DEP has been
>> moved to the "accepted" state.
>>
>> As some may have seen, I've started work on adding async support to views
>> (
>> https://github.com/django/django/compare/master...andrewgodwin:async_views)
>> - this is, as the DEP states, the last "blocking" change before we can open
>> up lots of parallel work on making other parts of Django async, and so now
>> the DEP is approved the next step is to work out funding and organisation
>> for future async work.
>>
>> If you are interested in helping with fundraising, then please get in
>> touch with me directly; I have some ideas about how to structure it, but I
>> could do with some people to help out. Otherwise, stay tuned for more
>> information about how to get involved contributing and what to work on!
>>
>> Andrew
>>
>> On Sat, Jun 8, 2019 at 9:14 AM Andrew Godwin  wrote:
>>
>>>
>>>
>>> On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon 
>>> wrote:
>>>
 Hello,

 There is something a little scary for me, in changing all the core of
 Django to async, when this really helps only, imho, a tiny fraction of
 users : websocket/long polling services, and reddit-like sites with
 thousands+ hits per second. For most webpages and webservices, async
 artillery sounds quite overkill.

 Are cpython threads inefficient ? As far as I know they are only kernel
 threads constrained by the Gil, so they shouldnt wake up when they are
 blocked on io syscalls/mutexes (or do they?), and context switches remain
 acceptable compared to the slowness of python itself.

>>>
>>> It's fine when you only at 5/10 threads - which, notably, is what most
>>> WSGI servers run at. When you get to the hundreds, though, you start losing
>>> a large proportion of your execution time (tens of percent, in some cases).
>>>
>>>

 We used to provide provisioning and automatic authentication for 20
 million users, with partner webservices tar-pitting us for sometimes 1mn.
 The nightmare scenario. But with 2 machines, 1 process by core, and 800
 threads by process, it did the job, enough for us to answer millions of
 hits a day. Without even relying on other no-recoding optimizations like
 pypy or gevent.

 Async would certainly have been a relevant techno if we had known in
 advance that our partners would be so slow, but avoiding the extra
 complexity burden of this style (where a single buggy dependency can block
 all requests in a process, where all modules have to be recoded for it) was
 also a huge benefit. And the limited thread pool also protected our DB from
 unbearable loads.

>>>
>>> Please remember that even after this change, Django will still expect
>>> you to write synchronously by default, and not impose any of that extra
>>> complexity on you. We will only swap out the "native" implementation of
>>> things if the performance matches (within ~10%) or exceeds that of the
>>> synchronous version when there's a couple of threads going; it's expected
>>> this will largely be the case due to the direct benefits of idling less.
>>>
>>> But - the plan is not to make it more complex by default (you only have
>>> to interact with the async if you want to) or slower.
>>>
>>>

 It's very nice if a proper async ecosystem emerges in python, but I
 fear lots of people are currently jumping into it without a need for such
 performance, and at the expense of lots of much more important matters like
 robust ess, correctness, compatibility... like it happened for docker and
 microservices, transforming into fragile bloatwares simple intranets, which
 just needed a single django codebase deployed in a single container.

 A few days ago I audited a well used django module, the current user
 was stored in a global variable (!!!). People might eventually fix that
 ticket, use threadlocals, and then switch to a future django-async without
 realizing that the security issue has come back due to the way async works.

 Still I hope I'm wrong, that the performance gains will prove worth the
 software fragmentation and complexity brought by asyncio, but I still dont
 understand them for 99% users... Especially as long as key-in-hand
 solutions like greenlets exist for power users.


>>> I agree with you that there's a chance this is all useless and 

Re: Django Async DEP

[Jacob Kaplan-Moss]

Congratulations, and great news!

I hope the TB will consider sharing details and/or a summary of the "long
and involved vote"; I'll bet there's a bunch the broader community could
learn from the specifics.

Jacob

On Sun, Jul 21, 2019 at 3:54 PM Andrew Godwin  wrote:

> Hi everyone,
>
> After a long and involved vote, I can announce that the Technical Board
> has voted in favour of DEP 0009 (Async Django), and so the DEP has been
> moved to the "accepted" state.
>
> As some may have seen, I've started work on adding async support to views (
> https://github.com/django/django/compare/master...andrewgodwin:async_views)
> - this is, as the DEP states, the last "blocking" change before we can open
> up lots of parallel work on making other parts of Django async, and so now
> the DEP is approved the next step is to work out funding and organisation
> for future async work.
>
> If you are interested in helping with fundraising, then please get in
> touch with me directly; I have some ideas about how to structure it, but I
> could do with some people to help out. Otherwise, stay tuned for more
> information about how to get involved contributing and what to work on!
>
> Andrew
>
> On Sat, Jun 8, 2019 at 9:14 AM Andrew Godwin  wrote:
>
>>
>>
>> On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon 
>> wrote:
>>
>>> Hello,
>>>
>>> There is something a little scary for me, in changing all the core of
>>> Django to async, when this really helps only, imho, a tiny fraction of
>>> users : websocket/long polling services, and reddit-like sites with
>>> thousands+ hits per second. For most webpages and webservices, async
>>> artillery sounds quite overkill.
>>>
>>> Are cpython threads inefficient ? As far as I know they are only kernel
>>> threads constrained by the Gil, so they shouldnt wake up when they are
>>> blocked on io syscalls/mutexes (or do they?), and context switches remain
>>> acceptable compared to the slowness of python itself.
>>>
>>
>> It's fine when you only at 5/10 threads - which, notably, is what most
>> WSGI servers run at. When you get to the hundreds, though, you start losing
>> a large proportion of your execution time (tens of percent, in some cases).
>>
>>
>>>
>>> We used to provide provisioning and automatic authentication for 20
>>> million users, with partner webservices tar-pitting us for sometimes 1mn.
>>> The nightmare scenario. But with 2 machines, 1 process by core, and 800
>>> threads by process, it did the job, enough for us to answer millions of
>>> hits a day. Without even relying on other no-recoding optimizations like
>>> pypy or gevent.
>>>
>>> Async would certainly have been a relevant techno if we had known in
>>> advance that our partners would be so slow, but avoiding the extra
>>> complexity burden of this style (where a single buggy dependency can block
>>> all requests in a process, where all modules have to be recoded for it) was
>>> also a huge benefit. And the limited thread pool also protected our DB from
>>> unbearable loads.
>>>
>>
>> Please remember that even after this change, Django will still expect you
>> to write synchronously by default, and not impose any of that extra
>> complexity on you. We will only swap out the "native" implementation of
>> things if the performance matches (within ~10%) or exceeds that of the
>> synchronous version when there's a couple of threads going; it's expected
>> this will largely be the case due to the direct benefits of idling less.
>>
>> But - the plan is not to make it more complex by default (you only have
>> to interact with the async if you want to) or slower.
>>
>>
>>>
>>> It's very nice if a proper async ecosystem emerges in python, but I fear
>>> lots of people are currently jumping into it without a need for such
>>> performance, and at the expense of lots of much more important matters like
>>> robust ess, correctness, compatibility... like it happened for docker and
>>> microservices, transforming into fragile bloatwares simple intranets, which
>>> just needed a single django codebase deployed in a single container.
>>>
>>> A few days ago I audited a well used django module, the current user was
>>> stored in a global variable (!!!). People might eventually fix that ticket,
>>> use threadlocals, and then switch to a future django-async without
>>> realizing that the security issue has come back due to the way async works.
>>>
>>> Still I hope I'm wrong, that the performance gains will prove worth the
>>> software fragmentation and complexity brought by asyncio, but I still dont
>>> understand them for 99% users... Especially as long as key-in-hand
>>> solutions like greenlets exist for power users.
>>>
>>>
>> I agree with you that there's a chance this is all useless and doesn't
>> bear fruit, in which case I will be the first person to pull the plug and
>> say that Python async isn't ready. However, I've been working with it for
>> the last four years, including on several very large deployments, and there
>> are some 

Re: Django Async DEP

[Andrew Godwin]

Hi everyone,

After a long and involved vote, I can announce that the Technical Board has
voted in favour of DEP 0009 (Async Django), and so the DEP has been moved
to the "accepted" state.

As some may have seen, I've started work on adding async support to views (
https://github.com/django/django/compare/master...andrewgodwin:async_views)
- this is, as the DEP states, the last "blocking" change before we can open
up lots of parallel work on making other parts of Django async, and so now
the DEP is approved the next step is to work out funding and organisation
for future async work.

If you are interested in helping with fundraising, then please get in touch
with me directly; I have some ideas about how to structure it, but I could
do with some people to help out. Otherwise, stay tuned for more information
about how to get involved contributing and what to work on!

Andrew

On Sat, Jun 8, 2019 at 9:14 AM Andrew Godwin  wrote:

>
>
> On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon 
> wrote:
>
>> Hello,
>>
>> There is something a little scary for me, in changing all the core of
>> Django to async, when this really helps only, imho, a tiny fraction of
>> users : websocket/long polling services, and reddit-like sites with
>> thousands+ hits per second. For most webpages and webservices, async
>> artillery sounds quite overkill.
>>
>> Are cpython threads inefficient ? As far as I know they are only kernel
>> threads constrained by the Gil, so they shouldnt wake up when they are
>> blocked on io syscalls/mutexes (or do they?), and context switches remain
>> acceptable compared to the slowness of python itself.
>>
>
> It's fine when you only at 5/10 threads - which, notably, is what most
> WSGI servers run at. When you get to the hundreds, though, you start losing
> a large proportion of your execution time (tens of percent, in some cases).
>
>
>>
>> We used to provide provisioning and automatic authentication for 20
>> million users, with partner webservices tar-pitting us for sometimes 1mn.
>> The nightmare scenario. But with 2 machines, 1 process by core, and 800
>> threads by process, it did the job, enough for us to answer millions of
>> hits a day. Without even relying on other no-recoding optimizations like
>> pypy or gevent.
>>
>> Async would certainly have been a relevant techno if we had known in
>> advance that our partners would be so slow, but avoiding the extra
>> complexity burden of this style (where a single buggy dependency can block
>> all requests in a process, where all modules have to be recoded for it) was
>> also a huge benefit. And the limited thread pool also protected our DB from
>> unbearable loads.
>>
>
> Please remember that even after this change, Django will still expect you
> to write synchronously by default, and not impose any of that extra
> complexity on you. We will only swap out the "native" implementation of
> things if the performance matches (within ~10%) or exceeds that of the
> synchronous version when there's a couple of threads going; it's expected
> this will largely be the case due to the direct benefits of idling less.
>
> But - the plan is not to make it more complex by default (you only have to
> interact with the async if you want to) or slower.
>
>
>>
>> It's very nice if a proper async ecosystem emerges in python, but I fear
>> lots of people are currently jumping into it without a need for such
>> performance, and at the expense of lots of much more important matters like
>> robust ess, correctness, compatibility... like it happened for docker and
>> microservices, transforming into fragile bloatwares simple intranets, which
>> just needed a single django codebase deployed in a single container.
>>
>> A few days ago I audited a well used django module, the current user was
>> stored in a global variable (!!!). People might eventually fix that ticket,
>> use threadlocals, and then switch to a future django-async without
>> realizing that the security issue has come back due to the way async works.
>>
>> Still I hope I'm wrong, that the performance gains will prove worth the
>> software fragmentation and complexity brought by asyncio, but I still dont
>> understand them for 99% users... Especially as long as key-in-hand
>> solutions like greenlets exist for power users.
>>
>>
> I agree with you that there's a chance this is all useless and doesn't
> bear fruit, in which case I will be the first person to pull the plug and
> say that Python async isn't ready. However, I've been working with it for
> the last four years, including on several very large deployments, and there
> are some direct benefits that I believe we can get without making things a
> lot more complex, even inside Django.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: Django Async DEP

[Andrew Godwin]

On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon 
wrote:

> Hello,
>
> There is something a little scary for me, in changing all the core of
> Django to async, when this really helps only, imho, a tiny fraction of
> users : websocket/long polling services, and reddit-like sites with
> thousands+ hits per second. For most webpages and webservices, async
> artillery sounds quite overkill.
>
> Are cpython threads inefficient ? As far as I know they are only kernel
> threads constrained by the Gil, so they shouldnt wake up when they are
> blocked on io syscalls/mutexes (or do they?), and context switches remain
> acceptable compared to the slowness of python itself.
>

It's fine when you only at 5/10 threads - which, notably, is what most WSGI
servers run at. When you get to the hundreds, though, you start losing a
large proportion of your execution time (tens of percent, in some cases).


>
> We used to provide provisioning and automatic authentication for 20
> million users, with partner webservices tar-pitting us for sometimes 1mn.
> The nightmare scenario. But with 2 machines, 1 process by core, and 800
> threads by process, it did the job, enough for us to answer millions of
> hits a day. Without even relying on other no-recoding optimizations like
> pypy or gevent.
>
> Async would certainly have been a relevant techno if we had known in
> advance that our partners would be so slow, but avoiding the extra
> complexity burden of this style (where a single buggy dependency can block
> all requests in a process, where all modules have to be recoded for it) was
> also a huge benefit. And the limited thread pool also protected our DB from
> unbearable loads.
>

Please remember that even after this change, Django will still expect you
to write synchronously by default, and not impose any of that extra
complexity on you. We will only swap out the "native" implementation of
things if the performance matches (within ~10%) or exceeds that of the
synchronous version when there's a couple of threads going; it's expected
this will largely be the case due to the direct benefits of idling less.

But - the plan is not to make it more complex by default (you only have to
interact with the async if you want to) or slower.


>
> It's very nice if a proper async ecosystem emerges in python, but I fear
> lots of people are currently jumping into it without a need for such
> performance, and at the expense of lots of much more important matters like
> robust ess, correctness, compatibility... like it happened for docker and
> microservices, transforming into fragile bloatwares simple intranets, which
> just needed a single django codebase deployed in a single container.
>
> A few days ago I audited a well used django module, the current user was
> stored in a global variable (!!!). People might eventually fix that ticket,
> use threadlocals, and then switch to a future django-async without
> realizing that the security issue has come back due to the way async works.
>
> Still I hope I'm wrong, that the performance gains will prove worth the
> software fragmentation and complexity brought by asyncio, but I still dont
> understand them for 99% users... Especially as long as key-in-hand
> solutions like greenlets exist for power users.
>
>
I agree with you that there's a chance this is all useless and doesn't bear
fruit, in which case I will be the first person to pull the plug and say
that Python async isn't ready. However, I've been working with it for the
last four years, including on several very large deployments, and there are
some direct benefits that I believe we can get without making things a lot
more complex, even inside Django.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upp%3D0L9diMZr5qTxLeri329PuWWdMeN8uxyfmTt2jZM1w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Pascal Chambon]

Hello,

There is something a little scary for me, in changing all the core of
Django to async, when this really helps only, imho, a tiny fraction of
users : websocket/long polling services, and reddit-like sites with
thousands+ hits per second. For most webpages and webservices, async
artillery sounds quite overkill.

Are cpython threads inefficient ? As far as I know they are only kernel
threads constrained by the Gil, so they shouldnt wake up when they are
blocked on io syscalls/mutexes (or do they?), and context switches remain
acceptable compared to the slowness of python itself.

We used to provide provisioning and automatic authentication for 20 million
users, with partner webservices tar-pitting us for sometimes 1mn. The
nightmare scenario. But with 2 machines, 1 process by core, and 800 threads
by process, it did the job, enough for us to answer millions of hits a day.
Without even relying on other no-recoding optimizations like pypy or gevent.

Async would certainly have been a relevant techno if we had known in
advance that our partners would be so slow, but avoiding the extra
complexity burden of this style (where a single buggy dependency can block
all requests in a process, where all modules have to be recoded for it) was
also a huge benefit. And the limited thread pool also protected our DB from
unbearable loads.

It's very nice if a proper async ecosystem emerges in python, but I fear
lots of people are currently jumping into it without a need for such
performance, and at the expense of lots of much more important matters like
robust ess, correctness, compatibility... like it happened for docker and
microservices, transforming into fragile bloatwares simple intranets, which
just needed a single django codebase deployed in a single container.

A few days ago I audited a well used django module, the current user was
stored in a global variable (!!!). People might eventually fix that ticket,
use threadlocals, and then switch to a future django-async without
realizing that the security issue has come back due to the way async works.

Still I hope I'm wrong, that the performance gains will prove worth the
software fragmentation and complexity brought by asyncio, but I still dont
understand them for 99% users... Especially as long as key-in-hand
solutions like greenlets exist for power users.

Regards,
Pascal





Le ven. 7 juin 2019 à 19:41, Andrew Godwin  a écrit :

>
>
> On Fri, Jun 7, 2019 at 9:19 AM John Obelenus  wrote:
>
>> I wonder about the end-result payoff of this approach. In general,
>> Django/Python code is not going to be I/O bound, which is where
>> asynchronous approaches are going to get the bang for your buck. Even when
>> it comes to DB access—the DB is a lot faster than the python and django
>> code running against the result set. And too much context-switching (as you
>> noted) has painful ramifications for performance.
>>
>
> To the contrary, I have found that as you scale up, a large amount of your
> time becomes I/O (either HTTP calls to other components/hosted serviecs or
> database calls). Our APM at work shows me that it's around 80% of request
> time.
>
> Obviously we don't design Django just for large use cases, which is why
> it's not going to be the default, but with the massive growth of hosted
> services, I suspect this trend will continue to trickle down to smaller
> deploys too. And ultimately, for smaller deploys performance is rarely a
> concern anyway.
>
>
>>
>> I can absolutely see why creating a layer that handles asgi, websockets,
>> and http requests asynchronously is going to pay off. Bit time. But I'm
>> less certain that the ORM access will benefit from an asyncio approach. Do
>> we have anything that approaches a hard number that would tell us re-doing
>> the ORM layer in asyncio would get us X% performance benefit?
>>
>> I'm basing my thoughts off this well-reasoned look at performance:
>> https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/
>>
>>
> I do not personally have hard numbers that I am allowed to share,
> unfortunately, but I would encourage you to look at results on benchmarks
> that include database access - like this one (
> https://twitter.com/_tomchristie/status/1005001902092967936) using Python
> asyncio/ASGI - and see that it does make a difference. Obviously it doesn't
> matter for all deploys, but I believe it matters for the majority of site
> architectures as they scale up.
>
> Andrew
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/5CVsR9FSqmg/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at 

Re: Django Async DEP

[Andrew Godwin]

On Fri, Jun 7, 2019 at 9:19 AM John Obelenus  wrote:

> I wonder about the end-result payoff of this approach. In general,
> Django/Python code is not going to be I/O bound, which is where
> asynchronous approaches are going to get the bang for your buck. Even when
> it comes to DB access—the DB is a lot faster than the python and django
> code running against the result set. And too much context-switching (as you
> noted) has painful ramifications for performance.
>

To the contrary, I have found that as you scale up, a large amount of your
time becomes I/O (either HTTP calls to other components/hosted serviecs or
database calls). Our APM at work shows me that it's around 80% of request
time.

Obviously we don't design Django just for large use cases, which is why
it's not going to be the default, but with the massive growth of hosted
services, I suspect this trend will continue to trickle down to smaller
deploys too. And ultimately, for smaller deploys performance is rarely a
concern anyway.


>
> I can absolutely see why creating a layer that handles asgi, websockets,
> and http requests asynchronously is going to pay off. Bit time. But I'm
> less certain that the ORM access will benefit from an asyncio approach. Do
> we have anything that approaches a hard number that would tell us re-doing
> the ORM layer in asyncio would get us X% performance benefit?
>
> I'm basing my thoughts off this well-reasoned look at performance:
> https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/
>
>
I do not personally have hard numbers that I am allowed to share,
unfortunately, but I would encourage you to look at results on benchmarks
that include database access - like this one (
https://twitter.com/_tomchristie/status/1005001902092967936) using Python
asyncio/ASGI - and see that it does make a difference. Obviously it doesn't
matter for all deploys, but I believe it matters for the majority of site
architectures as they scale up.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urJVeamhLWLLDpkz_-e1fAM59x6b6rzyAV%3DZAUgg9qrvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[John Obelenus]

I wonder about the end-result payoff of this approach. In general, 
Django/Python code is not going to be I/O bound, which is where 
asynchronous approaches are going to get the bang for your buck. Even when 
it comes to DB access—the DB is a lot faster than the python and django 
code running against the result set. And too much context-switching (as you 
noted) has painful ramifications for performance.

I can absolutely see why creating a layer that handles asgi, websockets, 
and http requests asynchronously is going to pay off. Bit time. But I'm 
less certain that the ORM access will benefit from an asyncio approach. Do 
we have anything that approaches a hard number that would tell us re-doing 
the ORM layer in asyncio would get us X% performance benefit?

I'm basing my thoughts off this well-reasoned look at performance: 
https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/

On Thursday, May 9, 2019 at 12:11:19 PM UTC-4, Andrew Godwin wrote:
>
> Hello everyone,
>
> While the ASGI patch (https://github.com/django/django/pull/11209) is 
> maybe the first small step in a long road to async, it's the only real one 
> we could do without a DEP as it purely pulls on existing specs and bugs.
>
> To that end, I have drafted a DEP (provisionally #0009), "Async-capable 
> Django". It is a summary - and I use that word very lightly - of the rest 
> of the work to be done to make a version of Django that one could call 
> "async capable".
>
> You can view it here: 
> https://github.com/andrewgodwin/deps/blob/async/draft/0009-async.rst
>
> It is approximately 7,000 words long - while it could probably do with 
> some editing, the subject matter means it has to be quite in-depth. Even 
> then, it does not propose an exact plan to follow; instead, it proposes the 
> overall strategy and the high-level ideas about what needs to be done.
>
> If you are short on time, please read the Foreword and the High-Level 
> Summary, and then jump around using the table of contents to the sections 
> that interest you.
>
> Feedback on this is very much appreciated; you can either reply here or, 
> if you have comments that would benefit from precise location on the diff, 
> use the pull request here: https://github.com/django/deps/pull/56
>
> This is a very complex topic, and I appreciate feedback might take a 
> while; at least a couple of weeks to get everyone's thoughts. If you want a 
> more private channel to discuss concerns or design questions you're not 
> happy to talk about publicly, feel free to email me directly.
>
> This proposal does not quite fit the DEP template - we never really 
> planned for something of this scale - but I have done my best to make it 
> work; I think we still benefit from discussing and voting on it in the DEP 
> format, but suggestions on how to make it work better are welcome, provided 
> they fit within how Django operates as a community now.
>
> Finally, if there is positive feedback on this, we have the separate 
> question of funding it that I will discuss separately, if and when we get 
> that far. The DEP covers some of the topics, but it's notable that the 
> implementation speed of this varies considerably; if we could get, say, me 
> and someone else on it full-time (which is not a cheap or likely prospect), 
> we can probably be done inside of a single release. If we rely just on 
> volunteer time, it may take years. There is hopefully somewhere between 
> those two that works.
>
> Thanks for taking the time to read through!
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a51fac9e-5632-432c-adb1-74e9335c8264%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Thu, Jun 6, 2019 at 3:20 PM Pkl  wrote:

> Hello,
>
> I'm a little late to the party, thanks for the big overview on this
> complex matter.
>
> There are lots of thinks I still don't understand though, for example
> regarding "what it unlocks". What asyncio structures would allow to run
> several DB queries concurrently safely and easily, that can't be achieved
> with some threadpool-like mechanism in nowadays' cod ?
>

Python threads perform badly, as they force context-switches even when no
work is pending; you can't sensibly run more than 20/30 threads before the
overheads seriously start eating into your performance.


>
> Most importantly, I'm not getting why gevent/eventlet-style solutions are
> systematically being dismissed in favor of asyncio. And the more I
> read/talk about it, the less I understand what all that hype around asyncio
> is. This is a whole new language, which forces people to trashbin half of
> the Python ecosystem, and remake it with similar code but filled with
> incompatible async/await statements.
>
> ...
>
> Granted, I have little experience with Geven and Asyncio, but all
> experience feedbacks I've read so far mainly insist on minor limitations of
> greenlets, and on the fact that "people are mainly going with asyncio" (a
> self-fulfilling prophecy?). Considered the dramatic difference in workload
> between the two, I'd really love to understand what killer-features justify
> to go for the "recode everything" solution (or what greenlet limitations
> would be show-stoppers on the long term).
>

I can't give you a full answer about why they are being dismissed, but the
move is clearly systematic. My own personal experience is that writing with
gevent in particular ends up being very difficult, as it is not very
explicit about what is async, what causes a context switch, and so you end
up wrapping a lot of your code in locks to even try and get code that isn't
susceptible to nasty race conditions.

I suspect asyncio fits much more with the Zen of Python - you know exactly
when a context switch might occur (when you see an await), and modules
explicitly add support for it rather than having it monkey-patched in. In
addition, most people I know in the Python community who are actively
working on async libraries are doing so against asyncio.

You could make the same argument with trio - it's arguably a better,
cleaner async implementation. But, again, it's not where Python is at. This
change is already big enough that it's very important we keep consistency
with Python core to lower the workload. If Python core turned around and
blessed greenlets and gevent as the chosen async solution, I'd change my
mind, but I haven't seen any evidence of that over many years.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqgUUwL_%3DXS%2Be6tzieycn%3DjDZK6wFCKzHKxRtfukacfQw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Pkl]

Hello,

I'm a little late to the party, thanks for the big overview on this complex 
matter.

There are lots of thinks I still don't understand though, for example 
regarding "what it unlocks". What asyncio structures would allow to run 
several DB queries concurrently safely and easily, that can't be achieved 
with some threadpool-like mechanism in nowadays' cod ?

Most importantly, I'm not getting why gevent/eventlet-style solutions are 
systematically being dismissed in favor of asyncio. And the more I 
read/talk about it, the less I understand what all that hype around asyncio 
is. This is a whole new language, which forces people to trashbin half of 
the Python ecosystem, and remake it with similar code but filled with 
incompatible async/await statements. 

For sure, having so obstrusive keywords everywhere can be handy when 
writing highly concurrent code without mutexes. But who writes highly 
concurrent code in webservers ? The very principle of webservers is imho 
precisely to never interfere with other requests, never modify 
process-global structures, and at worst delegate some heavy or concurrent 
tasks to some dedicated executor.

Greenlet-style parallelism would bring long-running requests and high 
parallelism to Django without having to touch the bulk of the code code, 
just the I/O parts (DB connections...) ; and thread-local is already 
greenified by Gevent for exemple, according to docs. Why would it bring 
"much higher risk of race conditions and deadlocks without careful 
programming" ? Preemptive threads are much more dangerous than the implicit 
but deterministic context-switching which occurs when greenlets reach 
socket/disk/sleep operations, and yet race conditions seem to be the least 
of the problems of the huge majority of Django programmers.

I'd love to be wrong, but I have the feeling that with a fraction of the 
work required by django-asyncio, it would be possible to greenify the whole 
of Django, fix lots of current limitations and corner-cases of gevent (lack 
of builtin executor to offload tasks to real threads, lack of support in 
python package X/Y/Z...), and without having to recode any of the "middle" 
parts of existing modules. How can we say "third-party support for this 
style of concurrency is much weaker", whereas through monkey-patching, 
about ANY python module (except those using blocking C extensions) can be 
used in a Gevent project ?

Granted, I have little experience with Geven and Asyncio, but all 
experience feedbacks I've read so far mainly insist on minor limitations of 
greenlets, and on the fact that "people are mainly going with asyncio" (a 
self-fulfilling prophecy?). Considered the dramatic difference in workload 
between the two, I'd really love to understand what killer-features justify 
to go for the "recode everything" solution (or what greenlet limitations 
would be show-stoppers on the long term). 

Thanks in advance everyone for your feedback on this issue.

regards,
Pascal Chambon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/605d78af-f24a-46eb-9cac-4ad75547b604%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

Just a heads up that feedback on this draft DEP has slowed down, and so I
have merged it into the deps repository.

This is an invitation for any additional feedback before I take the DEP to
the Technical Board to get their opinion in a week or two. In the meantime,
I am going to start work on writing up a funding plan for this, including
various options for how we can pay people for their work.

Andrew

On Tue, May 14, 2019 at 2:55 AM Andrew Godwin  wrote:

>
>
> On Mon, May 13, 2019 at 4:31 AM Tobias Kunze  wrote:
>
>> Hi Andrew (and everybody following the discussion, of course),
>>
>> First off, thank you for your work here. DEP9 is an excellent technical
>> document, and it was as easy and pleasant to read as a document of this
>> scope and depth can be.
>>
>> Especially the Motivation section was very insightful – it might be
>> worth moving it up a bit, as I found myself dropping about a third of my
>> notes and questions from the first half of the DEP after reaching the
>> motivation section.
>> Also, discussing the Why first and the How later is a bit better
>> argumentatively – if I just spent half an hour reading about the How,
>> I'll start to see the Why as a given, so it's harder to reason about
>> alternatives upon reaching the motivation section. Maybe that's just me,
>> though.
>>
>
> The ordering that's there is defined by the DEP template, alas - though
> maybe that's just because of historical reasons. I agree it might be better
> to swap them.
>
>
>>
>> The Sequencing section is equally helpful to get a feeling for the
>> implementation work. It might be worth including a note on additional
>> future DEPs there, as those are only mentioned in the high level summary
>> (and in the templating section, I think).
>>
>
> I've added another paragraph in there to flesh it out a bit.
>
>
>>
>> I have a couple of questions and comments – none of which are meant to
>> criticise the direction of the DEP or arguments. It's just niggles,
>> really. I also left some comments on the PR that concern only some
>> phrasing.
>>
>> The DEP doesn't really include a discussion of potential downsides. Even
>> if that's not in scope of a DEP, I'd like to ask this here:
>>
>> - Is there a potential negative impact for Django users who just
>>   continue to use Django? The only mention of this says "Running code in
>>   threads is likely not going to increase performance - the overhead
>>   added will probably decrease it very slightly in the case where you're
>>   just running normal, linear code". Do you have any details on that? I
>>   would have expected some (even just a sentence or two) discussion of
>>   potential downsides in the Rationale section next to the alternatives.
>> - Is there a potential negative impact on Django if work on this
>>   proposal takes a long time/is abandoned?
>>
>
> These are both good questions - I have added in explainers about them to
> the "Motivation" section. (In short: Yes, but we'll keep it to a small
> impact; No, as long as we're good about code committing and being
> sustainable).
>
>
>>
>>
>> Less general remarks:
>>
>> > Every single feature that is converted to be async internally will also
>> present
>> > a synchronous interface that is backwards-compatible with the API as it
>> stands
>> > today (in 2.2), at least for the normal deprecation period.
>>
>> As it stands, this will sound a lot like "we'll deprecate lots of
>> synchronous interfaces soon" to people who are afraid of exactly that.
>> It's probably worth to clarify this here, or to choose different
>> phrasing, unless that's what you're planning to do.
>>
>
> Good point, I've made this a lot clearer.
>
>
>>
>> > This general overview works on nearly all features on Django that need
>> to be
>> > async, with the exceptions mostly being places where the Python
>> language itself
>> > does not provide async equivalents to features we already use.
>>
>> Can you give examples for this? They don't need to be in the draft, I
>> think, I'd just like to understand this part better.
>>
>
> The classic one is attribute access - Django uses "
> model_instance.related_field.name" to do work in the background.
> Attribute access can't be async, though, so we can no longer call the
> database.
>
> It also exists for length - doing "len(queryset)" does not pass down an
> async context, so you can't do blocking work down there in a subthread. The
> pattern continues for most things that Python makes objects supply
> __special__ methods for; unless serious work is done to Python as an async
> language, these will likely stay the same indefinitely.
>
>
>>
>> > Asynchronous views will continue to be wrapped in an ``atomic()`` block
>> by
>> > default - while this reduces immediate performance gains, as it will
>> lock all
>> > ORM queries to a single subthread (see "The ORM" below), it is what our
>> users
>> > will expect and much safer. If they want to run ORM queries
>> concurrently, they
>> > will have to explicitly 

Re: Django Async DEP

[Andrew Godwin]

On Mon, May 13, 2019 at 4:31 AM Tobias Kunze  wrote:

> Hi Andrew (and everybody following the discussion, of course),
>
> First off, thank you for your work here. DEP9 is an excellent technical
> document, and it was as easy and pleasant to read as a document of this
> scope and depth can be.
>
> Especially the Motivation section was very insightful – it might be
> worth moving it up a bit, as I found myself dropping about a third of my
> notes and questions from the first half of the DEP after reaching the
> motivation section.
> Also, discussing the Why first and the How later is a bit better
> argumentatively – if I just spent half an hour reading about the How,
> I'll start to see the Why as a given, so it's harder to reason about
> alternatives upon reaching the motivation section. Maybe that's just me,
> though.
>

The ordering that's there is defined by the DEP template, alas - though
maybe that's just because of historical reasons. I agree it might be better
to swap them.


>
> The Sequencing section is equally helpful to get a feeling for the
> implementation work. It might be worth including a note on additional
> future DEPs there, as those are only mentioned in the high level summary
> (and in the templating section, I think).
>

I've added another paragraph in there to flesh it out a bit.


>
> I have a couple of questions and comments – none of which are meant to
> criticise the direction of the DEP or arguments. It's just niggles,
> really. I also left some comments on the PR that concern only some
> phrasing.
>
> The DEP doesn't really include a discussion of potential downsides. Even
> if that's not in scope of a DEP, I'd like to ask this here:
>
> - Is there a potential negative impact for Django users who just
>   continue to use Django? The only mention of this says "Running code in
>   threads is likely not going to increase performance - the overhead
>   added will probably decrease it very slightly in the case where you're
>   just running normal, linear code". Do you have any details on that? I
>   would have expected some (even just a sentence or two) discussion of
>   potential downsides in the Rationale section next to the alternatives.
> - Is there a potential negative impact on Django if work on this
>   proposal takes a long time/is abandoned?
>

These are both good questions - I have added in explainers about them to
the "Motivation" section. (In short: Yes, but we'll keep it to a small
impact; No, as long as we're good about code committing and being
sustainable).


>
>
> Less general remarks:
>
> > Every single feature that is converted to be async internally will also
> present
> > a synchronous interface that is backwards-compatible with the API as it
> stands
> > today (in 2.2), at least for the normal deprecation period.
>
> As it stands, this will sound a lot like "we'll deprecate lots of
> synchronous interfaces soon" to people who are afraid of exactly that.
> It's probably worth to clarify this here, or to choose different
> phrasing, unless that's what you're planning to do.
>

Good point, I've made this a lot clearer.


>
> > This general overview works on nearly all features on Django that need
> to be
> > async, with the exceptions mostly being places where the Python language
> itself
> > does not provide async equivalents to features we already use.
>
> Can you give examples for this? They don't need to be in the draft, I
> think, I'd just like to understand this part better.
>

The classic one is attribute access - Django uses "
model_instance.related_field.name" to do work in the background. Attribute
access can't be async, though, so we can no longer call the database.

It also exists for length - doing "len(queryset)" does not pass down an
async context, so you can't do blocking work down there in a subthread. The
pattern continues for most things that Python makes objects supply
__special__ methods for; unless serious work is done to Python as an async
language, these will likely stay the same indefinitely.


>
> > Asynchronous views will continue to be wrapped in an ``atomic()`` block
> by
> > default - while this reduces immediate performance gains, as it will
> lock all
> > ORM queries to a single subthread (see "The ORM" below), it is what our
> users
> > will expect and much safer. If they want to run ORM queries
> concurrently, they
> > will have to explicitly opt out of having the transaction around the
> view using
> > the existing ``non_atomic_requests`` mechanism, though we will need to
> improve
> > the documentation around it.
>
> By default, Django's views are not wrapped in ``atomic()`` blocks. This
> is only the case if ``ATOMIC_REQUESTS`` is ``True``, which it isn't by
> default. Not sure if an off-by-default feature is worth an entire
> paragraph here, but in any case, please make it clear that not every
> async view will be wrapped in an ``atomic()`` block (unless I'm mistaken
> and they will be?).
>

I have updated this. I have kept 

Re: Django Async DEP

[Tobias Kunze]

Hi Andrew (and everybody following the discussion, of course),

First off, thank you for your work here. DEP9 is an excellent technical
document, and it was as easy and pleasant to read as a document of this
scope and depth can be.

Especially the Motivation section was very insightful – it might be
worth moving it up a bit, as I found myself dropping about a third of my
notes and questions from the first half of the DEP after reaching the
motivation section.
Also, discussing the Why first and the How later is a bit better
argumentatively – if I just spent half an hour reading about the How,
I'll start to see the Why as a given, so it's harder to reason about
alternatives upon reaching the motivation section. Maybe that's just me,
though.

The Sequencing section is equally helpful to get a feeling for the
implementation work. It might be worth including a note on additional
future DEPs there, as those are only mentioned in the high level summary
(and in the templating section, I think).

I have a couple of questions and comments – none of which are meant to
criticise the direction of the DEP or arguments. It's just niggles,
really. I also left some comments on the PR that concern only some
phrasing.

The DEP doesn't really include a discussion of potential downsides. Even
if that's not in scope of a DEP, I'd like to ask this here:

- Is there a potential negative impact for Django users who just
  continue to use Django? The only mention of this says "Running code in
  threads is likely not going to increase performance - the overhead
  added will probably decrease it very slightly in the case where you're
  just running normal, linear code". Do you have any details on that? I
  would have expected some (even just a sentence or two) discussion of
  potential downsides in the Rationale section next to the alternatives.
- Is there a potential negative impact on Django if work on this
  proposal takes a long time/is abandoned?


Less general remarks:

> Every single feature that is converted to be async internally will also 
> present
> a synchronous interface that is backwards-compatible with the API as it stands
> today (in 2.2), at least for the normal deprecation period.

As it stands, this will sound a lot like "we'll deprecate lots of
synchronous interfaces soon" to people who are afraid of exactly that.
It's probably worth to clarify this here, or to choose different
phrasing, unless that's what you're planning to do.

> This general overview works on nearly all features on Django that need to be
> async, with the exceptions mostly being places where the Python language 
> itself
> does not provide async equivalents to features we already use.

Can you give examples for this? They don't need to be in the draft, I
think, I'd just like to understand this part better.

> Asynchronous views will continue to be wrapped in an ``atomic()`` block by
> default - while this reduces immediate performance gains, as it will lock all
> ORM queries to a single subthread (see "The ORM" below), it is what our users
> will expect and much safer. If they want to run ORM queries concurrently, they
> will have to explicitly opt out of having the transaction around the view 
> using
> the existing ``non_atomic_requests`` mechanism, though we will need to improve
> the documentation around it.

By default, Django's views are not wrapped in ``atomic()`` blocks. This
is only the case if ``ATOMIC_REQUESTS`` is ``True``, which it isn't by
default. Not sure if an off-by-default feature is worth an entire
paragraph here, but in any case, please make it clear that not every
async view will be wrapped in an ``atomic()`` block (unless I'm mistaken
and they will be?).

> In some ways, this
> will end up looking more like Django 1.0 era middleware again from an internal
> perspective.

It might be worth to make it clear that the middleware interface doesn't
change on the user-facing side, though.

> Whenever a
> ``new_connections()`` block is entered, the transactions do not persist 
> inside,
> but transactions can be made inside the ``new_connections()`` block and run
> against those connections.

I think this was the most complicated sentence in the entire document.
It took me several tries to parse it in a way that could be correct.
Could you try to clarify? I think the missing reference for the "the
transactions" and "those connections" probably led to my confusion.

> This means that, at some point, the ``valid`` methods and ``save``, at
> minimum, need to be able to be called in an async fashion.

The ``valid`` methods? Did you mean the ``clean`` (and ``clean_*``)
methods, or am I missing something?

> While Python is not a perfect asynchronous language, and there are some flaws
> in the core design of ``asyncio``,

This leads me to a question I haven't really seen discussed so far: How
stable is the asyncio API by now? In the 3.x releases so far, asyncio
API has shifted quite a bit. I looked through the 3.8 release notes, but

Re: Django Async DEP

[Andrew Godwin]

On Fri, May 10, 2019 at 3:51 AM Asif Saif Uddin  wrote:

> I have a separate question. Is it possible to get the django 3.0 asgi
> things into a different package to use with django 2.2?
>
> Thanks for the great work.
>
>
Unfortunately not - this is covered a little in the section in the DEP that
says why we can't do it in a separate package, but basically, the changes
required to Django are too deep to do separately (or even as a long-running
fork).

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uppnH8r_%3Dgk_o%2BuuP%2BpxR7Da6HQzR8%2B%3DmyM3yHkEZcyDw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Asif Saif Uddin]

I have a separate question. Is it possible to get the django 3.0 asgi 
things into a different package to use with django 2.2?

Thanks for the great work.

On Friday, May 10, 2019 at 3:42:08 PM UTC+6, Patryk Zawadzki wrote:
>
> We've always considered that implicit queries on attribute access were an 
>> intractable problem. I said it on stage an DjangoCon US 2013. I'm now 
>> wondering if I was wrong all along! In an async ORM context, every time we 
>> traverse a relation, we could create a Future that would execute the query 
>> and resolve to its result. This would require one await per SQL query so 
>> we'd still get the benefit of making queries explicit, although to a lesser 
>> extent than with an explicit select/prefetch_related.
>>
>
> In theory there's nothing stopping you from turning prefetch_related int 
> on explicit fetch_related:
>
> queryset = Foo.objects.filter(Foo.publish_date <= datetime.date.today())
>
> queryset = fetch_related(queryset[:10], Foo.author)  # with a future
>
> results = await queryset.all()
>
> fetch_related(results, Foo.category)  # with a result set, returns identity
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e738b8a8-89eb-4883-8e59-14647d0ecc17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

>
> We've always considered that implicit queries on attribute access were an 
> intractable problem. I said it on stage an DjangoCon US 2013. I'm now 
> wondering if I was wrong all along! In an async ORM context, every time we 
> traverse a relation, we could create a Future that would execute the query 
> and resolve to its result. This would require one await per SQL query so 
> we'd still get the benefit of making queries explicit, although to a lesser 
> extent than with an explicit select/prefetch_related.
>

In theory there's nothing stopping you from turning prefetch_related int on 
explicit fetch_related:

queryset = Foo.objects.filter(Foo.publish_date <= datetime.date.today())

queryset = fetch_related(queryset[:10], Foo.author)  # with a future

results = await queryset.all()

fetch_related(results, Foo.category)  # with a result set, returns identity

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6c2d073d-fa21-43f2-94aa-db12c0f817da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Thu, May 9, 2019 at 2:44 PM KimSia Sim  wrote:

> I have a question that strays slightly away from the main topic. I have
> looked at Tom's repos. Is it encode/databases
> https://github.com/encode/databases that you're referring to?
>
> Or do you mean Tom's working on an async ORM that works within Django?
>

It is indeed encode/databases!

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoRf2Ug5b870zspfudncKPK%3DaWjff-z%2BGCFbW53pGsxxQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Thu, May 9, 2019 at 2:15 PM Patryk Zawadzki  wrote:

> I don't think it makes a significant difference from a readability
>> perspective in this example. It does have some advantages, though:
>>
>> - It could be easier to implement a sync version based on the async one,
>> or vice-versa, if each one has its own class. It will probably be more DRY.
>> - It could also be a bit more usable by developers, especially in IDEs
>>
>
> Speaking about convenience, it would be great if we could make the sync
> versions raise exceptions when used with an event loop attached and the
> async ones fail when there isn't a loop running. I assume synchronous views
> would be executed within a worker thread with the event loop explicitly set
> to None.
>

My ASGI patch already does this with the @async_unsafe decorator all over
the ORM so people can't screw up by accident. It would be quite easy to
extend this to enforcement on both the sync and async versions - there's
maybe an edge case that you can call an async function from a thread you
have not started an event loop in _yet_, but I'd rather see if and when
that happens and provide a workaround, maybe.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uq4HX%3DHQa9Gmymckv-T_qXbGgA%3DtUrHn2pG8ejVSML3GA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

>
> I don't think it makes a significant difference from a readability
> perspective in this example. It does have some advantages, though:
>
> - It could be easier to implement a sync version based on the async one,
> or vice-versa, if each one has its own class. It will probably be more DRY.
> - It could also be a bit more usable by developers, especially in IDEs
>

Right. It could also be a sub-class on the class in question to make the
implementation easier - e.g. "core.cache.cache.sync" is a literal
passthrough, and "core.cache.cache.async" is the async wrapper
implementation.


>
> Also it could provide an alternative solution for the async attribute
> access problem: `model_instance.async.related_field` could be a Future
> that, when awaited, resolves to the related field. So you could `await
> model_instance.async.related_field` instead of having to `select_related`.
> To be honest I didn't investigate all the consequences of this idea. It
> seemed worth mentioning, though, even if it turns out to be impractical :-)
>

I think this might be feasible? I'd want to probably push it to have its
own research and proposal, probably, but it might be a nice way out of the
initial thing of requiring select_related. I just don't know enough about
how that might cascade down the ORM internals to judge it at this point!

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upEsCpnL-81guSw_vKLn%3DWmfM3pamuFY%2BSZWLMerNC0yA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[KimSia Sim]

Hi Andrew, 

I joined this group and chat because I saw the twitter post you made about 
this DEP. I  find that interesting.


On Friday, May 10, 2019 at 3:49:30 AM UTC+8, Andrew Godwin wrote:
>
>  Tom Christie, for example, has already started work on an asynchronous 
> ORM. Some of Django's biggest sites don't use the ORM - for example, 
> Instagram.
>
>>
>>
I have a question that strays slightly away from the main topic. I have 
looked at Tom's repos. Is it encode/databases 
 https://github.com/encode/databases that you're referring to?

Or do you mean Tom's working on an async ORM that works within Django?

Thank you. I think Django will be even more awesome with async features

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/521a0e83-a23d-4d0b-a40f-45ecd30c4ef6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

>
> I don't think it makes a significant difference from a readability 
> perspective in this example. It does have some advantages, though: 
>
> - It could be easier to implement a sync version based on the async one, 
> or vice-versa, if each one has its own class. It will probably be more DRY. 
> - It could also be a bit more usable by developers, especially in IDEs 
>

Speaking about convenience, it would be great if we could make the sync 
versions raise exceptions when used with an event loop attached and the 
async ones fail when there isn't a loop running. I assume synchronous views 
would be executed within a worker thread with the event loop explicitly set 
to None. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5b6222c3-9f76-4eb3-8619-e3f44a17d6a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

W dniu czwartek, 9 maja 2019 22:47:48 UTC+2 użytkownik J. Pic napisał:
>
> I'm a bit confused here, what benefit are you getting from async emails if 
> you're already retrying emails in the background in production ?
>

Anything that uses I/O should be async to unblock the worker to process 
other things while for example waiting for an SMTP server to return. 
Network I/O like that often requires tens to hundreds of milliseconds, in 
this time you could process another request from another browser.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2ce12106-b423-4dc1-a95a-56f112e63a0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[J. Pic]

Nevermind my question you will get a lot more out of the workers, that
Django 3.0 is going to be really blazing fast like channels that calls for
a celebration xD

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op1-1apTUFZq9z5%3DEanpL%2B%2BA9nx9%3D3zS6_C-ZGNtJyBacog%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Aymeric Augustin]

> On 9 May 2019, at 22:06, Andrew Godwin  wrote:
> 
> Jannis suggested what I think might be a nicer approach on Twitter, which is 
> to add an async "proxy object" to access methods with, e.g.:
> 
> cache.get("foo")
> cache.async.get("foo")
> 
> This is still explicit but looks less ugly, and gives us a way in future to 
> make some modules transition to async-by-default (by e.g. supplying 
> cache.sync.get). What do you think?


I don't think it makes a significant difference from a readability perspective 
in this example. It does have some advantages, though:

- It could be easier to implement a sync version based on the async one, or 
vice-versa, if each one has its own class. It will probably be more DRY.
- It could also be a bit more usable by developers, especially in IDEs

Also it could provide an alternative solution for the async attribute access 
problem: `model_instance.async.related_field` could be a Future that, when 
awaited, resolves to the related field. So you could `await 
model_instance.async.related_field` instead of having to `select_related`. To 
be honest I didn't investigate all the consequences of this idea. It seemed 
worth mentioning, though, even if it turns out to be impractical :-)

We've always considered that implicit queries on attribute access were an 
intractable problem. I said it on stage an DjangoCon US 2013. I'm now wondering 
if I was wrong all along! In an async ORM context, every time we traverse a 
relation, we could create a Future that would execute the query and resolve to 
its result. This would require one await per SQL query so we'd still get the 
benefit of making queries explicit, although to a lesser extent than with an 
explicit select/prefetch_related.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/258CBCB2-2456-439F-82DF-E8760AE5EEF0%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[J. Pic]

Hi Andrew,

On Thu, May 9, 2019 at 10:04 PM Andrew Godwin  wrote:

>
> If you want guaranteed email delivery, that's a task for something like
> Celery or a third-party API; any method of sending emails in the background
> in the same process, be it threads or async coroutines, is going to be
> unreliable as the server may die any time. I don't think it's sensible for
> Django to try and solve this problem internally, at least not as part of
> this async push.
>

I'm a bit confused here, what benefit are you getting from async emails if
you're already retrying emails in the background in production ?

Thanks for your kind reply,

Have a great day

-- 
∞

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op18AA3Q7wguoZn%3D%2BfqHTAquXU0cESxF2i0GYzp%2BjB_ft%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

>
> I am also not a fan of the approach, but I did err towards being explicit. 
> Jannis suggested what I think might be a nicer approach on Twitter, which 
> is to add an async "proxy object" to access methods with, e.g.:
>
> cache.get("foo")
> cache.async.get("foo")
>
> This is still explicit but looks less ugly, and gives us a way in future 
> to make some modules transition to async-by-default (by e.g. supplying 
> cache.sync.get). What do you think?
>

I think I'd personally prefer to have to do `from django.asyncore.cache 
import cache` and if the synchronous cache raised an exception when 
attempting to use it with an event loop active. Like Aymeric, I don't see 
myself mixing async and sync code within one module.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d2c879cb-fad7-4e56-a1d9-1a31d5134e96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Thu, May 9, 2019 at 1:04 PM Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Hello Andrew,
>
> Thanks for your work putting together this plan. Within our constraints,
> it's a good plan.
>
> Regarding templating, I would say it isn't a priority because a developer
> who knows how to parallelize I/O bound operations will prefer (or at least
> accept) to perform these operations in the view, not in the template.
>
> I'm on the fence about the convention for async APIs. I'm not super
> excited by spraying async code with _async prefixes. The namespacing
> approach would allow for cleaner async code. Most Python modules should be
> either sync or async, not a mix of both styles. But I might be
> underestimating the importance of explicitness...
>
>
I am also not a fan of the approach, but I did err towards being explicit.
Jannis suggested what I think might be a nicer approach on Twitter, which
is to add an async "proxy object" to access methods with, e.g.:

cache.get("foo")
cache.async.get("foo")

This is still explicit but looks less ugly, and gives us a way in future to
make some modules transition to async-by-default (by e.g. supplying
cache.sync.get). What do you think?

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uowvL3FH9BR80hpOoBdRw-qUwpu5bA%3DBDLXmhY0WzBM%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

> Slightly off-topic but once we have an async ORM, making it synchronous
is not impossible (I believe either Channels or Daphne already have shims
that use a worker thread to spin the event loop until a future is
fulfilled).

This is in fact the proposed way to make the ORM work with both sync and
async worlds once the interior is fully-async.

> In one project I really enjoyed using channels just as a background
worker instead of celery, and the DEP does not talk much in the Email
section: use the async variant, can be tackled separately, low priority.
>
> For now the ORM works, but if sending an email fails (ie. SMTP down) it
will raise an exception that can end up in a 500 error, it would be great
if Django could retry so that emails leave when its SMTP is up again, would
that maybe weight in priority ?

If you want guaranteed email delivery, that's a task for something like
Celery or a third-party API; any method of sending emails in the background
in the same process, be it threads or async coroutines, is going to be
unreliable as the server may die any time. I don't think it's sensible for
Django to try and solve this problem internally, at least not as part of
this async push.

Andrew

On Thu, May 9, 2019 at 12:56 PM Patryk Zawadzki  wrote:

> That said, I also think it's important to allow the ORM to support both
>> modes in the long term. I truly believe the best way to be able to write
>> async code is to _have the choice to write it_, rather than being made to
>> all the time; if we make people use a separate, async ORM, then we force
>> them to write every view asynchronously, with all the extra danger and
>> thinking that requires. It's much better for Django to do the hard work,
>> and say "hey, if you want to write asynchronously or synchronously, that's
>> fine - it takes literally zero extra effort to go either way".
>>
>
> Slightly off-topic but once we have an async ORM, making it synchronous is
> not impossible (I believe either Channels or Daphne already have shims that
> use a worker thread to spin the event loop until a future is fulfilled).
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/402dbe77-44fc-485b-b058-e619a1576ddb%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urOnhEJDb6Sm45U59iF5hCZmN%3Dw-J%2B0er4UCnqh3qsmCw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Aymeric Augustin]

Hello Andrew,

Thanks for your work putting together this plan. Within our constraints, it's a 
good plan.

Regarding templating, I would say it isn't a priority because a developer who 
knows how to parallelize I/O bound operations will prefer (or at least accept) 
to perform these operations in the view, not in the template.

I'm on the fence about the convention for async APIs. I'm not super excited by 
spraying async code with _async prefixes. The namespacing approach would allow 
for cleaner async code. Most Python modules should be either sync or async, not 
a mix of both styles. But I might be underestimating the importance of 
explicitness...

I don't have much else to say on the DEP. It makes a lot of sense. I'm happy to 
see this happening!

Cheers,

-- 
Aymeric.



> On 9 May 2019, at 18:10, Andrew Godwin  wrote:
> 
> Hello everyone,
> 
> While the ASGI patch (https://github.com/django/django/pull/11209 
> ) is maybe the first small step 
> in a long road to async, it's the only real one we could do without a DEP as 
> it purely pulls on existing specs and bugs.
> 
> To that end, I have drafted a DEP (provisionally #0009), "Async-capable 
> Django". It is a summary - and I use that word very lightly - of the rest of 
> the work to be done to make a version of Django that one could call "async 
> capable".
> 
> You can view it here: 
> https://github.com/andrewgodwin/deps/blob/async/draft/0009-async.rst 
> 
> 
> It is approximately 7,000 words long - while it could probably do with some 
> editing, the subject matter means it has to be quite in-depth. Even then, it 
> does not propose an exact plan to follow; instead, it proposes the overall 
> strategy and the high-level ideas about what needs to be done.
> 
> If you are short on time, please read the Foreword and the High-Level 
> Summary, and then jump around using the table of contents to the sections 
> that interest you.
> 
> Feedback on this is very much appreciated; you can either reply here or, if 
> you have comments that would benefit from precise location on the diff, use 
> the pull request here: https://github.com/django/deps/pull/56 
> 
> 
> This is a very complex topic, and I appreciate feedback might take a while; 
> at least a couple of weeks to get everyone's thoughts. If you want a more 
> private channel to discuss concerns or design questions you're not happy to 
> talk about publicly, feel free to email me directly.
> 
> This proposal does not quite fit the DEP template - we never really planned 
> for something of this scale - but I have done my best to make it work; I 
> think we still benefit from discussing and voting on it in the DEP format, 
> but suggestions on how to make it work better are welcome, provided they fit 
> within how Django operates as a community now.
> 
> Finally, if there is positive feedback on this, we have the separate question 
> of funding it that I will discuss separately, if and when we get that far. 
> The DEP covers some of the topics, but it's notable that the implementation 
> speed of this varies considerably; if we could get, say, me and someone else 
> on it full-time (which is not a cheap or likely prospect), we can probably be 
> done inside of a single release. If we rely just on volunteer time, it may 
> take years. There is hopefully somewhere between those two that works.
> 
> Thanks for taking the time to read through!
> 
> Andrew
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAFwN1uqav3JXQA3m%2Bajf0Bd15QQX3JTZTfmH3Hc3ECWyRR3CVg%40mail.gmail.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at 

Re: Django Async DEP

[Patryk Zawadzki]

>
> That said, I also think it's important to allow the ORM to support both 
> modes in the long term. I truly believe the best way to be able to write 
> async code is to _have the choice to write it_, rather than being made to 
> all the time; if we make people use a separate, async ORM, then we force 
> them to write every view asynchronously, with all the extra danger and 
> thinking that requires. It's much better for Django to do the hard work, 
> and say "hey, if you want to write asynchronously or synchronously, that's 
> fine - it takes literally zero extra effort to go either way".
>

Slightly off-topic but once we have an async ORM, making it synchronous is 
not impossible (I believe either Channels or Daphne already have shims that 
use a worker thread to spin the event loop until a future is fulfilled).

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/402dbe77-44fc-485b-b058-e619a1576ddb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[J. Pic]

In one project I really enjoyed using channels just as a background worker
instead of celery, and the DEP does not talk much in the Email section: use
the async variant, can be tackled separately, low priority.

For now the ORM works, but if sending an email fails (ie. SMTP down) it
will raise an exception that can end up in a 500 error, it would be great
if Django could retry so that emails leave when its SMTP is up again, would
that maybe weight in priority ?

Also, you would probably like to view the queue in the admin, maybe make an
unmanaged Model for tasks. So, as simple as Email would look like, I think
efforts in it could benefit in a much wider way too.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op18rkxnz04hVciJs_pHEvmfG2ir_bMafPaYQ7d9HGTJhoQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

I would agree with both of you - I think the most important thing is to get
the view layer async-capable, as that then lets sites use any manner of
asynchronous libraries that already exist to get experiments and unique
things going. Tom Christie, for example, has already started work on an
asynchronous ORM. Some of Django's biggest sites don't use the ORM - for
example, Instagram.

That said, I also think it's important to allow the ORM to support both
modes in the long term. I truly believe the best way to be able to write
async code is to _have the choice to write it_, rather than being made to
all the time; if we make people use a separate, async ORM, then we force
them to write every view asynchronously, with all the extra danger and
thinking that requires. It's much better for Django to do the hard work,
and say "hey, if you want to write asynchronously or synchronously, that's
fine - it takes literally zero extra effort to go either way".

This is why I propose in the DEP that we do the view layer first, and then
move onto the ORM as a second wave.

Andrew

On Thu, May 9, 2019 at 12:29 PM Patryk Zawadzki  wrote:

> I'm not sure but for me the "What is Django" section answers the question.
>> For me Django is full of philosophy that seeds a great ecosystem of apps of
>> all sorts with a growing user base nonetheless, and a bunch of brilliant
>> hackers to look up to and inspire for more. Of course if you're into fixing
>> a particular technical issue such as "I want to make an efficient GraphQL
>> server", then even Django works even if it's not necessarily the most fit
>> for X reason. But for "I want to get a web page going with some forms and a
>> database and see where that goes" then Django is definitely enjoyable, and
>> at this point in time where tons of more sophisticated frameworks are born
>> it's clear that Django is still in the game and will be for the next 10
>> years. So, why not also get more features out of it while we've not been
>> abandoned by all the talented contributors ? It's not because we like
>> Django that we cant haz nice things ;)
>>
>
> I'm arguing the opposite, that by limiting the scope of the MVP to
> asynchronous views (even if middleware stays synchronous for now) and using
> an existing ORM we can make Django a viable solution for many new
> applications. And some of them don't need the full stack.
>
> I'd go as far as arguing that fewer and fewer applications benefit from
> Django providing all of the pieces (as many new applications only implement
> an API and render in the client) so treating them all as blockers here may
> not be beneficial.
>
> Best regards
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/314d7b71-1bf1-441b-ae49-e4fbeb5e1891%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urQVr%3DZ55Xqn7E%2B5-AeZU-egJAZ_%2BooAp_3nSSYnbNX7w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

>
> I'm not sure but for me the "What is Django" section answers the question. 
> For me Django is full of philosophy that seeds a great ecosystem of apps of 
> all sorts with a growing user base nonetheless, and a bunch of brilliant 
> hackers to look up to and inspire for more. Of course if you're into fixing 
> a particular technical issue such as "I want to make an efficient GraphQL 
> server", then even Django works even if it's not necessarily the most fit 
> for X reason. But for "I want to get a web page going with some forms and a 
> database and see where that goes" then Django is definitely enjoyable, and 
> at this point in time where tons of more sophisticated frameworks are born 
> it's clear that Django is still in the game and will be for the next 10 
> years. So, why not also get more features out of it while we've not been 
> abandoned by all the talented contributors ? It's not because we like 
> Django that we cant haz nice things ;)
>

I'm arguing the opposite, that by limiting the scope of the MVP to 
asynchronous views (even if middleware stays synchronous for now) and using 
an existing ORM we can make Django a viable solution for many new 
applications. And some of them don't need the full stack.

I'd go as far as arguing that fewer and fewer applications benefit from 
Django providing all of the pieces (as many new applications only implement 
an API and render in the client) so treating them all as blockers here may 
not be beneficial.

Best regards

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/314d7b71-1bf1-441b-ae49-e4fbeb5e1891%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[J. Pic]

Oops too fast, if it's possible to split the DEP and delay the ORM as
advised by Patrick it could make it a lot easier to distribute the work, I
don't know really sry. Have a great day !

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op1-tDvM-raHfOLGCQYFFk7Aoz2CAa%3DY_bXcaj30Y887wzw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[J. Pic]

Hi Patryk,

I'm not sure but for me the "What is Django" section answers the question.
For me Django is full of philosophy that seeds a great ecosystem of apps of
all sorts with a growing user base nonetheless, and a bunch of brilliant
hackers to look up to and inspire for more. Of course if you're into fixing
a particular technical issue such as "I want to make an efficient GraphQL
server", then even Django works even if it's not necessarily the most fit
for X reason. But for "I want to get a web page going with some forms and a
database and see where that goes" then Django is definitely enjoyable, and
at this point in time where tons of more sophisticated frameworks are born
it's clear that Django is still in the game and will be for the next 10
years. So, why not also get more features out of it while we've not been
abandoned by all the talented contributors ? It's not because we like
Django that we cant haz nice things ;)

Great move Andrew and very exciting, keep it up and best luck to you !

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op19DRYB%2BtUJX-9fG_9EOeo0oqGZayn5SHHBdec9qweq8ng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

Hey Andrew,

Great work on the DEP, the task at hand is humongous.

Do you think it's worth it to try and make the ORM async? It contains tons 
of magic that is inherently incompatible with explicit I/O that is required 
for async/await to work, things like silently fetching relations on first 
access, silently fetching deferred fields, queryset laziness etc. Those 
need to be explicitly awaited now and having to await regular field access 
for the sake of making deferred fields work is (I believe) not worth it.

Conveniently, all of these magic properties are things that silently break 
performance of applications leading to the n+1 queries problem so I'm more 
than happy to get rid of them ;)

Seeing that we can't get rid of anything in the old synchronous ORM, one 
idea would be to just depend on a separate, asyncio-based ORM and configure 
it to use the same tables and relations that the old ORM uses. It could 
work like model forms or formset factories, give it a "classic" model 
class, a list of overrides (we can't hope to automatically match old field 
types to anything understood by another engine) and receive an 
asyncio-capable table. There's an obvious candidate that Tom Christie is 
working on.

We could even improve the querying syntax to use the SQL Alchemy's 
filtering operators with the hope of eventually reaching full static type 
safety[2] (one can only dream).

The new ORM would only be usable in new async views and other features 
could be implemented gradually. You don't need to async forms (but with Tom 
Christie's typesystem[3] you don't really need async forms) and async 
template loading to achieve great things.

You can already use things like requests-async[4] to access external 
resources, you can use libraries like Ariadne[5] to implement a GraphQL API 
with WebSocket-based subscriptions without having any of those (disclaimer: 
I work on Ariadne). It's okay if some things can't be ported initially or 
if certain features are only available in their sync flavor. The community 
will pave the way and provide many missing pieces before we get to 
implementing them. It's okay if new async counterparts have different APIs 
if these APIs are either better or are a closer match to how async/await 
works.

[1] https://github.com/encode/orm
[2] https://docs.sqlalchemy.org/en/13/core/tutorial.html#operators
[3] https://github.com/encode/typesystem
[4] https://github.com/encode/requests-async
[5] https://github.com/mirumee/ariadne


PS: If Django provided abstractions for common event loop operations such 
as "await this collections of futures" and "ensure this task is executed 
even if not explicitly awaited" then I think it may be possible to avoid 
explicitly depending on asyncio.

All the best

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/da0d4c65-562b-44cc-abe0-e5c31e28d91d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.