Re: A Django Async Roadmap

[Andrew Godwin]

The DEP is drafted and in the DEPs repo, and awaiting approval by the
freshly-elected Technical Board once I submit it. In the meantime, we
landed the ASGI patch, as well.

Andrew

On Tue, Jun 25, 2019 at 3:30 PM Chris Barry 
wrote:

> Hey all,
>
> Just wondering what the future of this is looking like?
>
> CB
>
> On Friday, 12 April 2019 07:33:35 UTC+1, Shaggy wrote:
>>
>> and how it is going ?
>> is there some interest from django devs?
>>
>> On Monday, 4 June 2018 15:18:23 UTC+2, Andrew Godwin wrote:
>>>
>>> Hello everyone,
>>>
>>> For a while now I have been working on potential plans for making Django
>>> async-capable, and I finally have a plan I am reasonably happy with and
>>> which I think we can actually do.
>>>
>>> This proposed roadmap, in its great length, is here:
>>>
>>> https://www.aeracode.org/2018/06/04/django-async-roadmap/
>>>
>>> I'd like to invite discussion on this potential plan - including:
>>>
>>>  - Do we think async is worth going after? Note that this is just async
>>> HTTP capability, not WebSockets (that would remain in Channels)
>>>
>>>  - Can we do this in a reasonable timeframe? If not, is there a way
>>> around that?
>>>
>>>  - Are the proposed modifications to how Django runs sensible?
>>>
>>>  - How should we fund this?
>>>
>>> There's many more potential questions, and I really would love feedback
>>> on this. I'm personally pretty convinced that we can and should do this,
>>> but this is a decision we cannot take lightly, and I would love to hear
>>> what you have to say.
>>>
>>> Andrew
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/6ea76507-4041-4850-ac6c-bb13a09af941%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoPkM6bZUnAvkK6Yq7P0%2BvtRm4yFHW-ZxzuQKOvr%3DPv%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Chris Barry]

Hey all,

Just wondering what the future of this is looking like?

CB

On Friday, 12 April 2019 07:33:35 UTC+1, Shaggy wrote:
>
> and how it is going ?
> is there some interest from django devs?
>
> On Monday, 4 June 2018 15:18:23 UTC+2, Andrew Godwin wrote:
>>
>> Hello everyone,
>>
>> For a while now I have been working on potential plans for making Django 
>> async-capable, and I finally have a plan I am reasonably happy with and 
>> which I think we can actually do.
>>
>> This proposed roadmap, in its great length, is here:
>>
>> https://www.aeracode.org/2018/06/04/django-async-roadmap/
>>
>> I'd like to invite discussion on this potential plan - including:
>>
>>  - Do we think async is worth going after? Note that this is just async 
>> HTTP capability, not WebSockets (that would remain in Channels)
>>
>>  - Can we do this in a reasonable timeframe? If not, is there a way 
>> around that?
>>
>>  - Are the proposed modifications to how Django runs sensible?
>>
>>  - How should we fund this?
>>
>> There's many more potential questions, and I really would love feedback 
>> on this. I'm personally pretty convinced that we can and should do this, 
>> but this is a decision we cannot take lightly, and I would love to hear 
>> what you have to say.
>>
>> Andrew
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6ea76507-4041-4850-ac6c-bb13a09af941%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Shaggy]

and how it is going ?
is there some interest from django devs?

On Monday, 4 June 2018 15:18:23 UTC+2, Andrew Godwin wrote:
>
> Hello everyone,
>
> For a while now I have been working on potential plans for making Django 
> async-capable, and I finally have a plan I am reasonably happy with and 
> which I think we can actually do.
>
> This proposed roadmap, in its great length, is here:
>
> https://www.aeracode.org/2018/06/04/django-async-roadmap/
>
> I'd like to invite discussion on this potential plan - including:
>
>  - Do we think async is worth going after? Note that this is just async 
> HTTP capability, not WebSockets (that would remain in Channels)
>
>  - Can we do this in a reasonable timeframe? If not, is there a way around 
> that?
>
>  - Are the proposed modifications to how Django runs sensible?
>
>  - How should we fund this?
>
> There's many more potential questions, and I really would love feedback on 
> this. I'm personally pretty convinced that we can and should do this, but 
> this is a decision we cannot take lightly, and I would love to hear what 
> you have to say.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/98e4a315-17a5-4cf9-9ea7-b83234480ec9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[James Addison]

If you're following that 'medium' article, then you'll see it 
uses https://github.com/jneight/django-db-geventpool, which has 
instructions for using `psycogreen` with gunicorn... I believe what it 
states is correct.

On Thursday, December 13, 2018 at 3:21:32 AM UTC, Ray Marceau wrote:
>
> James Addison,
>
>  I also currently use `psycogreen.gevent` for postgresql async behaviour.
>>
>
> How did you go about using psycogreen.gevent for async postgres?  I tried 
> using various tools and was unsuccessful in seeing any performance 
> improvements.  I was following the info here: 
> https://medium.com/@bfirsh/squeezing-every-drop-of-performance-out-of-a-django-app-on-heroku-4b5b1e5a3d44
>
> Have any suggestions/tips?  django blocking while waiting for postgres is 
> a huge bottleneck for my app.
>
> Thanks!
>
>
> How did you go about using psycogreen.gevent for postgresql async 
> behavior?  I'm 
> On Tuesday, August 21, 2018 at 10:16:03 AM UTC-7, James Addison wrote:
>>
>> On Monday, June 4, 2018 at 6:18:23 AM UTC-7, Andrew Godwin wrote:
>>>
>>> For a while now I have been working on potential plans for making Django 
>>> async-capable, and I finally have a plan I am reasonably happy with and 
>>> which I think we can actually do.
>>>
>>
>> Andrew,
>>
>> Out of curiosity, how would these kinds of changes interact with gunicorn 
>> running with a `gevent` worker? I also currently use `psycogreen.gevent` 
>> for postgresql async behaviour.
>>
>> Thanks,
>> James
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f765848c-6e8f-4402-8baf-ca5ed0c96b4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Ray Marceau]

James Addison,

 I also currently use `psycogreen.gevent` for postgresql async behaviour.
>

How did you go about using psycogreen.gevent for async postgres?  I tried 
using various tools and was unsuccessful in seeing any performance 
improvements.  I was following the info 
here: 
https://medium.com/@bfirsh/squeezing-every-drop-of-performance-out-of-a-django-app-on-heroku-4b5b1e5a3d44

Have any suggestions/tips?  django blocking while waiting for postgres is a 
huge bottleneck for my app.

Thanks!


How did you go about using psycogreen.gevent for postgresql async 
behavior?  I'm 
On Tuesday, August 21, 2018 at 10:16:03 AM UTC-7, James Addison wrote:
>
> On Monday, June 4, 2018 at 6:18:23 AM UTC-7, Andrew Godwin wrote:
>>
>> For a while now I have been working on potential plans for making Django 
>> async-capable, and I finally have a plan I am reasonably happy with and 
>> which I think we can actually do.
>>
>
> Andrew,
>
> Out of curiosity, how would these kinds of changes interact with gunicorn 
> running with a `gevent` worker? I also currently use `psycogreen.gevent` 
> for postgresql async behaviour.
>
> Thanks,
> James
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3f676d11-7b2c-41f7-a19f-d34f3b0a1b50%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[James Addison]

On Monday, June 4, 2018 at 6:18:23 AM UTC-7, Andrew Godwin wrote:
>
> For a while now I have been working on potential plans for making Django 
> async-capable, and I finally have a plan I am reasonably happy with and 
> which I think we can actually do.
>

Andrew,

Out of curiosity, how would these kinds of changes interact with gunicorn 
running with a `gevent` worker? I also currently use `psycogreen.gevent` 
for postgresql async behaviour.

Thanks,
James

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bfcaec8a-179d-4e9d-b72b-0fd07d7afbe0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[charettes]

AFAICT the performance hit is minimal unless you are doing something
slow when logging warnings (e.g. -Walways with a slow I/O py.warnings 
handler).

The Python/Django instrumentation simply adds a value to Model._state
and look it up on attribute accesses.

Cheers,
Simon

Le lundi 20 août 2018 19:34:11 UTC-4, Curtis Maloney a écrit :
>
> In general this sounds like a tremendously useful tool... I'm caused to 
> wonder, however... what, if any, are the performance impacts? 
>
> -- 
> Curtis 
>
>
>
> On 08/21/2018 08:10 AM, charettes wrote: 
> > Regarding the lazy loading of deferred fields and foreign keys 
> > I wanted to mention I've been working on a third-party application 
> > that allows overriding the default behavior[0]. 
> > 
> > The project works by tainting objects retrieved from "sealed" 
> > querysets and having fields descriptors lookup whether or not 
> > the object is "sealed" on attribute access and warn the developer 
> > about it if it's the case. Warnings can be elevated to errors 
> > using `filterwarnings` when deemed appropriated (e.g. CI, staging). 
> > 
> > It has been an useful tool to assist in figuring out where 
> > `select_related()` and `prefetch_related()` should be used 
> > to adjust complex projects database interactions. 
> > 
> > I assume a similar pattern could be used to mark objects retrieved 
> > from `QuerySet.__aiter__` to prevent non-async queries from being 
> > performed on attribute accesses; on `Model._state.async = True` 
> > field descriptors would error out. 
> > 
> > 
> > Cheers, 
> > Simon 
> > 
> > P.-S. 
> > 
> > While the project might look complex most of the code takes care 
> > of the delicate tasks of replacing fields descriptors once models 
> > are configured which could be significantly simplified if it was 
> > part of Django core. 
> > 
> > [0] https://github.com/charettes/django-seal 
> > Le samedi 9 juin 2018 02:30:59 UTC-4, Josh Smeaton a écrit : 
> > 
> > I think most of what you've laid out sounds great and that pursuing 
> > async Django is in the projects best interests. The sync to async 
> > and async to sync wrappers that have come out of channels give me 
> > much more confidence that this project is doable in a reasonable 
> > amount of time with backwards compatibility being preserved. 
> > 
> > The only real concern I have at the moment is around your comments 
> > regarding on demand foreign key traversal. If existing code running 
> > synchronously is going to be impacted, that's going to be very 
> > difficult for a lot of people. If it's only asynchronous traversal 
> > that'll have issues, then I have no real concern, as on demand 
> > foreign key fetching is usually a bug anyway. 
> > 
> > Having a brief read through the psycopg asynchronous docs[0], it 
> > looks like a number of features will be impossible or troublesome to 
> > use, like transactions, executemany, and named cursors (.iterator() 
> > with server side cursors). We'd also need to investigate how 
> > pgbouncer would work in async mode, as most large sites using 
> > postgres are also using pgbouncer. I would expect support can only 
> > further improve, especially if there is a driver like django 
> > pushing. Fallback would just be to run inside a thread pool though, 
> > so it's not a blocker for the rest of the proposal. 
> > 
> > Very exciting times ahead! 
> > 
> > [0] http://initd.org/psycopg/docs/advanced.html#asynchronous-support 
> >  
> > 
> > On Monday, 4 June 2018 23:18:23 UTC+10, Andrew Godwin wrote: 
> > 
> > Hello everyone, 
> > 
> > For a while now I have been working on potential plans for 
> > making Django async-capable, and I finally have a plan I am 
> > reasonably happy with and which I think we can actually do. 
> > 
> > This proposed roadmap, in its great length, is here: 
> > 
> > https://www.aeracode.org/2018/06/04/django-async-roadmap/ 
> >  
> > 
> > I'd like to invite discussion on this potential plan - 
> including: 
> > 
> >   - Do we think async is worth going after? Note that this is 
> > just async HTTP capability, not WebSockets (that would remain in 
> > Channels) 
> > 
> >   - Can we do this in a reasonable timeframe? If not, is there a 
> > way around that? 
> > 
> >   - Are the proposed modifications to how Django runs sensible? 
> > 
> >   - How should we fund this? 
> > 
> > There's many more potential questions, and I really would love 
> > feedback on this. I'm personally pretty convinced that we can 
> > and should do this, but this is a decision we cannot take 
> > lightly, and I would love to hear what you have to say. 
> > 
> > Andrew 

Re: A Django Async Roadmap

[Curtis Maloney]

In general this sounds like a tremendously useful tool... I'm caused to 
wonder, however... what, if any, are the performance impacts?


--
Curtis



On 08/21/2018 08:10 AM, charettes wrote:

Regarding the lazy loading of deferred fields and foreign keys
I wanted to mention I've been working on a third-party application
that allows overriding the default behavior[0].

The project works by tainting objects retrieved from "sealed"
querysets and having fields descriptors lookup whether or not
the object is "sealed" on attribute access and warn the developer
about it if it's the case. Warnings can be elevated to errors
using `filterwarnings` when deemed appropriated (e.g. CI, staging).

It has been an useful tool to assist in figuring out where
`select_related()` and `prefetch_related()` should be used
to adjust complex projects database interactions.

I assume a similar pattern could be used to mark objects retrieved
from `QuerySet.__aiter__` to prevent non-async queries from being
performed on attribute accesses; on `Model._state.async = True`
field descriptors would error out.


Cheers,
Simon

P.-S.

While the project might look complex most of the code takes care
of the delicate tasks of replacing fields descriptors once models
are configured which could be significantly simplified if it was
part of Django core.

[0] https://github.com/charettes/django-seal
Le samedi 9 juin 2018 02:30:59 UTC-4, Josh Smeaton a écrit :

I think most of what you've laid out sounds great and that pursuing
async Django is in the projects best interests. The sync to async
and async to sync wrappers that have come out of channels give me
much more confidence that this project is doable in a reasonable
amount of time with backwards compatibility being preserved.

The only real concern I have at the moment is around your comments
regarding on demand foreign key traversal. If existing code running
synchronously is going to be impacted, that's going to be very
difficult for a lot of people. If it's only asynchronous traversal
that'll have issues, then I have no real concern, as on demand
foreign key fetching is usually a bug anyway.

Having a brief read through the psycopg asynchronous docs[0], it
looks like a number of features will be impossible or troublesome to
use, like transactions, executemany, and named cursors (.iterator()
with server side cursors). We'd also need to investigate how
pgbouncer would work in async mode, as most large sites using
postgres are also using pgbouncer. I would expect support can only
further improve, especially if there is a driver like django
pushing. Fallback would just be to run inside a thread pool though,
so it's not a blocker for the rest of the proposal.

Very exciting times ahead!

[0] http://initd.org/psycopg/docs/advanced.html#asynchronous-support


On Monday, 4 June 2018 23:18:23 UTC+10, Andrew Godwin wrote:

Hello everyone,

For a while now I have been working on potential plans for
making Django async-capable, and I finally have a plan I am
reasonably happy with and which I think we can actually do.

This proposed roadmap, in its great length, is here:

https://www.aeracode.org/2018/06/04/django-async-roadmap/


I'd like to invite discussion on this potential plan - including:

  - Do we think async is worth going after? Note that this is
just async HTTP capability, not WebSockets (that would remain in
Channels)

  - Can we do this in a reasonable timeframe? If not, is there a
way around that?

  - Are the proposed modifications to how Django runs sensible?

  - How should we fund this?

There's many more potential questions, and I really would love
feedback on this. I'm personally pretty convinced that we can
and should do this, but this is a decision we cannot take
lightly, and I would love to hear what you have to say.

Andrew

--
You received this message because you are subscribed to the Google 
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-developers+unsubscr...@googlegroups.com 
.
To post to this group, send email to django-developers@googlegroups.com 
.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0b3b92b9-eee7-4909-82d6-c0138e8b5760%40googlegroups.com 

Re: A Django Async Roadmap

[charettes]

Regarding the lazy loading of deferred fields and foreign keys
I wanted to mention I've been working on a third-party application
that allows overriding the default behavior[0].

The project works by tainting objects retrieved from "sealed"
querysets and having fields descriptors lookup whether or not
the object is "sealed" on attribute access and warn the developer
about it if it's the case. Warnings can be elevated to errors
using `filterwarnings` when deemed appropriated (e.g. CI, staging).

It has been an useful tool to assist in figuring out where
`select_related()` and `prefetch_related()` should be used
to adjust complex projects database interactions.

I assume a similar pattern could be used to mark objects retrieved
from `QuerySet.__aiter__` to prevent non-async queries from being
performed on attribute accesses; on `Model._state.async = True`
field descriptors would error out.


Cheers,
Simon

P.-S.

While the project might look complex most of the code takes care
of the delicate tasks of replacing fields descriptors once models
are configured which could be significantly simplified if it was
part of Django core.

[0] https://github.com/charettes/django-seal
Le samedi 9 juin 2018 02:30:59 UTC-4, Josh Smeaton a écrit :
>
> I think most of what you've laid out sounds great and that pursuing async 
> Django is in the projects best interests. The sync to async and async to 
> sync wrappers that have come out of channels give me much more confidence 
> that this project is doable in a reasonable amount of time with backwards 
> compatibility being preserved.
>
> The only real concern I have at the moment is around your comments 
> regarding on demand foreign key traversal. If existing code running 
> synchronously is going to be impacted, that's going to be very difficult 
> for a lot of people. If it's only asynchronous traversal that'll have 
> issues, then I have no real concern, as on demand foreign key fetching is 
> usually a bug anyway.
>
> Having a brief read through the psycopg asynchronous docs[0], it looks 
> like a number of features will be impossible or troublesome to use, like 
> transactions, executemany, and named cursors (.iterator() with server side 
> cursors). We'd also need to investigate how pgbouncer would work in async 
> mode, as most large sites using postgres are also using pgbouncer. I would 
> expect support can only further improve, especially if there is a driver 
> like django pushing. Fallback would just be to run inside a thread pool 
> though, so it's not a blocker for the rest of the proposal.
>
> Very exciting times ahead!
>
> [0] http://initd.org/psycopg/docs/advanced.html#asynchronous-support
>
> On Monday, 4 June 2018 23:18:23 UTC+10, Andrew Godwin wrote:
>>
>> Hello everyone,
>>
>> For a while now I have been working on potential plans for making Django 
>> async-capable, and I finally have a plan I am reasonably happy with and 
>> which I think we can actually do.
>>
>> This proposed roadmap, in its great length, is here:
>>
>> https://www.aeracode.org/2018/06/04/django-async-roadmap/
>>
>> I'd like to invite discussion on this potential plan - including:
>>
>>  - Do we think async is worth going after? Note that this is just async 
>> HTTP capability, not WebSockets (that would remain in Channels)
>>
>>  - Can we do this in a reasonable timeframe? If not, is there a way 
>> around that?
>>
>>  - Are the proposed modifications to how Django runs sensible?
>>
>>  - How should we fund this?
>>
>> There's many more potential questions, and I really would love feedback 
>> on this. I'm personally pretty convinced that we can and should do this, 
>> but this is a decision we cannot take lightly, and I would love to hear 
>> what you have to say.
>>
>> Andrew
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0b3b92b9-eee7-4909-82d6-c0138e8b5760%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Emil Stenström]

I just wanted to add that I'm really excited for this change. 

On PyCon Sweden 2015 I held a "Why Django Sucks"-presentation where I 
talked about where the web is heading, and why django is lagging behind. 
The main point was lack of support for async. After the presentation lots 
of people came forward, and made clear that:

   1. They would really like async support in Django
   2. Django would NEVER be what I'd hope for. "That would need a separate 
   deamon process", "That would be too much work", "Nah, Django should just 
   leave the web stuff for Javascript and become an API". 

That Andrew is pushing the project in this direction, makes me more excited 
for Django than I've been in a long time. 

On Monday, 4 June 2018 15:18:23 UTC+2, Andrew Godwin wrote:
>
> Hello everyone,
>
> For a while now I have been working on potential plans for making Django 
> async-capable, and I finally have a plan I am reasonably happy with and 
> which I think we can actually do.
>
> This proposed roadmap, in its great length, is here:
>
> https://www.aeracode.org/2018/06/04/django-async-roadmap/
>
> I'd like to invite discussion on this potential plan - including:
>
>  - Do we think async is worth going after? Note that this is just async 
> HTTP capability, not WebSockets (that would remain in Channels)
>
>  - Can we do this in a reasonable timeframe? If not, is there a way around 
> that?
>
>  - Are the proposed modifications to how Django runs sensible?
>
>  - How should we fund this?
>
> There's many more potential questions, and I really would love feedback on 
> this. I'm personally pretty convinced that we can and should do this, but 
> this is a decision we cannot take lightly, and I would love to hear what 
> you have to say.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1a5bc51c-d67b-4ff6-9823-4e3fc2dd2715%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Andrew Godwin]

On Fri, Jun 8, 2018 at 11:31 PM Josh Smeaton  wrote:

>
> The only real concern I have at the moment is around your comments
> regarding on demand foreign key traversal. If existing code running
> synchronously is going to be impacted, that's going to be very difficult
> for a lot of people. If it's only asynchronous traversal that'll have
> issues, then I have no real concern, as on demand foreign key fetching is
> usually a bug anyway.
>

It would only be for asynchronous contexts - it would still work exactly as
today for synchronous ones. Backwards compatability is VERY important to me
during this whole thing.


>
> Having a brief read through the psycopg asynchronous docs[0], it looks
> like a number of features will be impossible or troublesome to use, like
> transactions, executemany, and named cursors (.iterator() with server side
> cursors). We'd also need to investigate how pgbouncer would work in async
> mode, as most large sites using postgres are also using pgbouncer. I would
> expect support can only further improve, especially if there is a driver
> like django pushing. Fallback would just be to run inside a thread pool
> though, so it's not a blocker for the rest of the proposal.
>
>
Yes, there's a lot of specifics that need to be worked out for each driver
(e.g. does it even make sense with SQLite at all?), but I think providing
that space and framework to experiment in is going to be very beneficial.
The aiomysql package implemented everything in pure Python, as well - it's
possible that similar would be needed for full async PostgreSQL support,
where the non-compiled speed reduction might be outweighed by the
parallelism improvements.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upvo5suu_7FQXi6cWDau1xK04NdQG2m-sbuX%3D26k1Xoew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Ran Benita]

> Loading a lazy ForeignKey/RelatedField attribute on a model instance

Also deferred fields.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANeXs%3D2ZdvCChgps-i0jqonFT5OZseK45vVtKsT8rTxDQX3dqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Josh Smeaton]

I think most of what you've laid out sounds great and that pursuing async 
Django is in the projects best interests. The sync to async and async to 
sync wrappers that have come out of channels give me much more confidence 
that this project is doable in a reasonable amount of time with backwards 
compatibility being preserved.

The only real concern I have at the moment is around your comments 
regarding on demand foreign key traversal. If existing code running 
synchronously is going to be impacted, that's going to be very difficult 
for a lot of people. If it's only asynchronous traversal that'll have 
issues, then I have no real concern, as on demand foreign key fetching is 
usually a bug anyway.

Having a brief read through the psycopg asynchronous docs[0], it looks like 
a number of features will be impossible or troublesome to use, like 
transactions, executemany, and named cursors (.iterator() with server side 
cursors). We'd also need to investigate how pgbouncer would work in async 
mode, as most large sites using postgres are also using pgbouncer. I would 
expect support can only further improve, especially if there is a driver 
like django pushing. Fallback would just be to run inside a thread pool 
though, so it's not a blocker for the rest of the proposal.

Very exciting times ahead!

[0] http://initd.org/psycopg/docs/advanced.html#asynchronous-support

On Monday, 4 June 2018 23:18:23 UTC+10, Andrew Godwin wrote:
>
> Hello everyone,
>
> For a while now I have been working on potential plans for making Django 
> async-capable, and I finally have a plan I am reasonably happy with and 
> which I think we can actually do.
>
> This proposed roadmap, in its great length, is here:
>
> https://www.aeracode.org/2018/06/04/django-async-roadmap/
>
> I'd like to invite discussion on this potential plan - including:
>
>  - Do we think async is worth going after? Note that this is just async 
> HTTP capability, not WebSockets (that would remain in Channels)
>
>  - Can we do this in a reasonable timeframe? If not, is there a way around 
> that?
>
>  - Are the proposed modifications to how Django runs sensible?
>
>  - How should we fund this?
>
> There's many more potential questions, and I really would love feedback on 
> this. I'm personally pretty convinced that we can and should do this, but 
> this is a decision we cannot take lightly, and I would love to hear what 
> you have to say.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d3b3c04e-9db1-4a41-af3a-ad50209a07e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Andrew Godwin]

Chaim: For references to get up to speed on async concepts, there's not a
lot of great beginner-level stuff out there. I have a practical guide to
async functionality at
https://www.aeracode.org/2018/02/19/python-async-simplified/, but it's not
great at going into detail on Web stuff and why it has big advantages. I'd
love if other people could chime in with resources that helped them (rather
than me, who just started writing async code for fun 8 years ago and have
sort of kept up with things that way)

Tom F:

>  I’m just spitballing here but could we not conditionally return a
future/promise if some conditions are met, then implement some kind of
chaining based on the attribute accesses?

We could, but as you can see from the length of the paragraph you've
written this is incredibly _tricky_ to do correctly, and I'm not a fan of
lazy objects in the first place. Something we should investigate maybe, but
ultimately I'd prefer a simpler solution with less magic.

>  If we go with a `_async` suffix we could reduce a lot of boilerplate
with a bit of magic

Sure, though honestly I wouldn't say it needs magic, just:

def function_async(self, foo):
   ...

function = async_to_sync(function_async)

Tom C:

> Supporting an ASGI/asyncio request-response cycle in 2.2 might be a
perfectly achievable goal with or without funding. Users wouldn't be able
to  use the ORM, or much of the default middleware, but for some
gateway-type services I can still see it being useful.

Indeed. The time I personally spend each week on Channels (around 2 days
fulltime-equivalent) could easily be partially or wholly diverted into this
project, as Channels is pretty stable right now.

That said, my time for that project has been recompensed by the Mozilla OSS
grant they gave us a while back up until now - the funds for that are
finally drying out, and while I could do the work for free, I take a paycut
from my normal work to get the time that I use to work on Django, so
funding would be nice to have - and I don't expect anyone else to work for
free. I suspect the funding level needed would not be too great for the
initial work.

Given we have some MOSS funds left, I may use the last of those to start
getting an async pathway ready for 2.2 once this whole thing gets approved.
It's still within the scope of the original grant.

Jordan:

I'm not sure how we're going to develop this yet, but it'll likely be a
branch in the main Django repo once this proposal is approved (there's
still a little while before that happens - there needs to be consensus here
and/or a technical board vote).

Andrew


On Tue, Jun 5, 2018 at 2:21 PM C. Kirby  wrote:

> Andrew,
>
> I don't know enough about async web to comment directly on your proposal,
> but like most of your proposals it looks well thought out and
> comprehensive. Kudos to you.
>
> As you state, the scope of this project is one that can raise the profile
> of developers in the community - that appeals to me in ways that bug fixing
> has not, mostly because I have not encountered bugs in django.
>
> Towards that end, can you point to any of your preferred references to get
> up to speed on async concepts to help understand the scope and scale of
> this undertaking? Even if I don't work on the project I think intuitively
> that I could make good use of it.
>
> Thanks,
> Chaim
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/7a958407-7dcf-4fd8-b09f-67e28346cb65%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upLVX6hsH_tgooRUgh9Mtvn%2Bqcn9_9_S1pKOAQ44S-PyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Jordan Eremieff]

The proposal looks great, thanks for putting it together and pushing async 
in Django forward.

I'm not sure the best approach to take here, but I've just started 
experimenting on a fork: 
https://github.com/erm/django/tree/async-experiment-2. I am unsure how much 
of the more complex items I may be able to find solutions for, but it seems 
like a good first step is to get basic ASGI support running alongside the 
current behavior and then bring in the middleware from that point. Open to 
any suggestions for other directions to go in, otherwise I'll just be 
digging away at it this way for now.

On Monday, June 4, 2018 at 11:18:23 PM UTC+10, Andrew Godwin wrote:
>
> Hello everyone,
>
> For a while now I have been working on potential plans for making Django 
> async-capable, and I finally have a plan I am reasonably happy with and 
> which I think we can actually do.
>
> This proposed roadmap, in its great length, is here:
>
> https://www.aeracode.org/2018/06/04/django-async-roadmap/
>
> I'd like to invite discussion on this potential plan - including:
>
>  - Do we think async is worth going after? Note that this is just async 
> HTTP capability, not WebSockets (that would remain in Channels)
>
>  - Can we do this in a reasonable timeframe? If not, is there a way around 
> that?
>
>  - Are the proposed modifications to how Django runs sensible?
>
>  - How should we fund this?
>
> There's many more potential questions, and I really would love feedback on 
> this. I'm personally pretty convinced that we can and should do this, but 
> this is a decision we cannot take lightly, and I would love to hear what 
> you have to say.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/58c0ee80-7dbe-4d91-9bf8-17a6e28556a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Tom Christie]

I wonder if a sensible first tack onto this might be:

Supporting an ASGI/asyncio request-response cycle in 2.2 might be a 
perfectly achievable goal with or without funding. Users wouldn't be able 
to  use the ORM, or much of the default middleware, but for some 
gateway-type services I can still see it being useful.

A proof of concept there might help validate (or otherwise) both the 
motivation (run some actually tests against a service that makes outgoing 
HTTP calls and then returns a response - what sort of throughput can we 
achieve there vs. the threaded case?) and the feasibility (what aspects 
that we haven't considered do we bump into along the way to 
implementation?).

Perhaps we could work on the basis of (1) pulling together a proof of 
concept there, and then (2) attempting to get that into the 2.2 alpha 
freeze, without being blocked on getting together the rest of the plan & 
funding for the more substantial parts such as the ORM work?

That wouldn't necessarily preclude also getting asnycio ORM work into 2.2, 
or achieving funding early on, but would at least give us something 
valuable to start working towards immediately?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b994f620-cfb9-4e4c-aac7-c601be809b5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Tom Forbes]

> Unfortunately even if you have context variables, you simply can't await
inside of an attribute reference because there's a synchronous call in your
stack. I even chatted to some Python core devs at PyCon US about this and
we couldn't really think of a way out of this problem without some very
serious changes to the language.

I’m just spitballing here but could we not conditionally return a
future/promise if some conditions are met, then implement some kind of
chaining based on the attribute accesses? So
`instance.related_field.other_related_field` breaks down into the
`related_field` returning some kind of awaitable instance, and we return a
nested instance when `other_related_field` is accessed? This last future
would resolve to ’the right thing’. Perhaps just throwing an exception if
we are in an async loop is an acceptable and simpler trade off however just
as long as the existing synchronous implementation works as-is.

>  keyword arguments (like "my_method(..., async=True)"), suffixes
("my_method_async()", as you suggest), automatically changing based on if
you're in an async thread (which is a bit... magic) or different module
namespaces ("from django.db.async import foo”).

If we go with a `_async` suffix we could reduce a lot of boilerplate with a
bit of magic. We could potentially auto-generate the sync versions of the
methods with a decorator, seeing as they would just call the async version
and wait for the result? I’m not sure if we could do the same nicely with a
different namespace, but that might be cleaner.

On 5 June 2018 at 07:36:49, Andrew Godwin (and...@aeracode.org) wrote:


> I think getting rid of the related field references could be a big issue
> here and cause a lot of headaches for existing applications. Could we do
> this in a backwards compatible way at all? I wonder if PEP 567
>  could help here, could we
> define some kind of ‘Django async context’ and do different things
> depending on if this is true or false?
>
Unfortunately even if you have context variables, you simply can't await
inside of an attribute reference because there's a synchronous call in your
stack. I even chatted to some Python core devs at PyCon US about this and
we couldn't really think of a way out of this problem without some very
serious changes to the language.

What will work, though, is attribute access from a sync context - the
related field references will only error out of they know they're in an
async loop, and we can detect that by looking for an active event loop on
the current thread (no context varaible needed).


> Regarding exposing an async interface alongside a synchronous one: are you
> envisaging something like appending _async to methods or having some kind
> of wrapper class that could be optionally included to go from async->sync?
> I guess it would have to be appending _async, as a wrapper class could be
> used in different contexts.
>

I'm not quite sure on this one. For backwards compatibility, we have to
keep sync methods working with the same name, but there are several options
as to how to separate them - keyword arguments (like "my_method(...,
async=True)"), suffixes ("my_method_async()", as you suggest),
automatically changing based on if you're in an async thread (which is a
bit... magic) or different module namespaces ("from django.db.async import
foo").

I want to sketch out what all of these look like as part of this project
and then work out which is best for Django.


> Async templates seem particularly powerful if we work out the details. We
> could eventually render different parts of the template concurrently, i.e
> example each iteration of a for loop could be it’s own future resolved
> independently, but this is likely a pipe dream.
>

That would very much be a long-term thing, and honestly something I might
consider handing off to something like Jinja (which already has full async
support, I learnt yesterday)

Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAFwN1upPVQCjHTxgJmzJSUzaevsfQ7evwi89hFt4OvHmMe5m2A%40mail.gmail.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: A Django Async Roadmap

[Mattia Procopio]

I'm really excited that the django community is starting to think of 
async-friendly django. I think this is needed for the community and will be 
a strenght point for the framework itself for next couple of years if this 
will take place.
Now that asyncio is a thing in the Python world I think more and more devs 
will try to use it, I personally tried to use it within django a couple of 
times and I ended up hacking lot of things,I really would like to have an 
available async loop
that I can use to run my coroutines and do other stuff. These are just 
hacks of course but I think that if django won't be more async friendly 
lots of people will end write their own hacks ending with a mess, while if
async will be baked into it there will be no need of custom async-like 
code. Adding initial support for coroutine views will be a very cool and 
interesting starting point for experiments.
Really hope this plan will become real 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/798268a9-1fb5-47ae-b8d3-c9658f7b9692%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[C. Kirby]

Andrew,

I don't know enough about async web to comment directly on your proposal, 
but like most of your proposals it looks well thought out and 
comprehensive. Kudos to you.

As you state, the scope of this project is one that can raise the profile 
of developers in the community - that appeals to me in ways that bug fixing 
has not, mostly because I have not encountered bugs in django.

Towards that end, can you point to any of your preferred references to get 
up to speed on async concepts to help understand the scope and scale of 
this undertaking? Even if I don't work on the project I think intuitively 
that I could make good use of it.

Thanks,
Chaim

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7a958407-7dcf-4fd8-b09f-67e28346cb65%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Andrew Godwin]

>
>
> I think getting rid of the related field references could be a big issue
> here and cause a lot of headaches for existing applications. Could we do
> this in a backwards compatible way at all? I wonder if PEP 567
>  could help here, could we
> define some kind of ‘Django async context’ and do different things
> depending on if this is true or false?
>
Unfortunately even if you have context variables, you simply can't await
inside of an attribute reference because there's a synchronous call in your
stack. I even chatted to some Python core devs at PyCon US about this and
we couldn't really think of a way out of this problem without some very
serious changes to the language.

What will work, though, is attribute access from a sync context - the
related field references will only error out of they know they're in an
async loop, and we can detect that by looking for an active event loop on
the current thread (no context varaible needed).


> Regarding exposing an async interface alongside a synchronous one: are you
> envisaging something like appending _async to methods or having some kind
> of wrapper class that could be optionally included to go from async->sync?
> I guess it would have to be appending _async, as a wrapper class could be
> used in different contexts.
>

I'm not quite sure on this one. For backwards compatibility, we have to
keep sync methods working with the same name, but there are several options
as to how to separate them - keyword arguments (like "my_method(...,
async=True)"), suffixes ("my_method_async()", as you suggest),
automatically changing based on if you're in an async thread (which is a
bit... magic) or different module namespaces ("from django.db.async import
foo").

I want to sketch out what all of these look like as part of this project
and then work out which is best for Django.


> Async templates seem particularly powerful if we work out the details. We
> could eventually render different parts of the template concurrently, i.e
> example each iteration of a for loop could be it’s own future resolved
> independently, but this is likely a pipe dream.
>

That would very much be a long-term thing, and honestly something I might
consider handing off to something like Jinja (which already has full async
support, I learnt yesterday)

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upPVQCjHTxgJmzJSUzaevsfQ7evwi89hFt4OvHmMe5m2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Curtis Maloney]

On 06/05/2018 07:54 AM, Tom Forbes wrote:

Are the proposed modifications to how Django runs sensible?


I had a few random thoughts while reading your proposal:

I think getting rid of the related field references could be a big issue 
here and cause a lot of headaches for existing applications. Could we do 
this in a backwards compatible way at all? I wonder if PEP 567 
 could help here, could we 
define some kind of ‘Django async context’ and do different things 
depending on if this is true or false?


One thought about the related field stuff... whilst having it makes it 
easy to get started, any site at scale will sensibly obviate their need 
in almost all cases by using select_related, for performance reasons.


--
C

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4447805e-f035-8242-957e-ffb1e0c57d51%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Tom Forbes]

Hey Andrew, thank you for the very exciting proposal and the ongoing work
on Django channels! There is a lot to consider here and some very
interesting problems to work through - I’d love to help out wherever I can.

Do we think async is worth going after?

I think this is very much worth doing. Async interfaces to the ORM and
templates would be a huge, huge improvement if we can get it right. I think
Tom Christie summed it up really well however I would add that I think
teams excluding Django as a contender may happen sooner rather than later
if it’s not happening already.

Are the proposed modifications to how Django runs sensible?

I had a few random thoughts while reading your proposal:

I think getting rid of the related field references could be a big issue
here and cause a lot of headaches for existing applications. Could we do
this in a backwards compatible way at all? I wonder if PEP 567
 could help here, could we
define some kind of ‘Django async context’ and do different things
depending on if this is true or false?

Regarding exposing an async interface alongside a synchronous one: are you
envisaging something like appending _async to methods or having some kind
of wrapper class that could be optionally included to go from async->sync?
I guess it would have to be appending _async, as a wrapper class could be
used in different contexts.

Async templates seem particularly powerful if we work out the details. We
could eventually render different parts of the template concurrently, i.e
example each iteration of a for loop could be it’s own future resolved
independently, but this is likely a pipe dream.




On 4 June 2018 at 14:18:19, Andrew Godwin (and...@aeracode.org) wrote:

Hello everyone,

For a while now I have been working on potential plans for making Django
async-capable, and I finally have a plan I am reasonably happy with and
which I think we can actually do.

This proposed roadmap, in its great length, is here:

https://www.aeracode.org/2018/06/04/django-async-roadmap/

I'd like to invite discussion on this potential plan - including:

 - Do we think async is worth going after? Note that this is just async
HTTP capability, not WebSockets (that would remain in Channels)

 - Can we do this in a reasonable timeframe? If not, is there a way around
that?

 - Are the proposed modifications to how Django runs sensible?

 - How should we fund this?

There's many more potential questions, and I really would love feedback on
this. I'm personally pretty convinced that we can and should do this, but
this is a decision we cannot take lightly, and I would love to hear what
you have to say.

Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAFwN1up4%2BP%2B4bCSSiG6t845idoM8dvktnuTs4dH2R5aAuCz3iA%40mail.gmail.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMTTXPK5vjE7uKm8nKcZJOJTSyWA3xGVtxaGEtrQuUkfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Tom Christie]

> Do we think async is worth going after?

I think this is hugely important. Django's advantage in the field currently 
is in it's developer productivity and maintainability. As frameworks based 
on Node or on languages with co-operative concurrency built-in continue to 
mature it's going to face more of a squeeze there, while being less 
resource efficient than many alternatives.

Although performance characteristics are often over-stated or over-valued, 
it's still a substantive point against a team choosing Django in some 
circumstances. It's particularly relevant in super successful high-scale 
cases, and I want Django to continue to be a great choice both for single 
person building out an MVP on a couple of instances, all the way up to 
large teams working on massive systems, with many running services.

If we don't attempt to tackle this I think we may end up very slowly 
starting to write Django out as a contender for some teams.

> Can we do this in a reasonable timeframe? If not, is there a way around 
that?

I can't speak to the ORM work, as I don't know the internals well enough, 
but I think how everything else is proposed makes sense. The minimal first 
step, of tackling the request/response path is already potentially valuable 
for any teams that include an API gateway as one of their services - being 
able to have a Django-based app that can make non-blocking requests out to 
other services would immediately benefit a decent chunk of real-world use 
cases.

> How should we fund this?

It's a pretty serious amount of funding you'd want to see, so I guess 
ideally not as a single backer.
A joint combination of Mozilla's MOSS program, PSF grant, and corporate 
sponsorships all orchestrated together? (Waves hands magically)
Wrt. corporate sponsorships I think we might well want to set realistic 
expectations about how much developer time money actually buys, and a set a 
decently high minimum tier. What I think works especially well is tying in 
contributions, so. eg. an agreed-in-principle grant that's conditional on 
also achieving a certain amount of corporate sponsorships might be a great 
motivator for companies to make a shared, collaborative investment. ??

Thanks Andrew - *really* exciting proposal.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/72d08b8d-06e8-4447-8017-9f3629b43656%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Rigel]

Hello Andrew.

I like your proposal and want to help out. In particular, what I'm really
fond of is the approach of making async available only if you want it,
while keeping Django backwards-compatible. Django is brilliant in that,
like the Python language, it's easy and fun to pick and learn, but flexible
and powerful at the same time.

Rigel.


On Mon, Jun 4, 2018 at 3:17 PM, Andrew Godwin  wrote:

> Hello everyone,
>
> For a while now I have been working on potential plans for making Django
> async-capable, and I finally have a plan I am reasonably happy with and
> which I think we can actually do.
>
> This proposed roadmap, in its great length, is here:
>
> https://www.aeracode.org/2018/06/04/django-async-roadmap/
>
> I'd like to invite discussion on this potential plan - including:
>
>  - Do we think async is worth going after? Note that this is just async
> HTTP capability, not WebSockets (that would remain in Channels)
>
>  - Can we do this in a reasonable timeframe? If not, is there a way around
> that?
>
>  - Are the proposed modifications to how Django runs sensible?
>
>  - How should we fund this?
>
> There's many more potential questions, and I really would love feedback on
> this. I'm personally pretty convinced that we can and should do this, but
> this is a decision we cannot take lightly, and I would love to hear what
> you have to say.
>
> Andrew
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/CAFwN1up4%2BP%2B4bCSSiG6t845idoM8dvktnuTs4dH
> 2R5aAuCz3iA%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAD9P0JrY%3DwdtPsk9-xLhALM_4Y4ZK2uSaT6MEjKt7Nd2-DDjUw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.