Re: Channels integration plan, first draft

[Anssi Kääriäinen]

I have a gut feeling this isn't going to work that well. The reasons
include:
  - Backwards compatibility: how is a large site going to upgrade from 1.9
to 1.10?
  - Complexity of setup.
  - Error conditions: for example, what happens when an interface server
sends a request to worker, and then dies (that is, the response channel has
zero listeners). Similarly for chunked messages.
  - Does the architecture really scale enough? The channel backend is going
to be a bottleneck, it needs the ability to handle a huge amount of data
and a huge amount of individual messages. In particular, the request
channel is going to be contested. We know classic http scales, but is the
same true for interface server architecture?
  - Performance. Each request and response needs two additional network
roundtrips. One to save to the channel server, one to fetch from the
channel server. If the messages are large, this adds a lot of latency.
  - Untested architecture: does any big site use this kind of architecture
for all http handling?

A realistic test for this is to push a scalable amount of scalable sized
requests through the stack. The stack should recover even if you shut down
parts of the network, any single interface server, channel backend server
or worker server. Of course, when a server or a part of the network
recovers, the stack would need to recover from that. Compare the
performance, simplicity of setup and ability to recover from error
conditions to a setup with only classic Django http servers.

I'm sorry if this feels negative. But, you are planning to change the very
core of what a Django server is, and I feel we need to know with certainty
that the new architecture really works. And not only that, it needs to be
at least as good as classic http handling for existing users.

 - Anssi

On Thursday, December 17, 2015, Andrew Godwin  wrote:

> Yes, that is the idea. While it obviously adds overhead (a millisecond or
> two in my first tests), it also adds natural load balancing between workers
> and then lets us have the same architecture for websockets and normal HTTP.
>
> (The interface server does do all the HTTP parsing, so what gets sent over
> is slightly less verbose than normal HTTP and needs less work to use, but
> it's not a big saving)
>
> Andrew
>
> On Thu, Dec 17, 2015 at 9:01 PM, Anssi Kääriäinen  > wrote:
>
>> Is the idea a large site using classic request-response architecture
>> would get the requests at interface servers, these would then push the HTTP
>> requests through channels to worker processes, which process the message
>> and push the response through the channel backend back to the interface
>> server and from there back to the client?
>>
>>  - Anssi
>>
>> On Thursday, December 17, 2015, Andrew Godwin > > wrote:
>>
>>> To address the points so far:
>>>
>>>  - I'm not yet sure whether "traditional" WSGI mode would actually run
>>> through the in memory backend or just be plugged in directly to the
>>> existing code path; it really depends on how much code would need to be
>>> moved around in either case. I'm pretty keen on keeping a raw-WSGI path
>>> around for performance/compatability reasons, and so we can hard fail if
>>> you try *any* channels use (right now the failure mode for trying to use
>>> channels with the wsgi emulation is silent failure)
>>>
>>> - Streaming HTTP responses are already in the channels spec as chunked
>>> messages; you just keep sending response-style messages with a flag saying
>>> "there's more".
>>>
>>> - File uploads are more difficult, due to the nature of the worker model
>>> (you can't guarantee all the messages will go to the same worker). My
>>> current plan here is to revise the message spec to allow infinite size
>>> messages and make the channel backend handle chunking in the best way
>>> (write to shared disk, use lots of keys, etc), but if there are other
>>> suggestions I'm open. This would also let people return large http
>>> responses without having to worry about size limits.
>>>
>>> - Alternative serialisation formats will be looked into; it's up to the
>>> channel backend what to use, I just chose JSON as our previous research
>>> into this at work showed that it was actually the fastest overall due to
>>> the fact it has a pure C implementation, but that's a year or two old.
>>> Whatever is chosen needs large support and forwards compatability, however.
>>> The message format is deliberately specified as JSON-capable structures
>>> (dicts, lists, strings) as it's assumed any serialisation format can handle
>>> this, and so it can be portable across backends.
>>>
>>> - I thought SCRIPT_NAME was basically unused by anyone these days, but
>>> hey, happy to be proved wrong. Do we have any usage numbers on it to know
>>> if we'd need it for a new standalone server to 

Re: Decoupling the ORM

[Curtis Maloney]

I've identified one new "issue".
There is an implicit assumption that primary keys are useful for
ordering by the current QuerySet API methods `.first()` and `.last()`.


I believe the case here is that first and last are meaningless without 
an ordering, so in lieu of any programmer supplied ordering, a 
"fallback" consistent ordering of 'pk' is used.


Remember the DBMS is under no obligation to return a query in consistent 
order without an ORDER BY clause.


So, without an ordering, two successive calls to first() [or last()] 
may, in fact, return different records even without modifications to the 
table.


It's not expected people should _rely_ on the ordering of PK, and, 
indeed, it's frequently recommended against inferring any meaning from 
PK values [sqlite, IIRC, assigns them pseudo-ramdomly]


That said, the assumption that a PK is sortable and will provide a 
deterministic ordering shouldn't be much to ask, surely?


--
Curtis

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/567353FA.6030105%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Re: Channels integration plan, first draft

[Andrew Godwin]

The problem is that even with a specially named channel you can't guarantee
the messages will all go to the same worker so you can re-assemble them on
local disk, which kind of makes it still a hard problem. It works for
responses because there's only one interface server listening to any
response channel.

The current plan I think is best is to change the message spec to allow for
any size message, and then have the channel backend handle chunking itself
(then, the backend can do what is best for the way it's designed, and
stream bigger messages to disk before presenting them to the client).

It does mean going a bit beyond the normal dictionary interface for a
message to allow a way to stream the contents of a key rather than just
read it, but I think that would be easy-ish to do.

Andrew

On Thu, Dec 17, 2015 at 11:13 PM, Sam Willis  wrote:

> Hi,
>
> To support file uploads or a large message body the http.request message
> could have an file_channel or body_channel (much like its reply_channel)?
> These would be something like http.request.file.Dj3Hd9J and would stream
> chunked file or body content in the same way as the http.response message
> with a more_content attribute.
>
> Sam
>
> On Thursday, December 17, 2015 at 9:48:31 PM UTC, Andrew Godwin wrote:
>>
>> Yes, that is the idea. While it obviously adds overhead (a millisecond or
>> two in my first tests), it also adds natural load balancing between workers
>> and then lets us have the same architecture for websockets and normal HTTP.
>>
>> (The interface server does do all the HTTP parsing, so what gets sent
>> over is slightly less verbose than normal HTTP and needs less work to use,
>> but it's not a big saving)
>>
>> Andrew
>>
>> On Thu, Dec 17, 2015 at 9:01 PM, Anssi Kääriäinen 
>> wrote:
>>
>>> Is the idea a large site using classic request-response architecture
>>> would get the requests at interface servers, these would then push the HTTP
>>> requests through channels to worker processes, which process the message
>>> and push the response through the channel backend back to the interface
>>> server and from there back to the client?
>>>
>>>  - Anssi
>>>
>>> On Thursday, December 17, 2015, Andrew Godwin 
>>> wrote:
>>>
 To address the points so far:

  - I'm not yet sure whether "traditional" WSGI mode would actually run
 through the in memory backend or just be plugged in directly to the
 existing code path; it really depends on how much code would need to be
 moved around in either case. I'm pretty keen on keeping a raw-WSGI path
 around for performance/compatability reasons, and so we can hard fail if
 you try *any* channels use (right now the failure mode for trying to use
 channels with the wsgi emulation is silent failure)

 - Streaming HTTP responses are already in the channels spec as chunked
 messages; you just keep sending response-style messages with a flag saying
 "there's more".

 - File uploads are more difficult, due to the nature of the worker
 model (you can't guarantee all the messages will go to the same worker). My
 current plan here is to revise the message spec to allow infinite size
 messages and make the channel backend handle chunking in the best way
 (write to shared disk, use lots of keys, etc), but if there are other
 suggestions I'm open. This would also let people return large http
 responses without having to worry about size limits.

 - Alternative serialisation formats will be looked into; it's up to the
 channel backend what to use, I just chose JSON as our previous research
 into this at work showed that it was actually the fastest overall due to
 the fact it has a pure C implementation, but that's a year or two old.
 Whatever is chosen needs large support and forwards compatability, however.
 The message format is deliberately specified as JSON-capable structures
 (dicts, lists, strings) as it's assumed any serialisation format can handle
 this, and so it can be portable across backends.

 - I thought SCRIPT_NAME was basically unused by anyone these days, but
 hey, happy to be proved wrong. Do we have any usage numbers on it to know
 if we'd need it for a new standalone server to implement? It's really not
 hard to add it into the request format, just thought it was one of those
 CGI remnants we might finally be able to kill.

 Andrew

 On Thu, Dec 17, 2015 at 6:32 PM, Anssi Kääriäinen 
 wrote:

> On Thursday, December 17, 2015, Carl Meyer  wrote:
>
>> Hi Andrew,
>
>
>> - I share Mark's concern about the performance (latency, specifically)
>> implications for projects that want to keep deploying old-style, given
>> all the new serialization that would now be in the request path. I
>> think
>> some 

Re: Channels integration plan, first draft

[Sam Willis]

Hi,

To support file uploads or a large message body the http.request message 
could have an file_channel or body_channel (much like its reply_channel)? 
These would be something like http.request.file.Dj3Hd9J and would stream 
chunked file or body content in the same way as the http.response message 
with a more_content attribute.

Sam

On Thursday, December 17, 2015 at 9:48:31 PM UTC, Andrew Godwin wrote:
>
> Yes, that is the idea. While it obviously adds overhead (a millisecond or 
> two in my first tests), it also adds natural load balancing between workers 
> and then lets us have the same architecture for websockets and normal HTTP.
>
> (The interface server does do all the HTTP parsing, so what gets sent over 
> is slightly less verbose than normal HTTP and needs less work to use, but 
> it's not a big saving)
>
> Andrew
>
> On Thu, Dec 17, 2015 at 9:01 PM, Anssi Kääriäinen  > wrote:
>
>> Is the idea a large site using classic request-response architecture 
>> would get the requests at interface servers, these would then push the HTTP 
>> requests through channels to worker processes, which process the message 
>> and push the response through the channel backend back to the interface 
>> server and from there back to the client?
>>
>>  - Anssi
>>
>> On Thursday, December 17, 2015, Andrew Godwin > > wrote:
>>
>>> To address the points so far:
>>>
>>>  - I'm not yet sure whether "traditional" WSGI mode would actually run 
>>> through the in memory backend or just be plugged in directly to the 
>>> existing code path; it really depends on how much code would need to be 
>>> moved around in either case. I'm pretty keen on keeping a raw-WSGI path 
>>> around for performance/compatability reasons, and so we can hard fail if 
>>> you try *any* channels use (right now the failure mode for trying to use 
>>> channels with the wsgi emulation is silent failure)
>>>
>>> - Streaming HTTP responses are already in the channels spec as chunked 
>>> messages; you just keep sending response-style messages with a flag saying 
>>> "there's more".
>>>
>>> - File uploads are more difficult, due to the nature of the worker model 
>>> (you can't guarantee all the messages will go to the same worker). My 
>>> current plan here is to revise the message spec to allow infinite size 
>>> messages and make the channel backend handle chunking in the best way 
>>> (write to shared disk, use lots of keys, etc), but if there are other 
>>> suggestions I'm open. This would also let people return large http 
>>> responses without having to worry about size limits.
>>>
>>> - Alternative serialisation formats will be looked into; it's up to the 
>>> channel backend what to use, I just chose JSON as our previous research 
>>> into this at work showed that it was actually the fastest overall due to 
>>> the fact it has a pure C implementation, but that's a year or two old. 
>>> Whatever is chosen needs large support and forwards compatability, however. 
>>> The message format is deliberately specified as JSON-capable structures 
>>> (dicts, lists, strings) as it's assumed any serialisation format can handle 
>>> this, and so it can be portable across backends.
>>>
>>> - I thought SCRIPT_NAME was basically unused by anyone these days, but 
>>> hey, happy to be proved wrong. Do we have any usage numbers on it to know 
>>> if we'd need it for a new standalone server to implement? It's really not 
>>> hard to add it into the request format, just thought it was one of those 
>>> CGI remnants we might finally be able to kill.
>>>
>>> Andrew
>>>
>>> On Thu, Dec 17, 2015 at 6:32 PM, Anssi Kääriäinen  
>>> wrote:
>>>
 On Thursday, December 17, 2015, Carl Meyer  wrote:

> Hi Andrew,


> - I share Mark's concern about the performance (latency, specifically)
> implications for projects that want to keep deploying old-style, given
> all the new serialization that would now be in the request path. I 
> think
> some further discussion of this, with real benchmark numbers to refer
> to, is a prerequisite to considering Channels as a candidate for Django
> 1.10. To take a parallel from Python, Guido has always said that he
> won't consider removing the GIL unless it can be done without 
> penalizing
> single-threaded code. If you think a different approach makes sense 
> here
> (that is, that it's OK to penalize the simple cases in order to
> facilitate the less-simple ones), can you explain your reasons for that
> position?
>

 We would also need some form of streamed messages for streamed http 
 responses.

 Is it possible to handle old-style http the way it has always been 
 handled?

  - Anssi 

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Django developers (Contributions to Django itself)" group.
 To 

Adding name to fieldsets to improve their validation error text

[Benjamin W Stookey]

Right now, if min or max validation doesn't pass on a fieldset, we provide 
something 
like the following error 

:

Please submit 1 or more forms.


I'd like to propose that we provide a way to generate a slightly more 
specific error. For example:

Please submit 1 or more authors.


I'm eager to write the code to do this. There are a number of ways to 
implement it so I figured it would need discussion first and a ticket 
before I could issue a pull request. If people think this is a good idea, 
I'll document a few potential approaches for discussion.

A few options that come to mind before diving in.

   - Adding a `verbose_name_plural` property to the formset
   - Pulling the `verbose_name_plural` from the form class (if forms are 
   model forms)

Thanks,
-Ben

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c4e1f78a-32ab-4a35-b4d2-4c0aa7af47e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels integration plan, first draft

[Andrew Godwin]

Yes, that is the idea. While it obviously adds overhead (a millisecond or
two in my first tests), it also adds natural load balancing between workers
and then lets us have the same architecture for websockets and normal HTTP.

(The interface server does do all the HTTP parsing, so what gets sent over
is slightly less verbose than normal HTTP and needs less work to use, but
it's not a big saving)

Andrew

On Thu, Dec 17, 2015 at 9:01 PM, Anssi Kääriäinen 
wrote:

> Is the idea a large site using classic request-response architecture would
> get the requests at interface servers, these would then push the HTTP
> requests through channels to worker processes, which process the message
> and push the response through the channel backend back to the interface
> server and from there back to the client?
>
>  - Anssi
>
> On Thursday, December 17, 2015, Andrew Godwin  wrote:
>
>> To address the points so far:
>>
>>  - I'm not yet sure whether "traditional" WSGI mode would actually run
>> through the in memory backend or just be plugged in directly to the
>> existing code path; it really depends on how much code would need to be
>> moved around in either case. I'm pretty keen on keeping a raw-WSGI path
>> around for performance/compatability reasons, and so we can hard fail if
>> you try *any* channels use (right now the failure mode for trying to use
>> channels with the wsgi emulation is silent failure)
>>
>> - Streaming HTTP responses are already in the channels spec as chunked
>> messages; you just keep sending response-style messages with a flag saying
>> "there's more".
>>
>> - File uploads are more difficult, due to the nature of the worker model
>> (you can't guarantee all the messages will go to the same worker). My
>> current plan here is to revise the message spec to allow infinite size
>> messages and make the channel backend handle chunking in the best way
>> (write to shared disk, use lots of keys, etc), but if there are other
>> suggestions I'm open. This would also let people return large http
>> responses without having to worry about size limits.
>>
>> - Alternative serialisation formats will be looked into; it's up to the
>> channel backend what to use, I just chose JSON as our previous research
>> into this at work showed that it was actually the fastest overall due to
>> the fact it has a pure C implementation, but that's a year or two old.
>> Whatever is chosen needs large support and forwards compatability, however.
>> The message format is deliberately specified as JSON-capable structures
>> (dicts, lists, strings) as it's assumed any serialisation format can handle
>> this, and so it can be portable across backends.
>>
>> - I thought SCRIPT_NAME was basically unused by anyone these days, but
>> hey, happy to be proved wrong. Do we have any usage numbers on it to know
>> if we'd need it for a new standalone server to implement? It's really not
>> hard to add it into the request format, just thought it was one of those
>> CGI remnants we might finally be able to kill.
>>
>> Andrew
>>
>> On Thu, Dec 17, 2015 at 6:32 PM, Anssi Kääriäinen 
>> wrote:
>>
>>> On Thursday, December 17, 2015, Carl Meyer  wrote:
>>>
 Hi Andrew,
>>>
>>>
 - I share Mark's concern about the performance (latency, specifically)
 implications for projects that want to keep deploying old-style, given
 all the new serialization that would now be in the request path. I think
 some further discussion of this, with real benchmark numbers to refer
 to, is a prerequisite to considering Channels as a candidate for Django
 1.10. To take a parallel from Python, Guido has always said that he
 won't consider removing the GIL unless it can be done without penalizing
 single-threaded code. If you think a different approach makes sense here
 (that is, that it's OK to penalize the simple cases in order to
 facilitate the less-simple ones), can you explain your reasons for that
 position?

>>>
>>> We would also need some form of streamed messages for streamed http
>>> responses.
>>>
>>> Is it possible to handle old-style http the way it has always been
>>> handled?
>>>
>>>  - Anssi
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-developers+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-developers@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/CALMtK1Gz%3DaYMLyFW2da2C6Wo_-c_V2T_4p6K9eh0vwrKB91dKw%40mail.gmail.com
>>> 

Re: Channels integration plan, first draft

[Anssi Kääriäinen]

Is the idea a large site using classic request-response architecture would
get the requests at interface servers, these would then push the HTTP
requests through channels to worker processes, which process the message
and push the response through the channel backend back to the interface
server and from there back to the client?

 - Anssi

On Thursday, December 17, 2015, Andrew Godwin  wrote:

> To address the points so far:
>
>  - I'm not yet sure whether "traditional" WSGI mode would actually run
> through the in memory backend or just be plugged in directly to the
> existing code path; it really depends on how much code would need to be
> moved around in either case. I'm pretty keen on keeping a raw-WSGI path
> around for performance/compatability reasons, and so we can hard fail if
> you try *any* channels use (right now the failure mode for trying to use
> channels with the wsgi emulation is silent failure)
>
> - Streaming HTTP responses are already in the channels spec as chunked
> messages; you just keep sending response-style messages with a flag saying
> "there's more".
>
> - File uploads are more difficult, due to the nature of the worker model
> (you can't guarantee all the messages will go to the same worker). My
> current plan here is to revise the message spec to allow infinite size
> messages and make the channel backend handle chunking in the best way
> (write to shared disk, use lots of keys, etc), but if there are other
> suggestions I'm open. This would also let people return large http
> responses without having to worry about size limits.
>
> - Alternative serialisation formats will be looked into; it's up to the
> channel backend what to use, I just chose JSON as our previous research
> into this at work showed that it was actually the fastest overall due to
> the fact it has a pure C implementation, but that's a year or two old.
> Whatever is chosen needs large support and forwards compatability, however.
> The message format is deliberately specified as JSON-capable structures
> (dicts, lists, strings) as it's assumed any serialisation format can handle
> this, and so it can be portable across backends.
>
> - I thought SCRIPT_NAME was basically unused by anyone these days, but
> hey, happy to be proved wrong. Do we have any usage numbers on it to know
> if we'd need it for a new standalone server to implement? It's really not
> hard to add it into the request format, just thought it was one of those
> CGI remnants we might finally be able to kill.
>
> Andrew
>
> On Thu, Dec 17, 2015 at 6:32 PM, Anssi Kääriäinen  > wrote:
>
>> On Thursday, December 17, 2015, Carl Meyer > > wrote:
>>
>>> Hi Andrew,
>>
>>
>>> - I share Mark's concern about the performance (latency, specifically)
>>> implications for projects that want to keep deploying old-style, given
>>> all the new serialization that would now be in the request path. I think
>>> some further discussion of this, with real benchmark numbers to refer
>>> to, is a prerequisite to considering Channels as a candidate for Django
>>> 1.10. To take a parallel from Python, Guido has always said that he
>>> won't consider removing the GIL unless it can be done without penalizing
>>> single-threaded code. If you think a different approach makes sense here
>>> (that is, that it's OK to penalize the simple cases in order to
>>> facilitate the less-simple ones), can you explain your reasons for that
>>> position?
>>>
>>
>> We would also need some form of streamed messages for streamed http
>> responses.
>>
>> Is it possible to handle old-style http the way it has always been
>> handled?
>>
>>  - Anssi
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com
>> 
>> .
>> To post to this group, send email to django-developers@googlegroups.com
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CALMtK1Gz%3DaYMLyFW2da2C6Wo_-c_V2T_4p6K9eh0vwrKB91dKw%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails 

Re: Channels integration plan, first draft

[Raphael Michel]

Hi,

Am Thu, 17 Dec 2015 12:27:07 -0700
> I'll admit to not being an expert on this use case at all, since I
> don't generally do it, but AFAIK SCRIPT_NAME remains pretty key for
> transparently deploying a Django site at non-root URL paths.

I used  this before and can confirm that it is currently very
conveniently possible to deploy a Django application on a non-root URL
using SCRIPT_PATH without modifying the application. It is not really
important to me, but I think it would be worth keeping it in, if it is
not too much hassle.

Cheers
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20151217203754.4f15a4cf%40arlen.
For more options, visit https://groups.google.com/d/optout.


pgpkpxtq5I83X.pgp
Description: Digitale Signatur von OpenPGP

Re: Channels integration plan, first draft

[Andrew Godwin]

Yeah, I definitely see the need for it, and things like making the
webserver transparently change the paths isn't going to work since it can't
rewrite output. At least we can rename it and call it request['root_path']
or something.

On Thu, Dec 17, 2015 at 7:27 PM, Carl Meyer  wrote:

> On 12/17/2015 11:50 AM, Andrew Godwin wrote:
> > - I thought SCRIPT_NAME was basically unused by anyone these days, but
> > hey, happy to be proved wrong. Do we have any usage numbers on it to
> > know if we'd need it for a new standalone server to implement? It's
> > really not hard to add it into the request format, just thought it was
> > one of those CGI remnants we might finally be able to kill.
>
> I'll admit to not being an expert on this use case at all, since I don't
> generally do it, but AFAIK SCRIPT_NAME remains pretty key for
> transparently deploying a Django site at non-root URL paths. If you grep
> for SCRIPT_NAME in Django, you'll see that Django itself pays attention
> to it (in order to support this use case) in the core WSGIHandler and in
> the url reverser. Although it may be that passing SCRIPT_NAME in META to
> user view code isn't actually critical to continuing to support non-root
> deploys. Needs exploration.
>
> Carl
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/56730C8B.1050309%40oddbird.net
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uo66Kix5HPdPH3WSoo4NQYOeHhDPu09Vamfup8iUSN2tg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels integration plan, first draft

[Carl Meyer]

On 12/17/2015 11:50 AM, Andrew Godwin wrote:
> - I thought SCRIPT_NAME was basically unused by anyone these days, but
> hey, happy to be proved wrong. Do we have any usage numbers on it to
> know if we'd need it for a new standalone server to implement? It's
> really not hard to add it into the request format, just thought it was
> one of those CGI remnants we might finally be able to kill.

I'll admit to not being an expert on this use case at all, since I don't
generally do it, but AFAIK SCRIPT_NAME remains pretty key for
transparently deploying a Django site at non-root URL paths. If you grep
for SCRIPT_NAME in Django, you'll see that Django itself pays attention
to it (in order to support this use case) in the core WSGIHandler and in
the url reverser. Although it may be that passing SCRIPT_NAME in META to
user view code isn't actually critical to continuing to support non-root
deploys. Needs exploration.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/56730C8B.1050309%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: Channels integration plan, first draft

[Andrew Godwin]

To address the points so far:

 - I'm not yet sure whether "traditional" WSGI mode would actually run
through the in memory backend or just be plugged in directly to the
existing code path; it really depends on how much code would need to be
moved around in either case. I'm pretty keen on keeping a raw-WSGI path
around for performance/compatability reasons, and so we can hard fail if
you try *any* channels use (right now the failure mode for trying to use
channels with the wsgi emulation is silent failure)

- Streaming HTTP responses are already in the channels spec as chunked
messages; you just keep sending response-style messages with a flag saying
"there's more".

- File uploads are more difficult, due to the nature of the worker model
(you can't guarantee all the messages will go to the same worker). My
current plan here is to revise the message spec to allow infinite size
messages and make the channel backend handle chunking in the best way
(write to shared disk, use lots of keys, etc), but if there are other
suggestions I'm open. This would also let people return large http
responses without having to worry about size limits.

- Alternative serialisation formats will be looked into; it's up to the
channel backend what to use, I just chose JSON as our previous research
into this at work showed that it was actually the fastest overall due to
the fact it has a pure C implementation, but that's a year or two old.
Whatever is chosen needs large support and forwards compatability, however.
The message format is deliberately specified as JSON-capable structures
(dicts, lists, strings) as it's assumed any serialisation format can handle
this, and so it can be portable across backends.

- I thought SCRIPT_NAME was basically unused by anyone these days, but hey,
happy to be proved wrong. Do we have any usage numbers on it to know if
we'd need it for a new standalone server to implement? It's really not hard
to add it into the request format, just thought it was one of those CGI
remnants we might finally be able to kill.

Andrew

On Thu, Dec 17, 2015 at 6:32 PM, Anssi Kääriäinen 
wrote:

> On Thursday, December 17, 2015, Carl Meyer  wrote:
>
>> Hi Andrew,
>
>
>> - I share Mark's concern about the performance (latency, specifically)
>> implications for projects that want to keep deploying old-style, given
>> all the new serialization that would now be in the request path. I think
>> some further discussion of this, with real benchmark numbers to refer
>> to, is a prerequisite to considering Channels as a candidate for Django
>> 1.10. To take a parallel from Python, Guido has always said that he
>> won't consider removing the GIL unless it can be done without penalizing
>> single-threaded code. If you think a different approach makes sense here
>> (that is, that it's OK to penalize the simple cases in order to
>> facilitate the less-simple ones), can you explain your reasons for that
>> position?
>>
>
> We would also need some form of streamed messages for streamed http
> responses.
>
> Is it possible to handle old-style http the way it has always been handled?
>
>  - Anssi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CALMtK1Gz%3DaYMLyFW2da2C6Wo_-c_V2T_4p6K9eh0vwrKB91dKw%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upm8N8tOhCdAgPEbYbBO6MU%2BmnEQ%3D%3Dp6EmW75%3DbNXHkfg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels integration plan, first draft

[Anssi Kääriäinen]

On Thursday, December 17, 2015, Carl Meyer  wrote:

> Hi Andrew,


> - I share Mark's concern about the performance (latency, specifically)
> implications for projects that want to keep deploying old-style, given
> all the new serialization that would now be in the request path. I think
> some further discussion of this, with real benchmark numbers to refer
> to, is a prerequisite to considering Channels as a candidate for Django
> 1.10. To take a parallel from Python, Guido has always said that he
> won't consider removing the GIL unless it can be done without penalizing
> single-threaded code. If you think a different approach makes sense here
> (that is, that it's OK to penalize the simple cases in order to
> facilitate the less-simple ones), can you explain your reasons for that
> position?
>

We would also need some form of streamed messages for streamed http
responses.

Is it possible to handle old-style http the way it has always been handled?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1Gz%3DaYMLyFW2da2C6Wo_-c_V2T_4p6K9eh0vwrKB91dKw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels integration plan, first draft

[Florian Apolloner]

On Thursday, December 17, 2015 at 5:50:39 PM UTC+1, Carl Meyer wrote:
>
> - I share Mark's concern about the performance (latency, specifically) 
> implications for projects that want to keep deploying old-style, given 
> all the new serialization that would now be in the request path.
>

It would be worth investigating more involved protocols like 
https://capnproto.org/ and benchmark those in comparison to JSON. While 
JSON is nice and good, I am not sure it is the perfect transport structure 
for what we are trying to achieve (especially not for in memory transport).

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e910e8c3-bbe5-4b51-ab3b-40903e6b935a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels integration plan, first draft

[Florian Apolloner]

It would be interesting to add a few sentences about file uploads and how 
they are going to work with the new system.

On Thursday, December 17, 2015 at 12:35:18 PM UTC+1, Andrew Godwin wrote:
>
> Hi everyone,
>
> One of the first steps I want to get done for Channels is get a rough plan 
> in place for how things are going to work in terms of where code goes, 
> supported versions, etc. I've written up my thoughts on this into a first 
> draft of what I'm calling the "integration plan":
>
> http://channels.readthedocs.org/en/latest/integration-plan.html
>
> Feedback is much welcomed - I'd particularly like to hear people's 
> thoughts on the ideas of releasing both natively in Django 1.10 and as a 
> third-party addon for 1.8/1.9, plus the new (for Django) concept of 
> releasing part of it as a separate codebase, though still under the Django 
> umbrella.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/22f1296f-a951-4f69-baa8-bcfe59504b01%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels integration plan, first draft

[Carl Meyer]

Hi Andrew,

On 12/17/2015 04:34 AM, Andrew Godwin wrote:
> Hi everyone,
> 
> One of the first steps I want to get done for Channels is get a rough
> plan in place for how things are going to work in terms of where code
> goes, supported versions, etc. I've written up my thoughts on this into
> a first draft of what I'm calling the "integration plan":
> 
> http://channels.readthedocs.org/en/latest/integration-plan.html
> 
> Feedback is much welcomed - I'd particularly like to hear people's
> thoughts on the ideas of releasing both natively in Django 1.10 and as a
> third-party addon for 1.8/1.9, plus the new (for Django) concept of
> releasing part of it as a separate codebase, though still under the
> Django umbrella.

Thanks for your work on this. A few quick thoughts on first reading:

- On the packaging side, we don't really support or offer instructions
for any installation technique other than pip -- even if you manually
download a tarball, you should still install it with pip. Even our
"install the dev version from git" instructions use "pip install -e .".
So I think you should simplify the packaging/code-reuse plan to
"interface server and channel backends will be separate packages, listed
as dependencies where needed." AFAIK we've said for a while now that
we're ready for required dependencies, just waiting for a case where we
actually need them. I think this is that case, and it's a better option
than adding a bunch of build complexity where sometimes things are
bundled and sometimes they are not.

- I share Mark's concern about the performance (latency, specifically)
implications for projects that want to keep deploying old-style, given
all the new serialization that would now be in the request path. I think
some further discussion of this, with real benchmark numbers to refer
to, is a prerequisite to considering Channels as a candidate for Django
1.10. To take a parallel from Python, Guido has always said that he
won't consider removing the GIL unless it can be done without penalizing
single-threaded code. If you think a different approach makes sense here
(that is, that it's OK to penalize the simple cases in order to
facilitate the less-simple ones), can you explain your reasons for that
position?

- Re imports and builtin vs third-party, I think we should avoid magic
and just require try/except for portable code. It's a bit of
boilerplate, but only authors of reusable apps are likely to need it,
and it keeps things simple and explicit. The boilerplate can be easily
packaged up in a `compat` module if someone cares enough.

- I also share Mark's concern about the SCRIPT_NAME change. Non-root
deploys are a key use case for many people (and a lot of work over the
years has gone into making them work!). I don't think it's an option to
just shrug and break that entirely in Django 1.10, which is what it
currently sounds like you're proposing. What's the technical obstacle
here, exactly?

Carl


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5672E7C3.1000704%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: Channels integration plan, first draft

[Mark Lavin]

I have concerns about "built-in feature" means for existing applications 
(and future applications really). Under "Preserving Simplicty" you note 
that you should be able to run Django as simply as you do now. Is there a 
consideration to run "classic" WSGI applications without channels? 
This in-memory channel isn't free. It still requires serializing the 
incoming HTTP request to JSON, putting it into the channel queue, 
deserializing the JSON, , serializing the 
response to JSON, putting it into the response channel, deserializing the 
response, and sending to the client. That's two JSON serialization round 
trips which don't currently exist with no gain for existing applications. 
As you note later on this currently breaks non-root mounted (those using 
SCRIPT_NAME) applications but doesn't appear to have a plan to resolve it.

Best,

Mark


On Thursday, December 17, 2015 at 6:35:18 AM UTC-5, Andrew Godwin wrote:
>
> Hi everyone,
>
> One of the first steps I want to get done for Channels is get a rough plan 
> in place for how things are going to work in terms of where code goes, 
> supported versions, etc. I've written up my thoughts on this into a first 
> draft of what I'm calling the "integration plan":
>
> http://channels.readthedocs.org/en/latest/integration-plan.html
>
> Feedback is much welcomed - I'd particularly like to hear people's 
> thoughts on the ideas of releasing both natively in Django 1.10 and as a 
> third-party addon for 1.8/1.9, plus the new (for Django) concept of 
> releasing part of it as a separate codebase, though still under the Django 
> umbrella.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ab27553c-7ebe-4a97-83ad-0fcd4dcc5023%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Channels integration plan, first draft

[Andrew Godwin]

Hi everyone,

One of the first steps I want to get done for Channels is get a rough plan
in place for how things are going to work in terms of where code goes,
supported versions, etc. I've written up my thoughts on this into a first
draft of what I'm calling the "integration plan":

http://channels.readthedocs.org/en/latest/integration-plan.html

Feedback is much welcomed - I'd particularly like to hear people's thoughts
on the ideas of releasing both natively in Django 1.10 and as a third-party
addon for 1.8/1.9, plus the new (for Django) concept of releasing part of
it as a separate codebase, though still under the Django umbrella.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urujzF77_f2_yy2Xc63ATn%3DFaTC%3Dgjv3QFW6cQ26Z-4oA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: MOSS Award to Django

[Andrew Godwin]

Hi Samuel,

You'll notice if you look carefully that I avoid saying "async" almost
anywhere in the Channels announcements or documentation, and when I do,
it's qualified very carefully. Channels makes Django event-driven, but it
does not provide full "in-process" async like Twisted, asyncio, or
something like Pulsar.

Channels is actually kind of related to the actor model in some ways, but
it's simpler; imagine it as a pool of identical actors that are all
stateless, apart from the interface servers, which have the incoming socket
and thus the state.

The key thing is that Channels does nothing to make Django asynchronous.
All the Django code will remain synchronous (the ORM, templating, etc.),
and run in synchronous processes that just process one message after the
next. The only part of the system that will require an async framework is
the interface servers, as to be anywhere near efficient they need to serve
more than one socket at once; right now we're using Twisted and asyncio for
those, purely as they're incredibly mature and well-supported by the
community, but you could swap out anything there that achieved the same
goal (greenlets, for example, though I personally dislike their
implicitness).

Channels essentially takes the worker-prefork model that's been used in
webservers for years and applies it to a lower-level message concept rather
than HTTP requests (http requests are just one kind of message). This not
only has the advantage of a lot of previous work and research into
architecting and scaling these kinds of systems, it also means the actual
amount of work in core Django isn't crazy; it's more about providing nice
places to wrap up the worker loop and nice utilities around authentication
and session management.

If you read through the docs there's a bit more about this, but basically,
the fundamental design goal here is that Channels be no harder to use than
normal Django; indeed, it's almost exactly the same view concept, but
instead of a request you get a message, and the newer developers I've
interviewed about it seem to be comfortable with the extension of the
current abstraction straight away. I don't ever want people to have to
write asynchronous code if they don't want to; it's currently so easy to
shoot yourself in the foot in almost any Python async solution that I don't
think that makes for a good developer experience.

The "complexity" of the message backends you describe is because they're
network-transparent; there is indeed a simple, in-process one that uses
threads too, but the whole point of this design is that you run Django over
multiple processes on many machines, and in order to get something like
that you need a place to funnel messages through (another potential backend
design could use direct connection and process discovery, perhaps, or an
Erlang-like process spawn-and-track design with an overarching controller
process with access to many pre-selected machines).

The idea of making an asynchronous Django is tangential to making Channels;
this is more about changing the design abstraction of Django so we can fit
the concept in, and then providing a neat solution that drops into the gap.
There's nothing stopping someone making the worker processes for Channels
async-aware as well, so they can run multiple message consumers at once,
but that's such a complex and risk-fraught task that I don't think it's
worth it when we can get 90% of the benefits with a much simpler and
easier-to-design-and-understand system. If you want to see an example of
actual async Django, see Amber's talk from Django Under the Hood this year,
where she managed to get it working decently well under Twisted.

I expect Channels to smooth out Django's performance mostly as a
side-effect of the worker model providing demand-based load balancing by
design, but I doubt it will be a huge increase until people start using the
primitives it provides to offload tasks to other workers.

I hope that somewhat long-winded explanation helps - if not, let me know
what's still unclear and I'll try to tackle it. Clearing up what we're
trying to do and communicating it is important to me; I want to make sure
everyone knows so we can get good feedback on this stuff. For what it's
worth, I did consider the actor model along with several other kinds of
concurrency approaches (I've been prototyping bits of this for years); this
approach ended up being the nicest design and the one I have the most
confidence in that we can scale and shard to large work volumes.

Andrew

On Thu, Dec 17, 2015 at 10:30 AM, Samuel Bishop 
wrote:

> I'm uncertain how popular the suggestion will be but ... are "channels"
> the right solution to our async/concurrency problems? ( I'm all for
> django-channels the project and the work to solve these issues)
> All the talk of channels, workers, queues and back ends in the
> explanations I'm reading as I follow along with the progress of 'Django
> Channels", just feels 

Re: MOSS Award to Django

[Samuel Bishop]

I'm uncertain how popular the suggestion will be but ... are "channels" the 
right solution to our async/concurrency problems? ( I'm all for 
django-channels the project and the work to solve these issues)
All the talk of channels, workers, queues and back ends in the explanations 
I'm reading as I follow along with the progress of 'Django Channels", just 
feels "less good" than solving this problem via Actor Model Concurrency 
(championed by Erlang, Elixir, Stackless Python, etc). 

There is an excellent actor model concurrency library for python that 
already supports 'async django'. "Pulsar" 
(http://pythonhosted.org/pulsar/index.html) and the documentation here 
http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways to 
use actor model concurrency to attack the issue of using Django with 
asynchronous web stuff like Websockets, etc.
Some of what's being described in the explanations of Django channels 
sounds very similar to what they provide to work around the limitations of 
django's blocking request processing cycle other parts sound much more 
complicated, for instance the need for various back ends 
in-memory/redis/etc.

I know Pulsar is currently Python 3.4 and newer only, but in the past it 
supported 2.7, so anything we can learn from their concurrency approach can 
work on Python 2.7 to maintain our backwards compatibility going forward. 
You can see an example of Pulsar using python 2 if you look on the 
benchmark page for their python powered 'redis clone' 
- https://gist.github.com/lsbardel/8068579 - One of the benchmarks is using 
pypy so the code definitely worked in a python 2.7 environment.

Would the work to integrate channels as part of Django improve the ability 
to run Django in a completely async mode and enhance its performance when 
run under Pulsar? or would it just be something that needed to be turned 
off if we wanted to use something like Pulsar to power our "async django". 
Is there anything to be gained from using a more actor driven concurrency 
model or adopting any of the other methods used in their 'pulse' django app?
I suppose I'm just trying to work out how to compare using Django-Channels 
with using Pulse from the Pulsar project.

On Wednesday, 16 December 2015 19:23:56 UTC+8, Andrew Godwin wrote:

>
> On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann <
> in...@markusholtermann.eu > wrote:
>
>>
>> >If I get it right -- Curtis' description is spot-on; some tasks will
>> >still
>> >need Celery, Channels will take care of many others.
>>
>> For me it's more a question of "is a client directly involved".
>>
>
> I don't agree; I think it's more "do you have enough workers to handle 
> this". Channels makes all workers serve all requests, so if you're doing 
> e.g. video encoding you can't segregate it from request handling. However, 
> you could have multiple channel backends and thus multiple worker pools to 
> achieve this.
>
> Andrew 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Decoupling the ORM

[Samuel Bishop]

By the time I opened the issue ticket I had become convinced that the DB 
Compiler was effectively an impossible route. I completely agree with your 
sentiments about implementing Compiler. I'd go as far as to suggest that 
few small documentation changes may be warranted in order to suitably 
explain to future developers that they should not take this route if their 
database is not "relational enough".

Using different models has some advantages in that it can take full 
advantage of the underlying database's capabilities. But it does sacrifice 
compatibility with a significant amount of existing Django packages, so 
putting aside the additional complexity level, its not the target I'm 
aiming for. 

I'm definitely coming to the conclusion that Queryset is the correct place 
to start work. 
I think there are a number of issues this will expose/create, such as 
issues related to UUID usage especially as Primary Keys, and from just a 
few minutes re-reading the queryset class I also think there may be a need 
to clarify  

So far, I've found the following problems related to UUIDs that might get 
in the way of 'finishing' this work.

Existing issues: 
- https://code.djangoproject.com/ticket/24691 
- https://code.djangoproject.com/ticket/24954
- https://code.djangoproject.com/ticket/6663

I've identified one new "issue".
There is an implicit assumption that primary keys are useful for ordering 
by the current QuerySet API methods `.first()` and `.last()`.
https://docs.djangoproject.com/en/1.9/ref/models/querysets/#first / 
https://docs.djangoproject.com/en/1.9/ref/models/querysets/#last 
I'll raise an issue for this item after I give an opportunity for further 
discussion here since I'd like to have more of an idea regarding typical 
usage of these two queryset methods. I'm currently unsure how often these 
are used on unordered QuerySet objects. If the current behaviour of 
implicitly falling back to ordering by the primary key is in heavy use, I 
will need to take that into consideration. In the shorter term I currently 
have a few possible workarounds in mind to replicate the existing behaviour 
but the performance implications of these different methods become 
significantly more important if the implicit order by primary key behaviour 
is heavily used. Longer term, this behaviour might be good to deprecate by 
documenting that without an integer primary key, this behaviour cannot be 
relied upon, and removing any workarounds that emulate integer ordering 
type behaviour.

Ticket 6663 was closed quite some time ago, however in order to get the 
most from any attempt to support non relational databases, via QuerySet or 
otherwise, it will need to be revisited and either reopened or a new issue 
created to address the point I'm about to make that I feel is encompassed 
by 6663. I hope I can avoid any confusion and be clear what I feel is 
covered by this.

The current UUIDField that was recently added to Django is not always 
suitable for use as a database primary key because:
- The UUIDField generates the UUID with Python code and this is less than 
optimal in some circumstances. Many databases can or do generate document 
or row UUID 'primary keys' automatically. It should be possible to let 
Django defer the creation of the UUID and rely on the database for the 
creation of UUID primary keys just like it currently does for automatically 
incrementing integer primary keys. 
- Existing Django applications/libraries were not written with UUID primary 
keys. Supporting existing Django applications and models is one of my 
goals, so requiring explicit use of something like `id = 
UUIDField(primary_key=True)` on a model in order to make it compatible, 
represents an issue to me. 

Ticket 6663 was about the ability to use a UUID as the primary key. While 
on the surface this appears solved, we can do `id = 
UUIDField(primary_key=True)` and we have a UUID as the primary key, what 
hasn't been addressed is the ability to say "I want to use UUIDs for 
primary keys", I feel this was the intent behind Ticket 6663 and it should 
be reopened with an explicit focus on fixing the following two things:
- The default AutoField that Django provides any model that doesn't 
explicitly create its own id field, should not "force" the use of an 
automatically incrementing integer based primary key.
- A mechanism for configuring what kind of primary keys should be used. The 
two most likely configurations are all integer primary keys and all UUID 
primary keys, so my initial thoughts are that this mechanism should reside 
at the public QuerySet API layer, probably as a boolean value set during 
QuerySet class `__init__`.

In addition to this, in order for this to be most effective, there needs to 
be a way to specify that you want to use an alternative QuerySet class. 
There are several places one could override this for their own application 
and models very easily, however no convenient way to modify the 'default 
QuerySet'