Re: structural & functional review of django documentation

[Samuel Bishop]

Broadly speaking, I think the 'optimal' goal is not going to be one that 
changes our existing documentation structure in any disruptive way.
 
I think the general concept would be covered by either creating a "fourth 
division". So we would go from "topics", "reference", and "how-to", to 
"topics", "reference", "how-to", and "implementation"/"internals"/"APIs"/etc
Or enhancing the content in the reference section so that in addition to 
our existing handwritten documentation, we expose the 'api-docs' tree as an 
addendum to the reference section.

Sphinx gives us a lot of power to work with for trying either approach. I 
cant remember which project I saw it on, but I recall seeing some sphinx 
powered projects that had Django style hand written documentation, and each 
heading (module, class, function, etc) in their hand written documentation 
had an automatically generated link to the matcihng "implementation docs" 
that took you to that segment of the sphinx autodoc output. It worked quite 
well from what I recall. 
I'm pretty busy for a few days, but if no one else does this first and 
shares some results, I'll definitely take a look at the output from 
sphinx's autodoc tools when run over the Django repo and post some 
findings. 
Since there is certainly the big question of "is the information we have 
useful"? 
The docstrings and sundry that sphinx uses for autodoc generation may need 
work before it measures up to the quality standards of our existing hand 
written documentation. 

On Monday, 28 December 2015 06:42:49 UTC+8, Tim Graham wrote:
>
> Adding a survey link is not difficult. We conducted a community survey [1] 
> earlier this year with one question related to documentation, "What parts 
> of the Django documentation do you find most useful?" What questions to ask 
> and how to turn the answers into actionable items is the part I'm not sure 
> about and maybe you could advise on.
>
> In my view, Django's docs haven't strayed from the "topics", "reference", 
> and "how-to" division that we've had since 1.0 or so. Are you aware of this 
> grouping? Maybe a "how the docs are organized" section on the index page 
> would help communicate that and make it more intuitive where to look for 
> something?
>
> I'll admit I'm skeptical of a wholesale reorganization to this structure 
> for several reasons:
> 1. It'll be confusing for existing users who are familiar with the current 
> section.
> 2. It'll make it more difficult or impossible to backport documentation 
> enhancements to the stable version of the docs (assuming we don't also 
> reorganize them with same structure)
> 3. It'll create an opportunity for broken links (obviously we could 
> mitigate this to some extent by adding redirects to the new locations).
>
> It seems to me you were pretty close to finding what you were looking for 
> at https://docs.djangoproject.com/en/1.9/ref/forms/ (first bullet, I 
> think), but I didn't understand what you meant by the page being "the Joy 
> of Cooking with Django."
>
> [1] https://www.djangoproject.com/weblog/2015/may/07/community-survey/
>
> On Sunday, December 27, 2015 at 2:47:35 PM UTC-5, Doug Epling wrote:
>>
>> Again, I am sorry if my comments have ruffled anyone's feathers.  I am 
>> not going to argue.  My sole intent is a positive one.  And, indeed, I am 
>> humbled by the ongoing work of this community over a period of time that I, 
>> until now, have not been involved.
>>
>> I beleive, it is my impression, that between Django 1.1 and now, on the 
>> verge of its second major version, there has been a tremendous amount of 
>> Python software develpment.  And the internal commenting as well as the 
>> public documentation has trailed along ad hoc.
>>
>> It can be said without legitimate reproach that any system whether it is 
>> thermodynamics or a system of communication, such as our documentation, 
>> will naturally tend toward entropy unless something actively intervenes. 
>>  And we have developed a fairly complex system compared to, say, werksgeud. 
>>
>> That patchwork approach has disrupted a flow of utility for users in both 
>> public documentation and internal commenting.  If this is true, Django has 
>> strayed from principles of its foundation.  And our motto: "The framework 
>> for perfectionists with deadlines."; holds true only until fininding 
>> oneself lost in the documentation.
>>
>> Tim is exactly right; this is with no doubt a non-trivial issue.  Is 
>> Django capable of tackling non-trivial issues?  If not I am in the wrong 
>> place (a challenge to Django, relax, it's not personal) because I believe 
>> Django should be setting the standard.  And this issue will not be resolved 
>> by an ad hoc approach; meaning our traditional methodology of a problem 
>> ticket reporting process is not amenable.  This calls for something else if 
>> it calls for anything.
>>
>> However, Wim has a good idea!  Some exploratory research is a very 
>> reasonable first step toward an 

Re: MOSS Award to Django

[Samuel Bishop]

Once again I have to thank you for taking the time to keep replying. These 
longer explanations are proving very helpful (at least to me) in 
understanding Django Channels. :-)

I definitely understand the compatibility reasons, I'm also pleased to see 
that signalling via a channel will be as simple as I had hoped, at least 
simple as long as we put aside compatibility issues with Django's built in 
signals. I'd still be pleased with only the ability to make my custom 
signals asynchronous.

Digging into the implementation details, you've sparked my curiosity. For 
some reason the simplicity of the having just send() and receive_many() 
didn't fully dawn on me until now. Once I've made progress with my 
decoupling, I think I'll have to try experiment with cleaving the major 
parts of the entire request/response cycle into a chain of asynchronous 
actors.
I'll likely test how hard it is to 'wrap' each part of the cycle in a way 
that allows it to be called by synchronously or asynchronous code. I'd like 
to avoid creating "yet another async django, that isnt really django 
anymore", keep the existing synchronous code pathway seems like a good 
start at a "useful experiment".

As for 'could you call the end result Django'... I think "Django" is 
whatever we make of it ;-) 
I don't see why we wont still call the combination of all the parts, the 
ecosystem of compatible libraries, etc, "Django", regardless of how much we 
might change between now and far future versions like 4.0 or 9.0. 

Regards
Sam

On Monday, 21 December 2015 19:31:54 UTC+8, Andrew Godwin wrote:
>
>
>
> On Mon, Dec 21, 2015 at 6:05 AM, Samuel Bishop <lucra...@gmail.com 
> > wrote:
>
>> Hi Andrew, 
>>
>> Thanks for the detailed response. I'm glad to hear you have tried a few 
>> different ways to implement this, and having read through your reply, and 
>> the channels docs again, I have more questions :-)  
>>
>> Your point about asynchronous code introducing needless complexity is 
>> very well founded, its one of the reasons I like Actor Model Concurrency 
>> over other concurrency models, it scales well computationally, and 
>> *mentally*, as a programmer, I am trying break down my real world goals 
>> into discrete steps of computations, comparisons, iterations and by 
>> combining these steps, I construct my program. Actor Model Concurrency, is 
>> no more complicated than saying "This step can be done without affecting 
>> anything else thats going on", which makes the 'actor/task' much easier to 
>> introduce to existing synchronous code than most other concurrency methods.
>>
>
> I agree - I'm a big fan of actors in general and it's usually what I turn 
> to to write complex simulations when I'm mucking around with game backends.
>  
>
>>
>> One thing that sticks with me as I think about the current channels 
>> design, relates to the idea I've raised in another discussion ( 
>> https://groups.google.com/forum/#!topic/django-developers/-BOsuVdPFXc ).
>> If I put my architect hat on and think about the future a bit, a 'more' 
>> actor based design to channels might simplify later additional async work. 
>> Since all the websocket stuff is new, I'll stick to the existing HTTP cycle 
>> in this example. In version 1 a 'vanilla' Django task/actor would be a full 
>> HTTP request response cycle, like how Django and WSGI currently work. Our 
>> Django task is effectively our WSGI task, much like Pulsar Pulse. If all of 
>> Django is contained inside our new task/worker/supervisor Actor layer, then 
>> it becomes fairly trivial to do things like create separate request 
>> handler, database query, and template rendering task/actors and chain their 
>> calls. Since in the previously linked to discussion thread, I'm talking 
>> about decoupling the ORM from the 'upper' layers of Django, its hard to not 
>> also think what opportunities that would open up for async efforts. How do 
>> you see the architecture of Django channels accommodating such improvements 
>> in future? At the moment your focused on keeping things compatible and 
>> avoiding the need for any of the existing synchronous Django code to 
>> change, is this introducing any design tradeoffs that you are aware of with 
>> regards to hypothetical future developments?
>>
>
> So, what you propose obviously isn't feasible in the short term - I'm 
> doing as much as I can to decouple Channels from "rewrite all of Django" - 
> but notably, channels is a subset of the actor model. In particular, you 
> could see consumers as stateless, single-action actors - but, this means 
> that any future abstraction would be able to build on that.
&

Re: MOSS Award to Django

[Samuel Bishop]

, and the newer developers I've 
> interviewed about it seem to be comfortable with the extension of the 
> current abstraction straight away. I don't ever want people to have to 
> write asynchronous code if they don't want to; it's currently so easy to 
> shoot yourself in the foot in almost any Python async solution that I don't 
> think that makes for a good developer experience.
>
> The "complexity" of the message backends you describe is because they're 
> network-transparent; there is indeed a simple, in-process one that uses 
> threads too, but the whole point of this design is that you run Django over 
> multiple processes on many machines, and in order to get something like 
> that you need a place to funnel messages through (another potential backend 
> design could use direct connection and process discovery, perhaps, or an 
> Erlang-like process spawn-and-track design with an overarching controller 
> process with access to many pre-selected machines).
>
> The idea of making an asynchronous Django is tangential to making 
> Channels; this is more about changing the design abstraction of Django so 
> we can fit the concept in, and then providing a neat solution that drops 
> into the gap. There's nothing stopping someone making the worker processes 
> for Channels async-aware as well, so they can run multiple message 
> consumers at once, but that's such a complex and risk-fraught task that I 
> don't think it's worth it when we can get 90% of the benefits with a much 
> simpler and easier-to-design-and-understand system. If you want to see an 
> example of actual async Django, see Amber's talk from Django Under the Hood 
> this year, where she managed to get it working decently well under Twisted.
>
> I expect Channels to smooth out Django's performance mostly as a 
> side-effect of the worker model providing demand-based load balancing by 
> design, but I doubt it will be a huge increase until people start using the 
> primitives it provides to offload tasks to other workers.
>
> I hope that somewhat long-winded explanation helps - if not, let me know 
> what's still unclear and I'll try to tackle it. Clearing up what we're 
> trying to do and communicating it is important to me; I want to make sure 
> everyone knows so we can get good feedback on this stuff. For what it's 
> worth, I did consider the actor model along with several other kinds of 
> concurrency approaches (I've been prototyping bits of this for years); this 
> approach ended up being the nicest design and the one I have the most 
> confidence in that we can scale and shard to large work volumes.
>
> Andrew
>
> On Thu, Dec 17, 2015 at 10:30 AM, Samuel Bishop <lucra...@gmail.com 
> > wrote:
>
>> I'm uncertain how popular the suggestion will be but ... are "channels" 
>> the right solution to our async/concurrency problems? ( I'm all for 
>> django-channels the project and the work to solve these issues)
>> All the talk of channels, workers, queues and back ends in the 
>> explanations I'm reading as I follow along with the progress of 'Django 
>> Channels", just feels "less good" than solving this problem via Actor Model 
>> Concurrency (championed by Erlang, Elixir, Stackless Python, etc). 
>>
>> There is an excellent actor model concurrency library for python that 
>> already supports 'async django'. "Pulsar" (
>> http://pythonhosted.org/pulsar/index.html) and the documentation here 
>> http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways 
>> to use actor model concurrency to attack the issue of using Django with 
>> asynchronous web stuff like Websockets, etc.
>> Some of what's being described in the explanations of Django channels 
>> sounds very similar to what they provide to work around the limitations of 
>> django's blocking request processing cycle other parts sound much more 
>> complicated, for instance the need for various back ends 
>> in-memory/redis/etc.
>>
>> I know Pulsar is currently Python 3.4 and newer only, but in the past it 
>> supported 2.7, so anything we can learn from their concurrency approach can 
>> work on Python 2.7 to maintain our backwards compatibility going forward. 
>> You can see an example of Pulsar using python 2 if you look on the 
>> benchmark page for their python powered 'redis clone' - 
>> https://gist.github.com/lsbardel/8068579 - One of the benchmarks is 
>> using pypy so the code definitely worked in a python 2.7 environment.
>>
>> Would the work to integrate channels as part of Django improve the 
>> ability to run Django in a completely async mode and enhance its 
>> performance 

Re: Decoupling the ORM

[Samuel Bishop]

I was more referring to 'above' in a broad way, as the fields are 
constructed from the data returned by the ORM when building the model. 
which should also be above the ORM. 

Pulling open the Django code in PyCharm and taking a look at things, with 
an eye towards "lots of smaller incremental improvements" has me 
wondering... how small is small?

As part of restructuring the layers, it would make sense to create a higher 
level module, stub the 'new' models and fields into the old locations for 
backwards compatibility, and separate the ORM related and non ORM related 
parts of the model and field classes, placing the non ORM part in the new 
module, and likely keeping the rest in the current place since 
`django.db.models` since 'django database models' feels like the right 
place for database specific parts of model code. Splitting it like that 
would allow the placement of our new 'chose your back end' logic to exist 
between the non ORM parts and the ORM parts by the use of dynamic 
subclassing based on configuration options much as we do inside the ORM.

But while that could happen with little new code needing to be written, 
since its primarily about restructuring the existing code. It doesn't 
however feel like a small change, and no matter how I cut it up, at some 
point there would likely need to be a fairly big "cut over" PR. Not really 
in the spirit of lots of small increments.  

On Sunday, 20 December 2015 07:31:48 UTC+8, Josh Smeaton wrote:
>
> As far as I'm concerned, anything we can do to simplify and decouple 
> abstraction layers is a good thing and would be welcomed, with the usual 
> caveats of backwards compatibility in public and pseudo-public APIs.
>
> models.query is a thin layer over models.sql.query. That's a good thing. 
> Ideally, models.query should be able to use any sql.query underneath, but 
> there's currently no good way to swap out implementations. I think 
> simplifying models.query so that it is *only* a thin wrapper over the 
> backend query class would be a great first start. 
>
> I think there's a tonne of room for improvement with sql.query also. It 
> does way more than just store the current state of the query. I feel a lot 
> of the functionality of sql.query should be pushed down to the Compiler 
> class.
>
> As far as fields themselves go, they're distinct from the 
> queryset/query/compiler layer, so I wouldn't say they are "above" or 
> "below" queryset itself. Perhaps fields would benefit from a similar 
> abstraction where there is a very thin layer up top, and a "backend" 
> implementation that does the heavy lifting. Again though, figuring out a 
> nice way to swap out implementations would be a challenge, and preserving 
> backwards compatibility of fields is going to be even harder than that of 
> query and compiler, because fields are most definitely public/pseudo-public.
>
> But going back to my original point, we'd welcome lots of smaller 
> incremental improvements to the abstraction layers. Fields are going to be 
> hard to improve though. They offer so many different methods that aren't 
> well documented but are definitely relied on from all backends.
>
> Cheers
>
>
> On Sunday, 20 December 2015 02:39:13 UTC+11, Samuel Bishop wrote:
>>
>> You've raise a very good point, one that has been on my mind the entire 
>> time I've been exploring this. 
>> Much of Django has been designed 'on the back of' the ORM, reliant on its 
>> limitations and without need to go beyond the scope of what it can provide, 
>> its very probable that over time this has introduced some failures to fully 
>> adhere to the best possible separation of concerns.
>>
>> Autofield does seem to suffer from this a little with its assumption of 
>> Integer only keys. 
>> Fundamentally, when it comes to letting the database do the work for us, 
>> Django doesn't need a lot of code to make a field... I've seen alternate 
>> (non primary key) autofields as short as 3 lines. If we took the simplest 
>> approach possible, a 'project wide setting'. Such a setting could 'toggle' 
>> between integer and string/UUID behavior everywhere we need to introduce 
>> changes. However I feel that may be too simplistic an approach if I want to 
>> try and get broader support for NoSQL/Alternate DB support via the approach 
>> I'm so determined to try and build. 
>>
>> My biggest worry is really, just how many places check the DB parameters. 
>> A cursory reading of the base Field class, has literally dozens of 
>> references to both the default connection wrapper object `connection`, and 
>> to the database connectionhandler object `connections`. 55 references at 
>> last check in `django.db.models.field

Re: Decoupling the ORM

[Samuel Bishop]

You've raise a very good point, one that has been on my mind the entire 
time I've been exploring this. 
Much of Django has been designed 'on the back of' the ORM, reliant on its 
limitations and without need to go beyond the scope of what it can provide, 
its very probable that over time this has introduced some failures to fully 
adhere to the best possible separation of concerns.

Autofield does seem to suffer from this a little with its assumption of 
Integer only keys. 
Fundamentally, when it comes to letting the database do the work for us, 
Django doesn't need a lot of code to make a field... I've seen alternate 
(non primary key) autofields as short as 3 lines. If we took the simplest 
approach possible, a 'project wide setting'. Such a setting could 'toggle' 
between integer and string/UUID behavior everywhere we need to introduce 
changes. However I feel that may be too simplistic an approach if I want to 
try and get broader support for NoSQL/Alternate DB support via the approach 
I'm so determined to try and build. 

My biggest worry is really, just how many places check the DB parameters. A 
cursory reading of the base Field class, has literally dozens of references 
to both the default connection wrapper object `connection`, and to the 
database connectionhandler object `connections`. 55 references at last 
check in `django.db.models.fields` alone. It worries me that I may still 
need to create essentially a fake database backend full of dozens of stub 
entries to satisfy the introspection being performed by various classes 
despite using a queryset that never even makes a database call. If anyone 
with more knowledge about this wants to comment, it would be most useful. 
Currently, I'm operating under the assumption, educated at least in part by 
studying the DB connection classes, testing and debugging various 
alternative database backends, and debugging my way through ORM calls,that 
the queryset interface is the uppermost level of ORM interfacing code. In 
principle nothing above the queryset should rely on any particular 
database, or any database at all. If there are parts of Django "above" the 
queryset that rely on knowing things about the database, how defective 
should we consider this? Should I be temporarily working around such 
issues, commenting things out, logging bugs etc, or implementing 
workarounds in my own queryset... such as the aforementioned 'fake database 
backend'. 


On Saturday, 19 December 2015 01:42:40 UTC+8, Marc Tamlyn wrote:
>
> I agree that the current uuidfield is a simplistic solution to be used as 
> autofield, and having a way to specify your autofield type globally would 
> be good, equally something like a biginteger field. The complexity involved 
> with doing this (custom RETURNING statements, database level defaults, 
> integration with migrations) is an order of magnitude more complex than the 
> solution committed. I knew that was the intention of the original issue but 
> it was also why it hadn't been solved for the "simple" case.
>
> I did spend some time looking at the more complete solution and honestly 
> could not work out how to approach it. I found if hit many more parts of 
> the internals of the ORM than I expected.
> On 18 Dec 2015 12:32 a.m., "Curtis Maloney"  > wrote:
>
>> I've identified one new "issue".
>>> There is an implicit assumption that primary keys are useful for
>>> ordering by the current QuerySet API methods `.first()` and `.last()`.
>>>
>>
>> I believe the case here is that first and last are meaningless without an 
>> ordering, so in lieu of any programmer supplied ordering, a "fallback" 
>> consistent ordering of 'pk' is used.
>>
>> Remember the DBMS is under no obligation to return a query in consistent 
>> order without an ORDER BY clause.
>>
>> So, without an ordering, two successive calls to first() [or last()] may, 
>> in fact, return different records even without modifications to the table.
>>
>> It's not expected people should _rely_ on the ordering of PK, and, 
>> indeed, it's frequently recommended against inferring any meaning from PK 
>> values [sqlite, IIRC, assigns them pseudo-ramdomly]
>>
>> That said, the assumption that a PK is sortable and will provide a 
>> deterministic ordering shouldn't be much to ask, surely?
>>
>> --
>> Curtis
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/567353FA.6030105%40tinbrain.net
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this 

Re: MOSS Award to Django

[Samuel Bishop]

I'm uncertain how popular the suggestion will be but ... are "channels" the 
right solution to our async/concurrency problems? ( I'm all for 
django-channels the project and the work to solve these issues)
All the talk of channels, workers, queues and back ends in the explanations 
I'm reading as I follow along with the progress of 'Django Channels", just 
feels "less good" than solving this problem via Actor Model Concurrency 
(championed by Erlang, Elixir, Stackless Python, etc). 

There is an excellent actor model concurrency library for python that 
already supports 'async django'. "Pulsar" 
(http://pythonhosted.org/pulsar/index.html) and the documentation here 
http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways to 
use actor model concurrency to attack the issue of using Django with 
asynchronous web stuff like Websockets, etc.
Some of what's being described in the explanations of Django channels 
sounds very similar to what they provide to work around the limitations of 
django's blocking request processing cycle other parts sound much more 
complicated, for instance the need for various back ends 
in-memory/redis/etc.

I know Pulsar is currently Python 3.4 and newer only, but in the past it 
supported 2.7, so anything we can learn from their concurrency approach can 
work on Python 2.7 to maintain our backwards compatibility going forward. 
You can see an example of Pulsar using python 2 if you look on the 
benchmark page for their python powered 'redis clone' 
- https://gist.github.com/lsbardel/8068579 - One of the benchmarks is using 
pypy so the code definitely worked in a python 2.7 environment.

Would the work to integrate channels as part of Django improve the ability 
to run Django in a completely async mode and enhance its performance when 
run under Pulsar? or would it just be something that needed to be turned 
off if we wanted to use something like Pulsar to power our "async django". 
Is there anything to be gained from using a more actor driven concurrency 
model or adopting any of the other methods used in their 'pulse' django app?
I suppose I'm just trying to work out how to compare using Django-Channels 
with using Pulse from the Pulsar project.

On Wednesday, 16 December 2015 19:23:56 UTC+8, Andrew Godwin wrote:

>
> On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann <
> in...@markusholtermann.eu > wrote:
>
>>
>> >If I get it right -- Curtis' description is spot-on; some tasks will
>> >still
>> >need Celery, Channels will take care of many others.
>>
>> For me it's more a question of "is a client directly involved".
>>
>
> I don't agree; I think it's more "do you have enough workers to handle 
> this". Channels makes all workers serve all requests, so if you're doing 
> e.g. video encoding you can't segregate it from request handling. However, 
> you could have multiple channel backends and thus multiple worker pools to 
> achieve this.
>
> Andrew 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Decoupling the ORM

[Samuel Bishop]

ent way to modify the 'default 
QuerySet' class provided by `Manager`. While my first thought is modify 
`BaseManager.from_queryset()` here 
https://github.com/django/django/blob/stable/1.9.x/django/db/models/manager.py#L143
 
so that the definition of `Manager` no longer has to explicitly pass 
QuerySet, like it does here 
https://github.com/django/django/blob/stable/1.9.x/django/db/models/manager.py#L238
 
the potential impact of such changes is definitely something I'm unfamiliar 
with, so I would greatly appreciate any feedback on how appropriate this 
approach would be.

- Sam 


On Wednesday, 16 December 2015 15:33:08 UTC+8, Anssi Kääriäinen wrote:

> On Tuesday, December 15, 2015 at 5:43:55 PM UTC+2, Samuel Bishop wrote:
>>
>> Having worked through the code of several Django nosql/alternative 
>> database backend libraries, forks, etc... 
>>
>> I've noticed that that one of the biggest challenges they run into, is 
>> 'conforming' to many of the things Django expects these lowest layers to do.
>>
>> I opened this ticket https://code.djangoproject.com/ticket/25265 to 
>> begin getting feedback on an initial idea for how to 'fix' the problem.
>> Since then I've had further time to ponder the problem and while it still 
>> seems to me that the best mechanism is to draw a line between the 'upper' 
>> and 'lower' layers of Django, 
>> I'm no longer 100% sure the correct place to enable this is the queryset 
>> via an additional method, because I've realized that this is not just an 
>> opportunity to get NoSQL databases into Django, but also an opportunity to 
>> finally provide support for alternative Python ORMs, such as SQLAlchemy. 
>>
>> I've been digging around the code for this so I dont mind writing up the 
>> code for this, but there is the big question of 'where to decouple' things. 
>> Initial feedback in the thread 
>> https://code.djangoproject.com/ticket/25265#comment:4 
>> <https://www.google.com/url?q=https%3A%2F%2Fcode.djangoproject.com%2Fticket%2F25265%23comment%3A4=D=1=AFQjCNEeTJV5U_vgQBRIQIppH6F1Hf991Q>
>>  
>> has raised the suggestion that moving one layer further up may be the right 
>> place to go. It would be very helpful for me to get extra input from Django 
>> developers familiar with the QuerySet and Query, before I start writing, so 
>> I would love to hear feedback on the idea.
>>
>
> Assume the goal is perfect admin integration with a MongoDB backend. The 
> approach can be either:
> 1) Use Django's standard models, create a QuerySet compatible 
> MongoDBQuerySet.
> 2) Use completely different models, which respond to the APIs needed by 
> Admin. This includes implementing a QuerySet compatible MongoDBQuerySet.
>
> There is a lot more work to 2), but the benefit is that you get to use 
> models actually meant to be used with a non-relational backend. For 
> example, Django's User, Permission and Group models are implemented in a 
> way that makes sense for a relational backend. If you use relational schema 
> on non-relational database you are going to face big problems if you try to 
> run the site with any non-trivial amount of data. For this reason I believe 
> 2) to be the right approach.
>
> But, to get there, a QuerySet compatible MongoDBQuerySet is needed 
> anyways. Here the choices are those mentioned in 
> https://code.djangoproject.com/ticket/25265#comment:4 
> <https://www.google.com/url?q=https%3A%2F%2Fcode.djangoproject.com%2Fticket%2F25265%23comment%3A4=D=1=AFQjCNEeTJV5U_vgQBRIQIppH6F1Hf991Q>.
>  
> That is, you can go with Django's QuerySet and Query, and just implement a 
> MongoDBCompiler. Or, you can use QuerySet with MongoDBQuery class. Or, 
> finally, you can implement MongoDBQuerySet directly from scratch.
>
> If you implement Compiler or Query, you are targeting internal APIs which 
> we *will* change in the future, maybe even in dramatic ways. If you target 
> QuerySet, you are targeting a public API that doesn't change often. And, 
> even if it changes, you will get a nice deprecation period for the changes.
>
> It might seem a lot of work to start from QuerySet, but for a 
> non-relational backend there isn't actually *that* much work involved. Most 
> of Django's Query and Compiler classes deal with joins, SQL's NULL 
> peculiarities or SQL's way of doing aggregations. All of these are 
> non-issues for non-relational backends.
>
> So, I think you should start with implementing a custom QuerySet for your 
> wanted backend. You can also try to make it work with all Django models, 
> but that approach is very likely to fail. For starters, Django's models use 
> an autoincrementing integer field for primary key, whereas most (if not 
> all) nonrelat

Re: Decoupling the ORM

[Samuel Bishop]

After many years idly wanting the feature, and having looked at the 
existing "state of the art" when it came to Django and alternative 
Databases, it was Djangae that convinced me that this is doable and got me 
"over the ditch" to where I now stand, Djangae on one side of me as proof, 
a working example that Django can exist on this 'brave new soil' so to 
speak, looking forward at several concrete options for how I can push even 
further ahead. So you've been a lot more helpful than you think ;-) 

On Wednesday, 16 December 2015 16:01:44 UTC+8, Ola Sitarska wrote:
>
> I'm definitely not going to be more helpful than Anssi, but feel free to 
> take a look on Djangae <https://github.com/potatolondon/djangae>, a 
> Django database backend for non-relational datastore on Google App Engine. 
> We've also got a chat and a mailing list if you wanna have a chat with us. 
>
> Here are also some implementation details 
> <http://blog.kazade.co.uk/2015/06/how-djangaes-app-engine-datastore.html>.
>
> On Wed, Dec 16, 2015 at 7:33 AM, Anssi Kääriäinen <akaa...@gmail.com 
> > wrote:
>
>> On Tuesday, December 15, 2015 at 5:43:55 PM UTC+2, Samuel Bishop wrote:
>>>
>>> Having worked through the code of several Django nosql/alternative 
>>> database backend libraries, forks, etc... 
>>>
>>> I've noticed that that one of the biggest challenges they run into, is 
>>> 'conforming' to many of the things Django expects these lowest layers to do.
>>>
>>> I opened this ticket https://code.djangoproject.com/ticket/25265 to 
>>> begin getting feedback on an initial idea for how to 'fix' the problem.
>>> Since then I've had further time to ponder the problem and while it 
>>> still seems to me that the best mechanism is to draw a line between the 
>>> 'upper' and 'lower' layers of Django, 
>>> I'm no longer 100% sure the correct place to enable this is the queryset 
>>> via an additional method, because I've realized that this is not just an 
>>> opportunity to get NoSQL databases into Django, but also an opportunity to 
>>> finally provide support for alternative Python ORMs, such as SQLAlchemy. 
>>>
>>> I've been digging around the code for this so I dont mind writing up the 
>>> code for this, but there is the big question of 'where to decouple' things. 
>>> Initial feedback in the thread 
>>> https://code.djangoproject.com/ticket/25265#comment:4 
>>> <https://www.google.com/url?q=https%3A%2F%2Fcode.djangoproject.com%2Fticket%2F25265%23comment%3A4=D=1=AFQjCNEeTJV5U_vgQBRIQIppH6F1Hf991Q>
>>>  
>>> has raised the suggestion that moving one layer further up may be the right 
>>> place to go. It would be very helpful for me to get extra input from Django 
>>> developers familiar with the QuerySet and Query, before I start writing, so 
>>> I would love to hear feedback on the idea.
>>>
>>
>> Assume the goal is perfect admin integration with a MongoDB backend. The 
>> approach can be either:
>> 1) Use Django's standard models, create a QuerySet compatible 
>> MongoDBQuerySet.
>> 2) Use completely different models, which respond to the APIs needed by 
>> Admin. This includes implementing a QuerySet compatible MongoDBQuerySet.
>>
>> There is a lot more work to 2), but the benefit is that you get to use 
>> models actually meant to be used with a non-relational backend. For 
>> example, Django's User, Permission and Group models are implemented in a 
>> way that makes sense for a relational backend. If you use relational schema 
>> on non-relational database you are going to face big problems if you try to 
>> run the site with any non-trivial amount of data. For this reason I believe 
>> 2) to be the right approach.
>>
>> But, to get there, a QuerySet compatible MongoDBQuerySet is needed 
>> anyways. Here the choices are those mentioned in 
>> https://code.djangoproject.com/ticket/25265#comment:4 
>> <https://www.google.com/url?q=https%3A%2F%2Fcode.djangoproject.com%2Fticket%2F25265%23comment%3A4=D=1=AFQjCNEeTJV5U_vgQBRIQIppH6F1Hf991Q>.
>>  
>> That is, you can go with Django's QuerySet and Query, and just implement a 
>> MongoDBCompiler. Or, you can use QuerySet with MongoDBQuery class. Or, 
>> finally, you can implement MongoDBQuerySet directly from scratch.
>>
>> If you implement Compiler or Query, you are targeting internal APIs which 
>> we *will* change in the future, maybe even in dramatic ways. If you target 
>> QuerySet, you are targeting a public API that doesn't change often. And, 
>> even if it changes, you 

Decoupling the ORM

[Samuel Bishop]

Having worked through the code of several Django nosql/alternative database 
backend libraries, forks, etc... 

I've noticed that that one of the biggest challenges they run into, is 
'conforming' to many of the things Django expects these lowest layers to do.

I opened this ticket https://code.djangoproject.com/ticket/25265 to begin 
getting feedback on an initial idea for how to 'fix' the problem.
Since then I've had further time to ponder the problem and while it still 
seems to me that the best mechanism is to draw a line between the 'upper' 
and 'lower' layers of Django, 
I'm no longer 100% sure the correct place to enable this is the queryset 
via an additional method, because I've realized that this is not just an 
opportunity to get NoSQL databases into Django, but also an opportunity to 
finally provide support for alternative Python ORMs, such as SQLAlchemy. 

I've been digging around the code for this so I dont mind writing up the 
code for this, but there is the big question of 'where to decouple' things. 
Initial feedback in the 
thread https://code.djangoproject.com/ticket/25265#comment:4 has raised the 
suggestion that moving one layer further up may be the right place to go. 
It would be very helpful for me to get extra input from Django developers 
familiar with the QuerySet and Query, before I start writing, so I would 
love to hear feedback on the idea.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ad4200bc-086d-4b23-a5b6-49e412ac55ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.