http://code.google.com/p/live-android/
On 22 Okt., 13:35, Waldemar Kornewald <wkornew...@gmail.com> wrote: > Hi everyone, > this rather long mail contains a status report and instructions for > contributors and implementation notes for Django core developers. If > you only want to know the status you can stop after the first section. > If you want to contribute I hope this provides a good starting point > into our port. > > --------------------------------------- > Status report > > We've got pretty far with our App Engine port. For example, the > sessions db and cached_db backends both work unmodified on App Engine. > You can also order results and use basic filter()s as supported by the > low-level App Engine API (gt, gte, lt, lte, exact, pk__in). You can > also use QuerySet.order_by(), .delete(), .count(), Model.save(), > .delete(). > > This is our second porting attempt (it's not in the old repository). > Our first attempt had too many conflicts with the multi-db branch > (esp. the one on github). This time we just hacked everything > together. We didn't concentrate on cleaning up the current backend > API. We've also disabled SQL support. > > The next step is to move all the hacks into a nice backend API (at the > same time making sure that it won't conflict with multi-db) and > re-enable SQL support. That's where we need help. Also, if you want to > work on SimpleDB support this is the right time to join. The App > Engine backend itself can be handled by Thomas Wanschik and me - > contributions in this area are not absolutely necessary, so please > concentrate on the cleanup if you want to help. > > Now to the details (for those who want to contribute). > > --------------------------------------- > Introducing QueryGlue > > The old Django code was distributed across three layers: > * django.db.models.queryset.QuerySet > * django.db.models.sql.query.Query (from now on just sql.Query) > * backend > > When a new QuerySet is instantiated (e.g. by calling > Model.objects.all()) it asks the backend for its Query class and then > creates an instance of that class. By default, this class is > sql.Query. Only the Oracle backend has its own Query which subclasses > sql.Query. I think Waldermar wanted to say: 'Only the Oracle backend has its own Query which subclasses sql.query.BaseQuery.' instead of 'sql.Query'. We added a module called 'nonrelational' to django.db.models. There we define query.py which itself defines a BaseQuery class (the equivalent to sql.query.BaseQuery). Just like Oracle subclasses sql.query.BaseQuery we could use the same mechanism and subclass nonrelational.BaseQuery in order to get the GAEQuery. For now this subclassing mechanism is called in the module sql.query (not sql.Query from above) by connection.ops.query_class(BaseQuery) and uses sql.query.BaseQuery as the BaseQuery. We need some way of telling which class should be the BaseQuery. This could be done via the settings, something similar to settings.DATABASE_TYPE = 'nonrelational' and for sql it would be settings.DATABASE_TYPE= 'sql'. According to this the right BaseQuery could be chosen. Additionally these mechanisme shouldn't be called in sql.query. But that's only a proposal of making the port a little bit more clean. I don't know if there are any conflicts with multi-db this way. What do you guys think of it? > > Normally, sql.Query builds the query on-the-fly. Whenever you call > QuerySet.filter(<filters>) the filters get put into a > Q(<filters>) and passed to > sql.Query.add_q( Q(...) ). > This function iterates over all filter rules in the Q object and calls > sql.Query.add_filter() for each individual filter. > This in turn directly modifies sql.Query.where which is a tree > structure that represents the WHERE clause. It already contains > information about the JOIN type for each filter (INNER, OUTER), the > fields that get referenced by the filter, the column and table > aliases, and so on. It already does a lot of what we need for > non-relational backends, but it's too SQL-specific. > > The current behavior is also a problem for multi-db because it makes > too many assumptions about the storage format of the filter rules. The > user could call QuerySet.using(other_connection) anytime, so QuerySet > shouldn't really work with the low-level sql.Query class before it > actually executes the query. > > We've solved this problem by introducing a backend-independent query > representation between QuerySet and the low-level Query (sql.Query, > appengine.Query, etc.). This representation is called QueryGlue. You > can find it in django.db.models.queryglue. It > provides almost exactly the same "public" API as sql.Query (so it can > easily be integrated with QuerySet). Each filter() call gets > translated into a tree structure that is inspired by sql.Query.where, > but it doesn't contain any information about the kind of JOIN. > Instead, it stores high-level important information like whether we're > filtering on a primary key, which columns and tables are involved in a > JOIN, etc. > > --------------------------------------- > The low-level Query class > > Once the query needs to be executed (e.g., by calling .count() or by > iterating over the query) the QueryGlue instance creates a new > low-level Query instance which gets the QueryGlue as its only > parameter. Currently, the low-level Query class is hard-coded to > GAEQuery/BaseQuery in django.db.models.nonrelational.query. > > Then, QueryGlue calls the Query's respective execution function > (results_iter(), count(), etc.). The > constructor only gets the QueryGlue instance. Then, we call the > respective execution function (results_iter(), count(), etc.) on the > instantiated low-level Query. Our GAEQuery can now iterate over all > filters in QueryGlue.filters and convert them to an App Engine Query > object. > This should be done cleaner too. Instead of using QueryGlue for QuerySet.query, QuerySet.query should be an instance of the actual Query class (see above for a proposal of a loading mechanism for the Query class) and the actual QueryGlue instance should be passed to QuerySet.query somehow. QuerySet's methods will update the QueryGlue instance only. This mechanism could be used for the existing sql backends too (QueryGlue has not to be used for this but at least some high-level information tree. Much code of QueryGlue could be reused for this tree but has to be extended too). Of course that will result in changes to the existing backends but would result in a flexible way to write backends by letting backends traverse the tree and form the actual query for the specified database as soon as the query is executed). > --------------------------------------- > subqueries > > Instead of working with subquery classes we've added delete_bulk(), > insert(), etc. directly to QueryGlue and the low-level Query class. If > sql.Query really needs the current design those functions can still be > routed to the respective subquery instance, but on App Engine it's > easier to handle those operations in a separate function. > > --------------------------------------- > The cleanup > > We made a few not-so-clean changes to Django itself. I've attached a > diff, so contributors can easily find all the changes we did to Django > (they're also commented with TODO and GAE): > > ............................ > * disabled multi-table inheritance; > this could be emulated as described on the Django > wikihttp://code.djangoproject.com/wiki/NonSqlBackends > > See > django/db/models/base.py: line 147 > > ............................ > * disabled deletion of related objects in Model.delete() and QuerySet.delete() > > See > django/db/models/query.py: lines 1036, 1065 > > ............................ > * replaced sql.subqueries.*Query usage with simple functions on a > single Query class (insert_or_update() instead of InsertQuery and > UpdateQuery) > > See > django/db/models/query.py: lines 1058, 1088 > > ............................ > * commented out distinction between insert and update in > Model.save_base() because there's no such concept in App Engine (and > SimpleDB, AFAIK) > > See > django/db/models/base.py: lines 470, 475 > > ............................ > The long-term goal is of course to clean this up and move most of > these changes into the backend API. > Looking at the diff you can see that we really made small changes to django and in the majority of cases we simply commented some django code out (like the deletion of related objects). So moving these parts into the backend shouldn't be hard and would enable users to write nonrelational databases backends for django in a clean way without manipulating much existent code of django (existing parts only would have to be moved into the sql database backend). > --------------------------------------- > Common non-relational features > > The plan is to add support for simple joins and select_related to all > non-relational backends by > either subclassing the backend's Query class on-the-fly with a > JoinQuery or by supporting something like query pre-processors which > can be added above the low-level Query class. We haven't thought about > the details, yet, but I hope you get the idea. > > --------------------------------------- > SQL layer details: > > The ugly detail is that sql.subqueries contains specialized query > classes like InsertQuery, DeleteQuery, etc. which subclass the > backend's Query class. This means that currently, the module loading > process jumps around: > * sql/__init__.py imports sql.query and then sql.subqueries > * sql.query creates the base Query class > * after that, sql.query allows the backend to override the Query class > * sql.subqueries creates subclasses which derive from Query > > In multi-db in SVN this is uglier because the subquery classes don't > have just one single sql.Query base class from which to derive, > anymore. There can be multiple backends, each with their own sql.Query > class, so the subqueries have to be maintained by the backend (with > some multi-inheritance magic and manual caching of the custom > subclasses). > > In multi-db on github this is much cleaner: The backends can't > override sql.Query, anymore. Instead, there's an SQLCompiler class > which can be overridden by the backend to take care of > backend-specific details. sql.Query stores a slightly more abstract > representation of the query. This multi-db branch moves a lot of code > around. That's why we should try to keep as much code as possible > where it is (at least until the branch gets merged into trunk). > > --------------------------------------- > The source > > The test project and our unit tests are > here:http://bitbucket.org/wkornewald/django-testapp/ > > The modified Django source and the backend is > here:http://bitbucket.org/wkornewald/django-nonrel-hacked/ > > We've patched the trunk branch. Unforunately, the branches are > unnamed (I converted the git mirror because the hg mirror's branches > on bitbucket are broken). You should be able to find the right branch > with "hg heads" > and "hg up -C" to it. Normally our branch should be at tip, anyway, so > you don't need to do anything. > > When merging you need to find the trunk branch with "hg heads" and "hg > merge <revnum>" with the trunk head. If this becomes a huge problem > we'll switch to the django-trunk mirror, but I wanted to keep the > option to switch to Alex' multidb branch if that's better, so I chose > this sub-optimal Django mirroring solution. > > --------------------------------------- > Task management > > Our tasks are managed in a Google > Spreadsheet:https://spreadsheets.google.com/ccc?key=0AnLqunL-SCJJdE1fM0NzY1JQTXJu... > > The task list isn't complete, yet. We're working on that. > > Bye, > Waldemar Kornewald > > django.diff > 9KAnzeigenHerunterladen I hope this will give contributers an idea of where to start. I will add the solution ideas to the spreadsheet as soon as i find some time. It would be nice to hear of some thoughts from django-developers too :) Bye Thomas Wanschik --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
