Re: non-relational DB

[Waldemar Kornewald]

Hey,
a little status update:

We've switched our work to Alex' github multi-db branch because we
depend on that to make a clean non-relational backend API. Otherwise
we'd have to rewrite too much code once multi-db gets merged into
trunk. The new branch is at:
http://bitbucket.org/wkornewald/django-nonrel-multidb/

Our django-testapp project finally contains unit tests for all
supported DB features (Model.save(), QuerySet.get(), .count(),
.filter(), .exclude(), etc.).

Now we can implement a query_class() backend API and begin to move the
hacked-in App Engine code out of Django.

Bye,
Waldemar Kornewald

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=.

Re: non-relational DB

[Waldemar Kornewald]

On Thu, Oct 29, 2009 at 9:51 PM, Alex Gaynor  wrote:
> I haven't had a chance to look at it, and I probably won't until at
> least a few of the items on my plate are dealt with.  That being said
> I am extremely leery about investing time in something with names like
> "QueryGlue" as to me they imply a lack of organization in the code,
> and that may of may not be true, but giving things name thats are at
> least somewhat explanatory to outside users really helps.

I've renamed it to QueryData. With that huge roadblock out of our way,
I hope you're much more likely to help. ;)

Bye,
Waldemar Kornewald

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Alex Gaynor]

On Thu, Oct 29, 2009 at 2:44 PM, Waldemar Kornewald
 wrote:
>
> Hi,
> Russell and Alex, did you already look at QueryGlue? We really need to
> discuss which branch the new query_class() should be in.
>
> Bye,
> Waldemar Kornewald
>
> >
>

I haven't had a chance to look at it, and I probably won't until at
least a few of the items on my plate are dealt with.  That being said
I am extremely leery about investing time in something with names like
"QueryGlue" as to me they imply a lack of organization in the code,
and that may of may not be true, but giving things name thats are at
least somewhat explanatory to outside users really helps.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Waldemar Kornewald]

Hi,
Russell and Alex, did you already look at QueryGlue? We really need to
discuss which branch the new query_class() should be in.

Bye,
Waldemar Kornewald

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Waldemar Kornewald]

On Mon, Oct 26, 2009 at 3:05 PM, Russell Keith-Magee
 wrote:
>
> On Mon, Oct 26, 2009 at 8:46 PM, Waldemar Kornewald
>  wrote:
>>
>> On Mon, Oct 26, 2009 at 1:12 PM, Russell Keith-Magee
>>  wrote:
>>> To date, sql.Query is the right structure for all Django's supported
>>> backends. It might even be the right structure for a non-SQL backend
>>> that provides a SQL-like query layer (AppEngine possibly falls into
>>> this category, as might a SimpleDB backend). However, a CouchDB,
>>> Cassandra or MongoDB backend probably won't get much traction using an
>>> internal query structure that talks about Joins and Where clauses.
>>
>> App Engine's datastore API is more similar to MongoDB than SQL. Even
>> on SimpleDB I don't think that the Where tree is a good idea because
>> it's way too SQL-specific.
>
> Exactly my point. There is no such thing as a "generic" internal
> query. The closest we can hope for is a common interface for objects
> that can have Qs, filters, et all added to them. sql.Query interprets
> those Q's and filters as joins. Other backends will require other
> interpretations.
> [...]
> I need to take a closer look at QueryGlue to be able to offer any
> deeper critique of this. I'll put this on my todo list.

Yes, that'll help in our discussions and I hope it'll make clearer why
query_class() should rather be implemented in our branch instead of
multi-db (which already works the way it is - withour query_class()).

Here's the link:
http://bitbucket.org/wkornewald/django-nonrel-hacked/src/tip/django/db/models/queryglue.py
What QueryGlue does is something like this (though, it's simplified):
queryset.filter(bla__attr=3)
=> gets translated to =>
queryglue.filters_tree.add(( ['bla', 'attr'], 'exact', 3 ))

As you can see, there isn't anything backend-specific in the
filters_tree. It's actually not even that much different from what
sql.Query.add_filter() already does - just without adding information
about joins and other SQL-specific stuff.

Now, an SQL backend can just iterate over filters_tree and call
sql.Query.add_filter() for each child in the tree - this would be the
easiest way to make sql.Query work again in our code. OTOH, the
non-relational backends could inspect the tree and possibly execute
multiple queries - one for each table involved in the query - and then
join the result set in memory (depending on the query and your data
this can be inefficient - or efficient).

Bye,
Waldemar Kornewald

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Russell Keith-Magee]

On Mon, Oct 26, 2009 at 8:46 PM, Waldemar Kornewald
 wrote:
>
> On Mon, Oct 26, 2009 at 1:12 PM, Russell Keith-Magee
>  wrote:
>> To date, sql.Query is the right structure for all Django's supported
>> backends. It might even be the right structure for a non-SQL backend
>> that provides a SQL-like query layer (AppEngine possibly falls into
>> this category, as might a SimpleDB backend). However, a CouchDB,
>> Cassandra or MongoDB backend probably won't get much traction using an
>> internal query structure that talks about Joins and Where clauses.
>
> App Engine's datastore API is more similar to MongoDB than SQL. Even
> on SimpleDB I don't think that the Where tree is a good idea because
> it's way too SQL-specific.

Exactly my point. There is no such thing as a "generic" internal
query. The closest we can hope for is a common interface for objects
that can have Qs, filters, et all added to them. sql.Query interprets
those Q's and filters as joins. Other backends will require other
interpretations.

>> So - the intention is to repurpose query_class() slightly. Once
>> refactored, query_class() will be required to return a class that
>> implements the Query interface. sql.Query is the only example at
>> present, but other backends can provide other internal
>> representations. The call to query_class() will be made in QuerySet -
>> not as part of the sql.Query construction. In this way, query_class()
>> becomes the "get me the actual implementation" method on the backend.
>
> Why do you want to implement this in multi-db if it's only useful for
> non-SQL support? Shouldn't you better keep multi-db as-is and add the
> query_class() feature to our branch? That would save us lots of
> conflicts because won't have to implement our code twice (once for the
> old query_class and once for your version) and we'll probably have to
> change your query_class, anyway.

Because the way query_class() is currently used causes other problems.
Providing an entry point for multi-db is a bonus.

>> We're *not* trying to build a completely generic internal query
>> representation. I'm not convinced that such an animal is even possible
>> in the general case - again, JOIN means something to relational
>> databases, but doesn't mean much to non-SQL databases. If AppEngine is
>> able to leverage some of the sql.Query internals, thats great - but I
>> don't expect that this will be the default situation.
>
> Does this mean you'll remove QuerySet.using()? Otherwise you'd have to
> transform an sql.Query to an appengine.Query.

QuerySet.using() will continue to exist. However, I expect there will
be some restrictions on when you can call it. Retasking across backend
types will be one of those restrictions.

> If the generic query representation is not much more detailed than Q
> objects then I don't see a big problem, anyway (our QueryGlue can be
> easily transformed into sql.Query or any other query type exactly for
> that reason). The point why we need QueryGlue is that the queries will
> have to be manipulated and interpreted in order to emulate certain
> features (e.g., joins) and its much easier to do this on the final
> query tree than on its intermediate states.

I need to take a closer look at QueryGlue to be able to offer any
deeper critique of this. I'll put this on my todo list.

Yours,
Russ Magee %-)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Waldemar Kornewald]

On Mon, Oct 26, 2009 at 1:12 PM, Russell Keith-Magee
 wrote:
> To date, sql.Query is the right structure for all Django's supported
> backends. It might even be the right structure for a non-SQL backend
> that provides a SQL-like query layer (AppEngine possibly falls into
> this category, as might a SimpleDB backend). However, a CouchDB,
> Cassandra or MongoDB backend probably won't get much traction using an
> internal query structure that talks about Joins and Where clauses.

App Engine's datastore API is more similar to MongoDB than SQL. Even
on SimpleDB I don't think that the Where tree is a good idea because
it's way too SQL-specific.

> So - the intention is to repurpose query_class() slightly. Once
> refactored, query_class() will be required to return a class that
> implements the Query interface. sql.Query is the only example at
> present, but other backends can provide other internal
> representations. The call to query_class() will be made in QuerySet -
> not as part of the sql.Query construction. In this way, query_class()
> becomes the "get me the actual implementation" method on the backend.

Why do you want to implement this in multi-db if it's only useful for
non-SQL support? Shouldn't you better keep multi-db as-is and add the
query_class() feature to our branch? That would save us lots of
conflicts because won't have to implement our code twice (once for the
old query_class and once for your version) and we'll probably have to
change your query_class, anyway.

> We're *not* trying to build a completely generic internal query
> representation. I'm not convinced that such an animal is even possible
> in the general case - again, JOIN means something to relational
> databases, but doesn't mean much to non-SQL databases. If AppEngine is
> able to leverage some of the sql.Query internals, thats great - but I
> don't expect that this will be the default situation.

Does this mean you'll remove QuerySet.using()? Otherwise you'd have to
transform an sql.Query to an appengine.Query.

If the generic query representation is not much more detailed than Q
objects then I don't see a big problem, anyway (our QueryGlue can be
easily transformed into sql.Query or any other query type exactly for
that reason). The point why we need QueryGlue is that the queries will
have to be manipulated and interpreted in order to emulate certain
features (e.g., joins) and its much easier to do this on the final
query tree than on its intermediate states.

Bye,
Waldemar Kornewald

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Russell Keith-Magee]

On Mon, Oct 26, 2009 at 4:52 PM, Waldemar Kornewald
 wrote:
>
> On Mon, Oct 26, 2009 at 1:46 AM, Russell Keith-Magee
>  wrote:
>> The current query_class will need to change slightly to support
>> multi-db, so anything you implement against that interface will
>> require some rework later on. That said, the fundamental approach
>> (i.e., the backend tells you what class to use for queries) will still
>> be there - it will just be used in a slightly different way.
>
> In the SVN multi-db branch there is a modified query_class() API.
> OTOH, on github it got replaced with SQLCompiler. Are the
> query_class() changes already committed somewhere?

No, they haven't been developed yet. Alex and I did the initial design
work at the DjangoCon sprints, but we haven't actually implemented
anything yet.

> Why do you still need query_class() if you already have SQLCompiler?
> If this is just about making non-SQL backends work then you'll need
> some kind of backend-independent query representation, so
> QuerySet.using() can be supported. That's exactly what we've already
> done with QueryGlue, so maybe you should better reuse what we've
> started and finish that together with us, so we all don't waste time
> on refactoring everything twice?

There are two different agents at work here.

We need to split sql.Query from QueryCompiler to support the fact that
the same SQL-like query needs to be rendered in different ways by
different backends. This can be as simple as the character used for
quoting, or as complex as wrapper clauses needed to handle LIMIT and
OFFSET.

There is a separate issue of determining if sql.Query is the right
internal structure to use for representing a query.

To date, sql.Query is the right structure for all Django's supported
backends. It might even be the right structure for a non-SQL backend
that provides a SQL-like query layer (AppEngine possibly falls into
this category, as might a SimpleDB backend). However, a CouchDB,
Cassandra or MongoDB backend probably won't get much traction using an
internal query structure that talks about Joins and Where clauses.

So - the intention is to repurpose query_class() slightly. Once
refactored, query_class() will be required to return a class that
implements the Query interface. sql.Query is the only example at
present, but other backends can provide other internal
representations. The call to query_class() will be made in QuerySet -
not as part of the sql.Query construction. In this way, query_class()
becomes the "get me the actual implementation" method on the backend.

We're *not* trying to build a completely generic internal query
representation. I'm not convinced that such an animal is even possible
in the general case - again, JOIN means something to relational
databases, but doesn't mean much to non-SQL databases. If AppEngine is
able to leverage some of the sql.Query internals, thats great - but I
don't expect that this will be the default situation.

Yours,
Russ Magee %-)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Waldemar Kornewald]

On Mon, Oct 26, 2009 at 1:46 AM, Russell Keith-Magee
 wrote:
> The current query_class will need to change slightly to support
> multi-db, so anything you implement against that interface will
> require some rework later on. That said, the fundamental approach
> (i.e., the backend tells you what class to use for queries) will still
> be there - it will just be used in a slightly different way.

In the SVN multi-db branch there is a modified query_class() API.
OTOH, on github it got replaced with SQLCompiler. Are the
query_class() changes already committed somewhere?

Why do you still need query_class() if you already have SQLCompiler?
If this is just about making non-SQL backends work then you'll need
some kind of backend-independent query representation, so
QuerySet.using() can be supported. That's exactly what we've already
done with QueryGlue, so maybe you should better reuse what we've
started and finish that together with us, so we all don't waste time
on refactoring everything twice?

Bye,
Waldemar Kornewald

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Russell Keith-Magee]

On Mon, Oct 26, 2009 at 12:42 AM, Thomas Wanschik
 wrote:
>
>
>
> On 22 Okt., 23:52, Russell Keith-Magee  wrote:
>> On Fri, Oct 23, 2009 at 5:09 AM, Thomas Wanschik
>>
>>  wrote:
>>
>> >> When a new QuerySet is instantiated (e.g. by calling
>> >> Model.objects.all()) it asks the backend for its Query class and then
>> >> creates an instance of that class. By default, this class is
>> >> sql.Query. Only the Oracle backend has its own Query which subclasses
>> >> sql.Query.
>>
>> > I think Waldermar wanted to say: 'Only the Oracle backend has its own
>> > Query which subclasses sql.query.BaseQuery.' instead of 'sql.Query'.
>>
>> I should point out that this is one of the specific problems Alex and
>> I are trying to address in the multi-db refactor. When we've finished,
>> returning the right query class should be as simple as implementing an
>> API on the backend.
>>
>
> Thanks for your answer Russell. But i have one question left. Should
> we make the effort and clean the app engine backend up in the way the
> oracle backend is done (using a query_class) or should we wait for the
> multi-db refactor and then clean up our code according to multi-db?
> Will it be easier to merge the backend into django then?

The current query_class will need to change slightly to support
multi-db, so anything you implement against that interface will
require some rework later on. That said, the fundamental approach
(i.e., the backend tells you what class to use for queries) will still
be there - it will just be used in a slightly different way.

If you want to write (and test) code now, my suggestion would be to
try making your code as clean as possible against the current
interface, with the expectation that there will be some rework once
multi-db lands. The corollary to this is that if you find yourself
needing to make weird and widespread engineering decisions in order to
support the query_class approach, you should stop and wait for
multi-db to land.

Yours
Russ Magee %-)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Thomas Wanschik]

On 22 Okt., 23:52, Russell Keith-Magee  wrote:
> On Fri, Oct 23, 2009 at 5:09 AM, Thomas Wanschik
>
>  wrote:
>
> >> When a new QuerySet is instantiated (e.g. by calling
> >> Model.objects.all()) it asks the backend for its Query class and then
> >> creates an instance of that class. By default, this class is
> >> sql.Query. Only the Oracle backend has its own Query which subclasses
> >> sql.Query.
>
> > I think Waldermar wanted to say: 'Only the Oracle backend has its own
> > Query which subclasses sql.query.BaseQuery.' instead of 'sql.Query'.
>
> I should point out that this is one of the specific problems Alex and
> I are trying to address in the multi-db refactor. When we've finished,
> returning the right query class should be as simple as implementing an
> API on the backend.
>

Thanks for your answer Russell. But i have one question left. Should
we make the effort and clean the app engine backend up in the way the
oracle backend is done (using a query_class) or should we wait for the
multi-db refactor and then clean up our code according to multi-db?
Will it be easier to merge the backend into django then?

Bye,
Thomas Wanschik

> Yours,
> Russ Magee %-)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Thomas Wanschik]

I just want to remind contributers to fill in the cell "Assigned to"
and "Status" in the task spreadsheet while working on a specific task
in order to prefend problems.
Here is the link:
https://spreadsheets.google.com/ccc?key=0AnLqunL-SCJJdE1fM0NzY1JQTXJuZGdEa0huODVfRHc=en

Bye,
Thomas Wanschik

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Russell Keith-Magee]

On Fri, Oct 23, 2009 at 5:09 AM, Thomas Wanschik
 wrote:
>
>> When a new QuerySet is instantiated (e.g. by calling
>> Model.objects.all()) it asks the backend for its Query class and then
>> creates an instance of that class. By default, this class is
>> sql.Query. Only the Oracle backend has its own Query which subclasses
>> sql.Query.
>
> I think Waldermar wanted to say: 'Only the Oracle backend has its own
> Query which subclasses sql.query.BaseQuery.' instead of 'sql.Query'.

I should point out that this is one of the specific problems Alex and
I are trying to address in the multi-db refactor. When we've finished,
returning the right query class should be as simple as implementing an
API on the backend.

Yours,
Russ Magee %-)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Thomas Wanschik]

http://code.google.com/p/live-android/



On 22 Okt., 13:35, Waldemar Kornewald  wrote:
> Hi everyone,
> this rather long mail contains a status report and instructions for
> contributors and implementation notes for Django core developers. If
> you only want to know the status you can stop after the first section.
> If you want to contribute I hope this provides a good starting point
> into our port.
>
> ---
> Status report
>
> We've got pretty far with our App Engine port. For example, the
> sessions db and cached_db backends both work unmodified on App Engine.
> You can also order results and use basic filter()s as supported by the
> low-level App Engine API (gt, gte, lt, lte, exact, pk__in). You can
> also use QuerySet.order_by(), .delete(), .count(), Model.save(),
> .delete().
>
> This is our second porting attempt (it's not in the old repository).
> Our first attempt had too many conflicts with the multi-db branch
> (esp. the one on github). This time we just hacked everything
> together. We didn't concentrate on cleaning up the current backend
> API. We've also disabled SQL support.
>
> The next step is to move all the hacks into a nice backend API (at the
> same time making sure that it won't conflict with multi-db) and
> re-enable SQL support. That's where we need help. Also, if you want to
> work on SimpleDB support this is the right time to join. The App
> Engine backend itself can be handled by Thomas Wanschik and me -
> contributions in this area are not absolutely necessary, so please
> concentrate on the cleanup if you want to help.
>
> Now to the details (for those who want to contribute).
>
> ---
> Introducing QueryGlue
>
> The old Django code was distributed across three layers:
> * django.db.models.queryset.QuerySet
> * django.db.models.sql.query.Query (from now on just sql.Query)
> * backend
>
> When a new QuerySet is instantiated (e.g. by calling
> Model.objects.all()) it asks the backend for its Query class and then
> creates an instance of that class. By default, this class is
> sql.Query. Only the Oracle backend has its own Query which subclasses
> sql.Query.

I think Waldermar wanted to say: 'Only the Oracle backend has its own
Query which subclasses sql.query.BaseQuery.' instead of 'sql.Query'.

We added a module called 'nonrelational' to django.db.models. There we
define query.py which itself defines a BaseQuery class (the equivalent
to sql.query.BaseQuery). Just like Oracle subclasses
sql.query.BaseQuery we could use the same mechanism and subclass
nonrelational.BaseQuery in order to get the GAEQuery. For now this
subclassing mechanism is called in the module sql.query (not sql.Query
from above) by connection.ops.query_class(BaseQuery) and uses
sql.query.BaseQuery as the BaseQuery. We need some way of telling
which class should be the BaseQuery. This could be done via the
settings, something similar to settings.DATABASE_TYPE =
'nonrelational' and for sql it would be settings.DATABASE_TYPE= 'sql'.
According to this the right BaseQuery could be chosen. Additionally
these mechanisme shouldn't be called in sql.query. But that's only a
proposal of making the port a little bit more clean. I don't know if
there are any conflicts with multi-db this way. What do you guys think
of it?

>
> Normally, sql.Query builds the query on-the-fly. Whenever you call
> QuerySet.filter() the filters get put into a
> Q() and passed to
> sql.Query.add_q( Q(...) ).
> This function iterates over all filter rules in the Q object and calls
> sql.Query.add_filter() for each individual filter.
> This in turn directly modifies sql.Query.where which is a tree
> structure that represents the WHERE clause. It already contains
> information about the JOIN type for each filter (INNER, OUTER), the
> fields that get referenced by the filter, the column and table
> aliases, and so on. It already does a lot of what we need for
> non-relational backends, but it's too SQL-specific.
>
> The current behavior is also a problem for multi-db because it makes
> too many assumptions about the storage format of the filter rules. The
> user could call QuerySet.using(other_connection) anytime, so QuerySet
> shouldn't really work with the low-level sql.Query class before it
> actually executes the query.
>
> We've solved this problem by introducing a backend-independent query
> representation between QuerySet and the low-level Query (sql.Query,
> appengine.Query, etc.). This representation is called QueryGlue. You
> can find it in django.db.models.queryglue. It
> provides almost exactly the same "public" API as sql.Query (so it can
> easily be integrated with QuerySet). Each filter() call gets
> translated into a tree structure that is inspired by sql.Query.where,
> but it doesn't contain any information about the kind of JOIN.
> Instead, it stores high-level important information like whether we're
> filtering on a primary key, which 

Re: non-relational DB

[Andi Albrecht]

On Thu, Oct 22, 2009 at 2:07 PM, Russell Keith-Magee
 wrote:
>
> On Thu, Oct 22, 2009 at 7:46 PM, Waldemar Kornewald
>  wrote:
>>
>> Hi again,
>> now a little question:
>>
>> Some fields do type conversions. For example, TimeField converts
>> datetime objects into time objects.
>> App Engine doesn't support time, but only datetime, so should we do
>> such conversions at the backend level or should we expect the field to
>> handle it (esp. if it already has such conversion code)?
>
> I'm unsure what problem you're having here. The backend needs to
> return a type that the TimeField can turn into a Python Time object.
> TimeField is fairly liberal in what it will accept - DateTime objects,
> Time objects, and strings that express a time will all be handled.
>
> As long as your backend returns one of these acceptable types, you're done.
>
>> What's the status of the email backends ticket? There hasn't been any
>> reply to Andi Albrecht's latest patch and comment.
>> http://code.djangoproject.com/ticket/10355
>> This is essential for supporting all kinds of cloud platforms.
>
> We're in the process of doing feature voting for v1.2. Personally, I'm
> happy with the state of the patch, but there have been a couple of -1
> votes for the patch, which means that some people still need to be
> convinced that it's the right thing to do. Once voting is finished, we
> may need to revisit this issue on django-dev.

To give a short feedback. I'm still there and I've read most of the
comments given in the voting sheet. I'd happy to address the concerns
once voting is finished - in a different thread, of course :)

>
> Yours,
> Russ Magee %-)
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Waldemar Kornewald]

On Thu, Oct 22, 2009 at 2:07 PM, Russell Keith-Magee
 wrote:
>
> On Thu, Oct 22, 2009 at 7:46 PM, Waldemar Kornewald
>  wrote:
>>
>> Hi again,
>> now a little question:
>>
>> Some fields do type conversions. For example, TimeField converts
>> datetime objects into time objects.
>> App Engine doesn't support time, but only datetime, so should we do
>> such conversions at the backend level or should we expect the field to
>> handle it (esp. if it already has such conversion code)?
>
> I'm unsure what problem you're having here. The backend needs to
> return a type that the TimeField can turn into a Python Time object.
> TimeField is fairly liberal in what it will accept - DateTime objects,
> Time objects, and strings that express a time will all be handled.
>
> As long as your backend returns one of these acceptable types, you're done.

Great. I just wasn't sure if this was just an internal implementation
detail which we better shouldn't rely on in our backends.

Bye,
Waldemar Kornewald

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Russell Keith-Magee]

On Thu, Oct 22, 2009 at 7:46 PM, Waldemar Kornewald
 wrote:
>
> Hi again,
> now a little question:
>
> Some fields do type conversions. For example, TimeField converts
> datetime objects into time objects.
> App Engine doesn't support time, but only datetime, so should we do
> such conversions at the backend level or should we expect the field to
> handle it (esp. if it already has such conversion code)?

I'm unsure what problem you're having here. The backend needs to
return a type that the TimeField can turn into a Python Time object.
TimeField is fairly liberal in what it will accept - DateTime objects,
Time objects, and strings that express a time will all be handled.

As long as your backend returns one of these acceptable types, you're done.

> What's the status of the email backends ticket? There hasn't been any
> reply to Andi Albrecht's latest patch and comment.
> http://code.djangoproject.com/ticket/10355
> This is essential for supporting all kinds of cloud platforms.

We're in the process of doing feature voting for v1.2. Personally, I'm
happy with the state of the patch, but there have been a couple of -1
votes for the patch, which means that some people still need to be
convinced that it's the right thing to do. Once voting is finished, we
may need to revisit this issue on django-dev.

Yours,
Russ Magee %-)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: non-relational DB

[Waldemar Kornewald]

Hi again,
now a little question:

Some fields do type conversions. For example, TimeField converts
datetime objects into time objects.
App Engine doesn't support time, but only datetime, so should we do
such conversions at the backend level or should we expect the field to
handle it (esp. if it already has such conversion code)?

What's the status of the email backends ticket? There hasn't been any
reply to Andi Albrecht's latest patch and comment.
http://code.djangoproject.com/ticket/10355
This is essential for supporting all kinds of cloud platforms.

Bye,
Waldemar Kornewald

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---