Re: More multi-database plumbing (WAS Re: Changing DatabaseWrapper._cursor() to take a dict instead of settings)

[Alex Gaynor]

On Sat, Mar 14, 2009 at 8:35 AM, koenb  wrote:

>
> > Is there any downside to checking this in immediately? It's a simple
> > enough change that I don't see it introducing any bugs (famous last
> > words, right?)...
>
> Please don't allow using connections from different database backends
> unless the problems of using the correct database options are resolved
> first. Both model fields and query creation still have a lot of uses
> of the 'default' connection to get to these options. This implies that
> there is a possibility to get hard to debug weird sql trouble when
> mixing database engines.
> Enforcing the same backend for the different connections might be a
> good idea for now.
>
> Koen
> >
>
How is that any different from the present scenario if one assigns to
query.query.connection?

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." --Voltaire
"The people's good is the highest law."--Cicero

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: More multi-database plumbing (WAS Re: Changing DatabaseWrapper._cursor() to take a dict instead of settings)

[koenb]

> Is there any downside to checking this in immediately? It's a simple
> enough change that I don't see it introducing any bugs (famous last
> words, right?)...

Please don't allow using connections from different database backends
unless the problems of using the correct database options are resolved
first. Both model fields and query creation still have a lot of uses
of the 'default' connection to get to these options. This implies that
there is a possibility to get hard to debug weird sql trouble when
mixing database engines.
Enforcing the same backend for the different connections might be a
good idea for now.

Koen
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: More multi-database plumbing (WAS Re: Changing DatabaseWrapper._cursor() to take a dict instead of settings)

[Adrian Holovaty]

On Fri, Mar 13, 2009 at 11:12 AM, Alex Gaynor  wrote:
> To give an idea of what I've suggesting I drew up a small diff(full test
> passage): http://dpaste.com/14018/

I like this. It's straightforward, and it's backwards-compatible. One
particularly nice thing about it is that it gives people the freedom
to register their own connections dynamically by assigning to
django.db.connections -- i.e., they wouldn't be bound to what's
hard-coded in the DATABASES setting.

Is there any downside to checking this in immediately? It's a simple
enough change that I don't see it introducing any bugs (famous last
words, right?)...

Then, the next step would be to change django.db.transactions to use
this connection list, as Alex pointed out in a previous e-mail in this
thread.

Adrian

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: More multi-database plumbing (WAS Re: Changing DatabaseWrapper._cursor() to take a dict instead of settings)

[Alex Gaynor]

On Fri, Mar 13, 2009 at 7:57 AM, Jeremy Dunck  wrote:

>
> On Fri, Mar 13, 2009 at 7:55 AM, Shai Berger  wrote:
> ...
> > Or has some other solution surfaced lately for distributed transactions?
>
> A big fat documentation warning and careful selection of distribution
> strategy?
>
> >
>
To give an idea of what I've suggesting I drew up a small diff(full test
passage): http://dpaste.com/14018/ .  This would let someone who created
their DB tables access the hooks in the ORM fairly seamlessly(since they can
grab connections off of here and put them into their queries in all of 3
lines of code).  The question would be is it reasonable to have *just* this
in, and if not where is the line for stuff we want to put in without having
a full public API.  The other question is does everyone actually like this
API for defining connections, I know the previous multi db suggestion used a
class based approach with registration and this doesn't lend itself to the
registration mechanism as easily, although it is more inline with how other
settings are defined, especially since I view this is as more a way to just
define the connections, and the mechanisms for defining their usage is
elsewhere.

I realize I offered more questions than answers, but the code should count
for somethign :).

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." --Voltaire
"The people's good is the highest law."--Cicero

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: More multi-database plumbing (WAS Re: Changing DatabaseWrapper._cursor() to take a dict instead of settings)

[koenb]

> My "nuts and bolts" document ...

Hi Alex,

about a year ago, I spent a few weeks working on multiple database
support myself (building further on the code in ticket #4747).

I used a mercurial trac setup by Ben Ford to store the work and some
notes. It can be found at http://trac.woe-beti.de. (It was down for a
while, but Ben was so kind to put it up again).

Obviously, most of what was done then is very much out of date now
(last commit is from 5/29/08), but it may help you get some extra
ideas.

You may want to check out the notes in the wiki to see what was done.
There is also a discussion of use cases and some API ideas.

Some of the items that were covered were:
- declarative style connection definitions
- model registration
- per connection transaction management
- changes to the sql generation (WhereNode) in order to produce
correct sql on different database backends
- modification to field definitions: there were/are a lot of backend
specific things in there (quoting and stuff)
- definition of a ConnectionCommand management command class so you
can specify a connection for certain management commands
- ...

Important to note is I wanted to get this to work with different
database backends (getting data out of legacy databases), hence the
focus on the backend specific differences.

I was discouraged by the lack of interest back then so I gave up
(getting my data for my project was easy enough without it), I hope
you have some more success.

Good luck,

Koen
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: More multi-database plumbing (WAS Re: Changing DatabaseWrapper._cursor() to take a dict instead of settings)

[Alex Gaynor]

On Fri, Mar 13, 2009 at 12:26 AM, Adrian Holovaty wrote:

>
> On Thu, Mar 12, 2009 at 11:53 PM, Alex Gaynor 
> wrote:
> > Great, since we've moved so quickly on improving our plumbing for this
> there
> > was one other idea I wanted to put forth(since I believe it needs to
> happen
> > for any multidb implementation).  In terms of plumbing the cornerstone of
> my
> > multidb proposal is the addition of a django.db.connections object which
> > lazily returns connections by subscripting based on a DATABASES option
> which
> > is a dictionary of dictionaries.  I think we could add support for this
> > relatively easily, with full backwards compatibility.  However, we only
> have
> > 1 week until feature freeze so I wouldn't want to proceed with this
> unless
> > everyone was completely comfortable with that proposed API, but I figured
> > I'd bring it up since we have moved so quickly on these past bits.
>
> Yeah, a collection of connections (called django.db.connections or
> whatever) is one of the next logical steps for multi-database support.
> But for the time being, that's 90% just a pretty API and only 10%
> plumbing. It can be worked-around completely without changing Django
> internals. (I've got a DATABASES setting in my application, along with
> a custom manager that checks the setting to get connection-specific
> parameters -- code very similar to
> http://www.eflorenzano.com/blog/post/easy-multi-database-support-django/
> )
>
> Regarding the one-week-until-feature-freeze reality -- we haven't
> *hit* the feature freeze yet, so it seems like it's still fair game,
> but this *would* be a bigger change than the previous few, so it'd be
> worth discussing the pluses and minuses.
>
> A bigger issue (one that requires plumbing, unless I'm completely
> missing something) is to get django/db/transaction.py to be aware of
> the multiple connections. All of the functions in there deal directly
> with the global connection, and I don't immediately see an easy way to
> refactor things to work on separate connections. Any ideas? Alex, have
> you addressed this in the Grand Proposal of yours? :-)
>
> Adrian
>
> >
>
My "nuts and bolts" document(which I've alluded to several times but haven't
actually posted since my laptop has been dead :( ) has fixing up the
transcation stuff, specifically I put it after we had a basic public API,
but before we get into the more complicated bits in terms of the
implementation schedule.  That being said it looks like all the states there
is handled in a series of dictionaries keyed on thread id, so my instinct
would be to turn those into dictionaries of dictionaries keyed on thread id
and connection alias respectively and have each of the relevant functions
take an extra parameter, which defaults to None, which should be a list of
db aliases that the op should be preformed on(None indicicating all dbs),
this preserves backwards compatibility and makes sense in terms of the
common usage, for example at the end of a request I want to commit against
all dbs but when I create a savepoint I only actually create that on 1 db.

Having said all that I don't think that's stuff that necessarily needs to
happen when we add the connections object, since as it is our transaction
stuff doesn't come into play when someone grabs a cursor themselves with
even our single DB(this is filed as a bug and I imagine will get fixed for
1.1).

If no one has any seroius objections other than it's a sort of big addition
with not a ton of time and we don't want to mess of the public/private API
bridge I'll probably start coding something up tommorow.


Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." --Voltaire
"The people's good is the highest law."--Cicero

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

More multi-database plumbing (WAS Re: Changing DatabaseWrapper._cursor() to take a dict instead of settings)

[Adrian Holovaty]

On Thu, Mar 12, 2009 at 11:53 PM, Alex Gaynor  wrote:
> Great, since we've moved so quickly on improving our plumbing for this there
> was one other idea I wanted to put forth(since I believe it needs to happen
> for any multidb implementation).  In terms of plumbing the cornerstone of my
> multidb proposal is the addition of a django.db.connections object which
> lazily returns connections by subscripting based on a DATABASES option which
> is a dictionary of dictionaries.  I think we could add support for this
> relatively easily, with full backwards compatibility.  However, we only have
> 1 week until feature freeze so I wouldn't want to proceed with this unless
> everyone was completely comfortable with that proposed API, but I figured
> I'd bring it up since we have moved so quickly on these past bits.

Yeah, a collection of connections (called django.db.connections or
whatever) is one of the next logical steps for multi-database support.
But for the time being, that's 90% just a pretty API and only 10%
plumbing. It can be worked-around completely without changing Django
internals. (I've got a DATABASES setting in my application, along with
a custom manager that checks the setting to get connection-specific
parameters -- code very similar to
http://www.eflorenzano.com/blog/post/easy-multi-database-support-django/
)

Regarding the one-week-until-feature-freeze reality -- we haven't
*hit* the feature freeze yet, so it seems like it's still fair game,
but this *would* be a bigger change than the previous few, so it'd be
worth discussing the pluses and minuses.

A bigger issue (one that requires plumbing, unless I'm completely
missing something) is to get django/db/transaction.py to be aware of
the multiple connections. All of the functions in there deal directly
with the global connection, and I don't immediately see an easy way to
refactor things to work on separate connections. Any ideas? Alex, have
you addressed this in the Grand Proposal of yours? :-)

Adrian

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---