Re: multi-db branch
Hi Ed, There are a few people looking at multi-db at the moment. I've had to put my own contributions on hold because I've had rather a hectic personal life lately, but hopefully I'll be able to crack on with it next week or so. There has also been some discussion about rethinking some of the design of multi-db, especially in light of recent changes to the way backends work and forthcoming changes to querying, which is something I'd like to have a look at when I get back into it. Watch this space at the moment it seems! :-) Cheers Ben On 31/10/2007, Ed Summers <[EMAIL PROTECTED]> wrote: > > > On Oct 29, 3:31 pm, "James Bennett" <[EMAIL PROTECTED]> wrote: > > You probably want to start by reading messages here: > > > > http://groups.google.com/group/django-developers/search?q=multi-db... > > So it sounds as if the branch is dead, and there are patches being > attached to #4747? I'd be willing to pitch in some time to get the > multi-db code up to snuff, if there is a clear road ahead for what > needs to be done. If there isn't I can live with that too :-) > > //Ed > > > > > -- Regards, Ben Ford [EMAIL PROTECTED] +6281317958862 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: multi-db branch
On Oct 29, 3:31 pm, "James Bennett" <[EMAIL PROTECTED]> wrote: > You probably want to start by reading messages here: > > http://groups.google.com/group/django-developers/search?q=multi-db... So it sounds as if the branch is dead, and there are patches being attached to #4747? I'd be willing to pitch in some time to get the multi-db code up to snuff, if there is a clear road ahead for what needs to be done. If there isn't I can live with that too :-) //Ed --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: multi-db branch
On 10/29/07, Ed Summers <[EMAIL PROTECTED]> wrote: > Does anyone know if there are any plans to merge the multi-db branch > [1]. I see in ticket #1142 that things are kind of quiet on the > development side. Did this come up at all during the big spring of > 14-Sep-2007? You probably want to start by reading messages here: http://groups.google.com/group/django-developers/search?q=multi-db=0=d; -- "Bureaucrat Conrad, you are technically correct -- the best kind of correct." --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
multi-db branch
I posted this over in django users as well...was getting errors trying to post in this group. But I'd like to go ahead and attempt to take over maintenance of the multi-db branch I haven't done anything with the Django project as of yet, but I've got 10 yrs under my belt as a professional programmer, 2.5 yrs with Python. I've been using this branch since November, and would like to get it active again, so that it doesn't die off. I'm a bit lacking in subversion knowledge, as we don't do much with branches at my work and have just started moving to subversion from CVS, so I'm going to have to read the docs on that, but am going to attempt to start merging the changes from the trunk that have occured since JP's last merge with this branch. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: multi-db branch: looking for a new maintainer
Just wanted to put a note on here...that there are people actively using this branch. I've been using it since the beginning of December...and have found it to work fantastically... ( well I did send JP a bug report today ... with a sample project ... when he told me this news ). I do hope in the future to be able to pitch in on this branch and help out ( I did submit a how-to dock in the wiki) ...but am under some serious deadlines right now... so also wouldn't be able to devote the time needed to fully understand everything quickly to take it over... until I meet some deadlines in Feb/March At any rate...I did want to chime in and thank JP for his help when I was getting started...and tell him that this branch of the project is great!! On Jan 24, 3:50 pm, "JP" <[EMAIL PROTECTED]> wrote: > Hi all, > > I'm afraid that I am not going to have time in the future to continue > maintaining and merging up themulti-dbbranch. It's diverged so far > from trunk that any trunk change that touches management.py or models > results in hours of hand-merging and svn detective work... hours that I > just don't have to give right now. > > I'd hate to see the branch die, since I think it will be of use to > people, so I'm hoping that someone else will step forward to take over > ownership, until such a time as the branch is reviewed and merged or > rejected. Of course I'll be more than happy to help the new maintainer > get up to speed on the changes and new features in the branch, and will > always be available to answer questions about it -- as much as time > allows anyway. :) > > Any takers? > > JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
multi-db branch: looking for a new maintainer
Hi all, I'm afraid that I am not going to have time in the future to continue maintaining and merging up the multi-db branch. It's diverged so far from trunk that any trunk change that touches management.py or models results in hours of hand-merging and svn detective work... hours that I just don't have to give right now. I'd hate to see the branch die, since I think it will be of use to people, so I'm hoping that someone else will step forward to take over ownership, until such a time as the branch is reviewed and merged or rejected. Of course I'll be more than happy to help the new maintainer get up to speed on the changes and new features in the branch, and will always be available to answer questions about it -- as much as time allows anyway. :) Any takers? JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
> OK, thoughts, anyone? When i first heard of multi-db i more thought about support for clustering then moving some apps/models to other databases, which is great, too of course. for example: having one master db and some slave dbs that mirror the master db. requests should go to slaves in some sort of changeable round robin style and changes should go directly to the master db. this scenario popped into my mind the first time i heard of multi-db. is such a scenario supported somehow. i think it will be useful for very large sites and should be kept in mind when the db layer is redesigned. kindly regards daniel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
Replying to myself... here's what I've come up with to explain the problems I see in my current implementation and what I think should be done to fix them. Apologies in advance -- it's quite long. I've implemented a bit of this just to make sure it would work, mainly the basic parts in django.db.__init__.py. Still need to figure out how to come up with some tests for these issues that won't be slow as boiled slugs to run. Anyway, here's the writeup: Django can run as a WSGI app within a container like Paste, where the same app may be run with different settings at the same time in multiple simultaneous threads, and mutiple serial requests within the same thread. Below are listed a few changes that I think are required to make the multi-db branch's settings and connection handling safe for serving in that kind of enviroment. For the purposes of this discussion, assume that the app myproj.myapp is configured within the container with two sets of settings, SA and SB. Imagine that we have a server running threads T1 and T2, and answering requests R1, R2, ... etc. Proposed change: Clear django.db.connections on request finish, make django.db.connections thread-local. Where: django.db.__init__.py Rationale: Settings may change between requests, including what DATABASE_* settings are labeled with a given key in OTHER_DATABASES. django.db.connections is a LazyConnectionManager, so resetting its _connections property between requests will clear named connections for the next request. Of course, R1 may finish on T1 while R2 is running on T2, so we only want to clear the django.db.connections._connections in T1 -- so, django.db.connections._connections must become a thread-local value. Proposed change: Make django.db.connection, .backend, etc proxies Where: django.db.__init__.py Rationale: Apply the same isolation to the default connection and its backend, etc (which are all module-level variables in django.db) as to named connections in django.db.connections. Unfortunately in this case we don't have a convenient dict to clear out at the end of each request. And users may have 'from django.db import connection' anywhere in their code. So django.db.connection, django.db.backend, etc must be thread-local and also cleared at the end of each request, but without rebinding their names. All of these variables are references to attributes in django.db.connection_info, which is a ConnectionInfo instance intialized from the default settings (settings.DATABASE_*). The shortest route to making them request- and thread-safe is to make them instead attributes of a new DefaultConnectionInfoProxy instance, in which each attribute delegates to the same attribute in a ConnectionInfo stored in a thread-local container and reset at the end of each request. Proposed change: Move connection information into Manager.db Where: django.db.models.manager Rationale: Moving the per-model connection information from model._meta to model._default_manager has already been discussed and I think is generally approved -- this is just to flesh out a bit more what that will mean. Since the connection for a model will now be wholly specified by settings, the model's default manager will have to look up the connection to use for the model in settings.OTHER_DATABASES by examining the MODELS entry in each value in that dict for the best match to the model. A manager instance may be shared across threads, so the storage for the current connection must be thread-local. And the same manager may encounter different settings on different requests in the same thread, if (say) R1 in T1 loads SA but R2 also in T1 loads SB, so not only the ConnectionInfo but the whole connection scenario (named or default, name of connection if named, settings to use) must be reset at the end of each request. To do this, I propose making manager.db a descriptor that serves as a thread-local ConnectionInfo proxy. The proxy can be initialized during contribute_to_class and attached only to the instance (not the Manager class). In use, where the current multi-db code has an access pattern like: connection = model._meta.connection The new pattern would be: connection = model._default_manager.db.connection Or in the manager itself: connection = self.db.connection backend = self.db.backend and so on. The model._default_manager.db descriptor will (on first access) initialize and return the correct ConnectionInfo for the current settings state that applies to the manager's model, caching that ConnectionInfo in thread-local storage for the rest of the request and connecting to the request_finished signal to clear the cache. This way, only Managers used during a request would reset at the end of the request. Proposed change: Add a thread-local cache of connections Where: django.db.__init__.py Rationale: Efficiency. Settings may change between requests, but the actual database connection picked out by the tuple
Re: Multi-db branch feedback/questions
[snip most of OTHER_DATABASES proposal] > * ``MODELS`` is a list. Each item in the list may be: > > * ``app_label.module_name`` if a single model should be use this > connection. > * ``app_label`` if the entire app should use this connection > > How does this sound? Good enough for me. Though I weep for all of the tests that are about to break. :) I'm working on a long writeup of the thread/request isolation issues and ideas for how to deal with them that I'll post up later today. I'd like to get some feedback on that stuff before I start hacking away, the changes required look large and tricky. JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
> What about making Manager callable? It could take a database > identifier, or even an object with connection info, and return a new > Manager instance with the specified database info. This would let you > keep using Model.objects with a default database, or for the case where > multiple databases are not specified. But then you could do something > like this: > > Person.objects(db).filter(...) +1 from me. JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Jul 11, 2006, at 8:08 AM, jason pellerin wrote:
> Here's another case that I'd like to see handled gracefully, that I
> think can be handled better with your proposed syntax than mine: one
> app with many models, some of which use a named connection. To me, the
> ideal way to handle that would be to allow entries in APPS to be
> tuples of (app name, [model, model, ...]), like: ('myproject',
> ['ModelA', 'ModelB']) to indicate that the setting applies only to the
> selected models from the app. If apps are keys in DATABASES, you'd
> wind up with keys something like (appname, (model, model, ...)), which
> is pretty grim.
Yeah, that was pretty much my thinking.
After sleeping on it, here's my concrete proposal on how all this
should work. Much of it I think meshes with what's you've already
coded, but there are a few changes:
* All of the current ``DATABASE_*`` remain the same -- they specify
the "main" database. NO change from what you've already god.
* Additionally, the ``DATABASE_*`` settings can be considered
"defaults" for the extra databases (so if you have a bunch of
PostgreSQL databases with the same connection info you only need to
define ``DATABASE_NAME`` for each of the extra DBs. Again, I think
this is the same as what you've already done.
* Other databases are defined in a dict ``OTHER_DATABASES``. I want
to rename this because ``DATABASES`` implies that *all* the databases
go in the dict, instead of just the extra ones. I'm open to other
names, of course, but I want something that makes its secondary
nature pretty obvious.
* The structure of ``OTHER_DATABASES`` is as you've made it, modulo
the name change.
* By default, any app in ``INSTALLED_APPS`` is installed into the
main database. However, each value in ``OTHER_DATABASES`` contains
an ``MODELS`` key which defines which models are installed in that
database.
* ``MODELS`` is a list. Each item in the list may be:
* ``app_label.module_name`` if a single model should be use this
connection.
* ``app_label`` if the entire app should use this connection
How does this sound?
Jacob
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
A wrinkle occurred to me last night that I really should have thought
of earlier, and that calls into question my whole connection-handling
setup.
When served as a WSGI app in a container like Paste, settings can be
different on every request. That means, first off, that however we
decide to structure the DATABASES setting, the lookup of app.Model ->
named connection has to be done at request startup. Second, by making
each connection (that is, backend.DatabaseWrapper) take a settings
argument and keep it around, I've broken the current mechanism for
connection request isolation (close the connection at the end of the
request, so it is forced to get a new cursor with possibly new
settings at the start of the next request). Though I'm not sure how
well that method works when app A sees two different DATABASE_ENGINE
settings on two different requests served from the same thread, so
maybe I'm misunderstanding something there.
Anyway, I hope that we can come up with some ideas here for how to
efficiently handle resetting all connection particulars on each
request for both named connections in db.connections, and the default
connection in db.connection (and all of its accessory bits like
backend, etc). I don't have anything concrete to propose right now,
still trying to think it all through. Any ideas, anyone?
> > I'd like to see this done the other way around, which the DATABASES
> > settings defining which *models* are used. I'm picturing something
> > like this::
> >
> > DATABASES = {
> > 'a' : {
> > 'DATABASE_ENGINE' : 'sqlite3',
> > 'DATABASE_NAME': '/whatever/foo.db',
> > 'APPS' : ['myproject', 'yourproject.widget',
> > 'someotherapp']
> > }
> > }
Here's another case that I'd like to see handled gracefully, that I
think can be handled better with your proposed syntax than mine: one
app with many models, some of which use a named connection. To me, the
ideal way to handle that would be to allow entries in APPS to be
tuples of (app name, [model, model, ...]), like: ('myproject',
['ModelA', 'ModelB']) to indicate that the setting applies only to the
selected models from the app. If apps are keys in DATABASES, you'd
wind up with keys something like (appname, (model, model, ...)), which
is pretty grim.
JP
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
Trying not to repeat myself too much while arguing for my fast-fading
favorite... :)
> Yeah, you really better, since I don't like it one bit :)
>
> Currently the settings file for lawrence.com has... (/me checks)...
> 48 entries. You mean I've got to duplicate my database settings 48
> times!?
Only if they use 48 different databases. Anything not in DATABASES
would use the default connection. Any database settings that are shared
among > 1 app could be put into a variable ... or the keys in DATABASES
could be tuples.
> I like the idea of keys in ``DATABASES`` being "identifiers" for the
> database connection; this lets something like the following work::
>
> from django import db
> conn = db.get_connection("main")
>
> Which I very much like...
Me too, but I don't know how to reconcile it with configuring which
apps use which settings. I'd be equally happy with:
from django import db
conn = db.get_connection_for('myproj.myapp')
Which is almost as good, with a little extra goodness in not having to
remember anything but the app name to get the connection for that app.
> That makes ``DATABASES`` the canonical location
> for, well, database info, and that's good.
Yes, definitely.
JP
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
Another use case to mull around: I'm currently using django for basically everything *but* it's ORM, in large part due to not supporting multiple databases. We have one software package with many nearly-identical databases (for different customers). The models would be the same, but I'd want to decide where to save them at runtime, not as a Meta option. Creating a custom manager for each database would work, but with 10+ databases, it could pollute the namespace, and be tedious to write. > > I posted a couple of example use-cases in response to Malcolm. > > Manually setting ``db_connection`` is ugly but workable; I'll think > > some more about a more elegant way of doing it. What about making Manager callable? It could take a database identifier, or even an object with connection info, and return a new Manager instance with the specified database info. This would let you keep using Model.objects with a default database, or for the case where multiple databases are not specified. But then you could do something like this: Person.objects(db).filter(...) Dan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
> Under your proposal, I need to write DATABASE_ENGINE and DATABASE_NAME
> once for each application, because there is no way to group the
> information is even one of my applications needs a different database.
> That seems like a lot more configuration (longer config files are harder
> to debug, too).
A few counter-arguments...
If you have five apps that all use the same db, then you don't need
DATABASES at all.
Tuples are hashable:
DATABASES = {
('myproj.fred', 'myproj.ethel', 'myproj.lucy', 'myproj.ricky'): {
'DATABASE_ENGINE': ... }
}
Settings are python files: :)
common_db_settings = { 'DATABASE_ENGINE': ... }
DATABASES = {
'myproj.fred': common_db_settings,
'myproj.ethel': common_db_settings,
'myproj.ricky': common_db_settings,
'myproj.lucy': { 'DATABASE_ENGINE': ... } # she's uncommon
}
Lastly, as I have it now, items in DATABASES inherit from the default
any settings they don't specify themselves, so that should also reduce
config file length when N out of M apps use the same connection.
> I would prefer to get rid of the option of putting it in models
> altogether
You're right, the connection information should belong to the manager.
In my reply to Jacob, I suggested manager.db as the holder for all of
the connection info, which would give you manager.db.connection,
manager.db.backend, etc.
> I am unconvinced. :-)
I have more arguments. Or I will in the morning. :)
JP
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Mon, 2006-07-10 at 22:24 -0500, Jacob Kaplan-Moss wrote: > On Jul 10, 2006, at 10:09 PM, jason pellerin wrote: [...] > >> Finally: will there be syntax to use a specific connection for some > >> operations even if that model's set up to use a different backend? > >> I'd hope so, but I'm not sure I can come up with a good syntax... > > > > Well, you can always set Model._meta.db_connection = 'foo' at any > > time. Or create a manager that pulls the connection to use from > > someplace other than self.model._meta.connection, though that would be > > difficult without some additional abstraction. I'm not quite sure what > > the use of this would be, but that may mean there's a need that I just > > can't see. > > I posted a couple of example use-cases in response to Malcolm. > Manually setting ``db_connection`` is ugly but workable; I'll think > some more about a more elegant way of doing it. One possibility based on what you (Jacob) suggested earlier: if the db connection is a property of the model manager, then it is something you can override by using a different manager. We already have get_query_set() as the "default query set" for a manager. We could have get_db_connection() as the way to get the db connection (or make it a property or an attribute or whatever; the concept is the thing here). Then you can use various managers to get data from different databases. True, you could end having code that looks like this: if preferred_connection = 'first_db': manager = 'first' else: manager = 'second' data = getattr(MyModel, manager).filter(...) which could get old after the first two or three times, although wrapping that in a convenience function (MyModel.get_manager(manager).filter(...)) is not crazy in those cases. In passing, storing the *default* db connection in the Option class doesn't feel too ugly to me: it's a place to store model-specific (as opposed to instance-specific) information. We just happen to populate it mostly from Meta (I'm rewriting history saying it this way, I realise, but does that sound like a reasoning that makes sense). Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Jul 10, 2006, at 10:09 PM, jason pellerin wrote:
> Thanks, Jacob. And yes please, more eyes are badly needed. Aside from
> dropping tables, the branch is fully functional -- please try it out!
I've svn swich'd and I'm fooling with it now...
> This is interesting. How would it work -- look up the APPS in each
> settings entry at model class creation time, and fill in
> opts.db_connection that way? Or do you want to get rid of
> opts.db_connection entirely? I'd prefer to keep it -- for one thing,
> without that attribute around, the work to find the proper connection
> will have to be repeated every time the connection is requested. (If
> it's understood to be more of a connection-name cache rather than the
> prefered point of configuration, that's ok with me.)
Hmm...
I'd imagine the best way to do it would be to have the ``Manager``
look up the database connection it's supposed to use (at ``__init__``
time, probably). That way the lookup would only happen at model
definition time, not repeatedly for each query. Storing it in
``opts`` would be OK, but IMHO a bit ugly since ``opts`` is usually
reserved for "stuff defined in ``Meta``". Thus you could find out
which connection a model is using by checking
``Model.objects.connection`` (or something other than ``connection``,
of course).
> I'm not sure that tucking APPS into the db-definition dict is the
> right thing, though.
Yeah, I'm not really that happy with my syntax; it's just a strawman
to get us thinking about ways to define the connection in the
settings file.
> This may be too "magical", but how about: the
> keys in DATABASES could be app or model names, and in that case,
> matching apps/models will use the connection for that key. Example:
>
> DATABASES = {
># symbolic names are still ok, but you'd have to set them in the
> model
>'a': { ... },
># but specify an app or model and that app or model will use
> this connection
>'myproj.myapp': { ... },
>'thatproj.thatapp.ThatModel': { }
> }
>
> That seems pretty natural and clear to me, at least.
Yeah, I'm not sure I like that... what if I have an app with the same
name as my database? There's at least one Ellington install I know
of that has "ellington" in ``INSTALLED_APPS`` and uses a database
named "ellington"...
I'm running a bit dry on other ideas, though...
> Maybe a
> warning in management.validation if a model references a model that
> the current settings put until a different connection?
Yeah, that sounds perfect to me.
>> Finally: will there be syntax to use a specific connection for some
>> operations even if that model's set up to use a different backend?
>> I'd hope so, but I'm not sure I can come up with a good syntax...
>
> Well, you can always set Model._meta.db_connection = 'foo' at any
> time. Or create a manager that pulls the connection to use from
> someplace other than self.model._meta.connection, though that would be
> difficult without some additional abstraction. I'm not quite sure what
> the use of this would be, but that may mean there's a need that I just
> can't see.
I posted a couple of example use-cases in response to Malcolm.
Manually setting ``db_connection`` is ugly but workable; I'll think
some more about a more elegant way of doing it.
Jacob
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Mon, 2006-07-10 at 22:09 -0500, jason pellerin wrote:
> Hi all,
>
> Thanks, Jacob. And yes please, more eyes are badly needed. Aside from
> dropping tables, the branch is fully functional -- please try it out!
>
> > I'd like to see this done the other way around, which the DATABASES
> > settings defining which *models* are used. I'm picturing something
> > like this::
> >
> > DATABASES = {
> > 'a' : {
> > 'DATABASE_ENGINE' : 'sqlite3',
> > 'DATABASE_NAME': '/whatever/foo.db',
> > 'APPS' : ['myproject', 'yourproject.widget',
> > 'someotherapp']
> > }
> > }
> >
> > So each member of APPS is either an app specifier (as you'd pass to
> > models.get_app) or a model specifier (as you'd pass to
> > models.get_model). This way each site can specify different
> > databases for different apps or even individual models.
>
> This is interesting. How would it work -- look up the APPS in each
> settings entry at model class creation time, and fill in
> opts.db_connection that way? Or do you want to get rid of
> opts.db_connection entirely? I'd prefer to keep it -- for one thing,
> without that attribute around, the work to find the proper connection
> will have to be repeated every time the connection is requested. (If
> it's understood to be more of a connection-name cache rather than the
> prefered point of configuration, that's ok with me.)
>
> I'm not sure that tucking APPS into the db-definition dict is the
> right thing, though. This may be too "magical", but how about: the
> keys in DATABASES could be app or model names, and in that case,
> matching apps/models will use the connection for that key. Example:
>
> DATABASES = {
> # symbolic names are still ok, but you'd have to set them in the model
> 'a': { ... },
> # but specify an app or model and that app or model will use this
> connection
> 'myproj.myapp': { ... },
> 'thatproj.thatapp.ThatModel': { }
> }
Under Jacob's proposal, if I have five applications and all the models
in each application use the same connection, I need to write out (at
most), DATABASE_ENGINE and DATABASE_NAME once for each engine/name pair
and I need write the application names once.
Under your proposal, I need to write DATABASE_ENGINE and DATABASE_NAME
once for each application, because there is no way to group the
information is even one of my applications needs a different database.
That seems like a lot more configuration (longer config files are harder
to debug, too).
I would prefer to get rid of the option of putting it in models
altogether, having read Jacob's reasoning: having multiple ways to do
something seems unnecessary here (is it necessary for some reason I
haven't thought of?) and it is encouraging non-portable, difficult to
debug problems when moving applications around. What advantages does it
add that you see?
Jacob's approach still leaves you with a natural key to use when trying
to refer to the connection directly (so keeping DATABASES as a
dictionary, rather than just a list of dictionaries is worthwhile).
>
> That seems pretty natural and clear to me, at least.
I am unconvinced. :-)
Best wishes,
Malcolm
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Jul 10, 2006, at 9:55 PM, Malcolm Tredinnick wrote: > I agree with most of your points (the settings one is actually kind of > obvious that it may be backwards now that you mention it). > Inter-database relations would be nice to have at some point, but it > could be a post-first-release kind of feature. I agree it would be cool, but it would be (possibly horrifically) slow, and (for me at least) fringes on YAGNI. > On Mon, 2006-07-10 at 21:38 -0500, Jacob Kaplan-Moss wrote: > [...] >> Finally: will there be syntax to use a specific connection for some >> operations even if that model's set up to use a different backend? >> I'd hope so, but I'm not sure I can come up with a good syntax... > > What sort of use cases do you have in mind for this? Constructing > inter-db relations by hand? Not that in particular; I was thinking of things like: - importing data from another database - "feeding" data between two Django installations - saving data from a feed to multiple databases in one script (All of these are real-world problems I've found a way around in the past). > Thinking of some of his earlier mails, I think Jason already had in > mind > some way of getting the specific connections via db.backend, since we > would need that for manually constructing SQL in any case. But the way > you phrased the above query makes me think you have some trickier > use in > mind. I'm not sure it's really that complicated, frankly. Just being able to use the ORM against to database simultaneously would be a huge win for me. Jacob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
Hi all,
Thanks, Jacob. And yes please, more eyes are badly needed. Aside from
dropping tables, the branch is fully functional -- please try it out!
> I'd like to see this done the other way around, which the DATABASES
> settings defining which *models* are used. I'm picturing something
> like this::
>
> DATABASES = {
> 'a' : {
> 'DATABASE_ENGINE' : 'sqlite3',
> 'DATABASE_NAME': '/whatever/foo.db',
> 'APPS' : ['myproject', 'yourproject.widget',
> 'someotherapp']
> }
> }
>
> So each member of APPS is either an app specifier (as you'd pass to
> models.get_app) or a model specifier (as you'd pass to
> models.get_model). This way each site can specify different
> databases for different apps or even individual models.
This is interesting. How would it work -- look up the APPS in each
settings entry at model class creation time, and fill in
opts.db_connection that way? Or do you want to get rid of
opts.db_connection entirely? I'd prefer to keep it -- for one thing,
without that attribute around, the work to find the proper connection
will have to be repeated every time the connection is requested. (If
it's understood to be more of a connection-name cache rather than the
prefered point of configuration, that's ok with me.)
I'm not sure that tucking APPS into the db-definition dict is the
right thing, though. This may be too "magical", but how about: the
keys in DATABASES could be app or model names, and in that case,
matching apps/models will use the connection for that key. Example:
DATABASES = {
# symbolic names are still ok, but you'd have to set them in the model
'a': { ... },
# but specify an app or model and that app or model will use this connection
'myproj.myapp': { ... },
'thatproj.thatapp.ThatModel': { }
}
That seems pretty natural and clear to me, at least.
> Second, there's no mention about how relations will work across
> databases. I'm sure we *could* emulate them, but I'd suggest
> throwing a nice error if you try to reference a model that's in
> another database.
There's a *little* mention in the tests. :) My preference is not to
enforce this sort of thing, but let the database do it when needed.
Mainly because one of the big use cases for me is the ability to use a
local sqlite cache on a spread of webservers. The caches are published
from an admin db, but only the appropriate public tables are sent; so
there may be models and relations that appear to cross connections on
the public side but share a connection on the admin side. Maybe a
warning in management.validation if a model references a model that
the current settings put until a different connection?
> Finally: will there be syntax to use a specific connection for some
> operations even if that model's set up to use a different backend?
> I'd hope so, but I'm not sure I can come up with a good syntax...
Well, you can always set Model._meta.db_connection = 'foo' at any
time. Or create a manager that pulls the connection to use from
someplace other than self.model._meta.connection, though that would be
difficult without some additional abstraction. I'm not quite sure what
the use of this would be, but that may mean there's a need that I just
can't see.
> OK, thoughts, anyone?
What he said. :)
JP
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
I agree with most of your points (the settings one is actually kind of obvious that it may be backwards now that you mention it). Inter-database relations would be nice to have at some point, but it could be a post-first-release kind of feature. On Mon, 2006-07-10 at 21:38 -0500, Jacob Kaplan-Moss wrote: [...] > Finally: will there be syntax to use a specific connection for some > operations even if that model's set up to use a different backend? > I'd hope so, but I'm not sure I can come up with a good syntax... What sort of use cases do you have in mind for this? Constructing inter-db relations by hand? Thinking of some of his earlier mails, I think Jason already had in mind some way of getting the specific connections via db.backend, since we would need that for manually constructing SQL in any case. But the way you phrased the above query makes me think you have some trickier use in mind. Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
