Sorry for the forward, forgot to add the dev group when I sent the mail! ---------- Forwarded message ---------- From: Ben Ford <[EMAIL PROTECTED]> Date: 11 Oct 2007 10:14 Subject: Re: Multiple DB-Support at django To: BIERMANS Koen <[EMAIL PROTECTED]> Cc: "Adam, Mario AD/LGN-IL" <[EMAIL PROTECTED]>
Hi Guys, Yes it just so happens that I've been working on just this issue. Yesterday I put my work so far into a djangosnippet here<http://http://www.djangosnippets.org/snippets/442/>. It relies on a couple of changes that I've made to my copy of the multi-db trunk, ( e.g as an interim measure model_connection_name now checks for a db attribute on the model itself which is a key in django.db._init__.connections). I believe that we should ditch most of the logic currently in db.__init__.. A lot of it has now been negated by the stuff in the backend refactor from what I can see... Maybe we should just think about starting again from a vanilla trunk We should provide the ability to regester models with different databases (similar to how templatetags works), maybe in a separate module (project/connections.py perhaps) with the database defaulting to the details found in the normal setting module. I think that would work nicely for all cases, if you have a model in more than one db, just register it more than once with different connection settings. With the backend refactor it should be relatively simple to map models to DatabaseWrappers. I also agree with you Koen that the reference to the database would probably be better attached to the model somehow. Mario, The snippet mentioned above probably won't work as is with the existing copy of multi-db, I have had to make a couple of changes. Also it's undocumented and I didn't write tests for it. (yes yes i know, but I've been under a lot of time constraint and I haven't had time to learn to be productive with TDD yet - next project for sure :-)! ) But I use it quite happily with my multiple databases with the same schema and it solves my problems perfectly :-) django-dev, FYI and for any comments. Cheers Ben On 10/10/2007, BIERMANS Koen <[EMAIL PROTECTED] > wrote: > > *Ben, I put you in CC because I think you know more about this than I do. > * > > > > Hi Mario, > > > > I am glad it is working. > > > > About foreign keys to other databases: we will have to wait for queryset > refactoring to land before we can see if that can be solved I think. You are > best off doing the queries separately yourself for now. > > > > As to using different connections for one model: (I may be playing out of > my league trying to answer this, so forgive me for any inaccuracies): > > > > The connections are used in query.py via the manager of the model > (connection = model._default_manager.db.connection). > > > > The "db" var is a ConnectionInfoDescriptor object from db.__init__.py. > > > > The nice thing about this object is that you can simply "SET" this to any > named connection you want (use the "connections" proxy to get them. The only > thing that bugs me is the _default connection (I don"t know why this one was > implemented the way it is via this dummy object, but you can just import > _default and call connections[_default]). > > Beware: the connection that is linked to this manager is reset after every > request via the request_finished signal. > > > > The connections themselves are lazy: they are only really connected when > there is a query to perform (django does not do any connection pooling > beyond the request). > > > > So my guess is: yes, you can define all your databases in your settings > files (OTHER_DATABASES, no need to add models) and set the correct > connection to your models at the start of every request, eg via middleware. > ( something like modelX._default_manager.db = connections['connectionY'] > should work I think). > > > > */me takes a note: we should be passing the connection to the query > instead of using the _**default** _manager's one, that way you could > simply define different managers (eg modelx.objects.all() uses the default > connection, modelx.objectsfromY.all() uses the 'connectionY' connection > etc...(Ben, can we do this when we define the queryset class for the models > manager? I'll need to look at this.)* > > > > I have however no experience with all this, maybe Ben has more info on > this ? (I seem to remember him using it to copy data from one db to another > ?) > > > > Koen > > > ------------------------------ > > *Van:* Adam, Mario AD/LGN-IL [mailto:[EMAIL PROTECTED] > *Verzonden:* woensdag 10 oktober 2007 10:21 > *Aan:* BIERMANS Koen > *Onderwerp:* AW: Multiple DB-Support at django > > > > Hello Koen, > > > > some greetings from germany. :-) > > So far this patch works great. I encountered one thing (but I think it is > not solvable, due to the database backend): I have one model stored at DB 1 > and one model stored at DB 2. Model 1 has a foreign key to model 2. Using > syncdb (and I think all other commands as well) would lead to an error, as > django tries to create a foreign key relation to a table, that does not > exist in model-1's database but in the second one. Anyway I can live with > such a limitation. > > > > I have one other question. My currently developed project (which makes > heavy use of this patch) is going to use 80 databases at the moment. The > number of databases is increasing with the time. Depending on the subdomain > I have to use another database connection. To explain it more detailed: > imagine a browsergame with different gameworlds. The useraccounts are stored > in the master database, the gameworld-specific data are stored in the > relevant gameworld-database (e.g. player data, village data, etc). This > means, that I have several databases with the same structure. I need to > separate the databases due to some technical reasons. Thatswhy the usage of > database connections is depending on the subdomain. Now my question is: when > does django initializes the db-connection for each model? At the moment, > when I start the django project (via "manage.py runfcgi") or at the > moment, the model is created? (out of my head I'd guess the first one). > > At the moment this would mean, that I need to have ~80 files settings.pyand > therefor need to start ~80 django instances. Starting them with prefork > would result in 5 python-processes for each instance consuming each ~15k > memory - making more than 5GB memory usage after all. If I could change the > database connection via a middleware (I currently HAVE a middleware, that > initializes another object - the gameworld-model itself - depending on the > subdomain and appending it to the request), I'd only need a hand full > instances (or even only a single instance) of django starting up. But this > is only possible, if I could change a model's connection at runtime (via a > middleware). Is this possible? > > > > Mit freundlichen Grüßen / Kind regards > > i.A. Mario Adam > IT Application Operations > > LuK-Aftermarket Service oHG > Paul-Ehrlich-Str. 21 > 63225 Langen > Tel. +49 6103 753 133 · Fax +49 6103 753 307 > mailto:[EMAIL PROTECTED] <[EMAIL PROTECTED]> · * > http://www.luk-as.com* > Sitz Langen > Registergericht: AG Offenbach am Main HRA 31654 > > Diese Information ist für den Gebrauch durch die Person oder die > Firma/Organisation bestimmt, die in der Empfängeradresse benannt ist und > unterliegt u. U. dem Betriebsgeheimnis, dem Schutz von Arbeitsergebnissen > oder anderweitigem rechtlichen Schutz. Wenn Sie nicht der angegebene > Empfänger sind, nehmen Sie bitte zur Kenntnis, dass Weitergabe, Kopieren, > Verteilung oder Nutzung des Inhalts dieser E-Mail-Übertragung unzulässig > ist. Falls Sie diese E-Mail irrtümlich erhalten haben, benachrichtigen Sie > den Absender bitte unverzüglich telefonisch oder durch eine E-Mail und > löschen Sie diese Information aus Ihrem EDV-System. > > This e-mail message is intended only for the use of the named recipient(s) > and contains information which may be confidential or privileged. If you are > not the intended recipient, be aware that any distribution, or use of the > contents of this information is prohibited. If you have received this > electronic transmission in error, please notify the sender and delete the > material from the computer. > > > > > ------------------------------ > > *Von:* BIERMANS Koen [mailto:[EMAIL PROTECTED] > *Gesendet:* Donnerstag, 4. Oktober 2007 13:31 > *An:* Adam, Mario AD/LGN-IL > *Betreff:* RE: Multiple DB-Support at django > > Mario, > > > > I modified the get_db_prep_save functions to accept also the > model_instance (this was needed to have the correct quoting function in the > mysql exception in datetimefield. > > Looking into it now I see I forgot to add it to the get_db_prep_save of > the FileField (see models/fields/__init__.py, line 722). > > Does your userprofile use a filefield ? (if not, there may be more items > missing). > > > > Koen > > > ------------------------------ > > *Van:* Adam, Mario AD/LGN-IL [mailto:[EMAIL PROTECTED] > *Verzonden:* donderdag 4 oktober 2007 12:35 > *Aan:* BIERMANS Koen > *Onderwerp:* AW: Multiple DB-Support at django > > > > Hello Koen, > > > > Just as a first result: when trying to do: > > request.user.get_profile().save() > > I get following error. > > "get_db_prep_save() takes exactly 2 arguments (3 given)" > > > > Profile exists, user too... Saving worked when using the trunk with no > applied patch. I'll try now to figure it out, why that happens. (Other > updates to the database are working well...) > > > > Mit freundlichen Grüßen / Kind regards > > i.A. Mario Adam > IT Application Operations > > LuK-Aftermarket Service oHG > Paul-Ehrlich-Str. 21 > 63225 Langen > Tel. +49 6103 753 133 · Fax +49 6103 753 307 > mailto:[EMAIL PROTECTED] <[EMAIL PROTECTED]> · * > http://www.luk-as.com* > Sitz Langen > Registergericht: AG Offenbach am Main HRA 31654 > > Diese Information ist für den Gebrauch durch die Person oder die > Firma/Organisation bestimmt, die in der Empfängeradresse benannt ist und > unterliegt u. U. dem Betriebsgeheimnis, dem Schutz von Arbeitsergebnissen > oder anderweitigem rechtlichen Schutz. Wenn Sie nicht der angegebene > Empfänger sind, nehmen Sie bitte zur Kenntnis, dass Weitergabe, Kopieren, > Verteilung oder Nutzung des Inhalts dieser E-Mail-Übertragung unzulässig > ist. Falls Sie diese E-Mail irrtümlich erhalten haben, benachrichtigen Sie > den Absender bitte unverzüglich telefonisch oder durch eine E-Mail und > löschen Sie diese Information aus Ihrem EDV-System. > > This e-mail message is intended only for the use of the named recipient(s) > and contains information which may be confidential or privileged. If you are > not the intended recipient, be aware that any distribution, or use of the > contents of this information is prohibited. If you have received this > electronic transmission in error, please notify the sender and delete the > material from the computer. > > > > > ------------------------------ > > *Von:* BIERMANS Koen [mailto:[EMAIL PROTECTED] > *Gesendet:* Donnerstag, 4. Oktober 2007 11:59 > *An:* Adam, Mario AD/LGN-IL > *Betreff:* RE: Multiple DB-Support at django > > Mario, > > > > Could you let me know whether this one worked ? I will put it on the > ticket then. > > > > Koen > > > ------------------------------ > > *Van:* Adam, Mario AD/LGN-IL [mailto:[EMAIL PROTECTED] > *Verzonden:* donderdag 4 oktober 2007 11:54 > *Aan:* BIERMANS Koen > *Onderwerp:* Multiple DB-Support at django > > > > Hello Koen, > > I encountered some really weird things when I try to apply your attached > patch ( http://code.djangoproject.com/ticket/4747). The most weird thing > is: I cannot apply this patch. I've checked out revision 6433 from trunk and > tried to apply the patch: > > > ********************************************************************************************************** > Hmm... Looks like a unified diff to me... > The text leading up to this was: > -------------------------- > |=== django/test/utils.py > |================================================================== > |--- django/test/utils.py (/mirror/django/trunk) (revision 4191) > | > |+++ django/test/utils.py (/local/django/multidb) (revision 4191) > | > -------------------------- > Patching file django/test/utils.py using Plan A... > Hunk #1 FAILED at 1. > 1 out of 1 hunk FAILED -- saving rejects to file django/test/utils.py.rej > Hmm...missing header for unified diff at line 16 of patch > The next patch looks like a unified diff to me... > can't find file to patch at input line 16 > Perhaps you used the wrong -p or --strip option? > The text leading up to this was: > -------------------------- > |ispatch import dispatcher > -------------------------- > File to patch: > > ********************************************************************************************************** > > OK - the next weird thing is: there were no changes at utils.py between > revision 6110 and 6443. Comparing your patch to the patch for revision 6110 > there are no differences in the first 24 lines (except the revision number). > But as listed above the patching process already fails at the very first > beginning. Can you find any reason for this? I tried following things with > --dry-run: > > 1) apply patch for rev. 6110 to rev. 6110 -> suceeded > 2) apply patch for rev. 6110 to rev. 6433 -> suceeded > 3) apply patch for rev. 6433 to rev. 6110 -> failed > 4) apply patch for rev. 6433 to rev. 6433 -> failed > > I did it both on Windows and Linux without different results. > > Or can you hand me out a patched source of django? (per download, per SVN, > per <whatever>) > > With best regards > Mario Adam (Germany) > -- Regards, Ben Ford [EMAIL PROTECTED] +6281317958862 -- Regards, Ben Ford [EMAIL PROTECTED] +6281317958862 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
