Re: More efficient m2m assignment SQL
On 27 heinä, 08:15, Jeremy Dunck wrote: > I have 2 models: > > class District(Model): > pass > > class Voter(Model): > districts = ManyToManyField(District) > > Sometimes I want to reset the m2m list: > > voter.districts = other_districts > > I noticed that this uses ManyRelatedManager._clear_items, which, among > other things, boils down to: > > voter.districts.through._default_manager.filter(voter_id=voter.pk).delete() > > I would have expected something along the lines of : > > DELETE FROM `api_voter_districts` WHERE `voter_id` = 1 > > But instead it's 2 queries: > > SELECT `api_voter_districts`.`id`, `api_voter_districts`.`voter_id`, > `api_voter_districts`.`district_id` FROM `api_voter_districts` WHERE > `api_voter_districts`.`voter_id` = 1 > > DELETE FROM `api_voter_districts` WHERE `id` IN (2, 3, 4, 5, 6,...) > > Was it intentional to take 2 queries? That is, is there a use case > where avoiding the single DELETE query is preferable? To me it seems there is no reason to do two queries in delete if the following conditions hold: - there are no signals sent for the deleted objects - the delete does not cascade further (actually you don't need to collect even in this case - you can do the cascade query completely in the DB without fetching the IDs first). For automatic M2M tables the above conditions always hold, but it is not guaranteed if you are using the "through" argument. To me it seems the correct fix is checking the above conditions. If they hold, issue immediate delete query, if not, you need to collect the objects, then delete. Doing the delete in single query can be a huge win if you are deleting a lot of objects in one go. Actually, if you are deleting too many objects, you will likely go OOM. I am not objecting fixing this just for automatic m2m tables if checking the conditions mentioned above turns out to be complicated. - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
More efficient m2m assignment SQL
I have 2 models: class District(Model): pass class Voter(Model): districts = ManyToManyField(District) Sometimes I want to reset the m2m list: voter.districts = other_districts I noticed that this uses ManyRelatedManager._clear_items, which, among other things, boils down to: voter.districts.through._default_manager.filter(voter_id=voter.pk).delete() I would have expected something along the lines of : DELETE FROM `api_voter_districts` WHERE `voter_id` = 1 But instead it's 2 queries: SELECT `api_voter_districts`.`id`, `api_voter_districts`.`voter_id`, `api_voter_districts`.`district_id` FROM `api_voter_districts` WHERE `api_voter_districts`.`voter_id` = 1 DELETE FROM `api_voter_districts` WHERE `id` IN (2, 3, 4, 5, 6,...) Was it intentional to take 2 queries? That is, is there a use case where avoiding the single DELETE query is preferable? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Add a form to admin side
Please ask questions about using Django on django-users. The topic of this list is the discussion of the development of Django itself. Karen -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Add a form to admin side
I need to create a form to admin-side with two fields 1. Number of code: integer 2. Value of code: float How can I do that. This form is not related to any model -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/VHLeWPO-kw8J. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django should load custom SQL when testing
On 26 heinä, 19:35, Andrei Antoukh wrote: > Having the hooks as a method or function, I find it much more interesting > than having them in files. Above that allows logic in these methods, which > with flat files is not possible. I also like raw SQL in Python hooks better. An example why this is preferred is that we will likely have settings configurable db_schema one day in the future. How to use the correct schema in raw SQL files when you don't control the db_schema? This could happen for example in 3rd party apps. While I am likely making this feature request more complex than it needs to be, I do think the interaction with migrations is an interesting aspect. For normal model definition changes the migrations framework spots the changes, and you get the migrations for free. Thus, you need to define the changes in just one place. For raw SQL it is impossible to automatically create migrations based on changes in the SQL. Maybe the answer to this is that you have to define the migrations separately, and the current state separately. Unfortunately this could lead to situations where you test on different schema than what your production setup will be after migrations. Thus, it would be nice to define the changes just once to ensure you can't have state errors between production and testing. And, as you can't do the "define just once" in the raw SQL hooks, it must be the migrations which contain the just once setup. - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django should load custom SQL when testing
2012/7/26 Anssi Kääriäinen > On 26 heinä, 15:44, Danilo Bargen wrote: > > As the Tickethttps://code.djangoproject.com/ticket/16550is closed, I > will > > continue the discussion here on the mailing list. > > > > The reason for the ticket is that the Django test runner does not > provide a > > way to run custom SQL. This makes sense when users want to load custom > SQL > > *data*, but not when modifying the way the database works. > > > > Here's my comment from the issue comment thread: > > > > I agree very much with the comment above, in that there should be a way > to > > execute custom SQL before and after running syncdb. This should not be > used > > to load custom data, which is truncated anyways, but to alter the schema, > > create new types or load extensions. > > > > In my case, I have to load the Postgres citext extension: > > > > CREATE EXTENSION IF NOT EXISTS citext; > > > > Because I have to create a database manually before running syncdb, I can > > run my custom SQL in my database and then use it normally with Django. > But > > because the extensions are database specific, when running the tests > (which > > create their own database) they fail because the extension is not > present. > > This means that contrary to the comment by russellm, the *test database* > > introduces an inconsistency between the way tables work during a test > case > > and the way they work in production. > > > > Another widely used extension is the hstore extension. > > > > As a workaround, I loaded the extension in the default postgres template. > > > > psql -d template1 -c 'CREATE EXTENSION citext;' > > > > But that means that any new database I create will contain the citext > > extension, which might not be what I want. > > Therefore I suggest that you add some pre syncdb and post syncdb hooks to > > run custom SQL, and to mention in the docs that this should *not* be used > > to load custom data, but to create new types and alter the schema, so > that > > it *matches the production database*. The data is truncated anyways. > > > > I vote for reopen. > > > > What do you think about this issue? Does anyone else agree that there > > should be a way to run custom SQL in order not to create inconsistencies > > between the test and production database? > > I agree, the ability to run raw SQL on syncdb would be useful to me. > > There is an idea for allowing post_sync SQL for models (see > > http://groups.google.com/group/django-developers/browse_thread/thread/2a1b068f56faae75 > ). > It is clear one might want to define something else than post-sync SQL > per model. The full set of hooks seems to be: > - per project (pre + post) > - per app (pre + post). > - per model (per + post) > I wonder if we want to support all of those... > I think all three are useful, but I think the most useful: per model and per project. > > The main gripe against any of the above is that once you start doing > schema migrations, then you need to define the raw SQL in multiple > places. Both in the app and in the migrations files. This raises the > question if migration files is the right place to do this to begin > with. The syntax could be something like: > > class Migration: > apply_at = 'pre_sync'/'post_sync'/'migrate' > > Where 'migrate' is the default and means the migration is only applied > on "manage.py migrate". pre_sync and post_sync would be applied on > syncdb, and also on migrate if they are not applied before to the DB. > The idea is that you could define the raw SQL only once, and it would > be applied both to testdb on sync, and to production db on migrate. > > Still another idea: maybe we want to take the easy route and add > special "pre_sync/post_sync" initial SQL file names. Document that > these files should not contain data, just schema definition. Once we > get the migrations framework in, then deprecate the whole "initial SQL > in a directory" concept. > Having the hooks as a method or function, I find it much more interesting than having them in files. Above that allows logic in these methods, which with flat files is not possible. > > - Anssi Andrey. -- Andrei Antoukh - http://www.niwi.be/page/about/ http://www.kaleidos.net/A5694F/ "Linux is for people who hate Windows, BSD is for people who love UNIX" "Social Engineer -> Because there is no patch for human stupidity" -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: About: Feature request: Comments in Django models saved to database schema
On 26 heinä, 18:55, Andrei Antoukh wrote:
> 2012/7/24 Anssi Kääriäinen
>
> > On 22 heinä, 23:01, Andrei Antoukh wrote:
> > > Hello!
>
> Hello!
>
>
>
>
>
>
>
>
>
>
>
> > > I started working on what had been proposed. In the commithttps://
> > github.com/niwibe/django/commit/af887029integeintege614ca53d8...<
> >https://github.com/niwibe/django/commit/af887029614ca53d8bdab23f0a40e...>
> > > is
> > > the initial implementation. It is based on the Anssi idea.
>
> > > I have some doubts about how to make things better. The proposed method
> > > should return a list of sql statements:
> > > ["COMMENT ON TABLE fooapp_sampletable IS 'Foo comment';"]
>
> > > or a list of tuples with sql statements with possible arguments:
>
> > > [("COMMENT ON TABLE fooapp_sampletable IS %s", ["Foo comment"])]
>
> > > Currently, the initial implementation, is a list of tuples, but I'm
> > > wondering if the best option. I think that this is not usabe for commands
> > > such as sqlall because the returned sql is not complete, if only partial,
> > > failing to place them arguments.
>
> > I do think the sqlall output should be usable as is. That is, the
> > return value should be list of strings. This might mean that one has
> > to create the SQL in a way that is susceptible to SQL injection
> > attacks, but I don't see that as a problem. The post_sync method's SQL
> > can't be ran or controlled by users, so there is no attack vector.
>
> OK!
>
>
>
> > I have a couple of questions:
> > 1) How does this work in model inheritance situations? If the base
> > model defines post_sync_sql(), wouldn't that be called again for the
> > child models? Maybe it is possible to run the method only if it is
> > defined in the current class directly. This should be possible by
> > inspecting the model's __dict__ for the method instead of using
> > hasattr.
>
> Good observation!
>
> > 2) What is the order in which the post_sync_sql methods are called?
> > This could be important for some use cases (one view depends on
> > another view for example). I guess the order is app for app in
> > INSTALLED_APPS, models in the order they are found from app's
> > models.py? The order should be tested and documented.
>
> I'll think about it. Currently uses the same order as syncdb, even if not
> the most appropriate for this.
> We could think of a way to define the order of execution by the user.
>
> I created the asociated ticket for this
> feature:https://code.djangoproject.com/tidcket/18672
>
> Andrey.
Before proceeding too far in the morel pre/post sync hooks, lets see
if the other current thread about a similar issues yields a better
API. The other thread is about raw SQL on pre/post sync, and while it
isn't dealing with the exact same issue as this thread, there is a lot
of overlap. See
http://groups.google.com/group/django-developers/browse_thread/thread/c4dd1a333ae60356
- Anssi
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Django should load custom SQL when testing
On 26 heinä, 15:44, Danilo Bargen wrote: > As the Tickethttps://code.djangoproject.com/ticket/16550is closed, I will > continue the discussion here on the mailing list. > > The reason for the ticket is that the Django test runner does not provide a > way to run custom SQL. This makes sense when users want to load custom SQL > *data*, but not when modifying the way the database works. > > Here's my comment from the issue comment thread: > > I agree very much with the comment above, in that there should be a way to > execute custom SQL before and after running syncdb. This should not be used > to load custom data, which is truncated anyways, but to alter the schema, > create new types or load extensions. > > In my case, I have to load the Postgres citext extension: > > CREATE EXTENSION IF NOT EXISTS citext; > > Because I have to create a database manually before running syncdb, I can > run my custom SQL in my database and then use it normally with Django. But > because the extensions are database specific, when running the tests (which > create their own database) they fail because the extension is not present. > This means that contrary to the comment by russellm, the *test database* > introduces an inconsistency between the way tables work during a test case > and the way they work in production. > > Another widely used extension is the hstore extension. > > As a workaround, I loaded the extension in the default postgres template. > > psql -d template1 -c 'CREATE EXTENSION citext;' > > But that means that any new database I create will contain the citext > extension, which might not be what I want. > Therefore I suggest that you add some pre syncdb and post syncdb hooks to > run custom SQL, and to mention in the docs that this should *not* be used > to load custom data, but to create new types and alter the schema, so that > it *matches the production database*. The data is truncated anyways. > > I vote for reopen. > > What do you think about this issue? Does anyone else agree that there > should be a way to run custom SQL in order not to create inconsistencies > between the test and production database? I agree, the ability to run raw SQL on syncdb would be useful to me. There is an idea for allowing post_sync SQL for models (see http://groups.google.com/group/django-developers/browse_thread/thread/2a1b068f56faae75). It is clear one might want to define something else than post-sync SQL per model. The full set of hooks seems to be: - per project (pre + post) - per app (pre + post). - per model (per + post) I wonder if we want to support all of those... The main gripe against any of the above is that once you start doing schema migrations, then you need to define the raw SQL in multiple places. Both in the app and in the migrations files. This raises the question if migration files is the right place to do this to begin with. The syntax could be something like: class Migration: apply_at = 'pre_sync'/'post_sync'/'migrate' Where 'migrate' is the default and means the migration is only applied on "manage.py migrate". pre_sync and post_sync would be applied on syncdb, and also on migrate if they are not applied before to the DB. The idea is that you could define the raw SQL only once, and it would be applied both to testdb on sync, and to production db on migrate. Still another idea: maybe we want to take the easy route and add special "pre_sync/post_sync" initial SQL file names. Document that these files should not contain data, just schema definition. Once we get the migrations framework in, then deprecate the whole "initial SQL in a directory" concept. - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: About: Feature request: Comments in Django models saved to database schema
2012/7/24 Anssi Kääriäinen
> On 22 heinä, 23:01, Andrei Antoukh wrote:
> > Hello!
>
Hello!
> >
> > I started working on what had been proposed. In the commithttps://
> github.com/niwibe/django/commit/af887029integeintege614ca53d8...<
> https://github.com/niwibe/django/commit/af887029614ca53d8bdab23f0a40e...>
> > is
> > the initial implementation. It is based on the Anssi idea.
> >
> > I have some doubts about how to make things better. The proposed method
> > should return a list of sql statements:
> > ["COMMENT ON TABLE fooapp_sampletable IS 'Foo comment';"]
> >
> > or a list of tuples with sql statements with possible arguments:
> >
> > [("COMMENT ON TABLE fooapp_sampletable IS %s", ["Foo comment"])]
> >
> > Currently, the initial implementation, is a list of tuples, but I'm
> > wondering if the best option. I think that this is not usabe for commands
> > such as sqlall because the returned sql is not complete, if only partial,
> > failing to place them arguments.
>
> I do think the sqlall output should be usable as is. That is, the
> return value should be list of strings. This might mean that one has
> to create the SQL in a way that is susceptible to SQL injection
> attacks, but I don't see that as a problem. The post_sync method's SQL
> can't be ran or controlled by users, so there is no attack vector.
>
OK!
>
> I have a couple of questions:
> 1) How does this work in model inheritance situations? If the base
> model defines post_sync_sql(), wouldn't that be called again for the
> child models? Maybe it is possible to run the method only if it is
> defined in the current class directly. This should be possible by
> inspecting the model's __dict__ for the method instead of using
> hasattr.
>
Good observation!
> 2) What is the order in which the post_sync_sql methods are called?
> This could be important for some use cases (one view depends on
> another view for example). I guess the order is app for app in
> INSTALLED_APPS, models in the order they are found from app's
> models.py? The order should be tested and documented.
>
I'll think about it. Currently uses the same order as syncdb, even if not
the most appropriate for this.
We could think of a way to define the order of execution by the user.
I created the asociated ticket for this feature:
https://code.djangoproject.com/tidcket/18672
Andrey.
>
> - Anssi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>
--
Andrei Antoukh -
http://www.niwi.be/page/about/
http://www.kaleidos.net/A5694F/
"Linux is for people who hate Windows, BSD is for people who love UNIX"
"Social Engineer -> Because there is no patch for human stupidity"
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Django should load custom SQL when testing
As the Ticket https://code.djangoproject.com/ticket/16550 is closed, I will continue the discussion here on the mailing list. The reason for the ticket is that the Django test runner does not provide a way to run custom SQL. This makes sense when users want to load custom SQL *data*, but not when modifying the way the database works. Here's my comment from the issue comment thread: I agree very much with the comment above, in that there should be a way to execute custom SQL before and after running syncdb. This should not be used to load custom data, which is truncated anyways, but to alter the schema, create new types or load extensions. In my case, I have to load the Postgres citext extension: CREATE EXTENSION IF NOT EXISTS citext; Because I have to create a database manually before running syncdb, I can run my custom SQL in my database and then use it normally with Django. But because the extensions are database specific, when running the tests (which create their own database) they fail because the extension is not present. This means that contrary to the comment by russellm, the *test database* introduces an inconsistency between the way tables work during a test case and the way they work in production. Another widely used extension is the hstore extension. As a workaround, I loaded the extension in the default postgres template. psql -d template1 -c 'CREATE EXTENSION citext;' But that means that any new database I create will contain the citext extension, which might not be what I want. Therefore I suggest that you add some pre syncdb and post syncdb hooks to run custom SQL, and to mention in the docs that this should *not* be used to load custom data, but to create new types and alter the schema, so that it *matches the production database*. The data is truncated anyways. I vote for reopen. What do you think about this issue? Does anyone else agree that there should be a way to run custom SQL in order not to create inconsistencies between the test and production database? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/KsrsEf7XOA8J. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
