QuerySet.contains() for .values() and .values_list()
https://github.com/django/django/pull/13038
Above pull request (Ticket #24141) has a discussion on how to handle
qs.contains() for QuerySets with qs.values() or qs.values_list() calls.
Current patch checks self._result_cache if exists and iterable class is
ModelIterable, otherwise returns .filter(pk=obj.pk).exists(), i.e. hitting
the database.
These are the options I see:
1. As above.
2. Throw an exception if iterable class is not ModelIterable (when using
.values or .values_list).
3. Accept dict lookup for .values and tuple lookup for .values_list
querysets, so that the .contains() query matches what you get from the
iterable.
It seems kind of neat to be able to check if an object is in a QuerySet,
even if you group that QuerySet using
.values('group').annotate(sum=Sum('value')).order_by(). But that was never
the case I saw for QuerySet.contains(), so I have no real preference here.
Johan
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/5d77428e-a4a4-4bda-a3bb-2811a88e3797n%40googlegroups.com.
Re: Implement QuerySet.__contains__?
Shai does make a good point about changing a well documented behaviour. That argument wins me over. Adding .contains() and updating the documentation goes a long way to make it easier for developers. Best case would be that Dajngo does not make assumptions about what the developer wants, but to implement __len__(), __bool__() and __contains__(), we have to assume one method or the other. I still think the wrong call was made here, but changing it now is too much pain. *(Assuming queryset should be evaluated is probably correct in most cases, but sometimes adds a big performance hit when the code goes into production and the dataset grows - code that looks reasonable and works like a charm in dev will cause trouble in production. Assuming the opposite would have added a tiny performance hit in most cases, but avoided the big one.)* Since I'm new here: If people agree QuerySet.contains() should be added, how do we move forward? I'd be willing to write some code with tests and add a ticket, if that's helpful. Le ven. 5 juin 2020 à 11:42, אורי a écrit : > Hi, > > I'm thinking, maybe instead of: > > if obj in queryset: > > Users should write: > > if obj in list(queryset): > > Which I understand is already working? Why does the first line (without > list()) should work anyway? > > And if users want performance, why not write: > > if queryset.filter(pk=obj.pk).exists(): > אורי > u...@speedy.net > > > On Tue, Jun 2, 2020 at 11:57 AM Johan Schiff wrote: > >> Is there a specific reason Djangos QuerySet does not implement >> __contains__? It doesn't seem very complicated to me, but maybe I'm missing >> something. >> >> When checking if an object is in e queryset I always use code lite this: >> if queryset.filter(pk=obj.pk).exists(): >> >> The pythonic way would be: >> if obj in queryset: >> >> The way I understand it this latter version will fetch all objects from >> queryset, while the first one is a much leaner db query. >> >> So, is there a reason QuerySet doesn't have a __contains__ method, or >> would it be a good addition? My guess is that a lot of people use the "obj >> in queryset" syntax, believing it does "the right thing". >> >> //Johan >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-developers+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/88c83b8d-658c-47cc-9162-fcfebebe9c4a%40googlegroups.com >> <https://groups.google.com/d/msgid/django-developers/88c83b8d-658c-47cc-9162-fcfebebe9c4a%40googlegroups.com?utm_medium=email_source=footer> >> . >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "Django developers (Contributions to Django itself)" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/django-developers/NZaMq9BALrs/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > django-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/CABD5YeE7_ORUfJcnt6f4aB4J0j-%3DyDK_BowEt_fefcaFMGdB1g%40mail.gmail.com > <https://groups.google.com/d/msgid/django-developers/CABD5YeE7_ORUfJcnt6f4aB4J0j-%3DyDK_BowEt_fefcaFMGdB1g%40mail.gmail.com?utm_medium=email_source=footer> > . > -- Vänligen, Johan Schiff Radkompaniet AB 072-229 61 19 -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAD69rUBKOse%2Be2RedPT%2B-KWkcdC%3DiaG0zyt1Hakefp6gQ7vdLA%40mail.gmail.com.
Re: Implement QuerySet.__contains__?
To answer my own question: No, I wasn't doing it correctly. I should have
done a sanity check before posting.
New timeit code and results at bottom, now using a smaller dataset (~900
objects).
Notable:
- An .exists() query is about 1/100 the time of full fetch in this case.
This difference would ofc be much bigger if I did it on the 100 000+
objects.
- If already prefetched, it should be worth it to search prefetched data
if the dataset is <700 objects. It seems rare to prefetch that many
objects. If so, a developer could easily be explicit and use .exists()
method.
*Based on this exact scenario: 100 times slower for using obj in
queryset is quite a performance hit. On the other hand, an extra 1 % query
time for a queryset that is later evaluated is marginal.*
If a developer knows they want to evaluate the queryset later, it would be
nice to have a .evaluate() that does self._fetch_all() and returns self. I
think that's preferable over list(queryset)-method, because we don't need
to create an extra list.
# Old explicit fetch method
queryset = Stuff.objects.all()
if obj in list(queryset):
pass
# My suggestion
queryset = Stuff.objects.all().evaluate()
if obj in queryset:
pass
Updated timeit function and results:
import timeit
def time_fn(title, stmt, number=10_000):
setup = [
'from movies.models import Movie',
'qs=Movie.objects.all()',
'obj_first = qs.first()',
'obj_mid = qs[300]',
'obj_last = qs.last()',
'list(qs)',
]
result_time = timeit.timeit(stmt, setup='\n'.join(setup), number=number)
print(f'{title}: {result_time}')
time_fn('Database fetch', 'qs._result_cache=None\nlist(qs)')
time_fn('Prefetched obj_first in qs', 'obj_first in qs')
time_fn('Prefetched obj_mid in qs', 'obj_mid in qs')
time_fn('Prefetched obj_last in qs', 'obj_last in qs')
time_fn('Database obj_first exists()', 'qs.filter(pk=obj_first.pk
).exists()')
time_fn('Database obj_mid exists()', 'qs.filter(pk=obj_mid.pk).exists()')
time_fn('Database obj_last exists()', 'qs.filter(pk=obj_last.pk).exists()')
# Results
Database fetch: 616.097227364
Prefetched obj_first in qs: 0.030961711003328674
Prefetched obj_mid in qs: 6.6988333979970776
Prefetched obj_last in qs: 24.18991441695
Database obj_first exists(): 6.468764332996216
Database obj_mid exists(): 6.167532913001196
Database obj_last exists(): 6.0190791100030765
Le mer. 3 juin 2020 à 10:52, Johan Schiff a écrit :
> Thanks for great info.
>
> First, I'm leaning towards Aymeric's proposition here. I do recognize that
> there is a lot to consider.
>
> This seems to be important:
>
>1. Developers must be able to explicitly choose methods to optimize
>for their environment. (Considering database latency, dataset size,
>prefetch preferred, etc.)
>2. Behaviour should preferably be consistent across methods. (len(qs),
>bool(qs), obj in qs)
>3. Syntax should be pythonic.
>
> I think what Aymeric is proposing would fulfill those demands.
>
> I have done some timeit-tests on a model with 100 000+ records, based on
> Rogers work, on a single host setup using postgresql. My finding is that a
> separate database query is generally faster than current *obj in qs*
> behaviour,
> even on a prefetched queryset (unless your dataset is really small). This
> tells me that we should prioritize .exists() query unless explicitly stated
> (i.e. *obj in iter(qs)*), even when we have prefetched data. For len(qs)
> and bool(qs) that should not be the case.
>
> It would be interesting to get timings from other setups, so I'm including
> the code I used. (Also, am I doing this correctly?)
>
> import timeit
>
> def time_fn(title, stmt, number=1, prefetch=False):
> setup = [
> 'from tickets.models import Ticket',
> 'qs=Ticket.objects.all()',
> 'obj_first = qs.first()',
> 'obj_mid = qs[5]',
> 'obj_last = qs.last()',
> ]
> if prefetch:
> setup.append('list(qs)')
> result_time = timeit.timeit(stmt, setup='\n'.join(setup),
> number=number)
> print(f'{title}: {result_time}')
>
> time_fn('Database fetch', 'list(qs)')
> time_fn('Prefetched obj_first in qs', 'obj_first in qs', prefetch=True)
> time_fn('Prefetched obj_mid in qs', 'obj_mid in qs', prefetch=True)
> time_fn('Prefetched obj_last in qs', 'obj_last in qs', prefetch=True)
> time_fn('Database obj_first exists()', 'qs.filter(pk=obj_first.pk
> ).exists()')
> time_fn('Database obj_mid exists()', 'qs.filter(pk=obj_mid.pk).exists()')
> time_fn('Database obj_last exists()', 'qs.filter(pk=obj_last.pk
> ).exists()')
>
>
> My results:
>
> Database fetch: 25.667968138001015
> Prefetched obj_first in qs: 0.027538340998944477
> Prefetched obj_mid in qs: 1051.1691511649988
> Prefetched o
Re: Implement QuerySet.__contains__?
o unsubscribe from this group and all its topics, send an email to > django-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/3928039038bac9b52279294f7efcac318dc80388.camel%40gammascience.co.uk > <https://groups.google.com/d/msgid/django-developers/3928039038bac9b52279294f7efcac318dc80388.camel%40gammascience.co.uk?utm_medium=email_source=footer> > . > -- Vänligen, Johan Schiff Radkompaniet AB 072-229 61 19 -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAD69rUA1E1Vfdm7kuv32r1Ppyw%3D7CCqCzfSW9itda88th2Uw3g%40mail.gmail.com.
Implement QuerySet.__contains__?
Is there a specific reason Djangos QuerySet does not implement __contains__? It doesn't seem very complicated to me, but maybe I'm missing something. When checking if an object is in e queryset I always use code lite this: if queryset.filter(pk=obj.pk).exists(): The pythonic way would be: if obj in queryset: The way I understand it this latter version will fetch all objects from queryset, while the first one is a much leaner db query. So, is there a reason QuerySet doesn't have a __contains__ method, or would it be a good addition? My guess is that a lot of people use the "obj in queryset" syntax, believing it does "the right thing". //Johan -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/88c83b8d-658c-47cc-9162-fcfebebe9c4a%40googlegroups.com.
Customizable urlize
Hi Django. I'm currently working on a project where we want to use Django template filter *urlize*, but with a custom url formatter. The way the Django code works atm, we can't extend *urlize* customize output. That means we have to go non-DRY and copy code. Generally I like using class based coding for this kind of thing, but Djangos template filters seems to prefer functions and discourage instantiation. Therefore, it would be nice to solve this in a functional style, in line with current design. Wouldn't it be a good idea to extend django.utils.html.urlize, or am I looking at this the wrong way? The function signature would be smthng like this: def urlize( text, trim_url_limit=None, *# for compat reasons, though it would make sense to accomplish that with the formatter functions in the urlizetrunc filter.* nofollow=False, autoescape=False, email_formatter: callable = None, url_formatter: callable = None, ): ... Johan Schiff Betahaus (SE) -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5d2877db-83e9-4bda-8087-98f5cd8fc57c%40googlegroups.com.
RE: Tagging 1.4 django release in Subversion
>>good news: tag is there (https://code.djangoproject.com/changeset/17810) Thanks! >>Out of curiosity, what's the benefit of using an svn tag over the released tarball? For my project I need to apply some patches to the django code (eg for #8280). Applying a patch on svn checkout allows me to track my changes better, and upgrade them between releases. Johan From: django-developers@googlegroups.com [mailto:django-developers@googlegroups.com] On Behalf Of Florian Apolloner Sent: Monday, March 26, 2012 5:48 PM To: django-developers@googlegroups.com Subject: Re: Tagging 1.4 django release in Subversion Hi, good news: tag is there (https://code.djangoproject.com/changeset/17810) On Monday, March 26, 2012 6:05:47 AM UTC+2, Tai Lee wrote: How come? The release that can be downloaded from the site already must correspond to an SVN revision number, right? Why not tag it as such so that people can easily get the same code from SVN as from the release tarball? Out of curiosity, what's the benefit of using an svn tag over the released tarball? Cheers, Florian -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/dZ-7tRLAXswJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: How to create a new command in django-admin.py
Well I can think of a whole lot of ways in which this can be done 'better' :) ... But, I don't have the time to change django atm so I will have to live with it :) ... Thanks for the help ... On Feb 23, 4:14 pm, Tom Evans <tevans...@googlemail.com> wrote: > On Thu, Feb 23, 2012 at 1:22 PM, Johan <djjord...@gmail.com> wrote: > > Hi > > > I know how to create a new command in manage.py, this link explains it > > nicely > > :https://docs.djangoproject.com/en/dev/howto/custom-management-command... > > . But how do I create new command which would be available in django- > > admin.py or in manage.py BUT without having to add the application to > > the INSTALLED_APPS tuple? > > > Regards > > Johan > > This should be on django-users, unless you are proposing a new way of > doing this. > > The answer to your question is "you can't", btw. > > Cheers > > Tom -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: How to create a new command in django-admin.py
Nope there is not. I just thought it could be done as easily as for manage.py. Thanks On Feb 23, 4:13 pm, Jani Tiainen <rede...@gmail.com> wrote: > 23.2.2012 15:22, Johan kirjoitti: > > > Hi > > > I know how to create a new command in manage.py, this link explains it > > nicely > > :https://docs.djangoproject.com/en/dev/howto/custom-management-command... > > . But how do I create new command which would be available in django- > > admin.py or in manage.py BUT without having to add the application to > > the INSTALLED_APPS tuple? > > > Regards > > Johan > > Only way to achieve what you're asking is to modify Django source code - > basically building in your command into Django core. > > Is there real rationale to do that? > > -- > > Jani Tiainen -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
How to create a new command in django-admin.py
Hi I know how to create a new command in manage.py, this link explains it nicely : https://docs.djangoproject.com/en/dev/howto/custom-management-commands/#command-objects . But how do I create new command which would be available in django- admin.py or in manage.py BUT without having to add the application to the INSTALLED_APPS tuple? Regards Johan -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Trouble joining IRC
Got it! Thank you so much! Johan On Feb 12, 11:17 pm, Alex Gaynor <alex.gay...@gmail.com> wrote: > On Fri, Feb 12, 2010 at 11:14 PM, Johan <johanharj...@gmail.com> wrote: > > Hello, > > > I am trying to log into the IRC channel at #django-dev but I kept on > > getting this error: > > > #django-dev :Cannot join channel (+r) - you need to be identified > > with services > > > I can join other channels, so I think the problem is with #django-dev > > specifically, is there something special I need to do before I can > > join the channel? I'm not sure what it means by "identified with > > services" > > > also, I'm quite new to IRC, so perhaps it's something completely > > obvious that I overlooked? > > > thank you for the trouble, > > Johan > > > -- > > You received this message because you are subscribed to the Google Groups > > "Django developers" group. > > To post to this group, send email to django-develop...@googlegroups.com. > > To unsubscribe from this group, send email to > > django-developers+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/django-developers?hl=en. > > As the message says, #django-dev requires you to be authenticated with > the nick server:http://freenode.net/faq.shtml#contents-userregistration > > Alex > > -- > "I disapprove of what you say, but I will defend to the death your > right to say it." -- Voltaire > "The people's good is the highest law." -- Cicero > "Code can always be simpler than you think, but never as simple as you > want" -- Me -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Trouble joining IRC
Hello, I am trying to log into the IRC channel at #django-dev but I kept on getting this error: #django-dev :Cannot join channel (+r) - you need to be identified with services I can join other channels, so I think the problem is with #django-dev specifically, is there something special I need to do before I can join the channel? I'm not sure what it means by "identified with services" also, I'm quite new to IRC, so perhaps it's something completely obvious that I overlooked? thank you for the trouble, Johan -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Bug in django\template\__init__.py ??
Thanks. This link is exactly what i need. On Oct 29, 5:07 pm, Waylan Limberg <way...@gmail.com> wrote: > On Thu, Oct 29, 2009 at 10:37 AM, Johan <djjord...@gmail.com> wrote: > > > I am > > wanting to use the template engine outside the context of a django > > project so I would not have a settings file anywhere on my path. > > This is documented here: > > http://docs.djangoproject.com/en/dev/topics/settings/#using-settings-... > > > I am > > assuming that the code works in a project context since the project > > would import settings and this 'broken' import would just fail > > silently > > Pay special attention to the last section (Either configure() or > DJANGO_SETTINGS_MODULE is required) of the docs linked above. As > Russell mentioned settings is lazy so you don't get an import error > but you will get a RuntimeError if settings have not been configured > properly when you actually try to use your templates. > -- > > \X/ /-\ `/ |_ /-\ |\| > Waylan Limberg --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Bug in django\template\__init__.py ??
Thanks for the quick reply. I discovered exactly what you said about 2 seconds after pressing the submit on my query. On Oct 29, 4:50 pm, Russell Keith-Magee <freakboy3...@gmail.com> wrote: > On Thu, Oct 29, 2009 at 10:37 PM, Johan <djjord...@gmail.com> wrote: > > > Hi. In django\template\__init__.py on line 54 there is a line of > > code : from django.conf import settings. Firstly there is no > > settings.py file in django\conf, ther is however a file called > > global_settings.py. In my context the is issue is fxed by changing the > > line to from django.conf import global_settings as settings. I am > > wanting to use the template engine outside the context of a django > > project so I would not have a settings file anywhere on my path. I am > > assuming that the code works in a project context since the project > > would import settings and this 'broken' import would just fail > > silently ?? Is this a bug? Is my fix going to break anything else? > > What you are reporting is not a bug. You need to look closer at what > is happening with django.conf. django/conf/__init__.py contains a > variable named settings that is the lazy evaluator of the setting > file. THis is the object that is obtained when you call 'from > django.conf import settings" > > Your "fix" will break quite a bit of code - including, potentially, > your own. It would results in the template system only ever using the > global defaults. This wouldn't be a problem, except for the fact that > there are a couple of settings that affect the way that templates are > rendered. > > Django's dependence on DJANGO_SETTINGS_MODULE is an oft-lamented > problem. Suggestions on how to address this constraint are most > welcome. However, it isn't a simple problem to fix. > > Yours, > Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Bug in django\template\__init__.py ??
Hi. In django\template\__init__.py on line 54 there is a line of code : from django.conf import settings. Firstly there is no settings.py file in django\conf, ther is however a file called global_settings.py. In my context the is issue is fxed by changing the line to from django.conf import global_settings as settings. I am wanting to use the template engine outside the context of a django project so I would not have a settings file anywhere on my path. I am assuming that the code works in a project context since the project would import settings and this 'broken' import would just fail silently ?? Is this a bug? Is my fix going to break anything else? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django, initial data and custom SQL
ith, since they in that case most likely should update it anyway) but understand the Way of Django and fully respect that. This patch is the result. Just for reference, this is what I think should be done for an optimal workflow: * create table * create m2m * run custom data * run fixtures * run indexes * emit "database done" signal This way, you can still insert your favorite SQL with custom data, use fixtures for data, then for advanced users - remove indexes with their hooks. I know this (kind of) retracts my above patch - but you generally want to throw in functions before inserting data (think pgmemcached), and the above behavior is way too untested (at my place, anyway) to be sent as a real patch. > > - Ludvig Thanks for your time, Johan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django, initial data and custom SQL
On Feb 10, 5:07 pm, Johan Bergström <b...@bergstroem.nu> wrote: > Hey, > > On Feb 10, 4:51 pm, "ludvig.ericson" <ludvig.eric...@gmail.com> wrote: > > > > > > > On Feb 10, 1:13 pm, Johan Bergström <b...@bergstroem.nu> wrote: > > > > Since Django executes my custom SQL before creating indexes, it's > > > impossible to achieve something that hooks into initdb/syncdb. I know > > > that it is "good custom" to create indexes after inserting data – but > > > fixtures in Django is already executed after creating indexes, so that > > > can't be the reason.. So, without further ado, what I would like to > > > know is if there's a reason to why custom SQL is executed before index > > > creation. > > > Isn't this doable with initial > > SQL?http://docs.djangoproject.com/en/dev/howto/initial-data/#initial-sql > > > Testing here with SQLite, it'd seem it runs the custom SQL at the very > > last point, so you could actually add some ALTER TABLE statements, I > > guess. Again, this is testing with SQLite, and SQLite doesn't do > > indexing. > > Actually it doesn't. I think you just did a reset/sqlall instead of > sync/initdb: > > # cat settings.py | grep DATABASE_E > DATABASE_ENGINE = "sqlite3" > > # python manage.py syncdb > > Creating table testapp_message > Creating table testapp_avatar > Installing custom SQL for testapp.Message model > Failed to install custom SQL for testapp.Message model: no such index: > testapp_message_avatar_id > Installing index testapp.Message model > > Installing json fixture 'initial_data' from '/fixtures'. > > As you most likely can tell from above, sql/message.sql contains a > "drop index ..." operation. > > (nitpick: SQlite has indexes - you could of course argue their > effectiveness :-) > > > > > Maybe I misunderstood? > > Perhaps I should've been more verbose :-) Thanks for your input > though. > > > - Ludvig > > Regards, > Johan Bergström I took the liberty of creating a ticket with attached patch at: http://code.djangoproject.com/ticket/10236 Thanks, Johan Bergström --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django, initial data and custom SQL
Hey, On Feb 10, 4:51 pm, "ludvig.ericson" <ludvig.eric...@gmail.com> wrote: > On Feb 10, 1:13 pm, Johan Bergström <b...@bergstroem.nu> wrote: > > > Since Django executes my custom SQL before creating indexes, it's > > impossible to achieve something that hooks into initdb/syncdb. I know > > that it is "good custom" to create indexes after inserting data – but > > fixtures in Django is already executed after creating indexes, so that > > can't be the reason.. So, without further ado, what I would like to > > know is if there's a reason to why custom SQL is executed before index > > creation. > > Isn't this doable with initial > SQL?http://docs.djangoproject.com/en/dev/howto/initial-data/#initial-sql > > Testing here with SQLite, it'd seem it runs the custom SQL at the very > last point, so you could actually add some ALTER TABLE statements, I > guess. Again, this is testing with SQLite, and SQLite doesn't do > indexing. Actually it doesn't. I think you just did a reset/sqlall instead of sync/initdb: # cat settings.py | grep DATABASE_E DATABASE_ENGINE = "sqlite3" # python manage.py syncdb Creating table testapp_message Creating table testapp_avatar Installing custom SQL for testapp.Message model Failed to install custom SQL for testapp.Message model: no such index: testapp_message_avatar_id Installing index testapp.Message model Installing json fixture 'initial_data' from '/fixtures'. As you most likely can tell from above, sql/message.sql contains a "drop index ..." operation. (nitpick: SQlite has indexes - you could of course argue their effectiveness :-) > > Maybe I misunderstood? > Perhaps I should've been more verbose :-) Thanks for your input though. > - Ludvig Regards, Johan Bergström --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Django, initial data and custom SQL
Hello, I would like to suggest (patch will follow if someone concurs) that custom sql is executed after Django has created indexes. Django is (in my opinion) a bit optimistic regarding index creation, and by looking at pg_stat_* output I see that at least a couple of indexes on busy tables hasn't been used - ever. The obvious thing to do is to drop these. Next step would be to sync this with my test and staging environment so I can run tests against my data/application and verify that something hasn't broken - and here's basically where it gets interesting. Since Django executes my custom SQL before creating indexes, it's impossible to achieve something that hooks into initdb/syncdb. I know that it is "good custom" to create indexes after inserting data – but fixtures in Django is already executed after creating indexes, so that can't be the reason.. So, without further ado, what I would like to know is if there's a reason to why custom SQL is executed before index creation. Kind regards, Johan bergström --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Django, Simplejson and speedups
Simplejson has been getting some version increments lately and Django hasn't been including them (there are tickets), which is - in my opinion - okay. Constantly upgrading the included simplejson library is an never ending rabbit chase where required effort doesn't motivate possible benefits (read: speedups) if no serious bugs should surface. I still wonder if there's a better solution for somehow allowing users to drop in their own simplejson without resorting to edit django.utils.simplejson or overriding sys.modules. This would reduce the need for upgrading the bundled package more than once each year something as well as allowing C-speedups come into play. The current code for C-speedups looks a bit messed up as well. Including native simplejson libraries on the fly feels a bit scary since it allows for messing up both tests and running code, so that's out of the question. I know I'm short of solutions here, but the current behavior could improve. Any ideas? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django & memcache hashing
On Nov 20, 8:58 am, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote: > On Thu, 2008-11-20 at 07:20 +0300, Ivan Sagalaev wrote: > > Malcolm Tredinnick wrote: > > > Okay. If we go this path, it's something to include in Django, rather > > > than recommending yet another caching package. We either make it a > > > configuration option to force python-memcache or cmemcache or we just > > > "Do The Right Thing", with the latter being preferable. > > > What concerns me is that this will break the usage of memcached without > > Django's cache API. I had the need a couple of times to do plain > > instantiation of memcache.Client and work with it. If it won't see the > > cache the same way as Django does it would be that very issue, hard to > > debug, that started this thread. > > This is indeed a concern. I was intending to put in a module that you > can import to get the same behaviour as Django. So instead of > > import memcached > > you write > > from django.core.cache import memcached > > I'm not 100% certain, though, that this is the way to go. I'm letting it > bounce around for a few days. Both options have their drawbacks and it's > kind of a matter of weighing up which inconvenience is more likely to > occur, given that they're both relatively uncommon (after all, if you're > accessing Django objects via direct usage, you need to be using Django's > get_cache_key() and the like anyway). I'm not sure this is the way to go. Personally, I use memcached from lots of distributed applications where (only) one of them is backed by Django. It would be a bit inconvenient to import Django into my other applications in order to make sure that I consistently use the same hashing algorithm. Hashing should be up to the library itself. Modern memcached libraries nowadays also give you alternatives for consistent distributions (ketama and others) which makes python-memcached look a bit old. For reasons raised in this thread (as well as beeing linked to the crash-prone libmemcache library), I don't think that cmemcache belongs as an alternative in Django. Either offer an alternative (pylibmc comes to mind, albeit at very young age) for performance or remove it. > > Regards, > Malcolm Thanks, Johan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Pluggable filestorage [was Heads-up: doing a bit of triage work]
On Sep 18, 1:21 pm, "Marty Alchin" <[EMAIL PROTECTED]> wrote: > On 9/18/07, Johan Bergström <[EMAIL PROTECTED]> wrote: > > > What is wrong with using FTP for intranet file shuffling? I don't > > think frameworks should decide or advocate how you design your > > application from a security point of view. Offering both SFTP/FTP with > > documented recommendations towards SFTP is in my opinion the way > > forward. > > There's nothing stopping you from implementing FTP in your own > environment. Jacob's just recommending that we don't include it in the > core distribution. One big point of this whole filestorage thing is so > that you don't have to rely solely on what's provided. Point taken. Since FTP and SFTP are so closely related i still think that the 'why' or 'how do i' would show more often than not if FTP was omitted. My view of frameworks is generally based on DRY - but i guess at some point it is wiser to pick 'better' (in this case more secure) instead of 'most common'. > > -Gul Thanks, Johan Bergström --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Pluggable filestorage [was Heads-up: doing a bit of triage work]
On Sep 17, 6:13 pm, "Jacob Kaplan-Moss" <[EMAIL PROTECTED]> wrote: > On 9/17/07, jedie <[EMAIL PROTECTED]> wrote: > > > What's about a FTP backend? ;) > > I'd be -1 on including one with Django for the simple reason that FTP > is dangerously insecure. Anyone still using FTP should be encouraged > strongly to switch. What is wrong with using FTP for intranet file shuffling? I don't think frameworks should decide or advocate how you design your application from a security point of view. Offering both SFTP/FTP with documented recommendations towards SFTP is in my opinion the way forward. > The is the same reasoning behind not shipping with a CGI handler -- > Django shouldn't make it easy to do stupid things. I don't think this is the same thing. Why use encryption when you trust the sender? > Now, a SSH/SFTP backend... that would rock. > > Jacob best regards, Johan Bergström --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Decouple simplejson from Django?
Why not put simplejson as a depedency in setup.py, so easy_install fetches the latest avaliable version instead of the possibly older bundled version then? I get the point, as simplejson is a very integrated part in core functionality. It is still a third party though and the 'requirement' can be solved on a similar fashion as above. thanks, Johan Bergström --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Decouple simplejson from Django?
> Can you use the ORM without a database adapter? Ok, that might sound a bit stupid since you already kind of answered that in the post above. What i meant was that Django has lot of different imports, and without some of them (say a database adapter) Django is crippled. I personally don't see adding simplejson to "requirements" an issue - all we do is moving updates to the user. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Decouple simplejson from Django?
On Jul 12, 8:55 am, "James Bennett" <[EMAIL PROTECTED]> wrote: > The other modules you mention are optional to a certain extent (don't > technically need flup to do Django-as-WSGI, can use Django without a > database or with your choice of DB adapter), but simplejson is > absolutely required for the serialization system (and hence test > fixtures and initial data loading) to work. Similarly, Django's > internal dispatch system relies on pydispatcher and will not function > without it, so pydisptatcher is bundled directly in Django. > > -- > "Bureaucrat Conrad, you are technically correct -- the best kind of correct." Can you use the ORM without a database adapter? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Decouple simplejson from Django?
Is there a specific reason simplejson is bundled with Django? The way i see it, there are two main reasons to decouple this: * Concurrency (for instance: flup and all db connectors are decoupled) * Less updates to merge into tree (user already handles other decoupled updates him/herself) cheers, Johan Bergström --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: make db form settings
I recently made a patch that can be used to get a similar result.
You can find the patch here: http://code.djangoproject.com/ticket/4528
regards,
Johan Bergström
On Jul 4, 7:54 pm, Carl Karsten <[EMAIL PROTECTED]> wrote:
> I have the begging of something that may eventually get submitted, and looking
> for guidance as I build it.
>
> In the spirit of DRY, I have a nifty script that helps create the db defined
> in
> settings.py
>
> #!/usr/bin/env python
> # mkdbuser.py
> # prints the CREATE DATABASE and GRANT commands based on the local settings.py
>
> import settings
>
> SQL = """
> DROP DATABASE IF EXISTS %(db)s;
> CREATE DATABASE %(db)s;
> GRANT ALL
> ON %(db)s.*
> TO %(user)s
> IDENTIFIED BY '%(pw)s'
> with grant option;
>
> """
>
> print SQL % {
> 'db':settings.DATABASE_NAME,
> 'user':settings.DATABASE_USER,
> 'pw':settings.DATABASE_PASSWORD }
>
> # end: mkdbuser.py
>
> I pipe it into the client:
> $ ./mkdbuser.py | mysql -u root -p
> and presto! I have the db that syncdb wants.
>
> I am thinking it would be nice if it was hung off of db/backends/mysql so that
> it was driven by DATABASE_ENGINE and the various backends could spit out the
> appropriate dialect.
>
> I am also researching exactly what rights are needed by syncdb and runserver
> or
> any production use so that what is spit out isn't quite so wide open. If
> anyone
> has already done this, I would rather not repeat it. :) however, if someone
> wants me to anyway so we can compare notes to make sure we didn't mis
> anything,
> thats fine too.
>
> Carl K
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
