Re: Sealing or locking QuerySets -- a suggestion to prevent surprise N+1 queries.

[Shai Berger]

Hi all,

Django 2.0 has a new feature[1] which allows you to say "I expect no actual 
database queries in this piece of code". It is not designed to stop a specific 
queryset from spawning requests, so getting it to do exactly what's asked for 
in this thread may be a little involved, but if your goal is to prevent 
surprise queries, I think it is actually better than sealing a single 
queryset.

HTH,
Shai

[1] https://docs.djangoproject.com/en/2.0/topics/db/instrumentation/

Re: Sealing or locking QuerySets -- a suggestion to prevent surprise N+1 queries.

[Andres Osinski]

I'm very interested in this feature and would love to assist in making it
happen.

On Thu, Jan 4, 2018 at 12:34 AM, charettes  wrote:

> Hey everyone,
>
> I also believe this is a feature that should be in core
> judging by the number of times I had to enable query logging
> in the past or rely on assertNumQueries to make sure no extra
> queries would be inadvertently introduced in a critical code
> path.
>
> In the light of Anssi's comments on #26481 I tried implementing
> a PoC as a third party app and while it required a bit of hacking
> with internal APIs it was easier than I initially expected[0].
>
> I only had to monkey patch ManyToManyField.contribute_to_class()
> because unlike ForeignObject it doesn't rely on class level attributes
> to create it's forward descriptor[1].
>
> That gives me hope that it should be feasible to add such a feature
> to core if deemed useful without invasive changes.
>
> Cheers,
> Simon
>
> [0] https://github.com/charettes/django-seal
> [1] https://github.com/django/django/blob/602481d0c9edb79692955e073fa481
> c322f1df47/django/db/models/fields/related.py#L1590
>
>
> Le mercredi 3 janvier 2018 12:34:21 UTC-5, Bryan Helmig a écrit :
>>
>> At Zapier we're working with some rather complex and performance
>> sensitive QuerySet building (most currently around experimenting with
>> GraphQL) and I am constantly worried that the laziness of the ORM will
>> surprise us in production (after a few levels of related nesting, the math
>> of a mistake starts to really, really hurt). While not difficult to
>> prevent, it does get a bit tedious maintaining constant vigilance :tm:. I
>> was curious if sealing or locking a QuerySet has ever been seriously
>> discussed before.
>>
>> For example, you might do something like:
>>
>> class Tag(models.Model):
>> name = models.CharField()
>> slug = models.SlugField()
>> # etc...
>>
>> def get_url(self):
>> return reverse('tag_index', args=[self.slug])
>>
>> class Post(models.Model):
>> title = models.CharField()
>> tags = models.ManyToManyField(Tag)
>> # etc...
>>
>> posts = Post.objects.only(
>> 'id', 'title',
>> ).prefetch_related(
>> Prefetch('tags', queryset=Tag.objects.only('id', 'name')),
>> )
>>
>> # we'd .seal() or .strict() and evaluate the queryset here:
>> for post in posts.seal():
>> # get_url() would normally execute the classic N+1 queries
>> # but after qs.seal() would instead raise some exception
>> print ', '.join(tag.get_url() for tag in post.tags.all())
>>
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> SealedQuerySetException: Cannot load sealed deferred attribute 'slug' on
>> prefetched Tag model.
>>
>> Of course the obvious solution is to just add 'slug' to only(), but in a
>> sufficiently complex application with many engineers working across various
>> app boundaries it is both difficult to predict and test against. It
>> requires lots of explicit self.assertNumQueries() and in the worse case can
>> cause "production surprise" as deeply nested QuerySets can potentially
>> explode in queries.
>>
>> This doesn't apply only to deferred attributes, but also related
>> OneToOne, ManyToOne, ManyToMany, etc. lazy resolution. FWIW, I would image
>> the FK/M2M N+1 pattern is a much more common surprise than opting into
>> .only() or .defer() as my example above outlines.
>>
>> I had a go at implementing it outside of Django core, but ended up having
>> to go down a monkey patching rabbit hole so I thought I'd reach out to the
>> mailing list. I'd imagine setting a flag when evaluating a QuerySet to
>> disable the laziness of a Model's fields and relations would be sufficient
>> (IE: if it isn't in cache, raise SealedQuerySetException). It also seems
>> backwards compatible, if you don't use .seal() you are still lazy like
>> before.
>>
>> So, I guess my questions are:
>>
>> 1. Are there any other past discussions around this topic I can look at?
>> I did find https://code.djangoproject.com/ticket/26481 which seems
>> similar, but doesn't mention relationships.
>> 2. Does this seem useful to anyone else? Specifically opting to raise
>> explicit exceptions instead of automatic (and sometimes surprising) queries.
>> 3. Anyone have tips on implementing this as a library external to Django
>> with lots of monkey patching?
>>
>> I'd be happy to take a swing at it if there was a >50% chance that it
>> would be at least considered.
>>
>> -bryan, cto & cofounder @ zapier
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 

Re: Sealing or locking QuerySets -- a suggestion to prevent surprise N+1 queries.

[charettes]

Hey everyone,

I also believe this is a feature that should be in core
judging by the number of times I had to enable query logging
in the past or rely on assertNumQueries to make sure no extra
queries would be inadvertently introduced in a critical code
path.

In the light of Anssi's comments on #26481 I tried implementing
a PoC as a third party app and while it required a bit of hacking
with internal APIs it was easier than I initially expected[0].

I only had to monkey patch ManyToManyField.contribute_to_class()
because unlike ForeignObject it doesn't rely on class level attributes
to create it's forward descriptor[1].

That gives me hope that it should be feasible to add such a feature
to core if deemed useful without invasive changes.

Cheers,
Simon

[0] https://github.com/charettes/django-seal
[1] 
https://github.com/django/django/blob/602481d0c9edb79692955e073fa481c322f1df47/django/db/models/fields/related.py#L1590

Le mercredi 3 janvier 2018 12:34:21 UTC-5, Bryan Helmig a écrit :
>
> At Zapier we're working with some rather complex and performance sensitive 
> QuerySet building (most currently around experimenting with GraphQL) and I 
> am constantly worried that the laziness of the ORM will surprise us in 
> production (after a few levels of related nesting, the math of a mistake 
> starts to really, really hurt). While not difficult to prevent, it does get 
> a bit tedious maintaining constant vigilance :tm:. I was curious if sealing 
> or locking a QuerySet has ever been seriously discussed before.
>
> For example, you might do something like:
>
> class Tag(models.Model):
> name = models.CharField()
> slug = models.SlugField()
> # etc...
>
> def get_url(self):
> return reverse('tag_index', args=[self.slug])
>
> class Post(models.Model):
> title = models.CharField()
> tags = models.ManyToManyField(Tag)
> # etc...
>
> posts = Post.objects.only(
> 'id', 'title',
> ).prefetch_related(
> Prefetch('tags', queryset=Tag.objects.only('id', 'name')),
> )
>
> # we'd .seal() or .strict() and evaluate the queryset here:
> for post in posts.seal():
> # get_url() would normally execute the classic N+1 queries
> # but after qs.seal() would instead raise some exception
> print ', '.join(tag.get_url() for tag in post.tags.all())
>
> Traceback (most recent call last):
>   File "", line 1, in 
> SealedQuerySetException: Cannot load sealed deferred attribute 'slug' on 
> prefetched Tag model.
>
> Of course the obvious solution is to just add 'slug' to only(), but in a 
> sufficiently complex application with many engineers working across various 
> app boundaries it is both difficult to predict and test against. It 
> requires lots of explicit self.assertNumQueries() and in the worse case can 
> cause "production surprise" as deeply nested QuerySets can potentially 
> explode in queries.
>
> This doesn't apply only to deferred attributes, but also related 
> OneToOne, ManyToOne, ManyToMany, etc. lazy resolution. FWIW, I would image 
> the FK/M2M N+1 pattern is a much more common surprise than opting into 
> .only() or .defer() as my example above outlines.
>
> I had a go at implementing it outside of Django core, but ended up having 
> to go down a monkey patching rabbit hole so I thought I'd reach out to the 
> mailing list. I'd imagine setting a flag when evaluating a QuerySet to 
> disable the laziness of a Model's fields and relations would be sufficient 
> (IE: if it isn't in cache, raise SealedQuerySetException). It also seems 
> backwards compatible, if you don't use .seal() you are still lazy like 
> before.
>
> So, I guess my questions are:
>
> 1. Are there any other past discussions around this topic I can look at? I 
> did find https://code.djangoproject.com/ticket/26481 which seems similar, 
> but doesn't mention relationships.
> 2. Does this seem useful to anyone else? Specifically opting to raise 
> explicit exceptions instead of automatic (and sometimes surprising) queries.
> 3. Anyone have tips on implementing this as a library external to Django 
> with lots of monkey patching?
>
> I'd be happy to take a swing at it if there was a >50% chance that it 
> would be at least considered.
>
> -bryan, cto & cofounder @ zapier
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3c332e75-d262-43ad-8882-e4e54bb738a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sealing or locking QuerySets -- a suggestion to prevent surprise N+1 queries.

[Tobias McNulty]

1. I also could not find anything other than #26481 and a brief discussion
 of
it on the list.

2. I've often wished for something like this, but ended up resorting to
assertNumQueries() for lack of a better solution. So yes, it'd certainly be
useful to me/us, and I'd be curious to see how one might go about
implementing it.

3. My gut says that an API, in Django core, that builds off of #26481
(e.g., .only(strict=True) and .defer(strict=True)) is the right direction
(even for related fields), and my preference would be for a loud failure
(i.e., an exception rather than a log message) when this "strict" mode is
enabled. One can easily downgrade that to a log message when needed, less
so the other way around.



*Tobias McNulty*Chief Executive Officer

tob...@caktusgroup.com
www.caktusgroup.com

On Wed, Jan 3, 2018 at 12:33 PM, 'Bryan Helmig' via Django developers
(Contributions to Django itself)  wrote:

> At Zapier we're working with some rather complex and performance sensitive
> QuerySet building (most currently around experimenting with GraphQL) and I
> am constantly worried that the laziness of the ORM will surprise us in
> production (after a few levels of related nesting, the math of a mistake
> starts to really, really hurt). While not difficult to prevent, it does get
> a bit tedious maintaining constant vigilance :tm:. I was curious if sealing
> or locking a QuerySet has ever been seriously discussed before.
>
> For example, you might do something like:
>
> class Tag(models.Model):
> name = models.CharField()
> slug = models.SlugField()
> # etc...
>
> def get_url(self):
> return reverse('tag_index', args=[self.slug])
>
> class Post(models.Model):
> title = models.CharField()
> tags = models.ManyToManyField(Tag)
> # etc...
>
> posts = Post.objects.only(
> 'id', 'title',
> ).prefetch_related(
> Prefetch('tags', queryset=Tag.objects.only('id', 'name')),
> )
>
> # we'd .seal() or .strict() and evaluate the queryset here:
> for post in posts.seal():
> # get_url() would normally execute the classic N+1 queries
> # but after qs.seal() would instead raise some exception
> print ', '.join(tag.get_url() for tag in post.tags.all())
>
> Traceback (most recent call last):
>   File "", line 1, in 
> SealedQuerySetException: Cannot load sealed deferred attribute 'slug' on
> prefetched Tag model.
>
> Of course the obvious solution is to just add 'slug' to only(), but in a
> sufficiently complex application with many engineers working across various
> app boundaries it is both difficult to predict and test against. It
> requires lots of explicit self.assertNumQueries() and in the worse case can
> cause "production surprise" as deeply nested QuerySets can potentially
> explode in queries.
>
> This doesn't apply only to deferred attributes, but also related
> OneToOne, ManyToOne, ManyToMany, etc. lazy resolution. FWIW, I would image
> the FK/M2M N+1 pattern is a much more common surprise than opting into
> .only() or .defer() as my example above outlines.
>
> I had a go at implementing it outside of Django core, but ended up having
> to go down a monkey patching rabbit hole so I thought I'd reach out to the
> mailing list. I'd imagine setting a flag when evaluating a QuerySet to
> disable the laziness of a Model's fields and relations would be sufficient
> (IE: if it isn't in cache, raise SealedQuerySetException). It also seems
> backwards compatible, if you don't use .seal() you are still lazy like
> before.
>
> So, I guess my questions are:
>
> 1. Are there any other past discussions around this topic I can look at? I
> did find https://code.djangoproject.com/ticket/26481 which seems similar,
> but doesn't mention relationships.
> 2. Does this seem useful to anyone else? Specifically opting to raise
> explicit exceptions instead of automatic (and sometimes surprising) queries.
> 3. Anyone have tips on implementing this as a library external to Django
> with lots of monkey patching?
>
> I'd be happy to take a swing at it if there was a >50% chance that it
> would be at least considered.
>
> -bryan, cto & cofounder @ zapier
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/962ca077-cde8-476e-92db-
> 47adf5785866%40googlegroups.com
> 
> .
> For 

Sealing or locking QuerySets -- a suggestion to prevent surprise N+1 queries.

['Bryan Helmig' via Django developers (Contributions to Django itself)]

At Zapier we're working with some rather complex and performance sensitive 
QuerySet building (most currently around experimenting with GraphQL) and I 
am constantly worried that the laziness of the ORM will surprise us in 
production (after a few levels of related nesting, the math of a mistake 
starts to really, really hurt). While not difficult to prevent, it does get 
a bit tedious maintaining constant vigilance :tm:. I was curious if sealing 
or locking a QuerySet has ever been seriously discussed before.

For example, you might do something like:

class Tag(models.Model):
name = models.CharField()
slug = models.SlugField()
# etc...

def get_url(self):
return reverse('tag_index', args=[self.slug])

class Post(models.Model):
title = models.CharField()
tags = models.ManyToManyField(Tag)
# etc...

posts = Post.objects.only(
'id', 'title',
).prefetch_related(
Prefetch('tags', queryset=Tag.objects.only('id', 'name')),
)

# we'd .seal() or .strict() and evaluate the queryset here:
for post in posts.seal():
# get_url() would normally execute the classic N+1 queries
# but after qs.seal() would instead raise some exception
print ', '.join(tag.get_url() for tag in post.tags.all())

Traceback (most recent call last):
  File "", line 1, in 
SealedQuerySetException: Cannot load sealed deferred attribute 'slug' on 
prefetched Tag model.

Of course the obvious solution is to just add 'slug' to only(), but in a 
sufficiently complex application with many engineers working across various 
app boundaries it is both difficult to predict and test against. It 
requires lots of explicit self.assertNumQueries() and in the worse case can 
cause "production surprise" as deeply nested QuerySets can potentially 
explode in queries.

This doesn't apply only to deferred attributes, but also related 
OneToOne, ManyToOne, ManyToMany, etc. lazy resolution. FWIW, I would image 
the FK/M2M N+1 pattern is a much more common surprise than opting into 
.only() or .defer() as my example above outlines.

I had a go at implementing it outside of Django core, but ended up having 
to go down a monkey patching rabbit hole so I thought I'd reach out to the 
mailing list. I'd imagine setting a flag when evaluating a QuerySet to 
disable the laziness of a Model's fields and relations would be sufficient 
(IE: if it isn't in cache, raise SealedQuerySetException). It also seems 
backwards compatible, if you don't use .seal() you are still lazy like 
before.

So, I guess my questions are:

1. Are there any other past discussions around this topic I can look at? I 
did find https://code.djangoproject.com/ticket/26481 which seems similar, 
but doesn't mention relationships.
2. Does this seem useful to anyone else? Specifically opting to raise 
explicit exceptions instead of automatic (and sometimes surprising) queries.
3. Anyone have tips on implementing this as a library external to Django 
with lots of monkey patching?

I'd be happy to take a swing at it if there was a >50% chance that it would 
be at least considered.

-bryan, cto & cofounder @ zapier

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/962ca077-cde8-476e-92db-47adf5785866%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Get all the models referring to object

[Tim Graham]

This mailing list is for the development of Django itself, not for support. 
Use the django-users mailing list for that, or IRC #django on freenode, or 
a site like Stack Overflow.

On Wednesday, January 3, 2018 at 12:12:31 PM UTC-5, mohitdu...@gmail.com 
wrote:
>
> How can i get all the Models referring to the field as foreign key.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7c805fc9-df9b-45f0-abcd-b32e9f3401bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Get all the models referring to object

[mohitdubey2695]

How can i get all the Models referring to the field as foreign key.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/202e67d1-a484-4c36-8195-708c847f47c5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Database session deletion

[George-Cristian Bîrzan]

The documentation explicitly mentions that expired sessions aren't
automatically deleted from the database, except when the user manually logs
out. [1]

That, however, isn't the case. Firstly, when trying to access a user whose
password was changed, the session is deleted from the database. [2] This,
to me, seems the correct behaviour, and I think there should be a bug filed
against the docs.

There's also a second case where this happens. [3] This one is a bit harder
to follow. The way I read it, if you're already logged in and log in again,
OR if the password is different (this is the part I have trouble
understanding, I guess this can only happen when you call login for the
same user, but don't verify the password). The latter part was added in
https://github.com/django/django/commit/fd23c06023a0585ee743c0752dc94da66694cf63
.

The first part, logging in as another user should act like a logout/login,
but, the docs need a mention of it. The second, I don't really understand,
so not sure whether what the change to the docs should say.


[1] -
https://docs.djangoproject.com/en/2.0/topics/http/sessions/#clearing-the-session-store
[2] -
https://github.com/django/django/blob/master/django/contrib/auth/__init__.py#L194
[3] -
https://github.com/django/django/blob/master/django/contrib/auth/__init__.py#L99

-- 
George-Cristian Bîrzan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMxNYabP8ayBao6FNeGMGZGAvLN%2BSQt1Cd-gcco6V6xM1k_RNQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.