Re: Sealing or locking QuerySets -- a suggestion to prevent surprise N+1 queries.

[Andres Osinski]

I'm very interested in this feature and would love to assist in making it
happen.

On Thu, Jan 4, 2018 at 12:34 AM, charettes  wrote:

> Hey everyone,
>
> I also believe this is a feature that should be in core
> judging by the number of times I had to enable query logging
> in the past or rely on assertNumQueries to make sure no extra
> queries would be inadvertently introduced in a critical code
> path.
>
> In the light of Anssi's comments on #26481 I tried implementing
> a PoC as a third party app and while it required a bit of hacking
> with internal APIs it was easier than I initially expected[0].
>
> I only had to monkey patch ManyToManyField.contribute_to_class()
> because unlike ForeignObject it doesn't rely on class level attributes
> to create it's forward descriptor[1].
>
> That gives me hope that it should be feasible to add such a feature
> to core if deemed useful without invasive changes.
>
> Cheers,
> Simon
>
> [0] https://github.com/charettes/django-seal
> [1] https://github.com/django/django/blob/602481d0c9edb79692955e073fa481
> c322f1df47/django/db/models/fields/related.py#L1590
>
>
> Le mercredi 3 janvier 2018 12:34:21 UTC-5, Bryan Helmig a écrit :
>>
>> At Zapier we're working with some rather complex and performance
>> sensitive QuerySet building (most currently around experimenting with
>> GraphQL) and I am constantly worried that the laziness of the ORM will
>> surprise us in production (after a few levels of related nesting, the math
>> of a mistake starts to really, really hurt). While not difficult to
>> prevent, it does get a bit tedious maintaining constant vigilance :tm:. I
>> was curious if sealing or locking a QuerySet has ever been seriously
>> discussed before.
>>
>> For example, you might do something like:
>>
>> class Tag(models.Model):
>> name = models.CharField()
>> slug = models.SlugField()
>> # etc...
>>
>> def get_url(self):
>> return reverse('tag_index', args=[self.slug])
>>
>> class Post(models.Model):
>> title = models.CharField()
>> tags = models.ManyToManyField(Tag)
>> # etc...
>>
>> posts = Post.objects.only(
>> 'id', 'title',
>> ).prefetch_related(
>> Prefetch('tags', queryset=Tag.objects.only('id', 'name')),
>> )
>>
>> # we'd .seal() or .strict() and evaluate the queryset here:
>> for post in posts.seal():
>> # get_url() would normally execute the classic N+1 queries
>> # but after qs.seal() would instead raise some exception
>> print ', '.join(tag.get_url() for tag in post.tags.all())
>>
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> SealedQuerySetException: Cannot load sealed deferred attribute 'slug' on
>> prefetched Tag model.
>>
>> Of course the obvious solution is to just add 'slug' to only(), but in a
>> sufficiently complex application with many engineers working across various
>> app boundaries it is both difficult to predict and test against. It
>> requires lots of explicit self.assertNumQueries() and in the worse case can
>> cause "production surprise" as deeply nested QuerySets can potentially
>> explode in queries.
>>
>> This doesn't apply only to deferred attributes, but also related
>> OneToOne, ManyToOne, ManyToMany, etc. lazy resolution. FWIW, I would image
>> the FK/M2M N+1 pattern is a much more common surprise than opting into
>> .only() or .defer() as my example above outlines.
>>
>> I had a go at implementing it outside of Django core, but ended up having
>> to go down a monkey patching rabbit hole so I thought I'd reach out to the
>> mailing list. I'd imagine setting a flag when evaluating a QuerySet to
>> disable the laziness of a Model's fields and relations would be sufficient
>> (IE: if it isn't in cache, raise SealedQuerySetException). It also seems
>> backwards compatible, if you don't use .seal() you are still lazy like
>> before.
>>
>> So, I guess my questions are:
>>
>> 1. Are there any other past discussions around this topic I can look at?
>> I did find https://code.djangoproject.com/ticket/26481 which seems
>> similar, but doesn't mention relationships.
>> 2. Does this seem useful to anyone else? Specifically opting to raise
>> explicit exceptions instead of automatic (and sometimes surprising) queries.
>> 3. Anyone have tips on implementing this as a library external to Django
>> with lots of monkey patching?
>>
>> I'd be happy to take a swing at it if there was a >50% chance that it
>> would be at least considered.
>>
>> -bryan, cto & cofounder @ zapier
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 

Re: future of QuerySet.extra()?

[Andres Osinski]

Would the expressions API be able to define fields that are aggregations of
complex foreign relationships? I keep using it when I need fields that are
the result of weird aggregations over multiple JOINs with some unusual
stuff like CASE fields and subqueries.

On Fri, Jul 31, 2015 at 2:58 PM, Tim Graham  wrote:

> In light of the new expressions API, the idea of deprecating
> QuerySet.extra() has been informally discussed in IRC and elsewhere. I
> wonder if there is consensus to mark extra() as "unmaintained" and to
> suggest filing feature requests for functionality that can be performed
> through extra() but not through other existing QuerySet methods? There are
> at least several tickets (examples below) of edge cases that don't work
> with extra(). It seems like a waste of time to leave these tickets as
> accepted and to triage new issues with extra() if they won't be fixed.
>
> https://code.djangoproject.com/ticket/24142
> https://code.djangoproject.com/ticket/19434
> https://code.djangoproject.com/ticket/12890
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/6e1be326-3b17-49ca-accf-03eec5ad41ef%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Andrés Osinski

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BxF-PwnnPNyOOUzUHPaYG1mRQS7Fynys0o%3DoK9_g2ESTBRM2w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Consider reopening ticket #11385

[Andres Osinski]

Hi everyone, I wanted to bring up a request: Ticket #11385 dealt with
Django's DateTimeField being unable to handle ISO 8601 date formats.

At the time it seemed reasonable to not support this feature given,
however, given the proliferation of web service APIs that return dates
using this format, it is natural to use forms to validate the outputs of
such services, and having it not baked in while depending on strftime's
formatting behavior (which seems to fail tremendously for such datetime
formats) makes working with such data very awkward.

I can volunteer to do the bulk of the effort if there's any chance to have
a more experienced member of the community go over my work to make sure a
decent patch can be submitted.

Cheers

-- 
Andrés Osinski

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BxF-PzQ1Vg4aTReu0ZW%3DdOAc2HCzysaLN73%3Dk%3D9-2%2BQwWv8Ow%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: When to use single quotes and double quotes

[Andres Osinski]

I don't know where I got this from, but I consider it to be the best
convention for choosing quote types:

Use double quotes for strings meant for human consumption, use single
quotes for everything else.

Text, trans strings, model verbose_names: double quotes
Dict keys, HTML tags, flags, and similar: single quotes

Use triple quotes for long blocks of text, ideally only necessary for
human-readable strings.

On Tue, Oct 29, 2013 at 12:31 AM, Zach Borboa  wrote:
>
>> Here's an example from the django source where we are using double quotes
>> in python instead of single quotes. The following code prints out html using
>> single quotes instead of the double quotes regularly seen in html.
>>
>> return format_html("> value='{0}' />", csrf_token)
>>
>> instead of:
>>
>> return format_html('> value="{0}" />', csrf_token)
>
>
>
> This message inadvertently got deleted. If it was supposed to be removed. My
> apologies; feel free to again remove it.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/ca45bbf4-5167-4f6b-ad3b-517957e10ced%40googlegroups.com.
>
> For more options, visit https://groups.google.com/groups/opt_out.



-- 
Andrés Osinski
http://www.andresosinski.com.ar/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BxF-Pyr6NHLd4AC8us%3DCAk4%3DpXht_tKBOxTeL9RVkhc9rjUfA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Add strutctured settings module to django 1.7?

[Andres Osinski]

Honestly, I find the idea of *not* using a settings file to be
inconvenient, and the notion of using environment variables for doing
anything aside from pointing to a settings file to be pretty disgusting.

Here are my thoughts, in no particular order:
- Pretty much everyone out there uses some form of settings +
local_settings.py for configuration, keeping the installation site-specific
in the local settings, and checking in a local_settings.example.py file for
instructions. This has never really been an issue for me except for
database-specific content settings, which is not something environment
variables solve.
- The system environment variables is a flat namespace, completely polluted
by different shell's variables, with high potential for unintended name
clashes.
- For development, reloading an env var does not reset the development
server; I usually *want* the dev server to be reset whenever there's a
config change.
- You have to set your env vars from somewhere, adding an extra layer of
indirection and complexity.
- Not all shells have facilities to easily navigate settings; it's easier
if they're stored in a startup shell file, but that offers no advantage
from a Django file
- I like the fact that you can use Python facilities like os.path.join to
generate config variables on startup.


On Thu, Sep 26, 2013 at 9:55 AM, VernonCole  wrote:

> (Memo to self: always think about it for a day before replying.)
>
> Okay, I see.  And I certainly agree that a few strategically placed
> instances of (for example)
> >>> psw = os.environ.get('DBPASSWORD', 'DumbDefault')
> would help security in most cases.
>
> I will get back to the drawing board, then, and see if I can still get the
> optimization I am looking for without making the system more difficult for
> those who don't want it.
>
> My use case, is, I'm afraid, even more wrong than you suggest, Marc.  I
> develop on Linux, but have to plan for instances where unfortunate
> individuals must deploy on Windows.  Administrators of those systems are
> often neophytes at systems administration in general and Python in
> particular.  Installation of something like envdir would be far beyond
> hope.   More frequent will be deployments by people who may know Windows
> systems administration, but this will be their first experience with
> Linux.  Simplicity is everything.
>
> Thank you, everyone, for the quality input.
> --
> Vernon
>
> On Thursday, September 26, 2013 11:26:01 AM UTC+1, Marc Tamlyn wrote:
>>
>> No, when 12 factor says in the environment, it means exactly that. *NOT*
>> checked into version control, and in the environment of the shell.
>> 12-factor is talking *only* about number 1 for differences in your
>> environments.
>>
>> Yes windows is not very good at environment variables in general, but I
>> would argue if you're developing on windows and deploying to *nix you're
>> already doing it wrong. I refer you to point 10. You may however be
>> interested in Jannis' python port of envdir - https://github.com/jezdez/*
>> *envdir  - which works on Windows.
>>
>> Personally, I find the multiple settings files (local_settings.py,
>> structured settings) to be a pattern I'm moving away from, and I'm unlikely
>> to support changes to Django encouraging it more. Jannis' configurations is
>> a bit different, I find it's a nicer way of grouping your settings
>> together, making them overrideable from external apps etc. The end game is
>> still one settings file, with environment switches.
>>
>> The advantage of the current settings.py is that it's very simple and
>> flexible, which allows these patterns to all work together. At some point
>> we may change to another system, but I don't feel we should be actively
>> encouraging any particular path at present. There isn't a clear consensus
>> as to the best way forwards.
>>
>>
>> On 26 September 2013 11:11, VernonCole  wrote:
>>
>>>
>>> Okay, I read it.  I think that it argues for structured settings, not
>>> against them.
>>>
>>> What I read was a set of guidelines, rather like the Zen of Python. They
>>> look like good, reasonable rules-of-thumb to me, and I have been around
>>> long enough to recognize such.  One of the reasons I adopted Python was
>>> that I found my self in such complete agreement with the Zen the first time
>>> I read it. I particularly liked its built-in admission that there will be
>>> exceptions to its principles.
>>>
>>> Let's talk about three of these twelve.  I have cut-and-paste-ed them
>>> from the web page.
>>>
>>> I. Codebase  One codebase tracked in
 revision control, many deploysIII. Config  
 Store
 config in the environmentX. Dev/prod 
 parity Keep
 development, staging, and production as similar as possible
>>>
>>>
>>> Note that number three says "in the environment" not "in a 

Re: Admin for proxy model for custom User

[Andres Osinski]

Sorry, it was my intention but had the wrong autocomplete. My apologies.

On Tue, May 28, 2013 at 10:00 PM, Angus Turner <angusisf...@gmail.com> wrote:
> You'd be better off posting this to django-users as this list is for the
> development of django itself.
>
> Thanks
> Angus Turner
> angusisf...@gmail.com
>
>
> On Wed, May 29, 2013 at 10:33 AM, Andres Osinski <andres.osin...@gmail.com>
> wrote:
>>
>> Hi everyone, I've created a custom User model and reated a proxy for
>> it, like this:
>>
>> class User(AbstractUser):
>> USER_KINDS = (
>> ('d', 'doctor'),
>> ('p', 'patient')
>> )
>>  kind = models.CharField(max_length=2, choices=USER_KINDS)
>>
>> class DoctorManager(BaseUserManager):
>> def get_queryset(self):
>> return super(DoctorManager, self).get_queryset().filter(kind='d')
>>
>>
>> class PatientManager(BaseUserManager):
>> def get_queryset(self):
>> return super(DoctorManager, self).get_queryset().filter(kind='p')
>>
>>
>> class Doctor(User):
>> objects = DoctorManager()
>>
>> class Meta:
>> proxy = True
>>
>>
>> class Patient(User):
>> objects = PatientManager()
>>
>> class Meta:
>> proxy = True
>>
>> 
>>
>>
>> Now, my admin has the following code:
>>
>> # Forms subclassing default auth forms
>> class DemographicsUserAdmin(reversion.VersionAdmin, UserAdmin):
>> form = DemographicsUserChangeForm
>> add_form = DemographicsUserCreationForm
>>
>> class PatientAdmin(admin.ModelAdmin):
>> add_form_template = 'admin/auth/user/add_form.html'
>> form = PatientChangeForm
>> add_form = PatientCreationForm
>>
>> def get_queryset(self, request):
>> return Patient.objects.all()
>>
>> class DoctorAdmin(admin.ModelAdmin):
>> add_form_template = 'admin/auth/user/add_form.html'
>> form = PatientChangeForm
>> add_form = PatientCreationForm
>>
>> def get_queryset(self, request):
>> return Doctor.objects.all()
>>
>>
>>
>> admin.site.register(User, DemographicsUserAdmin)
>> admin.site.register(Doctor, DoctorAdmin)
>> admin.site.register(Patient, PatientAdmin)
>>
>> --
>>
>>
>> Now, this would seem to indicate that the proxy model admins would be
>> filtering only those users with the proper kind. However, not only is
>> this not happening, but the get_queryset() methods for either admins
>> is not running, nor the default objects() manager. Any idea why this
>> would be happening?
>>
>> Thanks
>> --
>> Andrés Osinski
>> http://www.andresosinski.com.ar/
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at
>> http://groups.google.com/group/django-developers?hl=en.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



-- 
Andrés Osinski
http://www.andresosinski.com.ar/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Admin for proxy model for custom User

[Andres Osinski]

Hi everyone, I've created a custom User model and reated a proxy for
it, like this:

class User(AbstractUser):
USER_KINDS = (
('d', 'doctor'),
('p', 'patient')
)
 kind = models.CharField(max_length=2, choices=USER_KINDS)

class DoctorManager(BaseUserManager):
def get_queryset(self):
return super(DoctorManager, self).get_queryset().filter(kind='d')


class PatientManager(BaseUserManager):
def get_queryset(self):
return super(DoctorManager, self).get_queryset().filter(kind='p')


class Doctor(User):
objects = DoctorManager()

class Meta:
proxy = True


class Patient(User):
objects = PatientManager()

class Meta:
proxy = True




Now, my admin has the following code:

# Forms subclassing default auth forms
class DemographicsUserAdmin(reversion.VersionAdmin, UserAdmin):
form = DemographicsUserChangeForm
add_form = DemographicsUserCreationForm

class PatientAdmin(admin.ModelAdmin):
add_form_template = 'admin/auth/user/add_form.html'
form = PatientChangeForm
add_form = PatientCreationForm

def get_queryset(self, request):
return Patient.objects.all()

class DoctorAdmin(admin.ModelAdmin):
add_form_template = 'admin/auth/user/add_form.html'
form = PatientChangeForm
add_form = PatientCreationForm

def get_queryset(self, request):
return Doctor.objects.all()



admin.site.register(User, DemographicsUserAdmin)
admin.site.register(Doctor, DoctorAdmin)
admin.site.register(Patient, PatientAdmin)

--


Now, this would seem to indicate that the proxy model admins would be
filtering only those users with the proper kind. However, not only is
this not happening, but the get_queryset() methods for either admins
is not running, nor the default objects() manager. Any idea why this
would be happening?

Thanks
--
Andrés Osinski
http://www.andresosinski.com.ar/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Perception of attitude in tickets

[Andres Osinski]

I really don't know how you think you'll be getting a different response in
other frameworks, because the core developers' attitude on this is correct.

There are over 2 tickets in the bug tracker for a project used by tens
of thousands of people. Code has to pass style guidelines, regression
tests, documentation must be made, features must be integrated into a
release schedule, obscure platform bugs and overall consistency with the
rest of the code has to be verified.

This takes a *long* time. Adding an extra method to the Model class is
something that may take weeks to discuss and to consider in a myriad
possible aspects. And what's worse, someone after you will have to maintain
the whole thing, because retrospectively removing things is a *really bad*
practice on code that you don't own.

Discussions take place on the mailing list and chat rooms because bug
trackers are not and have never been a place to discuss design decisions or
implementation details; just a place to track the progress in the
development and status of a bug or feature request.

Finally, Django as a framework is *massive*, and really good reasons have
to be had in order to increment the line count even more for something that
is ostensibly a corner case for people who don't wish to do a bit of
additional work by working within the framework's constraints and opinions
instead of against them.

Unfortunately, given that no framework is as large or feature-complete
(Pyramid supports a lot of things, but is against including too much in the
core framework and instead depends on external libs), you'll have even
*less* luck with other pieces of code. And I would certainly distrust any
framework that has less rigor than Django for including features; what core
devs giveth, core devs may deprecate on the next release.

Also, I beleive your perception of Django's openness if flat-out erroneous.
Go try to add an unrequested feature in Rails or Yii without discussion and
see how you're welcomed, never mind the fact that both of these framework's
development is done completely in the dark for many of the most important
features.


On Fri, May 10, 2013 at 12:45 PM, Simon  wrote:

> Hi.
>
> I'm a newcomer to both Python and Django and just wanted to share my
> experience trying to solve a couple of problems.
>
> When I started coding in Python a month ago, Django was sufficiently
> common in Google searches that it was my first port of call. I've found
> quite a few features which I love and a few which seem a little quirky.
>
> Unfortunately, every time I come across a problem, I seem to end up back
> at the ticket system which almost invariably results in lots of people
> saying "It's important to me" and a core developer saying (rather
> arbitrarily) "I don't see the need" and closing as wontfix.
>
> Of course, once the ticket has been closed, the only way to appeal is
> through the mailing lists. To myself as a newcomer, that just feels like a
> way of making further dissent less visible. I'm sure this isn't the case
> but that's certainly the feeling I got.
>
> The most recent example, which prompted this post is
> https://code.djangoproject.com/ticket/901 but this is the 5th or 6th time
> I've ended up at similar tickets with similar outcomes. I think comment
> #20  demonstrates
> my point perfectly - While the point being made is valid, the tone and
> general attitude is poor to say the least.
>
> In any case, I shall be moving away from Django before I get too tied to
> it. I know this won't impact you in the slightest but I suspect other
> newcomers will be doing the same and not taking the time to post.
>
> All the best
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>



-- 
Andrés Osinski
http://www.andresosinski.com.ar/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.