Hi Josh,
As https://github.com/django/django/pull/5090 pull request merged into
master, I wanted to extend django-orm-sugar library with some
functionality, related to passing transforms or lookup objects. Currently
it's not clear to me how lookups or transforms can be used in that way.
I'm talking about mentioned earlier syntax:
GameSession.objects.filter(GreaterThan('user__profile__last_login_date',
> yesterday))
>
Could you give me a starter point for that?
Thanks,
Alexey
On Wednesday, August 26, 2015 at 6:04:02 PM UTC+3, Alexey Zankevich wrote:
>
> The patch allowing F objects for select_related and prefetch_related
> methods was trivial, unfortunately F uses "name" field as a part of
> Expression interface, so dotted querying will conflict with internal field
> (as "name" is quite a popular name for model field).
>
> >>> Contains(F.user.name, "Bob") # conflict
> Will work on another patch with custom class.
>
>
> On Tuesday, August 25, 2015 at 1:12:56 PM UTC+3, Alexey Zankevich wrote:
>
> Gotcha.
> So, F objects seem to be pretty good as generic interface to specify query
> paths, the only remaining patch - adding select_related and
> prefetch_related support. I'll preapare a preliminary pull request soon.
>
>
> On Tuesday, August 25, 2015 at 2:48:10 AM UTC+3, Josh Smeaton wrote:
>
> Expressions have built in support for ordering with asc or desc, see here:
> https://docs.djangoproject.com/en/1.8/ref/models/expressions/#django.db.models.Expression.asc
>
> Meaning that -F() is never ambiguous, which is the main reason we went for
> the F().desc() API rather than using the minus sign as an indicator of
> ordering.
>
> To use the `P` example in an order by statement:
>
> Model.objects.order_by(P.related.date_field().desc())
>
> I've "called" date_field here, but that's just assuming an API. As long as
> an expression is being returned, you can call `desc` on it.
>
> What some other ORMs do is reference fields directly on the model, though
> I'm not sure I like the verboseness (but it gives you safety):
>
> Model.objects.order_by(Model.related.date_field.desc())
>
> The extra `Model` is annoying, but it does mean that you can actually look
> up fields as you go rather than just crafting an `F()` that is eventually
> resolved into the `Col()`. This would also allow us to return expressions
> directly. Not sure how or if this API would work with the current
> metaclasses though.
>
> For what it's worth, the names `TreePath` and `FieldTree` are too low
> level for a public facing API that is supposed to make it easier/safer to
> craft field paths. One letter class names are also a bad design in general,
> so we'd need something half way between both if going for a new class.
>
>
> On Tuesday, 25 August 2015 04:13:32 UTC+10, Alexey Zankevich wrote:
>
> Hey Josh,
>
> Here is the current state of F support by different functions:
>
> * annotations (Count, Sum etc) - work
> * order_by - works partially (only asc order supported)
> * select_related, prefetch_related - don't work
>
> So, F is not a universal interface for getting paths right now. Also,
> there is another problem with F - it overloads python operators already, so
> if we overload unary operation "-F", it will confuse users as in one case
> it will return CombinedExpression (ex. 0-F('fieldname')), in the other -
> OrderBy (ex. -F('fieldname')).
> Since F originally designed to describe arithmetic upon fields during
> filtering, it will not be a perfect match to denote query paths.
> As for accepting callable as a query path factory, I don't think it's a
> perfect match as well, as callable interface is too general, at least to be
> an unnamed argument.
> For example, fields.CharField(default=callable) is a clear approach, as
> it's obvious what was implied.
> ModelClass.objects.all().order_by(callable) - still clear, but
> YearGte(callable, 40) isn't (since I'm talking about general interface to
> specify query paths in the whole project).
> So, I suggest a separate class for that purposes, and it doesn't
> necessarily need to have short name like "P". Even better if it has
> meaningful name like "TreePath" or "FieldTree".
>
>
> On Friday, August 21, 2015 at 4:26:34 AM UTC+3, Josh Smeaton wrote:
>
> Most expressions already support strings, and most that support strings
> expect them to be field_paths that can be wrapped in F() expressions. So
> you have two options really, without built in support from other
> expressions.
>
> 1. P.field.other.some_field() returns an F()
> 2. P.field.other.some_field() returns a str that will be internally
> wrapped in an F().
>
> The callable part of your syntax is just something I'm suggesting as an
> option. The requirement is that a string or F() be returned in some way
> though. I'm also using "P." as a stand-in as you used previously.
>
> Supporting callables is an interesting idea, and I don't think that'll
> cause issues with existing expressions. The callable must return an
> expression though. If you wanted to write a patch to support callables, I'd
> be onboard to review ASAP.
>
> Cheers
>
> On Friday, 21 August 2015 00:16:45 UTC+10, Alexey Zankevich wrote:
>
> What about the idea to add interface to specify paths with special class
> or callable?
>
>
> On Wednesday, August 19, 2015 at 10:49:07 AM UTC+3, Josh Smeaton wrote:
>
> If I finish the patch in time (I think I have about a month left), then
> it'll be included in 1.9. Review and comments on the PR will go a long way
> to helping me tidy it up sooner rather than later, so please feel free to
> review.
>
> Regards,
>
> On Wednesday, 19 August 2015 04:55:21 UTC+10, Alexey Zankevich wrote:
>
> Once Josh completes this patch https://github.com/django/django/pull/5090
> (.filter method accepting class-based lookups and transforms), it will be
> almost everything required by third-party apps. Is it going to be a part of
> Django 1.9, by the way?
>
> Additionally, for pure flexibility, next method and classes need to accept
> either a callable or an object supporting special path interface (for
> example,
> just a single method get_path() returning string path).
> They listed below:
>
> 1. The current methods.select_related, .prefetch_related, .order_by
> 2. Annotation classes Min, Max, Count, Sum
> 3. Future transforms and lookups classes
>
>
> Examples:
>
> >>> path = lambda x: 'user__last_login_date'
> >>> GameSession.objects.all().order_by(path)
>
> or
>
> >>> class LoginDatePath(object):
> ... def get_path(self):
> ... return 'user__last_login_date'
> >>> path = LoginDatePath()
> >>> GameSession.objects.all().order_by(path)
>
> Path generation is a critical part of external query syntax as it used in
> almost all aspects of the ORM, meanwhile each related method accepts
> either a
> string or (in some cases) a specific kind of class - OrderBy for order_by
> method, Prefetch for prefetch_related and etc.
>
>
> On Tuesday, August 18, 2015 at 7:54:48 PM UTC+3, Michael Manfre wrote:
>
> +1 for making it doable for 3rd party apps.
>
> Regards,
> Michael Manfre
>
> On Tue, Aug 18, 2015 at 12:49 PM, Anssi Kääriäinen <akaa...@gmail.com>
> wrote:
>
> I'm still thinking we shouldn't integrate any new query syntax into
> 1.9. Instead lets make it easy to create 3rd party apps that offer
> different querying syntax, and then lets see which solution (if any)
> gets popular enough to be integrated into Django.
>
> - Anssi
>
>
>
> On Tue, Aug 18, 2015 at 5:54 PM, Collin Anderson <cmawe...@gmail.com>
> wrote:
> > Just a quick thought: I could imagine some newbies could get confused by
> the
> > python-like syntax and try to do:
> >
> > Equal(P.user.get_full_name(), 'Bob Someone')
> >
> > I don't think it's not a huge deal, but worth noting.
> >
> > On Tuesday, August 18, 2015 at 8:00:17 AM UTC-4, Alexey Zankevich wrote:
> >>
> >> Hi all,
> >>
> >> Thanks for detailed response. I thought over the described expressions/
> >> transforms patches and here are my thoughts about the best way to
> >> implement simplified lookups.
> >>
> >> Firstly, I want to describe which properties of the new syntax seem to
> be
> >> important:
> >>
> >> 1. Using Python operators to do basic lookups as it's a natural way in
> >> Python
> >> for that.
> >>
> >> 2. Syntax should be minimalistic for basic lookups, the use of that
> >> approach
> >> will be more noticeable on multiple filter conditions. In other words,
> >> next
> >> syntax is better:
> >>
> >> >>> GameSession.objects.filter(Q.user.profile.last_login.date() ==
> >> >>> datetime.now().date)
> >>
> >> than this one
> >>
> >> >>> GameSession.objects.filter(E(F.user.profile.last_login).date() ==
> >> >>> datetime.now().date)
> >>
> >> as it requires additional calls, which makes expressions less readable.
> >>
> >> 3. I definitely like the idea of having explicit classes for lookups and
> >> transforms and think it's worth to bundle dotted query syntax with the
> >> patch.
> >> Explicit classes will separate functionality and simplify functions
> >> signature
> >> checks.
> >>
> >> I suggest a mixed approach, with next properties.
> >>
> >> 1. We will have a separate class to define query paths (let's call it P,
> >> we
> >> can still use F as "field", but having F as multipurpose class may
> confuse
> >> users, as well as implementation may become more complicated). And it
> will
> >> be
> >> the only purpose of the class. Below I'll reference it as "P" no matter
> we
> >> call
> >> it in future.
> >>
> >> 2. Any transforms and lookups will take query string or P class, as well
> >> as
> >> existing methods "select_related", "prefetch_related" and "order_by".
> The
> >> simplest implementation will be overriding __str__ method of the path
> >> class
> >> to generate related strings.
> >>
> >> >>> path = P.user.last_login_date
> >> >>> GameSession.objects.all().order_by(path)[:10]
> >>
> >> >>> print(path)
> >> user__last_login_date
> >>
> >> 3. Implement transforms and lookups as classes or functions (not bound
> to
> >> P class):
> >>
> >> >>> GameSession.objects.filter(Unaccent(P.user.location.name) == "Cote
> >> >>> d'Ivoire")
> >>
> >> It will simplify cases with parametrized transforms (ex. mentioned
> >> collate('fi')).
> >> Also eliminate fields collision with util functions like "date", which
> may
> >> be a
> >> so-called field.
> >>
> >> 4. Below is a table describing accepted passed and returned parameters:
> >>
> >> +-------------------+---------------------------+----------------------+
> >> | Class/Function | Allowed Param Types | Comparison Operators |
> >> +-------------------+---------------------------+----------------------+
> >> | Transform | str, P, Transform, Lookup | Return lookup |
> >> | Lookup | str, P, Transform | Raise exception |
> >> | P | | Return lookup |
> >> | .order_by | str, P | |
> >> | .select_related | str, P | |
> >> | .prefetch_related | str, P, Prefetch | |
> >> +-------------------+---------------------------+----------------------+
> >>
> >>
> >> Samples:
> >>
> >> >>> P.user.name == 'Bob'
> >> Equal('user__name', 'Bob')
> >>
> >> >>> Unaccent(P.user.name)
> >> Unaccent('user__name')
> >>
> >> >>> Collate(P.user.name, 'fi')
> >> Collate('user__name', 'fi')
> >>
> >> >>> Unaccent(P.user.name) == 'Bob'
> >> Equal(Unaccent('user__name'), 'Bob')
> >>
> >> >>> Equal(P.user.name, 'Bob') == 'Bob'
> >> Traceback (most recent call last):
> >> File "<stdin>", line 1, in <module>
> >> Exception: Lookups comparison is not allowed
> >>
> >> >>> Contains(P.user.name, 'Bo') # lookup
> >> >>> Date(P.user.last_login_date, datetime.now().date) # transform
> >>
> >>
> >> Questions to discuss and possible pitfalls:
> >>
> >> 1. Handling descended ordering in order_by
> >>
> >> >>> path = P.user.last_login_date
> >> >>> -path
> >> '-user__last_login_date'
> >>
> >> or even
> >>
> >> >>> -p
> >> NegP('user__last_login_date')
> >>
> >> Is it possible path subtraction is required in future by any reason? In
> >> that
> >> case the given approach will be inconvenient. Or may be better to just
> let
> >> users to format desc ordering themselves?
> >>
> >> >>> '-{}'.format(path)
> >>
> >> 2. Don't think it's a big deal, but users might do next thing:
> >>
> >> >>> path = P.user.last_login_date
> >> >>> GameSession.objects.filter(path=today)
> >> Traceback (most recent call last):
> >> File "<console>", line 1, in <module>
> >> FieldError: Cannot resolve keyword 'path' into field.
> >>
> >> At least need to mention it in documentation
> >>
> >> 3. Classes vs factory functions:
> >>
> >> >>> Equal(P.user.name, 'Bob')
> >> Equal('user__name', 'Bob')
> >>
> >> vs
> >>
> >> >>> equal(P.user.name, 'Bob')
> >> Equal('user__name', 'Bob')
> >>
> >>
> >> If you are fine with described syntax I can start with DEP.
> >>
> >> Regards,
> >> Alexey
> >>
> >> On Monday, August 17, 2015 at 12:26:45 PM UTC+3, Anssi Kääriäinen wrote:
> >>>
> >>> I like this idea, too.
> >>>
> >>> The work I and Loic have been doing has been about transforms that can
> >>> accept arguments. I don't see why the two styles couldn't be mixed
> >>> directly.
> >>>
> >>> .filter(Q.user.profile.last_login.date(timezone='Europe/Helsinki') <
> >>> '2015-01-01')
> >>> .filter(Q.user.last_name.collate('fi').unaccent() == 'Kaariainen')
> >>>
> >>> OTOH if you want to use expressions directly, that should be OK, too
> >>> (if you don't mind the non-optimal readability):
> >>> .filter(Exact(unaccent(collate(Q.user.last_name, 'fi'))),
> 'Kaariainen'))
> >>>
> >>> All in all, +1 to working towards these goals. I'm not yet convinced
> >>> we need all this in Django core directly. Maybe we should first try
> >>> these ideas in 3rd party apps, and then pick a proven solution into
> >>> Django core?
> >>>
> >>> - Anssi
> >>>
> >>> On Mon, Aug 17, 2015 at 3:33 AM, Josh Smeaton <josh.s...@gmail.com>
> >>> wrote:
> >>> > Hi Alexey,
> >>> >
> >>> > I find this approach really interesting, and I don't mind the API
> >>> > you've
> >>> > created at all. Even if this doesn't make it into Django, I think
> >>> > there's
> >>> > enough utility in your library that other people would want to use
> it.
> >>> > I'm
> >>> > not going to comment on specifics just yet like method names,
> because I
> >>> > figure that if we can agree on an approach, the bike-shedding can
> come
> >>> > later.
> >>> >
> >>> > So first things first. I think we can all, or at least mostly, agree
> >>> > that
> >>> > having an alternative syntax for building filters is a good idea.
> >>> > String
> >>> > based lookups are cool, and won't go away, but sometimes you want to
> >>> > have
> >>> > greater control over what kind of filter you want to build without
> >>> > having to
> >>> > register global lookups and transforms. I see similarities with
> >>> > prefetch_related. You can provide a string based field lookup, or
> use a
> >>> > Prefetch() object to give users more control over the behaviour of
> the
> >>> > prefetch.
> >>> >
> >>> > I've been working on a patch
> >>> > (https://github.com/django/django/pull/5090) to
> >>> > unify Transforms and Expressions (Func specifically). The work behind
> >>> > that
> >>> > can be (and should be) extended to include Lookups -- making them
> fully
> >>> > fledged expression objects. That work will need to happen regardless.
> >>> > Once
> >>> > this patch is finished and lands, then we should be able to extend
> >>> > .filter()
> >>> > and .exclude() to allow expressions to be composed as filters:
> >>> >
> >>> >
> >>> >
> GameSession.objects.filter(GreaterThan('user__profile__last_login_date',
> >>> > yesterday))
> >>> > # or, if last_login was a datetime, and we wanted to compare the date
> >>> > part
> >>> > only
> >>> > GameSession.objects.filter(Equal(Date('user__profile__last_login'),
> >>> > datetime.now().date))
> >>> > # or if we wanted to implement __gt__ and __eq__ etc:
> >>> > GameSession.objects.filter(Date('user__profile__last_login') ==
> >>> > datetime.now().date))
> >>> >
> >>> > Loic and Anssi have also been working on alternative syntaxes
> >>> > (https://github.com/django/django/pull/4953):
> >>> >
> >>> > GameSession.objects.filter(E('user__profile__last_login').date() ==
> >>> > datetime.now().date)
> >>> >
> >>> >
> >>> > Article.objects.filter(E('headline').collate('fi') == 'Article1')
> >>> >
> >>> > Both of these approaches still suffer from "the string problem" that
> >>> > you're
> >>> > trying to address, but minimises the final component by extracting
> the
> >>> > lookup and transform components as objects instead. So I think your
> >>> > idea
> >>> > here could nicely coexist:
> >>> >
> >>> > GameSession.objects.filter(Equal(Date(Q.user.profile.last_login),
> >>> > datetime.now().date))
> >>> > GameSession.objects.filter(E(Q.user.profile.last_login).date() ==
> >>> > datetime.now().date)
> >>> >
> >>> > Or, even building this into F expressions rather than Q expressions:
> >>> >
> >>> > GameSession.objects.filter(Equal(Date(F.user.profile.last_login),
> >>> > datetime.now().date))
> >>> > GameSession.objects.filter(E(F.user.profile.last_login).date() ==
> >>> > datetime.now().date)
> >>> >
> >>> > I think it may look better with F objects, considering they are Field
> >>> > references, and since the Lookups (Equal/GTE) parts accept F
> >>> > expressions
> >>> > anyway. I'm not too concerned about this particular detail though.
> >>> >
> >>> > A DEP is probably the right way to go here but I wonder if we should
> >>> > expand
> >>> > the scope to include alternative filter syntax as a whole
> >>> > (expressions/callable transforms) as well as the dot field reference
> >>> > notation you've proposed above. Then we can consider how all these
> >>> > things
> >>> > might work together, and clearly document why we've gone one way and
> >>> > not
> >>> > another. Obviously, alternatives can exist outside of core where the
> >>> > API is
> >>> > available.
> >>> >
> >>> > I'll be happy to work as the shepherd if needed. But I'd also like
> some
> >>> > input from Loic and Anssi especially, as well as others in the core
> >>> > team
> >>> > with some interest in the ORM.
> >>> >
> >>> > Regards,
> >>> >
> >>> >
> >>> > On Sunday, 16 August 2015 23:18:26 UTC+10, Alexey Zankevich wrote:
> >>> >>
> >>> >> Hi all,
> >>> >>
> >>> >> This topic is related to the current ORM query syntax with
> >>> >> underscores.
> >>> >> There are lots of arguing related to it, anyway it has pros and
> cons.
> >>> >>
> >>> >> Let's take a concrete example of querying a model:
> >>> >>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)
> >>> >>
> >>> >>
> >>> >> Pros:
> >>> >>
> >>> >> 1. The syntax is easy to understand
> >>> >> 2. Can be extended with custom transforms and lookups
> >>> >>
> >>> >> However, there are several cons:
> >>> >>
> >>> >> 1. Long strings is hard to read, especially if we have fields with
> >>> >> underscores.
> >>> >> It's really easy to make a mistake by missing one:
> >>> >>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> GameSession.objects.filter(user_profile__last_login_date__gte=yesterday)
> >>> >>
> >>> >> Not easy to catch missing underscore between user and profile, is
> it?
> >>> >> Even
> >>> >> though, it's not easy to say whether it should be "user_profile"
> >>> >> attribute
> >>> >> or
> >>> >> user.profile foreign key.
> >>> >>
> >>> >> 2. Query strings can't be reused, thus the approach violates DRY
> >>> >> principle.
> >>> >> For example, we need to order results by last_login_date:
> >>> >>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday) \
> >>> >> .order_by('user__profile__last_login_date')
> >>> >>
> >>> >> We can't keep user__profile_login_date as a variable as in the first
> >>> >> part
> >>> >> of the
> >>> >> expression we use a keyword argument, meanwhile in the second part -
> >>> >> just
> >>> >> a
> >>> >> string. And thus we just have to type query path twice.
> >>> >>
> >>> >> 3. Lookup names not natural to Python language and require to be
> >>> >> remembered or
> >>> >> looked up in documentation. For example, "__gte" or "__lte" lookups
> >>> >> tend
> >>> >> to be
> >>> >> confused with "ge" and "le" due to similarity to methods "__ge__"
> and
> >>> >> "__le__".
> >>> >>
> >>> >> 4. Lookup keywords limited to a single argument only, very
> >>> >> inconvenient
> >>> >> when
> >>> >> necessary to filter objects by range.
> >>> >>
> >>> >> I was thinking a lot trying to solve those issues, keeping in mind
> >>> >> Django
> >>> >> approaches. Finally I came up with solution to extend Q objects with
> >>> >> dot
> >>> >> expression syntax:
> >>> >>
> >>> >> >>> GameSession.objecs.filter(Q.user.profile.last_login_date >=
> >>> >> >>> yesterday)
> >>> >>
> >>> >> Q is a factory instance for old-style Q objects. Accessing attribute
> >>> >> by
> >>> >> dot
> >>> >> returns a child factory, calling factory will instantiate old-style
> Q
> >>> >> object.
> >>> >>
> >>> >> >>> Q
> >>> >> <QFactory object at 0x7f407298ee10>
> >>> >>
> >>> >> >>> Q.user.profile
> >>> >> <QFactory object at 0x7f40765da310>
> >>> >>
> >>> >> >>> Q(user__name='Bob')
> >>> >> <Q: (AND: ('user__name', 'Bob'))>
> >>> >>
> >>> >> It overrides operators, so comparing factory with value returns a
> >>> >> related
> >>> >> Q
> >>> >> object:
> >>> >>
> >>> >> >>> Q.user.name == 'Bob'
> >>> >> <Q: (AND: ('user__name', 'Bob'))>
> >>> >>
> >>> >> Factory has several helper functions for lookups which aren't
> related
> >>> >> to
> >>> >> any
> >>> >> Python operators directly:
> >>> >>
> >>> >> >>> Q.user.name.icontains('Bob')
> >>> >> <Q: (AND: ('user__name__icontains', 'Bob'))>
> >>> >>
> >>> >> And helper to get query path as string, which requred by order_by or
> >>> >> select_related queryset methods:
> >>> >>
> >>> >> >>> Q.user.profile.last_login_date.get_path()
> >>> >> 'user__profile__last_login_date'
> >>> >>
> >>> >> You can check implementation and more examples here
> >>> >> https://github.com/Nepherhotep/django-orm-sugar
> >>> >>
> >>> >> How it solves issues:
> >>> >>
> >>> >> #1. Dots hard to confuse with underscores
> >>> >> #2. Query paths can be reused:
> >>> >>
> >>> >> >>> factory = Q.user.profile.last_login_date
> >>> >> >>> query = GameSession.objects.filter(factory >= yesterday)
> >>> >> >>> query = query.order_by(factory.get_path())
> >>> >>
> >>> >> #3. Not neccessary to remember most of lookup names and use
> comparison
> >>> >> operators
> >>> >> instead.
> >>> >> #4. Possible to use multiple keyword arguments:
> >>> >>
> >>> >> >>> Q.user.profile.last_login_date.in_range(from_date, to_date)
> >>> >> <Q: (AND: ('user__profile__last_login_date__lte', from_date),
> >>> >> ('user__profile__last_login_date__gte', to_date))>
> >>> >>
> >>> >>
> >>> >> This approach looked the best for me due to several reasons:
> >>> >>
> >>> >> 1. It's explicit - it doesn't do anything but generating
> appropriate Q
> >>> >> object.
> >>> >> The result of comparison can be saved as Q object variable.
> >>> >>
> >>> >> 2. It's short - variants with using model for that will look much
> >>> >> longer,
> >>> >> when
> >>> >> joining two or more filters:
>
> ...
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/9be6e6f3-11f5-441f-a7c2-0e4d6d4965e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
