+1 to not requiring all transforms to handle __underscore__ syntax.
I think what we want to do is allow users choose which syntax they
prefer. The idea is that Django will support both JSONExtract('data',
path=['owner', 'other_pets', 0, 'name']) and
data__owner__other_pets__0__name out of the box.
We will also make it possible to support callable transforms. For example:
filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')))
is equivalent to
filter(F('name').decode('utf-8') == Value(u'Бармаглот'))
The callable transforms syntax will not be a part of Django, but it
will be possible to create an extension for this (it is actually
surprisingly easy to do once we have support for expressions in
filter).
- Anssi
On Thu, Oct 1, 2015 at 4:00 AM, Josh Smeaton <josh.smea...@gmail.com> wrote:
> No, not all Lookups or Transforms are required to handle __underscore__
> syntax. The entire point of supporting object based lookups is to handle
> cases that get more complex than a single argument transform or a left and
> right hand side lookup.
>
> In particular, I think your Decode(utf8) example is a good one. It shows
> that you could maybe shoehorn multiple arg transforms into the
> __underscore__ api, but it wouldn't be entirely obvious how to do so. You'd
> probably need to register partial transformations for each encoding you
> wanted to support. The contrib.postgres module has (from memory)
> KeyTransform classes that do a similar thing.
>
> Cheers
>
>
> On Thursday, 1 October 2015 02:13:41 UTC+10, Alexey Zankevich wrote:
>>
>> I'll try to turn lookups into expression (will use master branch).
>> I also have a question. Meanwhile the old query syntax with underscores is
>> pretty good for simple queries (like Model.objects.filter(name='Bob'), it
>> gets really ugly for parametrized calls, a new JSONField is a good example
>> of that:
>>
>> >>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy')
>> [<Dog: Rufus>]
>>
>> It will get even more messy if we want to pass a string as a second param
>> of the func.
>>
>> ex.:
>>
>> 1. filter(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')) # <- neat
>> 2. filter(name__decode__utf8=u'Бармаглот') # <- ?? ambiguous and not nice
>> at all
>>
>> So question - is it implied all the funcs, transforms and lookups to have
>> underscore-based equivalent? It can affect the final implementation pretty
>> much. In my opinion underscore-based equivalent should not be really
>> required for funcs (the problem doesn't seem to affect transforms as they
>> will not accept multiple params according to another thread).
>>
>> Thanks,
>> Alexey
>>
>>
>> On Wednesday, September 30, 2015 at 9:19:51 AM UTC+3, Anssi Kääriäinen
>> wrote:
>>>
>>> I don't think we need split-to-subq support for Lookups before we make
>>> them expressions. Lookups as expressions are usable outside .filter(),
>>> and we need the split-to-subq support only in .filter(expression).
>>>
>>> - Anssi
>>>
>>> On Wed, Sep 30, 2015 at 8:46 AM, Josh Smeaton <josh.s...@gmail.com>
>>> wrote:
>>> > I'm mixing my versions, sorry to those following along. 1.9 has just
>>> > reached
>>> > alpha. Lookups as Expressions should be doable for 1.10 which master is
>>> > currently tracking.
>>> >
>>> > Cheers
>>> >
>>> >
>>> > On Wednesday, 30 September 2015 15:31:24 UTC+10, Josh Smeaton wrote:
>>> >>
>>> >> The alpha for 1.10 has already been cut, and I'm not sure that the
>>> >> kinds
>>> >> of changes needed here are appropriate to add now that the alpha is
>>> >> out. One
>>> >> could *maybe* make the argument that changing Lookup to an Expression
>>> >> now
>>> >> rather than later is the right move considering Transforms just
>>> >> underwent
>>> >> the same change for 1.10. Personally though, I don't think I have the
>>> >> time
>>> >> right now to do this change. I would support you if you were able, but
>>> >> we'd
>>> >> still be at the mercy of the technical board (I assume) for getting
>>> >> this
>>> >> change in for 1.10.
>>> >>
>>> >> Do you think Lookup as Expressions requires the subquery/exclude fix
>>> >> you
>>> >> mention above? I would think not -- not until we were ready to
>>> >> document and
>>> >> support .filter(Lookup(F(), Value()). If it wasn't a requirement, it'd
>>> >> make
>>> >> the Lookup->Expression work much easier. It wouldn't even need to be
>>> >> documented (other than the release notes), as it'd just be an
>>> >> implementation
>>> >> change.
>>> >>
>>> >> Cheers,
>>> >>
>>> >> On Wednesday, 30 September 2015 14:33:14 UTC+10, Anssi Kääriäinen
>>> >> wrote:
>>> >>>
>>> >>> On the core ORM side we need to make
>>> >>> .exclude(LessThan(F('friends__age'), 30)) do a subquery. This way
>>> >>> .exclude(friends__age__lt=30) does the same thing as the expression
>>> >>> version. This isn't that easy to do. If we just use
>>> >>> resolve_expression, then the friends relation will generate a join,
>>> >>> and then as second step do a negated filter on the joined value.
>>> >>> Instead we want to detect that the LessThan expression needs to be
>>> >>> pushed in to a subquery.
>>> >>>
>>> >>> So, we need to solve:
>>> >>>
>>> >>> A) A way to ask an expression if it is referencing a multijoin
>>> >>> (possible approach is to just have a method
>>> >>> "refs_multi_valued_relation(query)")
>>> >>> B) When the ORM sees an expression that is reffing a multijoin in an
>>> >>> exclude filter, then we need to push the expression in to a subquery.
>>> >>>
>>> >>> A) requires some new work. This shouldn't be that hard to implement,
>>> >>> we just recursively ask subexpressions if they reference a multijoin.
>>> >>>
>>> >>> Something like https://github.com/django/django/pull/4385 will make
>>> >>> B)
>>> >>> much easier to implement.
>>> >>>
>>> >>> I've been working on making Q-objects responsible for resolving
>>> >>> themselves. See https://github.com/django/django/pull/4801. This
>>> >>> should solve 3).
>>> >>>
>>> >>> We don't seem to be missing any major parts. I (or some volunteer)
>>> >>> just need to finish the PRs, and then we should be really close to
>>> >>> full support for expressions in filter.
>>> >>>
>>> >>> Josh: do you think we could get Lookup as expressions in to 1.10
>>> >>> instead of 1.11?
>>> >>>
>>> >>> - Anssi
>>> >>>
>>> >>> On Wed, Sep 30, 2015 at 3:46 AM, Josh Smeaton <josh.s...@gmail.com>
>>> >>> wrote:
>>> >>> > 1. Lookups should become Expressions, just as Transforms have
>>> >>> > become
>>> >>> > Expressions. This will let us process Lookup arguments as
>>> >>> > Expressions
>>> >>> > all
>>> >>> > the way the way through. I think this should be a major goal for
>>> >>> > version
>>> >>> > 1.11.
>>> >>> >
>>> >>> > 2. Chaining transforms is now possible since they are just Func
>>> >>> > expressions.
>>> >>> > Func(Func(Func('field_name'))) is no issue.
>>> >>> >
>>> >>> > 3. Sounds like an OK idea, but I haven't looked into the details
>>> >>> > enough
>>> >>> > to
>>> >>> > really comment. I do think we should create the correct form as
>>> >>> > early
>>> >>> > as
>>> >>> > possible (parsing into a chain of Lookup/Transform expressions) so
>>> >>> > we
>>> >>> > don't
>>> >>> > have to do parsing in multiple places. The entry points to
>>> >>> > .filter()
>>> >>> > and
>>> >>> > .exclude(), or their direct counterparts in sql.query sound ideal.
>>> >>> > Anssi has
>>> >>> > mentioned elsewhere that WhereNode's should only contain fully
>>> >>> > resolved
>>> >>> > expressions, so resolving will need to be done directly after
>>> >>> > parsing
>>> >>> > (or
>>> >>> > during).
>>> >>> >
>>> >>> > Part 1 above can be started now if you have the time or interest.
>>> >>> > We
>>> >>> > can
>>> >>> > nail down the particulars of part 3 while we're solving part 1.
>>> >>> > Part 1
>>> >>> > may
>>> >>> > drive some of part 3.
>>> >>> >
>>> >>> > Cheers
>>> >>> >
>>> >>> >
>>> >>> > On Wednesday, 30 September 2015 04:49:54 UTC+10, Alexey Zankevich
>>> >>> > wrote:
>>> >>> >>
>>> >>> >> Here is a list of issues to solve to support explicit transforms
>>> >>> >> and
>>> >>> >> lookups by filter (and exclude) methods.
>>> >>> >>
>>> >>> >> 1. Make Lookup.__init__ signature to support initialization with F
>>> >>> >> objects
>>> >>> >> or string path (e.g. GreaterThan(F('user__id'), 10) or
>>> >>> >> GreaterThan('user__id', 10)), not sure it's possible to use
>>> >>> >> simultaneously
>>> >>> >> with the current approach with lhs, rhs initialization (even with
>>> >>> >> moving it
>>> >>> >> to a separate class method, e.g Lookup.build(lhs, rhs)), so I
>>> >>> >> assume
>>> >>> >> creating so-called util classes which will delegate SQL-related
>>> >>> >> functionality to existing Lookup classes.
>>> >>> >>
>>> >>> >> 2. Chain transforms by passing them as argument:
>>> >>> >>
>>> >>> >> Lower(Unaccent(F('user__name)))
>>> >>> >>
>>> >>> >> 3. Decide if Q objects shall support explicit lookups/transforms
>>> >>> >> as
>>> >>> >> argument as well - it's a kind of logical step, as without Q
>>> >>> >> objects
>>> >>> >> it will
>>> >>> >> not be possible to perform complicated conditions (AND, OR, NOT).
>>> >>> >> In that case lookup/transform parsing should be moved from
>>> >>> >> QuerySet
>>> >>> >> object
>>> >>> >> to Q object - filter will take already parsed lookup tree.
>>> >>> >> Example:
>>> >>> >>
>>> >>> >> Q(user__name__lower__unaccent__icontains='Bob') will internally
>>> >>> >> parse
>>> >>> >> it
>>> >>> >> and build next structure:
>>> >>> >>
>>> >>> >> Q(Icontains(Lower(Unaccent(F('user__name')))), 'Bob')
>>> >>> >>
>>> >>> >>
>>> >>> >> On Sunday, August 16, 2015 at 4:18:26 PM UTC+3, Alexey Zankevich
>>> >>> >> wrote:
>>> >>> >>>
>>> >>> >>> Hi all,
>>> >>> >>>
>>> >>> >>> This topic is related to the current ORM query syntax with
>>> >>> >>> underscores.
>>> >>> >>> There are lots of arguing related to it, anyway it has pros and
>>> >>> >>> cons.
>>> >>> >>>
>>> >>> >>> Let's take a concrete example of querying a model:
>>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> Pros:
>>> >>> >>>
>>> >>> >>> 1. The syntax is easy to understand
>>> >>> >>> 2. Can be extended with custom transforms and lookups
>>> >>> >>>
>>> >>> >>> However, there are several cons:
>>> >>> >>>
>>> >>> >>> 1. Long strings is hard to read, especially if we have fields
>>> >>> >>> with
>>> >>> >>> underscores.
>>> >>> >>> It's really easy to make a mistake by missing one:
>>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>> GameSession.objects.filter(user_profile__last_login_date__gte=yesterday)
>>> >>> >>>
>>> >>> >>> Not easy to catch missing underscore between user and profile, is
>>> >>> >>> it?
>>> >>> >>> Even
>>> >>> >>> though, it's not easy to say whether it should be "user_profile"
>>> >>> >>> attribute or
>>> >>> >>> user.profile foreign key.
>>> >>> >>>
>>> >>> >>> 2. Query strings can't be reused, thus the approach violates DRY
>>> >>> >>> principle.
>>> >>> >>> For example, we need to order results by last_login_date:
>>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)
>>> >>> >>> >>> \
>>> >>> >>> .order_by('user__profile__last_login_date')
>>> >>> >>>
>>> >>> >>> We can't keep user__profile_login_date as a variable as in the
>>> >>> >>> first
>>> >>> >>> part
>>> >>> >>> of the
>>> >>> >>> expression we use a keyword argument, meanwhile in the second
>>> >>> >>> part -
>>> >>> >>> just
>>> >>> >>> a
>>> >>> >>> string. And thus we just have to type query path twice.
>>> >>> >>>
>>> >>> >>> 3. Lookup names not natural to Python language and require to be
>>> >>> >>> remembered or
>>> >>> >>> looked up in documentation. For example, "__gte" or "__lte"
>>> >>> >>> lookups
>>> >>> >>> tend
>>> >>> >>> to be
>>> >>> >>> confused with "ge" and "le" due to similarity to methods "__ge__"
>>> >>> >>> and
>>> >>> >>> "__le__".
>>> >>> >>>
>>> >>> >>> 4. Lookup keywords limited to a single argument only, very
>>> >>> >>> inconvenient
>>> >>> >>> when
>>> >>> >>> necessary to filter objects by range.
>>> >>> >>>
>>> >>> >>> I was thinking a lot trying to solve those issues, keeping in
>>> >>> >>> mind
>>> >>> >>> Django
>>> >>> >>> approaches. Finally I came up with solution to extend Q objects
>>> >>> >>> with
>>> >>> >>> dot
>>> >>> >>> expression syntax:
>>> >>> >>>
>>> >>> >>> >>> GameSession.objecs.filter(Q.user.profile.last_login_date >=
>>> >>> >>> >>> yesterday)
>>> >>> >>>
>>> >>> >>> Q is a factory instance for old-style Q objects. Accessing
>>> >>> >>> attribute
>>> >>> >>> by
>>> >>> >>> dot
>>> >>> >>> returns a child factory, calling factory will instantiate
>>> >>> >>> old-style Q
>>> >>> >>> object.
>>> >>> >>>
>>> >>> >>> >>> Q
>>> >>> >>> <QFactory object at 0x7f407298ee10>
>>> >>> >>>
>>> >>> >>> >>> Q.user.profile
>>> >>> >>> <QFactory object at 0x7f40765da310>
>>> >>> >>>
>>> >>> >>> >>> Q(user__name='Bob')
>>> >>> >>> <Q: (AND: ('user__name', 'Bob'))>
>>> >>> >>>
>>> >>> >>> It overrides operators, so comparing factory with value returns a
>>> >>> >>> related
>>> >>> >>> Q
>>> >>> >>> object:
>>> >>> >>>
>>> >>> >>> >>> Q.user.name == 'Bob'
>>> >>> >>> <Q: (AND: ('user__name', 'Bob'))>
>>> >>> >>>
>>> >>> >>> Factory has several helper functions for lookups which aren't
>>> >>> >>> related
>>> >>> >>> to
>>> >>> >>> any
>>> >>> >>> Python operators directly:
>>> >>> >>>
>>> >>> >>> >>> Q.user.name.icontains('Bob')
>>> >>> >>> <Q: (AND: ('user__name__icontains', 'Bob'))>
>>> >>> >>>
>>> >>> >>> And helper to get query path as string, which requred by order_by
>>> >>> >>> or
>>> >>> >>> select_related queryset methods:
>>> >>> >>>
>>> >>> >>> >>> Q.user.profile.last_login_date.get_path()
>>> >>> >>> 'user__profile__last_login_date'
>>> >>> >>>
>>> >>> >>> You can check implementation and more examples here
>>> >>> >>> https://github.com/Nepherhotep/django-orm-sugar
>>> >>> >>>
>>> >>> >>> How it solves issues:
>>> >>> >>>
>>> >>> >>> #1. Dots hard to confuse with underscores
>>> >>> >>> #2. Query paths can be reused:
>>> >>> >>>
>>> >>> >>> >>> factory = Q.user.profile.last_login_date
>>> >>> >>> >>> query = GameSession.objects.filter(factory >= yesterday)
>>> >>> >>> >>> query = query.order_by(factory.get_path())
>>> >>> >>>
>>> >>> >>> #3. Not neccessary to remember most of lookup names and use
>>> >>> >>> comparison
>>> >>> >>> operators
>>> >>> >>> instead.
>>> >>> >>> #4. Possible to use multiple keyword arguments:
>>> >>> >>>
>>> >>> >>> >>> Q.user.profile.last_login_date.in_range(from_date, to_date)
>>> >>> >>> <Q: (AND: ('user__profile__last_login_date__lte', from_date),
>>> >>> >>> ('user__profile__last_login_date__gte', to_date))>
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> This approach looked the best for me due to several reasons:
>>> >>> >>>
>>> >>> >>> 1. It's explicit - it doesn't do anything but generating
>>> >>> >>> appropriate
>>> >>> >>> Q
>>> >>> >>> object.
>>> >>> >>> The result of comparison can be saved as Q object variable.
>>> >>> >>>
>>> >>> >>> 2. It's short - variants with using model for that will look much
>>> >>> >>> longer,
>>> >>> >>> when
>>> >>> >>> joining two or more filters:
>>> >>> >>>
>>> >>> >>> >>> GameSession.objects.user.profile_last_login_date >= yesterday
>>> >>> >>> >>> #
>>> >>> >>> >>> awkward
>>> >>> >>>
>>> >>> >>> 3. Implementation will not require to change querset manager or
>>> >>> >>> model
>>> >>> >>> classes
>>> >>> >>>
>>> >>> >>> 4. Will still allow to use filters and Q class in the old way:
>>> >>> >>>
>>> >>> >>> >>> q = Q(user__profile__last_login_date__gte=yesterday)
>>> >>> >>>
>>> >>> >>> or
>>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)
>>> >>> >>>
>>> >>> >>> I'd like to make it as a part of Django ORM syntax and it will
>>> >>> >>> not be
>>> >>> >>> hard to
>>> >>> >>> do, especially taking into account the library is already done
>>> >>> >>> and
>>> >>> >>> working.
>>> >>> >>> Anyway, I need your thought about the idea in general, as well as
>>> >>> >>> about
>>> >>> >>> particular things like chosen method names - "get_path",
>>> >>> >>> "in_range"
>>> >>> >>> and
>>> >>> >>> etc.
>>> >>> >>> As next step I can create a ticket in the issue tracker, or
>>> >>> >>> prepare
>>> >>> >>> DEP
>>> >>> >>> first.
>>> >>> >>> In latter case I need to find a shepherd to work with.
>>> >>> >>>
>>> >>> >>> Best regards,
>>> >>> >>> Alexey
>>> >>> >
>>> >>> > --
>>> >>> > You received this message because you are subscribed to the Google
>>> >>> > Groups
>>> >>> > "Django developers (Contributions to Django itself)" group.
>>> >>> > To unsubscribe from this group and stop receiving emails from it,
>>> >>> > send
>>> >>> > an
>>> >>> > email to django-develop...@googlegroups.com.
>>> >>> > To post to this group, send email to django-d...@googlegroups.com.
>>> >>> > Visit this group at
>>> >>> > http://groups.google.com/group/django-developers.
>>> >>> > To view this discussion on the web visit
>>> >>> >
>>> >>> >
>>> >>> > https://groups.google.com/d/msgid/django-developers/3bad8371-f9b4-47ff-a681-0108b320e9b5%40googlegroups.com.
>>> >>> >
>>> >>> > For more options, visit https://groups.google.com/d/optout.
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "Django developers (Contributions to Django itself)" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an
>>> > email to django-develop...@googlegroups.com.
>>> > To post to this group, send email to django-d...@googlegroups.com.
>>> > Visit this group at http://groups.google.com/group/django-developers.
>>> > To view this discussion on the web visit
>>> >
>>> > https://groups.google.com/d/msgid/django-developers/5e7ee87b-6253-420b-9688-51c39d657cab%40googlegroups.com.
>>> >
>>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/b97ea866-514c-4d03-bf60-e46a06e5fc5a%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CALMtK1H8OGkXJys4Z-XiZ9kFspZ_naqNS%2BbOigsHCK40Snhj1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
