Re: Proposal: relationships based on arbitrary predicates
Hello Alex, While investigating an issue with reverse known related objects assignment[0] I stumbled upon a little something that might interest you. It looks like there's currently some tested cases of abusing the ForeignObject interface to achieve something similar to what you are suggesting here[1][2]. The contenttypes.GenericRelation object happens to implement a similar interface while it's actually a many-to-many field. I assume it's not setting many_to_many=True because a ton of internal Django checks duck-type this attribute to figure whether or not "through" exists but it should really be marked this way. I'd say giving a shot at reimplementing GenericRelation on top of your suggested Relationship object would be a good way to test whether or not it's flexible enough and be a strong argument for inclusion in Django core to get rid of the current hacks GenericRelation currently employs. Simon [0] https://code.djangoproject.com/ticket/29908 [1] https://github.com/django/django/blob/ecac6d7a2a510cb0a5d772deeca633c99c9687e5/tests/foreign_object/models/empty_join.py#L22-L84 [2] https://github.com/django/django/blob/ecac6d7a2a510cb0a5d772deeca633c99c9687e5/tests/foreign_object/models/empty_join.py#L98-L103 Le mardi 18 septembre 2018 21:43:56 UTC-4, Alex Hill a écrit : > > Hi Silvio, > > Thanks for your feedback. David Sanders brought up something similar to > ComputedField on GitHub[0] - a ForwardedField which can traverse > relationships and present a field from a remote instance as if it were a > local field. As long as it can traverse relationships, that use case would > be covered by ComputedField, so that's another tick for that idea. > > I think these two features are quite distinct - except that they may both > require a similar change to the ORM to allow traversing relations in the > referred-to fields. One limitation of relativity now is that you can only > form conditions with local fields of your models. I'd like to be able to > traverse relationships in those conditions. > > Anybody who's interested - please try out relativity and see if it works > for you. The mptt and treebeard helpers are a good place to start :) > > Simon - any further thoughts on this before I start working up a patch? > > Thanks, > Alex > > [0] > https://github.com/AlexHill/django-relativity/pull/1#issuecomment-418239406 > > On Mon, 10 Sep 2018 at 4:59 am, Silvio > > wrote: > >> Alex, >> >> This is a very useful pattern, that like many others, I've also >> implemented in an ad-hoc fashion using a ton of undocumented internal APIs. >> So I fully agree standardizing it would be great. Something similar is: >> >> https://groups.google.com/forum/#!topic/django-developers/ADSuUUuZp3Q >> >> Essentially, I've ended up with the need for: >> >> ComputedField >> >> and >> >> ComputedRelationship >> >> where both have all of the niceties that regular fields and foreign >> relationships have. >> >> So I'd love to see this in Django. >> >> - >> Silvio >> >> >> On Sunday, September 2, 2018 at 10:55:58 PM UTC-4, Alex Hill wrote: >>> >>> Hi Simon, >>> >>> Thanks for looking at this and for providing some context - I had looked >>> at FilteredRelation but I hadn't seen reverse-unique. It makes me more >>> confident that this is a good direction to take. I've reimplemented >>> ReverseUnique using Relationship [0] and the tests pass, with the only code >>> carried over that for discovery of the FK link. >>> >>> > I'm not totally sold on the API but having an analog of what >>> ForeignObject is to ForeignKey for ManyToManyField would definitely be >>> useful. >>> >>> I'm not tied to the API, but I think passing a Q as a predicate makes >>> sense especially given that it's what both FilteredRelation and >>> ReverseUnique do. The core of the idea is that we can express a >>> relationship as a combination of predicate and arity. In practise I don't >>> think this would be used all that much by users directly - more by >>> third-party apps like mptt, and perhaps Django internally. >>> >>> > From what I can see in relativity.fields[0] most of the additional >>> logic revolves around the extra filtering capabilites through Restriction. >>> >>> Yeah that's what it boils down to. We return no columns to join against, >>> and return a compilable Restriction from get_extra_restriction to provide >>> all the ON conditions. The rest of it is making the descriptors, rels, >>> prefetch, etc work. >>> >>> > Do you have an idea of what the fields.related inheritance chain would >>> look like if it was part of core? >>> >>> The least intrusive, and probably a good starting point, would be to >>> introduce Relationship alongside the other relation fields as a standalone >>> feature, modifying the ORM to allow the implementation to be less hacky. It >>> would remain a subclass of ForeignObject (or perhaps RelatedField - I'll >>> give that a try). >>> >>> In the future there's potential for a nice refactor o
Re: Proposal: relationships based on arbitrary predicates
Hi Silvio, Thanks for your feedback. David Sanders brought up something similar to ComputedField on GitHub[0] - a ForwardedField which can traverse relationships and present a field from a remote instance as if it were a local field. As long as it can traverse relationships, that use case would be covered by ComputedField, so that's another tick for that idea. I think these two features are quite distinct - except that they may both require a similar change to the ORM to allow traversing relations in the referred-to fields. One limitation of relativity now is that you can only form conditions with local fields of your models. I'd like to be able to traverse relationships in those conditions. Anybody who's interested - please try out relativity and see if it works for you. The mptt and treebeard helpers are a good place to start :) Simon - any further thoughts on this before I start working up a patch? Thanks, Alex [0] https://github.com/AlexHill/django-relativity/pull/1#issuecomment-418239406 On Mon, 10 Sep 2018 at 4:59 am, Silvio wrote: > Alex, > > This is a very useful pattern, that like many others, I've also > implemented in an ad-hoc fashion using a ton of undocumented internal APIs. > So I fully agree standardizing it would be great. Something similar is: > > https://groups.google.com/forum/#!topic/django-developers/ADSuUUuZp3Q > > Essentially, I've ended up with the need for: > > ComputedField > > and > > ComputedRelationship > > where both have all of the niceties that regular fields and foreign > relationships have. > > So I'd love to see this in Django. > > - > Silvio > > > On Sunday, September 2, 2018 at 10:55:58 PM UTC-4, Alex Hill wrote: >> >> Hi Simon, >> >> Thanks for looking at this and for providing some context - I had looked >> at FilteredRelation but I hadn't seen reverse-unique. It makes me more >> confident that this is a good direction to take. I've reimplemented >> ReverseUnique using Relationship [0] and the tests pass, with the only code >> carried over that for discovery of the FK link. >> >> > I'm not totally sold on the API but having an analog of what >> ForeignObject is to ForeignKey for ManyToManyField would definitely be >> useful. >> >> I'm not tied to the API, but I think passing a Q as a predicate makes >> sense especially given that it's what both FilteredRelation and >> ReverseUnique do. The core of the idea is that we can express a >> relationship as a combination of predicate and arity. In practise I don't >> think this would be used all that much by users directly - more by >> third-party apps like mptt, and perhaps Django internally. >> >> > From what I can see in relativity.fields[0] most of the additional >> logic revolves around the extra filtering capabilites through Restriction. >> >> Yeah that's what it boils down to. We return no columns to join against, >> and return a compilable Restriction from get_extra_restriction to provide >> all the ON conditions. The rest of it is making the descriptors, rels, >> prefetch, etc work. >> >> > Do you have an idea of what the fields.related inheritance chain would >> look like if it was part of core? >> >> The least intrusive, and probably a good starting point, would be to >> introduce Relationship alongside the other relation fields as a standalone >> feature, modifying the ORM to allow the implementation to be less hacky. It >> would remain a subclass of ForeignObject (or perhaps RelatedField - I'll >> give that a try). >> >> In the future there's potential for a nice refactor of the ORM to >> generalise join conditions from key-equality to arbitrary predicates of >> which key equality is just one case, at which point Relationship could sit >> comfortably as a base class of all the other relations. The assumption that >> join==key-equality is pervasive and I think that's an unnecessarily large >> chunk of work to take on at this point - it would be better to get the >> feature in, then have a release cycle or so to think about the best way to >> approach that problem and if we even want to. >> >> I would be happy to write up a DEP expanding on an implementation plan >> and potential future work. >> >> Thanks, >> Alex >> >> [0] >> https://github.com/AlexHill/django-reverse-unique/blob/624c8b19406a40b8e1a2c969c23a6b45bed5a896/reverse_unique/fields.py >> >> >> >> >> >> >> On Fri, 31 Aug 2018 at 12:12 am, charettes wrote: >> > Hello Alex! >>> >>> Thanks for your work on this project, this is definitely something that >>> I believe would be useful in Django's core based on the number of times I >>> implemented a filtered queryset getter on Models. >>> >>> I'm not totally sold on the API but having an analog of what >>> ForeignObject is to ForeignKey for ManyToManyField would definitely be >>> useful. >>> >>> From what I can see in relativity.fields[0] most of the additional logic >>> revolves around the extra filtering capabilites through Restriction. >>> >>> Do you have an idea of what the fields.related inheri
Re: Proposal: relationships based on arbitrary predicates
Alex, This is a very useful pattern, that like many others, I've also implemented in an ad-hoc fashion using a ton of undocumented internal APIs. So I fully agree standardizing it would be great. Something similar is: https://groups.google.com/forum/#!topic/django-developers/ADSuUUuZp3Q Essentially, I've ended up with the need for: ComputedField and ComputedRelationship where both have all of the niceties that regular fields and foreign relationships have. So I'd love to see this in Django. - Silvio On Sunday, September 2, 2018 at 10:55:58 PM UTC-4, Alex Hill wrote: > > Hi Simon, > > Thanks for looking at this and for providing some context - I had looked > at FilteredRelation but I hadn't seen reverse-unique. It makes me more > confident that this is a good direction to take. I've reimplemented > ReverseUnique using Relationship [0] and the tests pass, with the only code > carried over that for discovery of the FK link. > > > I'm not totally sold on the API but having an analog of what > ForeignObject is to ForeignKey for ManyToManyField would definitely be > useful. > > I'm not tied to the API, but I think passing a Q as a predicate makes > sense especially given that it's what both FilteredRelation and > ReverseUnique do. The core of the idea is that we can express a > relationship as a combination of predicate and arity. In practise I don't > think this would be used all that much by users directly - more by > third-party apps like mptt, and perhaps Django internally. > > > From what I can see in relativity.fields[0] most of the additional logic > revolves around the extra filtering capabilites through Restriction. > > Yeah that's what it boils down to. We return no columns to join against, > and return a compilable Restriction from get_extra_restriction to provide > all the ON conditions. The rest of it is making the descriptors, rels, > prefetch, etc work. > > > Do you have an idea of what the fields.related inheritance chain would > look like if it was part of core? > > The least intrusive, and probably a good starting point, would be to > introduce Relationship alongside the other relation fields as a standalone > feature, modifying the ORM to allow the implementation to be less hacky. It > would remain a subclass of ForeignObject (or perhaps RelatedField - I'll > give that a try). > > In the future there's potential for a nice refactor of the ORM to > generalise join conditions from key-equality to arbitrary predicates of > which key equality is just one case, at which point Relationship could sit > comfortably as a base class of all the other relations. The assumption that > join==key-equality is pervasive and I think that's an unnecessarily large > chunk of work to take on at this point - it would be better to get the > feature in, then have a release cycle or so to think about the best way to > approach that problem and if we even want to. > > I would be happy to write up a DEP expanding on an implementation plan and > potential future work. > > Thanks, > Alex > > [0] > https://github.com/AlexHill/django-reverse-unique/blob/624c8b19406a40b8e1a2c969c23a6b45bed5a896/reverse_unique/fields.py > > > > > > > On Fri, 31 Aug 2018 at 12:12 am, charettes > wrote: > >> Hello Alex! >> >> Thanks for your work on this project, this is definitely something that I >> believe would be useful in Django's core based on the number of times I >> implemented a filtered queryset getter on Models. >> >> I'm not totally sold on the API but having an analog of what >> ForeignObject is to ForeignKey for ManyToManyField would definitely be >> useful. >> >> From what I can see in relativity.fields[0] most of the additional logic >> revolves around the extra filtering capabilites through Restriction. >> >> Do you have an idea of what the fields.related inheritance chain would >> look like if it was part of core? I feel like having >> Relation(RelatedField), ForeignObject(Relation), ManyToManyField(Relation) >> and adding the filtering logic to Relation could work but I'd be interested >> to hear what you think here. FWIW Anssi implemented something similar[1] >> for reverse unique relationship before FilteredRelation() was introduced. >> >> In a sense Relation would be form of virtual field like ForeignObject >> since it's not in charge of any database field handling. >> >> Simon >> >> [0] >> https://github.com/AlexHill/django-relativity/blob/3802608c64e86c62ab6268efc37a3f5ca8539221/relativity/fields.py >> [1] https://github.com/akaariai/django-reverse-unique >> >> >> >> Le jeudi 30 août 2018 11:38:16 UTC-4, Alex Hill a écrit : >>> >>> Hi all, >>> >>> I've run into many situations during my time using Django where I've >>> wanted to be able to express relations based on some other criteria than >>> foreign key equality. A few examples: >>> - descendants or children of a node in a tree structure >>> - saved search terms to search results >>> - a model containing a dat
Re: Proposal: relationships based on arbitrary predicates
Hi Simon, Thanks for looking at this and for providing some context - I had looked at FilteredRelation but I hadn't seen reverse-unique. It makes me more confident that this is a good direction to take. I've reimplemented ReverseUnique using Relationship [0] and the tests pass, with the only code carried over that for discovery of the FK link. > I'm not totally sold on the API but having an analog of what ForeignObject is to ForeignKey for ManyToManyField would definitely be useful. I'm not tied to the API, but I think passing a Q as a predicate makes sense especially given that it's what both FilteredRelation and ReverseUnique do. The core of the idea is that we can express a relationship as a combination of predicate and arity. In practise I don't think this would be used all that much by users directly - more by third-party apps like mptt, and perhaps Django internally. > From what I can see in relativity.fields[0] most of the additional logic revolves around the extra filtering capabilites through Restriction. Yeah that's what it boils down to. We return no columns to join against, and return a compilable Restriction from get_extra_restriction to provide all the ON conditions. The rest of it is making the descriptors, rels, prefetch, etc work. > Do you have an idea of what the fields.related inheritance chain would look like if it was part of core? The least intrusive, and probably a good starting point, would be to introduce Relationship alongside the other relation fields as a standalone feature, modifying the ORM to allow the implementation to be less hacky. It would remain a subclass of ForeignObject (or perhaps RelatedField - I'll give that a try). In the future there's potential for a nice refactor of the ORM to generalise join conditions from key-equality to arbitrary predicates of which key equality is just one case, at which point Relationship could sit comfortably as a base class of all the other relations. The assumption that join==key-equality is pervasive and I think that's an unnecessarily large chunk of work to take on at this point - it would be better to get the feature in, then have a release cycle or so to think about the best way to approach that problem and if we even want to. I would be happy to write up a DEP expanding on an implementation plan and potential future work. Thanks, Alex [0] https://github.com/AlexHill/django-reverse-unique/blob/624c8b19406a40b8e1a2c969c23a6b45bed5a896/reverse_unique/fields.py On Fri, 31 Aug 2018 at 12:12 am, charettes wrote: > Hello Alex! > > Thanks for your work on this project, this is definitely something that I > believe would be useful in Django's core based on the number of times I > implemented a filtered queryset getter on Models. > > I'm not totally sold on the API but having an analog of what ForeignObject > is to ForeignKey for ManyToManyField would definitely be useful. > > From what I can see in relativity.fields[0] most of the additional logic > revolves around the extra filtering capabilites through Restriction. > > Do you have an idea of what the fields.related inheritance chain would > look like if it was part of core? I feel like having > Relation(RelatedField), ForeignObject(Relation), ManyToManyField(Relation) > and adding the filtering logic to Relation could work but I'd be interested > to hear what you think here. FWIW Anssi implemented something similar[1] > for reverse unique relationship before FilteredRelation() was introduced. > > In a sense Relation would be form of virtual field like ForeignObject > since it's not in charge of any database field handling. > > Simon > > [0] > https://github.com/AlexHill/django-relativity/blob/3802608c64e86c62ab6268efc37a3f5ca8539221/relativity/fields.py > [1] https://github.com/akaariai/django-reverse-unique > > > > Le jeudi 30 août 2018 11:38:16 UTC-4, Alex Hill a écrit : >> >> Hi all, >> >> I've run into many situations during my time using Django where I've >> wanted to be able to express relations based on some other criteria than >> foreign key equality. A few examples: >> - descendants or children of a node in a tree structure >> - saved search terms to search results >> - a model containing a date range to timestamped items falling within >> that date range >> >> Currently to do this kind of thing, you might write a getter which >> returns a queryset - think for example mptt's get_descendants(). But you >> don't get any of the nice things a real relation field gives you - you >> can't use that relationship in filters, you can't select/prefetch_related() >> or values(), there's no reverse relationship, etc. >> >> I've written a Relationship field[0] that lets you define relations in >> terms of arbitrary Q filters containing objects of a new type, L. An L is >> like an F, but represents a field on the "local" or "left" side of the >> relation, where the Q is filtering against the remote "to" side of the >> relation. For example, in a materialised path tree, this
Re: Proposal: relationships based on arbitrary predicates
Hello Alex!
Thanks for your work on this project, this is definitely something that I
believe would be useful in Django's core based on the number of times I
implemented a filtered queryset getter on Models.
I'm not totally sold on the API but having an analog of what ForeignObject
is to ForeignKey for ManyToManyField would definitely be useful.
>From what I can see in relativity.fields[0] most of the additional logic
revolves around the extra filtering capabilites through Restriction.
Do you have an idea of what the fields.related inheritance chain would look
like if it was part of core? I feel like having Relation(RelatedField),
ForeignObject(Relation), ManyToManyField(Relation) and adding the filtering
logic to Relation could work but I'd be interested to hear what you think
here. FWIW Anssi implemented something similar[1] for reverse unique
relationship before FilteredRelation() was introduced.
In a sense Relation would be form of virtual field like ForeignObject since
it's not in charge of any database field handling.
Simon
[0]
https://github.com/AlexHill/django-relativity/blob/3802608c64e86c62ab6268efc37a3f5ca8539221/relativity/fields.py
[1] https://github.com/akaariai/django-reverse-unique
Le jeudi 30 août 2018 11:38:16 UTC-4, Alex Hill a écrit :
>
> Hi all,
>
> I've run into many situations during my time using Django where I've
> wanted to be able to express relations based on some other criteria than
> foreign key equality. A few examples:
> - descendants or children of a node in a tree structure
> - saved search terms to search results
> - a model containing a date range to timestamped items falling within that
> date range
>
> Currently to do this kind of thing, you might write a getter which returns
> a queryset - think for example mptt's get_descendants(). But you don't get
> any of the nice things a real relation field gives you - you can't use that
> relationship in filters, you can't select/prefetch_related() or values(),
> there's no reverse relationship, etc.
>
> I've written a Relationship field[0] that lets you define relations in
> terms of arbitrary Q filters containing objects of a new type, L. An L is
> like an F, but represents a field on the "local" or "left" side of the
> relation, where the Q is filtering against the remote "to" side of the
> relation. For example, in a materialised path tree, this is how you might
> express descendants:
>
> class Node(models.Model):
> path = models.TextField()
> descendants = Relationship(
> "self",
> Q(path__startswith=L('path'), pk__ne=L('pk')),
> multiple=True,
> reverse_multiple=True,
> related_name='ascendants',
> )
>
> Now you can use the descendants field like any other many-to-many field in
> all the places I mentioned above, but the relationship is based purely on
> prefix-matching on the path field. You also get an ascendants field on
> Node, which represents the path back to the root and can be used in the
> same way.
>
> I think this could make a nice new feature for Django. It would give a
> usability boost to anyone using MPTT or treebeard, for example. It works OK
> as a third-party library, but the current implementation relies heavily on
> undocumented ORM internals, and there are a few features I'd like to
> implement that are impractical without making some ORM changes.
>
> Thoughts/feedback/questions welcome!
>
> Thanks,
> Alex
>
> [0] https://github.com/alexhill/django-relativity
>
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/c87af098-baa4-4d45-9a4b-757166b41734%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Proposal: relationships based on arbitrary predicates
Hi all,
I've run into many situations during my time using Django where I've wanted
to be able to express relations based on some other criteria than foreign
key equality. A few examples:
- descendants or children of a node in a tree structure
- saved search terms to search results
- a model containing a date range to timestamped items falling within that
date range
Currently to do this kind of thing, you might write a getter which returns
a queryset - think for example mptt's get_descendants(). But you don't get
any of the nice things a real relation field gives you - you can't use that
relationship in filters, you can't select/prefetch_related() or values(),
there's no reverse relationship, etc.
I've written a Relationship field[0] that lets you define relations in
terms of arbitrary Q filters containing objects of a new type, L. An L is
like an F, but represents a field on the "local" or "left" side of the
relation, where the Q is filtering against the remote "to" side of the
relation. For example, in a materialised path tree, this is how you might
express descendants:
class Node(models.Model):
path = models.TextField()
descendants = Relationship(
"self",
Q(path__startswith=L('path'), pk__ne=L('pk')),
multiple=True,
reverse_multiple=True,
related_name='ascendants',
)
Now you can use the descendants field like any other many-to-many field in
all the places I mentioned above, but the relationship is based purely on
prefix-matching on the path field. You also get an ascendants field on
Node, which represents the path back to the root and can be used in the
same way.
I think this could make a nice new feature for Django. It would give a
usability boost to anyone using MPTT or treebeard, for example. It works OK
as a third-party library, but the current implementation relies heavily on
undocumented ORM internals, and there are a few features I'd like to
implement that are impractical without making some ORM changes.
Thoughts/feedback/questions welcome!
Thanks,
Alex
[0] https://github.com/alexhill/django-relativity
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CA%2BKBOKzXDHJnO09Trkws2MJDu4DnOUsgxgA634xpCAKk1O5xJw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
