Re: Composite fields

[Victor Porton]

I have implemented something like this: 
https://bitbucket.org/portonv/composite-fields

(I did my implementation before reading my thread.)

Please feel free to use and modify my code.

Well, now I should read this thread and 
https://github.com/django/deps/blob/master/draft/0191-composite-fields.rst

On Wednesday, March 4, 2015 at 3:16:37 PM UTC+2, Thomas Stephenson wrote:
>
> Considering the past two proposals I've made here, I doubt I'll get more 
> than an echo chamber effect on this one.
>
> For the past week or so I've spent a bit of time on a feature I've always 
> wanted to see land in django -- composite fields. The tickets have been 
> open in the bug tracker for quite some time (and there's a few related 
> ones, such as multi-column primary keys that can all be killed with the one 
> stone).
>
> The work is available on this branch of my fork of django 
> <https://github.com/ovangle/django/tree/composite_fields> for the moment 
> -- I haven't opened up a PR yet because there's still some features that 
> are still to be implemented that will be explained below, but I want to 
> give everybody a chance to tell me where I can stick it before I spend 
> *too* much time on it.
>
> So, without further ado, the proposal.
>
>
> Composite Fields - Implemented
>
> A composite field is an encapsulation of the functionality of a subset of 
> fields on a model. Composite fields can be defined in one of two ways:
>
> 1. Statically declared composite fields
>
> A statically declared composite field is defined in the same way a django 
> model is defined. There are two customisable transformation functions, 
> CompositeField.value_to_dict(self, value) and 
> CompositeField.value_from_dict(self, value) which can be used to associate 
> the field with a python object.
>
> All the serialization functions are implemented via the implementations of 
> the subfields.
>
> For example,  
>
> class MoneyField(models.CompositeField):
>currency_code = models.CharacterField(max_length=3)
>amount = models.DecimalField(max_digits=16, decimal_digits=4)
>
>## Overriding __init__ can be used to pass field parameters to the 
> subfields
>
>def value_from_dict(self, value):
>if value is None:
>   return None
>return Money(**value)
>
>    def value_to_dict(self, value):
>   if value is None:
>  return None
>   return {attr: getattr(value, attr) for attr in ('currency_code', 
> 'amount')}
>
> 2. Inline composite fields.
>
> An inline composite field is declared at the field definition site on the 
> body of a model, by providing the subfields as the 'fields' argument of the 
> CompositeField constructor. There are no transformation parameters 
> available to override when declaring a composite field in this fashion -- 
> the value of the field is always available as a python `dict` as an 
> attribute on the MyModel
>
> class MyModel(models.Model):
> id = models.CompositeField(fields = [
>('a', models.IntegerField()),
>('b', models.CharField(max_length=30)
> ], primary_key=True)
>
> This method for defining composite fields has a few drawbacks, but can be 
> useful if the only reason to add the composite field to the model was to 
> implement a unique_together or index_together constraint *
>
> * Although it's still possible to do that directly on class Meta.
>
> 3. Null
>
> Setting the value of a multi-column field to NULL is different than 
> setting any of the individual subfields to NULL. But there are cases (e.g. 
> Money) where we would like to be able to set `null=True` on a composite 
> field, but still retain 'NOT NULL' constraints on each of the subfield 
> columns.
>
> To solve this problem, every table which implements a CompositeField will 
> also add an implicit (semi-hidden) `isnull` subfield on the attribute, 
> which keeps track of whether it is the value of the composite field that is 
> null, or any of the particular subfields.
>
> 3. Querying.
>
> The syntax for querying over the subfields of a composite field will be 
> familiar to anyone who has queried over a relationship attribute in django.
>
> model.objects.filter(price__currency_code='USD', 
> price__amount__lt=Decimal('50.00')).all()
>
> In addition, composite fields can be queried via EXACT and IN lookups. It 
> is possible to implement custom lookups for specific statically defined 
> fields, but not recommended and not part of the official API.
>
> 4. Restrictions
>
> The following restrictions are currently imposed on the use of composi

Re: Composite fields

[Tim Graham]

Thomas, Please be patient. As for me, I'm focused on finishing up the 1.8 
release. I've added composite fields and a link to your DEP to the 1.9 
roadmap to help ensure it gets attention for that release cycle: 
https://code.djangoproject.com/wiki/Version1.9Roadmap

On Wednesday, March 18, 2015 at 5:28:43 AM UTC-4, Curtis Maloney wrote:
>
> Dev discussions typically happen on #django-dev on the Freenode IRC 
> network.
>
> --
> C
>
>
> On 18 March 2015 at 19:54, Thomas Stephenson  > wrote:
>
>> IRC discussion sounds fine with me, but I'd like to be involved. What's 
>> the channel?
>>
>> Thomas
>>
>> On 18 March 2015 at 15:47, Asif Saifuddin 
>> > wrote:
>>
>>> but? IRC discussion?
>>>
>>> On Tuesday, March 17, 2015 at 9:09:49 PM UTC+6, Thomas Stephenson wrote:
>>>>
>>>> Not impatient or anything, but...
>>>>
>>>> Bump.
>>>>
>>>> On 13 March 2015 at 14:07, Thomas Stephenson  wrote:
>>>>
>>>>> All the null handling stuff has been removed from the specification 
>>>>> and replaced with slightly more stringent restrictions on 
>>>>> `value_to_dict`. 
>>>>> There is an updated version which includes that change, plus a couple 
>>>>> more 
>>>>> alterations that were discussed here and on the PR available here. 
>>>>> <https://github.com/django/deps/pull/12>
>>>>>
>>>>> Thomas
>>>>>
>>>>> On 9 March 2015 at 17:43, Anssi Kääriäinen  wrote:
>>>>>
>>>>>> On Mon, Mar 9, 2015 at 8:06 AM, Thomas Stephenson  
>>>>>> wrote:
>>>>>> >> It doesn't result in good table design and adds 
>>>>>> restrictions/complication
>>>>>> >> to the ORM/migrations.
>>>>>> >
>>>>>> > Well, honestly, making all subfields always nullable (whether it is 
>>>>>> or not)
>>>>>> > in order to support the composite field being potentially nullable 
>>>>>> doesn't
>>>>>> > really result in good table design either.
>>>>>> >
>>>>>> >> Enforcing that both x and y have a value should be handled with a
>>>>>> >> constraint and/or validators.
>>>>>> >
>>>>>> > There is _no_ problem that is better handled with validators than an
>>>>>> > equivalent database constraint unless you assume that the framework 
>>>>>> is the
>>>>>> > only client your database will ever have.
>>>>>>
>>>>>> You'll likely need both. The database constraint is there to catch all
>>>>>> errors, whether generated by the framework or some other client. The
>>>>>> validator will supply nice user-facing error messages. This is how
>>>>>> unique constraints work for example: there is a model validator that
>>>>>> says "this value already exists in the database", but we also have
>>>>>> database level constraint to make sure there is no way to store
>>>>>> invalid values.
>>>>>>
>>>>>> >> -1 on silently changing the field definition. At best this would 
>>>>>> result in
>>>>>> >> an unnecessary migration when the error is discovered. Django 
>>>>>> should error
>>>>>> >> out with a message that the composite field definition is invalid.
>>>>>> >
>>>>>> > Changing the subfield definitions in the composite field 
>>>>>> constructor has to
>>>>>> > be supported, there's no way of passing parameters to subfields if 
>>>>>> you
>>>>>> > don't. What mistake is there to be found? Either you want the 
>>>>>> composite
>>>>>> > field to be nullable or you don't. You're being explicit about it 
>>>>>> by passing
>>>>>> > `null=True` into the field definition and it's a standard field 
>>>>>> parameter.
>>>>>>
>>>>>> If the user supplies the fields, then we don't want to change them
>>>>>> silently. We could have checks framework error for incompatible
>>>>>> settings however. If the field auto-creates the subfie

Re: Composite fields

[Curtis Maloney]

Dev discussions typically happen on #django-dev on the Freenode IRC network.

--
C


On 18 March 2015 at 19:54, Thomas Stephenson  wrote:

> IRC discussion sounds fine with me, but I'd like to be involved. What's
> the channel?
>
> Thomas
>
> On 18 March 2015 at 15:47, Asif Saifuddin  wrote:
>
>> but? IRC discussion?
>>
>> On Tuesday, March 17, 2015 at 9:09:49 PM UTC+6, Thomas Stephenson wrote:
>>>
>>> Not impatient or anything, but...
>>>
>>> Bump.
>>>
>>> On 13 March 2015 at 14:07, Thomas Stephenson  wrote:
>>>
 All the null handling stuff has been removed from the specification and
 replaced with slightly more stringent restrictions on `value_to_dict`.
 There is an updated version which includes that change, plus a couple more
 alterations that were discussed here and on the PR available here.
 

 Thomas

 On 9 March 2015 at 17:43, Anssi Kääriäinen  wrote:

> On Mon, Mar 9, 2015 at 8:06 AM, Thomas Stephenson 
> wrote:
> >> It doesn't result in good table design and adds
> restrictions/complication
> >> to the ORM/migrations.
> >
> > Well, honestly, making all subfields always nullable (whether it is
> or not)
> > in order to support the composite field being potentially nullable
> doesn't
> > really result in good table design either.
> >
> >> Enforcing that both x and y have a value should be handled with a
> >> constraint and/or validators.
> >
> > There is _no_ problem that is better handled with validators than an
> > equivalent database constraint unless you assume that the framework
> is the
> > only client your database will ever have.
>
> You'll likely need both. The database constraint is there to catch all
> errors, whether generated by the framework or some other client. The
> validator will supply nice user-facing error messages. This is how
> unique constraints work for example: there is a model validator that
> says "this value already exists in the database", but we also have
> database level constraint to make sure there is no way to store
> invalid values.
>
> >> -1 on silently changing the field definition. At best this would
> result in
> >> an unnecessary migration when the error is discovered. Django
> should error
> >> out with a message that the composite field definition is invalid.
> >
> > Changing the subfield definitions in the composite field constructor
> has to
> > be supported, there's no way of passing parameters to subfields if
> you
> > don't. What mistake is there to be found? Either you want the
> composite
> > field to be nullable or you don't. You're being explicit about it by
> passing
> > `null=True` into the field definition and it's a standard field
> parameter.
>
> If the user supplies the fields, then we don't want to change them
> silently. We could have checks framework error for incompatible
> settings however. If the field auto-creates the subfields, then it can
> alter them in any way it wishes.
>
> Making the composite field null only when all of its subfields are
> null seems the right behavior to me. Otherwise you'd have strange
> cases where .filter(money__amount=10) is true, but
> .filter(money__isnull=True) is also true for the same row.
>
>  - Anssi
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/CALMtK1GCSfemy%2BN0HLPMbsovwgcSy%2BwEw6QF-%
> 2BoZ-tfBCTSkUw%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>


>>>  --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/0f8b0979-7e15-45d2-ab76-88dc747a0bf7%40googlegroups.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  -

Re: Composite fields

[Thomas Stephenson]

IRC discussion sounds fine with me, but I'd like to be involved. What's the
channel?

Thomas

On 18 March 2015 at 15:47, Asif Saifuddin  wrote:

> but? IRC discussion?
>
> On Tuesday, March 17, 2015 at 9:09:49 PM UTC+6, Thomas Stephenson wrote:
>>
>> Not impatient or anything, but...
>>
>> Bump.
>>
>> On 13 March 2015 at 14:07, Thomas Stephenson  wrote:
>>
>>> All the null handling stuff has been removed from the specification and
>>> replaced with slightly more stringent restrictions on `value_to_dict`.
>>> There is an updated version which includes that change, plus a couple more
>>> alterations that were discussed here and on the PR available here.
>>> 
>>>
>>> Thomas
>>>
>>> On 9 March 2015 at 17:43, Anssi Kääriäinen  wrote:
>>>
 On Mon, Mar 9, 2015 at 8:06 AM, Thomas Stephenson 
 wrote:
 >> It doesn't result in good table design and adds
 restrictions/complication
 >> to the ORM/migrations.
 >
 > Well, honestly, making all subfields always nullable (whether it is
 or not)
 > in order to support the composite field being potentially nullable
 doesn't
 > really result in good table design either.
 >
 >> Enforcing that both x and y have a value should be handled with a
 >> constraint and/or validators.
 >
 > There is _no_ problem that is better handled with validators than an
 > equivalent database constraint unless you assume that the framework
 is the
 > only client your database will ever have.

 You'll likely need both. The database constraint is there to catch all
 errors, whether generated by the framework or some other client. The
 validator will supply nice user-facing error messages. This is how
 unique constraints work for example: there is a model validator that
 says "this value already exists in the database", but we also have
 database level constraint to make sure there is no way to store
 invalid values.

 >> -1 on silently changing the field definition. At best this would
 result in
 >> an unnecessary migration when the error is discovered. Django should
 error
 >> out with a message that the composite field definition is invalid.
 >
 > Changing the subfield definitions in the composite field constructor
 has to
 > be supported, there's no way of passing parameters to subfields if you
 > don't. What mistake is there to be found? Either you want the
 composite
 > field to be nullable or you don't. You're being explicit about it by
 passing
 > `null=True` into the field definition and it's a standard field
 parameter.

 If the user supplies the fields, then we don't want to change them
 silently. We could have checks framework error for incompatible
 settings however. If the field auto-creates the subfields, then it can
 alter them in any way it wishes.

 Making the composite field null only when all of its subfields are
 null seems the right behavior to me. Otherwise you'd have strange
 cases where .filter(money__amount=10) is true, but
 .filter(money__isnull=True) is also true for the same row.

  - Anssi

 --
 You received this message because you are subscribed to the Google
 Groups "Django developers  (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at http://groups.google.com/group/django-developers.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/django-developers/CALMtK1GCSfemy%2BN0HLPMbsovwgcSy%2BwEw6QF-%
 2BoZ-tfBCTSkUw%40mail.gmail.com.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/0f8b0979-7e15-45d2-ab76-88dc747a0bf7%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroup

Re: Composite fields

[Asif Saifuddin]

but? IRC discussion?

On Tuesday, March 17, 2015 at 9:09:49 PM UTC+6, Thomas Stephenson wrote:
>
> Not impatient or anything, but...
>
> Bump.
>
> On 13 March 2015 at 14:07, Thomas Stephenson  > wrote:
>
>> All the null handling stuff has been removed from the specification and 
>> replaced with slightly more stringent restrictions on `value_to_dict`. 
>> There is an updated version which includes that change, plus a couple more 
>> alterations that were discussed here and on the PR available here. 
>> 
>>
>> Thomas
>>
>> On 9 March 2015 at 17:43, Anssi Kääriäinen > > wrote:
>>
>>> On Mon, Mar 9, 2015 at 8:06 AM, Thomas Stephenson >> > wrote:
>>> >> It doesn't result in good table design and adds 
>>> restrictions/complication
>>> >> to the ORM/migrations.
>>> >
>>> > Well, honestly, making all subfields always nullable (whether it is or 
>>> not)
>>> > in order to support the composite field being potentially nullable 
>>> doesn't
>>> > really result in good table design either.
>>> >
>>> >> Enforcing that both x and y have a value should be handled with a
>>> >> constraint and/or validators.
>>> >
>>> > There is _no_ problem that is better handled with validators than an
>>> > equivalent database constraint unless you assume that the framework is 
>>> the
>>> > only client your database will ever have.
>>>
>>> You'll likely need both. The database constraint is there to catch all
>>> errors, whether generated by the framework or some other client. The
>>> validator will supply nice user-facing error messages. This is how
>>> unique constraints work for example: there is a model validator that
>>> says "this value already exists in the database", but we also have
>>> database level constraint to make sure there is no way to store
>>> invalid values.
>>>
>>> >> -1 on silently changing the field definition. At best this would 
>>> result in
>>> >> an unnecessary migration when the error is discovered. Django should 
>>> error
>>> >> out with a message that the composite field definition is invalid.
>>> >
>>> > Changing the subfield definitions in the composite field constructor 
>>> has to
>>> > be supported, there's no way of passing parameters to subfields if you
>>> > don't. What mistake is there to be found? Either you want the composite
>>> > field to be nullable or you don't. You're being explicit about it by 
>>> passing
>>> > `null=True` into the field definition and it's a standard field 
>>> parameter.
>>>
>>> If the user supplies the fields, then we don't want to change them
>>> silently. We could have checks framework error for incompatible
>>> settings however. If the field auto-creates the subfields, then it can
>>> alter them in any way it wishes.
>>>
>>> Making the composite field null only when all of its subfields are
>>> null seems the right behavior to me. Otherwise you'd have strange
>>> cases where .filter(money__amount=10) is true, but
>>> .filter(money__isnull=True) is also true for the same row.
>>>
>>>  - Anssi
>>>
>>> --
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers  (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com .
>>> To post to this group, send email to django-d...@googlegroups.com 
>>> .
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/CALMtK1GCSfemy%2BN0HLPMbsovwgcSy%2BwEw6QF-%2BoZ-tfBCTSkUw%40mail.gmail.com
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0f8b0979-7e15-45d2-ab76-88dc747a0bf7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Composite fields

[Thomas Stephenson]

Not impatient or anything, but...

Bump.

On 13 March 2015 at 14:07, Thomas Stephenson  wrote:

> All the null handling stuff has been removed from the specification and
> replaced with slightly more stringent restrictions on `value_to_dict`.
> There is an updated version which includes that change, plus a couple more
> alterations that were discussed here and on the PR available here.
> 
>
> Thomas
>
> On 9 March 2015 at 17:43, Anssi Kääriäinen  wrote:
>
>> On Mon, Mar 9, 2015 at 8:06 AM, Thomas Stephenson 
>> wrote:
>> >> It doesn't result in good table design and adds
>> restrictions/complication
>> >> to the ORM/migrations.
>> >
>> > Well, honestly, making all subfields always nullable (whether it is or
>> not)
>> > in order to support the composite field being potentially nullable
>> doesn't
>> > really result in good table design either.
>> >
>> >> Enforcing that both x and y have a value should be handled with a
>> >> constraint and/or validators.
>> >
>> > There is _no_ problem that is better handled with validators than an
>> > equivalent database constraint unless you assume that the framework is
>> the
>> > only client your database will ever have.
>>
>> You'll likely need both. The database constraint is there to catch all
>> errors, whether generated by the framework or some other client. The
>> validator will supply nice user-facing error messages. This is how
>> unique constraints work for example: there is a model validator that
>> says "this value already exists in the database", but we also have
>> database level constraint to make sure there is no way to store
>> invalid values.
>>
>> >> -1 on silently changing the field definition. At best this would
>> result in
>> >> an unnecessary migration when the error is discovered. Django should
>> error
>> >> out with a message that the composite field definition is invalid.
>> >
>> > Changing the subfield definitions in the composite field constructor
>> has to
>> > be supported, there's no way of passing parameters to subfields if you
>> > don't. What mistake is there to be found? Either you want the composite
>> > field to be nullable or you don't. You're being explicit about it by
>> passing
>> > `null=True` into the field definition and it's a standard field
>> parameter.
>>
>> If the user supplies the fields, then we don't want to change them
>> silently. We could have checks framework error for incompatible
>> settings however. If the field auto-creates the subfields, then it can
>> alter them in any way it wishes.
>>
>> Making the composite field null only when all of its subfields are
>> null seems the right behavior to me. Otherwise you'd have strange
>> cases where .filter(money__amount=10) is true, but
>> .filter(money__isnull=True) is also true for the same row.
>>
>>  - Anssi
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CALMtK1GCSfemy%2BN0HLPMbsovwgcSy%2BwEw6QF-%2BoZ-tfBCTSkUw%40mail.gmail.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2Bm8oA9ALKEfWOOZCyLxqDawaBhy0QTAjjNXWFOB5foSd8e5%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Composite fields

[Thomas Stephenson]

All the null handling stuff has been removed from the specification and
replaced with slightly more stringent restrictions on `value_to_dict`.
There is an updated version which includes that change, plus a couple more
alterations that were discussed here and on the PR available here.


Thomas

On 9 March 2015 at 17:43, Anssi Kääriäinen  wrote:

> On Mon, Mar 9, 2015 at 8:06 AM, Thomas Stephenson 
> wrote:
> >> It doesn't result in good table design and adds
> restrictions/complication
> >> to the ORM/migrations.
> >
> > Well, honestly, making all subfields always nullable (whether it is or
> not)
> > in order to support the composite field being potentially nullable
> doesn't
> > really result in good table design either.
> >
> >> Enforcing that both x and y have a value should be handled with a
> >> constraint and/or validators.
> >
> > There is _no_ problem that is better handled with validators than an
> > equivalent database constraint unless you assume that the framework is
> the
> > only client your database will ever have.
>
> You'll likely need both. The database constraint is there to catch all
> errors, whether generated by the framework or some other client. The
> validator will supply nice user-facing error messages. This is how
> unique constraints work for example: there is a model validator that
> says "this value already exists in the database", but we also have
> database level constraint to make sure there is no way to store
> invalid values.
>
> >> -1 on silently changing the field definition. At best this would result
> in
> >> an unnecessary migration when the error is discovered. Django should
> error
> >> out with a message that the composite field definition is invalid.
> >
> > Changing the subfield definitions in the composite field constructor has
> to
> > be supported, there's no way of passing parameters to subfields if you
> > don't. What mistake is there to be found? Either you want the composite
> > field to be nullable or you don't. You're being explicit about it by
> passing
> > `null=True` into the field definition and it's a standard field
> parameter.
>
> If the user supplies the fields, then we don't want to change them
> silently. We could have checks framework error for incompatible
> settings however. If the field auto-creates the subfields, then it can
> alter them in any way it wishes.
>
> Making the composite field null only when all of its subfields are
> null seems the right behavior to me. Otherwise you'd have strange
> cases where .filter(money__amount=10) is true, but
> .filter(money__isnull=True) is also true for the same row.
>
>  - Anssi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CALMtK1GCSfemy%2BN0HLPMbsovwgcSy%2BwEw6QF-%2BoZ-tfBCTSkUw%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2Bm8oA_wa-CvNsKe5W_5bAWLtyHBQfTa77RDNYfg8kOfLM9kvg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Composite fields

[Anssi Kääriäinen]

On Mon, Mar 9, 2015 at 8:06 AM, Thomas Stephenson  wrote:
>> It doesn't result in good table design and adds restrictions/complication
>> to the ORM/migrations.
>
> Well, honestly, making all subfields always nullable (whether it is or not)
> in order to support the composite field being potentially nullable doesn't
> really result in good table design either.
>
>> Enforcing that both x and y have a value should be handled with a
>> constraint and/or validators.
>
> There is _no_ problem that is better handled with validators than an
> equivalent database constraint unless you assume that the framework is the
> only client your database will ever have.

You'll likely need both. The database constraint is there to catch all
errors, whether generated by the framework or some other client. The
validator will supply nice user-facing error messages. This is how
unique constraints work for example: there is a model validator that
says "this value already exists in the database", but we also have
database level constraint to make sure there is no way to store
invalid values.

>> -1 on silently changing the field definition. At best this would result in
>> an unnecessary migration when the error is discovered. Django should error
>> out with a message that the composite field definition is invalid.
>
> Changing the subfield definitions in the composite field constructor has to
> be supported, there's no way of passing parameters to subfields if you
> don't. What mistake is there to be found? Either you want the composite
> field to be nullable or you don't. You're being explicit about it by passing
> `null=True` into the field definition and it's a standard field parameter.

If the user supplies the fields, then we don't want to change them
silently. We could have checks framework error for incompatible
settings however. If the field auto-creates the subfields, then it can
alter them in any way it wishes.

Making the composite field null only when all of its subfields are
null seems the right behavior to me. Otherwise you'd have strange
cases where .filter(money__amount=10) is true, but
.filter(money__isnull=True) is also true for the same row.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1GCSfemy%2BN0HLPMbsovwgcSy%2BwEw6QF-%2BoZ-tfBCTSkUw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Composite fields

[Thomas Stephenson]

also can't say "well, look at the field definition and if a
>>> column is nullable, then use it as a marker, because what if you instead
>>> had:
>>>
>>> class Point(models.CompositeField):
>>>x = models.IntegerField(null=True)
>>>y = models.IntegerField()
>>>z = models.IntegerField(null=True)
>>>
>>> as your original class? `x` could still be used as the "marker" column,
>>> but any transformation you'd make with the above rule. So the framework
>>> can't actually define a workable `isnull` query (or an 'exact' query when
>>> the python value is `None`). The whole issue of the "marker" column is
>>> fraught anyway, because it breaks all the NOT NULL constraints on other
>>> columns anyway.
>>>
>>>
>>>
>>> So instead of all that, the user decides that they're going to solve the
>>> problem with
>>>
>>> class MaybeLocation(models.Model):
>>>has_point = models.BooleanField()
>>>point = models.Point(default=Point(0,0))
>>>
>>> which is exactly the same solution as including the implicit `isnull`
>>> field when the field is created with `null=True`, except the user has to do
>>> it explicitly, and implement the logic surrounding the definition
>>> themselves.
>>>
>>> They also can't define a new CompositeField like
>>>
>>> class MaybePoint(...):
>>>has_point = models.BooleanField()
>>>point = Point()
>>>
>>> because of the inheritance limitations (which I might have to lift in
>>> the medium-long term anyway, but which I'd like to preserve for as long as
>>> possible to keep the initial implementation and API simple).
>>>
>>> The point is (no pun intended) that I can't really think of a good way
>>> to map a python None to the values of a composite field _without_ the extra
>>> implicit column. It's not the nicest solution in the world, but it works.
>>>
>>> Thomas
>>>
>>> On 8 March 2015 at 16:29, Aron Podrigal 
>>> wrote:
>>>
>>>> Hi Thomas, I replied earlier before you posted, looks like my message
>>>> got sent actually a lot later.
>>>>
>>>> having a function like  *models.constrain(x, y, unique=True) *makes
>>>> sense for inline declarations.
>>>>
>>>> About null handling, If all columns on the database are null=True and
>>>> all columns have a null value, then the composite field value would be
>>>>  `None'. Otherwise it would be a dict mapping to the subfields, and a query
>>>> needs to specify the subfields it queries against.
>>>>
>>>>
>>>> Aron.
>>>>
>>>> On Saturday, March 7, 2015 at 11:16:12 PM UTC-5, Thomas Stephenson
>>>> wrote:
>>>>>
>>>>> Aymeric,
>>>>>
>>>>> Thanks for your input. I feel some of your concerns have been
>>>>> addressed in the DEPs I made, which have included quite a bit of input 
>>>>> from
>>>>> this thread along with the original design. That said, some of the points
>>>>> you've raised are new and haven't been raised by other people, so I'll 
>>>>> give
>>>>> you a full reply.
>>>>>
>>>>> 1) That's still something I have to do, but more time consuming than
>>>>> other parts of creating the proposal and I've put it off until I have a
>>>>> while to do it. The original implementation was done without too much
>>>>> consideration of prior attempts, because I was more interested in getting
>>>>> something working than I was in learning from the past.
>>>>>
>>>>> 2) I agree. The new syntax I've proposed is to add one or more
>>>>> class-level methods to the django.db.models API. The new syntax for
>>>>> "inline" fields is:
>>>>>
>>>>> class MyModel(models.Model):
>>>>> x = models.IntegerField()
>>>>> y = models.IntegerField()
>>>>> point = models.constrain(x, y, unique=True)
>>>>>
>>>> good
>>>>
>>>>>
>>>>> The reason for not using the CompositeField constructor is that a lot
>>>>> of the options which make sense for standalone field constructors (and all
>>>

Re: Composite fields

[Michael Manfre]

a
>> implicit column. It's not the nicest solution in the world, but it works.
>>
>> Thomas
>>
>> On 8 March 2015 at 16:29, Aron Podrigal  wrote:
>>
>>> Hi Thomas, I replied earlier before you posted, looks like my message
>>> got sent actually a lot later.
>>>
>>> having a function like  *models.constrain(x, y, unique=True) *makes
>>> sense for inline declarations.
>>>
>>> About null handling, If all columns on the database are null=True and
>>> all columns have a null value, then the composite field value would be
>>>  `None'. Otherwise it would be a dict mapping to the subfields, and a query
>>> needs to specify the subfields it queries against.
>>>
>>>
>>> Aron.
>>>
>>> On Saturday, March 7, 2015 at 11:16:12 PM UTC-5, Thomas Stephenson wrote:
>>>>
>>>> Aymeric,
>>>>
>>>> Thanks for your input. I feel some of your concerns have been addressed
>>>> in the DEPs I made, which have included quite a bit of input from this
>>>> thread along with the original design. That said, some of the points you've
>>>> raised are new and haven't been raised by other people, so I'll give you a
>>>> full reply.
>>>>
>>>> 1) That's still something I have to do, but more time consuming than
>>>> other parts of creating the proposal and I've put it off until I have a
>>>> while to do it. The original implementation was done without too much
>>>> consideration of prior attempts, because I was more interested in getting
>>>> something working than I was in learning from the past.
>>>>
>>>> 2) I agree. The new syntax I've proposed is to add one or more
>>>> class-level methods to the django.db.models API. The new syntax for
>>>> "inline" fields is:
>>>>
>>>> class MyModel(models.Model):
>>>> x = models.IntegerField()
>>>> y = models.IntegerField()
>>>> point = models.constrain(x, y, unique=True)
>>>>
>>> good
>>>
>>>>
>>>> The reason for not using the CompositeField constructor is that a lot
>>>> of the options which make sense for standalone field constructors (and all
>>>> the other field classes) make absolutely no sense when you're mainly
>>>> leveraging composite fields to provide a table level constraint to two
>>>> existing fields. Also there are some things that are commonly done in
>>>> practice (like providing `verbose_name` as a positional arg) that make
>>>> adding multiple leading positional arguments difficult.
>>>>
>>>> Note: It's a shame that we can't use py3's keyword only arguments here.
>>>>
>>>> 3) No, not all the field base API makes sense for composite fields --
>>>> in fact most of it doesn't. This is a huge problem with fields in general,
>>>> not just composite fields -- there's just too much functionality on the
>>>> basic Field class that assumes a one-to-one mapping between a Field and a
>>>> database column and too many parameters accepted by field base that only
>>>> make sense for subsets of the available field types.
>>>>
>>>> e.g.
>>>> - What does "blank" or "max_length" mean for an IntegerField?
>>>> - What does 'rel' mean for a NullBooleanField?
>>>> - What does get_col mean for a ManyToManyField?
>>>> etc.
>>>>
>>>> To fix this would mean introducing significant backwards incompatible
>>>> changes and, while I would support such a change, I'm not sure it's a
>>>> discussion that affects the ability to implement composite fields.
>>>>
>>>> 4) The addition of the `isnull` field is mainly to support the ability
>>>> to query for whether a composite field is NULL. If we leave it
>>>> implementation defined as to how to interpret whether a specific
>>>> configuration of subfields maps to a python `None` value for the composite
>>>> field, then it becomes really difficult to define lookup transformations to
>>>> query for null values in the table.
>>>>
>>>> But although it sounded like a good idea when I came up with it, it
>>>> raises more questions than it solves. I'm more than ready to go back to the
>>>> drawing board on that one.
>>>&g

Re: Composite fields

[Michael Manfre]

On Sun, Mar 8, 2015 at 12:12 PM, Thomas Stephenson 
wrote:

> @Aymeric
>
> Here's why I think the `isnull` column is necessary.
>
> Say you've got a custom field implementation:
>
> class Point(models.CompositeField):
>  x = models.IntegerField()
>  y = models.IntegerField()
>  # etc.
>
> and you use this to define a coordinate system to locate objects in your
> models. You justify that whenever you provide a point, you'll want both an
> x and y coordinate, so you make both of the subfields non-null.
>
> But then you come across a model which may or may not have a location.
> You'd like to say
>
> class MaybeLocatable(models.Model):
>point = Point(null=True)
>
> but you can't, because both of your subfields are non-nullable, so you
> define a nonsensical condition to apply to your Point
> "If I don't have an x-coordinate, then the point value should be
> considered null"
> so you update the Point field appropriately (and apply all the necessary
> migrations).
>

If a point can be nullable, then the subfields should be nullable.
Enforcing that both x and y have a value should be handled with a
constraint and/or validators. I really don't like "_isnull" subfield. It
doesn't result in good table design and adds restrictions/complication to
the ORM/migrations.


> You then want to query for all `MaybeLocatable` objects that are
> considered null. You can query for point__isnull=True (which is supposed to
> be supported). How does one define the lookup transformation for that?
>
> If there was an implicit 'isnull' column added, then the lookup
> transformation for both `point__isnull=True` and `point=None` become a
> lookup for the database column `point__isnull = True`. If you don't add
> that extra column, how does the framework what transform to make? It could
> query for both `point__x__isnull` and `point__y__isnull`, but that wouldn't
> match the semantics for the column.
>

The ORM would do the translation based upon the Composite field being
nullable. E.g. "point__isnull=True" and "point=None" would result in
"[point].[x] IS NULL AND [point].[y] IS NULL"


> And you also can't say "well, look at the field definition and if a column
> is nullable, then use it as a marker, because what if you instead had:
>
> class Point(models.CompositeField):
>x = models.IntegerField(null=True)
>y = models.IntegerField()
>z = models.IntegerField(null=True)
>
> as your original class? `x` could still be used as the "marker" column,
> but any transformation you'd make with the above rule. So the framework
> can't actually define a workable `isnull` query (or an 'exact' query when
> the python value is `None`). The whole issue of the "marker" column is
> fraught anyway, because it breaks all the NOT NULL constraints on other
> columns anyway.
>
>
If the composite field is nullable, then all subfields should be nullable.
We can add checks to identify any invalid configurations.


> So instead of all that, the user decides that they're going to solve the
> problem with
>
> class MaybeLocation(models.Model):
>has_point = models.BooleanField()
>point = models.Point(default=Point(0,0))
>
> which is exactly the same solution as including the implicit `isnull`
> field when the field is created with `null=True`, except the user has to do
> it explicitly, and implement the logic surrounding the definition
> themselves.
>

The more I think about it, my opinion is shifting solidly toward -1 on
implicit database fields in general. I can't think of any situations when
the benefit outweighs the extra complexity and standard support issues when
things are implicit. There are a lot of negatives to saving 1 line of code
in a model (or composite field) definition).

Your MaybeLocation example also imposes an extra database column on the
user for each nullable composite field they define, even if it may be
redundant for their data. For the below example, Publication would have
three implicit "isnull" fields added, even if "status" defines the business
rules about when deliverable, report, and poster fields are required.

class Publication(models.Modle):
status = model.CharField(max_length=20)
deliverable = MyDeliverableCompositeField(null=True)
report = MyReportCompositeField(null=True)
poster = MyPosterCompositeField(null=True)



> They also can't define a new CompositeField like
>
> class MaybePoint(...):
>has_point = models.BooleanField()
>point = Point()
>
> because of the inheritance limitations (which I might have to lift in the
> medium-long term anyway, but

Re: Composite fields

[Thomas Stephenson]

t;> Aron.
>>
>> On Saturday, March 7, 2015 at 11:16:12 PM UTC-5, Thomas Stephenson wrote:
>>>
>>> Aymeric,
>>>
>>> Thanks for your input. I feel some of your concerns have been addressed
>>> in the DEPs I made, which have included quite a bit of input from this
>>> thread along with the original design. That said, some of the points you've
>>> raised are new and haven't been raised by other people, so I'll give you a
>>> full reply.
>>>
>>> 1) That's still something I have to do, but more time consuming than
>>> other parts of creating the proposal and I've put it off until I have a
>>> while to do it. The original implementation was done without too much
>>> consideration of prior attempts, because I was more interested in getting
>>> something working than I was in learning from the past.
>>>
>>> 2) I agree. The new syntax I've proposed is to add one or more
>>> class-level methods to the django.db.models API. The new syntax for
>>> "inline" fields is:
>>>
>>> class MyModel(models.Model):
>>> x = models.IntegerField()
>>> y = models.IntegerField()
>>> point = models.constrain(x, y, unique=True)
>>>
>> good
>>
>>>
>>> The reason for not using the CompositeField constructor is that a lot of
>>> the options which make sense for standalone field constructors (and all the
>>> other field classes) make absolutely no sense when you're mainly leveraging
>>> composite fields to provide a table level constraint to two existing
>>> fields. Also there are some things that are commonly done in practice (like
>>> providing `verbose_name` as a positional arg) that make adding multiple
>>> leading positional arguments difficult.
>>>
>>> Note: It's a shame that we can't use py3's keyword only arguments here.
>>>
>>> 3) No, not all the field base API makes sense for composite fields -- in
>>> fact most of it doesn't. This is a huge problem with fields in general, not
>>> just composite fields -- there's just too much functionality on the basic
>>> Field class that assumes a one-to-one mapping between a Field and a
>>> database column and too many parameters accepted by field base that only
>>> make sense for subsets of the available field types.
>>>
>>> e.g.
>>> - What does "blank" or "max_length" mean for an IntegerField?
>>> - What does 'rel' mean for a NullBooleanField?
>>> - What does get_col mean for a ManyToManyField?
>>> etc.
>>>
>>> To fix this would mean introducing significant backwards incompatible
>>> changes and, while I would support such a change, I'm not sure it's a
>>> discussion that affects the ability to implement composite fields.
>>>
>>> 4) The addition of the `isnull` field is mainly to support the ability
>>> to query for whether a composite field is NULL. If we leave it
>>> implementation defined as to how to interpret whether a specific
>>> configuration of subfields maps to a python `None` value for the composite
>>> field, then it becomes really difficult to define lookup transformations to
>>> query for null values in the table.
>>>
>>> But although it sounded like a good idea when I came up with it, it
>>> raises more questions than it solves. I'm more than ready to go back to the
>>> drawing board on that one.
>>>
>> 5) Inheritance in the model API is complicated enough as it is without
>>> adding inheritance of field types. Yes, it could be supported by "beating
>>> the metaclass into submission", but it's not something that I think adds
>>> any particular value. I did change the "no subclassing at all" restriction
>>> to "only one class in an inheritance heirarchy can define subfields" when
>>> writing the DEP though, because Ansarri wants the ability for users to
>>> extend ForeignKey with their own python behaviour.
>>>
>>> Thomas
>>>
>>>
>>>
>>> On 7 March 2015 at 22:31, Aymeric Augustin >> polytechnique.org> wrote:
>>>
>>>> Hello Thomas,
>>>>
>>>> It’s hard for me to digest a two-page-long email on a complex topic
>>>> during
>>>> the week. I bookmarked your first email when it came in. It’s Saturday,
>>>> 11am,
>>>> and I dedicated m

Re: Composite fields

[Thomas Stephenson]

 class MyModel(models.Model):
>> x = models.IntegerField()
>> y = models.IntegerField()
>> point = models.constrain(x, y, unique=True)
>>
> good
>
>>
>> The reason for not using the CompositeField constructor is that a lot of
>> the options which make sense for standalone field constructors (and all the
>> other field classes) make absolutely no sense when you're mainly leveraging
>> composite fields to provide a table level constraint to two existing
>> fields. Also there are some things that are commonly done in practice (like
>> providing `verbose_name` as a positional arg) that make adding multiple
>> leading positional arguments difficult.
>>
>> Note: It's a shame that we can't use py3's keyword only arguments here.
>>
>> 3) No, not all the field base API makes sense for composite fields -- in
>> fact most of it doesn't. This is a huge problem with fields in general, not
>> just composite fields -- there's just too much functionality on the basic
>> Field class that assumes a one-to-one mapping between a Field and a
>> database column and too many parameters accepted by field base that only
>> make sense for subsets of the available field types.
>>
>> e.g.
>> - What does "blank" or "max_length" mean for an IntegerField?
>> - What does 'rel' mean for a NullBooleanField?
>> - What does get_col mean for a ManyToManyField?
>> etc.
>>
>> To fix this would mean introducing significant backwards incompatible
>> changes and, while I would support such a change, I'm not sure it's a
>> discussion that affects the ability to implement composite fields.
>>
>> 4) The addition of the `isnull` field is mainly to support the ability to
>> query for whether a composite field is NULL. If we leave it implementation
>> defined as to how to interpret whether a specific configuration of
>> subfields maps to a python `None` value for the composite field, then it
>> becomes really difficult to define lookup transformations to query for null
>> values in the table.
>>
>> But although it sounded like a good idea when I came up with it, it
>> raises more questions than it solves. I'm more than ready to go back to the
>> drawing board on that one.
>>
> 5) Inheritance in the model API is complicated enough as it is without
>> adding inheritance of field types. Yes, it could be supported by "beating
>> the metaclass into submission", but it's not something that I think adds
>> any particular value. I did change the "no subclassing at all" restriction
>> to "only one class in an inheritance heirarchy can define subfields" when
>> writing the DEP though, because Ansarri wants the ability for users to
>> extend ForeignKey with their own python behaviour.
>>
>> Thomas
>>
>>
>>
>> On 7 March 2015 at 22:31, Aymeric Augustin > > wrote:
>>
>>> Hello Thomas,
>>>
>>> It’s hard for me to digest a two-page-long email on a complex topic
>>> during
>>> the week. I bookmarked your first email when it came in. It’s Saturday,
>>> 11am,
>>> and I dedicated my first chunk of quality brain time to reading the
>>> entire
>>> thread. I’ll let you ponder what the effect of your second email was.
>>>
>>> In fact, if you propose something stupid, you’ll get a quick answer,
>>> because
>>> it’s easy to explain. Not getting answers right now means that your
>>> proposal
>>> is good enough to require consideration. Yes, that’s counter-intuitive.
>>>
>>> With that out of the way, here are my thoughts on your proposal. Some
>>> overlap
>>> with what other people have said in the thread.
>>>
>>> 1) I would find it reassuring if you described the landscape of past
>>> attempts,
>>> what good ideas you’re keeping, what bad ideas you’re throwing away — in
>>> short, if you convinced us that you’re building on top of past attempts.
>>> The
>>> main goal of this exercise is to guarantee that you don’t overlook a use
>>> case
>>> or an argument that made consensus in the past. (Don’t spend too much
>>> time on
>>> code; it my experience it’s harder to reuse and code matters much less
>>> than
>>> design.
>>>
>>> 2) The syntax for inline composite fields doesn't look very nice. Could
>>> you
>>> simplify it somehow? Anssi’s proposal is good. I assume that a composite
>>> fie

Re: Composite fields

[Aron Podrigal]

Hi Thomas, I replied earlier before you posted, looks like my message got 
sent actually a lot later.

having a function like  *models.constrain(x, y, unique=True) *makes sense 
for inline declarations.

About null handling, If all columns on the database are null=True and all 
columns have a null value, then the composite field value would be  `None'. 
Otherwise it would be a dict mapping to the subfields, and a query needs to 
specify the subfields it queries against.


Aron.

On Saturday, March 7, 2015 at 11:16:12 PM UTC-5, Thomas Stephenson wrote:
>
> Aymeric,
>
> Thanks for your input. I feel some of your concerns have been addressed in 
> the DEPs I made, which have included quite a bit of input from this thread 
> along with the original design. That said, some of the points you've raised 
> are new and haven't been raised by other people, so I'll give you a full 
> reply.
>
> 1) That's still something I have to do, but more time consuming than other 
> parts of creating the proposal and I've put it off until I have a while to 
> do it. The original implementation was done without too much consideration 
> of prior attempts, because I was more interested in getting something 
> working than I was in learning from the past. 
>
> 2) I agree. The new syntax I've proposed is to add one or more class-level 
> methods to the django.db.models API. The new syntax for "inline" fields is:
>
> class MyModel(models.Model):
> x = models.IntegerField()
> y = models.IntegerField()
> point = models.constrain(x, y, unique=True)
>
good 

>
> The reason for not using the CompositeField constructor is that a lot of 
> the options which make sense for standalone field constructors (and all the 
> other field classes) make absolutely no sense when you're mainly leveraging 
> composite fields to provide a table level constraint to two existing 
> fields. Also there are some things that are commonly done in practice (like 
> providing `verbose_name` as a positional arg) that make adding multiple 
> leading positional arguments difficult.
>
> Note: It's a shame that we can't use py3's keyword only arguments here.
>
> 3) No, not all the field base API makes sense for composite fields -- in 
> fact most of it doesn't. This is a huge problem with fields in general, not 
> just composite fields -- there's just too much functionality on the basic 
> Field class that assumes a one-to-one mapping between a Field and a 
> database column and too many parameters accepted by field base that only 
> make sense for subsets of the available field types.
>
> e.g. 
> - What does "blank" or "max_length" mean for an IntegerField?
> - What does 'rel' mean for a NullBooleanField?
> - What does get_col mean for a ManyToManyField?
> etc.
>
> To fix this would mean introducing significant backwards incompatible 
> changes and, while I would support such a change, I'm not sure it's a 
> discussion that affects the ability to implement composite fields.
>
> 4) The addition of the `isnull` field is mainly to support the ability to 
> query for whether a composite field is NULL. If we leave it implementation 
> defined as to how to interpret whether a specific configuration of 
> subfields maps to a python `None` value for the composite field, then it 
> becomes really difficult to define lookup transformations to query for null 
> values in the table. 
>
> But although it sounded like a good idea when I came up with it, it raises 
> more questions than it solves. I'm more than ready to go back to the 
> drawing board on that one. 
>
5) Inheritance in the model API is complicated enough as it is without 
> adding inheritance of field types. Yes, it could be supported by "beating 
> the metaclass into submission", but it's not something that I think adds 
> any particular value. I did change the "no subclassing at all" restriction 
> to "only one class in an inheritance heirarchy can define subfields" when 
> writing the DEP though, because Ansarri wants the ability for users to 
> extend ForeignKey with their own python behaviour. 
>
> Thomas
>
>
>
> On 7 March 2015 at 22:31, Aymeric Augustin  > wrote:
>
>> Hello Thomas,
>>
>> It’s hard for me to digest a two-page-long email on a complex topic during
>> the week. I bookmarked your first email when it came in. It’s Saturday, 
>> 11am,
>> and I dedicated my first chunk of quality brain time to reading the entire
>> thread. I’ll let you ponder what the effect of your second email was.
>>
>> In fact, if you propose something stupid, you’ll get a quick answer, 

Re: Composite fields

[Aron Podrigal]

Hi Thomas,

First, thanks for your work. I've had a plan to start working on this 
feature last week,
but I got too many things to work on with higher priority, Thankfully you 
opted for this, again Thanks.

just sharing my thoughts,

1)  I like the MoneyField model style definition.

2) I don't like the inline declaration, I would opt for Anssi's proposal.

3) as Aymeric mentioned, NOT NULL only makes sense for a real column on the 
database, so that should be only be handled on the SubFields.
Assigning python `None' to the CompositeField should be handled by the 
value_from_dict method.

4) querying syntax, sounds fare. However depending on how we decide on 
exposing the subfields through the Model Meta API, 
eg. if the subfields would be returned by default, then we should also 
allow directly querying of the subfield.

Aron


On Saturday, March 7, 2015 at 6:31:53 AM UTC-5, Aymeric Augustin wrote:
>
> Hello Thomas, 
>
> It’s hard for me to digest a two-page-long email on a complex topic during 
> the week. I bookmarked your first email when it came in. It’s Saturday, 
> 11am, 
> and I dedicated my first chunk of quality brain time to reading the entire 
> thread. I’ll let you ponder what the effect of your second email was. 
>
> In fact, if you propose something stupid, you’ll get a quick answer, 
> because 
> it’s easy to explain. Not getting answers right now means that your 
> proposal 
> is good enough to require consideration. Yes, that’s counter-intuitive. 
>
> With that out of the way, here are my thoughts on your proposal. Some 
> overlap 
> with what other people have said in the thread. 
>
> 1) I would find it reassuring if you described the landscape of past 
> attempts, 
> what good ideas you’re keeping, what bad ideas you’re throwing away — in 
> short, if you convinced us that you’re building on top of past attempts. 
> The 
> main goal of this exercise is to guarantee that you don’t overlook a use 
> case 
> or an argument that made consensus in the past. (Don’t spend too much time 
> on 
> code; it my experience it’s harder to reuse and code matters much less 
> than 
> design. 
>
> 2) The syntax for inline composite fields doesn't look very nice. Could 
> you 
> simplify it somehow? Anssi’s proposal is good. I assume that a composite 
> field 
> could add the subfields to the model class if they aren't defined 
> explicitly 
> and their names passed in arguments to the composite field. 
>
> 3) Have you checked that all the Field APIs make sense for CompositeField? 
> It's quite obvious that value_from/to_dict are needed. It's less obvious 
> that 
> everything else is still needed. 
>
> 4) I'm wary of the extra 'isnull' column. Couldn't we required that, if 
> the 
> composite field is NULL, at least one of the subfields is NULL? My idea is 
> that a nullable composite field would consider itself NULL if all nullable 
> subfields are NULL. Declaring a composite subfield nullable when all 
> subfields 
> are non-nullable would be an error. In your MoneyField example, the amount 
> subfield should be nullable when the composite field is nullable. 
>
> 5) I understand the first two restrictions. They required deeper 
> refactorings 
> to be lifted. The reason for the third one is less clear. Is it just a 
> matter 
> of beating the metaclass into cooperation? 
>
> Best, 
>
> -- 
> Aymeric. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e1024af3-2822-4136-979c-7980e8c99dc5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Composite fields

[Thomas Stephenson]

Aymeric,

Thanks for your input. I feel some of your concerns have been addressed in
the DEPs I made, which have included quite a bit of input from this thread
along with the original design. That said, some of the points you've raised
are new and haven't been raised by other people, so I'll give you a full
reply.

1) That's still something I have to do, but more time consuming than other
parts of creating the proposal and I've put it off until I have a while to
do it. The original implementation was done without too much consideration
of prior attempts, because I was more interested in getting something
working than I was in learning from the past.

2) I agree. The new syntax I've proposed is to add one or more class-level
methods to the django.db.models API. The new syntax for "inline" fields is:

class MyModel(models.Model):
x = models.IntegerField()
y = models.IntegerField()
point = models.constrain(x, y, unique=True)

The reason for not using the CompositeField constructor is that a lot of
the options which make sense for standalone field constructors (and all the
other field classes) make absolutely no sense when you're mainly leveraging
composite fields to provide a table level constraint to two existing
fields. Also there are some things that are commonly done in practice (like
providing `verbose_name` as a positional arg) that make adding multiple
leading positional arguments difficult.

Note: It's a shame that we can't use py3's keyword only arguments here.

3) No, not all the field base API makes sense for composite fields -- in
fact most of it doesn't. This is a huge problem with fields in general, not
just composite fields -- there's just too much functionality on the basic
Field class that assumes a one-to-one mapping between a Field and a
database column and too many parameters accepted by field base that only
make sense for subsets of the available field types.

e.g.
- What does "blank" or "max_length" mean for an IntegerField?
- What does 'rel' mean for a NullBooleanField?
- What does get_col mean for a ManyToManyField?
etc.

To fix this would mean introducing significant backwards incompatible
changes and, while I would support such a change, I'm not sure it's a
discussion that affects the ability to implement composite fields.

4) The addition of the `isnull` field is mainly to support the ability to
query for whether a composite field is NULL. If we leave it implementation
defined as to how to interpret whether a specific configuration of
subfields maps to a python `None` value for the composite field, then it
becomes really difficult to define lookup transformations to query for null
values in the table.

But although it sounded like a good idea when I came up with it, it raises
more questions than it solves. I'm more than ready to go back to the
drawing board on that one.

5) Inheritance in the model API is complicated enough as it is without
adding inheritance of field types. Yes, it could be supported by "beating
the metaclass into submission", but it's not something that I think adds
any particular value. I did change the "no subclassing at all" restriction
to "only one class in an inheritance heirarchy can define subfields" when
writing the DEP though, because Ansarri wants the ability for users to
extend ForeignKey with their own python behaviour.

Thomas



On 7 March 2015 at 22:31, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Hello Thomas,
>
> It’s hard for me to digest a two-page-long email on a complex topic during
> the week. I bookmarked your first email when it came in. It’s Saturday,
> 11am,
> and I dedicated my first chunk of quality brain time to reading the entire
> thread. I’ll let you ponder what the effect of your second email was.
>
> In fact, if you propose something stupid, you’ll get a quick answer,
> because
> it’s easy to explain. Not getting answers right now means that your
> proposal
> is good enough to require consideration. Yes, that’s counter-intuitive.
>
> With that out of the way, here are my thoughts on your proposal. Some
> overlap
> with what other people have said in the thread.
>
> 1) I would find it reassuring if you described the landscape of past
> attempts,
> what good ideas you’re keeping, what bad ideas you’re throwing away — in
> short, if you convinced us that you’re building on top of past attempts.
> The
> main goal of this exercise is to guarantee that you don’t overlook a use
> case
> or an argument that made consensus in the past. (Don’t spend too much time
> on
> code; it my experience it’s harder to reuse and code matters much less than
> design.
>
> 2) The syntax for inline composite fields doesn't look very nice. Could you
> simplify it somehow

Re: Composite fields

[Aymeric Augustin]

Hello Thomas,

It’s hard for me to digest a two-page-long email on a complex topic during
the week. I bookmarked your first email when it came in. It’s Saturday, 11am,
and I dedicated my first chunk of quality brain time to reading the entire
thread. I’ll let you ponder what the effect of your second email was.

In fact, if you propose something stupid, you’ll get a quick answer, because
it’s easy to explain. Not getting answers right now means that your proposal
is good enough to require consideration. Yes, that’s counter-intuitive.

With that out of the way, here are my thoughts on your proposal. Some overlap
with what other people have said in the thread.

1) I would find it reassuring if you described the landscape of past attempts,
what good ideas you’re keeping, what bad ideas you’re throwing away — in
short, if you convinced us that you’re building on top of past attempts. The
main goal of this exercise is to guarantee that you don’t overlook a use case
or an argument that made consensus in the past. (Don’t spend too much time on
code; it my experience it’s harder to reuse and code matters much less than
design.

2) The syntax for inline composite fields doesn't look very nice. Could you
simplify it somehow? Anssi’s proposal is good. I assume that a composite field
could add the subfields to the model class if they aren't defined explicitly
and their names passed in arguments to the composite field.

3) Have you checked that all the Field APIs make sense for CompositeField?
It's quite obvious that value_from/to_dict are needed. It's less obvious that
everything else is still needed.

4) I'm wary of the extra 'isnull' column. Couldn't we required that, if the
composite field is NULL, at least one of the subfields is NULL? My idea is
that a nullable composite field would consider itself NULL if all nullable
subfields are NULL. Declaring a composite subfield nullable when all subfields
are non-nullable would be an error. In your MoneyField example, the amount
subfield should be nullable when the composite field is nullable.

5) I understand the first two restrictions. They required deeper refactorings
to be lifted. The reason for the third one is less clear. Is it just a matter
of beating the metaclass into cooperation?

Best,

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2130B21B-41FA-4928-BC9C-5F5BAADAD7E0%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Composite fields

[Thomas Stephenson]

Whoops -- would be useful to provide a link 
<https://github.com/django/deps/pull/12>

On Saturday, March 7, 2015 at 6:13:19 PM UTC+11, Thomas Stephenson wrote:
>
> PR created. Unfortunately that's all the work I'm going to do on this over 
> the weekend, so I'll come back and review comments on monday. Some of the 
> reST syntax is slightly off. I will fix that up monday -- mainly just 
> posting it so people have something to review, if they're that way inclined.
>
> Thomas
>
> On Friday, March 6, 2015 at 9:52:56 PM UTC+11, Thomas Stephenson wrote:
>>
>> Y'OK. I'll create DEP 191 for the composite fields which point to model 
>> fields defined on the model (using the syntax you provided above) and DEP 
>> 192 for composite fields which define their own subfields (using the model 
>> syntax). 
>>
>> I was getting confused trying to create a specification which covers all 
>> the implementation details (so far) of both cases anyway.
>>
>> On 6 March 2015 at 18:48, Anssi Kääriäinen  wrote:
>>
>>> I think we can separate composite field implementation into two steps, 
>>> first implement composite fields for user defined existing fields, then 
>>> make it possible for fields to auto-create its subfields.
>>>
>>> I'm not sure of the meta issue. It seems in some cases you'll want 
>>> access to the subfields, especially if foreign keys are going to be split 
>>> using this same idea.
>>>
>>> +1 for DEP, this is exactly the kind of issue where writing a DEP will 
>>> be usefull, and getting an acceptance for the proposal will make 
>>> implementation easier.
>>>
>>>   - Anssi
>>>
>>> On Friday, March 6, 2015, Thomas Stephenson  wrote:
>>>
>>>> I'll reiterate that I mispoke when I said that "subfields wouldn't be 
>>>> added to model._meta". They would be, it's just that the contributor would 
>>>> be the CompositeField, not the model. Subfields are just wrappers for 
>>>> other 
>>>> field types and need a reference to the composite field. The difference 
>>>> that I was trying to make when I said "would not be available on 
>>>> model._meta" is that they wouldn't be available via the public API without 
>>>> passing special flags. 
>>>>
>>>> I'm all for creating a DEP for the proposal, it seems like a better 
>>>> place for nutting out these issues than a mailing list thread. What's the 
>>>> current index into the deps? I'm guessing that it's 191.
>>>>
>>>> Thomas
>>>>
>>>> On 6 March 2015 at 03:13, Markus Holtermann  
>>>> wrote:
>>>>
>>>>> While I like your approach, Thomas, I should also note that I'm not an 
>>>>> expert on those parts of Django and haven't kept up with the Michal's 
>>>>> proposal.
>>>>>
>>>>> With respect to migrations I like Anssi's approach to have both 
>>>>> `author` and `author_id` in _meta.fields. It will probably simplify a 
>>>>> couple of things and we can get rid of some workarounds.
>>>>>
>>>>> Furthermore, I suggest that we take the time and proceed with a DEP (
>>>>> https://github.com/django/deps) to find common ground what the API 
>>>>> should look like before we rush a concrete implementation.
>>>>>
>>>>> /Markus
>>>>>
>>>>>
>>>>> On Thursday, March 5, 2015 at 4:07:09 PM UTC+1, Thomas Stephenson 
>>>>> wrote:
>>>>>>
>>>>>> Well, I was thinking that although the subfields would still be added 
>>>>>> to Model._meta.fields, you could
>>>>>>
>>>>>> - add an 'include_subfields=False' default argument to get_fields()
>>>>>> - only allow direct lookups of subfields via Model._meta.get_field(
>>>>>> composite).get_subfield(subfield) or (alternately, depending on 
>>>>>> people's tastes) Model._meta.get_field(composite__subfield)
>>>>>>
>>>>>> Thomas
>>>>>>
>>>>>> On Friday, March 6, 2015 at 1:58:26 AM UTC+11, Marc Tamlyn wrote:
>>>>>>>
>>>>>>> I think `Model._meta` may well always want all the concrete 
>>>>>>> underlying fields included somehow,

Re: Composite fields

[Thomas Stephenson]

PR created. Unfortunately that's all the work I'm going to do on this over 
the weekend, so I'll come back and review comments on monday. Some of the 
reST syntax is slightly off. I will fix that up monday -- mainly just 
posting it so people have something to review, if they're that way inclined.

Thomas

On Friday, March 6, 2015 at 9:52:56 PM UTC+11, Thomas Stephenson wrote:
>
> Y'OK. I'll create DEP 191 for the composite fields which point to model 
> fields defined on the model (using the syntax you provided above) and DEP 
> 192 for composite fields which define their own subfields (using the model 
> syntax). 
>
> I was getting confused trying to create a specification which covers all 
> the implementation details (so far) of both cases anyway.
>
> On 6 March 2015 at 18:48, Anssi Kääriäinen  wrote:
>
>> I think we can separate composite field implementation into two steps, 
>> first implement composite fields for user defined existing fields, then 
>> make it possible for fields to auto-create its subfields.
>>
>> I'm not sure of the meta issue. It seems in some cases you'll want access 
>> to the subfields, especially if foreign keys are going to be split using 
>> this same idea.
>>
>> +1 for DEP, this is exactly the kind of issue where writing a DEP will be 
>> usefull, and getting an acceptance for the proposal will make 
>> implementation easier.
>>
>>   - Anssi
>>
>> On Friday, March 6, 2015, Thomas Stephenson  wrote:
>>
>>> I'll reiterate that I mispoke when I said that "subfields wouldn't be 
>>> added to model._meta". They would be, it's just that the contributor would 
>>> be the CompositeField, not the model. Subfields are just wrappers for other 
>>> field types and need a reference to the composite field. The difference 
>>> that I was trying to make when I said "would not be available on 
>>> model._meta" is that they wouldn't be available via the public API without 
>>> passing special flags. 
>>>
>>> I'm all for creating a DEP for the proposal, it seems like a better 
>>> place for nutting out these issues than a mailing list thread. What's the 
>>> current index into the deps? I'm guessing that it's 191.
>>>
>>> Thomas
>>>
>>> On 6 March 2015 at 03:13, Markus Holtermann  
>>> wrote:
>>>
>>>> While I like your approach, Thomas, I should also note that I'm not an 
>>>> expert on those parts of Django and haven't kept up with the Michal's 
>>>> proposal.
>>>>
>>>> With respect to migrations I like Anssi's approach to have both 
>>>> `author` and `author_id` in _meta.fields. It will probably simplify a 
>>>> couple of things and we can get rid of some workarounds.
>>>>
>>>> Furthermore, I suggest that we take the time and proceed with a DEP (
>>>> https://github.com/django/deps) to find common ground what the API 
>>>> should look like before we rush a concrete implementation.
>>>>
>>>> /Markus
>>>>
>>>>
>>>> On Thursday, March 5, 2015 at 4:07:09 PM UTC+1, Thomas Stephenson wrote:
>>>>>
>>>>> Well, I was thinking that although the subfields would still be added 
>>>>> to Model._meta.fields, you could
>>>>>
>>>>> - add an 'include_subfields=False' default argument to get_fields()
>>>>> - only allow direct lookups of subfields via Model._meta.get_field(
>>>>> composite).get_subfield(subfield) or (alternately, depending on 
>>>>> people's tastes) Model._meta.get_field(composite__subfield)
>>>>>
>>>>> Thomas
>>>>>
>>>>> On Friday, March 6, 2015 at 1:58:26 AM UTC+11, Marc Tamlyn wrote:
>>>>>>
>>>>>> I think `Model._meta` may well always want all the concrete 
>>>>>> underlying fields included somehow, especially for migrations work. A 
>>>>>> possibility for your external `MoneyField` could be to add other fields 
>>>>>> using `contribute_to_class`, though this may be a bad idea...
>>>>>>
>>>>>> On 5 March 2015 at 14:47, Thomas Stephenson  wrote:
>>>>>>
>>>>>>>  > Turns out this was a bad idea, after around 1700 lines changed 
>>>>>>> everything was broken and there were multiple hard to debug failures.
>>>>

Re: RE Composite fields-/ Multi Primary / Foreign keys

[Aron Podrigal]

yeah, turns out I got busy with some other work and did not have time to work 
on this. Fortunately, Thomas has done a lot of great work yet. I'll just follow 
up on that thread.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e6c3eea7-79fd-4a1e-8cb4-ecad9bde042d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Composite fields

[Thomas Stephenson]

Y'OK. I'll create DEP 191 for the composite fields which point to model
fields defined on the model (using the syntax you provided above) and DEP
192 for composite fields which define their own subfields (using the model
syntax).

I was getting confused trying to create a specification which covers all
the implementation details (so far) of both cases anyway.

On 6 March 2015 at 18:48, Anssi Kääriäinen  wrote:

> I think we can separate composite field implementation into two steps,
> first implement composite fields for user defined existing fields, then
> make it possible for fields to auto-create its subfields.
>
> I'm not sure of the meta issue. It seems in some cases you'll want access
> to the subfields, especially if foreign keys are going to be split using
> this same idea.
>
> +1 for DEP, this is exactly the kind of issue where writing a DEP will be
> usefull, and getting an acceptance for the proposal will make
> implementation easier.
>
>   - Anssi
>
> On Friday, March 6, 2015, Thomas Stephenson  wrote:
>
>> I'll reiterate that I mispoke when I said that "subfields wouldn't be
>> added to model._meta". They would be, it's just that the contributor would
>> be the CompositeField, not the model. Subfields are just wrappers for other
>> field types and need a reference to the composite field. The difference
>> that I was trying to make when I said "would not be available on
>> model._meta" is that they wouldn't be available via the public API without
>> passing special flags.
>>
>> I'm all for creating a DEP for the proposal, it seems like a better place
>> for nutting out these issues than a mailing list thread. What's the current
>> index into the deps? I'm guessing that it's 191.
>>
>> Thomas
>>
>> On 6 March 2015 at 03:13, Markus Holtermann 
>> wrote:
>>
>>> While I like your approach, Thomas, I should also note that I'm not an
>>> expert on those parts of Django and haven't kept up with the Michal's
>>> proposal.
>>>
>>> With respect to migrations I like Anssi's approach to have both `author`
>>> and `author_id` in _meta.fields. It will probably simplify a couple of
>>> things and we can get rid of some workarounds.
>>>
>>> Furthermore, I suggest that we take the time and proceed with a DEP (
>>> https://github.com/django/deps) to find common ground what the API
>>> should look like before we rush a concrete implementation.
>>>
>>> /Markus
>>>
>>>
>>> On Thursday, March 5, 2015 at 4:07:09 PM UTC+1, Thomas Stephenson wrote:
>>>>
>>>> Well, I was thinking that although the subfields would still be added
>>>> to Model._meta.fields, you could
>>>>
>>>> - add an 'include_subfields=False' default argument to get_fields()
>>>> - only allow direct lookups of subfields via Model._meta.get_field(
>>>> composite).get_subfield(subfield) or (alternately, depending on
>>>> people's tastes) Model._meta.get_field(composite__subfield)
>>>>
>>>> Thomas
>>>>
>>>> On Friday, March 6, 2015 at 1:58:26 AM UTC+11, Marc Tamlyn wrote:
>>>>>
>>>>> I think `Model._meta` may well always want all the concrete underlying
>>>>> fields included somehow, especially for migrations work. A possibility for
>>>>> your external `MoneyField` could be to add other fields using
>>>>> `contribute_to_class`, though this may be a bad idea...
>>>>>
>>>>> On 5 March 2015 at 14:47, Thomas Stephenson  wrote:
>>>>>
>>>>>>  > Turns out this was a bad idea, after around 1700 lines changed
>>>>>> everything was broken and there were multiple hard to debug failures.
>>>>>>
>>>>>> Yeah, been in this situation too many times to count.
>>>>>>
>>>>>> > Split ForeignKey to a concrete field instance, and a related field
>>>>>> instance. After this all related fields are purely virtual. This means 
>>>>>> that
>>>>>> author = models.ForeignKey(Author) will automatically generate a 
>>>>>> author_id
>>>>>> = IntegerField() on the model. Unfortunately this also means model._meta
>>>>>> will now contains two fields for each foreign key defined on the model
>>>>>> instead of just one.
>>>>>>
>>>>>> The foreign_key_i

Re: Composite fields

[Anssi Kääriäinen]

I think we can separate composite field implementation into two steps,
first implement composite fields for user defined existing fields, then
make it possible for fields to auto-create its subfields.

I'm not sure of the meta issue. It seems in some cases you'll want access
to the subfields, especially if foreign keys are going to be split using
this same idea.

+1 for DEP, this is exactly the kind of issue where writing a DEP will be
usefull, and getting an acceptance for the proposal will make
implementation easier.

  - Anssi

On Friday, March 6, 2015, Thomas Stephenson  wrote:

> I'll reiterate that I mispoke when I said that "subfields wouldn't be
> added to model._meta". They would be, it's just that the contributor would
> be the CompositeField, not the model. Subfields are just wrappers for other
> field types and need a reference to the composite field. The difference
> that I was trying to make when I said "would not be available on
> model._meta" is that they wouldn't be available via the public API without
> passing special flags.
>
> I'm all for creating a DEP for the proposal, it seems like a better place
> for nutting out these issues than a mailing list thread. What's the current
> index into the deps? I'm guessing that it's 191.
>
> Thomas
>
> On 6 March 2015 at 03:13, Markus Holtermann  > wrote:
>
>> While I like your approach, Thomas, I should also note that I'm not an
>> expert on those parts of Django and haven't kept up with the Michal's
>> proposal.
>>
>> With respect to migrations I like Anssi's approach to have both `author`
>> and `author_id` in _meta.fields. It will probably simplify a couple of
>> things and we can get rid of some workarounds.
>>
>> Furthermore, I suggest that we take the time and proceed with a DEP (
>> https://github.com/django/deps) to find common ground what the API
>> should look like before we rush a concrete implementation.
>>
>> /Markus
>>
>>
>> On Thursday, March 5, 2015 at 4:07:09 PM UTC+1, Thomas Stephenson wrote:
>>>
>>> Well, I was thinking that although the subfields would still be added to
>>> Model._meta.fields, you could
>>>
>>> - add an 'include_subfields=False' default argument to get_fields()
>>> - only allow direct lookups of subfields via Model._meta.get_field(
>>> composite).get_subfield(subfield) or (alternately, depending on
>>> people's tastes) Model._meta.get_field(composite__subfield)
>>>
>>> Thomas
>>>
>>> On Friday, March 6, 2015 at 1:58:26 AM UTC+11, Marc Tamlyn wrote:
>>>>
>>>> I think `Model._meta` may well always want all the concrete underlying
>>>> fields included somehow, especially for migrations work. A possibility for
>>>> your external `MoneyField` could be to add other fields using
>>>> `contribute_to_class`, though this may be a bad idea...
>>>>
>>>> On 5 March 2015 at 14:47, Thomas Stephenson  wrote:
>>>>
>>>>>  > Turns out this was a bad idea, after around 1700 lines changed
>>>>> everything was broken and there were multiple hard to debug failures.
>>>>>
>>>>> Yeah, been in this situation too many times to count.
>>>>>
>>>>> > Split ForeignKey to a concrete field instance, and a related field
>>>>> instance. After this all related fields are purely virtual. This means 
>>>>> that
>>>>> author = models.ForeignKey(Author) will automatically generate a author_id
>>>>> = IntegerField() on the model. Unfortunately this also means model._meta
>>>>> will now contains two fields for each foreign key defined on the model
>>>>> instead of just one.
>>>>>
>>>>> The foreign_key_id field could be a subfield of foreign key (then
>>>>> model._meta would not contain two entries for the foreign key).
>>>>> Unfortunately that would open the door to composite field inheritance, but
>>>>> it could be handled like enum inheritance is handled in python -- you can
>>>>> subclass as much as you want, but you can't declare additional fields on
>>>>> subclasses.
>>>>>
>>>>> If that happened, then composite foreign keys would be a lot easier
>>>>> (but still work that can be deferred).
>>>>>
>>>>> > Michal Petrucha did a lot of work to add composite fields to
>>>>> Django. The syntax he had was:
>>>&g

Re: Composite fields

[Thomas Stephenson]

I'll reiterate that I mispoke when I said that "subfields wouldn't be added
to model._meta". They would be, it's just that the contributor would be the
CompositeField, not the model. Subfields are just wrappers for other field
types and need a reference to the composite field. The difference that I
was trying to make when I said "would not be available on model._meta" is
that they wouldn't be available via the public API without passing special
flags.

I'm all for creating a DEP for the proposal, it seems like a better place
for nutting out these issues than a mailing list thread. What's the current
index into the deps? I'm guessing that it's 191.

Thomas

On 6 March 2015 at 03:13, Markus Holtermann 
wrote:

> While I like your approach, Thomas, I should also note that I'm not an
> expert on those parts of Django and haven't kept up with the Michal's
> proposal.
>
> With respect to migrations I like Anssi's approach to have both `author`
> and `author_id` in _meta.fields. It will probably simplify a couple of
> things and we can get rid of some workarounds.
>
> Furthermore, I suggest that we take the time and proceed with a DEP (
> https://github.com/django/deps) to find common ground what the API should
> look like before we rush a concrete implementation.
>
> /Markus
>
>
> On Thursday, March 5, 2015 at 4:07:09 PM UTC+1, Thomas Stephenson wrote:
>>
>> Well, I was thinking that although the subfields would still be added to
>> Model._meta.fields, you could
>>
>> - add an 'include_subfields=False' default argument to get_fields()
>> - only allow direct lookups of subfields via Model._meta.get_field(
>> composite).get_subfield(subfield) or (alternately, depending on people's
>> tastes) Model._meta.get_field(composite__subfield)
>>
>> Thomas
>>
>> On Friday, March 6, 2015 at 1:58:26 AM UTC+11, Marc Tamlyn wrote:
>>>
>>> I think `Model._meta` may well always want all the concrete underlying
>>> fields included somehow, especially for migrations work. A possibility for
>>> your external `MoneyField` could be to add other fields using
>>> `contribute_to_class`, though this may be a bad idea...
>>>
>>> On 5 March 2015 at 14:47, Thomas Stephenson  wrote:
>>>
>>>>  > Turns out this was a bad idea, after around 1700 lines changed
>>>> everything was broken and there were multiple hard to debug failures.
>>>>
>>>> Yeah, been in this situation too many times to count.
>>>>
>>>> > Split ForeignKey to a concrete field instance, and a related field
>>>> instance. After this all related fields are purely virtual. This means that
>>>> author = models.ForeignKey(Author) will automatically generate a author_id
>>>> = IntegerField() on the model. Unfortunately this also means model._meta
>>>> will now contains two fields for each foreign key defined on the model
>>>> instead of just one.
>>>>
>>>> The foreign_key_id field could be a subfield of foreign key (then
>>>> model._meta would not contain two entries for the foreign key).
>>>> Unfortunately that would open the door to composite field inheritance, but
>>>> it could be handled like enum inheritance is handled in python -- you can
>>>> subclass as much as you want, but you can't declare additional fields on
>>>> subclasses.
>>>>
>>>> If that happened, then composite foreign keys would be a lot easier
>>>> (but still work that can be deferred).
>>>>
>>>> > Michal Petrucha did a lot of work to add composite fields to Django.
>>>> The syntax he had was:
>>>>
>>>> I'm not exactly a fan of that syntax. It works for the unique_together
>>>> and index_together use cases (and the primary key use case), but it puts
>>>> all the subfields on model._meta and doesn't allow you encapsulate the
>>>> behaviour of composite fields outside the model definition. So you can't,
>>>> for example, define a reusable `MoneyField` that represents two columns in
>>>> the target model.
>>>>
>>>>
>>>> > The first part in the composite fields work should be making the
>>>> point field example to work. This will mean supporting
>>>> .filter(point__in=((1, 2), (2, 3))), and support for .values('point'). Both
>>>> of these will be surprisingly complex to do correctly. In addition there
>>>> will likely be a lot of work to do in 

Re: Composite fields

[Markus Holtermann]

While I like your approach, Thomas, I should also note that I'm not an 
expert on those parts of Django and haven't kept up with the Michal's 
proposal.

With respect to migrations I like Anssi's approach to have both `author` 
and `author_id` in _meta.fields. It will probably simplify a couple of 
things and we can get rid of some workarounds.

Furthermore, I suggest that we take the time and proceed with a DEP 
(https://github.com/django/deps) to find common ground what the API should 
look like before we rush a concrete implementation.

/Markus

On Thursday, March 5, 2015 at 4:07:09 PM UTC+1, Thomas Stephenson wrote:
>
> Well, I was thinking that although the subfields would still be added to 
> Model._meta.fields, you could
>
> - add an 'include_subfields=False' default argument to get_fields()
> - only allow direct lookups of subfields via 
> Model._meta.get_field(composite).get_subfield(subfield) or (alternately, 
> depending on people's tastes) Model._meta.get_field(composite__subfield)
>
> Thomas
>
> On Friday, March 6, 2015 at 1:58:26 AM UTC+11, Marc Tamlyn wrote:
>>
>> I think `Model._meta` may well always want all the concrete underlying 
>> fields included somehow, especially for migrations work. A possibility for 
>> your external `MoneyField` could be to add other fields using 
>> `contribute_to_class`, though this may be a bad idea...
>>
>> On 5 March 2015 at 14:47, Thomas Stephenson  wrote:
>>
>>>  > Turns out this was a bad idea, after around 1700 lines changed 
>>> everything was broken and there were multiple hard to debug failures.
>>>
>>> Yeah, been in this situation too many times to count. 
>>>
>>> > Split ForeignKey to a concrete field instance, and a related field 
>>> instance. After this all related fields are purely virtual. This means that 
>>> author = models.ForeignKey(Author) will automatically generate a author_id 
>>> = IntegerField() on the model. Unfortunately this also means model._meta 
>>> will now contains two fields for each foreign key defined on the model 
>>> instead of just one.
>>>
>>> The foreign_key_id field could be a subfield of foreign key (then 
>>> model._meta would not contain two entries for the foreign key). 
>>> Unfortunately that would open the door to composite field inheritance, but 
>>> it could be handled like enum inheritance is handled in python -- you can 
>>> subclass as much as you want, but you can't declare additional fields on 
>>> subclasses.
>>>
>>> If that happened, then composite foreign keys would be a lot easier (but 
>>> still work that can be deferred). 
>>>
>>> > Michal Petrucha did a lot of work to add composite fields to Django. 
>>> The syntax he had was:
>>>
>>> I'm not exactly a fan of that syntax. It works for the unique_together 
>>> and index_together use cases (and the primary key use case), but it puts 
>>> all the subfields on model._meta and doesn't allow you encapsulate the 
>>> behaviour of composite fields outside the model definition. So you can't, 
>>> for example, define a reusable `MoneyField` that represents two columns in 
>>> the target model.
>>>
>>>
>>> > The first part in the composite fields work should be making the 
>>> point field example to work. This will mean supporting 
>>> .filter(point__in=((1, 2), (2, 3))), and support for .values('point'). Both 
>>> of these will be surprisingly complex to do correctly. In addition there 
>>> will likely be a lot of work to do in other parts of Django, too (for 
>>> example in migrations), so implementing just "simple" composite fields will 
>>> be a lot of work.
>>>
>>> Well, I've already got that working (well, I've got point__exact working 
>>> and I can add point__in easily enough, it's just a matter of adding the 
>>> relevant lookup transformations to get_lookup_transform. There were some 
>>> comments surrounding that function which suggest it needs a refactoring, 
>>> but I don't think it does.
>>>
>>> Thomas
>>>
>>> On 5 March 2015 at 22:30, Anssi Kääriäinen  wrote:
>>>
>>>> I've started doing some refactorings to the fields/related.py. The 
>>>> ultimate goal is to have field.rel (ForeignObjectRel) instances to be 
>>>> Field 
>>>> subclasses.
>>>>
>>>> I first went ahead and did exactly this with the idea of changing 
>>>>

Re: Composite fields

[Thomas Stephenson]

Well, I was thinking that although the subfields would still be added to 
Model._meta.fields, you could

- add an 'include_subfields=False' default argument to get_fields()
- only allow direct lookups of subfields via 
Model._meta.get_field(composite).get_subfield(subfield) or (alternately, 
depending on people's tastes) Model._meta.get_field(composite__subfield)

Thomas

On Friday, March 6, 2015 at 1:58:26 AM UTC+11, Marc Tamlyn wrote:
>
> I think `Model._meta` may well always want all the concrete underlying 
> fields included somehow, especially for migrations work. A possibility for 
> your external `MoneyField` could be to add other fields using 
> `contribute_to_class`, though this may be a bad idea...
>
> On 5 March 2015 at 14:47, Thomas Stephenson  > wrote:
>
>>  > Turns out this was a bad idea, after around 1700 lines changed 
>> everything was broken and there were multiple hard to debug failures.
>>
>> Yeah, been in this situation too many times to count. 
>>
>> > Split ForeignKey to a concrete field instance, and a related field 
>> instance. After this all related fields are purely virtual. This means that 
>> author = models.ForeignKey(Author) will automatically generate a author_id 
>> = IntegerField() on the model. Unfortunately this also means model._meta 
>> will now contains two fields for each foreign key defined on the model 
>> instead of just one.
>>
>> The foreign_key_id field could be a subfield of foreign key (then 
>> model._meta would not contain two entries for the foreign key). 
>> Unfortunately that would open the door to composite field inheritance, but 
>> it could be handled like enum inheritance is handled in python -- you can 
>> subclass as much as you want, but you can't declare additional fields on 
>> subclasses.
>>
>> If that happened, then composite foreign keys would be a lot easier (but 
>> still work that can be deferred). 
>>
>> > Michal Petrucha did a lot of work to add composite fields to Django. 
>> The syntax he had was:
>>
>> I'm not exactly a fan of that syntax. It works for the unique_together 
>> and index_together use cases (and the primary key use case), but it puts 
>> all the subfields on model._meta and doesn't allow you encapsulate the 
>> behaviour of composite fields outside the model definition. So you can't, 
>> for example, define a reusable `MoneyField` that represents two columns in 
>> the target model.
>>
>>
>> > The first part in the composite fields work should be making the point 
>> field example to work. This will mean supporting .filter(point__in=((1, 2), 
>> (2, 3))), and support for .values('point'). Both of these will be 
>> surprisingly complex to do correctly. In addition there will likely be a 
>> lot of work to do in other parts of Django, too (for example in 
>> migrations), so implementing just "simple" composite fields will be a lot 
>> of work.
>>
>> Well, I've already got that working (well, I've got point__exact working 
>> and I can add point__in easily enough, it's just a matter of adding the 
>> relevant lookup transformations to get_lookup_transform. There were some 
>> comments surrounding that function which suggest it needs a refactoring, 
>> but I don't think it does.
>>
>> Thomas
>>
>> On 5 March 2015 at 22:30, Anssi Kääriäinen > > wrote:
>>
>>> I've started doing some refactorings to the fields/related.py. The 
>>> ultimate goal is to have field.rel (ForeignObjectRel) instances to be Field 
>>> subclasses.
>>>
>>> I first went ahead and did exactly this with the idea of changing 
>>> everything in one go. Turns out this was a bad idea, after around 1700 
>>> lines changed everything was broken and there were multiple hard to debug 
>>> failures.
>>>
>>> I did a fresh start, and my plan is now to do the following:
>>>   - First, make ForeignObjectRel to act like a Field instance (part of 
>>> this is done in https://github.com/django/django/pull/4241)
>>>   - Make ForeignObjectRel Field subclass. This will likely rename the 
>>> classes to something like ReverseForeignKey, ReverseManyToMany and so on.
>>>   - Finally, add the new reverse field instances directly to the remote 
>>> model's _meta
>>>
>>> This is just clean-up in the fields/related.py. The composite fields 
>>> work doesn't need to rely on this. To get to a state where we have 
>>> composite primary keys and composite joins we should:

Re: Composite fields

[Marc Tamlyn]

I think `Model._meta` may well always want all the concrete underlying
fields included somehow, especially for migrations work. A possibility for
your external `MoneyField` could be to add other fields using
`contribute_to_class`, though this may be a bad idea...

On 5 March 2015 at 14:47, Thomas Stephenson  wrote:

>  > Turns out this was a bad idea, after around 1700 lines changed
> everything was broken and there were multiple hard to debug failures.
>
> Yeah, been in this situation too many times to count.
>
> > Split ForeignKey to a concrete field instance, and a related field
> instance. After this all related fields are purely virtual. This means that
> author = models.ForeignKey(Author) will automatically generate a author_id
> = IntegerField() on the model. Unfortunately this also means model._meta
> will now contains two fields for each foreign key defined on the model
> instead of just one.
>
> The foreign_key_id field could be a subfield of foreign key (then
> model._meta would not contain two entries for the foreign key).
> Unfortunately that would open the door to composite field inheritance, but
> it could be handled like enum inheritance is handled in python -- you can
> subclass as much as you want, but you can't declare additional fields on
> subclasses.
>
> If that happened, then composite foreign keys would be a lot easier (but
> still work that can be deferred).
>
> > Michal Petrucha did a lot of work to add composite fields to Django.
> The syntax he had was:
>
> I'm not exactly a fan of that syntax. It works for the unique_together and
> index_together use cases (and the primary key use case), but it puts all
> the subfields on model._meta and doesn't allow you encapsulate the
> behaviour of composite fields outside the model definition. So you can't,
> for example, define a reusable `MoneyField` that represents two columns in
> the target model.
>
>
> > The first part in the composite fields work should be making the point
> field example to work. This will mean supporting .filter(point__in=((1, 2),
> (2, 3))), and support for .values('point'). Both of these will be
> surprisingly complex to do correctly. In addition there will likely be a
> lot of work to do in other parts of Django, too (for example in
> migrations), so implementing just "simple" composite fields will be a lot
> of work.
>
> Well, I've already got that working (well, I've got point__exact working
> and I can add point__in easily enough, it's just a matter of adding the
> relevant lookup transformations to get_lookup_transform. There were some
> comments surrounding that function which suggest it needs a refactoring,
> but I don't think it does.
>
> Thomas
>
> On 5 March 2015 at 22:30, Anssi Kääriäinen  wrote:
>
>> I've started doing some refactorings to the fields/related.py. The
>> ultimate goal is to have field.rel (ForeignObjectRel) instances to be Field
>> subclasses.
>>
>> I first went ahead and did exactly this with the idea of changing
>> everything in one go. Turns out this was a bad idea, after around 1700
>> lines changed everything was broken and there were multiple hard to debug
>> failures.
>>
>> I did a fresh start, and my plan is now to do the following:
>>   - First, make ForeignObjectRel to act like a Field instance (part of
>> this is done in https://github.com/django/django/pull/4241)
>>   - Make ForeignObjectRel Field subclass. This will likely rename the
>> classes to something like ReverseForeignKey, ReverseManyToMany and so on.
>>   - Finally, add the new reverse field instances directly to the remote
>> model's _meta
>>
>> This is just clean-up in the fields/related.py. The composite fields work
>> doesn't need to rely on this. To get to a state where we have composite
>> primary keys and composite joins we should:
>>   - Split ForeignKey to a concrete field instance, and a related field
>> instance. After this all related fields are purely virtual. This means that
>> author = models.ForeignKey(Author) will automatically generate a author_id
>> = IntegerField() on the model. Unfortunately this also means model._meta
>> will now contains two fields for each foreign key defined on the model
>> instead of just one.
>>   - Add composite fields (but not yet composite foreign keys or primary
>> keys)
>>   - Add composite primary keys
>>   - Add composite foreign keys
>>
>> Addition of composite fields can be done at the same time with changes to
>> fields/related.py, so it should be possible to start working on composite
>> fields right away.
>>
>

Re: Composite fields

[Thomas Stephenson]

 > Turns out this was a bad idea, after around 1700 lines changed
everything was broken and there were multiple hard to debug failures.

Yeah, been in this situation too many times to count.

> Split ForeignKey to a concrete field instance, and a related field
instance. After this all related fields are purely virtual. This means that
author = models.ForeignKey(Author) will automatically generate a author_id
= IntegerField() on the model. Unfortunately this also means model._meta
will now contains two fields for each foreign key defined on the model
instead of just one.

The foreign_key_id field could be a subfield of foreign key (then
model._meta would not contain two entries for the foreign key).
Unfortunately that would open the door to composite field inheritance, but
it could be handled like enum inheritance is handled in python -- you can
subclass as much as you want, but you can't declare additional fields on
subclasses.

If that happened, then composite foreign keys would be a lot easier (but
still work that can be deferred).

> Michal Petrucha did a lot of work to add composite fields to Django. The
syntax he had was:

I'm not exactly a fan of that syntax. It works for the unique_together and
index_together use cases (and the primary key use case), but it puts all
the subfields on model._meta and doesn't allow you encapsulate the
behaviour of composite fields outside the model definition. So you can't,
for example, define a reusable `MoneyField` that represents two columns in
the target model.


> The first part in the composite fields work should be making the point
field example to work. This will mean supporting .filter(point__in=((1, 2),
(2, 3))), and support for .values('point'). Both of these will be
surprisingly complex to do correctly. In addition there will likely be a
lot of work to do in other parts of Django, too (for example in
migrations), so implementing just "simple" composite fields will be a lot
of work.

Well, I've already got that working (well, I've got point__exact working
and I can add point__in easily enough, it's just a matter of adding the
relevant lookup transformations to get_lookup_transform. There were some
comments surrounding that function which suggest it needs a refactoring,
but I don't think it does.

Thomas

On 5 March 2015 at 22:30, Anssi Kääriäinen  wrote:

> I've started doing some refactorings to the fields/related.py. The
> ultimate goal is to have field.rel (ForeignObjectRel) instances to be Field
> subclasses.
>
> I first went ahead and did exactly this with the idea of changing
> everything in one go. Turns out this was a bad idea, after around 1700
> lines changed everything was broken and there were multiple hard to debug
> failures.
>
> I did a fresh start, and my plan is now to do the following:
>   - First, make ForeignObjectRel to act like a Field instance (part of
> this is done in https://github.com/django/django/pull/4241)
>   - Make ForeignObjectRel Field subclass. This will likely rename the
> classes to something like ReverseForeignKey, ReverseManyToMany and so on.
>   - Finally, add the new reverse field instances directly to the remote
> model's _meta
>
> This is just clean-up in the fields/related.py. The composite fields work
> doesn't need to rely on this. To get to a state where we have composite
> primary keys and composite joins we should:
>   - Split ForeignKey to a concrete field instance, and a related field
> instance. After this all related fields are purely virtual. This means that
> author = models.ForeignKey(Author) will automatically generate a author_id
> = IntegerField() on the model. Unfortunately this also means model._meta
> will now contains two fields for each foreign key defined on the model
> instead of just one.
>   - Add composite fields (but not yet composite foreign keys or primary
> keys)
>   - Add composite primary keys
>   - Add composite foreign keys
>
> Addition of composite fields can be done at the same time with changes to
> fields/related.py, so it should be possible to start working on composite
> fields right away.
>
> Michal Petrucha did a lot of work to add composite fields to Django. The
> syntax he had was:
>
> class MyModel(models.Model):
> x = models.IntegerField()
> y = models.IntegerField()
> point = models.CompositeField(x, y)
>
> I think we should stick to that.
>
> It is essential that we don't try to do too much in one go. Even small
> changes tend to be hard to do. Trying to do all this in one go will result
> in a patch that will be nearly impossible to review, and which will
> conflict badly with other changes once finished.
>
> If you want to work on the composite fields part of this, I suggest you
> should take a close look at

Re: RE Composite fields-/ Multi Primary / Foreign keys

[Asif Saifuddin]

Hi Aron,

this discussion thread might interest you to collaborate

https://groups.google.com/forum/?utm_source=digest&utm_medium=email#!topic/django-developers/MZUcOE6-7GY

On Tuesday, February 24, 2015 at 12:14:19 PM UTC+6, Aron Podrigal wrote:
>
> Hi, I just came across a project that requires  this functionality of 
> multi primary/foreign keys. So I'm bumping up this thread 
> again.
>  
> There were a lot of changes in django and the db api hasre-factored  since 
> this topic was discussed. It might be a time for another start on this 
> project. I'm willing to work on this as long it is ok with the core 
> developers. I didn't have much time yet to spend on this. I'm still reading 
> through the other earlier threads on possible implementations.
> I will post my proposal for this over the weekend. In the meantime I would 
> like to get some feedback from the community. 
>
> Aron.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/31b58c26-71be-4075-a4dc-e1aca33bcc66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Composite fields

[Anssi Kääriäinen]

I've started doing some refactorings to the fields/related.py. The ultimate 
goal is to have field.rel (ForeignObjectRel) instances to be Field 
subclasses.

I first went ahead and did exactly this with the idea of changing 
everything in one go. Turns out this was a bad idea, after around 1700 
lines changed everything was broken and there were multiple hard to debug 
failures.

I did a fresh start, and my plan is now to do the following:
  - First, make ForeignObjectRel to act like a Field instance (part of this 
is done in https://github.com/django/django/pull/4241)
  - Make ForeignObjectRel Field subclass. This will likely rename the 
classes to something like ReverseForeignKey, ReverseManyToMany and so on.
  - Finally, add the new reverse field instances directly to the remote 
model's _meta

This is just clean-up in the fields/related.py. The composite fields work 
doesn't need to rely on this. To get to a state where we have composite 
primary keys and composite joins we should:
  - Split ForeignKey to a concrete field instance, and a related field 
instance. After this all related fields are purely virtual. This means that 
author = models.ForeignKey(Author) will automatically generate a author_id 
= IntegerField() on the model. Unfortunately this also means model._meta 
will now contains two fields for each foreign key defined on the model 
instead of just one.
  - Add composite fields (but not yet composite foreign keys or primary 
keys)
  - Add composite primary keys
  - Add composite foreign keys

Addition of composite fields can be done at the same time with changes to 
fields/related.py, so it should be possible to start working on composite 
fields right away.

Michal Petrucha did a lot of work to add composite fields to Django. The 
syntax he had was:

class MyModel(models.Model):
x = models.IntegerField()
y = models.IntegerField()
point = models.CompositeField(x, y)

I think we should stick to that.

It is essential that we don't try to do too much in one go. Even small 
changes tend to be hard to do. Trying to do all this in one go will result 
in a patch that will be nearly impossible to review, and which will 
conflict badly with other changes once finished.

If you want to work on the composite fields part of this, I suggest you 
should take a close look at what was done in Petrucha's GSoC work 
(https://github.com/konk/django/, unfortunately I don't recall which 
branch contains the latest code). The code got to a point where it was 
pretty much commit ready. The commit didn't happen because migrations were 
getting ready for merge at the same time, and committing Petrucha's work 
would have caused severe problems for migrations.

The first part in the composite fields work should be making the point 
field example to work. This will mean supporting .filter(point__in=((1, 2), 
(2, 3))), and support for .values('point'). Both of these will be 
surprisingly complex to do correctly. In addition there will likely be a 
lot of work to do in other parts of Django, too (for example in 
migrations), so implementing just "simple" composite fields will be a lot 
of work.

I like the MoneyField idea, but lets punt composite field definitions in 
that way for later.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/673dc9ed-da09-46d5-902e-311756613c81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Composite fields

[Thomas Stephenson]

Thanks for that info. Just to be clear, I'm not interested in SoC grants or 
mentoring, I'm mostly writing this stuff for my own concerns (aimed at 
improving our internal use of django by contributing changes we want to see 
upstream). I've fought hard with management to be in a position to devote a 
portion of my time to open source projects we use in-house, so I'm getting 
paid for whatever contributions I make (assuming I can make any).

We mainly use the ORM in conjunction with the django-rest-framework, we're 
not interested in admin, forms, templating etc. so that's where most of my 
changes will be directed.

I'd be happy to put this work aside to help with any refactor of the 
fields/related.py module then come back to this at a later point, but I 
think my implementation (so far) is reasonably sound and would integrate 
well with whatever work that is going on with Relations.

Thomas 

On Thursday, March 5, 2015 at 2:47:51 PM UTC+11, Loïc Bistuer wrote:
>
> Hi Thomas, 
>   
> fields/related.py is begging for a refactor of the existing relational 
> fields, and ideally we'd build composite fields on top of this refactor. 
>   
> The ORM has seen huge changes in the recent past: migrations, expressions, 
> lookups, and last but not least a *public* footprint in Model._meta, each 
> having both positive and negative (mostly backwards compatibility concerns) 
> implications when it comes to redesigning relations. All of these didn't 
> exist at the time of the previous two SoC attempts. 
>
> Taking a step back and looking at the problem from this new perspective 
> takes time, and in the last couple of months both Anssi and I have been 
> experimenting with a few ideas. Anssi has dedicated a lot of time to it in 
> the last few weeks, let's wait for his feedback. Best case scenario, his 
> design and implementation work and we don't have a SoC project anymore; if 
> he can't follow through maybe he'll be interested in mentoring yet another 
> SoC on this issue. 
>
> -- 
> Loïc 
>
> > On Mar 5, 2015, at 09:58, Thomas Stephenson  > wrote: 
> > 
> > Hey there, 
> > 
> > Yeah, I've looked through some (probably not all) of the previous 
> proposals to support composite fields. I was going to wait until some more 
> work had been done to split concrete fields from virtual fields (ticket 
> 16508), because I thought that would be an ORM change that would have 
> drastically simplified things, but in the end I've found that I was able to 
> keep the implementation contained without it. 
> > 
> > Sorry if I sounded impatient -- it was just a bit of humour to kickstart 
> the conversation, since my previous thread (about a couple of small changes 
> to the field API I'd like to make to support this proposal) had been 
> languishing for four days without a single response. 
> > 
> > Thomas 
> > 
> > On 5 March 2015 at 11:52, Curtis Maloney  > wrote: 
> > Have you reviewed all the existing works on trying to add composite 
> fields? 
> > 
> > I know several managed to reach a level of maturity that was almost 
> merge quality, and am sure I heard some of the recent ORM changes would 
> ease support. 
> > 
> > On 5 March 2015 at 11:42, Thomas Stephenson  > wrote: 
> > OK, no need for everyone to shout -- your message is heard loud and 
> > clear. I'll go and find something else to work on. 
> > 
> > 
> > Wow... impatient much?  Not everyone works in your timezone, or to your 
> schedule.  I, for one, was planning to take some time to review your 
> proposal carefully, instead of "first thing in the morning whilst still 
> having my coffee", as I feel it's a topic that deserves careful 
> consideration. 
> > 
> > Don't you think it's worth giving everyone a chance to read your email 
> before you give up and move on?  It's only been 11 hours.  Many of us were 
> asleep for most of that. 
> > 
> > -- 
> > Curtis 
> > 
> >   
> > On 5 March 2015 at 00:16, Thomas Stephenson  > wrote: 
> > > Considering the past two proposals I've made here, I doubt I'll get 
> more 
> > > than an echo chamber effect on this one. 
> > > 
> > > For the past week or so I've spent a bit of time on a feature I've 
> always 
> > > wanted to see land in django -- composite fields. The tickets have 
> been open 
> > > in the bug tracker for quite some time (and there's a few related 
> ones, such 
> > > as multi-column primary keys that can all be killed with the one 
> stone). 
> > > 
> > > The w

Re: Composite fields

[Loïc Bistuer]

Hi Thomas,
 
fields/related.py is begging for a refactor of the existing relational fields, 
and ideally we'd build composite fields on top of this refactor.
 
The ORM has seen huge changes in the recent past: migrations, expressions, 
lookups, and last but not least a *public* footprint in Model._meta, each 
having both positive and negative (mostly backwards compatibility concerns) 
implications when it comes to redesigning relations. All of these didn't exist 
at the time of the previous two SoC attempts.

Taking a step back and looking at the problem from this new perspective takes 
time, and in the last couple of months both Anssi and I have been experimenting 
with a few ideas. Anssi has dedicated a lot of time to it in the last few 
weeks, let's wait for his feedback. Best case scenario, his design and 
implementation work and we don't have a SoC project anymore; if he can't follow 
through maybe he'll be interested in mentoring yet another SoC on this issue.

-- 
Loïc

> On Mar 5, 2015, at 09:58, Thomas Stephenson  wrote:
> 
> Hey there, 
> 
> Yeah, I've looked through some (probably not all) of the previous proposals 
> to support composite fields. I was going to wait until some more work had 
> been done to split concrete fields from virtual fields (ticket 16508), 
> because I thought that would be an ORM change that would have drastically 
> simplified things, but in the end I've found that I was able to keep the 
> implementation contained without it.
> 
> Sorry if I sounded impatient -- it was just a bit of humour to kickstart the 
> conversation, since my previous thread (about a couple of small changes to 
> the field API I'd like to make to support this proposal) had been languishing 
> for four days without a single response.
> 
> Thomas
> 
> On 5 March 2015 at 11:52, Curtis Maloney  wrote:
> Have you reviewed all the existing works on trying to add composite fields?
> 
> I know several managed to reach a level of maturity that was almost merge 
> quality, and am sure I heard some of the recent ORM changes would ease 
> support.
> 
> On 5 March 2015 at 11:42, Thomas Stephenson  wrote:
> OK, no need for everyone to shout -- your message is heard loud and
> clear. I'll go and find something else to work on.
> 
> 
> Wow... impatient much?  Not everyone works in your timezone, or to your 
> schedule.  I, for one, was planning to take some time to review your proposal 
> carefully, instead of "first thing in the morning whilst still having my 
> coffee", as I feel it's a topic that deserves careful consideration.
> 
> Don't you think it's worth giving everyone a chance to read your email before 
> you give up and move on?  It's only been 11 hours.  Many of us were asleep 
> for most of that.
> 
> --
> Curtis
> 
>  
> On 5 March 2015 at 00:16, Thomas Stephenson  wrote:
> > Considering the past two proposals I've made here, I doubt I'll get more
> > than an echo chamber effect on this one.
> >
> > For the past week or so I've spent a bit of time on a feature I've always
> > wanted to see land in django -- composite fields. The tickets have been open
> > in the bug tracker for quite some time (and there's a few related ones, such
> > as multi-column primary keys that can all be killed with the one stone).
> >
> > The work is available on this branch of my fork of django for the moment --
> > I haven't opened up a PR yet because there's still some features that are
> > still to be implemented that will be explained below, but I want to give
> > everybody a chance to tell me where I can stick it before I spend *too* much
> > time on it.
> >
> > So, without further ado, the proposal.
> >
> >
> > Composite Fields - Implemented
> >
> > A composite field is an encapsulation of the functionality of a subset of
> > fields on a model. Composite fields can be defined in one of two ways:
> >
> > 1. Statically declared composite fields
> >
> > A statically declared composite field is defined in the same way a django
> > model is defined. There are two customisable transformation functions,
> > CompositeField.value_to_dict(self, value) and
> > CompositeField.value_from_dict(self, value) which can be used to associate
> > the field with a python object.
> >
> > All the serialization functions are implemented via the implementations of
> > the subfields.
> >
> > For example,
> >
> > class MoneyField(models.CompositeField):
> >currency_code = models.CharacterField(max_length=3)
> >amount = models.DecimalField(max_digits=16, decimal

Re: Composite fields

[Thomas Stephenson]

> Have you thought about how CompositeFields and SubFields will interact
with migrations?

I've given it some though, but at the moment I've just hacked out the one
point in the code where it makes a difference, marked it with a FIXME and
was going to return to it after I could store and query values etc. The
subfield is just a wrapper class for another field, so it just requires the
composite field to be reconstructed.

The biggest complication is that a composite field has either:
- An argument which contains fields which need to be reconstructed; or
- The possibility of providing extra parameters to it's subfields.

Thomas

On 5 March 2015 at 14:21, Josh Smeaton  wrote:

> With regards to the field API - that probably doesn't need an ML
> discussion. Open a ticket on Trac, submit a PR, and the people that will
> want to see it will review it. If there's confusion about how to proceed,
> then an ML discussion can be had, when there are actually interested
> parties involved. I imagine Russ and Piros (if he's still kicking around, I
> think he is) will have some comments once there's something concrete to
> review.
>
> 2 cents
>
> On Thursday, 5 March 2015 13:59:44 UTC+11, Thomas Stephenson wrote:
>>
>> Hey there,
>>
>> Yeah, I've looked through some (probably not all) of the previous
>> proposals to support composite fields. I was going to wait until some more
>> work had been done to split concrete fields from virtual fields (ticket
>> 16508 <https://code.djangoproject.com/ticket/16508>), because I thought
>> that would be an ORM change that would have drastically simplified things,
>> but in the end I've found that I was able to keep the implementation
>> contained without it.
>>
>> Sorry if I sounded impatient -- it was just a bit of humour to kickstart
>> the conversation, since my previous thread (about a couple of small changes
>> to the field API I'd like to make to support this proposal) had been
>> languishing for four days without a single response.
>>
>> Thomas
>>
>> On 5 March 2015 at 11:52, Curtis Maloney 
>> wrote:
>>
>>> Have you reviewed all the existing works on trying to add composite
>>> fields?
>>>
>>> I know several managed to reach a level of maturity that was almost
>>> merge quality, and am sure I heard some of the recent ORM changes would
>>> ease support.
>>>
>>> On 5 March 2015 at 11:42, Thomas Stephenson  wrote:
>>>
>>>> OK, no need for everyone to shout -- your message is heard loud and
>>>> clear. I'll go and find something else to work on.
>>>>
>>>>
>>> Wow... impatient much?  Not everyone works in your timezone, or to your
>>> schedule.  I, for one, was planning to take some time to review your
>>> proposal carefully, instead of "first thing in the morning whilst still
>>> having my coffee", as I feel it's a topic that deserves careful
>>> consideration.
>>>
>>> Don't you think it's worth giving everyone a chance to read your email
>>> before you give up and move on?  It's only been 11 hours.  Many of us were
>>> asleep for most of that.
>>>
>>> --
>>> Curtis
>>>
>>>
>>>
>>>> On 5 March 2015 at 00:16, Thomas Stephenson  wrote:
>>>> > Considering the past two proposals I've made here, I doubt I'll get
>>>> more
>>>> > than an echo chamber effect on this one.
>>>> >
>>>> > For the past week or so I've spent a bit of time on a feature I've
>>>> always
>>>> > wanted to see land in django -- composite fields. The tickets have
>>>> been open
>>>> > in the bug tracker for quite some time (and there's a few related
>>>> ones, such
>>>> > as multi-column primary keys that can all be killed with the one
>>>> stone).
>>>> >
>>>> > The work is available on this branch of my fork of django for the
>>>> moment --
>>>> > I haven't opened up a PR yet because there's still some features that
>>>> are
>>>> > still to be implemented that will be explained below, but I want to
>>>> give
>>>> > everybody a chance to tell me where I can stick it before I spend
>>>> *too* much
>>>> > time on it.
>>>> >
>>>> > So, without further ado, the proposal.
>>>> >
>>>> >
>>>>

Re: Composite fields

[Josh Smeaton]

With regards to the field API - that probably doesn't need an ML 
discussion. Open a ticket on Trac, submit a PR, and the people that will 
want to see it will review it. If there's confusion about how to proceed, 
then an ML discussion can be had, when there are actually interested 
parties involved. I imagine Russ and Piros (if he's still kicking around, I 
think he is) will have some comments once there's something concrete to 
review.

2 cents

On Thursday, 5 March 2015 13:59:44 UTC+11, Thomas Stephenson wrote:
>
> Hey there, 
>
> Yeah, I've looked through some (probably not all) of the previous 
> proposals to support composite fields. I was going to wait until some more 
> work had been done to split concrete fields from virtual fields (ticket 
> 16508 <https://code.djangoproject.com/ticket/16508>), because I thought 
> that would be an ORM change that would have drastically simplified things, 
> but in the end I've found that I was able to keep the implementation 
> contained without it.
>
> Sorry if I sounded impatient -- it was just a bit of humour to kickstart 
> the conversation, since my previous thread (about a couple of small changes 
> to the field API I'd like to make to support this proposal) had been 
> languishing for four days without a single response.
>
> Thomas
>
> On 5 March 2015 at 11:52, Curtis Maloney  > wrote:
>
>> Have you reviewed all the existing works on trying to add composite 
>> fields?
>>
>> I know several managed to reach a level of maturity that was almost merge 
>> quality, and am sure I heard some of the recent ORM changes would ease 
>> support.
>>
>> On 5 March 2015 at 11:42, Thomas Stephenson > > wrote:
>>
>>> OK, no need for everyone to shout -- your message is heard loud and
>>> clear. I'll go and find something else to work on.
>>>
>>>
>> Wow... impatient much?  Not everyone works in your timezone, or to your 
>> schedule.  I, for one, was planning to take some time to review your 
>> proposal carefully, instead of "first thing in the morning whilst still 
>> having my coffee", as I feel it's a topic that deserves careful 
>> consideration.
>>
>> Don't you think it's worth giving everyone a chance to read your email 
>> before you give up and move on?  It's only been 11 hours.  Many of us were 
>> asleep for most of that.
>>
>> --
>> Curtis
>>
>>  
>>
>>> On 5 March 2015 at 00:16, Thomas Stephenson >> > wrote:
>>> > Considering the past two proposals I've made here, I doubt I'll get 
>>> more
>>> > than an echo chamber effect on this one.
>>> >
>>> > For the past week or so I've spent a bit of time on a feature I've 
>>> always
>>> > wanted to see land in django -- composite fields. The tickets have 
>>> been open
>>> > in the bug tracker for quite some time (and there's a few related 
>>> ones, such
>>> > as multi-column primary keys that can all be killed with the one 
>>> stone).
>>> >
>>> > The work is available on this branch of my fork of django for the 
>>> moment --
>>> > I haven't opened up a PR yet because there's still some features that 
>>> are
>>> > still to be implemented that will be explained below, but I want to 
>>> give
>>> > everybody a chance to tell me where I can stick it before I spend 
>>> *too* much
>>> > time on it.
>>> >
>>> > So, without further ado, the proposal.
>>> >
>>> >
>>> > Composite Fields - Implemented
>>> >
>>> > A composite field is an encapsulation of the functionality of a subset 
>>> of
>>> > fields on a model. Composite fields can be defined in one of two ways:
>>> >
>>> > 1. Statically declared composite fields
>>> >
>>> > A statically declared composite field is defined in the same way a 
>>> django
>>> > model is defined. There are two customisable transformation functions,
>>> > CompositeField.value_to_dict(self, value) and
>>> > CompositeField.value_from_dict(self, value) which can be used to 
>>> associate
>>> > the field with a python object.
>>> >
>>> > All the serialization functions are implemented via the 
>>> implementations of
>>> > the subfields.
>>> >
>>> > For example,
>>> >
>>> > class Mo

Re: Composite fields

[Michael Manfre]

As the others have already stated, patience is required if you want
meaningful feedback.

Have you thought about how CompositeFields and SubFields will interact with
migrations?

Regards,
Michael Manfre

On Wed, Mar 4, 2015 at 8:16 AM, Thomas Stephenson  wrote:

> Considering the past two proposals I've made here, I doubt I'll get more
> than an echo chamber effect on this one.
>
> For the past week or so I've spent a bit of time on a feature I've always
> wanted to see land in django -- composite fields. The tickets have been
> open in the bug tracker for quite some time (and there's a few related
> ones, such as multi-column primary keys that can all be killed with the one
> stone).
>
> The work is available on this branch of my fork of django
> <https://github.com/ovangle/django/tree/composite_fields> for the moment
> -- I haven't opened up a PR yet because there's still some features that
> are still to be implemented that will be explained below, but I want to
> give everybody a chance to tell me where I can stick it before I spend
> *too* much time on it.
>
> So, without further ado, the proposal.
>
>
> Composite Fields - Implemented
>
> A composite field is an encapsulation of the functionality of a subset of
> fields on a model. Composite fields can be defined in one of two ways:
>
> 1. Statically declared composite fields
>
> A statically declared composite field is defined in the same way a django
> model is defined. There are two customisable transformation functions,
> CompositeField.value_to_dict(self, value) and
> CompositeField.value_from_dict(self, value) which can be used to associate
> the field with a python object.
>
> All the serialization functions are implemented via the implementations of
> the subfields.
>
> For example,
>
> class MoneyField(models.CompositeField):
>currency_code = models.CharacterField(max_length=3)
>amount = models.DecimalField(max_digits=16, decimal_digits=4)
>
>## Overriding __init__ can be used to pass field parameters to the
> subfields
>
>def value_from_dict(self, value):
>if value is None:
>   return None
>return Money(**value)
>
>    def value_to_dict(self, value):
>   if value is None:
>  return None
>   return {attr: getattr(value, attr) for attr in ('currency_code',
> 'amount')}
>
> 2. Inline composite fields.
>
> An inline composite field is declared at the field definition site on the
> body of a model, by providing the subfields as the 'fields' argument of the
> CompositeField constructor. There are no transformation parameters
> available to override when declaring a composite field in this fashion --
> the value of the field is always available as a python `dict` as an
> attribute on the MyModel
>
> class MyModel(models.Model):
> id = models.CompositeField(fields = [
>('a', models.IntegerField()),
>('b', models.CharField(max_length=30)
> ], primary_key=True)
>
> This method for defining composite fields has a few drawbacks, but can be
> useful if the only reason to add the composite field to the model was to
> implement a unique_together or index_together constraint *
>
> * Although it's still possible to do that directly on class Meta.
>
> 3. Null
>
> Setting the value of a multi-column field to NULL is different than
> setting any of the individual subfields to NULL. But there are cases (e.g.
> Money) where we would like to be able to set `null=True` on a composite
> field, but still retain 'NOT NULL' constraints on each of the subfield
> columns.
>
> To solve this problem, every table which implements a CompositeField will
> also add an implicit (semi-hidden) `isnull` subfield on the attribute,
> which keeps track of whether it is the value of the composite field that is
> null, or any of the particular subfields.
>
> 3. Querying.
>
> The syntax for querying over the subfields of a composite field will be
> familiar to anyone who has queried over a relationship attribute in django.
>
> model.objects.filter(price__currency_code='USD',
> price__amount__lt=Decimal('50.00')).all()
>
> In addition, composite fields can be queried via EXACT and IN lookups. It
> is possible to implement custom lookups for specific statically defined
> fields, but not recommended and not part of the official API.
>
> 4. Restrictions
>
> The following restrictions are currently imposed on the use of composite
> fields. None of these are restrictions that can't be worked around in
> future extensions, but they're restrictions which consider

Re: Composite fields

[Thomas Stephenson]

Hey there,

Yeah, I've looked through some (probably not all) of the previous proposals
to support composite fields. I was going to wait until some more work had
been done to split concrete fields from virtual fields (ticket 16508
<https://code.djangoproject.com/ticket/16508>), because I thought that
would be an ORM change that would have drastically simplified things, but
in the end I've found that I was able to keep the implementation contained
without it.

Sorry if I sounded impatient -- it was just a bit of humour to kickstart
the conversation, since my previous thread (about a couple of small changes
to the field API I'd like to make to support this proposal) had been
languishing for four days without a single response.

Thomas

On 5 March 2015 at 11:52, Curtis Maloney  wrote:

> Have you reviewed all the existing works on trying to add composite fields?
>
> I know several managed to reach a level of maturity that was almost merge
> quality, and am sure I heard some of the recent ORM changes would ease
> support.
>
> On 5 March 2015 at 11:42, Thomas Stephenson  wrote:
>
>> OK, no need for everyone to shout -- your message is heard loud and
>> clear. I'll go and find something else to work on.
>>
>>
> Wow... impatient much?  Not everyone works in your timezone, or to your
> schedule.  I, for one, was planning to take some time to review your
> proposal carefully, instead of "first thing in the morning whilst still
> having my coffee", as I feel it's a topic that deserves careful
> consideration.
>
> Don't you think it's worth giving everyone a chance to read your email
> before you give up and move on?  It's only been 11 hours.  Many of us were
> asleep for most of that.
>
> --
> Curtis
>
>
>
>> On 5 March 2015 at 00:16, Thomas Stephenson  wrote:
>> > Considering the past two proposals I've made here, I doubt I'll get more
>> > than an echo chamber effect on this one.
>> >
>> > For the past week or so I've spent a bit of time on a feature I've
>> always
>> > wanted to see land in django -- composite fields. The tickets have been
>> open
>> > in the bug tracker for quite some time (and there's a few related ones,
>> such
>> > as multi-column primary keys that can all be killed with the one stone).
>> >
>> > The work is available on this branch of my fork of django for the
>> moment --
>> > I haven't opened up a PR yet because there's still some features that
>> are
>> > still to be implemented that will be explained below, but I want to give
>> > everybody a chance to tell me where I can stick it before I spend *too*
>> much
>> > time on it.
>> >
>> > So, without further ado, the proposal.
>> >
>> >
>> > Composite Fields - Implemented
>> >
>> > A composite field is an encapsulation of the functionality of a subset
>> of
>> > fields on a model. Composite fields can be defined in one of two ways:
>> >
>> > 1. Statically declared composite fields
>> >
>> > A statically declared composite field is defined in the same way a
>> django
>> > model is defined. There are two customisable transformation functions,
>> > CompositeField.value_to_dict(self, value) and
>> > CompositeField.value_from_dict(self, value) which can be used to
>> associate
>> > the field with a python object.
>> >
>> > All the serialization functions are implemented via the implementations
>> of
>> > the subfields.
>> >
>> > For example,
>> >
>> > class MoneyField(models.CompositeField):
>> >currency_code = models.CharacterField(max_length=3)
>> >amount = models.DecimalField(max_digits=16, decimal_digits=4)
>> >
>> >## Overriding __init__ can be used to pass field parameters to the
>> > subfields
>> >
>> >def value_from_dict(self, value):
>> >if value is None:
>> >   return None
>> >return Money(**value)
>> >
>> >def value_to_dict(self, value):
>> >   if value is None:
>> >  return None
>> >   return {attr: getattr(value, attr) for attr in ('currency_code',
>> > 'amount')}
>> >
>> > 2. Inline composite fields.
>> >
>> > An inline composite field is declared at the field definition site on
>> the
>> > body of a model, by providing the subfields as the 'fields' argument of
>> the
>> >

Re: Composite fields

[Josh Smeaton]

Hey Thomas,

You've only waited 11 hours for a response here. Your contributions are 
definitely welcome, and it's obvious that you've done a lot of work and put 
in a lot of effort. Personally, I thank you.

The reason you're not getting a response, at this point, is because the 
number of people able to intelligently comment on your proposal is very 
small. I can think of a few people that would have some interest, including 
Ansii, Loic, and probably Marc. I'm interested in the ORM in general, but I 
really have no opinion on Composite Fields at the moment, so I have nothing 
to really offer you except some positive wishes.

I know there is a desire to have composite fields and keys. The last brief 
discussion that I can find is 
here: 
https://groups.google.com/forum/#!searchin/django-developers/composite%7Csort:date/django-developers/ww6o-1kvI28/OE_V6e8qlWgJ
 
which calls out a particular branch where work had already commenced. 
AFAIK, it's also a Google Summer of Code suggested project.

So please, don't be disheartened. As I said, the number of users that could 
give you feedback is small, and there is a very cyclic pattern of activity 
amongst those developers. I would encourage you to read previous 
discussions and look at previous attempts, and continue with your work. It 
most definitely is wanted. Someone will be around, eventually, to offer 
help and guidance when needed.

As to your other messages on this mailing list that haven't seen a lot of 
activity, I wouldn't take that too personally. Open up a ticket on trac, 
and let the traditional triaging take place to let you know if you're on 
the right track or not. Feedback on the ML is usually reserved for ideas or 
features that are contentious, though that's not always the case. It's 
obvious that you have an interest in improving Django, and that you have 
the skill and desire to make that happen. I really do hope you continue, 
despite the lack of activity on the ML at the moment.

All the best :)

Josh

On Thursday, 5 March 2015 11:43:38 UTC+11, Thomas Stephenson wrote:
>
> OK, no need for everyone to shout -- your message is heard loud and 
> clear. I'll go and find something else to work on. 
>
> On 5 March 2015 at 00:16, Thomas Stephenson  > wrote: 
> > Considering the past two proposals I've made here, I doubt I'll get more 
> > than an echo chamber effect on this one. 
> > 
> > For the past week or so I've spent a bit of time on a feature I've 
> always 
> > wanted to see land in django -- composite fields. The tickets have been 
> open 
> > in the bug tracker for quite some time (and there's a few related ones, 
> such 
> > as multi-column primary keys that can all be killed with the one stone). 
> > 
> > The work is available on this branch of my fork of django for the moment 
> -- 
> > I haven't opened up a PR yet because there's still some features that 
> are 
> > still to be implemented that will be explained below, but I want to give 
> > everybody a chance to tell me where I can stick it before I spend *too* 
> much 
> > time on it. 
> > 
> > So, without further ado, the proposal. 
> > 
> > 
> > Composite Fields - Implemented 
> > 
> > A composite field is an encapsulation of the functionality of a subset 
> of 
> > fields on a model. Composite fields can be defined in one of two ways: 
> > 
> > 1. Statically declared composite fields 
> > 
> > A statically declared composite field is defined in the same way a 
> django 
> > model is defined. There are two customisable transformation functions, 
> > CompositeField.value_to_dict(self, value) and 
> > CompositeField.value_from_dict(self, value) which can be used to 
> associate 
> > the field with a python object. 
> > 
> > All the serialization functions are implemented via the implementations 
> of 
> > the subfields. 
> > 
> > For example, 
> > 
> > class MoneyField(models.CompositeField): 
> >currency_code = models.CharacterField(max_length=3) 
> >amount = models.DecimalField(max_digits=16, decimal_digits=4) 
> > 
> >## Overriding __init__ can be used to pass field parameters to the 
> > subfields 
> > 
> >def value_from_dict(self, value): 
> >if value is None: 
> >   return None 
> >return Money(**value) 
> > 
> >def value_to_dict(self, value): 
> >   if value is None: 
> >  return None 
> >   return {attr: getattr(value, attr) for attr in ('currency_code', 
> > 'amount')} 
> > 
> > 2. Inline composite fields. 
> > 
&

Re: Composite fields

[Curtis Maloney]

Have you reviewed all the existing works on trying to add composite fields?

I know several managed to reach a level of maturity that was almost merge
quality, and am sure I heard some of the recent ORM changes would ease
support.

On 5 March 2015 at 11:42, Thomas Stephenson  wrote:

> OK, no need for everyone to shout -- your message is heard loud and
> clear. I'll go and find something else to work on.
>
>
Wow... impatient much?  Not everyone works in your timezone, or to your
schedule.  I, for one, was planning to take some time to review your
proposal carefully, instead of "first thing in the morning whilst still
having my coffee", as I feel it's a topic that deserves careful
consideration.

Don't you think it's worth giving everyone a chance to read your email
before you give up and move on?  It's only been 11 hours.  Many of us were
asleep for most of that.

--
Curtis



> On 5 March 2015 at 00:16, Thomas Stephenson  wrote:
> > Considering the past two proposals I've made here, I doubt I'll get more
> > than an echo chamber effect on this one.
> >
> > For the past week or so I've spent a bit of time on a feature I've always
> > wanted to see land in django -- composite fields. The tickets have been
> open
> > in the bug tracker for quite some time (and there's a few related ones,
> such
> > as multi-column primary keys that can all be killed with the one stone).
> >
> > The work is available on this branch of my fork of django for the moment
> --
> > I haven't opened up a PR yet because there's still some features that are
> > still to be implemented that will be explained below, but I want to give
> > everybody a chance to tell me where I can stick it before I spend *too*
> much
> > time on it.
> >
> > So, without further ado, the proposal.
> >
> >
> > Composite Fields - Implemented
> >
> > A composite field is an encapsulation of the functionality of a subset of
> > fields on a model. Composite fields can be defined in one of two ways:
> >
> > 1. Statically declared composite fields
> >
> > A statically declared composite field is defined in the same way a django
> > model is defined. There are two customisable transformation functions,
> > CompositeField.value_to_dict(self, value) and
> > CompositeField.value_from_dict(self, value) which can be used to
> associate
> > the field with a python object.
> >
> > All the serialization functions are implemented via the implementations
> of
> > the subfields.
> >
> > For example,
> >
> > class MoneyField(models.CompositeField):
> >currency_code = models.CharacterField(max_length=3)
> >amount = models.DecimalField(max_digits=16, decimal_digits=4)
> >
> >## Overriding __init__ can be used to pass field parameters to the
> > subfields
> >
> >def value_from_dict(self, value):
> >if value is None:
> >   return None
> >return Money(**value)
> >
> >def value_to_dict(self, value):
> >   if value is None:
> >  return None
> >   return {attr: getattr(value, attr) for attr in ('currency_code',
> > 'amount')}
> >
> > 2. Inline composite fields.
> >
> > An inline composite field is declared at the field definition site on the
> > body of a model, by providing the subfields as the 'fields' argument of
> the
> > CompositeField constructor. There are no transformation parameters
> available
> > to override when declaring a composite field in this fashion -- the
> value of
> > the field is always available as a python `dict` as an attribute on the
> > MyModel
> >
> > class MyModel(models.Model):
> > id = models.CompositeField(fields = [
> >('a', models.IntegerField()),
> >('b', models.CharField(max_length=30)
> > ], primary_key=True)
> >
> > This method for defining composite fields has a few drawbacks, but can be
> > useful if the only reason to add the composite field to the model was to
> > implement a unique_together or index_together constraint *
> >
> > * Although it's still possible to do that directly on class Meta.
> >
> > 3. Null
> >
> > Setting the value of a multi-column field to NULL is different than
> setting
> > any of the individual subfields to NULL. But there are cases (e.g. Money)
> > where we would like to be able to set `null=True` on a composite field,
> but
> > still retain 'NOT NULL' constraints on each of t

Re: Composite fields

[Thomas Stephenson]

OK, no need for everyone to shout -- your message is heard loud and
clear. I'll go and find something else to work on.

On 5 March 2015 at 00:16, Thomas Stephenson  wrote:
> Considering the past two proposals I've made here, I doubt I'll get more
> than an echo chamber effect on this one.
>
> For the past week or so I've spent a bit of time on a feature I've always
> wanted to see land in django -- composite fields. The tickets have been open
> in the bug tracker for quite some time (and there's a few related ones, such
> as multi-column primary keys that can all be killed with the one stone).
>
> The work is available on this branch of my fork of django for the moment --
> I haven't opened up a PR yet because there's still some features that are
> still to be implemented that will be explained below, but I want to give
> everybody a chance to tell me where I can stick it before I spend *too* much
> time on it.
>
> So, without further ado, the proposal.
>
>
> Composite Fields - Implemented
>
> A composite field is an encapsulation of the functionality of a subset of
> fields on a model. Composite fields can be defined in one of two ways:
>
> 1. Statically declared composite fields
>
> A statically declared composite field is defined in the same way a django
> model is defined. There are two customisable transformation functions,
> CompositeField.value_to_dict(self, value) and
> CompositeField.value_from_dict(self, value) which can be used to associate
> the field with a python object.
>
> All the serialization functions are implemented via the implementations of
> the subfields.
>
> For example,
>
> class MoneyField(models.CompositeField):
>currency_code = models.CharacterField(max_length=3)
>amount = models.DecimalField(max_digits=16, decimal_digits=4)
>
>## Overriding __init__ can be used to pass field parameters to the
> subfields
>
>def value_from_dict(self, value):
>if value is None:
>   return None
>return Money(**value)
>
>    def value_to_dict(self, value):
>   if value is None:
>  return None
>   return {attr: getattr(value, attr) for attr in ('currency_code',
> 'amount')}
>
> 2. Inline composite fields.
>
> An inline composite field is declared at the field definition site on the
> body of a model, by providing the subfields as the 'fields' argument of the
> CompositeField constructor. There are no transformation parameters available
> to override when declaring a composite field in this fashion -- the value of
> the field is always available as a python `dict` as an attribute on the
> MyModel
>
> class MyModel(models.Model):
> id = models.CompositeField(fields = [
>('a', models.IntegerField()),
>('b', models.CharField(max_length=30)
> ], primary_key=True)
>
> This method for defining composite fields has a few drawbacks, but can be
> useful if the only reason to add the composite field to the model was to
> implement a unique_together or index_together constraint *
>
> * Although it's still possible to do that directly on class Meta.
>
> 3. Null
>
> Setting the value of a multi-column field to NULL is different than setting
> any of the individual subfields to NULL. But there are cases (e.g. Money)
> where we would like to be able to set `null=True` on a composite field, but
> still retain 'NOT NULL' constraints on each of the subfield columns.
>
> To solve this problem, every table which implements a CompositeField will
> also add an implicit (semi-hidden) `isnull` subfield on the attribute, which
> keeps track of whether it is the value of the composite field that is null,
> or any of the particular subfields.
>
> 3. Querying.
>
> The syntax for querying over the subfields of a composite field will be
> familiar to anyone who has queried over a relationship attribute in django.
>
> model.objects.filter(price__currency_code='USD',
> price__amount__lt=Decimal('50.00')).all()
>
> In addition, composite fields can be queried via EXACT and IN lookups. It is
> possible to implement custom lookups for specific statically defined fields,
> but not recommended and not part of the official API.
>
> 4. Restrictions
>
> The following restrictions are currently imposed on the use of composite
> fields. None of these are restrictions that can't be worked around in future
> extensions, but they're restrictions which considerably simplify both the
> implementation and API.
>
> - No related fields as a subfield of a composite field
> - No nested composite fields
> - No inher

Composite fields

[Thomas Stephenson]

Considering the past two proposals I've made here, I doubt I'll get more 
than an echo chamber effect on this one.

For the past week or so I've spent a bit of time on a feature I've always 
wanted to see land in django -- composite fields. The tickets have been 
open in the bug tracker for quite some time (and there's a few related 
ones, such as multi-column primary keys that can all be killed with the one 
stone).

The work is available on this branch of my fork of django 
<https://github.com/ovangle/django/tree/composite_fields> for the moment -- 
I haven't opened up a PR yet because there's still some features that are 
still to be implemented that will be explained below, but I want to give 
everybody a chance to tell me where I can stick it before I spend *too* 
much time on it.

So, without further ado, the proposal.


Composite Fields - Implemented

A composite field is an encapsulation of the functionality of a subset of 
fields on a model. Composite fields can be defined in one of two ways:

1. Statically declared composite fields

A statically declared composite field is defined in the same way a django 
model is defined. There are two customisable transformation functions, 
CompositeField.value_to_dict(self, value) and 
CompositeField.value_from_dict(self, value) which can be used to associate 
the field with a python object.

All the serialization functions are implemented via the implementations of 
the subfields.

For example,  

class MoneyField(models.CompositeField):
   currency_code = models.CharacterField(max_length=3)
   amount = models.DecimalField(max_digits=16, decimal_digits=4)

   ## Overriding __init__ can be used to pass field parameters to the 
subfields

   def value_from_dict(self, value):
   if value is None:
  return None
   return Money(**value)

   def value_to_dict(self, value):
  if value is None:
 return None
  return {attr: getattr(value, attr) for attr in ('currency_code', 
'amount')}

2. Inline composite fields.

An inline composite field is declared at the field definition site on the 
body of a model, by providing the subfields as the 'fields' argument of the 
CompositeField constructor. There are no transformation parameters 
available to override when declaring a composite field in this fashion -- 
the value of the field is always available as a python `dict` as an 
attribute on the MyModel

class MyModel(models.Model):
id = models.CompositeField(fields = [
   ('a', models.IntegerField()),
   ('b', models.CharField(max_length=30)
], primary_key=True)

This method for defining composite fields has a few drawbacks, but can be 
useful if the only reason to add the composite field to the model was to 
implement a unique_together or index_together constraint *

* Although it's still possible to do that directly on class Meta.

3. Null

Setting the value of a multi-column field to NULL is different than setting 
any of the individual subfields to NULL. But there are cases (e.g. Money) 
where we would like to be able to set `null=True` on a composite field, but 
still retain 'NOT NULL' constraints on each of the subfield columns.

To solve this problem, every table which implements a CompositeField will 
also add an implicit (semi-hidden) `isnull` subfield on the attribute, 
which keeps track of whether it is the value of the composite field that is 
null, or any of the particular subfields.

3. Querying.

The syntax for querying over the subfields of a composite field will be 
familiar to anyone who has queried over a relationship attribute in django.

model.objects.filter(price__currency_code='USD', 
price__amount__lt=Decimal('50.00')).all()

In addition, composite fields can be queried via EXACT and IN lookups. It 
is possible to implement custom lookups for specific statically defined 
fields, but not recommended and not part of the official API.

4. Restrictions

The following restrictions are currently imposed on the use of composite 
fields. None of these are restrictions that can't be worked around in 
future extensions, but they're restrictions which considerably simplify 
both the implementation and API.

- No related fields as a subfield of a composite field
- No nested composite fields
- No inheritance of composite fields (apart from inheriting from 
CompositeField itself).

5. Changes to the Field API

As discussed in the other thread I posted. I've changed the implementation 
so that _get_cache_name can still be dependent on the name, but I think 
using attname is more useful anyway.

Composite Fields -- unimplemented

These features are still not implemented

- multi column primary keys. unique_together and index_together are 
implemented and adding a primary key constraint should be a similar 
operation.
- some small issues with multi-table inheritance.
- more tes

Re: RE Composite fields-/ Multi Primary / Foreign keys

[Russell Keith-Magee]

Hi Aron,

Short version - if you're eager to look into this, we're eager to have
composite keys in Django - it's probably one of the last features I can
think of that was discussed for the 1.0 release but we bumped so that "we
can add it for 1.1".

(ahem. :-)

Yes, there have been a lot of changes since Michal Petrucha (koniiik) last
worked on his patch - most notably, migrations and the Meta refactor have
landed. Both of these will have a big impact on any composite key design.
However, there have also been a bunch of changes to the internals of
ForeignKeys that will aid in the implementation of composite keys; the
design of the meta refactor also kept the potential requirements of
composite keys in mind.

Michal's github branch is probably still the best place to start - he's
done a lot of work on this project, and at the very least should be able to
guide you as to what is left to be done.

Best of luck!

Yours,
Russ Magee %-)

On Tue, Feb 24, 2015 at 2:14 PM, Aron Podrigal 
wrote:

> Hi, I just came across a project that requires  this functionality of
> multi primary/foreign keys. So I'm bumping up this thread
> again.
> There were a lot of changes in django and the db api hasre-factored  since
> this topic was discussed. It might be a time for another start on this
> project. I'm willing to work on this as long it is ok with the core
> developers. I didn't have much time yet to spend on this. I'm still reading
> through the other earlier threads on possible implementations.
> I will post my proposal for this over the weekend. In the meantime I would
> like to get some feedback from the community.
>
> Aron.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/a4a31a7b-4759-4006-a280-34d9ac6b0abe%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAJxq84_c2j%3D_gvHVdYxBq81SHKrx5-rLcpox1txSKkrDNtePhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

RE Composite fields-/ Multi Primary / Foreign keys

[Aron Podrigal]

Hi, I just came across a project that requires  this functionality of multi 
primary/foreign keys. So I'm bumping up this thread 
again.
 
There were a lot of changes in django and the db api hasre-factored  since 
this topic was discussed. It might be a time for another start on this 
project. I'm willing to work on this as long it is ok with the core 
developers. I didn't have much time yet to spend on this. I'm still reading 
through the other earlier threads on possible implementations.
I will post my proposal for this over the weekend. In the meantime I would 
like to get some feedback from the community. 

Aron.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a4a31a7b-4759-4006-a280-34d9ac6b0abe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [GSoC] Composite fields once again

[Duke Nukem 3D is coming!]

This sounds exciting. I can't wait for this to land in master. Thanks, 
Michal!

On Sunday, June 2, 2013 10:22:36 AM UTC-4, Michal Petrucha wrote:
>
> Hello everyone, 
>
> I'm happy to announce that throughout the summer I'll be officially 
> working on composite fields under the supervision of Andrew Godwin as 
> part of this year's GSoC. 
>
> I'll use this thread to post all my weekly status updates regarding 
> this project. This email gives a brief overview of the current status 
> and a rough outline of my plans, as well as a few questions I'd like 
> to raise with the community. 
>
> So, as far as currently available code goes, there's a github repo 
> containing my work from the GSoC two years ago, which is outdated and 
> incomplete, but still contains a large amount of working code, most of 
> which should be possible to adapt to the current state of Django. 
>
> The work can be outlined roughly in the following four steps: 
>
> 1) create a solid implementation of virtual fields 
> 2) refactor ForeignKey/OneToOneField to become virtual fields 
> 3) implement CompositeField 
> 4) make CompositeField work with as many additional parts of Django as 
>possible, including GenericForeignKey and inspectdb 
> 5) possibly implement a mechanism to modify the primary key of a model 
>instance 
>
> All of the above steps are explained in more detail in the proposal I 
> submitted for the GSoC, available as a gist [1]. 
>
> Now, for the questions I wanted to raise. 
>
>
> ForeignKey customization API 
>  
>
> This one is mostly about the extent to which we want the internal 
> implementation of ForeignKey to affect the public API. To keep things 
> backwards compatible, attributes such as db_column or db_index will be 
> just passed to the auto-generated auxiliary field. 
>
> The question is, do we want to make it possible to specify a custom 
> auxiliary field to a ForeignKey instead of always creating one? 
>
> A related question, how should it be possible to customize the 
> database columns of composite ForeignKeys? Just make ForeignKey accept 
> a tuple instead of a string? Or just require the user to create the 
> fields (along with a CompositeField) by hand and pass that as the 
> auxiliary field? Any nicer possibility? 
>
> Either option is rather simple to implement, I just don't really have 
> a strong opinion, although I find both a little bit unpleasant. 
>
>
> GenericForeignKey and nontrivial field types 
>  
>
> As I've indicated in my proposal, just casting any value to a string 
> and then performing a reversible transformation on such strings may 
> work well enough for string and integer database columns, not so much 
> for things like dates, timestamps IP addresses or other similar types. 
>
> Any ideas on how to make this work? Should I try to extend the backend 
> API to include explicit casts for each nontrivial column type to a 
> string representation equal to the one used by Python? Or should I 
> just document this as unsupported? 
>
>
> Updatable primary keys 
> ------ 
>
> This feature is not directly related to the main objective of this 
> project, which is to implement composite fields. It is just easier for 
> people to get into a situation where this might be required when using 
> composite fields. 
>
> Full support for this feature would require quite massive changes to 
> the collectors used cascading deletes -- they'd have to be generalized 
> to support cascading updates as well. This would introduce a lot of 
> complexity, obviously. 
>
> Jeremy Tillman voiced his opinion against this feature in a comment to 
> my proposal. He gives valid arguments -- an update of a primary key 
> would be an expensive operation with potential to kill the performance 
> of any apps doing it. However, the argument that it is easily done 
> with a Model.objects.filter(...).update(...) call is not entirely true 
> as a full implementation of this feature would also cascade. Moreover, 
> it would make this possible in the admin without much hassle. 
>
> So, seeing that there is at least one voice against this feature, I 
> think it's better to let the community decide if we want it at all. 
>
> Either way, it's highly unlikely I'd be able to deliver this feature 
> as part of this GSoC, the best I can promise is an initial proof of 
> concept implementation. 
>
>
> Michal 
>
>
> [1] https://gist.github.com/konk/5408673 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [GSoC] Composite fields -- another status report

[Michal Petrucha]

Hello everyone,

a lot of time has passed since I last posted a status report here and
I apologize for that.

Composite ForeignKeys were a fairly simple thing to implement with all
the required parts already in place. There is one edge case that I'd
like to handle, where Django would detect attempts to traverse a
composite ForeignKey with select_related while deferring a subset of
its backing fields and raise an error. Other than that, the feature
only needs people to try it out now.

As for the other big thing I've been spending my time on during that
time, I was merging Andrew's migration code into my branch. Certain
parts of migrations need a little bit of tweaking to support virtual
fields. I fixed all regressions caught by the test suite, but there
are a lot of weird corner cases when it comes to virtual fields and
migrations we're already aware of where the code will just fail. This
is an area which will require a lot of polish, so we concentrated on
fixing it for simple (non-composite) ForeignKeys first, the rest can
come later.

I guess this is pretty much it for this year's Sumer of Code. To
recap, I managed to implement the core of the project and a few little
extra features as well. Unfortunately, I didn't manage to write
GenericForeignKey and inspectdb support, which is unfortunate, but a
rather large item popped up which wasn't taken into account in my
proposal, that is the migrations merge.

All in all, I think the project can be considered a success, although
there are still a lot of wrinkles to be ironed out. Of course, I
intend to stick around and get it to a mergeable state and eventually
get around to the features I didn't manage to pull off yet.

The current state of the code is that it doesn't cause any regressions
in the test suite compared to master in either branch. The next thing
to work on (apart from the check I mentioned earlier) is probably
giving more polish to migrations of virtual fields and I'll probably
document how people can write their own custom virtual fields as well.

That's it from me for now.

Michal


signature.asc
Description: Digital signature

[GSoC] Composite fields -- irregular status report #4

[Michal Petrucha]

Hello,

Last week I finished an implementation of CompositeField with support
for backend-dependent IN lookups. During the past few days I also
added support for composite primary keys to the admin. The latest work
is available in the soc2013/composite-fields branch [1].

As a reminder, the previous branch, soc2013/foreignkey-refactor, still
needs a few more eyes. I'll also be posting a separate thread for an
issue raised by Anssi while reviewing it, stay tuned.

Now I'm going to work on composite ForeignKey.

Michal


[1] https://github.com/konk/django/tree/soc2013/composite-fields


signature.asc
Description: Digital signature

Re: Composite fields -- (ir)regular status report #3

[Michal Petrucha]

> I did a quick review of the patch and didn't see anything that jumped out. 
> BTW when you want to merge a part of your work, please open a pull request. 
> Reviewing is much easier that way.

I just opened a pull request [1]. The branch is up to date and it does
not add any test failures that are not already there on master (I get
3 failures on master with anything but SQLite.)

It implements automatic auxiliary field creation as well as manual
auxiliary fields. I should probably add tests for select_related and
defer/only with custom auxiliary fields, which I'll try to do
tomorrow.

Michal


[1] https://github.com/django/django/pull/1407


signature.asc
Description: Digital signature

Re: Composite fields -- (ir)regular status report #3

[Anssi Kääriäinen]

On Monday, July 22, 2013 2:48:00 AM UTC+3, Michal Petrucha wrote:
>
> Hello, 
>
> I have some awesome news today. At long last I managed to finally get 
> the refactor of ForeignKey to pass the entire test suite. It's only 
> one configuration (CPython 2.7 + SQLite), but it's a start. Due to the 
> nature of my changes, I expect that only database creation should be 
> prone to errors on other backends, otherwise I didn't really touch any 
> of the SQL generating code. 
>
> So, the plan for the immediate future is that I'm going to spend the 
> next few days fixing any remaining regressions on other database 
> backends. When this is done, I guess I'll add some more tests for the 
> field cloning mechanism -- a few more tests can never hurt. 
> Afterwards, I'll proceed with CompositeField and generally try to 
> advance through the list of items in my project's timeline. 
>
> In the meantime, I think it's time to start reviewing the changes. 
> This is the first self-contained changeset delivered by this GSoC and 
> the sooner we get it into an acceptable state suitable for merging 
> into master, the less code built on top of it will have to be adapted 
> to changes warranted by reviews. So, if anyone finds some free time 
> and will to sift through the internals of the ORM, you're welcome to 
> have a look at my GitHub repo [1]. I can also create a pull request in 
> case it makes things easier for anyone. 


I did a quick review of the patch and didn't see anything that jumped out. 
BTW when you want to merge a part of your work, please open a pull request. 
Reviewing is much easier that way.

I hope to get some time to review the work in full next week.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Composite fields -- (ir)regular status report #3

[Chris Wilson]

On Mon, 22 Jul 2013, Michal Petrucha wrote:


I have some awesome news today. At long last I managed to finally get
the refactor of ForeignKey to pass the entire test suite.


That is awesome news! Well done Michal! I am really looking forward to 
having this feature available in Django.


Cheers, Chris.
--
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Citylife House, Sturton Street, Cambridge, CB1 2QF, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Composite fields -- (ir)regular status report #3

[Michal Petrucha]

Hello,

I have some awesome news today. At long last I managed to finally get
the refactor of ForeignKey to pass the entire test suite. It's only
one configuration (CPython 2.7 + SQLite), but it's a start. Due to the
nature of my changes, I expect that only database creation should be
prone to errors on other backends, otherwise I didn't really touch any
of the SQL generating code.

So, the plan for the immediate future is that I'm going to spend the
next few days fixing any remaining regressions on other database
backends. When this is done, I guess I'll add some more tests for the
field cloning mechanism -- a few more tests can never hurt.
Afterwards, I'll proceed with CompositeField and generally try to
advance through the list of items in my project's timeline.

In the meantime, I think it's time to start reviewing the changes.
This is the first self-contained changeset delivered by this GSoC and
the sooner we get it into an acceptable state suitable for merging
into master, the less code built on top of it will have to be adapted
to changes warranted by reviews. So, if anyone finds some free time
and will to sift through the internals of the ORM, you're welcome to
have a look at my GitHub repo [1]. I can also create a pull request in
case it makes things easier for anyone.

Have a great day.

Michal


[1] https://github.com/konk/django/tree/soc2013/foreignkey-refactor


signature.asc
Description: Digital signature

Re: [GSoC] Composite fields once again -- irregular status report #2

[Michal Petrucha]

Hello,

I apologize for the lack of updates on the progress of my project in
the past few weeks, there just wasn't much to report, unfortunately.

I spent the past two weeks porting the ForeignKey refactor on top of
master. It took significantly more time to do than I anticipated due
to all the changes in ForeignKey and other ORM internals. Currently
most of the code seems to work, however, I'm in a regression fixing
mode, running the test suite, picking a failing test, fixing and
repeating. I still have 32 failures and 56 errors remaining at the
moment, which is significantly less than the 200 failures and 1100
errors I started at, but still quite a lot.

The work is slow, tedious and quite tiring, each regression takes at
least a couple-three hours to figure out (often even longer) and
doesn't decrease the number of remaining errors all that much.

I'm one week behind my schedule right now, unfortunately, it seems
that this lag will increase further. Well, I was too optimistic in my
proposal...

The good news is that the changes I'm required to make at this stage
will make support for composite fields even easier to implement than I
anticipated.

Feel free to check out the code on Github [1].

Michal


[1] https://github.com/konk/django/tree/soc2013/foreignkey-refactor


signature.asc
Description: Digital signature

[GSoC] Composite fields -- weekly status report no. 1

[Michal Petrucha]

Hello,

The official coding period has started a week ago, therefore I guess a
progress report is in order.

As far as changes to the proposal are concerned, it seems there aren't
any. There haven't been any objections or other suggestions to improve
the proposal, other than the updatable primary key part, which is very
unlikely to happen as part of this project anyway (and was proposed as
such as well). That means I'll try to stick to the original plan as
outlined in my proposal [1].

Initially I had big plans to start a bit earlier in order to get a
head start, unfortunately, I only managed to pass my final exam last
Thursday, which means I spent most of this week revising. Luckily, I
had enough good sense while laying out my timeline not to put too much
work into the first week.

This week's goal was to port the basic virtual field functionality,
especially make most of the required changes in Model._meta. I think I
have most of this ready, though I had to modify GenericForeignKey and
GenericRelation a bit (since they are both handled as virtual fields,
although each in a different way) and I stumbled upon a couple of
unexpected complications with the handling of GenericRelation in joins
and cascading deletes.

I'm confident I should be able to sort these out tomorrow.

I also wrote some docs on the updated API of ForeignKey I have
planned. I'll also document the changes themselves in the release
notes as soon as I figure out what exactly is going to change and in
what way.

The code is accessible in a new branch of my github repo [2].

The plan for the next week is to begin porting the ForeignKey
refactor. Since this is easily the most extensive change that is a
part of this project, it is going to take more than a single week,
especially considering how the implementation of related fields has
changed since I first did the refactor. Anyway, the timeline takes
this into account, so let's hope my estimates weren't too optimistic.

That's it for this week's status report, have a great day.

Michal


[1] https://gist.github.com/konk/5408673
[2] https://github.com/konk/django/tree/soc2013/foreignkey-refactor


signature.asc
Description: Digital signature

Re: Composite fields once again

[Michal Petrucha]

Oh wow, have two weeks passed already? I seem to really suck at
email... \-: Anyway, thanks for the comments.

On Fri, Jun 07, 2013 at 03:55:12AM -0700, Anssi Kääriäinen wrote:
> Looks like a good plan. Getting CompositeFields, multicolumn primary
> keys and foreign keys to work as well as possible should of course be
> priority no.1, all other things are secondary.

That's the idea. (-:

> I think having a way to manually create the auxilary fields is a good
> idea. If you need to customize the internal fields for some reason,
> then having the ability to fully override the fields seems necessary.
> Be careful about the field ordering, if I recall correctly ImageField
> and its stored width/height values, and GenericForeignKey have some
> problems in model.__init__ due to this (need to use pre/post init
> signals...).

As for ImageField, I think its use of the post_init signal is not
really problematic in this regard. Regarding GenericForeignKey, as I
recall, I changed it into a VirtualField and modified Model.__init__
slightly -- first it processes its positional and keyword arguments as
usual and in the end if there are any unprocessed kwargs left, it
checks if any of them correspond to virtual fields and uses setattr on
those. This makes it possible to ditch pre/post_init signal handlers
in GenericForeignKey completely.

> GFKs can be fixed later on.
> 
> Document this as unsupported if you don't have time to work on this.
> This means that you can't do JOINs in the database using
> GenericForeignKeys and multicolumn primary keys. I wonder if this
> actually works correctly for all possible primary key types in current
> code...

That's exactly what I've been wondering. Okay then, I'll just document
this as unsupported and look into it only in case I manage to finish
everything else on time. (-;

> I don't see why updatable primary keys would cause big problems for
> performance. Why would these be more expensive that cascading deletes?

It depends on the usage, of course. But I agree that a lot of people
might find that useful without experiencing any significant
performance losses. (IOW, I'm not taking any side on this one. d-: )

> There is also the problem that foreign keys can be to other columns
> than primary key, and if we are going to handle foreign key cascades,
> then these should work, too.

I wonder, does the cascading delete code handle non-default to_fields?
This is not clear to me at the moment...

> I think the best way to do this is to have a model._meta (or maybe
> primary key field) option "updatable_primary_key=True/False". If
> updatable_primary_key=True, then store the original primary key into
> model._state.original_pk in Query.iterator() (and other places that
> "resurrect" objects from database). This will have a small cost on
> model initialization and should be avoided for autopk models. On save,
> check for changed primary key. If changed, then check for cascades.
> 
> The big problem is what to do for foreign key cascades when to_field
> isn't the primary key. One option is to check which fields are needed
> for cascades and store the original values for all of those. Another
> option is to fetch the old values on save. This will cost an extra
> database query on save when using cascading foreign keys, but this
> isn't too bad. By default all foreign keys are on_update=DO_NOTHING,
> that is cascades aren't handled.
> 
> My preferred option would be to have DB_CASCADE in addition to CASCADE
> and DO_NOTHING as on_update options. DB_CASCADE informs the database
> to create foreign keys as ON UPDATE CASCADE.

Well that escalated quickly... While this is mostly what I had in
mind, seeing it written down like this really makes me want to
reconsider this particular feature.

> In my opinion you don't need to handle any of this in the GSoC
> project. Implementing updatable primary keys, but not foreign key
> cascades is also possible. Users who want to use updatable primary
> keys with cascades will need to alter their database schema to use ON
> UPDATE CASCADE. Handling cascading foreign keys in full will be big
> enough project for full GSoC, so don't worry about this too much.

You're probably right, this was a feature I only included in my
proposal for the unlikely case that I'd be finished with the rest of
the work before the end of the program. I'll probably leave it open
and only dig into this if there's nothing else to work on.

Michal


signature.asc
Description: Digital signature

Re: Composite fields once again

[Anssi Kääriäinen]

As nobody else have responded, here are my opinions...

On 2 kesä, 17:22, Michal Petrucha  wrote:
> 1) create a solid implementation of virtual fields
> 2) refactor ForeignKey/OneToOneField to become virtual fields
> 3) implement CompositeField
> 4) make CompositeField work with as many additional parts of Django as
>    possible, including GenericForeignKey and inspectdb
> 5) possibly implement a mechanism to modify the primary key of a model
>    instance

Looks like a good plan. Getting CompositeFields, multicolumn primary
keys and foreign keys to work as well as possible should of course be
priority no.1, all other things are secondary.

> ForeignKey customization API
> 
>
> This one is mostly about the extent to which we want the internal
> implementation of ForeignKey to affect the public API. To keep things
> backwards compatible, attributes such as db_column or db_index will be
> just passed to the auto-generated auxiliary field.
>
> The question is, do we want to make it possible to specify a custom
> auxiliary field to a ForeignKey instead of always creating one?
>
> A related question, how should it be possible to customize the
> database columns of composite ForeignKeys? Just make ForeignKey accept
> a tuple instead of a string? Or just require the user to create the
> fields (along with a CompositeField) by hand and pass that as the
> auxiliary field? Any nicer possibility?
>
> Either option is rather simple to implement, I just don't really have
> a strong opinion, although I find both a little bit unpleasant.

I think having a way to manually create the auxilary fields is a good
idea. If you need to customize the internal fields for some reason,
then having the ability to fully override the fields seems necessary.
Be careful about the field ordering, if I recall correctly ImageField
and its stored width/height values, and GenericForeignKey have some
problems in model.__init__ due to this (need to use pre/post init
signals...).

> GenericForeignKey and nontrivial field types
> 
>
> As I've indicated in my proposal, just casting any value to a string
> and then performing a reversible transformation on such strings may
> work well enough for string and integer database columns, not so much
> for things like dates, timestamps IP addresses or other similar types.
>
> Any ideas on how to make this work? Should I try to extend the backend
> API to include explicit casts for each nontrivial column type to a
> string representation equal to the one used by Python? Or should I
> just document this as unsupported?

GFKs can be fixed later on.

Document this as unsupported if you don't have time to work on this.
This means that you can't do JOINs in the database using
GenericForeignKeys and multicolumn primary keys. I wonder if this
actually works correctly for all possible primary key types in current
code...

> Updatable primary keys
> ------
>
> This feature is not directly related to the main objective of this
> project, which is to implement composite fields. It is just easier for
> people to get into a situation where this might be required when using
> composite fields.
>
> Full support for this feature would require quite massive changes to
> the collectors used cascading deletes -- they'd have to be generalized
> to support cascading updates as well. This would introduce a lot of
> complexity, obviously.
>
> Jeremy Tillman voiced his opinion against this feature in a comment to
> my proposal. He gives valid arguments -- an update of a primary key
> would be an expensive operation with potential to kill the performance
> of any apps doing it. However, the argument that it is easily done
> with a Model.objects.filter(...).update(...) call is not entirely true
> as a full implementation of this feature would also cascade. Moreover,
> it would make this possible in the admin without much hassle.
>
> So, seeing that there is at least one voice against this feature, I
> think it's better to let the community decide if we want it at all.
>
> Either way, it's highly unlikely I'd be able to deliver this feature
> as part of this GSoC, the best I can promise is an initial proof of
> concept implementation.

I don't see why updatable primary keys would cause big problems for
performance. Why would these be more expensive that cascading deletes?

There is also the problem that foreign keys can be to other columns
than primary key, and if we are going to handle foreign key cascades,
then these should work, too.

I think the best way to do this is to have a model._meta (or maybe
primary key field) option "updatable_primary_key=True/False". If
updatable_primary_key=True, th

Re: Composite fields once again

[Anssi Kääriäinen]

On 2 kesä, 23:20, Luke Sneeringer  wrote:
> On Jun 2, 2013, at 8:22 AM, Michal Petrucha  wrote:
>
> > GenericForeignKey and nontrivial field types
> > 
>
> > As I've indicated in my proposal, just casting any value to a string
> > and then performing a reversible transformation on such strings may
> > work well enough for string and integer database columns, not so much
> > for things like dates, timestamps IP addresses or other similar types.
>
> > Any ideas on how to make this work? Should I try to extend the backend
> > API to include explicit casts for each nontrivial column type to a
> > string representation equal to the one used by Python? Or should I
> > just document this as unsupported?
>
> There's already a `db_type` method that you can override (that receives a 
> `connection` object) for the actual database type. It's pretty easy to do 
> something to the effect of (for instance):
>
>     if 'postgres' in connection['ENGINE']:
>         return 'uuid'
>     if 'mysql' in connection['ENGINE']:
>         return 'char(36)'
>     [...]
>
> However, having done some work myself on trying to create non-trivial field 
> subclasses, it's everything after that which gets difficult. Django provides 
> an overridable method to cast the value, but nothing for anything else in the 
> expression (the field or the operator), which are hard-coded into the 
> backend. (This is a source of frustration for me personally, because it makes 
> it very difficult to write classes for, say, PostgreSQL arrays, without 
> either resorting to massively ugly syntax or subclassing nearly every single 
> class involved in the process of creating a query (Manager, QuerySet, Query, 
> WhereNode...)
>
> I ultimately went through the subclass-half-the-world technique quite 
> recently (a couple of weeks ago), as I want some non-trivial custom fields 
> for a new project I am about to start for my company (sadly, the project is 
> private, although I'd be happy to share field code if it would help in any 
> way). What I ended up doing is checking the Field subclass for a custom 
> `get_db_lookup_expression` method (that's not a Django field method -- I made 
> it up), and then my Field subclasses could use that to return a full 
> expression in the form "{field} = {value}". If the method is present and I 
> get something (other than None) back from that method, then use it, otherwise 
> I pass it on to the DatabaseOperators class for its usual processing. Using 
> that method prevents me from having to modify a monolithic DatabaseOperators 
> subclass for each new field I add (it seems to me that fields should know how 
> to perform their lookups).
>
> The other challenge was defining the QuerySet lookup expressions. Django 
> essentially hard-codes the things it understands for lookups (e.g. 
> Foo.objects.filter(bar__gt=5) being transformed into "select ... from app_foo 
> where bar > 5"). The set of lookup suffices (exact, gt, gte, lt, lte, etc.) 
> is, sadly, also essentially hard-coded. I wrote an ArrayField to use 
> PostgreSQL arrays, and really wanted a way to be able to lookup based on the 
> length of the array (so, something like `Foo.objects.exclude(photos__len=0)`, 
> for instance, to give me all Foos with no photos). I did manage to make that 
> work, but it was a struggle. Also, the set of lookup suffices is universal, 
> even though some of them don't make sense on some fields ("year" on 
> IntegerField, for instance).
>
> So, my ideas on getting non-trivial field subclasses to work is basically:
>
> 1. Make fields be the arbiter of what lookup types they understand. 
> (IntegerFields shouldn't understand "year"; maybe someone's ArrayField 
> subclass does understand "len".) This probably needs to be something that can 
> be determined on the fly, as composite fields will probably need lookup types 
> corresponding to their individual sub-fields.
>
> 2. Make lookup types chainable. In the Array "len" example, 
> `photos__len__gt=5` makes sense.
>
> 3. Make it so fields define how they are looked up based on database engine 
> and lookup type.
>
> Moving these things into the Field implementation (rather than in the 
> backend) should mean that non-trivial field subclasses become much easier. 
> It'll also eliminate the need for, say, django.contrib.gis to have an 
> entirely different set of backends -- a large reason gis jumps through those 
> hoops (as best as I can tell from reading it; disclaimer: I am not a 
> contributor) 

Re: [GSoC] Composite fields once again

[Luke Sneeringer]

On Jun 2, 2013, at 8:22 AM, Michal Petrucha  wrote:

> GenericForeignKey and nontrivial field types
> 
> 
> As I've indicated in my proposal, just casting any value to a string
> and then performing a reversible transformation on such strings may
> work well enough for string and integer database columns, not so much
> for things like dates, timestamps IP addresses or other similar types.
> 
> Any ideas on how to make this work? Should I try to extend the backend
> API to include explicit casts for each nontrivial column type to a
> string representation equal to the one used by Python? Or should I
> just document this as unsupported?

There's already a `db_type` method that you can override (that receives a 
`connection` object) for the actual database type. It's pretty easy to do 
something to the effect of (for instance):

if 'postgres' in connection['ENGINE']:
return 'uuid'
if 'mysql' in connection['ENGINE']:
return 'char(36)'
[...]

However, having done some work myself on trying to create non-trivial field 
subclasses, it's everything after that which gets difficult. Django provides an 
overridable method to cast the value, but nothing for anything else in the 
expression (the field or the operator), which are hard-coded into the backend. 
(This is a source of frustration for me personally, because it makes it very 
difficult to write classes for, say, PostgreSQL arrays, without either 
resorting to massively ugly syntax or subclassing nearly every single class 
involved in the process of creating a query (Manager, QuerySet, Query, 
WhereNode...)

I ultimately went through the subclass-half-the-world technique quite recently 
(a couple of weeks ago), as I want some non-trivial custom fields for a new 
project I am about to start for my company (sadly, the project is private, 
although I'd be happy to share field code if it would help in any way). What I 
ended up doing is checking the Field subclass for a custom 
`get_db_lookup_expression` method (that's not a Django field method -- I made 
it up), and then my Field subclasses could use that to return a full expression 
in the form "{field} = {value}". If the method is present and I get something 
(other than None) back from that method, then use it, otherwise I pass it on to 
the DatabaseOperators class for its usual processing. Using that method 
prevents me from having to modify a monolithic DatabaseOperators subclass for 
each new field I add (it seems to me that fields should know how to perform 
their lookups).

The other challenge was defining the QuerySet lookup expressions. Django 
essentially hard-codes the things it understands for lookups (e.g. 
Foo.objects.filter(bar__gt=5) being transformed into "select ... from app_foo 
where bar > 5"). The set of lookup suffices (exact, gt, gte, lt, lte, etc.) is, 
sadly, also essentially hard-coded. I wrote an ArrayField to use PostgreSQL 
arrays, and really wanted a way to be able to lookup based on the length of the 
array (so, something like `Foo.objects.exclude(photos__len=0)`, for instance, 
to give me all Foos with no photos). I did manage to make that work, but it was 
a struggle. Also, the set of lookup suffices is universal, even though some of 
them don't make sense on some fields ("year" on IntegerField, for instance).

So, my ideas on getting non-trivial field subclasses to work is basically:

1. Make fields be the arbiter of what lookup types they understand. 
(IntegerFields shouldn't understand "year"; maybe someone's ArrayField subclass 
does understand "len".) This probably needs to be something that can be 
determined on the fly, as composite fields will probably need lookup types 
corresponding to their individual sub-fields.

2. Make lookup types chainable. In the Array "len" example, `photos__len__gt=5` 
makes sense.

3. Make it so fields define how they are looked up based on database engine and 
lookup type.

Moving these things into the Field implementation (rather than in the backend) 
should mean that non-trivial field subclasses become much easier. It'll also 
eliminate the need for, say, django.contrib.gis to have an entirely different 
set of backends -- a large reason gis jumps through those hoops (as best as I 
can tell from reading it; disclaimer: I am not a contributor) is to work around 
the restrictions being described.

I hope that helps your thinking. I have this stuff fresh in my head because 
I've just worked on an implementation for PostgreSQL arrays and composite 
fields that I need for my work. While I've thought a decent bit about 
extensibility (for my own purposes), I haven't open-sourced it largely because 
I know I haven't solved all the problems yet. Havin

[GSoC] Composite fields once again

[Michal Petrucha]

Hello everyone,

I'm happy to announce that throughout the summer I'll be officially
working on composite fields under the supervision of Andrew Godwin as
part of this year's GSoC.

I'll use this thread to post all my weekly status updates regarding
this project. This email gives a brief overview of the current status
and a rough outline of my plans, as well as a few questions I'd like
to raise with the community.

So, as far as currently available code goes, there's a github repo
containing my work from the GSoC two years ago, which is outdated and
incomplete, but still contains a large amount of working code, most of
which should be possible to adapt to the current state of Django.

The work can be outlined roughly in the following four steps:

1) create a solid implementation of virtual fields
2) refactor ForeignKey/OneToOneField to become virtual fields
3) implement CompositeField
4) make CompositeField work with as many additional parts of Django as
   possible, including GenericForeignKey and inspectdb
5) possibly implement a mechanism to modify the primary key of a model
   instance

All of the above steps are explained in more detail in the proposal I
submitted for the GSoC, available as a gist [1].

Now, for the questions I wanted to raise.


ForeignKey customization API


This one is mostly about the extent to which we want the internal
implementation of ForeignKey to affect the public API. To keep things
backwards compatible, attributes such as db_column or db_index will be
just passed to the auto-generated auxiliary field.

The question is, do we want to make it possible to specify a custom
auxiliary field to a ForeignKey instead of always creating one?

A related question, how should it be possible to customize the
database columns of composite ForeignKeys? Just make ForeignKey accept
a tuple instead of a string? Or just require the user to create the
fields (along with a CompositeField) by hand and pass that as the
auxiliary field? Any nicer possibility?

Either option is rather simple to implement, I just don't really have
a strong opinion, although I find both a little bit unpleasant.


GenericForeignKey and nontrivial field types


As I've indicated in my proposal, just casting any value to a string
and then performing a reversible transformation on such strings may
work well enough for string and integer database columns, not so much
for things like dates, timestamps IP addresses or other similar types.

Any ideas on how to make this work? Should I try to extend the backend
API to include explicit casts for each nontrivial column type to a
string representation equal to the one used by Python? Or should I
just document this as unsupported?


Updatable primary keys
--

This feature is not directly related to the main objective of this
project, which is to implement composite fields. It is just easier for
people to get into a situation where this might be required when using
composite fields.

Full support for this feature would require quite massive changes to
the collectors used cascading deletes -- they'd have to be generalized
to support cascading updates as well. This would introduce a lot of
complexity, obviously.

Jeremy Tillman voiced his opinion against this feature in a comment to
my proposal. He gives valid arguments -- an update of a primary key
would be an expensive operation with potential to kill the performance
of any apps doing it. However, the argument that it is easily done
with a Model.objects.filter(...).update(...) call is not entirely true
as a full implementation of this feature would also cascade. Moreover,
it would make this possible in the admin without much hassle.

So, seeing that there is at least one voice against this feature, I
think it's better to let the community decide if we want it at all.

Either way, it's highly unlikely I'd be able to deliver this feature
as part of this GSoC, the best I can promise is an initial proof of
concept implementation.


Michal


[1] https://gist.github.com/konk/5408673


signature.asc
Description: Digital signature

[GSoC 2013] Composite fields: first draft of the proposal

[Michal Petrucha]

Hello,

Last week I posted a draft of a timeline for my proposed project and
only afterward I realized that the timeline by itself does not convey
any meaningful information without explaining each item a little bit
deeper. Here comes the full version which I managed to finally
complete a while ago.

Again, for those who prefer reading the formatted version, the
proposal is also available as a gist:
https://gist.github.com/konk/5408673

Apologies for the huge amounts of text, I seem to have lost track
while writing.

Have an awesome day
Michal



Composite Fields, vol. 2
""""""""""""""""""""""""

Abstract


Two years ago, as part of GSoC 2011, I started working on an
implementation of composite fields as a means to support multi-column
primary keys in models. While the project was was successful, the
timeframe wasn't sufficient to finish it and get it into a state where it
could be merged into Django. This year I propose to take the last
remaining steps for this project to be ready and hopefully even extend it
with some extra features which were left out of the initial project.

Aim of this project
===

Since I started working on this two years ago, I managed (with the help of
a few people) to get most of the hard work done. The biggest part that I
didn't get around to implementing was support in the ORM for multi-column
joins. This has, however, been implemented recently by Jeremy Tillman and
Anssi, which means there are only a few things left to be done.

First of all, the code sitting in my github repo is badly out of date,
which means it needs to be updated to the current state of master. While
I'm at it, I also want to clean up the revision history to get it as close
to a state where it could be just reviewed and merged directly as
possible.

Beside this, there are only the juicier features left that we initially
wanted to leave unsupported and get back to them later. Those are the
following (in no particular order):

- ``GenericForeignKey`` support for ``CompositeField``
- more intelligent handling of ``__in`` lookups
- database instrospection and ``inspectdb``
- detection of a change of the primary key in model instances

Summary of ``CompositeField``
=

This section summarizes the basic API as established in the proposal for
GSoC 2011 [1]_.

A ``CompositeField`` requires a list of enclosed regular model fields as
positional arguments, as shown in this example::

class SomeModel(models.Model):
first_field = models.IntegerField()
second_field = models.CharField(max_length=100)
composite = models.CompositeField(first_field, second_field)

The model class then contains a descriptor for the composite field, which
returns a ``CompositeValue`` which is a customized namedtuple, the
descriptor accepts any iterable of the appropriate length. An example
interactive session::

>>> instance = new SomeModel(first_field=47, second_field="some string")
>>> instance.composite
CompositeObject(first_field=47, second_field='some string')
>>> instance.composite.first_field
47
>>> instance.composite[1]
'some string'
>>> instance.composite = (74, "other string")
>>> instance.first_field, instance.second_field
(74, 'other string')

``CompositeField`` supports the following standard field options:
``unique``, ``db_index``, ``primary_key``. The first two will simply add a
corresponding tuple to ``model._meta.unique_together`` or
``model._meta.index_together``. Other field options don't make much sense
in the context of composite fields.

Supported ``QuerySet`` filters will be ``exact`` and ``in``. The former
should be clear enough, the latter is elaborated in a separate section.

It will be possible to use a ``CompositeField`` as a target field of
``ForeignKey``, ``OneToOneField`` and ``ManyToManyField``. This is
described in more detail in the following section.

Changes in ``ForeignKey``
=

Currently ``ForeignKey`` is a regular concrete field which manages both
the raw value stored in the database and the higher-level relationship
semantics. Managing the raw value is simple enough for simple
(single-column) targets. However, in the case of a composite target field,
this task becomes more complex. The biggest problem is that many parts of
the ORM work under the assumption that for each database column there is a
model field it can assign the value from the column to. While it might be
possible to lift this restriction, it would be a really complex project by
itself.

On the other hand, there is the abstraction of virtual fields working on
top of other fields which is required for this project anyway. The way
forward would be to use th

Re: [GSoC 2013] Composite fields: first draft of the proposal

[Andrew Godwin]

Hi Michal,

This looks like a good starting point for a proposal (not to mention that
we don't doubt that you know about the problem area here!) - a few comments:

 - I'd like a bit more detail on some of your timeline points, in
particular what the introspection parts and primary key updates are
(personal bias, I suspect)
 - I'd like more details on what's been holding it back since GSOC 2011,
and how this GSOC is going to help serve those

Still, I like the general idea. You've mentioned tests and docs too. And I
really, really want to land this feature!

Andrew


On Fri, Apr 19, 2013 at 10:17 PM, Michal Petrucha wrote:

> Hello,
>
> as I promised a week ago, I'm posting a draft of my proposal. It is
> far from complete, specifically, it does not contain technical details
> on how things are going to be implemented. I'll try to fill these in
> as soon as possible, although I wrote more on most of them in my
> previous emails to this list. The most important thing is that it
> contains a timeline and a description of possible fallbacks.
>
> The proposal is accessible as a public gist [1] where it will be
> updated in the future, I'm posting the full text here as well for
> convenience.
>
> Have a nice weekend.
>
> Michal
>
> [1] https://gist.github.com/konk/5408673
>
>
> Composite Fields, vol. 2
> """"""""""""""""""""""""
>
> Aim of this project
> ===
>
> Since I started working on this two years ago, I managed (with the
> help of a few people) to get most of the hard work done. The biggest
> part that I didn't get around to implementing was support in the ORM
> for multi-column joins. This has, however, been implemented recently
> by Jeremy Tiller and Anssi, which means there are only a few things
> left to be done.
>
> First of all, the code sitting in my github repo is badly out of date,
> which means it needs to be updated to the current state of master.
> While I'm at it, I also want to clean up the revision history to get
> it as close to a state where it could be just reviewed and merged
> directly as possible.
>
> Beside this, there are only the juicier features left that we
> initially wanted to leave unsupported and get back to them later.
> Those are the following (in no particular order):
>
> - ``GenericForeignKey`` support for ``CompositeField``
> - more intelligent handling of ``__in`` lookups
> - database instrospection and ``inspectdb``
> - detection of a change of the primary key in model instances
>
> Timeline
> 
>
> Our exam period starts on May 20. and ends on June 28., which means
> that I can't guarantee I will be able to fully dedicate myself to the
> project for the first two weeks, however, if nothing goes wrong, I
> should be able to pass all exams before June 17.
>
> Also, I intend to go to EuroPython, which means the first week of July
> won't be as fruitful as the others, but otherwise I'm ready to work
> full time on the project.
>
> Week  1 (Jun 17. - Jun 23.)
> ~~
> - porting the required virtual field changes, like a ``VirtualField``
>   class and ``django.db.models.options`` changes
> - revisiting the documentation I wrote two years ago to reflect the
>   evolution this project has gone through
>
> Week  2 (Jun 24. - Jun 30.)
> ~~
> - porting the virtual ``ForeignKey`` patchset on top of master to get
>   most of the functionality right
>
> Week  3 (Jul  1. - Jul  7.)
> ~~
> - EuroPython 2013, Florence
> - I intend to spend the full two days of sprints working on this but I
>   can't promise much more during this week.
> - running through the tests and fixing those that need updating,
>   ironing out the remaining wrinkles
>
> Week  4 (Jul  8. - Jul 14.)
> ~~
> - basic ``CompositeField`` functionality, ``CompositeValue`` and
>   descriptor protocol
>
> Week  5 (Jul 15. - Jul 21.)
> ~~
> - ``db_index``, ``unique`` and ``primary_key`` for ``CompositeField``
> - backend-dependent SQL for ``__in`` lookups on ``CompositeField``
>
> Week  6 (Jul 22. - Jul 28.)
> ~~
> - the few patches required to make admin work with composite primary
>   keys
>
> Week  7 (Jul 29. - Aug  4.)
> ~~
> - composite ``ForeignKey``: basic functionality, descriptors, database
>   JOINs
>
> Week  8 (Aug  5. - Aug 11.)
> ~~
> - composite ``ForeignKey``: customization of auxiliary fields,
>   ad

[GSoC 2013] Composite fields: first draft of the proposal

[Michal Petrucha]

Hello,

as I promised a week ago, I'm posting a draft of my proposal. It is
far from complete, specifically, it does not contain technical details
on how things are going to be implemented. I'll try to fill these in
as soon as possible, although I wrote more on most of them in my
previous emails to this list. The most important thing is that it
contains a timeline and a description of possible fallbacks.

The proposal is accessible as a public gist [1] where it will be
updated in the future, I'm posting the full text here as well for
convenience.

Have a nice weekend.

Michal

[1] https://gist.github.com/koniiiik/5408673


Composite Fields, vol. 2
""""""""""""""""""""""""

Aim of this project
===

Since I started working on this two years ago, I managed (with the
help of a few people) to get most of the hard work done. The biggest
part that I didn't get around to implementing was support in the ORM
for multi-column joins. This has, however, been implemented recently
by Jeremy Tiller and Anssi, which means there are only a few things
left to be done.

First of all, the code sitting in my github repo is badly out of date,
which means it needs to be updated to the current state of master.
While I'm at it, I also want to clean up the revision history to get
it as close to a state where it could be just reviewed and merged
directly as possible.

Beside this, there are only the juicier features left that we
initially wanted to leave unsupported and get back to them later.
Those are the following (in no particular order):

- ``GenericForeignKey`` support for ``CompositeField``
- more intelligent handling of ``__in`` lookups
- database instrospection and ``inspectdb``
- detection of a change of the primary key in model instances

Timeline


Our exam period starts on May 20. and ends on June 28., which means
that I can't guarantee I will be able to fully dedicate myself to the
project for the first two weeks, however, if nothing goes wrong, I
should be able to pass all exams before June 17.

Also, I intend to go to EuroPython, which means the first week of July
won't be as fruitful as the others, but otherwise I'm ready to work
full time on the project.

Week  1 (Jun 17. - Jun 23.)
~~
- porting the required virtual field changes, like a ``VirtualField``
  class and ``django.db.models.options`` changes
- revisiting the documentation I wrote two years ago to reflect the
  evolution this project has gone through

Week  2 (Jun 24. - Jun 30.)
~~
- porting the virtual ``ForeignKey`` patchset on top of master to get
  most of the functionality right

Week  3 (Jul  1. - Jul  7.)
~~
- EuroPython 2013, Florence
- I intend to spend the full two days of sprints working on this but I
  can't promise much more during this week.
- running through the tests and fixing those that need updating,
  ironing out the remaining wrinkles

Week  4 (Jul  8. - Jul 14.)
~~
- basic ``CompositeField`` functionality, ``CompositeValue`` and
  descriptor protocol

Week  5 (Jul 15. - Jul 21.)
~~
- ``db_index``, ``unique`` and ``primary_key`` for ``CompositeField``
- backend-dependent SQL for ``__in`` lookups on ``CompositeField``

Week  6 (Jul 22. - Jul 28.)
~~
- the few patches required to make admin work with composite primary
  keys

Week  7 (Jul 29. - Aug  4.)
~~
- composite ``ForeignKey``: basic functionality, descriptors, database
  JOINs

Week  8 (Aug  5. - Aug 11.)
~~
- composite ``ForeignKey``: customization of auxiliary fields,
  adapting ``ModelForms`` and the admin in case it is necessary

Week  9 (Aug 12. - Aug 18.)
~~~~~~~~~~
- ``GenericForeignKey`` and ``GenericRelation`` support

Week 10 (Aug 19. - Aug 25.)
~~
- database introspection: create composite fields where necessary

Week 11 (Aug 26. - Sep  1.)
~~
- database introspection: composite ``ForeignKey``

Week 12 (Sep  2. - Sep  8.)
~~
- better handling of modifications to primary keys: detecting the fact
  and issuing an appropriate DB query

Week 13 (Sep  9. - Sep 15.)
~~
- revision log cleanup
- better handling of modifications to primary keys: cascading primary
  key updates

Week 14 (Sep 16. - Sep 22.)
~~
- this is after the suggested “soft pencils down” deadline, therefore
  I'll keep this week as a reserve

Deliverables and fallbacks
==

As the timeline shows, the first few weeks will be dedicated to
internal refactors of the relationship handling parts of the ORM,
which means the outcome won't be entirely evident or usefu

Re: A second stab at an implementation of composite fields

[Michal Petrucha]

On Wed, Apr 17, 2013 at 03:49:11AM -0700, Anssi Kääriäinen wrote:
> The basic idea is that there is a new ForeignObject class. A
> ForeignObject basically just takes a related model, and from_fields
> and to_fields which are the names of the fields used for the relation.
> Then, the ORM knows how to create JOINs, subqueries etc based on a
> ForeignObject. The ForeignObject itself is a virtual field, it doesn't
> have any concrete fields in the DB.
> 
> There is currently zero public API support for using the
> ForeignObject. The addition of ForeignObject is there to make the work
> on composite fields and multicolumn foreign keys in particular easier.

Thanks for the overview. For the sake of completeness, we had a
discussion on IRC earlier today, the outcome of which is that the
ForeignObject refactor will simplify the task immensely. We also came
upon a few concerns which I'll describe in a moment.

> I took a look at the code and I think the .Meta fields implementation
> is good. (Minor point is that there is no need for the field.virtual
> flag - it should be enough to check if it has a db column or not).

Agreed. Actually, the way I tried to go about the implementation, was
to replace field.column with field.columns which would be a tuple and
in case of virtual fields this would be a property which automatically
gathers the columns of all of its enclosed concrete fields, which
means a field.virtual would be necessary, but it looks like this
brings an inappropriate amount of complexity with little benefit.

> Do you have any plans for updatable primary keys? Updatable primary
> keys would be useful: the current api where obj.lastname = newval;
> obj.save() will result in save of new object instead of update of the
> lastname is plain ugly. OTOH This problem might be too complex to
> include in GSoC - cascading the updates is going to be hard. One
> possible solution is to add some way to create the foreign keys as ON
> UPDATE CASCADE and let the DB solve the problem.

Now I get to the issues that I still haven't figured out or would like
the opinion of others on. I'll separate them into sections for better
readability.

Updatable primary keys
--

The one you mention here is the way the ORM finds out whether to
perform an INSERT or UPDATE on model.save(). The current way this
works is just fine for models with an AutoField which is not visible
to the user. The problem is that as soon as you use an explicit
primary key and aren't extremely careful, you'll get unexpected
results.

This isn't directly related to composite primary keys, however, those
make it more apparent. I actually raised this issue two years ago
(IIRC) and I think I discussed it with Carl and possibly others; the
outcome was that we'd just put a note into the docs that would warn
people to be careful not to make any part of primary key user-editable
in the admin and generally be cautious when handling them directly or
otherwise bad things would happen.

I still think it would be good to change this behavior. The downsides
are that
 1) it would subtly change behavior for models with explicit primary
keys in a backwards-incompatible way
 2) it would require models to hold something like "the last known
value of the primary key in DB"
 3) it would cause weird behavior in situations like if you have two
separate instances corresponding to the same DB row, modify the PK
in one and save. I don't really know how much of an issue this
would be.
 4) as Anssi said, we would need a cascading mechanism for updates.
With certain backends, this could probably be done by the database
itself, with others, I believe we'd have to do it manually
    (SQLite?)...

Again, this is not entirely related to this project, but it would
still be nice to get this done before composite fields are released,
should we agree we want this implemented.

GenericForeignKey
-

Two years ago we gave this some thought and decided to leave this for
a later stage. I think the later stage is here.

It's fairly easy to represent composite values as strings, something
like the following works quite well::

",".join(escape(value) for value in composite_value)

Of course, we can just do this and stick it into the database as the
object_id. The problem is with GenericRelation. This is something that
works on the database level and we can't just ask the database server
to compare a string built this way with a tuple of other values.

Ideally, we'd need to come up with an encoding that would be possible
to reproduce as a SQL expression, preferably with the same result on
all DB backends. (The expression itself can be backend-dependent, the
result should probably not.)

As far as I know, Simone Federici had something that was close to what
we need, but IIRC, it had some issue

Re: A second stab at an implementation of composite fields

[Anssi Kääriäinen]

On 12 huhti, 18:34, Michal Petrucha  wrote:
> On Fri, Apr 12, 2013 at 07:35:45AM -0700, Anssi Kääriäinen wrote:
> > On 12 huhti, 16:44, Michal Petrucha  wrote:
> > ForeignKeys have been changed a lot since 2012-11-04. The introduction
> > of ForeignObject (which is base for ForeignKey) means that there is
> > support for multicolumn joins in the ORM now, and that ForeignObject
> > itself is a non-concrete field (not in ._meta.virtual_fields though).
> > ForeignObject currently works only on ORM level, so there is no API
> > for creating multicolumn foreign keys.
>
> > In any case, the GSoC proposal should take the introduction of
> > ForeignObject in account. tests/foreign_object contain some examples
> > related to ForeignObject functionality, exploring how those examples
> > work is a good idea.
>
> This has been on my TODO list for a few days actually, I was going
> through the work done regarding #19385 [1] and noticed this. However,
> I still need to give it a more thorough examination to fully get a
> grasp on this.

The basic idea is that there is a new ForeignObject class. A
ForeignObject basically just takes a related model, and from_fields
and to_fields which are the names of the fields used for the relation.
Then, the ORM knows how to create JOINs, subqueries etc based on a
ForeignObject. The ForeignObject itself is a virtual field, it doesn't
have any concrete fields in the DB.

There is currently zero public API support for using the
ForeignObject. The addition of ForeignObject is there to make the work
on composite fields and multicolumn foreign keys in particular easier.

> > Another big issue to solve is how to store the fields in model._meta.
> > IMO a good idea is to add all fields into some attribute ('all_fields'
> > as name might be good...), then separate the fields for different use
> > cases based on features the fields has. For example concrete_fields is
> > [f for f in self.all_fields if f.column], form fields are [f for f in
> > self.all_fields if f.form_field] (or maybe this should be called
> > logical fields - a multicolumn foreign key is logical, but not
> > concrete field, and the fields used by the foreign key are concrete
> > but not logical fields). The different field lists can be
> > cached_properties so there will be no problems from performance
> > perspective.
>
> As a matter of fact, this is one of the first things I did back in
> 2011. (-: Almost exactly like that -- _meta.fields contains all fields
> (except for M2M), _meta.concrete_fields, _meta.virtual_fields and I
> think I also needed a _meta.local_concrete (for non-inherited fields;
> I couldn't find a better name for it that wouldn't be too verbose).

I took a look at the code and I think the .Meta fields implementation
is good. (Minor point is that there is no need for the field.virtual
flag - it should be enough to check if it has a db column or not).

Do you have any plans for updatable primary keys? Updatable primary
keys would be useful: the current api where obj.lastname = newval;
obj.save() will result in save of new object instead of update of the
lastname is plain ugly. OTOH This problem might be too complex to
include in GSoC - cascading the updates is going to be hard. One
possible solution is to add some way to create the foreign keys as ON
UPDATE CASCADE and let the DB solve the problem.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: A second stab at an implementation of composite fields

[Michal Petrucha]

On Fri, Apr 12, 2013 at 07:35:45AM -0700, Anssi Kääriäinen wrote:
> On 12 huhti, 16:44, Michal Petrucha  wrote:
> ForeignKeys have been changed a lot since 2012-11-04. The introduction
> of ForeignObject (which is base for ForeignKey) means that there is
> support for multicolumn joins in the ORM now, and that ForeignObject
> itself is a non-concrete field (not in ._meta.virtual_fields though).
> ForeignObject currently works only on ORM level, so there is no API
> for creating multicolumn foreign keys.
> 
> In any case, the GSoC proposal should take the introduction of
> ForeignObject in account. tests/foreign_object contain some examples
> related to ForeignObject functionality, exploring how those examples
> work is a good idea.

This has been on my TODO list for a few days actually, I was going
through the work done regarding #19385 [1] and noticed this. However,
I still need to give it a more thorough examination to fully get a
grasp on this.

> Another big issue to solve is how to store the fields in model._meta.
> IMO a good idea is to add all fields into some attribute ('all_fields'
> as name might be good...), then separate the fields for different use
> cases based on features the fields has. For example concrete_fields is
> [f for f in self.all_fields if f.column], form fields are [f for f in
> self.all_fields if f.form_field] (or maybe this should be called
> logical fields - a multicolumn foreign key is logical, but not
> concrete field, and the fields used by the foreign key are concrete
> but not logical fields). The different field lists can be
> cached_properties so there will be no problems from performance
> perspective.

As a matter of fact, this is one of the first things I did back in
2011. (-: Almost exactly like that -- _meta.fields contains all fields
(except for M2M), _meta.concrete_fields, _meta.virtual_fields and I
think I also needed a _meta.local_concrete (for non-inherited fields;
I couldn't find a better name for it that wouldn't be too verbose).


[1] https://code.djangoproject.com/ticket/19385


signature.asc
Description: Digital signature

Re: A second stab at an implementation of composite fields

[Anssi Kääriäinen]

On 12 huhti, 16:44, Michal Petrucha  wrote:
 > As far as relationship fields go, we tried to ignore them at first
and
> get back to them during the second half of GSoC. Two approaches were
> considered, one was to special-case CompositeField targets in
> ForeignKey and in this case, make a single ForeignKey field manage
> multiple database columns directly. This turned out to be really
> painful and messy, so we tried another path, which was to turn
> ForeignKey into a virtual field and create an auxiliary copy of the
> target field in the local model which is supposed to manage the
> database column and its raw value. This way, ForeignKey only takes
> care of the higher-level relationship stuff.
>
> A large part of the ForeignKey refactor has been done. However, I was
> doing it on top of the CompositeField patchset, which makes it a real
> PITA to keep it in sync with master. I had managed to keep it in sync
> for about a year but this has become increasingly tedious, especially
> with all the Py3k changes and recent ORM improvements and cleanups.
> Currently my code is rebased on top of a commit from 2012-11-04, which
> is already five months in the past.

ForeignKeys have been changed a lot since 2012-11-04. The introduction
of ForeignObject (which is base for ForeignKey) means that there is
support for multicolumn joins in the ORM now, and that ForeignObject
itself is a non-concrete field (not in ._meta.virtual_fields though).
ForeignObject currently works only on ORM level, so there is no API
for creating multicolumn foreign keys.

In any case, the GSoC proposal should take the introduction of
ForeignObject in account. tests/foreign_object contain some examples
related to ForeignObject functionality, exploring how those examples
work is a good idea.

Another big issue to solve is how to store the fields in model._meta.
IMO a good idea is to add all fields into some attribute ('all_fields'
as name might be good...), then separate the fields for different use
cases based on features the fields has. For example concrete_fields is
[f for f in self.all_fields if f.column], form fields are [f for f in
self.all_fields if f.form_field] (or maybe this should be called
logical fields - a multicolumn foreign key is logical, but not
concrete field, and the fields used by the foreign key are concrete
but not logical fields). The different field lists can be
cached_properties so there will be no problems from performance
perspective.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[GSoC 2013] A second stab at an implementation of composite fields

[Michal Petrucha]

Hello everyone,

Hopefully some of you still remember me as the guy who's been trying
to implement support for composite primary keys by means of composite
model fields.

I haven't provided a whole lot of information on the progress of this
project for... quite a long time, so I'll try to rectify this now and
explain the current standing.

A quick summary of GSoC 2011: I started by implementing
CompositeField, which only acts as a dumb container. The obvious
problem was that it wouldn't work with relationship fields, but other
than that, it works just fine. Most importantly, using it as a primary
key should work just fine.

As far as relationship fields go, we tried to ignore them at first and
get back to them during the second half of GSoC. Two approaches were
considered, one was to special-case CompositeField targets in
ForeignKey and in this case, make a single ForeignKey field manage
multiple database columns directly. This turned out to be really
painful and messy, so we tried another path, which was to turn
ForeignKey into a virtual field and create an auxiliary copy of the
target field in the local model which is supposed to manage the
database column and its raw value. This way, ForeignKey only takes
care of the higher-level relationship stuff.

A large part of the ForeignKey refactor has been done. However, I was
doing it on top of the CompositeField patchset, which makes it a real
PITA to keep it in sync with master. I had managed to keep it in sync
for about a year but this has become increasingly tedious, especially
with all the Py3k changes and recent ORM improvements and cleanups.
Currently my code is rebased on top of a commit from 2012-11-04, which
is already five months in the past.

It might be the case that it's simply because of me abusing git in a
wrong way, but I've been keeping my work as a patchset on top of
master using git rebase -- it helps me keep track of the progress I
did and it is easier for me to find the commit where I did a certain
change and why I did it in case I'm not sure what to do when a merge
conflict appears. (If anyone is willing to teach me a better workflow,
I'd be more than happy to take advice from more experienced users.)

The more I think about it, the less I am convinced that trying to keep
my patchsets in sync with master is worth the effort. When I
implemented CompositeField itself, not having a clear idea of how
ForeignKey would be brought into the equation and what kinds of
abstraction would be required later, I implemented multiple things
(like database creation or SQL WHERE statements) in a way that worked
at that time, but require a more general handling with the
abstractions introduced in the ForeignKey refactor. This basically
means that I first implemented CompositeField and later broke most of
its internal handling within the ORM (which is one of the reasons why
keeping this in sync with master is difficult).

The course of action I'd suggest at this moment is to first port the
ForeignKey refactor on top of current master and make sure everything
is perfectly backwards compatible. This would also create all the
internal abstractions required to implement CompositeField, which
means porting CompositeField on top of the ForeignKey refactor will be
easier (at the very least, it will require less internal changes).


As the past few years have shown, this project is far too
time-consuming for an individual to make any significant progress just
working on it in one's free time. Therefore I'm proposing to do this
as part of the upcoming GSoC. That would mean I'd be working on it
full-time for three months which should be enough to finally push this
over the edge and at least finish this code-wise (and
documentation-wise, of course), hopefully even get the revision
history into a state where it'd be ready for review and eventually
inclusion into master.

If there are no serious objections to the proposed plan, in a couple
of days I'll start working on a timeline for the GSoC project and I'll
also write down the few more serious implementation issues I still
don't really know how to approach.


Have a great weekend everyone.

Michal


signature.asc
Description: Digital signature

Re: [GSoC composite fields] Late post-GSoC check-in #1

[Michal Petrucha]

First off, let me apologize for not reporting progress after the SoC
has ended. At first I thought I'd relax for a few days, but then the
work started piling up and I think anyone can guess how it went from
that point on... So, again, my apologies.

Second, I want to say a big thank you to the whole of the core team
for making this whole thing possible for me. Really, I can't thank you
enough, guys. Seriously, there seem to be no words strong enough for
this.

And third, the status of my project.

I implemented auxiliary field creation for ForeignKeys to the point
where it mostly works with regular targets. Mostly, because there are
still features that don't play well with my changes. At the moment,
according to the test suite, these are:
- ForeignKey subclassing
- ModelFormSets
- some other forms related code
- inspectdb
- deferred fields
- introspection

Also, some required changes broke most of what I had done during the
first phase, but I know exactly what happened and I'll get to it as
soon as the rest is ironed out.

Some of these failures will be fixed trivially by updating the tests,
some are harder to sort out. For instance, it took me about two weeks
of irregular work getting deferred fields to work at least in most
cases (until I decided I'd rather go for a probably suboptimal but
much simpler solution) but it still fails two mysterious test cases.

Anyway, at first I was getting several hundred test failures. I was
able to cut it down to 23 (on sqlite) but the process is getting more
and more tedious.

Thing is, I was lucky enough to get into one of our school's teams for
the upcoming ACM ICPC regionals, which means I'll have to spend most
of this month training and coding and training and I really won't have
enough free time to work on this feature.

Be that as it may, I'm definitely *not* abandoning the project, merely
postponing my work on it. I'll still try to keep it in sync with
trunk. If anyone feels like taking a look at it and trying to fix
a problem or two, I'll gladly help out, review patches etc., I just
don't have the resources to work on it on my own.

Thanks for bearing with me.

Michal

[1] https://github.com/konk/django


signature.asc
Description: Digital signature

Re: [GSoC composite fields] Weekly check-in #1

[Michal Petrucha]

Short summary of last week:
The first challenge was to make auxiliary field creation work with
ForeignKey targets. The first attempt was to move parts of the model
preparation mechanics to AppCache.

This turned out to be a dead end because when AppCache is being
populated, while importing models other things also get imported, like
ModelAdmin -- things that are often defined in the same modules as
models. These in turn require the models from contrib apps to be ready
on import.

So the other option was chosen, a field_prepared signal. After a few
days of work I can finally at least run the test suite with auxiliary
FK fields. The results show there's still a long road ahead of me, on
sqlite, I get 306 errors and 19 failures, but I suspect this can be
reduced drastically by fixing fixture loading.

That's about it for now, the next week will be a busy one.

Michal

[1] https://github.com/konk/django


signature.asc
Description: Digital signature

[GSoC composite fields] Weekly check-in #12 (Eh, is it that late already?)

[Michal Petrucha]

My first attempt at auxiliary field creation ended up worse than
expected, it turned out I had to iron out a few other issues, like
concrete model inheritance.

Anyway, I also encountered a problem with ForeignKeys whose target
field is also a ForeignKey. The latter has to be set up first in this
case and the machinery that's already in place doesn't really allow me
to ensure this. I'll probably need to define a new signal to be sent
after a ForeignKey has been fully initialized, I hope noone will mind.

Other than that, at least the basic support should be all right;
expect more fixes tomorrow.

(Note, if anyone is following and didn't notice yet, the auxiliary
field stuff is in a new branch in my repo. Just to avoid possible
confusion. (-; )

[1] https://github.com/konk/django


signature.asc
Description: Digital signature

Re: [GSoC composite fields] Weekly check-in #1

[Michal Petrucha]

The last week I've been looking into related fields and all that
stuff. As it turns out, the issue is much more complex than I
originally anticipated and at the moment I have some doubts whether
this can be achieved as part of this GSoC.

Basically, there are two approaches to the task:

1) Make ForeignKey and friends the field that manages all its rows
   (i. e. it will return multiple columns when asked for them and so
   on).

2) ForeignKey will be just a virtual field and it will create all
   required auxiliary fields in the local model. ForeignKey would then
   just handle the relationship stuff and leave everything else to the
   aux fields.

Some notes about both of them (I spent a few days trying to make
either support at least some basic features and both seem bloody
complex to me):

1) The changes required to ForeignKey itself can be kept to a minimum
   but this would require drastic changes in many parts of the
   internal code:

* a complete rewrite of the database creation code

* probably also a rewrite of the parts that match rows fetched
  from the database to model fields

* many internal changes in query code to support multi-column
  fields

   Basically, the problem is that practically all parts of the code
   rely heavily on the fact that each local field is backed by exactly
   one database column (except for M2M which is often special-cased).

   Now, all of this code would need to be rewritten to also work with
   fields spanning several database columns. I got completely lost
   somewhere around SQLCompiler.resolve_columns and
   DatabaseOperations.convert_values, though this is all just the tip
   of the iceberg that I encountered while looking into raw querysets;
   there is much more to it for regular ones.

2) This would require an extensive refactor of related fields. I can
   imagine making the aux field sit at ForeignKey's attname to manage
   the actual value. This would give us creation and row matching
   practically for free, but again, some internal query changes would
   still be necessary (multi-column joins, for one).

   The change could be made backwards-compatible if we made the
   default aux field use the ForeignKey's db_column.

Of course, it might be possible to make a half-assed hacky solution,
i. e. ForeignKey would be a full-featured field in some cases and a
virtual one otherwise but this would make a total mess out of
everything and it would require a serious amount of nasty hacks and
workarounds.

At any rate, I don't feel competent to make the decision in this
matter and I honestly believe there ought to be some discussion about
which route we'll take.

My personal favorite is the second option but I can imagine people not
liking code that adds local fields automagically. On the other hand,
there is already one such case (the id AutoField).

Anyway, now is the time that I'd like to see some comments and
opinions of other people who know the ORM code.

Michal


signature.asc
Description: Digital signature

[GSoC composite fields] Weekly check-in #7

[Michal Petrucha]

Basic support in the admin is now working: adding, editing and
deleting single instances with composite primary keys.
Obviously, there's no inline support yet since relationship fields are
not yet ready.

Michal

[1] https://github.com/konk/django


signature.asc
Description: Digital signature

[GSoC composite fields] Weekly check-in #6

[Michal Petrucha]

Hi folks,

Sorry about this delayed post. My repo now supports using
CompositeField as a model's primary key, as far as models are
concerned, that means, Model.pk and friends.

For the next week, expect admin to work. THe work is underway, nothing
worthy of publishing just yet.

Michal

[1] https://github.com/konk/django


signature.asc
Description: Digital signature

Re: [GSoC composite fields] Weekly check-in (this should be #5, right...?)

[Michal Petrucha]

Hi,

some visible progress on my project at long last. I spent most of the
last week digging deep inside the ORM's entrails to make composite
field lookups possible and finally it looks promising.

While working on this I found out the extra_filters approach I
intended to use was a dead end (which reminded me of what Russ wrote
in response to my proposal: "I'm almost completely certain you'll
find some gremlin lurking underneath some dark corner of the code").

In the end I had to make virtual fields part of the field list in
model options which required updating code in quite a few places to
reflect the change. Most of it should be all right by now, the test
suite runs well except for four GFK and admin related tests which I'll
look at tomorrow.

Setting a CompositeField's primary_key option still isn't supported,
however, most of it should already be prepared.

Watch my progress at GitHub [1].

Michal

[1] https://github.com/konk/django


signature.asc
Description: Digital signature

[GSoC composite fields] Another weekly check-in

[Michal Petrucha]

Another weekly update.

First off, I'd like to apologize for not posting one last week; I've
been completely buried under exams and other school-related stuff
during the past three weeks...
Things have settled now at last so I'm back at my SoC and cathing up.

My repo [1] now contains support for index creation over
CompositeFields and the ``unique`` attribute just results in the
fields being added to the ``unique_together`` model option.

About the code that generates the CREATE INDEX SQL, though, I haven't
found any way to properly test that (the tests usually don't mess with
backend-dependent SQL code). I'd expect there are some tests for this,
it's just that I haven't found them yet. I'd appreciate any pointers
in the right direction.

Slightly off-topic: I'm using rebases in my repo, mainly to see my set
of changes on the top (and not scattered around the history). Is it
okay if I continue using them or am I complicating someone else's work
with this?

Michal

[1] https://github.com/konk/django


signature.asc
Description: Digital signature

Re: [GSoC composite fields] Weekly check-in #1

[Michal Petrucha]

All right, time for another check-in.

This isn't a lengthy one -- unfortunately last Tuesday my laptop
failed me, leaving me with no replacement, which means I have to wait
for it to return from the service center to be fully up and running.

In the meantime, I managed to borrow an old one a couple of days ago
but it isn't set up yet; I'm working on that. At least I can keep my
repo in sync with trunk at the moment...

Anyway, I hope the next update will be a more productive one. Until
then...

Michal

> [1] https://github.com/konk/django


signature.asc
Description: Digital signature

[GSoC composite fields] Weekly check-in #1 (well, first after the coding has begun)

[Michal Petrucha]

Checking in with some actual results. At the moment, my repo [1]
contains a bit of documentation (look at the model field reference and
the model topic guide) and a basic implementation with some trivial
tests.

Some remarks:
* I ended up using namedtuple anyway, providing a fallback
  implementation for 2.5 taken from the Python docs, because after
  quite some time trying to figure out the best approach I concluded I
  would have to pretty much reimplement the same thing -- no need to
  reinvent the wheel.
* In my RFC [2] I apparently forgot about some things, like the new
  ForeignKey option you'll notice in my docs patch. I'm not too happy
  about its name, suggestions are welcome. (-:
* I'm not fully satisfied with the placement of the new section in the
  topic guide either -- it does reference relationships so it would
  be weird to place it directly after the AutoField section but it is
  more related to the AutoField section as far as the topic goes. Ah
  well, this is probably just a bikeshed anyway. (-;
* The docs will definitely need at least one more section in the
  reference, a discussion on supported lookup filters; I intend to
  write that in the near future.

What is possible with the implementation in its current state?
Basically, it can only do three things: add itself to a model, compose
a namedtuple with a value when retrieved and set the underlying fields
when assigned.
The next thing on the list: model validation, syncdb and database
indexes.

That's it for this week, suggestions, complaints, remarks and such are
welcome.

Michal

[1] https://github.com/konk/django
[2] https://groups.google.com/d/topic/django-developers/rF79c8z65cQ/discussion


signature.asc
Description: Digital signature

Re: RFC: Composite fields API

[Michael P. Jung]

About one year ago I wrote a CompositeField implementation, which I
proposed as a clean way of grouping fields together, while making it
easy to reuse that composition of field:

https://bitbucket.org/mp/django-composite-field

It's a slightly different approach and is centered around defining a
field which is composed of multiple field and can be reused in multiple
models.


One could also call that field MultiField, MixinField, GroupField,
FieldSet,...


I proposed this to be added in Django during the DjangoCon Europe
sprints, but sadly it didn't gain any attention. As a result I just
prepared a release on pypi including some documentation and test cases.


I'm very sorry that my comment to this RFC comes so late. I have
subscribed the django-developers mailing list, but don't read it on a
regular basis.


--mp

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: RFC: Composite fields API

[akaariai]

On May 17, 5:32 pm, Michal Petrucha  wrote:
> Proper subquery support is something that can be addressed once the
> rest of the implementation is stable.

To me the plan looks very reasonable (both disallowing subqueries and
converting to disjunction form), unless there is some part in the
internals which expects pk__in=qs to work. In that case it could just
be converted to something like:
if pk is multipart_pk:
qs = list(qs.values_list('pk_part1', 'pk_part2'))
continue as now.

In any case, in my opinion pushing as much of this work to later
patches is the way to go. The only question is how much can be pushed
to later patches. I do not know the answer to that, unfortunately...

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: RFC: Composite fields API

[Michal Petrucha]

On Tue, May 17, 2011 at 02:05:10AM -0700, akaariai wrote:
> On May 12, 2:41 pm, Michal Petrucha  wrote:
> > Due to the nature of this field type, other lookup filters than
> > ``exact`` and ``in`` would have unclear semantics and won't be
> > supported. The original plan was to also exclude support for ``in``
> > but as it turns out, ``in`` is used in several places under the
> > assumption that primary keys support it, for example DeleteQuery
> > or UpdateQuery. Therefore both filters will be implemented.
> 
> I wonder how to implement __in lookups in SQLite3. SQLite3 doesn't
> support where (col1, col2) in ((val3, val4),(val5, val6)). But other
> DBs do (at least MySQL, Oracle and PostgreSQL). I do not know what
> would be the best way to write something equivalent in SQLite3. The
> obvious choice is to rewrite it as an OR lookup (as mentioned in the
> full proposal). Maybe write it as an OR lookup for every DB for the
> initial patch, and later on this can be improved to have per database
> handling.

You're right, SQLite3 is the troublemaker here and the reason I wanted
to leave these lookups out initially.

Well, this depends on the level at which these lookups will be
handled. The doable, albeit somewhat hacky way is to handle
this when creating the SQL string for IN, recognize composite lookups
and turn them into a disjunction. The more robust and probably
"proper" way would be to delegate this to the database backend. The
backend could then decite whether it wants
(col1, col2) IN ((val1, val2), (val3, val4))
or
((col1 = val1) AND (col2 = val2)) OR ((col1 = val3) AND (col2 = val4))
This would, however, require enhancing the backend interface.

I think I'll go with the first option, the second one would require
even more non-trivial design decisions to be made regarding the
backend interface and I think I have enough on my plate anyway.

> In lookups with subselects are a harder problem. Those would
> need to be rewritten as joined subselects with a distinct clause. [1]
> Not in lookups could be still harder due to weird null handling. (1
> not in (null) -> Unknown). [2]
> 
> I hope there will be an easy solution to this problem, as this feature
> is something which would be really, really valuabe for Django (no more
> telling DBAs: by the way, no composite foreign keys...). One simple
> solution would be to disallow __in lookups with subselects (or run the
> subselects separately) and use OR lookups when given a list of values.
> This should be relatively easy to implement and could be improved
> later on.

Uh oh. This is black magic, probably heavily backend-dependent, too. I
can tell for sure that I don't intend to incorporate any subquery
support for composite lookups for now. Trying to do a composite __in
lookup using a subquery will probably just throw an exception for now,
the user will be required to evaluate it himself.

Proper subquery support is something that can be addressed once the
rest of the implementation is stable.

Michal


signature.asc
Description: Digital signature

Re: RFC: Composite fields API

[akaariai]

On May 12, 2:41 pm, Michal Petrucha  wrote:
> Due to the nature of this field type, other lookup filters than
> ``exact`` and ``in`` would have unclear semantics and won't be
> supported. The original plan was to also exclude support for ``in``
> but as it turns out, ``in`` is used in several places under the
> assumption that primary keys support it, for example DeleteQuery
> or UpdateQuery. Therefore both filters will be implemented.

I wonder how to implement __in lookups in SQLite3. SQLite3 doesn't
support where (col1, col2) in ((val3, val4),(val5, val6)). But other
DBs do (at least MySQL, Oracle and PostgreSQL). I do not know what
would be the best way to write something equivalent in SQLite3. The
obvious choice is to rewrite it as an OR lookup (as mentioned in the
full proposal). Maybe write it as an OR lookup for every DB for the
initial patch, and later on this can be improved to have per database
handling. In lookups with subselects are a harder problem. Those would
need to be rewritten as joined subselects with a distinct clause. [1]
Not in lookups could be still harder due to weird null handling. (1
not in (null) -> Unknown). [2]

I hope there will be an easy solution to this problem, as this feature
is something which would be really, really valuabe for Django (no more
telling DBAs: by the way, no composite foreign keys...). One simple
solution would be to disallow __in lookups with subselects (or run the
subselects separately) and use OR lookups when given a list of values.
This should be relatively easy to implement and could be improved
later on.

 - Anssi

[1] 
http://asktom.oracle.com/pls/asktom/f?p=100:11:0P11_QUESTION_ID:953229842074
[2] 
http://asktom.oracle.com/pls/asktom/f?p=100:11:1089369944141559P11_QUESTION_ID:442029737684

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: RFC: Composite fields API

[Ian Clelland]

On Thu, May 12, 2011 at 5:16 PM, Carl Meyer  wrote:

>
> On 05/12/2011 06:41 AM, Michal Petrucha wrote:
>
> On Thu, May 12, 2011 at 02:49:03PM +0100, Tom Evans wrote:

> The value of a CompositeField will be represented by an instance of a
> > CompositeValue class. This will be a descendant of tuple and will
> > resemble namedtuple present in Python >= 2.5. It will support
> > iteration, numbered indexing and access to individual field values
> > using attributes corresponding to underlying field names. The order of
> > values will be the same as the order of fields specified in the model
> > definition.
>
> Yes, Tom is right of course - now that Python 2.5 is minimal version, we
> can just use namedtuple.
>

As far as I can tell, namedtuple was added in Python 2.6, not 2.5, so a
compatibility class may still be necessary.

http://docs.python.org/library/collections.html#collections.namedtuple

-- 
Regards,
Ian Clelland


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: RFC: Composite fields API

[Luke Plant]

On 12/05/11 12:41, Michal Petrucha wrote:

> 1) Leave out the ``unique`` option and live with ``unique_together``.
>This would pribably imply also leaving out ``db_index``, otherwise
>the API would be a complete mess.
> 
> 2) Allow ``CompositeField.unique`` but also keep ``unique_together``.
>The problem I see with this approach is that there would be two
>quite different ways to achieve the same effect.
> 
> 3) Make ``CompositeField.unique`` the way to go and deprecate
>``unique_together``.
>This way, specifying a unique constraint on a tuple of fields would
>work the same way it works on single fields which is IMO a
>significant benefit. There's, however, the issue of breaking
>backwards compatibility. Furthermore, one would have to add a new
>field, albeit virtual, just to create a simple constraint, which
>may seem weird to some.

I'd go with (2), we can easily live with these two different ways to do
something, because, from a given starting point, there is actually only
"one obvious way" to achieve what you want i.e. if you have a composite
field already, there is one obvious way to make it unique, and if you
have two separate fields, there is one obvious way to make them 'unique
together'.

> I'm also thinking about implementing an abstract class, VirtualField.
> This could be useful mainly as a base class for fields with no direct
> database column. That means, it would mainly handle things like
> add_to_class (adding itself to the list of virtual fields instead of
> local ones), specifying arbitrary lookup filters when asked for one
> etc. CompositeField could then be a descendant of this class.
> 
> However, I can't currently imagine any other use-case for this
> abstract class than CompositeField. The question is, then, is there
> any interest in having an abstract mechanism like this? Can anyone
> imagine a use-case? (The question is, should I implement this
> functionality directly inside CompositeField or factor it out into
> something more general?)

Feel free to make such a class if it makes development easier, but keep
it private/undocumented until we have at least 2 use cases.

Luke

-- 
"Procrastination: Hard work often pays off after time, but laziness
always pays off now." (despair.com)

Luke Plant || http://lukeplant.me.uk/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: RFC: Composite fields API

[Michal Petrucha]

> > 2) Allow ``CompositeField.unique`` but also keep ``unique_together``.
> >The problem I see with this approach is that there would be two
> >quite different ways to achieve the same effect.
> 
> I agree with Javier - I favor option 2. In my mind, although the final
> result at the database level may be the same (a unique index across
> multiple database columns), in conceptual terms at the ORM level it is
> really two different things. There are many cases where I want to
> specify that two fields should be unique together, but they really are
> two separate fields; I'm never going to want to access it as a single
> field or composite value. In this case, specifying a CompositeField
> would confuse the intent and be more verbose than unique_together. I
> think the conceptual distinction is clear, and it will actually be less
> confusing to users to have both options available than to have
> CompositeField become the only way to specify an index on multiple columns.

One point I forgot to mention in the original e-mail is that there
would be an inconsistency: creating a unique index will be possible
using either a unique CompositeField or unique_together where as a
non-unique index will be possible only with an explicit
CompositeField. At least as far as I know there is currently no option
to create a non-unique index over several columns.

> > One minor detail, should the field silently ignore invalid options or
> > should it issue warnings?
> 
> Explicit is better than implicit, and errors should never pass silently
> unless explicitly silenced. If the option is invalid, it should not just
> be a warning, it should be an outright failure (though if the check is
> expensive, it could possibly happen in model-validation rather than
> always at runtime).

Thanks for the pointer, model validation looks like the right place
for this.

> I mentioned this in an earlier thread, but I'd really like to see the
> API allow me to specify my own class as the value class, as long as it
> satisfies some basic API requirements. In my mind the long-term goal
> here is that GFKs should be reasonably implementable as a CompositeField
> or a CompositeField subclass without exploiting undocumented internal APIs.
> 
> If there are implementation complexities that push this feature out of
> scope for GSoC, that's fine - but I want to make sure we don't make that
> future expansion difficult by design choices we make now.

I'll make it possible to insert custom hooks into the
assignment/retrieval routines which should be sufficient for the
purposes of GFKs.

> [...]
> > I'm also thinking about implementing an abstract class, VirtualField.
> > This could be useful mainly as a base class for fields with no direct
> > database column. That means, it would mainly handle things like
> > add_to_class (adding itself to the list of virtual fields instead of
> > local ones), specifying arbitrary lookup filters when asked for one
> > etc. CompositeField could then be a descendant of this class.
> > 
> > However, I can't currently imagine any other use-case for this
> > abstract class than CompositeField. The question is, then, is there
> > any interest in having an abstract mechanism like this? Can anyone
> > imagine a use-case? (The question is, should I implement this
> > functionality directly inside CompositeField or factor it out into
> > something more general?)
> 
> I wouldn't spend time on something we don't have any use case in mind
> for (unless making this split makes the code easier to read and
> understand). This is something that most likely could easily be done
> later, if we find we need it.

Fair enough, unless it turns out to be a better approach anyway. (-:

Michal


signature.asc
Description: Digital signature

Re: RFC: Composite fields API

[Michal Petrucha]

On Fri, May 13, 2011 at 09:01:19AM -0700, onelson wrote:
> I'm not that familiar with GFK's and how they work in django, but I just 
> wanted to check... 
> Will we have (non-generic) FK support for this, or is that another 
> can-o-worms that won't get touched for some time?

Adding support for composite PK targets into related fields (i. e.
ForeignKey, OneToOne and ManyToMany) is something I intend to devote
the whole second half of the program to. That means, if all goes well,
this should be all right by the end of this summer.

Michal


signature.asc
Description: Digital signature

Re: RFC: Composite fields API

[onelson]

I'm not that familiar with GFK's and how they work in django, but I just 
wanted to check... 
Will we have (non-generic) FK support for this, or is that another 
can-o-worms that won't get touched for some time?

Regards,
Owen

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: RFC: Composite fields API

[Carl Meyer]

Hi Michal,

I'm looking forward to seeing this project take shape! Comments below:

On 05/12/2011 06:41 AM, Michal Petrucha wrote:
[..]
> The constructor of a CompositeField will require at least two
> positional parameters, each positional parameter will be a single
> atomic field. The order of this parameters will be important as
> explained below. The parameters will have to be field instances, lazy
> loading won't be necessary (the recommended place of composite field
> definitions will be after atomic fields).

This sounds fine.

> CompositeField will accept these three field options:
> - db_index (creates a multi-column index across the underlying fields)
> - primary_key (creates a composite primary key in the model)
> - unique (creates a unique constraint for the set of fields)
> 
> Other field options either wouldn't make sense or would be too
> difficult to implement.
> 
> There is a clash with the current API here, in the ``unique`` option.
> This would supersede the current ``unique_together`` Meta option. I
> see three options possible:
> 
> 1) Leave out the ``unique`` option and live with ``unique_together``.
>This would pribably imply also leaving out ``db_index``, otherwise
>the API would be a complete mess.
> 
> 2) Allow ``CompositeField.unique`` but also keep ``unique_together``.
>The problem I see with this approach is that there would be two
>quite different ways to achieve the same effect.
> 
> 3) Make ``CompositeField.unique`` the way to go and deprecate
>``unique_together``.
>This way, specifying a unique constraint on a tuple of fields would
>work the same way it works on single fields which is IMO a
>significant benefit. There's, however, the issue of breaking
>backwards compatibility. Furthermore, one would have to add a new
>field, albeit virtual, just to create a simple constraint, which
>may seem weird to some.

I agree with Javier - I favor option 2. In my mind, although the final
result at the database level may be the same (a unique index across
multiple database columns), in conceptual terms at the ORM level it is
really two different things. There are many cases where I want to
specify that two fields should be unique together, but they really are
two separate fields; I'm never going to want to access it as a single
field or composite value. In this case, specifying a CompositeField
would confuse the intent and be more verbose than unique_together. I
think the conceptual distinction is clear, and it will actually be less
confusing to users to have both options available than to have
CompositeField become the only way to specify an index on multiple columns.

> One minor detail, should the field silently ignore invalid options or
> should it issue warnings?

Explicit is better than implicit, and errors should never pass silently
unless explicitly silenced. If the option is invalid, it should not just
be a warning, it should be an outright failure (though if the check is
expensive, it could possibly happen in model-validation rather than
always at runtime).

> The value of a CompositeField will be represented by an instance of a
> CompositeValue class. This will be a descendant of tuple and will
> resemble namedtuple present in Python >= 2.5. It will support
> iteration, numbered indexing and access to individual field values
> using attributes corresponding to underlying field names. The order of
> values will be the same as the order of fields specified in the model
> definition.

Yes, Tom is right of course - now that Python 2.5 is minimal version, we
can just use namedtuple.

> Assigning a value to a CompositeField will be possible using any
> iterable as long as its length equals the number of atomic fields (and
> the values can be assigned to the corresponding fields, obviously).

I mentioned this in an earlier thread, but I'd really like to see the
API allow me to specify my own class as the value class, as long as it
satisfies some basic API requirements. In my mind the long-term goal
here is that GFKs should be reasonably implementable as a CompositeField
or a CompositeField subclass without exploiting undocumented internal APIs.

If there are implementation complexities that push this feature out of
scope for GSoC, that's fine - but I want to make sure we don't make that
future expansion difficult by design choices we make now.

[...]
> I'm also thinking about implementing an abstract class, VirtualField.
> This could be useful mainly as a base class for fields with no direct
> database column. That means, it would mainly handle things like
> add_to_class (adding itself to the list of virtual fields instead of
> local ones), specifying arbitrary lookup filters when asked for one
> etc. CompositeField could then be a descendant of this class.
> 
> However, I can't currently imagine any other use-case for this
> abstract class than CompositeField. The question is, then, is there
> any interest in having an abstract mechanism

Re: RFC: Composite fields API

[Javier Guerra Giraldez]

On Thu, May 12, 2011 at 8:49 AM, Tom Evans  wrote:
> unique/unique_together: They should both be supported. unique_together
> should raise a PendingDeprecationWarning, and it should disappear
> according to the deprecation timeline. unique_together only exists as
> a Meta option as there is no field to attach that logic to - now there
> is.

while i'm +1 about using unique on the composite field, i'm -0 about
deprecating unique_together.  there are times when i want to ensure
composite uniqueness but don't consider those fields as part o a
composite.  in those cases, i wouldn't want to define the composite
field just to say its unique.

yes, i know about the Python Zen on "only one obvious way", but in
this case it seems to be two semantically different things (even if
the generated SQL is the same).  for me, readability wins

-- 
Javier

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: RFC: Composite fields API

[Michal Petrucha]

On Thu, May 12, 2011 at 02:49:03PM +0100, Tom Evans wrote:
> Hi Michal
> 
> This looks really, really good. A few comments:

Thanks for the response.

> Value of a composite field: It should descend from namedtuple. From
> 1.4 onwards, Django only supports 2.5+, so it's not necessary to fudge
> things for Python 2.4

Ah, great. That would actually make things easier, namedtuple could be
used directly without any special class factory. At the time of
writing the proposal, IIRC, it wasn't clear 2.4 would be dropped.

> unique/unique_together: They should both be supported. unique_together
> should raise a PendingDeprecationWarning, and it should disappear
> according to the deprecation timeline. unique_together only exists as
> a Meta option as there is no field to attach that logic to - now there
> is.

That's also my point of view, however, there have been objections that
I mentioned (having to name the index). That's why I'd like to hear a
few more opinions.

> I also have a few questions:
> 
> If a model has a composite field that is marked as primary_key=True,
> how will this affect instance.pk? Presumably this will now return a
> tuple - will this affect automatic URL generation in the admin?

The instance.pk property will work the same way as it works for any
other field, i. e. it will be an alias for the field with
primary_key=True. In this case, the CompositeField. That means,
retrieval and assignment will happen using iterables.

The admin will handle this by extending quote and unquote -- one
character will be picked as a separator (probably comma) and quoted
inside the atomic values.

Looking at the code, it is already quoted anyway so the only thing
that will change is that for models with composite primary keys this
character will appear in the primary key value.

Michal


signature.asc
Description: Digital signature

Re: RFC: Composite fields API

[Tom Evans]

On Thu, May 12, 2011 at 12:41 PM, Michal Petrucha
 wrote:
> As most of you have probably noticed by now, in a week and a half I'll
> start working on the implementation of composite fields. Before that
> we should probably agree on the final form of the API.
>
> 

Hi Michal

This looks really, really good. A few comments:

Value of a composite field: It should descend from namedtuple. From
1.4 onwards, Django only supports 2.5+, so it's not necessary to fudge
things for Python 2.4

unique/unique_together: They should both be supported. unique_together
should raise a PendingDeprecationWarning, and it should disappear
according to the deprecation timeline. unique_together only exists as
a Meta option as there is no field to attach that logic to - now there
is.

I also have a few questions:

If a model has a composite field that is marked as primary_key=True,
how will this affect instance.pk? Presumably this will now return a
tuple - will this affect automatic URL generation in the admin?

Cheers

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

RFC: Composite fields API

[Michal Petrucha]

As most of you have probably noticed by now, in a week and a half I'll
start working on the implementation of composite fields. Before that
we should probably agree on the final form of the API.

This lengthy mail is mostly a recapitulation of things mentioned in
the past, like [1], [2], [3] and questions raised in these discussions
that have not been answered so far.


CompositeField will be a new type of model fields. This type will be
virtual, i. e. it won't be backed by any real database column by
itself. Instead, it will act as a proxy to a given set of atomic
fields and will be used to set options for and perform operations on
the whole set as one field.


The constructor of a CompositeField will require at least two
positional parameters, each positional parameter will be a single
atomic field. The order of this parameters will be important as
explained below. The parameters will have to be field instances, lazy
loading won't be necessary (the recommended place of composite field
definitions will be after atomic fields).

CompositeField will accept these three field options:
- db_index (creates a multi-column index across the underlying fields)
- primary_key (creates a composite primary key in the model)
- unique (creates a unique constraint for the set of fields)

Other field options either wouldn't make sense or would be too
difficult to implement.

There is a clash with the current API here, in the ``unique`` option.
This would supersede the current ``unique_together`` Meta option. I
see three options possible:

1) Leave out the ``unique`` option and live with ``unique_together``.
   This would pribably imply also leaving out ``db_index``, otherwise
   the API would be a complete mess.

2) Allow ``CompositeField.unique`` but also keep ``unique_together``.
   The problem I see with this approach is that there would be two
   quite different ways to achieve the same effect.

3) Make ``CompositeField.unique`` the way to go and deprecate
   ``unique_together``.
   This way, specifying a unique constraint on a tuple of fields would
   work the same way it works on single fields which is IMO a
   significant benefit. There's, however, the issue of breaking
   backwards compatibility. Furthermore, one would have to add a new
   field, albeit virtual, just to create a simple constraint, which
   may seem weird to some.

I don't feel like deciding in this and either one is fine as far as
the implementation is concerned.

One minor detail, should the field silently ignore invalid options or
should it issue warnings?


Moving on...


The value of a CompositeField will be represented by an instance of a
CompositeValue class. This will be a descendant of tuple and will
resemble namedtuple present in Python >= 2.5. It will support
iteration, numbered indexing and access to individual field values
using attributes corresponding to underlying field names. The order of
values will be the same as the order of fields specified in the model
definition.

Assigning a value to a CompositeField will be possible using any
iterable as long as its length equals the number of atomic fields (and
the values can be assigned to the corresponding fields, obviously).


Due to the nature of this field type, other lookup filters than
``exact`` and ``in`` would have unclear semantics and won't be
supported. The original plan was to also exclude support for ``in``
but as it turns out, ``in`` is used in several places under the
assumption that primary keys support it, for example DeleteQuery
or UpdateQuery. Therefore both filters will be implemented.


This should be everything as far as the models API is concerned. As
for the other parts of Django, the changes will be kept to a working
minimum.

Forms: Only support in ModelChoiceFields will be added for composite
primary keys; there won't be any special form field type for now.

Admin: Again, only support for composite primary keys will be added in
the quoting/unquoting function to make it possible to access such
models.

GFK: For now, GenericForeignKey won't be able to reference models with
composite primary keys.


I'm also thinking about implementing an abstract class, VirtualField.
This could be useful mainly as a base class for fields with no direct
database column. That means, it would mainly handle things like
add_to_class (adding itself to the list of virtual fields instead of
local ones), specifying arbitrary lookup filters when asked for one
etc. CompositeField could then be a descendant of this class.

However, I can't currently imagine any other use-case for this
abstract class than CompositeField. The question is, then, is there
any interest in having an abstract mechanism like this? Can anyone
imagine a use-case? (The question is, should I implement this
functionality directly inside CompositeField or factor it out into
something more general?)


I'll really appreciate each comment.

Michal Petrucha

[GSoC composite fields] Weekly check-in #2

[Michal Petrucha]

Hello again,

time for the second weekly check-in.

I apologize for not having posted the API as I promised yet, there's a
lot of things going on in school. However, I started working on a
ticket [1] as we're supposed to for the interim period.

Not much else to report, hopefully the next week's check-in will be
richer.

Michal Petrucha


[1] http://code.djangoproject.com/ticket/5535


signature.asc
Description: Digital signature

[GSoC composite fields] Weekly check-in #1

[Michal Petrucha]

Hello everyone,

I hope the introductions aren't necessary since I've already
introduced myself in the past [1].

Just a recap, I'll be working, under the guidance of Jacob
Kaplan-Moss, on support for composite model fields which will allow
users to define models with composite primary keys. The full text of
my proposal is available for anyone interested to read [1][2].

You can reach me on #django-dev as 'konk', I'll be idling there
most of the time (unless the server running irssi crashes) and I tend
to check it quite often so feel free to ping me with any questions or
comments.

Same as Gregor and Jan, I created a fork on github [3] where I'll push
my progress. The branch name is, surprisingly,
soc2011/composite-fields. There's also the issue tracker for anyone
who prefers this form of communication.

In a couple days I'll hopefully post how I imagine the API with all
the bloody details. Just hang on those few days until I write it down.

I guess this is all I have to report for tonight, more coming soon.

Michal Petrucha


[1] 
http://groups.google.com/group/django-developers/browse_frm/thread/120d401e302f369b/82471d2407c87abf
[2] http://people.ksp.sk/~johnny64/GSoC-full-proposal
[3] https://github.com/konk/django


signature.asc
Description: Digital signature

Re: [GSoC] Composite fields: proposal draft 2

[Michal Petrucha]

I tried to incorporate the remarks into my proposal and I'm posting
the updated parts.

As usual, the full version is still available at
http://people.ksp.sk/~johnny64/GSoC-full-proposal

Retrieval and assignment


Jacob has actually already provided a skeleton of the code that takes care
of this as seen in [1]. I'll only summarize the behaviour in a brief
example of my own.

class SomeModel(models.Model):
first_field = models.IntegerField()
second_field = models.CharField(max_length=100)
composite = models.CompositeField(first_field, second_field)

>>> instance = new SomeModel(first_field=47, second_field="some string")
>>> instance.composite
CompositeObject(first_field=47, second_field='some string')
>>> instance.composite.first_field
47
>>> instance.composite[1]
'some string'
>>> instance.composite = (74, "other string")
>>> instance.first_field, instance.second_field
(74, 'other string')

Accessing the field attribute will create a CompositeObject instance which
will behave like a tuple but also with direct access to enclosed field
values via appropriately named attributes.

Assignment will be possible using any iterable. The order of the values in
the iterable will have to be the same as the order in which undelying
fields have been specified to the CompositeField.


QuerySet filtering
~~

[...]

Afterwards the handling inside Query is pretty straightforward. For
CompositeFields (and virtual fields in general) there is no value to be
used in the where node, the extra_filters are responsible for all
filtering, but since the filter should apply to a single object even after
join traversals, the aliases will be set up while handling the "root"
filter and then reused for each one of the extra_filters.

This way of extending the extra_filters mechanism will allow the field
class to create conjunctions of atomic conditions. This is sufficient for
the "__exact" lookup type which will be implemented.

Of the other lookup types, the only one that looks reasonable is "__in".
This will, however, have to be represented as a disjunction of multiple
"__exact" conditions since not all database backends support tuple
construction inside expressions. Therefore this lookup type will be left
out of this project as the mechanism would need much more work to make it
possible.


GenericForeignKeys
~~

Even though the admin uses the contenttypes framework to log the history
of actions, it turns out proper handling on the admin side will make
things work without the need to modify GenericForeignKey code at all. This
is thanks to the fact that the admin uses only the ContentType field and
handles the relations on its own. Making sure the unquoting function
recreates the whole CompositeObjects where necessary should suffice.

At a later stage, however, GenericForeignKeys could also be improved to
support composite primary keys. Using the same quoting solution as in the
admin could work in theory, although it would only allow fields capable of
storing arbitrary strings to be usable for object_id storage. This has
been left out of the scope of this project, though.


Estimates and timeline
--

The proposed timeline is as follows:

week  1 (May 23. - May 29.):
- basic CompositeField implementation with assignment and retrieval
- documentation for the new field type API

week  2 (May 30. - Jun  5.):
- creation of indexes on the database
- unique conditions checking regression tests

week  3 (Jun  6. - Jun 12.):
- query code refactoring to make it possible to support the required
  extra_filters
- lookups by CompositeFields

week  4 (Jun 13. - Jun 19.):
- creation of a composite primary key
- more tests and taking care of any missing/forgotten documentation so far

week  5 (Jun 20. - Jun 26.):
- ModelForms support for composite primary keys

week  6 (Jun 27. - Jul  3.):
- full support in the admin

week  7 (Jul  4. - Jul 10.):
- fixing any documentation discrepancies and making sure everything is
  tested thoroughly
- exploring the related fields in detail and working up a detailed plan
  for the following changes

> midterm
  By the time midterm evaluation arrives, everything except for
  relationship fields should be in production-ready state.

week  8 (Jul 11. - Jul 17.):
- implementing composite primary key support in all the
  RelatedObjectDescriptors

week  9 (Jul 18. - Jul 24.):
- query joins refactoring

week 10 (Jul 25. - Jul 31.):
- support for ForeignKey relationship traversals

week 11 (Aug  1. - Aug  7.):
- making sure OneToOne and ManyToMany work as well

week 12 (Aug  8. - Aug 14.):
- writing even more tests for the relationships
- finishing any missing documentation

> pencils down


signature.asc
Description: Digital signature

Re: [GSoC] Composite fields: proposal draft 1, full version

[Michal Petrucha]

On Thu, Apr 07, 2011 at 03:36:52PM +0800, Russell Keith-Magee wrote:
> On Thu, Apr 7, 2011 at 2:51 AM, Michal Petrucha  
> wrote:
> > GSoC 2011 Proposal: Composite Fields
> > 
> 
> Hi Michal,
> 
> This looks to be a fairly solid proposal. You've done a lot of
> detailed research, and while I'm almost completely certain you'll find
> some gremlin lurking underneath some dark corner of the code, you
> appear to have a good grasp on the scope of the problem you're
> proposing to solve.
> 
> A couple of quick comments:
> 
> > Relationship fields
> > ~~~
> >
> > This turns out to be, not too surprisingly, the toughest problem. The fact
> > that related fields are spread across about fifteen different classes,
> > most of which are quite nontrivial, makes the whole bundle pretty fragile,
> > which means the changes have to be made carefully not to break anything.
> 
> Feel free to do some housekeeping while you're in there :-) The
> internal structure of foreign keys and m2ms isn't officially stable
> API, so you have some liberty to do some cleanup.

There will certainly have to be some changes but I'll try to keep them
as small as possible. I already have enough on my plate, I think. (-:

> > This infrastructure will allow reimplementing the GenericForeignKey as a
> > CompositeField at a later stage. Thanks to the modifications in the
> > joining code it should also be possible to implement bidirectional generic
> > relationship traversal in QuerySet filters. This is, however, out of scope
> > of this project.
> 
> This, for me, is ultimately one of the biggest areas for gain. Adding
> the new features of composite FKs and PKs will certainly be a big
> benefit, cleaning up the special case handling for Generic Foreign
> Keys will be a much bigger architectural win.
> 
> I appreciate that you have to draw the line somewhere; but I suspect
> that you may be faced with the need to modify an interface in a way
> that will make it easier to handle CompositeFields, but breaks the
> existing implementation of Generic Keys. If this happens, updating
> GenericKeys may be an inevitable consequence.

My intention is to generalize the interfaces trying to keep them
backwards compatible where possible and modify them in a way that
requires least updates on this end.

I know this sounds awfully vague but most interfaces set up to support
GFKs are already capable of doing the tasks required by
CompositeFields.

> > As I will have quite a few exams at school throughout June, I won't be
> > able to commit myself fully to the project for the first month and will
> > spend approximately 20 hours per week during this period. By the end of
> > the exam period, however, I intend to have sped up to about 30-35 hours
> > per week.
> 
> I don't have any problem with this; it's an unfortunate consequence of
> European school terms. However, we've historically encouraged
> applicants to stretch the rules a little bit, and start early on their
> project so they can compensate for at least some of the time they lose
> during exams.

I'll try to find some time and begin earlier but there are a couple of
school project I will have to have finished by the end of the semester
which is May 13, and these will also require some time. In the end I
will probably only get a week's head start at most...

> > week  5 (Jun 20. - Jun 26.):
> > - ModelForms and GFK support for composite primary keys
> >
> > week  6 (Jun 27. - Jul  3.):
> > - full support in the admin
> 
> Weeks 5 and 6 seem particularly ambitious -- not completely
> impossible, mind; just very ambitious.

As I wrote in my reply to Jacob, I'll probably drop the GFK support
which would make the fifth week lighter.
As for the sixth week, a nontrivial part of the admin work is hidden
in ModelForms. Maybe I'm overlooking something but most of the code
should work as soon as the underlying forms and querysets support
composite fields.

> > week  7 (Jul  4. - Jul 10.):
> > - fixing any documentation discrepancies and making sure everything is
> >  tested thoroughly
> > - exploring the related fields in detail and working up a detailed plan
> >  for the following changes
> >
> > > midterm
> >  By the time midterm evaluation arrives, everything except for
> >  relationship fields should be in production-ready state.
> >
> > week  8 (Jul 11. - Jul 17.):
> > - implementing composite primary key support in all the
> >  RelatedObjectDescriptors
> >
> > week  9 (Jul 18. - Jul 24.):
> > - query joins refactoring
> > 

Re: [GSoC] Composite fields: proposal, part 1

[Michal Petrucha]

On Wed, Apr 06, 2011 at 04:57:39PM -0500, Jacob Kaplan-Moss wrote:
> So far this looks pretty good to me. Assuming you get the rest done
> with a similar level of detail I'll be voting to approve it (and
> possibly signing up to mentor, time-willing).

Thanks for the encouragement, I really appreciate it.

> One thing I'd like to see a bit more of is a roundup of your previous
> experience with Django and with open source in general. I can't keep
> track of everyone who participates, so it's nice to remind me of where
> I might have seen your name around before. If you haven't done much
> yet that's totally fine. But if you've got a track record, brag about
> it :)

I'll update this section of the proposal, but I think I'll prioritize
the implementation related parts today...

> > One thing that bugs me is the name of the namedtuple -- it is a class,
> > which means InitialCaps is the right way, however, its name is partly
> > composed of a field name which underscore-separated words fit better. This
> > is just a cosmetic detail though.
> 
> Another thing to consider: namedtuple is only available in Python 2.6
> and above, so we'd have to backport it if we used it literally. It may
> be better in the long-run to use a namedtuple-like "CompositeObject"
> implementation of our own. In other words, don't feel bound to
> namedtuple just because I used it. The code I wrote was a 10 minute
> hack.
> 
> So you may want in the proposal to describe the composite object as a
> "namedtuple-like object" or something to indicate that it might be a
> different type of object in the end. I don't think anyone particularly
> cares about the particular implementation as long as they can say
> ``person.name.first`` or whatever.

Writing a small class shouldn't be any problem, I'll update that.

> > QuerySet filtering
> > ~~
> >
> > This is where the real fun begins.
> 
> Man you ain't kiddin' :)
> 
> You've done a fine job describing the problem, and your solution
> sounds reasonable to me as a sketch.
> 
> But there's two things missing here for me: I'd like a description of which
> query operators will be supported by composite keys. Clearly ``__exact`` would
> be supported. Maybe ``__in``? But I'll bet composite fields will only support 
> a
> couple-three operators, right? I think that's fine -- I can always
> query the sub-fields directly themselves.
> 
> It's important for you to remember that you don't have to get every
> single feature done all in one fell swoop. Even if composite fields
> only support ``__exact`` that's a huge win, and we can always add
> support for other types of queries later. It's perfectly acceptable
> for you to say things like "this feature might be possible but isn't
> in scope".

Yeah, I forgot to make it more explicit. I was considering the
"__in" lookup at first, but in the end I decided it will only
support "__exact". "__in" might be implemented later but it will
require some more serious refactoring in the add_filter code and
making the extra_filters more powerful.

> Actually, now that I write that, it's not just acceptable -- it's
> awesome. Defining what's *not* in scope is great because it helps
> define what "success" is going to mean. Constraints are good :)
> 
> > GenericForeignKeys
> > ~~
> >
> > As I said, this is used in the admin, which means we can't have full admin
> > support without also making GenericForeignKeys work with CompositeFields.
> 
> Hm, I'm not sure I understand what you mean here -- can you explain a
> bit further? I'd think we can leave GFKs out as a to-be-done-later
> thing. Remember: you've got two months to get code to the point that
> we can merge it into trunk.

The thing is, the admin uses ContentType in the LogEntry model.
However, now that I'm looking at the code, it only uses
ContentType.get_object_for_this_type which means making sure the
primary key it passes as the parameter is an actual unpacked
CompositeObject should be enough.

I guess you're right, I'll leave this part for a later stage.

Thanks for the comments, I'll try to post an updated version soon.

Michal


signature.asc
Description: Digital signature

Re: [GSoC] Composite fields: proposal, part 1

[Michal Petrucha]

On Thu, Apr 07, 2011 at 12:38:07AM +0200, Johannes Dollinger wrote:
> The only downside is that you'll have to pick a name for the index –
> even if you don't really care (that's historically been a problem
> with `related_name`). But anyway, since Meta.unique_together
> probably cannot be deprecated any time soon, that's just a -0 from
> me.

You're right, having to choose a name for an index might seem weird.
Maybe the deprecation of unique_together wouldn't be necessary.

This reminds me of a certain comment in the related fields code, [1]

> I don't want to push my proposal into your GSoC project. It'd
> require more thought and discussion before I'd suggest that.
> To address your concern: fields are a central component of the ORM.
> But currently, instead of using methods to encapsulate field
> specific logic, everything is stuffed in a single monolithic class
> (Query), that has to know every possible field implementation. This
> leads to code that is hard to maintain and requires coupling with
> code that needs more control than standard fields (read:
> contenttypes). 
> And since `add_lookup` and a bit of public API on Query would make
> it possible to implement most of your proposal without touching
> Django, I wanted to propose it soon enough to have a possible
> influence on design decisions.

There is certainly a point in this. On the other hand, moving some of
the SQL logic would make life harder for the guys fighting on the
non-relational backend front. I don't know much about the state of
this area though...

> Regardless, I look forward to CompositeFields. Good luck with your
> application!

Thanks. (-;

Michal


[1] 
http://code.djangoproject.com/browser/django/trunk/django/db/models/fields/related.py#L1087


signature.asc
Description: Digital signature

Re: [GSoC] Composite fields: proposal draft 1, full version

[Russell Keith-Magee]

On Thu, Apr 7, 2011 at 2:51 AM, Michal Petrucha  wrote:
> GSoC 2011 Proposal: Composite Fields
> 

Hi Michal,

This looks to be a fairly solid proposal. You've done a lot of
detailed research, and while I'm almost completely certain you'll find
some gremlin lurking underneath some dark corner of the code, you
appear to have a good grasp on the scope of the problem you're
proposing to solve.

A couple of quick comments:

> Relationship fields
> ~~~
>
> This turns out to be, not too surprisingly, the toughest problem. The fact
> that related fields are spread across about fifteen different classes,
> most of which are quite nontrivial, makes the whole bundle pretty fragile,
> which means the changes have to be made carefully not to break anything.

Feel free to do some housekeeping while you're in there :-) The
internal structure of foreign keys and m2ms isn't officially stable
API, so you have some liberty to do some cleanup.

> This infrastructure will allow reimplementing the GenericForeignKey as a
> CompositeField at a later stage. Thanks to the modifications in the
> joining code it should also be possible to implement bidirectional generic
> relationship traversal in QuerySet filters. This is, however, out of scope
> of this project.

This, for me, is ultimately one of the biggest areas for gain. Adding
the new features of composite FKs and PKs will certainly be a big
benefit, cleaning up the special case handling for Generic Foreign
Keys will be a much bigger architectural win.

I appreciate that you have to draw the line somewhere; but I suspect
that you may be faced with the need to modify an interface in a way
that will make it easier to handle CompositeFields, but breaks the
existing implementation of Generic Keys. If this happens, updating
GenericKeys may be an inevitable consequence.

> As I will have quite a few exams at school throughout June, I won't be
> able to commit myself fully to the project for the first month and will
> spend approximately 20 hours per week during this period. By the end of
> the exam period, however, I intend to have sped up to about 30-35 hours
> per week.

I don't have any problem with this; it's an unfortunate consequence of
European school terms. However, we've historically encouraged
applicants to stretch the rules a little bit, and start early on their
project so they can compensate for at least some of the time they lose
during exams.

> The proposed timeline is as follows:
>
> week  1 (May 23. - May 29.):
> - basic CompositeField implementation with assignment and retrieval
> - documentation for the new field type API
>
> week  2 (May 30. - Jun  5.):
> - creation of indexes on the database
> - unique conditions checking regression tests
>
> week  3 (Jun  6. - Jun 12.):
> - query code refactoring to make it possible to support the required
>  extra_filters
> - lookups by CompositeFields
>
> week  4 (Jun 13. - Jun 19.):
> - creation of a composite primary key
> - more tests and taking care of any missing/forgotten documentation so far
>
> week  5 (Jun 20. - Jun 26.):
> - ModelForms and GFK support for composite primary keys
>
> week  6 (Jun 27. - Jul  3.):
> - full support in the admin

Weeks 5 and 6 seem particularly ambitious -- not completely
impossible, mind; just very ambitious.

> week  7 (Jul  4. - Jul 10.):
> - fixing any documentation discrepancies and making sure everything is
>  tested thoroughly
> - exploring the related fields in detail and working up a detailed plan
>  for the following changes
>
> > midterm
>  By the time midterm evaluation arrives, everything except for
>  relationship fields should be in production-ready state.
>
> week  8 (Jul 11. - Jul 17.):
> - implementing composite primary key support in all the
>  RelatedObjectDescriptors
>
> week  9 (Jul 18. - Jul 24.):
> - query joins refactoring
> - support for ForeignKey relationship traversals

Again -- another ambitious week.

> week 10 (Jul 25. - Jul 31.):
> - making sure OneToOne and ManyToMany work as well
>
> weeks 11&12 (Aug  1. - Aug  14.):
> - writing even more tests for the relationships
> - finishing any missing documentation
>
> > pencils down
>
> As can be seen from the proposed timeline, there is a separation between
> the part that leads up to admin support for composite primary keys and the
> relationship part. In my opinion the first part is more likely to be used
> in practice than the second part so the main emphasis will be put on it in
> case I discover unexpected difficulties. However, looking at the timeline
> broken down into small parts I'm confident all proposed features should be
> possible in the given time.

Re: [GSoC] Composite fields: proposal, part 1

[Johannes Dollinger]

t; Query.setup_joins that is in use by GenericRelations, this is
>>> unfortunately not enough. The optional extra_filters field method will be
>>> of great use here, though it will have to be extended.
>>> 
>>> Currently the only parameters it gets are the list of joins the
>>> filter traverses, the position in the list and a negate parameter
>>> specifying whether the filter is negated. The GenericRelation instance can
>>> determine the value of the content type (which is what the extra_filters
>>> method is used for) easily based on the model it belongs to.
>>> 
>>> This is not the case for a CompositeField -- it doesn't have any idea
>>> about the values used in the query. Therefore a new parameter has to be
>>> added to the method so that the CompositeField can construct all the
>>> actual filters from the iterable containing the values.
>>> 
>>> Afterwards the handling inside Query is pretty straightforward. For
>>> CompositeFields (and virtual fields in general) there is no value to be
>>> used in the where node, the extra_filters are responsible for all
>>> filtering, but since the filter should apply to a single object even after
>>> join traversals, the aliases will be set up while handling the "root"
>>> filter and then reused for each one of the extra_filters.
>> 
>> As you noted, the problem with GFK, CompositeFields, and virtual
>> fields in general is mostly that support for filtering (read: custom
>> lookups) requires changes deep inside the ORM.  I have a half-baked
>> proposal that would make such fields much simpler to implement but
>> would require a large refactoring of the ORM. It might well be that
>> it turns out to be impractical, impossible, or too big to be
>> included in your summer of code project. That being said, here's the
>> plan:
>> 
>> Lookups are handled by the field class. That involves moving
>> significant bits of code from db.models.sql.query into the field
>> classes. Ideally, Query knows anything about fieldtypes. It just
>> delegates lookups (including related field lookups) to the field
>> implementation via:
>> 
>>  def add_lookup(self, query, alias, lookup_bits, value, **kwargs):
>>  """
>>  - `query` is the db.sql.query.Query instance
>>  - `alias` is the table alias of the "current" table;
>>that is: the table which this lookup must be
>>performed on
>>  - `lookup_bits` is a sequence of lookups, e.g.
>>"related_field__some_field__icontains".split("__")
>>  - `value` is the filter argument
>>  - `**kwargs`: It might be necessary to pass more
>>information, but certainly not arbitrary data. I
>>just haven't thought that through yet.
>>  """
>> 
>> Examples: 
>> * CharField will assert that it receives at most one lookup bit. It
>>  adds a WHERE clause for `alias`.`fieldname`.
>> * ForeignKey looks for field `f` in its containing model with name
>>  lookup_bits[0]. If such a field exists it sets up a join and
>>  passes the resulting table alias along with lookup_bits[1:] to
>>  f.add_lookup().
>> * CompositeField checks if the first lookup bit is the name of one
>>  of its constituent fields. If that's the case, the lookup is
>>  delegated to that field. Otherwise, it adds a composite WHERE
>>  clause for its constituent fields.
>> * Reverse lookups would require that reverse descriptors are
>>  implemented as virtual fields.
>> 
>> This refactoring, which involves cleaning up (and splitting)
>> db.models.sql.query and db.models.fields.*, would probably be full
>> of dragons. Finding a clean public API for Field and Query alone
>> would be major task.
>> But it could result in a decoupled and comparably trivial
>> implementation of GFKs and Composite fields and would opens the ORM
>> for other virtual fields and custom lookups.
> 
> This idea looks interesting. The work to implement this might not even
> be as difficult as it may seem, just tedious since it would require
> shuffling around massive amounts of code and exhaustive regression
> tests.
> 
> Anyway, I'd rather leave it out of my proposal, because it would
> (IMHO) require a larger discussion since it is a pretty serious design
> decision and we definitely won't be able to discuss it until Friday.
> (-;
> 
> One thing that I do

Re: [GSoC] Composite fields: proposal, part 1

[Jacob Kaplan-Moss]

On Tue, Apr 5, 2011 at 7:45 PM, Michal Petrucha  wrote:
> Anyway, I'll post at least the part I have already written so that at
> least something can be commented on for now.

So far this looks pretty good to me. Assuming you get the rest done
with a similar level of detail I'll be voting to approve it (and
possibly signing up to mentor, time-willing).

Some specific comments inline:

> While developping an application for internal use by the organizers of
> several Slovak high school programming contests I got into a situation
> where having support for composite primary keys would help greatly,
> therefore I decided to implement it with some extra added value.

Just as an asside for other GSoC students, this sort of thing is
great. Open source gets done by a bunch of volunteers scratching their
own itches, so knowing that you've felt this particular one personally
really helps us see you've got the drive to finish this out.

One thing I'd like to see a bit more of is a roundup of your previous
experience with Django and with open source in general. I can't keep
track of everyone who participates, so it's nice to remind me of where
I might have seen your name around before. If you haven't done much
yet that's totally fine. But if you've got a track record, brag about
it :)

> One thing that bugs me is the name of the namedtuple -- it is a class,
> which means InitialCaps is the right way, however, its name is partly
> composed of a field name which underscore-separated words fit better. This
> is just a cosmetic detail though.

Another thing to consider: namedtuple is only available in Python 2.6
and above, so we'd have to backport it if we used it literally. It may
be better in the long-run to use a namedtuple-like "CompositeObject"
implementation of our own. In other words, don't feel bound to
namedtuple just because I used it. The code I wrote was a 10 minute
hack.

So you may want in the proposal to describe the composite object as a
"namedtuple-like object" or something to indicate that it might be a
different type of object in the end. I don't think anyone particularly
cares about the particular implementation as long as they can say
``person.name.first`` or whatever.

> QuerySet filtering
> ~~
>
> This is where the real fun begins.

Man you ain't kiddin' :)

You've done a fine job describing the problem, and your solution
sounds reasonable to me as a sketch.

But there's two things missing here for me: I'd like a description of which
query operators will be supported by composite keys. Clearly ``__exact`` would
be supported. Maybe ``__in``? But I'll bet composite fields will only support a
couple-three operators, right? I think that's fine -- I can always
query the sub-fields directly themselves.

It's important for you to remember that you don't have to get every
single feature done all in one fell swoop. Even if composite fields
only support ``__exact`` that's a huge win, and we can always add
support for other types of queries later. It's perfectly acceptable
for you to say things like "this feature might be possible but isn't
in scope".

Actually, now that I write that, it's not just acceptable -- it's
awesome. Defining what's *not* in scope is great because it helps
define what "success" is going to mean. Constraints are good :)

> Admin
> ~
>
> The solution that has been proposed so many times in the past is to extend
> the quote function used in the admin to also quote the comma and then use
> an unquoted comma as the separator. Even though this solution looks ugly
> to some, I don't think there is much choice -- there needs to be a way to
> separate the values and in theory, any character could be contained inside
> a value so we can't really avoid choosing one and escaping it.

Exactly. Just choose one, and don't let anyone bikeshed you into
spending more than 30 seconds on this choice.

> GenericForeignKeys
> ~~
>
> As I said, this is used in the admin, which means we can't have full admin
> support without also making GenericForeignKeys work with CompositeFields.

Hm, I'm not sure I understand what you mean here -- can you explain a
bit further? I'd think we can leave GFKs out as a to-be-done-later
thing. Remember: you've got two months to get code to the point that
we can merge it into trunk.

> This is a part I haven't finished yet, I will do that tomorrow since it is
> already late and I have yet to revise for tomorrow's exam.

Good luck on the exam. I look forward to seeing the rest of your
proposal - it's great so far.

Jacob

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSoC] Composite fields: proposal, part 1

[Michal Petrucha]

esn't have any idea
> > about the values used in the query. Therefore a new parameter has to be
> > added to the method so that the CompositeField can construct all the
> > actual filters from the iterable containing the values.
> > 
> > Afterwards the handling inside Query is pretty straightforward. For
> > CompositeFields (and virtual fields in general) there is no value to be
> > used in the where node, the extra_filters are responsible for all
> > filtering, but since the filter should apply to a single object even after
> > join traversals, the aliases will be set up while handling the "root"
> > filter and then reused for each one of the extra_filters.
> 
> As you noted, the problem with GFK, CompositeFields, and virtual
> fields in general is mostly that support for filtering (read: custom
> lookups) requires changes deep inside the ORM.  I have a half-baked
> proposal that would make such fields much simpler to implement but
> would require a large refactoring of the ORM. It might well be that
> it turns out to be impractical, impossible, or too big to be
> included in your summer of code project. That being said, here's the
> plan:
> 
> Lookups are handled by the field class. That involves moving
> significant bits of code from db.models.sql.query into the field
> classes. Ideally, Query knows anything about fieldtypes. It just
> delegates lookups (including related field lookups) to the field
> implementation via:
> 
>   def add_lookup(self, query, alias, lookup_bits, value, **kwargs):
>   """
>   - `query` is the db.sql.query.Query instance
>   - `alias` is the table alias of the "current" table;
> that is: the table which this lookup must be
> performed on
>   - `lookup_bits` is a sequence of lookups, e.g.
> "related_field__some_field__icontains".split("__")
>   - `value` is the filter argument
>   - `**kwargs`: It might be necessary to pass more
> information, but certainly not arbitrary data. I
> just haven't thought that through yet.
>   """
> 
> Examples: 
> * CharField will assert that it receives at most one lookup bit. It
>   adds a WHERE clause for `alias`.`fieldname`.
> * ForeignKey looks for field `f` in its containing model with name
>   lookup_bits[0]. If such a field exists it sets up a join and
>   passes the resulting table alias along with lookup_bits[1:] to
>   f.add_lookup().
> * CompositeField checks if the first lookup bit is the name of one
>   of its constituent fields. If that's the case, the lookup is
>   delegated to that field. Otherwise, it adds a composite WHERE
>   clause for its constituent fields.
> * Reverse lookups would require that reverse descriptors are
>   implemented as virtual fields.
> 
> This refactoring, which involves cleaning up (and splitting)
> db.models.sql.query and db.models.fields.*, would probably be full
> of dragons. Finding a clean public API for Field and Query alone
> would be major task.
> But it could result in a decoupled and comparably trivial
> implementation of GFKs and Composite fields and would opens the ORM
> for other virtual fields and custom lookups.

This idea looks interesting. The work to implement this might not even
be as difficult as it may seem, just tedious since it would require
shuffling around massive amounts of code and exhaustive regression
tests.

Anyway, I'd rather leave it out of my proposal, because it would
(IMHO) require a larger discussion since it is a pretty serious design
decision and we definitely won't be able to discuss it until Friday.
(-;

One thing that I don't like very much about this idea is that the
fields would have to be aware of low-level SQL notions, such as
aliases and joins. I think there was a reason these things were buried
deep under several layers of code under the public API.

For the purpose of this project the extra_filters should do just fine
once extended to also take the value.

Michal



[47] https://groups.google.com/d/msg/django-developers/MvhhyL1TZqU/QVAMX-T8pfkJ


signature.asc
Description: Digital signature

Re: [GSoC] Composite fields: proposal draft 1, full version

[Michal Petrucha]

> I started writing the draft for a full proposal, however, I don't have
> time to finish it today as I have to revise for tomorrow's exam. I
> will try to finish it in 12 hours at most since I know I'm already
> posting it a little bit too late to make it possible to review it
> thoroughly.

Heh, didn't manage the 12 hours, well, at least it is still
today... (hey, almost to the minute two days until deadline!)

I have yet to add some more references to the document.
The same URL as before still holds if anyone prefers it:
http://people.ksp.sk/~johnny64/GSoC-full-proposal


GSoC 2011 Proposal: Composite Fields


About me


My name is Michal Petrucha. I'm an undergrad student of computer science
at the Faculty of Mathematics, Physics and Informatics, Comenius
University, Bratislava, Slovakia. As a high school student I participated
in programming contests such as the Olympiad in Informatics.

While developping an application for internal use by the organizers of
several Slovak high school programming contests I got into a situation
where having support for composite primary keys would help greatly,
therefore I decided to implement it with some extra added value.


Synopsis


Django's ORM is a powerful tool which suits perfectly most use-cases,
however, there are cases where having exactly one primary key column per
table induces unnecessary redundancy.

One such case is the many-to-many intermediary model. Even though the pair
of ForeignKeys in this model identifies uniquely each relationship, an
additional field is required by the ORM to identify individual rows. While
this isn't a real problem when the underlying database schema is created
by Django, it becomes an obstacle as soon as one tries to develop a Django
application using a legacy database.

Since there is already a lot of code relying on the pk property of model
instances and the ability to use it in QuerySet filters, it is necessary
to implement a mechanism to allow filtering of several actual fields by
specifying a single filter.

The proposed solution is a virtual field type, CompositeField. This field
type will enclose several real fields within one single object. From the
public API perspective this field type will share several characteristics
of other field types, namely:

- CompositeField.unique
This will create a unique index on the enclosed fields in the
database, deprecating the 'unique_together' Meta attribute.

- CompositeField.db_index
This option will create a non-unique index on the enclosed fields.

- CompositeField.primary_key
This option will tell the ORM that the primary key for the model is
composed of the enclosed fields.

- Retrieval and assignment
Retrieval of the CompositeField value will return a namedtuple
containing the actual values of underlying fields; assignment will
assign given values to the underlying fields.

- QuerySet filtering
Supplying an iterable the same way as with assignment to an
'exact'-type filter will match only those instances where each
underlying field value equals the corresponding supplied value.


Implementation
--

Specifying a CompositeField in a Model
~~

The constructor of a CompositeField will accept the supported options as
keyword parameters and the enclosed fields will be specified as positional
parameters. The order in which they are specified will determine their
order in the namedtuple representing the CompositeField value (i. e. when
retrieving and assigning the CompositeField's value; see example below).

unique and db_index
~~~
Implementing these will require some modifications in the backend code.
The table creation code will have to handle virtual fields as well as
local fields in the table creation and index creation routines
respectively.

When the code handling CompositeField.unique is finished, the
models.options.Options class will have to be modified to create a unique
CompositeField for each tuple in the Meta.unique_together attribute. The
code handling unique checks in models.Model will also have to be updated
to reflect the change.

Retrieval and assignment


Jacob has actually already provided a skeleton of the code that takes care
of this as seen in [1]. I'll only summarize the behaviour in a brief
example of my own.

class SomeModel(models.Model):
first_field = models.IntegerField()
second_field = models.CharField(max_length=100)
composite = models.CompositeField(first_field, second_field)

>>> instance = new SomeModel(first_field=47, second_field="some string")
>>> instance.composite
SomeModel_composite(first_field=47, second_field='some string')
>>> instance.composite = (74, "other string")
>&g

Re: [GSoC] Composite fields: proposal, part 1

[Johannes Dollinger]

able which this lookup must be performed on
- `lookup_bits` is a sequence of lookups, e.g. 
"related_field__some_field__icontains".split("__")
- `value` is the filter argument
- `**kwargs`: It might be necessary to pass more information, 
but certainly not arbitrary data. I just haven't thought that through yet.
"""

Examples: 
* CharField will assert that it receives at most one lookup bit. It adds a 
WHERE clause for `alias`.`fieldname`.
* ForeignKey looks for field `f` in its containing model with name 
lookup_bits[0]. If such a field exists it sets up a join and passes the 
resulting table alias along with lookup_bits[1:] to f.add_lookup().
* CompositeField checks if the first lookup bit is the name of one of its 
constituent fields. If that's the case, the lookup is delegated to that field. 
Otherwise, it adds a composite WHERE clause for its constituent fields.
* Reverse lookups would require that reverse descriptors are implemented as 
virtual fields.

This refactoring, which involves cleaning up (and splitting) 
db.models.sql.query and db.models.fields.*, would probably be full of dragons. 
Finding a clean public API for Field and Query alone would be major task.
But it could result in a decoupled and comparably trivial implementation of 
GFKs and Composite fields and would opens the ORM for other virtual fields and 
custom lookups.

[snip]

__
Johannes

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSoC] Composite fields: proposal, part 1

[Michal Petrucha]

I started writing the draft for a full proposal, however, I don't have
time to finish it today as I have to revise for tomorrow's exam. I
will try to finish it in 12 hours at most since I know I'm already
posting it a little bit too late to make it possible to review it
thoroughly.

Anyway, I'll post at least the part I have already written so that at
least something can be commented on for now.

Just to make sure there won't be any formatting issues, I uploaded it
here: http://people.ksp.sk/~johnny64/GSoC-full-proposal


GSoC 2011 Proposal: Composite Fields


About me


My name is Michal Petrucha. I'm an undergrad student of computer science
at the Faculty of Mathematics, Physics and Informatics, Comenius
University, Bratislava, Slovakia. As a high school student I participated
in programming contests such as the Olympiad in Informatics.

While developping an application for internal use by the organizers of
several Slovak high school programming contests I got into a situation
where having support for composite primary keys would help greatly,
therefore I decided to implement it with some extra added value.


Synopsis


Django's ORM is a powerful tool which suits perfectly most use-cases,
however, there are cases where having exactly one primary key column per
table induces unnecessary redundancy.

One such case is the many-to-many intermediary model. Even though the pair
of ForeignKeys in this model identifies uniquely each relationship, an
additional field is required by the ORM to identify individual rows. While
this isn't a real problem when the underlying database schema is created
by Django, it becomes an obstacle as soon as one tries to develop a Django
application using a legacy database.

Since there is already a lot of code relying on the pk property of model
instances and the ability to use it in QuerySet filters, it is necessary
to implement a mechanism to allow filtering of several actual fields by
specifying a single filter.

The proposed solution is a virtual field type, CompositeField. This field
type will enclose several real fields within one single object. From the
public API perspective this field type will share several characteristics
of other field types, namely:

- CompositeField.unique
This will create a unique index on the enclosed fields in the
database, deprecating the 'unique_together' Meta attribute.

- CompositeField.db_index
This option will create a non-unique index on the enclosed fields.

- CompositeField.primary_key
This option will tell the ORM that the primary key for the model is
composed of the enclosed fields.

- Retrieval and assignment
Retrieval of the CompositeField value will return a namedtuple
containing the actual values of underlying fields; assignment will
assign given values to the underlying fields.

- QuerySet filtering
Supplying an iterable the same way as with assignment to an
'exact'-type filter will match only those instances where each
underlying field value equals the corresponding supplied value.


Implementation
--

Specifying a CompositeField in a Model
~~

The constructor of a CompositeField will accept the supported options as
keyword parameters and the enclosed fields will be specified as positional
parameters. The order in which they are specified will determine their
order in the namedtuple representing the CompositeField value (i. e. when
retrieving and assigning the CompositeField's value; see example below).

unique and db_index
~~~
Implementing these will require some modifications in the backend code.
The table creation code will have to handle virtual fields as well as
local fields in the table creation and index creation routines
respectively.

When the code handling CompositeField.unique is finished, the
models.options.Options class will have to be modified to create a unique
CompositeField for each tuple in the Meta.unique_together attribute. The
code handling unique checks in models.Model will also have to be updated
to reflect the change.

Retrieval and assignment


Jacob has actually already provided a skeleton of the code that takes care
of this as seen in [1]. I'll only summarize the behaviour in a brief
example of my own.

class SomeModel(models.Model):
first_field = models.IntegerField()
second_field = models.CharField(max_length=100)
composite = models.CompositeField(first_field, second_field)

>>> instance = new SomeModel(first_field=47, second_field="some string")
>>> instance.composite
SomeModel_composite(first_field=47, second_field='some string')
>>> instance.composite = (74, "other string")
>>> instance.first_field, instance.second_field
(74, '