Re: "Normalized" Data type for __month and __day lookups?

[Malcolm Tredinnick]

On Tue, 2009-01-20 at 02:16 -0300, Leo Soto M. wrote:
[...]
> I spent around an hour digging in the history of that code, and come
> to the following reasoning: the problems were caused by two mismatches
> between what the backend expected and what it got:
> 
> - After the get_db_prep_* refactoring, backend expected int, but the
> admin passed unicode. This caused #8424.
> - The fix for #8424 was to expect unicode on the backend. But most
> usages passed an int. Caused #8510.
> - The fix for #8510 was to normalize everything to unicode. Both sides
> talk happily now :)

Okay, that sounds very familiar. I remember lots of bouncing back and
forth and going a bit mental trying to keep everything happy. I think
that's the correct history.

> So, this little change is based on keep both sides synchronized but
> with a better contract.

If your change works, it's a little better, since we are really are
talking about integers there. Should use the right types if we can.

Regards,
Malcolm


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: "Normalized" Data type for __month and __day lookups?

[Leo Soto M.]

On Tue, Jan 20, 2009 at 2:02 AM, Malcolm Tredinnick
 wrote:
>
> On Tue, 2009-01-20 at 01:54 -0300, Leo Soto M. wrote:
> [...]
>> So I've uploaded a small patch on #10071[4] which does the
>> normalization to int instead of unicode. I tested it with sqlite
>> (which was the problematic backend) and it doesn't break any test.
>>
>> Now, my questions are:
>>
>>  - Is this change OK or would it be considered as backwards
>> incompatible for slightly changing the interface with DB backends? On
>> the pragmatic side I'd like to point that this doesn't break anything
>> on the built-in backends but it would be good get confirmation from
>> other external backend developers.
>
> That area of the code was really problematic just before 1.0. I'll dig
> through my memory and read the changes around that time to try and
> remember what the tricks were, but I have a recollection we tried to
> initially go directly to ints and something or other didn't work
> (something form or admin related).

I spent around an hour digging in the history of that code, and come
to the following reasoning: the problems were caused by two mismatches
between what the backend expected and what it got:

- After the get_db_prep_* refactoring, backend expected int, but the
admin passed unicode. This caused #8424.
- The fix for #8424 was to expect unicode on the backend. But most
usages passed an int. Caused #8510.
- The fix for #8510 was to normalize everything to unicode. Both sides
talk happily now :)

So, this little change is based on keep both sides synchronized but
with a better contract. However, people who can remember the actual
facts may be able to confirm or correct my reasoning.

>>  - If the change is OK, could it make it into the 1.0.X branch? That'd
>> be very good, as we are targeting Django 1.0 on the django-jython
>> project.
>
> I'd call it a bug fix for that reason. Jython support on 1.0 is/was one
> of our goals.
>
> Has something changed that made this only show up now? Or has it always
> been there and you just didn't notice?

It has always been there.

Regards,
-- 
Leo Soto M.
http://blog.leosoto.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: "Normalized" Data type for __month and __day lookups?

[Malcolm Tredinnick]

On Tue, 2009-01-20 at 01:54 -0300, Leo Soto M. wrote:
[...]
> So I've uploaded a small patch on #10071[4] which does the
> normalization to int instead of unicode. I tested it with sqlite
> (which was the problematic backend) and it doesn't break any test.
> 
> Now, my questions are:
> 
>  - Is this change OK or would it be considered as backwards
> incompatible for slightly changing the interface with DB backends? On
> the pragmatic side I'd like to point that this doesn't break anything
> on the built-in backends but it would be good get confirmation from
> other external backend developers.

That area of the code was really problematic just before 1.0. I'll dig
through my memory and read the changes around that time to try and
remember what the tricks were, but I have a recollection we tried to
initially go directly to ints and something or other didn't work
(something form or admin related). That might not have been the same as
the change you're making here, but I'll have a hard think. It might just
have been that the change we ended up making was the least intrusive for
that point in time and it didn't seem to break anything.

There were some people working on this (in person) at the Portland
sprint just before 1.0 and my recollection is that the patch they came
up with didn't quite work and I had to spend a while tweaking it. Don't
know if any of them are on this list, though.

>  - If the change is OK, could it make it into the 1.0.X branch? That'd
> be very good, as we are targeting Django 1.0 on the django-jython
> project.

I'd call it a bug fix for that reason. Jython support on 1.0 is/was one
of our goals.

Has something changed that made this only show up now? Or has it always
been there and you just didn't notice?

Regards,
Malcolm


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

"Normalized" Data type for __month and __day lookups?

[Leo Soto M.]

While resuming the Django/Jython work, I've been hit by a small
inconsistency on the types of lookup arguments, as received by DB
backends.

Basically, __year lookup arguments are converted to integers before
being passed to the backend, but for __month and __day it's unicode.
That's weird. And for JDBC backends which are picky about types (like
on PostgreSQL where operators may have specialized meanings for
different types) string arguments won't work.

Now, the blame is on my side here: I didn't saw this situation while
working on #7560[1] and just assumed that lookup arguments were always
integers. Later, when it was clear than unicode arguments should also
be supported (because the admin uses that), changesets 8424[2] and
8526[3] fixed the existing problems. Unfortunately, always passing
unicode to the backend was (part of) the solution. But looks like
normalizing to int should also work.

So I've uploaded a small patch on #10071[4] which does the
normalization to int instead of unicode. I tested it with sqlite
(which was the problematic backend) and it doesn't break any test.

Now, my questions are:

 - Is this change OK or would it be considered as backwards
incompatible for slightly changing the interface with DB backends? On
the pragmatic side I'd like to point that this doesn't break anything
on the built-in backends but it would be good get confirmation from
other external backend developers.

 - If the change is OK, could it make it into the 1.0.X branch? That'd
be very good, as we are targeting Django 1.0 on the django-jython
project.

 - Anyone else with other version of sqlite (I'm using 3.5.9) who
would like to test if the patch works?

 - Am I missing something here?

Regards,

[1] http://code.djangoproject.com/ticket/7560
[2] http://code.djangoproject.com/changeset/8494
[3] http://code.djangoproject.com/changeset/8526
[4] http://code.djangoproject.com/ticket/10071
-- 
Leo Soto M.
http://blog.leosoto.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Model-validation: call for discussions

[mrts]

On Jan 19, 1:43 pm, Malcolm Tredinnick 
wrote:
> one. Short version: when the form field validation would match what the
> model field is going to do anyway, don't do anything at the form level.
> The model field validation is about to be called anyway.

[snip]

> The solution here might not be too difficult and doesn't need the
> requirement of communication from the form to the model about what it's
> already done.
>
> To wit, when a formfield is constructed automatically from the model
> field, if the default formfield doesn't do any validation/normalisation
> beyond what the normal model field does, the formfield() method on the
> Field subclass returns a formfield with no validators. Thus, when the
> form cleaning process is run, all the validation for that field is done
> by the model. Thus form fields returned for model fields could well be
> slightly modified (in that their validator list is different) from what
> you get directly from using forms.fields.*.
>
> That dovetails nicely with the reuqirements of custom form fields. When
> a custom form field is created, the form field validation won't be
> skipped (unless the creator wants to remove any validation from the form
> field), so any differences between the default model field validation
> and the custom form field validation aren't overlooked: they both get
> run (which is the correct behaviour).

Looks like a good action plan.

> Attempts to abstract away similar-but-not-the-same algorithms can often
> lead to more difficult code than writing the same thing out directly in
> the two places. It tends to show up in the number of parameters you pass
> into the abstract algorithm and/or the complexity of the return types.
> And, in this case, it's only two places, not fourteen or anything like
> that. So whether this is worth it tends to be implementation specific.

Yeah, I won't pursue this further, although I still think the result
would be cleaner, not uglier.

> As a style issue, there's a bit of an arbitrary split going on there.
> Type conversion is really part of what we're calling "validation" (since
> the type coercion is part of the normalisation part of the process,
> which we tend to lump into validation). I'd lump things a bit closer
> together in the source. There's not really an external use-case scenario
> for the converters and, even within Django, they don't have a lot of
> user outside of the validators (we can't do a lot of sharing with the
> database-specific backend converters, since they do tend to be
> backend-specific).
>
> Maybe (and looking at my notes, I might upgrade that to probably, given
> the number of times I've mentioned it), we need to make a directory for
> django.core.validators instead of a single file. That gives us room to
> split out converters from things that work with standard types. But
> definitely try to keep them close together in the source, rather than
> putting them in django.utils, which can sometimes be a dumping ground
> for stuff that better belongs elsewhere.

As seen in the commit,
http://github.com/mrts/honza-django/commit/a8239b063591acc367add0a01785181a91a37971
,
my initial quick solution  was to lump them into the validation
module (as there's lot's of intermingling between validators and
coercers,
exemplified by merging to_pyton() and validate() to clean() in forms).

The directory-based approach is best, I'll go with it -- but it's yet
uncertain
when as I have to handle pressing matters at work during daytime.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Feedback from django-roa module

[David Larlet]

Hello,

I'll try to sum up my main issues with Django, trying to add a Django  
ROA/WOA/SOA/BuzzwordOA module which uses a custom backend in order to  
access remote resources in a RESTful way. Basically, it calls URLs  
instead of hitting database (a picture is worth a thousand words, look  
at the schema here: http://welldev.org/django-roa-diagram.png for  
details).

My first attempt was to create my own backend (as Oracle one) but it  
eventually looks way too complicated (I don't need all SQL machinery)  
and I end up with my own Model/Manager classes. I planed  to use Meta  
class in order to add resource URL but I can't because of Meta's  
restriction:
http://code.djangoproject.com/browser/django/trunk/django/db/models/options.py#L89
Eventually, I use a static method because it's easier but I don't know  
if this restriction is worth the pain.

I got an issue with M2M relations because SQL is hardcoded and I can't  
catch it in Query(Set) object: 
http://code.djangoproject.com/browser/django/trunk/django/db/models/fields/related.py#L428
It is/will be an issue for all non-sql backend (CouchDB for instance)  
and I think it should be fixed. Otherwise, I'll need to hack a large  
part of Django just to support those relations.

Integrating auth is not as hard as I thought thanks to inheritance, I  
just wrote my own remoteauth app and it works. Great. On the other  
side, integrating admin is more complicated. The main issues come from  
integrated ContentTypes (I'm still thinking about an elegant way to  
handle that) and hardcoded way to retrieve objects:
http://code.djangoproject.com/browser/django/trunk/django/contrib/admin/options.py#L584
A get_object method on ModelAdmin can solve this issue. In the same  
attempt, it's useful to be able to easily change the ChangeList class  
in order to customize url_for_result (for instance), it'll be solved  
once #9749 is commited. I'm sure people involved in composite PKs #373  
will agree with me :)

I hope it will not be considered as a rant, my English is pretty basic  
but I just want to discuss those issues in order to help the community.

Regards,
David Larlet

#9749 http://code.djangoproject.com/ticket/9749
#373 http://code.djangoproject.com/ticket/373


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Distributed workflow and the woes of slow testsuite

[Hanne Moa]

On Mon, Jan 12, 2009 at 18:04, mrts  wrote:
> As of now, I'll stop pursuing this further, but when I come back to
> this,
> would the approach outlined in http://dpaste.com/108140/ be
> acceptable?
>
> I.e. if
> 1) not specified explicitly (by the --multiprocessing option to test)
> 2) there's only a single core in the system
> 3) importing the multiprocessing module fails
> run the ordinary single-process test runner.

Django doesn't need this yet but to show what is happening elsewhere...

CPAN have plans for fully automated testing: an os-image with the
needed libraries is created and distributed, tests are run in the
vhost and results are sent to an accumulator, in essence: c...@home...

:)


HM

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Model-validation: call for discussions

[Kenneth Gonsalves]

On Monday 19 Jan 2009 6:52:15 pm Rajeev J Sebastian wrote:
> On Mon, Jan 19, 2009 at 4:17 PM, mrts  wrote:
> > And now something completely different
> > ==
> >
> > "Every problem in computer science can be solved by
> > another level of indirection."
>
>  - David Wheeler
>
> " ... except the problem of too many levels of indirection"
>  - Kevlin Henney
>
> http://en.wikipedia.org/wiki/Abstraction_layer
> http://en.wikipedia.org/wiki/Butler_Lampson

wikipedia is *really* different

-- 
regards
KG
http://lawgon.livejournal.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Model-validation: call for discussions

[Rajeev J Sebastian]

On Mon, Jan 19, 2009 at 4:17 PM, mrts  wrote:
> And now something completely different
> ==
>
> "Every problem in computer science can be solved by
> another level of indirection."
 - David Wheeler

" ... except the problem of too many levels of indirection"
 - Kevlin Henney

http://en.wikipedia.org/wiki/Abstraction_layer
http://en.wikipedia.org/wiki/Butler_Lampson

Regards
Rajeev J Sebastian

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Model-validation: call for discussions

[Malcolm Tredinnick]

As I understand it, this is primarily about avoiding duplicate
validation of some pieces of data when it's not necessary, right? So
it's really only applicable to the ModelForm case?

This is a pretty good summary of the situation, although I suspect
there's a fairly easy solution at hand, which I've outlined below.

On Mon, 2009-01-19 at 02:47 -0800, mrts wrote:

[...]
> 
> The main focus is on fields.
> 
> Form fields
> ===

This all sounds like a description of the current situation, which is
fine.

> Model fields
> 
> 
> Ditto for model fields, omitting the first step: format validation,
> coercion to Python and value validation is generally required.

Agreed.

> 
> If, however, we want to more radically depart from current
> behaviour, then it is possible and indeed reasonable to assume that
> the values assigned to model fields are already the expected Python
> types, as it is the application developer’s responsibility to assure
> that. This would get rid of the format validation and coercion step
> (these are already handled by the form layer). to_python() is
> currently used in model fields for controlling the conversion

As a practical matter, that ship has already sailed. Lots of code
assigns things like strings to datetime fields. I've always thought that
was an unfortunate decision, but it's hardly a showstopper. Model
validators will need to be able to convert from reasonable input
variations to the right Python object. It's not unreasonable to expect,
however, that after validation/cleaning has occurred on a model, all the
attributes contains things of the right type (assuming the model is
valid). That means that subsequent code working with, e.g, a datetime
attribute on a valid model doesn't have to worry that
mymodel.foo.strftime() will raise an exception because "foo" happens to
be a string, not a datetime.


> Interaction between form and model fields
> =
> 
> Forms and models should be orthogonal and oblivious of each other.
> Modelforms serves as the glue and mapping between the two, being
> aware of the internals of both.
> 
> A modelform should have the following responsibilites in validation
> context:
> 
>  * clean the form (performs format validation, coercion to a Python
>type and value validation),

>  * assign the form field values to the associated model fields,
> 
>  * inform the model fields that basic validation has already been done
> to
>avoid duplicated validation and call any additional validation
>methods.

That's one approach, but the alternative I've sketched below is another
one. Short version: when the form field validation would match what the
model field is going to do anyway, don't do anything at the form level.
The model field validation is about to be called anyway.

> Thus, the modelform should be able to invoke only the additional
> and custom model field validators, *skipping the default coercers
> and validators* that can be assumed to be the same in similar form
> and model field classes (e.g. an IntegerField in forms and
> IntegerField in models).

That's a slightly flawed assumption, as it won't necessarily be true.
Particularly in the case of custom form field overrides on a model form.
There could be any number of variations, from the range of data
permitted to the types.

[Aside: It's certainly reasonable to think about the duplicated effort,
but I wouldn't worry about it too much. It's not a huge amount of
duplicated computation, since the case when something is already of the
right type and passes normalisation and validation tends to be quite
quick (an if-check or two and then moving on). So if we don't have a
fantastically neat solution for this initially in the communcation
between ModelForms and Models, the world won't stop spinning. That's
only one use-case in the much larger scheme of things (again, I'm not
dismissing it, but let's not get too hung up on this case). That being
said... ]

The solution here might not be too difficult and doesn't need the
requirement of communication from the form to the model about what it's
already done.

To wit, when a formfield is constructed automatically from the model
field, if the default formfield doesn't do any validation/normalisation
beyond what the normal model field does, the formfield() method on the
Field subclass returns a formfield with no validators. Thus, when the
form cleaning process is run, all the validation for that field is done
by the model. Thus form fields returned for model fields could well be
slightly modified (in that their validator list is different) from what
you get directly from using forms.fields.*.

That dovetails nicely with the reuqirements of custom form fields. When
a custom form field is created, the form field validation won't be
skipped (unless the creator wants to remove any validation from the form
field), so any differences between the default model field validation
and the custom form field 

Re: Model-validation: call for discussions

[mrts]

On Jan 18, 5:17 am, Malcolm Tredinnick 
wrote:
> (b) Please do write it out and post it here so that we can have the
> discussion on the mailing list.

Let’s step back, distance ourselves from the current implementation
and look at how forms, models and modelforms should ideally
interact validation-wise at a somewhat abstract level.

The main focus is on fields.

Form fields
===

In validation context, a form field has the following
responsibilities:

 * get input value (a string) from user,

 * validate that the value is in expected format
   (*format validation*, preconditions),

 * convert the value to a Python type,

 * validate that the resulting value satisfies any conditions set
   for the value (*value validation*, postconditions).

Model fields


Ditto for model fields, omitting the first step: format validation,
coercion to Python and value validation is generally required.

If, however, we want to more radically depart from current
behaviour, then it is possible and indeed reasonable to assume that
the values assigned to model fields are already the expected Python
types, as it is the application developer’s responsibility to assure
that. This would get rid of the format validation and coercion step
(these are already handled by the form layer). to_python() is
currently used in model fields for controlling the conversion.

Interaction between form and model fields
=

Forms and models should be orthogonal and oblivious of each other.
Modelforms serves as the glue and mapping between the two, being
aware of the internals of both.

A modelform should have the following responsibilites in validation
context:

 * clean the form (performs format validation, coercion to a Python
   type and value validation),

 * assign the form field values to the associated model fields,

 * inform the model fields that basic validation has already been done
to
   avoid duplicated validation and call any additional validation
   methods.

Thus, the modelform should be able to invoke only the additional
and custom model field validators, *skipping the default coercers
and validators* that can be assumed to be the same in similar form
and model field classes (e.g. an IntegerField in forms and
IntegerField in models).

Let's look at an contrived example that illustrates the point.

Let IntegerForeignKeyField be an imaginary foreign key field in a
model that accepts only integer foreign keys. The corresponding
form field would be a PositiveIntegerField. Suppose that the
developer has provided a custom validator for the
IntegerForeignKeyField that allows foreign keys only to objects
where the is_active boolean field is True.

When a modelform uses them both, it is the form field's
responsibility to assure that the value is converted to a Python
int and that the int is positive. IntegerForeignKeyField should
not re-run neither the coercion nor basic validation routines.
However, IntegerForeignKeyField must

1) validate that the integer points to an existing row in the
   related table (i.e. run a foreignkey-specific validator),

2) validate that the row has is_active == True (i.e. run a custom
   validator).

In this case, coercion and basic validation is cheap, but that's
not always the case.

General validation principles
=

The extended conclusion of the sections above is as follows.

Double validation should never happen. There has to be a way to
convey that a value is already validated and no further validation
is necessary. No code duplication in type coercion or validation
should occur in models and forms. The following distinct
converters and validators participate in the coercion/validation
process:

 * format validators that validate that a given string can be
   converted to a Python type,

 * converters that coerce a given string to a Python type,

 * value validators that assure that the value returned by a
   converter satisfies any conditions set for the value.

It should be easy to add custom validators and intercept or
override the process at any step.

And now something completely different
==

"Every problem in computer science can be solved by
another level of indirection."
 --- source unknown

Both form and model fields need similar functionality that is not
well served by just duplicating validation and coercion functions
in them.

It would make sense to factor out the common functionality and
decouple it from both. A mixin class would serve that purpose well.

The following methods would live in the mixin:
 * clean() that calls to_python() and then validate(),
 * to_python() that takes care of format validation and coercion,
 * validate() that takes care of value validation and running
   custom validators.

It would have fields for storing the default and custom validators
and perhaps a state field for controlling which parts of the
validation