Re: [GSoC] Revamping validation framework and merging django-secure once again

[Christopher Medrela]

*Progress.*

- Deprecated `requires_model_validation` flag and `validate` method (both
  `BaseCommand` members) in favour of new `requires_system_checks` flag and
  `check` method.

- Finished working at `contenttypes` tests.

- Improved code thanks to Preston Holmes comments. Deleted dead code and 
added
  some new tests to improve code coverage.

It'd be nice to have checks of clashes between GenericForeignKey and
accessors. I didn't implemented it because little time left and I need to
hurry up.

When it was easy, I've added new tests to improve code coverage. However, 
there
are still some tests that would be nice to have:

- Test for raising an error while using an unique varchars longer than 255
  characters under MySQL backend. [1]

- Test for `ImageField`. When `Pillow` is not installed and `ImageField` is 
in
  use, an error should be raised. This should be tested. [2]

- Test for raising warning/ImproperlyConfigured in `BaseCommand.__init__`. 
[3]

[1] 
https://github.com/chrismedrela/django/blob/149800a8136ce839903f0fe9af7f548973da9244/django/db/backends/mysql/validation.py#L6
[2] 
https://github.com/chrismedrela/django/blob/149800a8136ce839903f0fe9af7f548973da9244/django/db/models/fields/files.py#L447
[3] 
https://github.com/chrismedrela/django/blob/149800a8136ce839903f0fe9af7f548973da9244/django/core/management/base.py#L202

*Filtering and silencing errors/warnings.* We need two features: ability
to run only a subset of checks (aka filtering errors) and ability to silence
some errors.

Silencing is easy. Every warning will have a unique name like "W027". The 
format
of the name is letter "W" followed by a unique number. The system check
framework is open ended and third-party apps can register its own checks. 
For
warnings raised by these apps, "Wnumber.applabel" (e.g. "W001.myapp") style 
will
be used. Of course, everything can be changed and I'm open to yours 
suggestions,
so feel free to comment and criticize it.

There will be a new setting called `SILENCED_WARNINGS`. If you want to 
silence a
warning, you put its name in this setting, e.g.:

SILENCED_ERRORS = [
'W027',
]

Only light messages (warnings, infos and debugs) can be silenced.

Running only a subset of check is a more complex task. We can associate 
every
check with a set of tags (that can be done while registering checks). To run
only checks tagged "mytag" or "mysecondtag", you need to type:

manage.py check mytag mysecondtag

However, we would like to run checks of an app (or a set of apps):

manage.py check auth admin

This is hard, because checks are no longer app-specific. I propose to solve 
this
by passing an `apps` optional keyword argument to every check function. The
function should only validate specified apps (or all apps if apps==None). 
The
only problem is how to determine if we deal with a tag or app name? Consider
this command:

manage.py check auth mytag

This should run all checks tagged "mytag". Only messages for `auth` app 
should
be reported. I propose to collect all tags while registering check 
functions and
if a string is one of them, then interpret it as a tag, otherwise assume 
it's an
app name (and check if an app with given name exists).

*Problems/questions.*

1. We decided to mimic message and logging framework, so every error is 
tagged
with a level. Its value is an integer indicating how important and serious 
is
a message. There are five predefined values: CRITICAL, ERROR, WARNING, INFO,
DEBUG. However, Preston Holmes noticed that this is unpractical because we
need only errors and warnings. I think we should discuss again if it's worth
mimicing these frameworks.

2. On the pull request we started discussing names. "System check" is 
better than
"check" but it suggest that it's connected with hardware. Preston proposed
"Startup checks". I don't have strong opinion. To be honest, I don't thing
"startup checks" is much more better than "system checks" so I will leave 
it as
it is, but I'm still open to suggestions and I would like to see yours 
opinion.

3. There are some problems with moving custom User model checks. Their 
first source
was that some tests override `INSTALLED_APPS` setting but don't override 
list of
registered checks. So checks of `auth` app were registered (because the app 
was
imported by some other tests), but this app wasn't installed. This ended in 
an
error, because checks of `auth` try to load User model which is, by default,
`auth.User` and `auth` is not installed. I've solved this by overriding 
list of
registered checks (I've introduced `override_system_checks` decorator).

However, there is still one red test -- `admin_scripts.test_complex_app` 
[4].
The problem is that this test spawns a new Django process and the decorator
cannot affect this new process. This test installs two apps (`complex_app` 
and
`simple_app`) and doesn't install `auth` app. However, `auth` app is 
imported,
because `complex_app` imports `admin` app which imports 

Re: Order of INSTALLED_APPS

[Stefano Crosta]

Done! https://code.djangoproject.com/ticket/20914#ticket

thanks!

On Tuesday, August 13, 2013 12:20:48 PM UTC+2, Aymeric Augustin wrote:
>
> 2013/8/13 Stefano Crosta 
>
>> My proposal would then be to simply add another box to the 
>> https://docs.djangoproject.com/en/dev/ref/settings/#installed-apps to 
>> say "order matters" once more and link the other two pages for translations 
>> and templates.
>> *if you think this would* help I could do it as well as a ticket. To save 
>> everybody's time no answer will mean it's not worth it!
>>
>
> Yes, please file one, and include a link to this discussion.
>
> -- 
> Aymeric. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Usage and documentation of F()

[Marc Tamlyn]

(Disclaimer: I didn't write any of this code, I'm just working from my own
possibly flawed understanding)

The main point of using F is not generally for performance per se, rather
for database consistency. In the first example, the performance increase is
negligable, but you get two significant benefits. The first is that you
minimise the risk of race conditions by effectively issuing "increment"
rather than "set" to the database. The second is that you can increase a
number of rows the same way -
Product.objects.update(number_sold=F('number_sold') + 1). These add
performance in that you end up doing fewer queries, but that's actually
what you're gaining.

It is worth noting that the *only* reason to set an attribute to be an F
expression is to then update the value in the database. In fact, you cannot
access the value after you've done this, you get a
"django.db.models.expressions.ExpressionNode" instance. In that sense,
doing  product.number_sold = F('number_sold') + 1 is really a long hand
version of the update() query.

As for what expressions are valid, I believe the things you suggested do
work, but I imagine exactly what works will depend on what operations the
database (and backend) support. Django itself isn't doing anything clever
here, it's just providing some architecture to allow you to delegate
functionality to the database.

I agree that the main documentation for F() should reference the update
case (IMO as an update() rather than the longhand version).

Marc


On 13 August 2013 20:22, Daniele Procida  wrote:

> I noticed while looking for material for <
> https://code.djangoproject.com/ticket/20877> that <
> https://docs.djangoproject.com/en/dev/models/instances.html#updating-attributes-based-on-existing-fields/>
> mentions that:
>
> product.number_sold = F('number_sold') + 1
>
> is faster than:
>
> product.number_sold += 1
>
> though this doesn't seem to be mentioned in the database optimisation page.
>
> That's easy enough to address, and <
> https://docs.djangoproject.com/en/dev/topics/db/optimization.html#do-database-work-in-the-database-rather-than-in-python/>
> seems like a sensible place for it.
>
> However the mentions of F() that I did find raised a number of questions.
>
> The F() class seems to be a handy general-purpose way to refer to the
> value of a model field..
>
>
> Firstly, it's not explained how, in expressions like:
>
> product.number_sold = F('number_sold') + 1
>
> (from <
> https://docs.djangoproject.com/en/dev/models/instances.html#updating-attributes-based-on-existing-fields/>)
> Django knows that F('number_sold') is refers to the product model.
>
> Does it know because product.number_sold is the field that this expression
> refers to? What would happen if we did:
>
> product.number_in_stock = F('number_in_stock') - F('number_sold)
>
> (i.e. can we make such calculations multiple other fields in one go?), or:
>
> product.number_to_reorder = F('number_sold)
>
> for example? What are the rules of the usage syntax of F()?
>
> Secondly, the main documentation for F() <
> https://docs.djangoproject.com/en/dev/topics/db/queries.html#query-expressions/>
> doesn't mention this kind of use at all: it only suggests that it might be
> useful in queries.
>
> Since this use seems to be just one of multiple uses for F(), shouldn't a
> more general description of F() belong somewhere else (where) instead?
>
>
> Finally, are there any other useful ways to use F() not covered by these
> two examples?
>
>
> Daniele
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Deprecation a little harsh?

[Jani Tiainen]

Hi,

We've been able to do quite painless upgrades to our software as well.

Considering that we started with Django 1.1, stuck quite while at 1.3 and 
recently we did upgrade to 1.5

Most notable change was {% url %} tag, but nothing really major. 

Though we don't use many external libs - we've built our own.

Only real problem we have is that our deployment system sucks with the fact 
that it had for a long time shared libraries (including Django) which did had 
some impact on upgrades but since moving towards virtualenv usage we've been 
able to get rid of that problem as well, but that has nothing to do with actual 
Django.

Line count wise we hit quite similar as Florian:

main project consists:
~200 models,

.py 44805 lines
.html 143937 lines
.js 101317 lines (this is explained that we use dojotoolkit/extjs4 based single 
page app)

And our library:
.py  25079 lines
.js 25500 lines

And my 2 cents here - 

What I really like that Django getting new, better features while keeping very 
good deprecation policy. And yes, that sometimes requires that I lose few hours 
of sleep while upgrading but eventually it has to be done - it's just that I 
can't sit forever on same version. Sooner it's done easier it is and less time 
it takes. 


On Tue, 13 Aug 2013 13:16:01 -0400
François Schiettecatte  wrote:

> Florian
> 
> Here are wc -l line counts:
> 
> Project 1
> .py 28574
> .html 16967
> (this is a little misleading because we don't use model.py but a separate 
> REST API to handle all the storage)
> 
> 
> Project 2
> .py 43199
> .html 91804
> 
> 
> Project 3
> .py 32441
> .html 86684
> 
> 
> Cheers
> 
> François
> 
> 
> On Aug 13, 2013, at 12:31 PM, Florian Apolloner  wrote:
> 
> > Hi François,
> > 
> > On Tuesday, August 13, 2013 5:46:10 PM UTC+2, François Schiettecatte wrote:
> > I have done 1.3.x -> 1.4.x -> 1.5.x and they have all been painless, each 
> > migration taking less than 1/2 day. Point being that back-porting is not 
> > something I would ever need. 
> > 
> > It's good to hear that some people are keeping up2date and it didn't cause 
> > any pain! Do you mind sharing how big (lines of code wise) those apps are 
> > (just a rough classification).
> > 
> > Regards,
> > Florian
> > 

-- 

Jani Tiainen

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.