Re: Tests of contrib apps

2013-03-25 Thread Aymeric Augustin 
On 25 mars 2013, at 20:20, Stephen Burrows  wrote:

> django-nose is pretty useful for handling test discovery issues, if you're 
> looking for a quick fix.


I don't suffer from this problem because I use a custom test runner to avoid 
it. My goal here is to improve the framework for others.

These days, about 5% of the new valid tickets are a variant of this. Here's one 
from three days ago: https://code.djangoproject.com/ticket/20114

I've committed a few patches of this kind, and now I regret it. Mechanically 
adding an override_settings for each case reported to us doesn't improve the 
code base. On the contrary, it adds noise.

I realize that when we started fixing such bugs with override_settings, we 
embarked on a sisyphean task that degrades the readability of the tests. It 
seems unrealistic to guarantee that all contrib apps tests will pass with any 
combination of settings, and this will be a constant source of bugs when we add 
new tests or new settings.

So, from now on, I'll dutifully watch these tickets pile up, and enjoy the 
little nudge about the deficiency in Django's testing tools, which I don't feel 
qualified to address.

I'm disappointed but let's move on, this isn't a big deal :)

-- 
Aymeric.



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Tests of contrib apps

2013-03-25 Thread Stephen Burrows 
The tests are nested whether they're in the core tests or in contrib. 
Philosophically this is more a question of coding at a distance. Putting 
the tests in /tests makes it less obvious which app they actually belong 
to. Additionally, I'd see the fact that it would be impossible to use 
manage.py test as a detriment rather than an advantage. If people want to 
not run the tests for an app, they can always just list the apps they do 
want to test. Also, django-nose is pretty useful for handling test 
discovery issues, if you're looking for a quick fix.

--Stephen

On Wednesday, March 20, 2013 2:26:33 AM UTC-7, Aymeric Augustin wrote:
>
> Hello, 
>
> Currently there are three locations for the tests of contrib apps: 
> - under tests/ — eg. admin 
> - inside the app — eg. auth 
> - both — eg. contenttypes 
>
> Following the modeltests / regressiontests merge, I propose to move all 
> contrib app tests under tests/. This has de following advantages: 
> - it makes them easier to discover and prevents accidental duplication 
> - they won't be run by './manage.py test' 
> - it'll dam up the stream of "if I change setting X then test Y in contrib 
> app Z fails" 
>
> I'm aware of the idea that contrib apps could include integration tests to 
> validate that they're properly used within projects, but I don't believe we 
> have any such tests currently. 
>
> What do you think? 
>
> -- 
> Aymeric. 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Tests of contrib apps

2013-03-20 Thread Mikhail Korobov 


среда, 20 марта 2013 г., 21:22:10 UTC+6 пользователь Aymeric Augustin 
написал:
>
> On 20 mars 2013, at 11:22, Russell Keith-Magee 
>  
> wrote: 
>
> > Personally, I'd rather see the opposite -- contrib apps containing their 
> own tests, and the tests directory containing just the tests for Django 
> itself. That enforces the fact that the apps really are self contained 
> apps, and can be tested independently. 
>
> > The catch -- and the reason why the tests are broken up the way they are 
> -- is that the test framework isn't currently rich enough to support this. 
> Admin is the best example of this -- in order to test admin, you need 
> different test models in different test apps, and the current test 
> framework doesn't support this very well. 
>
> Ah, I didn't know that was the reason. I assumed it was just because of 
> history... 
>
> > This really points at a deficiency in Django's testing tools that really 
> should be addressed. It's something I've wanted to look at for a while, but 
> I've never got around to it. It's also something that's likely to be easier 
> when the app refactor lands, because a lot of the problem lies in the app 
> cache. 
>
> > The point about not running the tests under ./manage.py test is a 
> slightly moot point if we move to the unittest2 discovery mechanism -- last 
> time I looked at that patch, unittest2 discovery would mean that contrib 
> tests wouldn't be run (by default) in a project anyway. 
>
> If these features were implemented, yes, they would bring the benefits I 
> listed. 
>
>
> There are a few other advantages in my proposal: 
>
> - Most users don't need the tests. Moving them apart will make Django 
> tarballs smaller. For example, auth has as more tests than code (120kb vs 
> 100kb), and admin has nearly twice as much tests than code (420kb vs 
> 220kb). 
>

Currently all tests are a part of tarball, so moving tests out of contrib 
won't make tarballs smaller.
 

>
> - Shipping tests for half of the contrib apps is worse than not shipping 
> any tests at all. Users who notice will think either we have poor coverage 
> or our packages aren't built correctly. Users who don't notice may falsely 
> believe that Django's tests pass for all the contrib apps they use with 
> their settings, which may not be true. 
>
> - The convenience of having all the tests in one directory at the toplevel 
> must be balanced with the philosophical correctness of dispatching them in 
> seventeen directories, four-levels-deep, which may themselves contain 
> entire app structures -- "flat is better than nested". 
>
> - It would help in the short term :) 
>
> -- 
> Aymeric. 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Tests of contrib apps

2013-03-20 Thread Aymeric Augustin 
On 20 mars 2013, at 11:22, Russell Keith-Magee  wrote:

> Personally, I'd rather see the opposite -- contrib apps containing their own 
> tests, and the tests directory containing just the tests for Django itself. 
> That enforces the fact that the apps really are self contained apps, and can 
> be tested independently.

> The catch -- and the reason why the tests are broken up the way they are -- 
> is that the test framework isn't currently rich enough to support this. Admin 
> is the best example of this -- in order to test admin, you need different 
> test models in different test apps, and the current test framework doesn't 
> support this very well. 

Ah, I didn't know that was the reason. I assumed it was just because of history…

> This really points at a deficiency in Django's testing tools that really 
> should be addressed. It's something I've wanted to look at for a while, but 
> I've never got around to it. It's also something that's likely to be easier 
> when the app refactor lands, because a lot of the problem lies in the app 
> cache. 

> The point about not running the tests under ./manage.py test is a slightly 
> moot point if we move to the unittest2 discovery mechanism -- last time I 
> looked at that patch, unittest2 discovery would mean that contrib tests 
> wouldn't be run (by default) in a project anyway.

If these features were implemented, yes, they would bring the benefits I 
listed. 


There are a few other advantages in my proposal:

- Most users don't need the tests. Moving them apart will make Django tarballs 
smaller. For example, auth has as more tests than code (120kb vs 100kb), and 
admin has nearly twice as much tests than code (420kb vs 220kb).

- Shipping tests for half of the contrib apps is worse than not shipping any 
tests at all. Users who notice will think either we have poor coverage or our 
packages aren't built correctly. Users who don't notice may falsely believe 
that Django's tests pass for all the contrib apps they use with their settings, 
which may not be true.

- The convenience of having all the tests in one directory at the toplevel must 
be balanced with the philosophical correctness of dispatching them in seventeen 
directories, four-levels-deep, which may themselves contain entire app 
structures — "flat is better than nested".

- It would help in the short term :)

-- 
Aymeric.



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Tests of contrib apps

2013-03-20 Thread Loic Bistuer 
I run a custom DjangoTestSuiteRunner specifically to filter these out.

+1

-- 
Loic

On Mar 20, 2013, at 4:26 PM, Aymeric Augustin 
 wrote:

> - they won't be run by './manage.py test'

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Tests of contrib apps

2013-03-20 Thread Russell Keith-Magee 
On Wed, Mar 20, 2013 at 5:26 PM, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Hello,
>
> Currently there are three locations for the tests of contrib apps:
> - under tests/ — eg. admin
> - inside the app — eg. auth
> - both — eg. contenttypes
>
> Following the modeltests / regressiontests merge, I propose to move all
> contrib app tests under tests/. This has de following advantages:
> - it makes them easier to discover and prevents accidental duplication
> - they won't be run by './manage.py test'
> - it'll dam up the stream of "if I change setting X then test Y in contrib
> app Z fails"
>
> I'm aware of the idea that contrib apps could include integration tests to
> validate that they're properly used within projects, but I don't believe we
> have any such tests currently.
>
> What do you think?
>

Personally, I'd rather see the opposite -- contrib apps containing their
own tests, and the tests directory containing just the tests for Django
itself. That enforces the fact that the apps really are self contained
apps, and can be tested independently.

The catch -- and the reason why the tests are broken up the way they are --
is that the test framework isn't currently rich enough to support this.
Admin is the best example of this -- in order to test admin, you need
different test models in different test apps, and the current test
framework doesn't support this very well.

This really points at a deficiency in Django's testing tools that really
should be addressed. It's something I've wanted to look at for a while, but
I've never got around to it. It's also something that's likely to be easier
when the app refactor lands, because a lot of the problem lies in the app
cache.

The point about not running the tests under ./manage.py test is a slightly
moot point if we move to the unittest2 discovery mechanism -- last time I
looked at that patch, unittest2 discovery would mean that contrib tests
wouldn't be run (by default) in a project anyway.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.