Re: Test runner with search

[Anssi Kääriäinen]

On 18 kesä, 12:23, Chris Wilson  wrote:
> > I guess this should work by giving a message like this on test name
> > clash: "ERROR: Multiple tests found: ". The
> > user could then copy-paste the correct test and run the test by full
> > identifier.
>
> I might deliberately want to run multiple tests in some cases. I was
> thinking of adding wildcard search support too. It shouldn't be an error,
> it should be explicitly supported.

It seems supporting both could achieved, for example using
".test_method" for exact match, if more than one match found then
error and print identifiers, "test_method" searches runs all matches.
The syntax is just an example, but it should be doable to have both.

I do have a need to run just a single test, as that way it is easier
to use pdb breakpoints. If there is an easy way to get a breakpoint
which fires only when running test_method_x, I am all ears.

> Fine by me. How about a contrib test runner? And does anyone have time to
> write some tests for a test runner?

Contrib isn't the right place for this. The idea is that if it is not
in core or contrib then you can change the interface easily. Even if
this feature is done in contrib, changes have to go through the
deprecation cycle. There are no experimental features in Django, so we
can't test the interface anywhere in core or contrib. Of course, if we
can get a consensus on the correct interface here on django-
developers, or from some existing test runner, then direct core could
be possible.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Chris Wilson]

Hi all,

On Mon, 18 Jun 2012, Anssi Kääriäinen wrote:
> On 16 kesä, 09:29, Russell Keith-Magee 
> >
> > Well, sure -- easier is better -- but is this actually going to be
> > easier? Are you saying that you've never written two test classes with
> > the same test name?

I don't think I have, but I don't mind either. The cost of accidentally 
running an additional unnecessary test (which should pass) is very low, 
about 1 second, compared to remembering the class name of the test. Also 
it's a good incentive to give your tests unique names :) 

> > Just looking at code I've got right now -- I've
> > got a dozen tests named "test_bad_date" scattered throughout my test
> > suite. How do I disambiguate between them?

You can still run tests the old way if you want to. The only difference is 
that if you do something that was previously invalid, i.e. specify a 
simple label that's not a package, it will search for tests instead of 
aborting.

> > We already have a problem with namespace collapsing -- if you have two 
> > test classes with the same name in an app, you can't differentiate 
> > between them. I'd rather not compound the problem by removing class 
> > names from the discovery process as well.

I'd rather not be forced to add yet another component to the test label 
when running it from the commnd line. Running tests should be as quick and 
easy as possible.

> I guess this should work by giving a message like this on test name 
> clash: "ERROR: Multiple tests found: ". The 
> user could then copy-paste the correct test and run the test by full 
> identifier.

I might deliberately want to run multiple tests in some cases. I was 
thinking of adding wildcard search support too. It shouldn't be an error, 
it should be explicitly supported.

> I have two specific needs for this feature:
>   1. When a test errors, I want to have as-easy-as-possible way to
> rerun that test. This could be achieved by just adding a line
> containing the full test identifier
> ("queries.Queries1Tests.test_something") in the error output. We
> currently don't have that, instead you need to collect pieces from the
> test output.

That's true, and annoying, but also requires using the mouse to rerun 
tests. That's where the Bash completion extension would be useful, I 
guess.

> I guess Jannis is right - this and the "record errors / replay" feature 
> belong to third party test runner. When we know the API is correct, then 
> adding these to core would make sense to me. But, as we can easily test 
> these outside core, lets start there.

Fine by me. How about a contrib test runner? And does anyone have time to 
write some tests for a test runner?

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cambridge City FC, Milton Road, Cambridge CB4 1UY

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Anssi Kääriäinen]

On 16 kesä, 09:29, Russell Keith-Magee 
wrote:
> On Thu, Jun 14, 2012 at 6:51 PM, Anssi Kääriäinen
>
>  wrote:
> > On 14 kesä, 13:35, Chris Wilson  wrote:
> >> I've made some improvements (in my view) to the DjangoTestSuiteRunner. I
> >> got tired of having to remember my test class names and of typing so much:
>
> >>   ./manage.py test binder.BinderTest.test_can_create_users
>
> >> This new version searches for tests with the given name in all
> >> INSTALLED_APPS, so you can just type this:
>
> >>   ./manage.py test test_can_create_users
>
> > A big +1 to easier running of single tests. I haven't checked the
> > implementation / API details at all. But I do know I would have
> > immediate use for this feature in running Django's tests.
>
> Well, sure -- easier is better -- but is this actually going to be
> easier? Are you saying that you've never written two test classes with
> the same test name? Just looking at code I've got right now -- I've
> got a dozen tests named "test_bad_date" scattered throughout my test
> suite. How do I disambiguate between them?
>
> We already have a problem with namespace collapsing -- if you have two
> test classes with the same name in an app, you can't differentiate
> between them. I'd rather not compound the problem by removing class
> names from the discovery process as well.

I guess this should work by giving a message like this on test name
clash: "ERROR: Multiple tests found: ". The
user could then copy-paste the correct test and run the test by full
identifier.

I have two specific needs for this feature:
  1. When a test errors, I want to have as-easy-as-possible way to
rerun that test. This could be achieved by just adding a line
containing the full test identifier
("queries.Queries1Tests.test_something") in the error output. We
currently don't have that, instead you need to collect pieces from the
test output.
  2. I am editing some test, and want to then run it. I don't have the
full test identifier anywhere easily copy-pasteable, and here this
feature would help me - I could copy-paste the method name, try to run
it, and if multiple matches were found I would get the full test
identifier and could then run that.

I guess Jannis is right - this and the "record errors / replay"
feature belong to third party test runner. When we know the API is
correct, then adding these to core would make sense to me. But, as we
can easily test these outside core, lets start there.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Jannis Leidel]

On 16.06.2012, at 08:29, Russell Keith-Magee  wrote:

> On Thu, Jun 14, 2012 at 6:51 PM, Anssi Kääriäinen
>  wrote:
>> On 14 kesä, 13:35, Chris Wilson  wrote:
>> 
>> A big +1 to easier running of single tests. I haven't checked the
>> implementation / API details at all. But I do know I would have
>> immediate use for this feature in running Django's tests.
> 
> Well, sure -- easier is better -- but is this actually going to be
> easier? Are you saying that you've never written two test classes with
> the same test name? Just looking at code I've got right now -- I've
> got a dozen tests named "test_bad_date" scattered throughout my test
> suite. How do I disambiguate between them?
> 
> We already have a problem with namespace collapsing -- if you have two
> test classes with the same name in an app, you can't differentiate
> between them. I'd rather not compound the problem by removing class
> names from the discovery process as well.
> 
> Although I'm not a big fan of Nose for testing, this is one thing I
> think they've got right -- explicit path naming of tests, not trying
> to collapse namespaces in order to make it "easier".

Exactly, but I think we can use the included unittest2 discovery for
this purpose instead of rolling our own. In that sense I share Carl's
concern about our test runner being an oddity compared to the rest of
the Python community. Do you see any blockers in moving our test runner
in the direction unittest2 has taken?

>> Another feature which I would find useful would be "--log-failed-
>> cases=somefile". Then you could re-run the failed tests using:
>> 
>>./manage.py test < somefile
>> 
>> I would find this very useful in fixing regressions - run the full
>> test suite - fix regressions - try to run the failed cases - fix more
>> - run the failed cases again - when done rerun full test suite.
>> 
>> Anybody else for this feature?
> 
> Bikeshedding, I'd rather just have a single flag that:
> * logs to a known location
> * searches for the existence of the log when the test run starts; if
> the log exists, only run the failed tests
> * if the test suite passes without error, deletes the log so that the
> next test run executes everything.
> 
> However, +1 to the general idea. "Run the tests that just failed" is a
> very common use case for me; it would be nice to have interface
> support for it.


Don't get me wrong, I also like the idea, but this seems entirely
something that would fit the bill of a 3rd party test runner, instead
of a built-in feature. At least I haven't had this particular
requirement yet and haven't heard of it being a common practice in
test driven development (like for example --failfast).

Jannis


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Russell Keith-Magee]

On Thu, Jun 14, 2012 at 6:51 PM, Anssi Kääriäinen
 wrote:
> On 14 kesä, 13:35, Chris Wilson  wrote:
>> I've made some improvements (in my view) to the DjangoTestSuiteRunner. I
>> got tired of having to remember my test class names and of typing so much:
>>
>>   ./manage.py test binder.BinderTest.test_can_create_users
>>
>> This new version searches for tests with the given name in all
>> INSTALLED_APPS, so you can just type this:
>>
>>   ./manage.py test test_can_create_users
>
> A big +1 to easier running of single tests. I haven't checked the
> implementation / API details at all. But I do know I would have
> immediate use for this feature in running Django's tests.

Well, sure -- easier is better -- but is this actually going to be
easier? Are you saying that you've never written two test classes with
the same test name? Just looking at code I've got right now -- I've
got a dozen tests named "test_bad_date" scattered throughout my test
suite. How do I disambiguate between them?

We already have a problem with namespace collapsing -- if you have two
test classes with the same name in an app, you can't differentiate
between them. I'd rather not compound the problem by removing class
names from the discovery process as well.

Although I'm not a big fan of Nose for testing, this is one thing I
think they've got right -- explicit path naming of tests, not trying
to collapse namespaces in order to make it "easier".

> Another feature which I would find useful would be "--log-failed-
> cases=somefile". Then you could re-run the failed tests using:
>
>    ./manage.py test < somefile
>
> I would find this very useful in fixing regressions - run the full
> test suite - fix regressions - try to run the failed cases - fix more
> - run the failed cases again - when done rerun full test suite.
>
> Anybody else for this feature?

Bikeshedding, I'd rather just have a single flag that:
 * logs to a known location
 * searches for the existence of the log when the test run starts; if
the log exists, only run the failed tests
 * if the test suite passes without error, deletes the log so that the
next test run executes everything.

However, +1 to the general idea. "Run the tests that just failed" is a
very common use case for me; it would be nice to have interface
support for it.

Russ %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Carl Meyer]

On 06/14/2012 07:25 AM, Jannis Leidel wrote:
> On 14.06.2012, at 12:35, Chris Wilson wrote:
>> Hi all,
>> 
>> I've made some improvements (in my view) to the
>> DjangoTestSuiteRunner. I got tired of having to remember my test
>> class names and of typing so much:
>> 
>> ./manage.py test binder.BinderTest.test_can_create_users
>> 
>> This new version searches for tests with the given name in all 
>> INSTALLED_APPS, so you can just type this:
>> 
>> ./manage.py test test_can_create_users
>> 
>> You can find the source code here:
>> 
>> https://github.com/aptivate/intranet-binder/blob/master/testing.py
>> 
>> Is this something that might be interesting to add to Django core?
> 
> 
> Yes! In fact there is a ticket for it already:
> https://code.djangoproject.com/ticket/17365
> 
> I've also released a almost similar tool called
> django-discover-runner based on Carl's work:
> http://pypi.python.org/pypi/django-discover-runner

It looks to me like Chris' added feature is actually kind of orthogonal
to #17365. #17365 is about lifting the "tests must be in tests.py of an
installed app" restriction in favor of unittest2 discovery. Chris' patch
allows searching for a test by name alone rather than full
app.TestCase.name.

Both seem useful enough, and could be combined. It pains me a bit to add
more specialized features to Django's test runner, rather than upstream
in one of the commonly-used Python test runners (such as unittest2), as
I'd rather move in the direction of less custom Django test-runner code
rather than more. But I guess that has to be balanced against the
convenience value of the feature.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Daniel Moisset]

On Thu, Jun 14, 2012 at 7:35 AM, Chris Wilson  wrote:
> Hi all,
>
> I've made some improvements (in my view) to the DjangoTestSuiteRunner. I
> got tired of having to remember my test class names and of typing so much:
>
>  ./manage.py test binder.BinderTest.test_can_create_users
>

In this same line, there's also
http://pypi.python.org/pypi/django-test-autocomplete ; although that's
more bash-oriented

Regards,
 D.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Chris Wilson]

Hi Andrew,

On Thu, 14 Jun 2012, Andrew Godwin wrote:

> +1 from me as well - the test runner you linked to looks reasonably 
> sane, only a few small things that I'm questioning (like why types is 
> imported inside the for loop).

I prefer to keep the imports as close to the code that uses them as 
possible, to make it easier to move code around and reduce "action at a 
distance". That's just my personal preference and of course if it's merged 
into Django then it can use the Django style.

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cambridge City FC, Milton Road, Cambridge CB4 1UY

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Jannis Leidel]

On 14.06.2012, at 12:35, Chris Wilson wrote:

> Hi all,
> 
> I've made some improvements (in my view) to the DjangoTestSuiteRunner. I 
> got tired of having to remember my test class names and of typing so much:
> 
>  ./manage.py test binder.BinderTest.test_can_create_users
> 
> This new version searches for tests with the given name in all 
> INSTALLED_APPS, so you can just type this:
> 
>  ./manage.py test test_can_create_users
> 
> You can find the source code here:
> 
>  https://github.com/aptivate/intranet-binder/blob/master/testing.py
> 
> Is this something that might be interesting to add to Django core?


Yes! In fact there is a ticket for it already: 
https://code.djangoproject.com/ticket/17365

I've also released a almost similar tool called django-discover-runner based on 
Carl's work: http://pypi.python.org/pypi/django-discover-runner

Jannis

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Andrew Godwin]

On 14/06/12 11:51, Anssi Kääriäinen wrote:
> On 14 kesä, 13:35, Chris Wilson  wrote:
>> I've made some improvements (in my view) to the DjangoTestSuiteRunner. I
>> got tired of having to remember my test class names and of typing so much:
>>
>>   ./manage.py test binder.BinderTest.test_can_create_users
>>
>> This new version searches for tests with the given name in all
>> INSTALLED_APPS, so you can just type this:
>>
>>   ./manage.py test test_can_create_users
> 
> A big +1 to easier running of single tests. I haven't checked the
> implementation / API details at all. But I do know I would have
> immediate use for this feature in running Django's tests.

+1 from me as well - the test runner you linked to looks reasonably
sane, only a few small things that I'm questioning (like why types is
imported inside the for loop).

> Another feature which I would find useful would be "--log-failed-
> cases=somefile". Then you could re-run the failed tests using:
> 
> ./manage.py test < somefile
> 
> I would find this very useful in fixing regressions - run the full
> test suite - fix regressions - try to run the failed cases - fix more
> - run the failed cases again - when done rerun full test suite.
> 
> Anybody else for this feature?

Interesting idea - I can see that being useful, too. Presumably we're
just talking an output with one test function reference per line?

Not sure about having to read from stdin - I can see that potentially
interfering with things like pdb.set_trace() - but there's always the
option of adding a command-line flag too.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Anssi Kääriäinen]

On 14 kesä, 13:35, Chris Wilson  wrote:
> I've made some improvements (in my view) to the DjangoTestSuiteRunner. I
> got tired of having to remember my test class names and of typing so much:
>
>   ./manage.py test binder.BinderTest.test_can_create_users
>
> This new version searches for tests with the given name in all
> INSTALLED_APPS, so you can just type this:
>
>   ./manage.py test test_can_create_users

A big +1 to easier running of single tests. I haven't checked the
implementation / API details at all. But I do know I would have
immediate use for this feature in running Django's tests.



Another feature which I would find useful would be "--log-failed-
cases=somefile". Then you could re-run the failed tests using:

./manage.py test < somefile

I would find this very useful in fixing regressions - run the full
test suite - fix regressions - try to run the failed cases - fix more
- run the failed cases again - when done rerun full test suite.

Anybody else for this feature?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.