Re: Replace the default django test runner to a more robust testing framework (like pytest)

[James Bennett]

On Tue, Dec 15, 2020 at 6:07 PM Diptesh Choudhuri
 wrote:
> Instead of removing the default test runner altogether, I can work on adding 
> a TESTING_BACKEND variable in settings.py with possible values being 
> 'django.testing.backends.pytest' (default) or 
> 'django.testing.backends.DefaultRunner'. This way we need not compromise on 
> the features that the default runner provides over pytest. Meanwhile I could 
> work on some the issues pytest has by contributing to pytest-dev.

I would be against both the original proposal to switch Django to
pytest, and this proposal.

And it's not just because I'm a little bit biased here -- though I am,
in that I personally dislike pytest and when I use it in personal
projects I use it solely as a runner, where the tests themselves all
use the unittest and mock modules in the Python standard library.

There are several independent reasons why I don't think either of
these proposals is a good fit for Django:

One is that even though it's easier at first to add dependencies to
Django now than it was 15 years ago, every dependency still should be
carefully considered. It wasn't too long ago that this proposal would
have been "switch the Django test runner to nose", and now nose is
unmaintained and deprecated (its last release was in 2015) and though
there's a maintained "nose2" fork, it's not compatible with the
original and doesn't have anywhere near the development momentum the
original had. You might observe that pytest shows no signs of having
that happen to it, but the fact that it *can* happen has to be
considered when we're talking about integrating it into one of
Django's core components; we can think it's likely, and be hopeful,
that pytest would still be around and actively maintained 5 years from
now, but we can be much more certain that the Python standard
library's unittest module will still be around and actively
maintained.

Another is that pytest would be effectively a test-only dependency,
and the history of *those* in Python projects is largely an unhappy
and complicated one. It would create a potential risk for unwary users
(forgetting to install the extra dependency, or to do "pip install
Django[pytest]" or similar), or force every installation of Django --
including production installations which often don't want a large
third-party framework that won't be used at all in production -- to
also install pytest.

Still another is the basic incompatibility between pytest and unittest
runners. While it's true that Django lets you configure a lot of
things, they all are handled through carefully normalized APIs, so
that you don't notice the difference. This is what lets you start out
development with a local SQLite database and switch to Postgres when
you deploy, for example, or swap out cache or session or
authentication or email-sending backends without needing to rewrite
your own code that uses those features, because Django provides
normalized APIs for accessing whichever backend it's been configured
to use. But pytest is not that simple: pytest can run tests written in
unittest style, but unittest cannot run tests written in pytest style.
Only pytest -- which understands its own conventions for setup,
fixture loading, and so on -- can do this. Which means that if I write
my Django app using pytest style, and you want to use it in your
project that's otherwise completely unittest style, you cannot
integrate with full testing support unless you also switch your whole
project's configuration to pytest. That's a *big* difference from,
say, switching to a new auth backend.

So I'm in agreement with others in this thread that if you want to
improve things for people who prefer to use pytest, contributing to
pytest, pytest-django, and related projects would be a fine way to do
so. But trying to switch Django itself to pytest, or to introduce a
"test backend" concept that would have a pytest runner, does not seem
to me like a good idea.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAL13Cg83sKgy7qrVMNBEuhrt4wRbnTv6DOnPJejc_0N0nhHmSA%40mail.gmail.com.

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Diptesh Choudhuri]

I understand what you mean. Here's a proposal.

Instead of removing the default test runner altogether, I can work on 
adding a *TESTING_BACKEND *variable in *settings.py *with possible values 
being *'django.testing.backends.pytest' *(default) or 
*'django.testing.backends.DefaultRunner'*. This way we need not compromise 
on the features that the default runner provides over pytest. Meanwhile I 
could work on some the issues pytest has by contributing to pytest-dev.

One small question- will a contribution to repositories such as *pytest-dev 
*or *graphene-django *count as contributions to the Django organization in 
GSoC?

Best
Diptesh

PS: I am also conducting a small survey about the same. I will post the 
results here once I get sufficient responses. It would be interesting to 
see how to common public reacts to such a change.

On Tuesday, December 15, 2020 at 10:56:37 PM UTC+5:30 Adam Johnson wrote:

> except the first one- this can be easily solved using a pytest hook
>
>
> It's not easy at all. I tried 4 years ago and failed to find a 
> satisfactory way of running the checks at the appropriate time since the 
> pytest lifecycle (db setup, reporting, etc.) is so different from unittest: 
> https://github.com/pytest-dev/pytest-django/pull/414
>
> Instead they are kind of addons that'd be nice to have.
>>
>
> That's not how we treat stability in Django. These features are necessary, 
> core features for the projects that use them, and we treat them as 
> necessary. Please read the Deprecation Policy: 
> https://docs.djangoproject.com/en/3.1/internals/release-process/#internal-release-deprecation-policy
>  
> .
>
> I appreciate your willingness to improve things and I agree generally 
> pytest is a better test runner experience. But please channel some of your 
> energy into understanding where Django is, where pytest is, how 
> contributions work etc. You will get further than having a long back and 
> forth where you keep asserting that it would be easy to implement.
>
> On Tue, 15 Dec 2020 at 16:58, Diptesh Choudhuri  
> wrote:
>
>> *Adam,*
>> Thanks for your detailed feedback.
>> I will be honest, I couldn't understand any of the points you mentioned 
>> (except the first one- this can be easily solved using a pytest hook). As 
>> you said, these are very rare cases indeed. As a django and pytest user for 
>> some time, I would have encountered atleast one of them when I wrote my 
>> tests. However I never have and hence I would say that these are very rare 
>> cases indeed. 
>> What I am trying to say is that if we include pytest in django core, the 
>> development of these features will only be accelerated. The default runner 
>> advantages you mentioned are not really features that are missing in 
>> pytest. Instead they are kind of addons that'd be nice to have. But the 
>> advantages that I mention of pytest over the default runner are significant 
>> and can tremendously improve developer experience while writing tests.
>>
>> Please don't think I am disregarding your points. I respect and value 
>> your opinions.
>>
>> Best
>> Diptesh
>>
>> On Tuesday, December 15, 2020 at 10:10:06 PM UTC+5:30 Diptesh Choudhuri 
>> wrote:
>>
>>> > A case could be made along the same lines as was made for South.
>>>
>>> Could you please elaborate a bit on this? Maybe provide the relevant 
>>> tickets? I am quite new to this community.
>>>
>>> Thanks!
>>> Best
>>> Diptesh
>>>
>>> On Tuesday, December 15, 2020 at 10:01:03 PM UTC+5:30 
>>> arvind...@gmail.com wrote:
>>>
>>>> A case could be made along the same lines as was made for South. 
>>>>
>>>> Get Outlook for Android <https://aka.ms/ghei36>
>>>>
>>>> --
>>>> *From:* django-d...@googlegroups.com  on 
>>>> behalf of Mariusz Felisiak 
>>>> *Sent:* Tuesday, December 15, 2020 9:58:56 PM
>>>> *To:* Django developers (Contributions to Django itself) <
>>>> django-d...@googlegroups.com>
>>>> *Subject:* Re: Replace the default django test runner to a more robust 
>>>> testing framework (like pytest) 
>>>>  
>>>> Hi,
>>>>
>>>> I agree with René and Adam, pytest and pytest-django already exist, 
>>>> and I don't see why we should copy it to Django or change the default test 
>>>> runner.
>>>>
>>>> Best,
>>>> Mariusz
>>>>
>>>> -- 
>>>> You received this message because you are subscribed to the Googl

Re: ARe: Replace the default django test runner to a more robust testing framework (like pytest)

[Adam Johnson]

>
> except the first one- this can be easily solved using a pytest hook


It's not easy at all. I tried 4 years ago and failed to find a satisfactory
way of running the checks at the appropriate time since the pytest
lifecycle (db setup, reporting, etc.) is so different from unittest:
https://github.com/pytest-dev/pytest-django/pull/414

Instead they are kind of addons that'd be nice to have.
>

That's not how we treat stability in Django. These features are necessary,
core features for the projects that use them, and we treat them as
necessary. Please read the Deprecation Policy:
https://docs.djangoproject.com/en/3.1/internals/release-process/#internal-release-deprecation-policy
.

I appreciate your willingness to improve things and I agree generally
pytest is a better test runner experience. But please channel some of your
energy into understanding where Django is, where pytest is, how
contributions work etc. You will get further than having a long back and
forth where you keep asserting that it would be easy to implement.

On Tue, 15 Dec 2020 at 16:58, Diptesh Choudhuri 
wrote:

> *Adam,*
> Thanks for your detailed feedback.
> I will be honest, I couldn't understand any of the points you mentioned
> (except the first one- this can be easily solved using a pytest hook). As
> you said, these are very rare cases indeed. As a django and pytest user for
> some time, I would have encountered atleast one of them when I wrote my
> tests. However I never have and hence I would say that these are very rare
> cases indeed.
> What I am trying to say is that if we include pytest in django core, the
> development of these features will only be accelerated. The default runner
> advantages you mentioned are not really features that are missing in
> pytest. Instead they are kind of addons that'd be nice to have. But the
> advantages that I mention of pytest over the default runner are significant
> and can tremendously improve developer experience while writing tests.
>
> Please don't think I am disregarding your points. I respect and value your
> opinions.
>
> Best
> Diptesh
>
> On Tuesday, December 15, 2020 at 10:10:06 PM UTC+5:30 Diptesh Choudhuri
> wrote:
>
>> > A case could be made along the same lines as was made for South.
>>
>> Could you please elaborate a bit on this? Maybe provide the relevant
>> tickets? I am quite new to this community.
>>
>> Thanks!
>> Best
>> Diptesh
>>
>> On Tuesday, December 15, 2020 at 10:01:03 PM UTC+5:30 arvind...@gmail.com
>> wrote:
>>
>>> A case could be made along the same lines as was made for South.
>>>
>>> Get Outlook for Android <https://aka.ms/ghei36>
>>>
>>> --
>>> *From:* django-d...@googlegroups.com  on
>>> behalf of Mariusz Felisiak 
>>> *Sent:* Tuesday, December 15, 2020 9:58:56 PM
>>> *To:* Django developers (Contributions to Django itself) <
>>> django-d...@googlegroups.com>
>>> *Subject:* Re: Replace the default django test runner to a more robust
>>> testing framework (like pytest)
>>>
>>> Hi,
>>>
>>> I agree with René and Adam, pytest and pytest-django already exist,
>>> and I don't see why we should copy it to Django or change the default test
>>> runner.
>>>
>>> Best,
>>> Mariusz
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/60a4a090-8ff5-477e-9651-db1fbf5c397bn%40googlegroups.com
>>> <https://groups.google.com/d/msgid/django-developers/60a4a090-8ff5-477e-9651-db1fbf5c397bn%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/39dc54fb-43e9-424f-9ca3-10938741d733n%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/39dc54fb-43e9-424f-9ca3-10938741d733n%40googlegroups.com?utm_medium=email_source=footer>
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0Wtg3rsLHrZtqoYke5e7Mg3Hc9b1xhHasBBvNmQdxU3A%40mail.gmail.com.

ARe: Replace the default django test runner to a more robust testing framework (like pytest)

[Diptesh Choudhuri]

*Adam,*
Thanks for your detailed feedback.
I will be honest, I couldn't understand any of the points you mentioned 
(except the first one- this can be easily solved using a pytest hook). As 
you said, these are very rare cases indeed. As a django and pytest user for 
some time, I would have encountered atleast one of them when I wrote my 
tests. However I never have and hence I would say that these are very rare 
cases indeed. 
What I am trying to say is that if we include pytest in django core, the 
development of these features will only be accelerated. The default runner 
advantages you mentioned are not really features that are missing in 
pytest. Instead they are kind of addons that'd be nice to have. But the 
advantages that I mention of pytest over the default runner are significant 
and can tremendously improve developer experience while writing tests.

Please don't think I am disregarding your points. I respect and value your 
opinions.

Best
Diptesh

On Tuesday, December 15, 2020 at 10:10:06 PM UTC+5:30 Diptesh Choudhuri 
wrote:

> > A case could be made along the same lines as was made for South.
>
> Could you please elaborate a bit on this? Maybe provide the relevant 
> tickets? I am quite new to this community.
>
> Thanks!
> Best
> Diptesh
>
> On Tuesday, December 15, 2020 at 10:01:03 PM UTC+5:30 arvind...@gmail.com 
> wrote:
>
>> A case could be made along the same lines as was made for South. 
>>
>> Get Outlook for Android <https://aka.ms/ghei36>
>>
>> --
>> *From:* django-d...@googlegroups.com  on 
>> behalf of Mariusz Felisiak 
>> *Sent:* Tuesday, December 15, 2020 9:58:56 PM
>> *To:* Django developers (Contributions to Django itself) <
>> django-d...@googlegroups.com>
>> *Subject:* Re: Replace the default django test runner to a more robust 
>> testing framework (like pytest) 
>>  
>> Hi,
>>
>> I agree with René and Adam, pytest and pytest-django already exist, 
>> and I don't see why we should copy it to Django or change the default test 
>> runner.
>>
>> Best,
>> Mariusz
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/60a4a090-8ff5-477e-9651-db1fbf5c397bn%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/60a4a090-8ff5-477e-9651-db1fbf5c397bn%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/39dc54fb-43e9-424f-9ca3-10938741d733n%40googlegroups.com.

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Diptesh Choudhuri]

> A case could be made along the same lines as was made for South.

Could you please elaborate a bit on this? Maybe provide the relevant 
tickets? I am quite new to this community.

Thanks!
Best
Diptesh

On Tuesday, December 15, 2020 at 10:01:03 PM UTC+5:30 arvind...@gmail.com 
wrote:

> A case could be made along the same lines as was made for South. 
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> --
> *From:* django-d...@googlegroups.com  on 
> behalf of Mariusz Felisiak 
> *Sent:* Tuesday, December 15, 2020 9:58:56 PM
> *To:* Django developers (Contributions to Django itself) <
> django-d...@googlegroups.com>
> *Subject:* Re: Replace the default django test runner to a more robust 
> testing framework (like pytest) 
>  
> Hi,
>
> I agree with René and Adam, pytest and pytest-django already exist, 
> and I don't see why we should copy it to Django or change the default test 
> runner.
>
> Best,
> Mariusz
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-develop...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/60a4a090-8ff5-477e-9651-db1fbf5c397bn%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/django-developers/60a4a090-8ff5-477e-9651-db1fbf5c397bn%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/13d8fdb1-74b3-4174-9cc9-c1b046ba124en%40googlegroups.com.

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Diptesh Choudhuri]

> I agree with René and Adam, pytest and pytest-django already exist, and I 
don't see why we should copy it to Django or change the default test runner.
I am not talking about replicating any code-base whatsoever. What I am 
proposing instead is official django support for pytest. Kind of like how 
django supports argon2 for password hashing. We can make this a feature 
using *pip install django[pytest]* and make appropriate changes to the 
*startproject 
*command to create some sane test and configuration files.

Thanks for your input!

Best
Diptesh

On Tuesday, December 15, 2020 at 9:58:57 PM UTC+5:30 Mariusz Felisiak wrote:

> Hi,
>
> I agree with René and Adam, pytest and pytest-django already exist, 
> and I don't see why we should copy it to Django or change the default test 
> runner.
>
> Best,
> Mariusz
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5bf973b5-c843-43e4-a32e-f63ad3c87c9cn%40googlegroups.com.

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Arvind Nedumaran]

A case could be made along the same lines as was made for South.

Get Outlook for Android<https://aka.ms/ghei36>


From: django-developers@googlegroups.com  
on behalf of Mariusz Felisiak 
Sent: Tuesday, December 15, 2020 9:58:56 PM
To: Django developers (Contributions to Django itself) 

Subject: Re: Replace the default django test runner to a more robust testing 
framework (like pytest)

Hi,

I agree with René and Adam, pytest and pytest-django already exist, and I 
don't see why we should copy it to Django or change the default test runner.

Best,
Mariusz


--
You received this message because you are subscribed to the Google Groups 
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
django-developers+unsubscr...@googlegroups.com<mailto:django-developers+unsubscr...@googlegroups.com>.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/60a4a090-8ff5-477e-9651-db1fbf5c397bn%40googlegroups.com<https://groups.google.com/d/msgid/django-developers/60a4a090-8ff5-477e-9651-db1fbf5c397bn%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/BYAPR14MB29187430911979D1A75E7E62A3C60%40BYAPR14MB2918.namprd14.prod.outlook.com.

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Mariusz Felisiak]

Hi,

I agree with René and Adam, pytest and pytest-django already exist, and 
I don't see why we should copy it to Django or change the default test 
runner.

Best,
Mariusz

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/60a4a090-8ff5-477e-9651-db1fbf5c397bn%40googlegroups.com.

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Diptesh Choudhuri]

*Ahmad*, thanks for your views! Here's what I think.

> I personally believe it should be in core because a lot of custom 
features we'd like to add to the test runner already exist in pytest's rich 
ecosystem. Just as Tom mentioned, adding timings was a lot harder than 
expected while such a feature already exists in pytest.
Adding to this, since features like this already exist in *pytest*, I don't 
think we should reinvent the wheel and try to implement them from scratch.

I would say the main reason I would like integrate pytest into the django 
core is that since it is much loved among the community, it only makes 
sense to make it official. New users won't have to wrack their heads over 
unittest's basics (it took me a long time to understand *setUp *and *tearDown 
*and even longer to actually use it in a project). *pytest-django *also 
requires you to write much less code for the actual test (compare 
*self.assertEquals(thing1, 
thing2) *with *assert thing1 == thing2*).

I don't intend to write documentation only. I believe this idea has a lot 
of scope, and by integrating the features I have already suggested and the 
ones that will be suggested in the near future, I will end up writing a lot 
of code. So it will fit GSoC's job description. However I thank you for 
mentioning Google Summer of Docs, I will definitely look into it.
On Tuesday, December 15, 2020 at 9:15:47 PM UTC+5:30 Ahmad A. Hussein wrote:

> Hi Diptesh,
>
> This is a great idea! (And one that has had a fair bit of discussion). I 
> don't think anyone holds an opinion against supporting pytest as a testing 
> tool, but I believe the main hurdle is laying a convincing argument why 
> this support should be in Django Core versus remaining in its current 
> third-party state. I personally believe it should be in core because a lot 
> of custom features we'd like to add to the test runner already exist in 
> pytest's rich ecosystem. Just as Tom mentioned, adding timings was a lot 
> harder than expected while such a feature already exists in pytest. 
>
> If we aren't going to add it to core, the question I'd like to raise (and 
> preferably get an answer from experienced pytest-django users/developers) 
> is what changes can we make in core to make it easier for the pytest-django 
> package.
>
> Lastly, if you intend to purely write documentation, I would recommend you 
> apply for next year's Google Season of Docs instead of Google Summer of 
> Code. It would be a better fit for the job description
>
> On Tue, Dec 15, 2020 at 3:52 PM Diptesh Choudhuri  
> wrote:
>
>> Though the django default runner gets the job done, I think its high time 
>> to leverage the power of dedicated frameworks for testing django apps. I 
>> will take up the case of *pytest* here because other frameworks like *nose 
>> *are not widely used.
>>
>> For everyone who has used pytest for testing in django, I think we will 
>> all agree that pytest is much better suited to this task. 
>>
>>1. The passing tests look better.
>>2. Failing tests are much more verbose with where the error actually 
>>occurred.
>>3. Parameterizing the test cases using the default runner requires a 
>>third party library and looks difficult 
>>
>> 
>>  (I 
>>will be honest, I haven't used this package myself at all). 
>>4. Injecting runtime conditions into the tests is quite difficult, if 
>>not impossible, and requires a lot of hacky workarounds.
>>5. I am not sure about this, but the default runner also doesn't 
>>cache results from previously run tests, making retesting unchanged tests 
>>much slower.
>>6. This might sound non-trivial but typing *pytest *is much more 
>>easier than typing *python manage.py test *into the command line.
>>7. IDEs like VSCode can discover tests written for *pytest *but not 
>>the ones written using the default runner. This might not sound a big 
>> deal, 
>>but it is infinitely more easier to run tests using the GUI that these 
>> IDEs 
>>have. They provide real-time feedback and show exactly which tests have 
>>passed and what haven't in the code source itself. Here 
>> is a picture showing what I mean.
>>8. The directory structure created by *python manage.py startapp 
>>my_app *creates a *tests.py *file. This goes against testing best 
>>practices which say that all tests should be aggregated at one place (as 
>>opposed to inside individual apps).
>>9. A single *tests.py *file encourages new, inexperienced developers 
>>to write all their tests in a single file. This, yet again, goes against 
>>testing best practices which put a lot of emphasis on modularization of 
>>tests. Models should be tested in *test_models.py*, views in 
>> *test_views.py 
>>*etc. 
>>10. Though *unittest*'s *setUp *and *tearDown *work 

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Adam Johnson]

Hi

Whilst I appreciate the effort and love pytest, I don't think this it's
feasible at the moment to even suggest to users that they use pytest, let
alone make it a default or port Django's unittest-based framework.

For a first step, we have our backwards compatibility policy. Any move away
from the unittest-based runner would break users' customizations such as
subclasses

Secondly, pytest-django may seem mature, but it does not have feature
parity with Django's test framework. Off the top of my head:

   - It doesn't run system checks before tests.
   - It doesn't support a setUpTestData-style shared database fixture with
   rollback between tests in the "functional" style of tests.
   - It only has one test database fixture, so it either sets up *all* test
   databases or none. Django's TestCase supports a "databases" attribute to
   list the required databases needed in the tests, allowing tests that only
   use a subset of the databases to only set up those databases.
   - It doesn't support the TestCase.available_apps attribute.
   - I don't believe it rearranges TransactionTestCase tests to the end of
   the test run, allowing them to slow down the run.

There are other examples in the pytest-django issues list. Naturally it's
the rare edge case, less-used features that few contributors want to work
on that are missing - those users who need them find that pytest-django
isn't for them and stick with the django testing framework. And there is a
fairly limited bunch of people contributing to both pytest-django and
django's test framework.

Third, pytest itself does not have complete feature parity with the basic
run-tests options of Django's test framework. For example, Django has the
--reverse flag to run tests in the reverse order. I suggested this be added
to pytest core, but it was decided to make it into a plugin instead,
pytest-reverse, which I now maintain.

I appreciate your willingness to contribute. I think any energy in this
space is best focussed on improving pytest-django to improve its feature
coverage, and that Django should not have any official support or
suggestion to use pytest-django at this time. We could perhaps link to
pytest as an alternative in one of the test topic guides, however we do
tend to avoid such documentation links since they are prone to bitrot.

Thanks,

Adam


On Tue, 15 Dec 2020 at 15:49, Diptesh Choudhuri 
wrote:

> Thank you everyone for your inputs! I don't know how you all are using the
> quote feature, so I will address you all by your names.
>
>
>- *Jacob*: Thank you for your support! The ticket you linked to is
>very helpful. It is upside of 5 years old. *pytest *has evolved a lot
>over that time, as has django itself. The points raised in the ticket
>itself are still valid 5 years later and as django is a mature project now,
>some of its development prowess must go to specifically fixing its
>weaknesses (I believe the default runner is one). That being said, I also
>opened another ticket on the same website today about this. Should I close
>that and bump the one you linked instead?
>- *Tobias*: I understand most of the issues raised are personal
>preferences. However these preferences have been cultivated by the django
>community itself, so it would be safe to assume that a lot of django users
>would have the same preferences. There is no harm in making these
>preferences official (by including them in the official docs). Once people
>have a common convention to follow, they will only appreciate it because it
>makes their lives easier. *pytest *indeed does a lot of magic behind
>the scenes but I intend to demystify this by writing extensive
>documentation in the *Advanced Testing *Topics page in the django
>docs. As for the *subTest *method, I admit to not having done enough
>research about it and it is completely new to me. However, since its not in
>common usage, we can safely assume that people don't like using it, don't
>you agree?
>- *Tom*: Totally agree with everything you said! Especially being able
>to write more "pythonic" code with *pytest*. Even though our
>definitions of pythonic might be different, I would boldly suggest that
>functional testing, which *pytest *prefers, looks much more beautiful
>and readable than unittest's. class based ones.
>- *René: *Though the option exists, I would humbly argue that its much
>easier to invoke *pytest *than it is to invoke *python manage.py test*.
>Most of my focus would be on improving the docs, along with a few changes
>to code-base itself to make pytest the default and generate a sane
>*pytest.ini*. On further thought, we can completely remove *pytest.ini
>*and instead do all configuration from *settings.py *itself. Though 
> *pytest-django
>*exists, I feel that it can be very unforgiving to users who are
>completely new to testing (I was one not long ago). It 

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Diptesh Choudhuri]

Thank you everyone for your inputs! I don't know how you all are using the 
quote feature, so I will address you all by your names.


   - *Jacob*: Thank you for your support! The ticket you linked to is very 
   helpful. It is upside of 5 years old. *pytest *has evolved a lot over 
   that time, as has django itself. The points raised in the ticket itself are 
   still valid 5 years later and as django is a mature project now, some of 
   its development prowess must go to specifically fixing its weaknesses (I 
   believe the default runner is one). That being said, I also opened another 
   ticket on the same website today about this. Should I close that and bump 
   the one you linked instead?
   - *Tobias*: I understand most of the issues raised are personal 
   preferences. However these preferences have been cultivated by the django 
   community itself, so it would be safe to assume that a lot of django users 
   would have the same preferences. There is no harm in making these 
   preferences official (by including them in the official docs). Once people 
   have a common convention to follow, they will only appreciate it because it 
   makes their lives easier. *pytest *indeed does a lot of magic behind the 
   scenes but I intend to demystify this by writing extensive documentation in 
   the *Advanced Testing *Topics page in the django docs. As for the *subTest 
   *method, I admit to not having done enough research about it and it is 
   completely new to me. However, since its not in common usage, we can safely 
   assume that people don't like using it, don't you agree?
   - *Tom*: Totally agree with everything you said! Especially being able 
   to write more "pythonic" code with *pytest*. Even though our definitions 
   of pythonic might be different, I would boldly suggest that functional 
   testing, which *pytest *prefers, looks much more beautiful and readable 
   than unittest's. class based ones.
   - *René: *Though the option exists, I would humbly argue that its much 
   easier to invoke *pytest *than it is to invoke *python manage.py test*. 
   Most of my focus would be on improving the docs, along with a few changes 
   to code-base itself to make pytest the default and generate a sane 
   *pytest.ini*. On further thought, we can completely remove *pytest.ini *and 
   instead do all configuration from *settings.py *itself. Though 
*pytest-django 
   *exists, I feel that it can be very unforgiving to users who are 
   completely new to testing (I was one not long ago). It doesn't work right 
   out of the box. I sincerely believe this must and should be worked upon. 
   
Thank you everyone for your inputs and I would love to hear some more 
feedback. Also if anyone could teach me how to use the quote function, I 
would highly appreciate it.

Best
Diptesh
On Tuesday, December 15, 2020 at 8:47:24 PM UTC+5:30 re...@fleschenberg.net 
wrote:

> Hi,
>
> On 15.12.20 15:48, Tobias Bengfort wrote:
> > My issue is mostly that pytest uses a lot of magic that is hard to 
> > debug, e.g. the overwritten assert statement or implicit injection of 
> > fixtures. Most pro arguments seem to depend on personal taste.
>
> It should be noted that you can use pytest / pytest-django as a test 
> runner without having to use any other pytest features (it can run 
> unittest-style tests, you don't have to rewrite them). I do think it has 
> some features in the test runner. For example, the -k command line 
> option to quickly select tests, or --last-failed to rerun failed tests.
>
> However, since pytest-django exists and has been working flawlessly for 
> me for several years, I am not sure how much benefit there would be in 
> having this in Django itself.
>
> Regards,
> René
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/64886590-2d95-4c11-ad8d-24d0a746001dn%40googlegroups.com.

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Ahmad A. Hussein]

Hi Diptesh,

This is a great idea! (And one that has had a fair bit of discussion). I
don't think anyone holds an opinion against supporting pytest as a testing
tool, but I believe the main hurdle is laying a convincing argument why
this support should be in Django Core versus remaining in its current
third-party state. I personally believe it should be in core because a lot
of custom features we'd like to add to the test runner already exist in
pytest's rich ecosystem. Just as Tom mentioned, adding timings was a lot
harder than expected while such a feature already exists in pytest.

If we aren't going to add it to core, the question I'd like to raise (and
preferably get an answer from experienced pytest-django users/developers)
is what changes can we make in core to make it easier for the pytest-django
package.

Lastly, if you intend to purely write documentation, I would recommend you
apply for next year's Google Season of Docs instead of Google Summer of
Code. It would be a better fit for the job description

On Tue, Dec 15, 2020 at 3:52 PM Diptesh Choudhuri <
diptesh.choudh...@gmail.com> wrote:

> Though the django default runner gets the job done, I think its high time
> to leverage the power of dedicated frameworks for testing django apps. I
> will take up the case of *pytest* here because other frameworks like *nose
> *are not widely used.
>
> For everyone who has used pytest for testing in django, I think we will
> all agree that pytest is much better suited to this task.
>
>1. The passing tests look better.
>2. Failing tests are much more verbose with where the error actually
>occurred.
>3. Parameterizing the test cases using the default runner requires a
>third party library and looks difficult
>
> 
>  (I
>will be honest, I haven't used this package myself at all).
>4. Injecting runtime conditions into the tests is quite difficult, if
>not impossible, and requires a lot of hacky workarounds.
>5. I am not sure about this, but the default runner also doesn't cache
>results from previously run tests, making retesting unchanged tests much
>slower.
>6. This might sound non-trivial but typing *pytest *is much more
>easier than typing *python manage.py test *into the command line.
>7. IDEs like VSCode can discover tests written for *pytest *but not
>the ones written using the default runner. This might not sound a big deal,
>but it is infinitely more easier to run tests using the GUI that these IDEs
>have. They provide real-time feedback and show exactly which tests have
>passed and what haven't in the code source itself. Here
> is a picture showing what I mean.
>8. The directory structure created by *python manage.py startapp
>my_app *creates a *tests.py *file. This goes against testing best
>practices which say that all tests should be aggregated at one place (as
>opposed to inside individual apps).
>9. A single *tests.py *file encourages new, inexperienced developers
>to write all their tests in a single file. This, yet again, goes against
>testing best practices which put a lot of emphasis on modularization of
>tests. Models should be tested in *test_models.py*, views in *test_views.py
>*etc.
>10. Though *unittest*'s *setUp *and *tearDown *work fine, I personally
>find that *pytest*'s fixture system provides a bit more freedom. I
>might want to put all my test methods in one *class*, but I might not
>want to run the *setUp *and *tearDown *for every test method. (Let me
>know if this point is a bit unclear.)
>
> *pytest *and *pytest-django *are quire mature frameworks and don't need a
> lot of changes. We could make it easier to setup tests right out of the
> box. For example, *django-admin startproject my_project *could create a 
> *pytest.ini
> *right out of the box in the project root.
>
> Most of my focus would be on rewriting and adding documentation. Even
> though widely used, *pytest *and *pytest-django *are pretty difficult to
> get started with for a newbie. I intend to fix this by adding a lot of
> examples to the docs (maybe even make a separate repository for this).
>
> *NOTE: I intend to work on this project for GSoC 2021. I am just scouting
> beforehand to see how the community receives it.*
>
> Please let me know what you think of this idea. Any critiques will be
> welcome.
>
> Best,
> Diptesh
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/8c5e0baa-5104-4876-a936-8792b3c8d5b8n%40googlegroups.com
> 

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[René Fleschenberg]

Hi,

On 15.12.20 15:48, Tobias Bengfort wrote:
My issue is mostly that pytest uses a lot of magic that is hard to 
debug, e.g. the overwritten assert statement or implicit injection of 
fixtures. Most pro arguments seem to depend on personal taste.


It should be noted that you can use pytest / pytest-django as a test 
runner without having to use any other pytest features (it can run 
unittest-style tests, you don't have to rewrite them). I do think it has 
some features in the test runner. For example, the -k command line 
option to quickly select tests, or --last-failed to rerun failed tests.


However, since pytest-django exists and has been working flawlessly for 
me for several years, I am not sure how much benefit there would be in 
having this in Django itself.


Regards,
René

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3810f1f8-fc22-6fea-51d1-5bfa1b72523f%40fleschenberg.net.

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Tom Forbes]

I would be very much in favour of this. The developer experience of pytest is 
far better than the unittest module, the tests look more Pythonic and there is 
a lot more of an ecosystem we can tap into.

There was some recent work done about adding timings to the Djanfo test suite 
as part of the multi-process test runner. This was way more difficult than it 
should have been.

Unittest, along with the logging module, are very old and archaic stdlib 
modules that are ok-ish, but definitely hobbled by legacy interfaces and a 
“Java inspired” pre-pythonic code style.

I think it’s safe to say that most of the Python ecosystem has settled on 
pytest, and if we go along with that as well then we can reduce our maintenance 
burden whilst making our users happier.

Tom

> On 15 Dec 2020, at 13:52, Diptesh Choudhuri  
> wrote:
> 
> Though the django default runner gets the job done, I think its high time to 
> leverage the power of dedicated frameworks for testing django apps. I will 
> take up the case of pytest here because other frameworks like nose are not 
> widely used.
> 
> For everyone who has used pytest for testing in django, I think we will all 
> agree that pytest is much better suited to this task. 
> The passing tests look better.
> Failing tests are much more verbose with where the error actually occurred.
> Parameterizing the test cases using the default runner requires a third party 
> library and looks difficult (I will be honest, I haven't used this package 
> myself at all). 
> Injecting runtime conditions into the tests is quite difficult, if not 
> impossible, and requires a lot of hacky workarounds.
> I am not sure about this, but the default runner also doesn't cache results 
> from previously run tests, making retesting unchanged tests much slower.
> This might sound non-trivial but typing pytest is much more easier than 
> typing python manage.py test into the command line.
> IDEs like VSCode can discover tests written for pytest but not the ones 
> written using the default runner. This might not sound a big deal, but it is 
> infinitely more easier to run tests using the GUI that these IDEs have. They 
> provide real-time feedback and show exactly which tests have passed and what 
> haven't in the code source itself. Here is a picture showing what I mean.
> The directory structure created by python manage.py startapp my_app creates a 
> tests.py file. This goes against testing best practices which say that all 
> tests should be aggregated at one place (as opposed to inside individual 
> apps).
> A single tests.py file encourages new, inexperienced developers to write all 
> their tests in a single file. This, yet again, goes against testing best 
> practices which put a lot of emphasis on modularization of tests. Models 
> should be tested in test_models.py, views in test_views.py etc. 
> Though unittest's setUp and tearDown work fine, I personally find that 
> pytest's fixture system provides a bit more freedom. I might want to put all 
> my test methods in one class, but I might not want to run the setUp and 
> tearDown for every test method. (Let me know if this point is a bit unclear.)
> pytest and pytest-django are quire mature frameworks and don't need a lot of 
> changes. We could make it easier to setup tests right out of the box. For 
> example, django-admin startproject my_project could create a pytest.ini right 
> out of the box in the project root. 
> 
> Most of my focus would be on rewriting and adding documentation. Even though 
> widely used, pytest and pytest-django are pretty difficult to get started 
> with for a newbie. I intend to fix this by adding a lot of examples to the 
> docs (maybe even make a separate repository for this).
> 
> NOTE: I intend to work on this project for GSoC 2021. I am just scouting 
> beforehand to see how the community receives it.
> 
> Please let me know what you think of this idea. Any critiques will be welcome.
> 
> Best,
> Diptesh
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/8c5e0baa-5104-4876-a936-8792b3c8d5b8n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/FDB4BE6D-226A-4B23-8A75-27267EC500D3%40tomforb.es.

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Tobias Bengfort]

Hi Diptesh,

I think it is great that you want to work on this. I also think asking 
for feedback now is the right way to go.


That said, I personall have not had the best experience with pytest. I 
have mostly worked on smaller projects though, so my perspective might 
be very limited.


My issue is mostly that pytest uses a lot of magic that is hard to 
debug, e.g. the overwritten assert statement or implicit injection of 
fixtures. Most pro arguments seem to depend on personal taste.


On 15/12/2020 04.37, Diptesh Choudhuri wrote:

 3. Parameterizing the test cases using the default runner requires a
third party library and looks difficult


 (I
will be honest, I haven't used this package myself at all).


`subTest` has been in the standard library since python 3.4.

https://docs.python.org/3/library/unittest.html?highlight=unittest#distinguishing-test-iterations-using-subtests

tobias

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f4b8a65c-5eff-1432-654d-3b2d782cd9bf%40posteo.de.

Re: Replace the default django test runner to a more robust testing framework (like pytest)

[Jacob Rief]

There is already a ticket for this:
https://code.djangoproject.com/ticket/25707

I'm actually very much in favor of switching to pytest. In my newer 
projects I use pytest exclusively.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/987d6da5-d1b0-455e-a33f-8e2865c18d40n%40googlegroups.com.

Replace the default django test runner to a more robust testing framework (like pytest)

[Diptesh Choudhuri]

Though the django default runner gets the job done, I think its high time 
to leverage the power of dedicated frameworks for testing django apps. I 
will take up the case of *pytest* here because other frameworks like *nose *
are not widely used.

For everyone who has used pytest for testing in django, I think we will all 
agree that pytest is much better suited to this task. 

   1. The passing tests look better.
   2. Failing tests are much more verbose with where the error actually 
   occurred.
   3. Parameterizing the test cases using the default runner requires a 
   third party library and looks difficult 
   

 (I 
   will be honest, I haven't used this package myself at all). 
   4. Injecting runtime conditions into the tests is quite difficult, if 
   not impossible, and requires a lot of hacky workarounds.
   5. I am not sure about this, but the default runner also doesn't cache 
   results from previously run tests, making retesting unchanged tests much 
   slower.
   6. This might sound non-trivial but typing *pytest *is much more easier 
   than typing *python manage.py test *into the command line.
   7. IDEs like VSCode can discover tests written for *pytest *but not the 
   ones written using the default runner. This might not sound a big deal, but 
   it is infinitely more easier to run tests using the GUI that these IDEs 
   have. They provide real-time feedback and show exactly which tests have 
   passed and what haven't in the code source itself. Here 
    is a picture showing what I mean.
   8. The directory structure created by *python manage.py startapp my_app 
*creates 
   a *tests.py *file. This goes against testing best practices which say 
   that all tests should be aggregated at one place (as opposed to inside 
   individual apps).
   9. A single *tests.py *file encourages new, inexperienced developers to 
   write all their tests in a single file. This, yet again, goes against 
   testing best practices which put a lot of emphasis on modularization of 
   tests. Models should be tested in *test_models.py*, views in *test_views.py 
   *etc. 
   10. Though *unittest*'s *setUp *and *tearDown *work fine, I personally 
   find that *pytest*'s fixture system provides a bit more freedom. I might 
   want to put all my test methods in one *class*, but I might not want to 
   run the *setUp *and *tearDown *for every test method. (Let me know if 
   this point is a bit unclear.)

*pytest *and *pytest-django *are quire mature frameworks and don't need a 
lot of changes. We could make it easier to setup tests right out of the 
box. For example, *django-admin startproject my_project *could create a 
*pytest.ini 
*right out of the box in the project root. 

Most of my focus would be on rewriting and adding documentation. Even 
though widely used, *pytest *and *pytest-django *are pretty difficult to 
get started with for a newbie. I intend to fix this by adding a lot of 
examples to the docs (maybe even make a separate repository for this).

*NOTE: I intend to work on this project for GSoC 2021. I am just scouting 
beforehand to see how the community receives it.*

Please let me know what you think of this idea. Any critiques will be 
welcome.

Best,
Diptesh

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8c5e0baa-5104-4876-a936-8792b3c8d5b8n%40googlegroups.com.

Re: django test-runner annoyances

[mvr]

On Sep 14, 4:16 pm, Carl Meyer  wrote:
>
> I'm generally in favor of updating Django's test runner to be more
> consistent with what the rest of the Python testing world does. Being
> able to reference test files, suites, and tests by
> fully-qualified-dotted-path rather than magical-applabel-path would be a
> good start, IMO.

I'll cut a ticket and submit a patch.

> Another piece would be adding support for unittest2 test discovery, to
> lift the requirement that all tests have to live directly in tests.py or
> models.py. It's not that I think putting tests inside a tests package is
> a bad convention, but if you have a lot of tests it's unfortunate that
> you currently have to manually import all the suites into tests/__init__.py.

We kind of have a fix for this too, and although it is not in the test-
runner itself or the management command it easily could be.

It involves adding one line to tests/__init__.py and never touching it
again.. that line looks like this:

__all__ = import_test_classes(__file__, __name__)

The implementation, which leverages unittest2 test loading magic, is
detailed here: http://dpaste.com/615679/

(Not sure whether dpaste or code in messages is worse form, but the
formatting on my first email looked bad..)

Regarding Swicegood's remarks about app-refactor, Yes, the first part
of what motivates full dotpaths for us is duplicate app names.  Second
is stuff like being able to run tests locally that the build process
won't discover (ie things defined in tests/some_file.py but not
imported in tests/__init__.py).  Third is running a very specific test
to save time iterating during TDD.

I also have a stand-alone workaround (as in, works with django 1.3 out
of the box) so that "dad test " still loads all tests even with
duplicate app names.  I didn't intend to open that can of worms here
since I'm not sure it's of interest with the app refactor up-and-
coming.  If anyone IS interested let me know.. it does require
modifying the test runner itself and not just patching some little
helper.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: django test-runner annoyances

[Filip Dupanović]

Like Carl and Travis said, Django's test runner supports the notion that a 
Django app has all it's test cases collected in the `models` and `tests` 
module.

Most of the time, If your app's `tests` module gets too crowded, fracturing 
the app into smaller apps works. However, if you have to go for the test 
package approach to maintain sanity, your missing the necessary tools to 
manage a growing number of test cases; for instance, if you organize your 
test cases into several modules, there's really no way you can run test 
cases from a specific module only, without providing a custom test runner or 
somehow passing arguments to a `suite()` implementation.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/76JePQ022ZEJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: django test-runner annoyances

[Travis Swicegood]

Hello;

I second all of what Carl said and would like to point out the app-refactor.
 I believe the most current code still lives in the app-loading branch on
jezdez's repository on GitHub[1].  The reason I point this out is because
the current testing structure is a legacy of the way Django internally finds
out what "apps" are tested.  I've gone down this rabbit hole before and the
solution I ended up with was modifying the way loading apps worked which has
all manner of side-effects.  One of the first steps down that path is the
app-loading branch which makes referring to an app by it's full name
possible.

-T

[1]: https://github.com/jezdez/django/tree/app-loading

On Wed, Sep 14, 2011 at 3:16 PM, Carl Meyer  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 09/13/2011 08:46 AM, mvr wrote:
> > Why doesn't the django test management command / test builder allow
> > fully-qualified package names instead of just app-relative ones?
> >
> > At work we've been using the method below to monkey-patch the test
> > builder, so that
> >
> >  $ django-admin.py test my_module.my_app.tests.some_test_file
> >
> > always works as expected.  We'd like to get rid of this monkey-patch,
> > and since this functionality can be added in such a way that it's
> > completely backwards compatible, where is the harm?  I'm also willing
> > to submit a diff that modifies django in-place, but the monkey patch
> > below should be easy to read and first I wanted to hear if anyone has
> > any thoughts on why the existing behaviour really is exactly what it
> > should be.
>
> I'm generally in favor of updating Django's test runner to be more
> consistent with what the rest of the Python testing world does. Being
> able to reference test files, suites, and tests by
> fully-qualified-dotted-path rather than magical-applabel-path would be a
> good start, IMO.
>
> Another piece would be adding support for unittest2 test discovery, to
> lift the requirement that all tests have to live directly in tests.py or
> models.py. It's not that I think putting tests inside a tests package is
> a bad convention, but if you have a lot of tests it's unfortunate that
> you currently have to manually import all the suites into
> tests/__init__.py.
>
> But I'm getting off-track -- these should be separate tickets anyway. If
> you'd like to file the first one and upload a backwards-compatible
> patch, I'm +1 on the concept.
>
> Carl
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk5xC54ACgkQ8W4rlRKtE2eNFgCg7gVEkO6Y+tmXcsWlidmh67ge
> SQwAn0PqFg74dy1yLsSPDYab1Jj+jNZ+
> =bAbE
> -END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: django test-runner annoyances

[Carl Meyer]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 09/13/2011 08:46 AM, mvr wrote:
> Why doesn't the django test management command / test builder allow
> fully-qualified package names instead of just app-relative ones?
> 
> At work we've been using the method below to monkey-patch the test
> builder, so that
> 
>  $ django-admin.py test my_module.my_app.tests.some_test_file
> 
> always works as expected.  We'd like to get rid of this monkey-patch,
> and since this functionality can be added in such a way that it's
> completely backwards compatible, where is the harm?  I'm also willing
> to submit a diff that modifies django in-place, but the monkey patch
> below should be easy to read and first I wanted to hear if anyone has
> any thoughts on why the existing behaviour really is exactly what it
> should be.

I'm generally in favor of updating Django's test runner to be more
consistent with what the rest of the Python testing world does. Being
able to reference test files, suites, and tests by
fully-qualified-dotted-path rather than magical-applabel-path would be a
good start, IMO.

Another piece would be adding support for unittest2 test discovery, to
lift the requirement that all tests have to live directly in tests.py or
models.py. It's not that I think putting tests inside a tests package is
a bad convention, but if you have a lot of tests it's unfortunate that
you currently have to manually import all the suites into tests/__init__.py.

But I'm getting off-track -- these should be separate tickets anyway. If
you'd like to file the first one and upload a backwards-compatible
patch, I'm +1 on the concept.

Carl
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5xC54ACgkQ8W4rlRKtE2eNFgCg7gVEkO6Y+tmXcsWlidmh67ge
SQwAn0PqFg74dy1yLsSPDYab1Jj+jNZ+
=bAbE
-END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

django test-runner annoyances

[mvr]

Why doesn't the django test management command / test builder allow
fully-qualified package names instead of just app-relative ones?

At work we've been using the method below to monkey-patch the test
builder, so that

 $ django-admin.py test my_module.my_app.tests.some_test_file

always works as expected.  We'd like to get rid of this monkey-patch,
and since this functionality can be added in such a way that it's
completely backwards compatible, where is the harm?  I'm also willing
to submit a diff that modifies django in-place, but the monkey patch
below should be easy to read and first I wanted to hear if anyone has
any thoughts on why the existing behaviour really is exactly what it
should be.



def _support_dotpaths():
""" allows real dotpaths to be used as test labels for
django's test runner
first we maintain the status quo by doing whatever django
does, then
just do it the way that they should have done it in the
first place.
"""
import unittest
from django.db.models.loading import ImproperlyConfigured
from django.test import simple
django_build_test_test = simple.build_test
def my_build_test(label):
fallback = lambda:
unittest.TestLoader().loadTestsFromName(label)
try:
return django_build_test_test(label)
except ValueError, v:
if 'should be of the form' in str(v):
return fallback()
else:
raise
except ImproperlyConfigured,e:
if 'App with label' in str(e):
return fallback()
else:
raise
simple.build_test = my_build_test
return my_build_test

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django test runner

[Ramiro Morales]

On Thu, May 28, 2009 at 12:44 PM, Alex Robbins
 wrote:
>
> It would be really great if the test runner had an excludes option. I
> would like to run tests from all apps, except for a couple I know are
> bad. Should I just make my own custom test runner or is this something
> others would like also?

Please, send posts of this kind to the django-users mailing list.
This list if for the coordination of Django development - related
activities.

See ticket 8363.

Also, I've been told nose has that feature.

Regards,

-- 
Ramiro Morales
http://rmorales.net

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Django test runner

[Alex Robbins]

It would be really great if the test runner had an excludes option. I
would like to run tests from all apps, except for a couple I know are
bad. Should I just make my own custom test runner or is this something
others would like also?

Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---