[issue10967] move regrtest over to using more unittest infrastructure

[STINNER Victor]

STINNER Victor added the comment:

This issue was opened 4 years ago and has no activity since 2 years :-/ Even if 
yes, it would be nice to enhance regrtest and unittest, this issue doesn't 
propose any concrete plan with a concrete patch. I understand that the current 
situation is not perfect but it's acceptable :-) I close the issue as out of 
date.

If someone is still motivated, please open a new issue with a concrete plan and 
maybe summarize this one.

In the meanwhile, the work on relying on unittest test discovery is still 
ongoing and made good progress last... years (see issue #16748).

--
dependencies:  -Make CPython test package discoverable
nosy: +haypo
resolution:  -> out of date
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Berker Peksag]

Changes by Berker Peksag :


--
nosy: +berker.peksag

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Zachary Ware]

Changes by Zachary Ware :


--
nosy: +zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[R. David Murray]

R. David Murray added the comment:

Ah, but for me the point of running directly under unit test (and unittest 
discovery) it to be able to use the standard unittest facilities for running a 
single testcase or individual test.  Converting them to skips without making 
them run if regrtest isn't active might make things worse for that use case.  
If we are going to convert them to skips, why not do it right to start with?  
It won't any harder, since the logic will be baked into the skip decorators.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Ezio Melotti]

Ezio Melotti added the comment:

Yes, that could be done as well, but even if they are not run by default under 
unittest, I'd be already happy to see them marked as skipped.  If I see that 
they are skipped I can always switch to regrtest to enable the necessary 
resource, or just execute the file directly if the resource is enabled just 
before unittest.main() (like the example in msg183314).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Ezio Melotti]

Ezio Melotti added the comment:

> So you are agreeing with my proposal?

I will have to see a concrete proposal first.  If a resource API is useful 
enough to be added to unittest, and we can come up with a decent API we can 
move it there, otherwise it can just stay in regrtest.  There's also the issue 
that the unittest API is already quite big, and adding even more things there 
might do more harm than good.

There are several other parts of regrtest that can take advantage of already 
existing unittest features first, and convert them might be enough to close 
this issue even if the existing resource infrastructure is kept in regrtest.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[R. David Murray]

R. David Murray added the comment:

My proposal is how to do this without adding anything to unittest.  It is about 
how to implement the skips so that all tests are run when run under unittest, 
but resource control still happens when the tests run under regrtest.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[R. David Murray]

R. David Murray added the comment:

So you are agreeing with my proposal?

And no, I don't think the goal is to get rid of regrtest, it is just to make it 
as small as practical.  Or to put it another way, to add the features of 
regrtest that are generally useful to unittest.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Ezio Melotti]

Ezio Melotti added the comment:

> I'd rather not have resources outside the local machine be accessed.

Fair enough.

> If you *also* want a way to skip tests based on resource when running 
> them outside regrtest, then unittest would have to grow a resource API.

This can already be done though -- adding a resource API to unittest would just 
provide an additional way to do it without going through regrtest.

I don't think we will ever get rid of regrtest.  It would be nice if most of 
the code could be removed and standard unittest features were used instead, but 
regrtest will always have more advanced/customized feature built on top of 
unittest (which is not to say that a resource API for unittest is useless, but 
that's not a blocker for this issue :).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[R. David Murray]

R. David Murray added the comment:

No controversy on (1), we should just do that.

I don't see a strong reason to change (2).  I always run the test suite with 
-uall, but if I were running a restricted set I'd rather not have resources 
outside the local machine be accessed.

My proposal is a way to provide (3): running the regrtest-resource-skipped 
tests when not running them through regrtest.  If you *also* want a way to skip 
tests based on resource when running them outside regrtest, then unittest would 
have to grow a resource API.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Ezio Melotti]

Ezio Melotti added the comment:

There are actually three separate issues here:
1) The resources don't use unittest skip (or at least not always), so while 
running the tests without regrtests there are no indication that some tests 
have been skipped.  This could be addressed by converting the current resources 
to proper skips (with something like @skip_unless_resource('network')) and 
shouldn't be too difficult.

2) The "network" resource could be enabled by default, given that network 
connectivity is nowadays fairly common (it would be even better if that was 
detected automatically at runtime).  This would enable more tests and shouldn't 
have negative side-effects (unless the network is not available and the tests 
have no way to realize that and fail).

3) unittest could have a "resource API", but that's not strictly necessary if 
we still provide a way to run the skipped tests through regrtest and/or while 
running the test file individually.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[R. David Murray]

R. David Murray added the comment:

We have previously discussed adding resource-awareness to unittest, but that is 
a much bigger project (API design, scope considerations, etc etc).  In the 
meantime, what we could do is modify the current resource-skip logic slightly: 
make it so that tests are *not* skipped for resource reasons *unless* some flag 
in support is set indicating that resource enforcement is in effect *and* the 
specified resource is not in the list of currently-asserted resources.  That 
way tests are only skipped for resource reasons if run via regrtest.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Ezio Melotti]

Ezio Melotti added the comment:

> For one thing, I think resources could be implemented in terms of skips

While working on #17333 I noticed a related problem.  Most of the tests in 
test_imaplib are skipped if the 'network' resource is not specified.  The test 
contains the following lines to enable those tests when the file is executed 
directly:

if __name__ == "__main__":
support.use_resources = ['network']
unittest.main()

but this clearly doesn't work while using unittest discovery, so those tests 
are skipped.

Implementing resources as skips makes sense, but we might also need a way to 
specify what resources are needed that works with unittest discovery.  Running 
these tests only when the test file is executed directly or through `regrtest 
-u` is also an option.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Chris Jerdonek]

Chris Jerdonek added the comment:

> Extending regrtest to support unittest test discovery directly is also a 
> worthwhile specific proposal.

Updating the tests to support discovery in all cases is discussed in (meta) 
issue 16748.  There are also many individual issues in the tracker (one per 
test module).

--
dependencies: +Make CPython test package discoverable

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Chris Jerdonek]

Chris Jerdonek added the comment:

One important piece is that regrtest currently has no tests (e.g. there is no 
test_regrtest.py), so changing it must be done more carefully.  How do people 
feel about new (or newly modified) regrtest classes and functions going into a 
different fully-tested module with its own test_* file.  Currently, regrtest is 
a mixture of definitions and "running code," so I would be leery of adding 
test_regrtest.

This could also be done by making regrtest a package, so that regrtest-related 
modules would be in a single directory.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Chris Jerdonek]

Changes by Chris Jerdonek :


--
nosy: +cjerdonek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Éric Araujo]

Éric Araujo  added the comment:

For one thing, I think resources could be implemented in terms of skips, or 
even included into stock unittest.

--
nosy: +eric.araujo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[R. David Murray]

R. David Murray  added the comment:

I also would put increasing test coverage at a higher priority, but this sort 
of refactoring can be a good step in the development path of new contributors, 
and doing it does decrease the future maintenance burden.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Wouldn't the time be better spent factoring the test suite or improving 
coverage?  I'm not sure how simply rearranging the tests makes us better-off.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Ezio Melotti]

Changes by Ezio Melotti :


--
keywords: +gsoc
nosy: +ezio.melotti

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Brett Cannon]

Brett Cannon  added the comment:

Yes, I somewhat view this as an index issue. I don't expect a wholesale move 
but a more step-by-step move.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[R. David Murray]

Changes by R. David Murray :


--
Removed message: http://bugs.python.org/msg126780

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[R. David Murray]

R. David Murray  added the comment:

I think we have already been moving in this direction for quite some time.  
Past policy is to only change things when we are working on that area of code 
anyway.  If someone wants to make some specific proposals to simplify regrtest 
by doing a wholesale move of test code away from a regrtest supported 
infrastructure to a unittest supported infrastructure, I think that's worth 
considering, but I think it should be done bit by bit with specific proposals 
to replace specific regrtest features.

Extending regrtest to support unittest test discovery directly is also a 
worthwhile specific proposal.

Do you intend this issue to be an "index issue" that links to issues with 
specific proposals?

(Note: see also the closely related issue 8273).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[R. David Murray]

R. David Murray  added the comment:

I think we have already been moving in this director for quite some time.  Past 
policy is to only change things when we are working on that area of code 
anyway.  If someone wants to make some specific proposals to simplify regrtest 
by doing a wholesale move of test code away from a regrtest supported 
infrastructure to a unittest supported infrastructure, I think that's worth 
considering, but I think it should be done bit by bit with specific proposals 
to replace specific regrtest features.

Extending regrtest to support unittest test discovery directly is also a 
worthwhile specific proposal.

Do you intend this issue to be an "index issue" that links to issues with 
specific proposals?

(Note: see also the closely related issue 8273).

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Sandro Tosi]

Changes by Sandro Tosi :


--
nosy: +sandro.tosi

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10967] move regrtest over to using more unittest infrastructure

[Brett Cannon]

New submission from Brett Cannon :

test.regrtest is rather old and has not been updated to take advantage of all 
the latest features in unittest (e.g., test discovery). It might be a rather 
large undertaking with various bits requiring some changes (e.g., getting away 
from raising exceptions for skipped tests and instead using unittest.skipIf), 
but for maintainability it might be good to try to use as much unittest code in 
regrtest as possible.

--
components: Tests
messages: 126665
nosy: brett.cannon
priority: low
severity: normal
stage: unit test needed
status: open
title: move regrtest over to using more unittest infrastructure
type: feature request
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com