[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:


New changeset 24c2c20873dc800c99d1dabf26419b40cadfe627 by Victor Stinner in 
branch '2.7':
bpo-30540, bpo-30523: Add --matchfile and --list-cases options to regrtest 
(#2249)
https://github.com/python/cpython/commit/24c2c20873dc800c99d1dabf26419b40cadfe627


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2301

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2299

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:


New changeset a0ccc54e6dffacf9e7c06f2a3e9056d2d35d21eb by Victor Stinner in 
branch '3.6':
Synchronize libregrtest from master to 3.6 (#2244)
https://github.com/python/cpython/commit/a0ccc54e6dffacf9e7c06f2a3e9056d2d35d21eb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:

I created bpo-30683: Enhance doctest support in regrtest --list-cases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2288

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:


New changeset b0c58d3c569d1b9060248e665857b5df94817340 by Victor Stinner (Louie 
Lu) in branch 'master':
bpo-30523: Add --list-cases unittest (#2243)
https://github.com/python/cpython/commit/b0c58d3c569d1b9060248e665857b5df94817340


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[Louie Lu]

Changes by Louie Lu :


--
pull_requests: +2287

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:

Ah something else: doctests are not fully supported.

./python -m test test_extcall --list-cases

This command has not output, whereas Lib/test/test_extcall.py contains doctests.

test_builtins works because it uses:

def load_tests(loader, tests, pattern):
from doctest import DocTestSuite
tests.addTest(DocTestSuite(builtins))
return tests

whereas test_extcall uses:

def test_main():
support.run_doctest(sys.modules[__name__], True)

I see two options:

* Replace support.run_doctest() with doctest.DocTestSuite() in all tests
* Enhance --list-cases to discover doctests. support.run_doctest() calls 
doctest.testmod(), but doctest.testmod() has no API to list tests. testmod() 
lists tests and directly runs them. Maybe --list-cases can reuse 
doctest.DocTestSuite(), I don't know.

I chose to merge Louie Lu's PR anyway, since I prefer to move step by step. 
Supporting test_extcall can be done later.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:

Yeah, Louie fixed most issues!

> I concur with Julien, the result should be printed to stdout.

Done by Louie. The code now uses a simple print().

> I tried:
>
> $ ./python -m unittest test.test_builtin --list-tests
> doctest.DocTestCase.runTest
> doctest.DocTestCase.runTest
> ...

Using Louie's implementation, it "just" works!

haypo@selma$ ./python -m test test_builtin --list-cases|grep ^builtins
builtins.bin
builtins.float.as_integer_ratio
...

Even if these test names are not fully qualified, they are accept by 
--match/--matchfile!

./python -m test test_builtin --match=builtins.bin -v
...
0:00:00 load avg: 0.12 [1/1] test_builtin
bin (builtins)
Doctest: builtins.bin ... ok
...

So it will be possible to implement my bisect feature on these doctests as well!

I pushed Louie Lu's implementation: add a new --list-cases option to regrtest. 
Serhiy proposed a different CLI:

* --list=methods
* --list=files
* --list=classes

But I chose to push Louie's patch anyway since its patch is super simple and 
doesn't break anything: you can still use the existing --list-tests to list 
test *files*. I don't need --list=classes yet, it may be added later.

Serhiy: I plan to backport --matchfile and --list-cases features to 2.7, 3.5 
and 3.6 branches, once Louie writes the unit test I requested him. Are you ok 
with that?

I really need to get my bisect tool on all branches, because more and more 
often I have to bisect refernece leaks, tests leaking resources (files, leaking 
memory, etc.), etc. --matchfile and --list-cases are the key features to 
implement such bisection.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:


New changeset a49c935cfd6b4f8ea1f750888dd9260bbf5b9980 by Victor Stinner 
(mlouielu) in branch 'master':
bpo-30523: regrtest: Add --list-cases option (#2238)
https://github.com/python/cpython/commit/a49c935cfd6b4f8ea1f750888dd9260bbf5b9980


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[Louie Lu]

Louie Lu added the comment:

Serhiy and Victor suggest moving this option to regrtest. open a new PR 2238 
addressed this.

Also, because moving into regrtest, some test that used `requires` will need to 
pass option when using it. e.g.:

  $ ./python -m test test.test_curses --list-cases -ucurses

otherwise, it will just passout and do nothing (maybe it should warn user?)

--
nosy: +louielu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[Louie Lu]

Changes by Louie Lu :


--
pull_requests: +2283

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The main problem is doctests. Numerous "doctest.DocTestCase.runTest" look 
useless and confusing.

Running tests instead of listing them is bad.

It is not clear why in one cases --list-tests outputs full qualified test 
names, while in other cases it outputs only class and method names.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I concur with Julien, the result should be printed to stdout.

I tried:

$ ./python -m unittest test.test_builtin --list-tests
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
test.test_builtin.BuiltinTest.test_abs
test.test_builtin.BuiltinTest.test_all
...

$ ./python -m test.test_builtin --list-tests
BuiltinTest.test_abs
BuiltinTest.test_all
BuiltinTest.test_any
...
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest

All these doctest.DocTestCase.runTest look useless.

But "./python -m unittest test.test_json --list-tests" and "./python -m 
test.test_json --list-tests" give the same output.

"./python -m test.test_doctest --list-tests" and "./python -m test.test_tk 
--list-tests" run tests instead of listing them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:

I used ./python -m test Lib/test/test_threading.py --list-tests.

I would like to be able to use this feature in regrtest too:

./python -m test test_threading --list-tests

Sadly, this option already exists and only return test files, not test
methods.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[Julien Palard]

Julien Palard added the comment:

It works in "discover" mode, I tested:

./python -m unittest discover ./Lib/unittest/test/ --list-tests

and it worked.

So maybe the --list-tests should be moved to _getDiscoveryArgParser?

Also spotted that tests are printed on stderr, typically when the users asks 
for something, it has to be printed to stdout. Typically usages are written on 
stderr in case of error, but on stdout when explicitly asked via --help, see:

$ ./python -m unittest discover ./Lib/unittest/test/ --tabayo | wc
usage: python -m unittest discover [-h] [-v] [-q] [--locals] [-f] [-c] [-b]
   [-s START] [-p PATTERN] [-t TOP]
   [--list-tests]
python -m unittest discover: error: unrecognized arguments: --tabayo
  0   0   0
$ ./python -m unittest discover ./Lib/unittest/test/ --help | wc
 23 1291110

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

How to use this feature? "./python -m unittest --list-tests" produces nothing.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:

I used testr which has a --list-tests option:
http://testrepository.readthedocs.io/en/latest/MANUAL.html#listing-tests

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[Julien Palard]

Julien Palard added the comment:

If it can be of any help, pytest have a "pytest --collect-only", it may make 
sense to use the same option name for consistency. I also though about 
"--dry-run" but I still prefer --list-tests or --collect-only.

--
nosy: +mdk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:

I wrote this patch because I'm trying to find which test of test_threading 
leaks a reference:

./python -m test -R 3:3 test_threading

Usually, I modify Lib/test/test_threading.py to remove tests one by one. I 
"bisect" the test manually. But this process is slow and painful, and it seems 
like we have many tests which leak references!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

STINNER Victor added the comment:

Maybe the feature already exists but I failed to find it. Since I'm suprised 
that nobody implements that before (in unittest), I wait for a first feedback 
before starting to document the new feature and to write an unit test for it.

support.run_unittest() doesn't use unittest.main() and so don't support the new 
--list-tests option.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +1961

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30523] unittest: add --list-tests option to only display the list of test names, don't run tests

[STINNER Victor]

New submission from STINNER Victor:

Attached pull requests adds a --list-tests option to only display the list of 
test names, don't run tests.

It's a first step to implement the issue #29512: "regrtest refleak: implement 
bisection feature".

--
components: Tests
messages: 294825
nosy: ezio.melotti, haypo, michael.foord, rbcollins
priority: normal
severity: normal
status: open
title: unittest: add --list-tests option to only display the list of test 
names, don't run tests
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com