[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Change by Inada Naoki :


--
resolution:  -> not a bug
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Géry]

Change by Géry :


--
nosy: +maggyero

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Inada Naoki  added the comment:


New changeset 0b2b9d251374c5ed94265e28039f82b37d039e3e by Inada Naoki in branch 
'main':
bpo-23882: unittest: Drop PEP 420 support from discovery. (GH-29745)
https://github.com/python/cpython/commit/0b2b9d251374c5ed94265e28039f82b37d039e3e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Change by Inada Naoki :


--
versions: +Python 3.11 -Python 3.10, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Change by Inada Naoki :


--
pull_requests: +27982
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/29745

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Adam Johnson]

Adam Johnson  added the comment:

I just reported https://bugs.python.org/issue45864 , and closed as duplicate of 
this.

--
nosy: +adamchainz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Eric V. Smith]

Eric V. Smith  added the comment:

As the author of PEP 420, I think having test discovery support namespace 
packages is a mistake, at least in the sense of pretending every directory is a 
namespace package.

--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Inada Naoki  added the comment:

I noticed that namespace package support has been broken since this commit.
https://github.com/python/cpython/commit/bbbcf8693b876daae4469765aa62f8924f39a7d2

Now namespace pacakge has __file__ attribute which is None. But...

try:
start_dir = os.path.abspath(
   os.path.dirname((the_module.__file__)))
except AttributeError:
# look for namespace packages

`the_module.__file__` doesn't raise AttributeError for now. But 
os.path.dirname(None) raise TypeError.

The commit is backported to 3.7 branch. So namespace package support has been 
broken since Python 3.7.

Shouldn't we drop namespace package support?
It is misleading. And we could not maintain it. We didn't notice that it is 
broken for 3 years!

--
nosy: +barry
resolution: fixed -> 
stage: resolved -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Md Jony Hossain Habib]

Md Jony Hossain Habib  added the comment:

diff -r 293d9964cf6e Lib/unittest/loader.py
--- a/Lib/unittest/loader.pyTue Apr 28 00:04:53 2015 -0400
+++ b/Lib/unittest/loader.pyTue Apr 28 10:12:07 2015 +0300
@@ -338,7 +338,7 @@
 raise ImportError('Start directory is not importable: %r' % 
start_dir)
 
 if not is_namespace:
-tests = list(self._find_tests(start_dir, pattern))
+tests = list(self._find_tests(start_dir, pattern, namespace=True))
 return self.suiteClass(tests)
 
 def _get_directory_containing_module(self, module_name):
@@ -403,7 +403,7 @@
 name = self._get_name_from_path(full_path)
 self._loading_packages.add(name)
 try:
-yield from self._find_tests(full_path, pattern, namespace)
+yield from self._find_tests(full_path, pattern, False)
 finally:
 self._loading_packages.discard(name)
 
diff -r 293d9964cf6e Lib/unittest/test/test_program.py
--- a/Lib/unittest/test/test_program.py Tue Apr 28 00:04:53 2015 -0400
+++ b/Lib/unittest/test/test_program.py Tue Apr 28 10:12:07 2015 +0300
@@ -16,7 +16,7 @@
 expectedPath = os.path.abspath(os.path.dirname(unittest.test.__file__))
 
 self.wasRun = False
-def _find_tests(start_dir, pattern):
+def _find_tests(start_dir, pattern, namespace):
 self.wasRun = True
 self.assertEqual(start_dir, expectedPath)
 return tests

--
nosy: +mdjonyhossainhabib

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Terry J. Reedy]

Terry J. Reedy  added the comment:

>From the merge:

+++ b/Doc/library/unittest.rst
@@ -330,7 +330,9 @@ Test modules and packages can customize test loading and 
discovery by through
 the `load_tests protocol`_.
 
 .. versionchanged:: 3.4
-   Test discovery supports :term:`namespace packages `.
+   Test discovery supports :term:`namespace packages `
+   for start directory. Note that you need to the top level directory too.
+   (e.g. ``python -m unittest discover -s root/namespace -t root``).
 
The last sentence is missing a verb after 'you need to' saying what to do about 
"the top level directory.  "be in"?  "go to"?  "later destroy"? (just kidding 
;-)

--
nosy: +terry.reedy
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Change by Inada Naoki :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[miss-islington]

miss-islington  added the comment:


New changeset 30fe3ee6d39fba8183db779f15936fe64cc5ec85 by Miss Islington (bot) 
in branch '3.9':
bpo-23882: Doc: Clarify unittest discovery document (GH-21560)
https://github.com/python/cpython/commit/30fe3ee6d39fba8183db779f15936fe64cc5ec85


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[miss-islington]

miss-islington  added the comment:


New changeset 9dd018e35cce30bc2545290b6083dbf6e50d7b61 by Miss Islington (bot) 
in branch '3.8':
bpo-23882: Doc: Clarify unittest discovery document (GH-21560)
https://github.com/python/cpython/commit/9dd018e35cce30bc2545290b6083dbf6e50d7b61


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[miss-islington]

Change by miss-islington :


--
pull_requests: +23405
pull_request: https://github.com/python/cpython/pull/24621

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington, miss-islington, miss-islington, miss-islington
nosy_count: 13.0 -> 14.0
pull_requests: +23400, 23401, 23403, 23404
pull_request: https://github.com/python/cpython/pull/24619

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington, miss-islington, miss-islington
nosy_count: 13.0 -> 14.0
pull_requests: +23400, 23401, 23403
pull_request: https://github.com/python/cpython/pull/24619

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington, miss-islington, miss-islington, miss-islington, 
miss-islington
nosy_count: 13.0 -> 14.0
pull_requests: +23400, 23401, 23402, 23403, 23404
pull_request: https://github.com/python/cpython/pull/24619

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 13.0 -> 14.0
pull_requests: +23400
pull_request: https://github.com/python/cpython/pull/24619

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington, miss-islington
nosy_count: 13.0 -> 14.0
pull_requests: +23400, 23401
pull_request: https://github.com/python/cpython/pull/24619

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Inada Naoki  added the comment:


New changeset 5a4aa4c03e27ca5007b86c9c1ee62c77ad08a120 by Inada Naoki in branch 
'master':
bpo-23882: Doc: Clarify unittest discovery document (GH-21560)
https://github.com/python/cpython/commit/5a4aa4c03e27ca5007b86c9c1ee62c77ad08a120


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Change by Inada Naoki :


--
pull_requests: +20706
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/21560

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Change by Inada Naoki :


--
assignee:  -> docs@python
components: +Documentation
nosy: +docs@python
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Inada Naoki  added the comment:

In such case, both directories must be specified explicitly.

Test discovery doesn't know that a directory is namespace package or
namespace package. So it shouldn't assume it is namespace package unless
specified.

Note that PEP 420 searches namespace package only when it is specified.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Roger Gammans]

Roger Gammans  added the comment:

But namespace packages are still useful for what PEP420 envisages and they 
should be able to have runnable tests.

For instance

 projectX/
   - interfaces/
 - proxyX.py
 - testX.py
 projectY/
   - intefaces/
 - proxyY.py
 - testY.py

Here interfaces is a namespace package and projectX and projectY are kept 
separate, perhaps to reduce dependency and/or for separation of concerns.

I don't think this is insurmountable - even taking into account Inada's very 
good point about dangerous_scripts. A full tests/ packages in on both projectX 
and projectY would allow their test to run. However, in some environments, it 
is normal to put the test alongside the code as shown here.

But some better documentation on this issue would advisable, and some guidance 
on the best practice.

Python has a complex ecosystem, and so an individual developer might hit this 
problem but be using a third party `framework`, which hasn't taken all of this 
on board. Eg. https://code.djangoproject.com/ticket/30403 . (I'm not really 
singling django out, although it's the one I hit) So it important that there is 
a clear understanding of the best way to deal with this case.

--
nosy: +rgamm...@gammascience.co.uk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Inada Naoki  added the comment:

Searching into directory without __init__.py recursively is not only 
inefficient, but also dangerous.

 project/
   - mylib/
  - __init__.py
  - foo.py
   - tests/
  - __init__.py
  - test_foo.py
   - tools/
  - bin/
  - dangerous_script.py

What happens if `python -m unittest` is run in the project root?
Who excepts tools/bin/dangarous.py is executed?

My conclution is:

* People shouldn't abuse PEP 420.  Omitting __init__.py is allowed only for 
namespace package.
* Namespace package should be searched based on PEP 420 rule. Don't search into 
regular directory unless it is specified.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Inada Naoki  added the comment:

I think people misunderstanding and misusing PEP 420, withouth knowing what is 
namespace package for.

I had wrote an article to describe namespace package is not a regular package.
https://dev.to/methane/don-t-omit-init-py-3hga

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Inada Naoki]

Inada Naoki  added the comment:

I had rejected this idea in #29642. This is a copy of my comments in the issue.

---

I'm afraid this change makes testloader searches unrelated directory contains 
massive files (like node_modules).

I don't think loading all tests from whole namespace package is not usual use 
case.

When using import, (namespace) package name is explicitly specified.
Only specified name is searched.

In test loader's case, there are no such limit.
Loader may search millions of completely unrelated directories.
It may include directories in NFS or samba over slow network.

--
nosy: +inada.naoki
versions: +Python 3.10 -Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Zbynek Winkler]

Change by Zbynek Winkler :


--
nosy: +Zbynek.Winkler

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Phil Connell]

Change by Phil Connell :


--
nosy: +pconnell

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Roger Gammans]

Change by Roger Gammans :


--
nosy: +rgammans

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Robert Collins]

Robert Collins added the comment:

(for the trivial case of CLI discover without a parameter - so translate that 
to the lower level API and then test that)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23882
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Robert Collins]

Robert Collins added the comment:

reviewed in rietvald, but here too just in case.

The hunk that saves/restores _top_level_dir feels wrong to me - and not part of 
this bug, please remove it.

The rest of the patch is fine today.

But it also needs to add two specifically namespace tests: concretely we need 
the following cases covered:


a) namespace subdir/namespace subdir/test_foo.py gets loaded
b) namespace subdir/test_foo.py with another instance of the namespace subdir 
on sys.path also containing e.g. test_bar.py, test_bar.py is not loaded.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23882
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Ned Deily]

Changes by Ned Deily n...@acm.org:


--
stage: needs patch - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23882
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Alex Shkop]

Alex Shkop added the comment:

Please, review the patch.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23882
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Alex Shkop]

Alex Shkop added the comment:

Yes. That is how issue23882_find_all.patch works. I just removed hte condition

if (not namespace and
not os.path.isfile(os.path.join(full_path, '__init__.py'))):
return None, False
 
This makes namespace parameter redundant. Can I remove it?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23882
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Robert Collins]

Robert Collins added the comment:

Ah the user model? 

I think the following:

If I run 'python -m unittest' in a directory, then I expect to run all of the 
tests contained within that directory tree, and no others.

Does that definition help?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23882
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Alex Shkop]

Changes by Alex Shkop a.v.sh...@gmail.com:


Added file: http://bugs.python.org/file39223/issue23882_find_one_level.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23882
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Alex Shkop]

Alex Shkop added the comment:

I'm still not sure which solution is the best. So I attach two simple patches.

First one enables full recursive scan of start_dir for namespace packages.

Second one loads tests only from top level namespace packages.

--
keywords: +patch
Added file: http://bugs.python.org/file39222/issue23882_find_all.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23882
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Robert Collins]

Robert Collins added the comment:

Ok, so here's whats happening:
the default behaviour is to do discovery of '.', which bypasses the namespace 
support code.

Running with tests as the first parameter works because it doesn't require the 
directory being directly scanned to be a package.

Equally, if the namespace isn't mapped to a local directory, 'python -m 
unittest tests' will load tests from the tests namespace.

So - this seems to be acting as designed. Namespace packages working fine.

However, I think what you're asking for is that namespace packages in your cwd 
are picked up and tested, without you having to supply the namespace name?

I think what we'd need for that to work sanely is a patch to teach 
loader.discover to determine the local namespaces which are present under 
start_dir. Right now this code:
elif os.path.isdir(full_path):
if (not namespace and
not os.path.isfile(os.path.join(full_path, '__init__.py'))):
return None, False

gets in your way: we assume that either we're processing a namespace, or there 
are no namespaces at all.

I think some more detangling of the generator could address this fairly cleanly.

--
stage:  - needs patch
title: unittest discovery and namespaced packages - unittest discovery doesn't 
detect namespace packages when given no parameters
versions: +Python 3.5, Python 3.6 -Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23882
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23882] unittest discovery doesn't detect namespace packages when given no parameters

[Alex Shkop]

Alex Shkop added the comment:

Thanks. I understand the code pretty well and I saw issue17457 that fixes 
discovery for explicitly specified namespace package.

What I need is to know how discovery has to work. Do we need to discover 
namespace packages inside discovery path? And should we do that recursively? 
I.e. should we pick namespace packages inside namespace packages? This will 
lead to recursive scan of all directories under discovery path.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23882
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com