[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Filipe Laíns]

Filipe Laíns  added the comment:

I opened up a new PR that should fix this properly. The root issue was that 
PathFinder was not setting the loader attribute for namespace packages in the 
spec, which it should. After fixing that, _find_module just needed to be 
updated to deal with NamespaceLoader.

```
$ tree namespace
namespace/
└── a.py

0 directories, 1 file
$ cat test-bpo-40350.py
import namespace.a
```

```
$ ./python
Python 3.11.0a1+ (heads/main:9e05da6224, Oct 23 2021, 20:36:14) [GCC 11.1.0] on 
linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import modulefinder
>>> modulefinder.ModuleFinder('test-bpo-40350.py')

>>> f = modulefinder.ModuleFinder('test-bpo-40350.py')
>>> f.
KeyboardInterrupt
>>> m = modulefinder.ModuleFinder()
>>> f = modulefinder.ModuleFinder()
>>> f.run_script('test-bpo-40350.py')
>>> f.modules.items()
dict_items([('__main__', Module('__main__', 'test-bpo-40350.py')), 
('namespace', Module('namespace', 
_NamespacePath(['/home/anubis/git/cpython/namespace'])))])
```

Previously:
```
$ python
Python 3.9.7 (default, Oct 10 2021, 15:13:22)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import modulefinder
>>> f = modulefinder.ModuleFinder()
>>> f.run_script('test-bpo-40350.py')
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.9/modulefinder.py", line 161, in run_script
self.load_module('__main__', fp, pathname, stuff)
  File "/usr/lib/python3.9/modulefinder.py", line 357, in load_module
self.scan_code(co, m)
  File "/usr/lib/python3.9/modulefinder.py", line 430, in scan_code
self._safe_import_hook(name, m, fromlist, level=0)
  File "/usr/lib/python3.9/modulefinder.py", line 375, in _safe_import_hook
self.import_hook(name, caller, level=level)
  File "/usr/lib/python3.9/modulefinder.py", line 173, in import_hook
q, tail = self.find_head_package(parent, name)
  File "/usr/lib/python3.9/modulefinder.py", line 229, in find_head_package
q = self.import_module(head, qname, parent)
  File "/usr/lib/python3.9/modulefinder.py", line 316, in import_module
fp, pathname, stuff = self.find_module(partname,
  File "/usr/lib/python3.9/modulefinder.py", line 508, in find_module
return _find_module(name, path)
  File "/usr/lib/python3.9/modulefinder.py", line 77, in _find_module
if spec.loader.is_package(name):
AttributeError: 'NoneType' object has no attribute 'is_package'
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Filipe Laíns]

Change by Filipe Laíns :


--
pull_requests: +27465
pull_request: https://github.com/python/cpython/pull/29196

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Filipe Laíns]

Change by Filipe Laíns :


--
nosy: +FFY00

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Denis Kasak]

Denis Kasak  added the comment:

Anything still left to do that is stalling this? I just got bitten by it when 
trying to use modulefinder.

--
nosy: +dkasak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Cajetan Rodrigues]

Cajetan Rodrigues  added the comment:

Turns out using _PKG_DIRECTORY as a type for namespace packages ended up making 
`_find_module` try to search for an __init__.py within them, since it had no 
understanding of the diff. between a namespace package and a regular one (the 
lack of __init__.py), and caused it to break.

I've raised a PR with a new type _NSP_DIRECTORY for namespace-directories.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Cajetan Rodrigues]

Change by Cajetan Rodrigues :


--
keywords: +patch
pull_requests: +19231
stage: test needed -> patch review
pull_request: https://github.com/python/cpython/pull/19917

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Cajetan Rodrigues]

Cajetan Rodrigues  added the comment:

Reproduced on Python3.9.0a5+

imp.find_module() simply raised an ImportError in my tests with an implicitly 
namespaced package (without an __init__.py)

About the "type_", I think it should be consistent with _PKG_DIRECTORY, since 
PEP 420 states the following[1]:

```
A namespace package is not fundamentally different from a regular package. It 
is just a different way of creating packages. Once a namespace package is 
created, there is no functional difference between it and a regular package.
```

I'd be happy to submit a patch if you think this is alright.


[1] https://www.python.org/dev/peps/pep-0420/#id24

--
nosy: +cajetan.rodrigues

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Eric Snow]

Eric Snow  added the comment:

Ah, namespace packages. :)  Yeah, the code is not taking the "spec.loader is 
None" case into account.  I expect the fix would be to add handling of that 
case a few lines up in the code, right after handling BuiltinImporter and 
FrozenImporter.  Offhand I'm not sure if the "type" should be _PKG_DIRECTORY or 
some new one just for namespace packages.  How does imp.find_module() (on which 
modulefinder._find_module() is based) respond to namespace packages?

[1] 
https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec.loader

--
nosy: +barry, brett.cannon, eric.smith, eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Eric Snow]

Change by Eric Snow :


--
stage:  -> test needed
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Greg Whiteley]

New submission from Greg Whiteley :

Issue:

Running ModuleFinder.run_script() on numpy versions 1.16.1 to 1.18.3 (maybe 
more) fails with backtrace.  See steps to reproduce below.

I do not see this problem on earlier versions of python than 3.8 (tested 3.4, 
3.5, 3.6 on ubuntu LTSs), but the code has changed around 3.8.

The failure comes to this line of modulefinder.py

https://github.com/python/cpython/blame/master/Lib/modulefinder.py#L79

if spec.loader.is_package(name):
return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY)

I can work around it by changing that to check for None

if spec.loader is not None and spec.loader.is_package(name):
return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY)


Environment:

Ubuntu 20.04 with default python3, python3-pip

Steps to reproduce:

# note any nump version I've tried 1.16.1 and greater fails - I included 1.18.3 
to be precise for reproduciton
$ pip3 install "numpy==1.18.3"
$ cat test.py
import numpy

$ python3
>>> from modulefinder import ModuleFinder
>>> finder = ModuleFinder()
>>> finder.run_script("test.py")
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.8/modulefinder.py", line 165, in run_script
self.load_module('__main__', fp, pathname, stuff)
...

300 lines of stack elided - see attached fulllog.txt

...
  File "/usr/lib/python3.8/modulefinder.py", line 433, in scan_code
self._safe_import_hook(name, m, fromlist, level=0)
  File "/usr/lib/python3.8/modulefinder.py", line 378, in _safe_import_hook
self.import_hook(name, caller, level=level)
  File "/usr/lib/python3.8/modulefinder.py", line 177, in import_hook
q, tail = self.find_head_package(parent, name)
  File "/usr/lib/python3.8/modulefinder.py", line 233, in find_head_package
q = self.import_module(head, qname, parent)
  File "/usr/lib/python3.8/modulefinder.py", line 320, in import_module
fp, pathname, stuff = self.find_module(partname,
  File "/usr/lib/python3.8/modulefinder.py", line 511, in find_module
return _find_module(name, path)
  File "/usr/lib/python3.8/modulefinder.py", line 78, in _find_module
if spec.loader.is_package(name):
AttributeError: 'NoneType' object has no attribute 'is_package'
>>> 


Obviously I can't tell if numpy or modulefinder is the real culprit.

Let me know if I can give any more information.

--
components: Library (Lib)
files: fulllog.txt
messages: 366912
nosy: Greg Whiteley
priority: normal
severity: normal
status: open
title: modulefinder chokes on numpy - dereferencing None in spec.loader
type: behavior
versions: Python 3.8
Added file: https://bugs.python.org/file49079/fulllog.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com