[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

_save_and_remove_module in the old code did too much and too little. It tested 
that the fresh module is importable, saved the current state of the fresh 
module and its submodules and removed the fresh module and its submodules. 
Since it tested importability before saving all states, it can have a side 
effect of importing other fresh module, and since it did it before blocking, 
the imported "fresh" module could import the blocked modules.

> 1. Switching the order of how "fresh" and "blocked" are resolved or

It does not help, because the step of resolving "fresh" includes saving the 
sys.modules state, and the step of resolving "blocked" modifies sys.modules. 
Also the step of resolving "fresh" can modify sys.modules before saving its 
sate.

2. Deleting `sys.modules[name]` if it exists immediately before calling 
`importlib.import_module(name)

It would be not enough, because the problem can occur in one of additional 
fresh modules imported by the initial module. We need to remove all fresh 
modules before attempting to import them.

The new import_fresh_module() consists of the following steps:

1. Save the state of all fresh and blocked modules and their submodules.
2. Remove the fresh modules and their submodules and block the blocked modules 
and their submodules.
3. Import all additional fresh modules and return None if any of them is not 
importable.
4. Import the initial module and raise ImportError if it is not importable.
5. Restore states of all fresh and blocked modules and their submodules.

It does not save and restore the state of all modules, because some indirectly 
imported modules can not support repeated importing. It can be reconsidered in 
new versions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset f7e99c98130a705f88348dde60adb98d5bfd8b98 by Serhiy Storchaka in 
branch '3.9':
[3.9] bpo-40173: Fix test.support.import_helper.import_fresh_module() 
(GH-28654) (GH-28658)
https://github.com/python/cpython/commit/f7e99c98130a705f88348dde60adb98d5bfd8b98


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 7873884d4730d7e637a968011b8958bd79fd3398 by Serhiy Storchaka in 
branch '3.10':
[3.10] bpo-40173: Fix test.support.import_helper.import_fresh_module() 
(GH-28654) (GH-28657)
https://github.com/python/cpython/commit/7873884d4730d7e637a968011b8958bd79fd3398


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +27024
pull_request: https://github.com/python/cpython/pull/28658

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +27023
pull_request: https://github.com/python/cpython/pull/28657

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset ec4d917a6a68824f1895f75d113add9410283da7 by Serhiy Storchaka in 
branch 'main':
bpo-40173: Fix test.support.import_helper.import_fresh_module() (GH-28654)
https://github.com/python/cpython/commit/ec4d917a6a68824f1895f75d113add9410283da7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +27020
pull_request: https://github.com/python/cpython/pull/28654

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I just encountered this bug after rewriting datetime tests in issue45229. It is 
also the cause of issue40058.

Not knowing about this issue I have wrote different patch than Hai Shi. It 
makes the code of import_fresh_module() clearer and fixes many other issues.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
versions: +Python 3.10, Python 3.11 -Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[hai shi]

hai shi  added the comment:

> I *think* the problem is that in the step where _save_and_remove_module is 
> called on fresh_name (see here: 
> https://github.com/python/cpython/blob/76db37b1d37a9daadd9e5b320f2d5a53cd1352ec/Lib/test/support/__init__.py#L328-L329)

Looks like deleting a module name after `__import__(name)` is not good enought 
in 
https://github.com/python/cpython/blob/master/Lib/test/support/__init__.py#L244.(some
 redundant submodules should be removed, no?)

paul's example can be passed in PR19390.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[hai shi]

Change by hai shi :


--
keywords: +patch
pull_requests: +18754
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19390

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[hai shi]

Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40173] test.support.import_fresh_module fails to correctly block submodules when fresh is specified

[Paul Ganssle]

New submission from Paul Ganssle :

It seems that test.support.import_fresh_module gets tripped up with its module 
blocking when you attempt to get a fresh copy of a submodule of a module where 
you are also importing the module that you are trying to block (bit of a doozy 
of a sentence there...). So, for example, with the following configuration in 
mymodule/__init__.py:

from .other import other

try:
from ._a import attr
except ImportError:
from ._b import attr

(Assuming _a.attr = "A" and _b.attr = "B"), if you attempt to do:

m = test.support.import_fresh_module("mymodule", 
fresh=("mymodule._other",), blocked=("mymodule._a"))

Then you'll find that m.attr is pulled from _a.attr. Here's a small script to 
demonstrate:

from test.support import import_fresh_module
import sys

def import_ab(fresh_other):
fresh = ("mymodule._other", ) if fresh_other else ()

mods_out = []
for to_block in "_b", "_a":
blocked = (f"mymodule.{to_block}",)

mods_out.append(import_fresh_module("mymodule",
fresh=fresh, blocked=blocked))
return mods_out


for fresh_other in [True, False]:
mymodule_a, mymodule_b = import_ab(fresh_other)

qualifier = "With" if fresh_other else "Without"
print(f"{qualifier} a fresh import of mymodule._other")

print(f"a: {mymodule_a.attr}")
print(f"b: {mymodule_b.attr}")
print()

When you run it with a suitably configured module on Python 3.8:

$ python importer.py 
With a fresh import of mymodule._other
a: A
b: A

Without a fresh import of mymodule._other
a: A
b: B

It also happens if you add `mymodule._a` or `mymodule._b` to the fresh list 
when you are trying to block the other one.

I *think* the problem is that in the step where _save_and_remove_module is 
called on fresh_name (see here: 
https://github.com/python/cpython/blob/76db37b1d37a9daadd9e5b320f2d5a53cd1352ec/Lib/test/support/__init__.py#L328-L329),
 it's necessarily populating `sys.modules` with a fresh import of the top-level 
module we're trying to import (mymodule) *before* the blocking goes into 
effect, then the final call to importlib.import_module just hits that cache.

I think either of the following options will fix this issue:

1. Switching the order of how "fresh" and "blocked" are resolved or
2. Deleting `sys.modules[name]` if it exists immediately before calling 
`importlib.import_module(name)

That said, I'm still having some weird statefulness problems if I block a C 
module's import and *then* block a Python module's import, so there may be some 
other underlying pathology to the current approach.

--
components: Tests
files: test_support_repro.zip
messages: 365702
nosy: brett.cannon, eric.snow, ncoghlan, p-ganssle
priority: normal
severity: normal
status: open
title: test.support.import_fresh_module fails to correctly block submodules 
when fresh is specified
type: behavior
versions: Python 3.7, Python 3.8, Python 3.9
Added file: https://bugs.python.org/file49031/test_support_repro.zip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com