[issue35943] PyImport_GetModule() can return partially-initialized module

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Created a new issue + fix in issue43517.

--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Antoine Pitrou]

Antoine Pitrou  added the comment:

After analysis, it may not need reversal.  There is a simple logic error it 
seems.  Will check.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Antoine Pitrou]

Change by Antoine Pitrou :


--
resolution: fixed -> 
stage: resolved -> needs patch
status: closed -> open
versions: +Python 3.10, Python 3.9 -Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Ok, going through other open issues including on third-party projects, I think 
these changes should unfortunately be reverted.  The regressions produced are 
far from trivial and most developers seem at a loss how to fix them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Note the conjunction of this change + issue32596 produces import fragility:
https://bugs.python.org/issue43515

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Cebtenzzre]

Change by Cebtenzzre :


--
nosy: +cebtenzzre

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Big Stone]

Big Stone  added the comment:

Is this bug causing the Dask-Jupyterlab failure ? 
https://github.com/dask/distributed/issues/4168

--
nosy: +Big Stone

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Atsuo Ishimoto]

Atsuo Ishimoto  added the comment:

After this fix, some functions like multiprocessing.Pool cannot be used in 
threaded code(https://bugs.python.org/issue41567).

importerror-sample.tgz contains simplified code to reproduce the same error 
without multiprocessing module. Is this an expected behaviour of this change?

Tested with Python 3.9.0/macOS 10.15.5.

--
nosy: +ishimoto
Added file: https://bugs.python.org/file49537/importerror-sample.tgz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Geoffrey Bache]

Geoffrey Bache  added the comment:

@Valentyn Tymofieiev - true, and thanks for the tip, though the symptoms 
described there are somewhat different from what I'm observing. Also, my 
problem seems to be dependent on zipping the Python code, which that one isn't.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Valentyn Tymofieiev]

Valentyn Tymofieiev  added the comment:

@gjb1002: see also https://bugs.python.org/issue38884, which demonstrates that 
concurrent imports are not thread-safe on Python 3.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Geoffrey Bache]

Geoffrey Bache  added the comment:

Oops, I mean we call PyImport_ImportModule and get these issues when the files 
are zipped. Unless that calls PyImport_GetModule internally I guess it's not 
related to this then.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Geoffrey Bache]

Geoffrey Bache  added the comment:

I have been experiencing what I thought was this issue in my embedded Python 
code. We have been using Python 3.7, so I thought upgrading to 3.8.1 would fix 
it, but it doesn't seem to have made any difference.

My C++ code essentially can call PyImport_GetModule() from two threads 
simultaneously on the same module A. The symptoms I see are that one of them 
then gets a stacktrace in module B (imported by A), saying that some symbol 
defined near the end of B does not exist.

I've also noticed that this happens far more often on deployed code (where 
Python modules end up in a zip file) than when run directly in development 
(where the modules are just normal files). I can't see any difference in the 
frequency between 3.7.5 and 3.8.1.

Any ideas? Should I reopen this?

--
nosy: +gjb1002

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Joannah Nanjekye]

Joannah Nanjekye  added the comment:

> Do you mean that there is a risk that the backport introduces a regression in 
> another part of the code? If yes, I would suggest to not backport the change 
> to *stable* branches.

My worry are the many changes that are required to ceval to make this back port 
work. Not that I think we can not successfully backport things. we can.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[STINNER Victor]

STINNER Victor  added the comment:

> The changes required to successfully do this backport are many and affect 
> critical areas. I am not in a hurry to do this. If anyone else wants to take 
> this up quickly, please do.

Do you mean that there is a risk that the backport introduces a regression in 
another part of the code? If yes, I would suggest to not backport the change to 
*stable* branches.

People survived with bug. Do you really *have to* backport the fix?

Note: this issue is closed. If you consider to backport it, I suggest to reopen 
the issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Joannah Nanjekye]

Joannah Nanjekye  added the comment:

The changes required to successfully do this backport are many and affect 
critical areas. I am not in a hurry to do this. If anyone else wants to take 
this up quickly, please do.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Valentyn Tymofieiev]

Valentyn Tymofieiev  added the comment:

Thanks. Is it possible that this issue and  https://bugs.python.org/issue38884 
are duplicates?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Brett Cannon]

Brett Cannon  added the comment:

I've assigned this to Joannah to decide if she wants to backport this.

--
assignee:  -> nanjekyejoannah
nosy: +nanjekyejoannah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Valentyn Tymofieiev]

Valentyn Tymofieiev  added the comment:

Do we plan to backport the change by nanjekyejoannah to 3.7 branch?

--
nosy: +Valentyn Tymofieiev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Brett Cannon]

Brett Cannon  added the comment:


New changeset 37c22206981f52ae35c28b39f7530f8438afbfdb by Brett Cannon (Joannah 
Nanjekye) in branch 'master':
bpo-35943: Prevent PyImport_GetModule() from returning a partially-initialized 
module (GH-15057)
https://github.com/python/cpython/commit/37c22206981f52ae35c28b39f7530f8438afbfdb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Brett Cannon]

Change by Brett Cannon :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Joannah Nanjekye]

Change by Joannah Nanjekye :


--
keywords: +patch
pull_requests: +14806
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/15057

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Paul Ganssle]

Change by Paul Ganssle :


--
nosy: +p-ganssle

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Eric Snow]

Eric Snow  added the comment:

Yeah, that makes sense.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35943] PyImport_GetModule() can return partially-initialized module

[Antoine Pitrou]

New submission from Antoine Pitrou :

PyImport_GetModule() returns whatever is in sys.modules, even if the module is 
still importing and therefore only partially initialized.

One possibility is to reuse the optimization already done in 
PyImport_ImportModuleLevelObject():

/* Optimization: only call _bootstrap._lock_unlock_module() if
   __spec__._initializing is true.
   NOTE: because of this, initializing must be set *before*
   stuffing the new module in sys.modules.
 */
spec = _PyObject_GetAttrId(mod, ___spec__);
if (_PyModuleSpec_IsInitializing(spec)) {
PyObject *value = _PyObject_CallMethodIdObjArgs(interp->importlib,
__lock_unlock_module, abs_name,
NULL);
if (value == NULL) {
Py_DECREF(spec);
goto error;
}
Py_DECREF(value);
}
Py_XDECREF(spec);

Issue originally mentioned in issue34572.

--
components: Interpreter Core
messages: 335097
nosy: brett.cannon, eric.snow, ncoghlan, pitrou
priority: normal
severity: normal
stage: needs patch
status: open
title: PyImport_GetModule() can return partially-initialized module
type: behavior
versions: Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com