[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Tal Einat]

Tal Einat  added the comment:

I am able to reproduce this with Python 3.6.3 and 3.7.0, but not with 3.7.4, 
3.8.0b3 or current master (to be 3.9).

This has been fixed since 3.7.3; see issue #34572.

--
nosy: +taleinat
resolution: not a bug -> fixed
stage:  -> resolved
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> not a bug
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +serhiy.storchaka
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12680
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Brett Cannon]

Brett Cannon added the comment:

I just checked and this is no longer an issue in Python 3.3.

The sys.modules bug isn't a bug as that's how it is supposed to work to 
prevent partially initialized modules. As for how pickle is doing stuff, that 
could change if it wouldn't break backwards-compatibility.

--
versions:  -Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12680
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Chris Kaynor]

Changes by Chris Kaynor ckay...@zindagigames.com:


--
nosy: +DragonFireCK

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12680
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Sagiv Malihi]

Sagiv Malihi sagivmal...@gmail.com added the comment:

As I said - it certainly happenes on 3.2 (tested).

@pitrou - what you suggested will not work since the actual import will block 
on the import lock.
The optimization there is not needed, it's already implemented in __import__.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12680
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 PyImport_ExecCodeModuleEx adds the module to sys.modules *before*
 actually executing the code. This is a design flaw (can it really be
 changed? )

I guess it is done so to allow for circular imports.

 The second bug: in cPickle.c: func_class() 
 cPickle 'manually' checks if a module is in sys.modules instead of
 letting the import mechanism do it for him (hence breaking the import
 lock's defense here).

I would guess it is an optimization shortcut. A solution (while keeping the 
optimization) would be to take the import lock before checking sys.modules.

Note that the _pickle module in 3.x has the same kind of logic, and therefore 
probably the same issue too (haven't tested).

--
nosy: +pitrou
versions: +Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12680
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Sagiv Malihi]

New submission from Sagiv Malihi sagivmal...@gmail.com:

When trying to cPickle.loads() from several threads at once, there is a race 
condition when threads try to import modules.

An example will explain it best:
suppose I have module foo.py which takes some time to load:

import time
class A(object):
def __setstate__(self, state):
self.x = x
time.sleep(1)
x = 5

and a pickled version of an A() object stored in 'A.pkl'.
the following code, when run for the first time, will raise a NameError about 
'x':

 p = open('A.pkl','rb').read()
 [thread.start_new(cPickle.loads, (p,)) for x in xrange(2)]

Unhandled exception in thread started by built-in function loads
Traceback (most recent call last):
  File foo.py, line 7, in __setstate__
self.x = x
NameError: global name 'x' is not defined


since the module is now loaded, subsequent calls to cPickle.loads will work as 
expected.

This was tested on 2.5.2, 2.7.1, and 3.2 on Ubuntu and on Windows 7.

please note that this bug was discovered when unpickling the standard 
'decimal.Decimal' class (decimal.py is quite long and takes some time to 
import), and this is not some corner case.

--
components: Extension Modules
messages: 141548
nosy: Sagiv.Malihi
priority: normal
severity: normal
status: open
title: cPickle.loads is not thread safe due to non-thread-safe imports
type: crash
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12680
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Sagiv Malihi]

Sagiv Malihi sagivmal...@gmail.com added the comment:

OK, digging deeper reveals that there are actually two bugs here, one is 
conceptual in the python importing mechanism, and the other is technical in 
cPickle.

The first bug: 
PyImport_ExecCodeModuleEx adds the module to sys.modules *before* actually 
executing the code. This is a design flaw (can it really be changed? )
Demonstrating this bug is easy using the foo.py module from the previous 
comment:

def f():
if 'bla' in sys.modules:
bla = sys.modules['bla']
else:
import bla
return bla.A()
running two instances of f in two threads results in the same error.

The second bug: in cPickle.c: func_class() 
cPickle 'manually' checks if a module is in sys.modules instead of letting the 
import mechanism do it for him (hence breaking the import lock's defense here).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12680
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
nosy: +brett.cannon, ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12680
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com