[issue27649] multiprocessing on Windows does not properly manage class attributes

[R. David Murray]

R. David Murray added the comment:

Right this is a fork vs spawn issue, not a bug.

I don't know why Eryk changed the versions the way he did, but FYI we use that 
field to indicate which version we will apply a patch to, so it is pretty much 
irrelevant when there isn't actually a bug :)

--
nosy: +r.david.murray
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27649] multiprocessing on Windows does not properly manage class attributes

[Ian Cordasco]

Ian Cordasco added the comment:

Why did you remove Python 3.3? It's still affected by this behaviour.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27649] multiprocessing on Windows does not properly manage class attributes

[Eryk Sun]

Eryk Sun added the comment:

Queuing the class doesn't do anything here. It gets pickled as a module 
attribute; the queue isn't marshaling the class dict. For example:

>>> pickletools.dis(pickle.dumps(mccabe.McCabeChecker))
0: \x80 PROTO  3
2: cGLOBAL 'mccabe McCabeChecker'
   24: qBINPUT 0
   26: .STOP
highest protocol among opcodes = 2

Unpickling this is basically getattr(sys.modules['mccabe'], 'McCabeChecker'). 
So you'll see the same result if you only put ('max_complexity', 10) in the 
queue and hard code the class in the _target function. 

Of course for a system that uses fork() the class attribute is the same in the 
child process. In 3.4+ you can force spawning by initially calling 
multiprocessing.set_start_method('spawn'). Then you'll see the same result as 
in Windows.

--
nosy: +eryksun
versions:  -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27649] multiprocessing on Windows does not properly manage class attributes

[Ian Cordasco]

New submission from Ian Cordasco:

In trying to add support for multiprocessing on Windows in Flake8, I found that 
the behaviour of the module is significantly different than on Unix. If you 
have a class that has class-level attributes on Unix, and you change them, put 
the class into a Queue (or have it attached to an object that moves through a 
Queue) then the values you set will be preserved [1]. On Windows [2], the 
attributes are reset.


As a minimal reproduction, I've thrown together this project: 
https://github.com/sigmavirus24/windows-multiprocessing-bug which is running 
tox on AppVeyor 
(https://ci.appveyor.com/project/sigmavirus24/windows-multiprocessing-bug) and 
Travis CI (https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug) so 
the behaviours can be compared.

For application developers, it would be wonderful if we could rely on the 
standard library behaving the same way in this case across both operating 
systems.

[1] 
https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276925#L158
 
https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276926#L158
 
https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276927#L157

[2] 
https://ci.appveyor.com/project/sigmavirus24/windows-multiprocessing-bug/build/1.0.2#L24

--
components: Windows
messages: 271618
nosy: icordasc, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: multiprocessing on Windows does not properly manage class attributes
type: behavior
versions: Python 3.3, Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com