[issue30018] multiprocessing.Pool garbles call stack for __new__

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +serhiy.storchaka
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30018] multiprocessing.Pool garbles call stack for __new__

[Charles McEachern]

Charles McEachern added the comment:

That seems to do it! Looks like the trick is to define __reduce__ to help out 
the serializer. Thanks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30018] multiprocessing.Pool garbles call stack for __new__

[R. David Murray]

R. David Murray added the comment:

I suspect you just need to add pickle support to your class.  When I subclassed 
str in the email package, I found I needed to do that.  I'd have to go through 
the docs again to remember how the code works, but you can take a look at the 
BaseHeader class in email.headerregistry to see what I did.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30018] multiprocessing.Pool garbles call stack for __new__

[Davin Potts]

Davin Potts added the comment:

> I am unfortunately not at liberty to share the code I'm working on.

I very much understand and am very thankful you took the time to create a 
simple example that you could share.  Honestly, that's the reason I felt 
inspired to stop what I was doing to look at this now rather than later.


> I suppose I should just work around it by checking right away if the input to 
> my constructor has already been constructed!

There are probably a number of different ways to address it but your suggestion 
of adding a check to see if this is the first time that object has been 
constructed sounds like it might be an easy win.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30018] multiprocessing.Pool garbles call stack for __new__

[Charles McEachern]

Charles McEachern added the comment:

This caused me several hours of misery yesterday, trying to isolate what was 
going wrong. 

I am unfortunately not at liberty to share the code I'm working on. The example 
on GitHub has the general thrust of it: my constructor was always called in a 
specific way, and didn't expect to be given something that was already 
processed. 

Interesting to see that this is a product of pickling. That makes me think that 
"fixing" this corner case would probably be a lot of work.

I suppose I should just work around it by checking right away if the input to 
my constructor has already been constructed!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30018] multiprocessing.Pool garbles call stack for __new__

[Davin Potts]

Davin Potts added the comment:

Expanding my above example to show how multiprocessing relates:
>>> import multiprocessing
>>> import os
>>> class Floof(object):
... def __new__(cls):
... print("New via pid=%d" % os.getpid())
... return object.__new__(cls)
... 
>>> os.getpid()   # parent pid
46560
>>> pool = multiprocessing.Pool(1)
>>> getter = pool.apply_async(Floof, (), {})  # output seen from child AND 
>>> parent
>>> New via pid=46583
New via pid=46560

>>> getter.get()  # everything seems to be working 
>>> as intended
<__main__.Floof object at 0x10866f250>


FWIW, near the end of my prior message:  s/it didn't merely/it merely/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30018] multiprocessing.Pool garbles call stack for __new__

[Davin Potts]

Davin Potts added the comment:

It looks like the first 'Called Foo.__new__' is being reported by the child 
(pool of 1) process and the second 'Called Foo.__new__' is being reported by 
the parent process.  In multiprocessing, because objects are by default 
serialized using pickle, this may be caused by the unpickling of the Foo object 
by the parent process which is something you would not experience when using 
ThreadPool because it does not have the same need for serialization.

Example showing invocation of __new__ as part of unpickling:
>>> class Foo(object):
... def __new__(cls):
... print("New")
... return object.__new__(cls)
... 
>>> import pickle
>>> f = Foo()
New
>>> pf = pickle.dumps(f, protocol=2)
>>> pickle.loads(pf)  # unpickling triggers __new__
New
<__main__.Foo object at 0x1084a06d0>



Having discovered this phenomenon, is this causing a problem for you somewhere 
in code?  (Your example code on github was helpful, thank you, but it didn't 
merely demonstrated the behavior and didn't show where this was causing you 
pain.)

--
nosy: +davin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30018] multiprocessing.Pool garbles call stack for __new__

[Charles McEachern]

New submission from Charles McEachern:

I'm calling the constructor of Foo, a subclass of str. Expected output:

Called Foo.__new__ with args = ('TIMESTAMP', 'INPUT0')
TIMESTAMP OUTPUT0

When I make the call using a multiprocessing.pool.ThreadPool, it works fine. 
But when I make the call using a multiprocessing.Pool (using the apply or 
apply_async method), I get:

Called Foo.__new__ with args = ('TIMESTAMP', 'INPUT0')
Called Foo.__new__ with args = ('TIMESTAMP OUTPUT0',)
Exception in thread Thread-3:
...
ValueError: Bad Foo input: ('TIMESTAMP OUTPUT0',)

That is, the object I just constructed seems to be getting shoved right back 
into the constructor. 

When I swap out the Foo class for the similar Goo class, which is not a str, 
and uses __init__ instead of __new__, I again see no problems:

Called Goo.__init__ with args = ('TIMESTAMP', 'INPUT0')


I see this in 2.7.9 as well as 3.4.5. Looks like it's present in 2.7.2 and 
3.5.2 as well:

https://github.com/charles-uno/python-new-pool-bug/issues/1

--
components: Library (Lib)
files: newpool.py
messages: 291278
nosy: Charles McEachern
priority: normal
severity: normal
status: open
title: multiprocessing.Pool garbles call stack for __new__
type: behavior
versions: Python 2.7, Python 3.4, Python 3.5
Added file: http://bugs.python.org/file46790/newpool.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com