[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ned Deily]

Ned Deily  added the comment:

In Python 3.8, the default start method for multiprocessing when run on macOS 
is now "spawn" instead of "fork".

"Changed in version 3.8: On macOS, the spawn start method is now the default. 
The fork start method should be considered unsafe as it can lead to crashes of 
the subprocess. See bpo-33725."

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> Python crashes on macOS after fork with no exec
versions:  -Python 2.7, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

That's annoying, I cannot reproduce the issue on macOS 10.13. I don't have 
access to my test VM running 10.11 at the moment, I'll work on documenting this 
when I get back home (and do have access to 10.11 again).

BTW. I don't know if providing a reliable test is possible other than testing 
that the documented procedure works in general.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Ronald Oussoren> I can provide a patch, but that will likely be during 
EuroPython as I’m currently taking a roundabout route towards Edinburgh.

Can you add a test of the procedure you document for using _tkinter with 
multiprocessing?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Antoine Pitrou]

Antoine Pitrou  added the comment:

It will definitely break *some* user code. Also, everyone not using tkinter 
(which I think is the majority of users) isn't affected by this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I can provide a patch, but that will likely be during EuroPython as I’m 
currently taking a roundabout route towards Edinburgh. 

Btw. I’m not sure yet how impactful changing the default spawning mechanism 
would be for 3.8.  How likely is that to break user code?   

Op 16 jul. 2018 om 10:24 heeft Antoine Pitrou  het 
volgende geschreven:

> 
> Antoine Pitrou  added the comment:
> 
> macOS users, feel free to propose a doc PR for multiprocessing.
> 
> --
> assignee:  -> docs@python
> components: +Documentation
> nosy: +docs@python
> versions: +Python 3.7, Python 3.8 -Python 3.5
> 
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Antoine Pitrou]

Antoine Pitrou  added the comment:

macOS users, feel free to propose a doc PR for multiprocessing.

--
assignee:  -> docs@python
components: +Documentation
nosy: +docs@python
versions: +Python 3.7, Python 3.8 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Terry J. Reedy]

Terry J. Reedy  added the comment:

I was uncertain from Ethan's post what does work best. "endless errors with 
both 'spawn", "Only 'fork' works...but without graphics", "method='spawn'and 
... does work".  Do the 1st and 3rd sentences mean that spawn + graphics fails 
without the proper additional code but does work with it?  Should the 2nd 
sentence be "fork only works without graphics" rather than starting with 'only 
fork'?

If so, this would suggest that the default should depend on whether there is a 
graphic import ;-).  In any case, how to use successfully use multiprocessing 
with graphics on Mac should be documented.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

As I mentioned in msg314304 this is almost certainly expected behavior on macOS 
because Apple’s GUI frameworks are not fork()-safe.  There is nothing we can do 
about this other than changing the default spawn mechanism for multiprocessing 
on macOS. 

--
On the road, hence brief. 

Op 14 jul. 2018 om 19:30 heeft Terry J. Reedy  het 
volgende geschreven:

> 
> Terry J. Reedy  added the comment:
> 
> On MacOS, 3.7.0 is compiled for and the installer loads tcl/tk 8.6.8.  The 
> same is true for the 3.6.6 64-bit installer.  Do tkinter and multiprocessing 
> work together better with these installations?
> 
> I want to consider using multiprocessing and pipes instead subprocess and 
> socket for IDLE's user-code execution process, started from and communicating 
> with the initial tkinter gui process.  idlelib.run, which runs in the 
> execution process to communicae with the gui process and supervise running 
> user code, imports tkinter.  Besides which, users have to be able to import 
> tkinter in their programs.
> 
> --
> nosy: +pitrou
> 
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Terry J. Reedy]

Terry J. Reedy  added the comment:

On MacOS, 3.7.0 is compiled for and the installer loads tcl/tk 8.6.8.  The same 
is true for the 3.6.6 64-bit installer.  Do tkinter and multiprocessing work 
together better with these installations?

I want to consider using multiprocessing and pipes instead subprocess and 
socket for IDLE's user-code execution process, started from and communicating 
with the initial tkinter gui process.  idlelib.run, which runs in the execution 
process to communicae with the gui process and supervise running user code, 
imports tkinter.  Besides which, users have to be able to import tkinter in 
their programs.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ethan Welty]

Ethan Welty  added the comment:

@ronaldoussoren: The order of the imports made no difference. Even with the 
call at the top, I got endless errors with both 'spawn' and 'forkserver', with 
or without importing a graphics backend. Only 'fork' works, and only if a 
graphics package is not imported.

** HOWEVER **

Setting method='spawn' or 'forkserver' and force=True at top, or calling 
multiprocessing.set_start_method in __main__ does work (when run as scripts 
from command line). It's a shame to have to give up the convenience of 'fork', 
but this is a start.

```
import multiprocessing
multiprocessing.set_start_method("spawn", force=True)
import numpy as np
# import _tkinter

def parallel_matmul(x):
R = np.random.randn(3, 3)
return np.matmul(R, x)

if __name__ == '__main__':
# multiprocessing.set_start_method("spawn", force=False)
pool = multiprocessing.Pool(4)
results = pool.map(parallel_matmul, [np.random.randn(3, 5000) for i in 
range(2)])
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

@ezwelty: The import of multiprocessing and the call to set_start_method should 
be at the very start of the code, before other imports, to avoid the annoying 
Apple behavior I mentioned.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ethan Welty]

Ethan Welty  added the comment:

Terry Reedy: I just tried your suggestion of a main clause (code below) and it 
made no difference.

```
import _tkinter
import numpy as np
# multiprocessing.set_start_method("spawn")
import multiprocessing

def parallel_matmul(x):
R = np.random.randn(3, 3)
return np.matmul(R, x)

if __name__ == '__main__':
pool = multiprocessing.Pool(4)
results = pool.map(parallel_matmul, [np.random.randn(3, 5000) for i in 
range(2)])
```

Ronald Oussoren: Good to know that perhaps upgrading to the latest OSX would 
resolve this issue. I tried 'spawn' and 'forkserver' with 
multiprocessing.set_start_method() (see above) but both result in an endless 
stream of errors (with or without importing a graphics backend), although that 
may be my not knowing how to properly use them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Terry J. Reedy]

Terry J. Reedy  added the comment:

The examples in the multiprocessing doc all put all multiprocessing calls 
within a "if __name__ == '__main__':" statement.  I know that this is necessary 
on Windows, but don't know if or when it helps on other OSes.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I cannot reproduce the problem on macOS 10.13.3 with python 3.6 (but haven't 
spent much time on this).

The combination of GUI code with multiprocessing is a known source of problems 
on macOS, the system libraries are hostile to that.

Changing the start method from "fork" to "spawn" or "forkserver" should help 
with this, using multiprocessing.set_start_method(). This should be changed 
before doing anything that might Apple system frameworks related to GUIs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ethan Welty]

Change by Ethan Welty :


--
components: +macOS

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ethan Welty]

Ethan Welty  added the comment:

I've tried with additional backends: WX, WXAgg, WXCairo, Qt5Agg (in matplotlib 
speak).

With these, I can at least import matplotlib.pyplot, but as soon as say 
matplotlib.pyplot.plot is called and the backend is loaded, the code breaks 
(error message for WXAgg below). Is multiprocessing in an interactive shell 
simply not meant to be supported on MacOS?

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File 
"/Users/Admin/.pyenv/versions/3.6.4/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/_pylab_helpers.py",
 line 78, in destroy_all
manager.destroy()
  File 
"/Users/Admin/.pyenv/versions/3.6.4/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/backends/backend_wx.py",
 line 1303, in destroy
self.frame.Destroy()
  File 
"/Users/Admin/.pyenv/versions/3.6.4/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/backends/backend_wx.py",
 line 1256, in Destroy
if not self.IsBeingDeleted():
RuntimeError: wrapped C/C++ object of type FigureFrameWxAgg has been deleted

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ethan Welty]

Ethan Welty  added the comment:

I have tried running the script with:

- command line (python ): Works without (breaks with) `import _tkinter`.
- basic python console (python): Same as with command line.
- ipython: Fails with or without `import _tkinter`. However, if for example I 
make the matrix in the scripts smaller or use only functions that don't resort 
to multithreading, code runs successfully in parallel in ipython.
- IDLE: Perhaps unsurprisingly, the code fails with or without `import 
_tkinter` (never terminates, or prompts endless "Your program is still running! 
Do you want to kill it?")

Suspecting a problem with Tcl/Tk (Apple's original 8.5.9), I rebuilt python 
pointing to tcl-tk installed with brew (8.6.8), which I checked with IDLE and 
_tkinter.TCL_VERSION() / _tkinter.TK_VERSION(). However, this did not fix the 
problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I'm fairly sure this is expected behavior on macOS: _tkinter loads the Tk 
library, which loads Apple GUI frameworks. Those frameworks are not save w.r.t. 
fork(2) and that can lead to all kinds of unwanted behavior (although I would 
have expected a more explicit crash than this)

--
nosy: +ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ned Deily]

Ned Deily  added the comment:

Are you by any chance trying to run this under IDLE?

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ethan Welty]

Change by Ethan Welty :


--
versions:  -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

[Ethan Welty]

New submission from Ethan Welty :

Merely importing tkinter breaks the use of parallel code on my system (Mac OSX 
10.11.6, tested on Python 2.7.13 / 2.7.14 / 3.5.0 / 3.6.4, all barebones 
distributions installed with pyenv). I've tested this with both multiprocessing 
and sharedmem (see minimal scripts below).

The issue seems to apply only to functions that evoke multithreading within 
their respective package (e.g. `numpy.matmul()`, 
`cv2.SIFT.detectAndCompute()`). If I make the matrix in the scripts below much 
smaller (e.g. change `5000` to `5`), avoiding internal multithreading, the 
scripts work.

## with `multiprocessing`

```python
import numpy as np
import multiprocessing
import _tkinter

def parallel_matmul(x):
R = np.random.randn(3, 3)
return np.matmul(R, x)

pool = multiprocessing.Pool(4)
results = pool.map(parallel_matmul, [np.random.randn(3, 5000) for i in 
range(2)])
```

> *Code never exits and Python has to be force quit*

## with `sharedmem`

```python
import numpy as np
import sharedmem
import _tkinter

def parallel_matmul(x):
R = np.random.randn(3, 3)
return np.matmul(R, x)

with sharedmem.MapReduce() as pool:
results = pool.map(parallel_matmul,
[np.random.randn(3, 5000) for i in range(2)])
```

> sharedmem.sharedmem.SlaveException: slave process 1 killed by signal 11

--
components: Tkinter
messages: 314160
nosy: ezwelty
priority: normal
severity: normal
status: open
title: Merely importing tkinter breaks parallel code (multiprocessing, 
sharedmem)
type: crash
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com