[issue31115] Py3.6 threading/reference counting issues with `numexpr`

[Mark Dickinson]

Mark Dickinson  added the comment:

Judging by the current state of https://github.com/pydata/numexpr/issues/252, 
it does seem as though this is resolved downstream, so probably safe to close 
here. Robert: feel free to reopen if I misunderstood.

--
nosy: +mark.dickinson
stage:  -> resolved
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31115] Py3.6 threading/reference counting issues with `numexpr`

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31115] Py3.6 threading/reference counting issues with `numexpr`

[Antoine Pitrou]

Antoine Pitrou added the comment:

If, judging by 
https://github.com/pydata/numexpr/commit/07d9245d88759f0c3dcabd88e6edefadc3061ee3,
 you are really calling a bunch of C API functions without holding the GIL, 
then it's not surprising you may get crashes all over the place.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31115] Py3.6 threading/reference counting issues with `numexpr`

[Robert McLeod]

Robert McLeod added the comment:

After building with Python3.7 I was able to get a useful error message that 
`PyMem` functions were being called inside GIL release.  I will replace these 
with C-equivalents and try with Python 3.6.

Fatal Python error: Python memory allocator called without holding the GIL

Thread 0x04044e00 (most recent call first):
  File 
"/home/rmcleod/py37/lib/python3.7/site-packages/numexpr3-3.0==19176== 
==19176== Process terminating with default action of signal 6 (SIGABRT)
==19176==at 0x579C428: raise (raise.c:54)
==19176==by 0x579E029: abort (abort.c:89)
==19176==by 0x422DF7: Py_FatalError (pylifecycle.c:1849)
==19176==by 0x41ED4C: _PyMem_DebugCheckGIL (obmalloc.c:1972)
==19176==by 0x41ED23: _PyMem_DebugMalloc (obmalloc.c:1980)
==19176==by 0x41FCAC: PyMem_Malloc (obmalloc.c:418)
==19176==by 0xCEA1920: NumExprObject_copy_threadsafe(NumExprObject 
const*) (interpreter.cpp:147)
==19176==by 0xCEA77CE: th_worker(void*) (module.cpp:73)
==19176==by 0x4E416B9: start_thread (pthread_create.c:333)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31115] Py3.6 threading/reference counting issues with `numexpr`

[Robert McLeod]

New submission from Robert McLeod:

I'm working on the development branch of the `numexpr` module and I've run into 
some problems on Python 3.6, where I seem to get a variety of errors relating 
to threading or reference counting errors. This module is commonly used for 
accelerating NumPy code, as it can parse an expression into a C-extension 
virtual machine program and break the calculations down into blocks which are 
then dispatched over multiple threads. I get similar errors on both Ubuntu 
16.04 with GCC5.4 and Win7 with MSVC2015.  I created an issue here where I 
provide some of the `gcc` error outputs.  

https://github.com/pydata/numexpr/issues/252

Typically I'm getting a different error on every run. 

We use a wrapper for Windows threads that emulated pthreads, and I found on 
Windows crashes occurred on calling the Windows system function 
`WaitForSingleObject(cond->sema, INFINITE);` in the file 
numexpr3/win32/pthread.c.  

I cannot replicate this problem in Python 2.7/3.4/3.5 on Windows or Linux. I'm 
using Anaconda in both instances, with nomkl NumPy on Linux and mkl NumPy on 
Windows.  

I tried valgrinding with Python 3.5 and 3.6, and I get numerous errors coming 
from places like pickle and ast (which the new NumExpr uses) in 3.6 and it's 
basically clean in 3.5.  The logs are attached to the issue linked above.

--
components: Extension Modules, Library (Lib)
messages: 299722
nosy: Robert McLeod
priority: normal
severity: normal
status: open
title: Py3.6 threading/reference counting issues with `numexpr`
type: crash
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com