[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

STINNER Victor  added the comment:

(Ah, Benjamin restarted the discussion, so I reopen this issue.)

> I understand the desire to compile Python with OpenMP.

I'm not sure that I understood the use case. Do you want to only compile Python 
core ("python3" binary") or just stdlib C extensions, or both?

> But the resolution here is hiding _Py_atomic symbols all the time, even when 
> OpenMP isn't involved, and even when building a standard extension module.

Sorry, but I don't understand the problem. Why is it an issue to hide 
_Py_atomic symbols?

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Benjamin Peterson]

Benjamin Peterson  added the comment:

Yes, why not fix this by putting the offending code under "#ifndef _OPENMP"?

What would be even better if is pyatomic.h wasn't included in Python.h, which 
should be fine since pyatomic.h doesn't have any public APIs. Maybe we can do 
that for 3.8.

--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

STINNER Victor  added the comment:

This issue is closed. Would you mind to either reopen it or create a new issue?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Antoine Pitrou]

Antoine Pitrou  added the comment:

(see issue34128 for context)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Case in point: if I want to include "internal/pystate.h" from _pickle.c, I get 
the following error:

"""
In file included from ./Include/internal/pystate.h:12:0,
 from /home/antoine/cpython/default/Modules/_pickle.c:10:
./Include/internal/ceval.h:14:5: error: unknown type name ‘_Py_atomic_int’
 _Py_atomic_int calls_to_do;
 ^
"""

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I think this should be reconsidered.  I understand the desire to compile Python 
with OpenMP.  But the resolution here is hiding _Py_atomic symbols all the 
time, even when OpenMP isn't involved, and even when building a standard 
extension module.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Alexander Heger]

Alexander Heger added the comment:

seems to work now with 3.5.1rc1

On 5 November 2015 at 23:01, STINNER Victor  wrote:
>
> STINNER Victor added the comment:
>
> "Pyatomic-2.patch solved the problem."
>
> Great! The good news is that the Python 3.5.1 release has now a schedule: 
> https://www.python.org/dev/peps/pep-0478/
>
> "3.5.1 final: December 6, 2015"
>
> --
>
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

STINNER Victor added the comment:

cool

2015-11-23 22:19 GMT+01:00 Alexander Heger :
>
> Alexander Heger added the comment:
>
> seems to work now with 3.5.1rc1
>
> On 5 November 2015 at 23:01, STINNER Victor  wrote:
>>
>> STINNER Victor added the comment:
>>
>> "Pyatomic-2.patch solved the problem."
>>
>> Great! The good news is that the Python 3.5.1 release has now a schedule: 
>> https://www.python.org/dev/peps/pep-0478/
>>
>> "3.5.1 final: December 6, 2015"
>>
>> --
>>
>> ___
>> Python tracker 
>> 
>> ___
>
> --
>
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[gul916]

gul916 added the comment:

Hi,
Just a few words to tell you that I had the same problem to compile scipy 
0.16.0 with mkl libraries under python 3.5 and linux fedora 22. 
Pyatomic-2.patch solved the problem.
Thanks

--
nosy: +gul916

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

STINNER Victor added the comment:

"Pyatomic-2.patch solved the problem."

Great! The good news is that the Python 3.5.1 release has now a schedule: 
https://www.python.org/dev/peps/pep-0478/

"3.5.1 final: December 6, 2015"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Arkadiusz Miskiewicz Arkadiusz Miskiewicz]

Arkadiusz MiskiewiczArkadiusz Miskiewicz added the 
comment:

Same for 3.5 branch from hg (git mirror actually).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Arkadiusz Miskiewicz Arkadiusz Miskiewicz]

Arkadiusz MiskiewiczArkadiusz Miskiewicz added the 
comment:

Should it work with /configure '--with-cxx-main=g++' && make?

Because currently it doesn't:

g++ -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv 
-O3 -Wall -Wstrict-prototypes-Werror=declaration-after-statement   -I. 
-IInclude -I./Include-DPy_BUILD_CORE -o Programs/python.o 
./Programs/python.c
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC 
but not for C++
In file included from Include/pyatomic.h:10:0,
 from Include/Python.h:53,
 from ./Programs/python.c:3:
/usr/lib64/gcc/x86_64-pld-linux/5.2.0/include/stdatomic.h:40:9: error: 
‘_Atomic’ does not name a type
[...]

(testing 3.5.0 + patch for this issue from hg)

--
nosy: +arekm

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

Changes by STINNER Victor :


--
keywords: +3.5regression

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

STINNER Victor added the comment:

Yes my fix will be part of Python 3.5.1 release.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Alexander Heger]

Alexander Heger added the comment:

Dear Victor,

yes, you patch seems to fix the yt install.  Thank you very much!

I hope this can be included in 3.5.1.

Best wishes,
Alexander

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Alexander Heger]

Alexander Heger added the comment:

So, apparently, more than just one spot needs to be fixed.  I also tried just 
modifying the Python.h after install, but that does not do the trick either.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

STINNER Victor added the comment:

/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c:2062:15: error: 
‘_PyThreadState_Current’ undeclared (first use in this function)
 if (Py_EnterRecursiveCall("while processing _as_parameter_")) {
   ^

Ah yes, if you check the fix for C++ (changeset cb05b6d7aacd), I also had to 
modify pystate.h.

Please try pyatomic-2.patch which hides more CPython internals in public 
headers.

--
Added file: http://bugs.python.org/file40497/pyatomic-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

STINNER Victor added the comment:

I created a venv with a Python patched with pyatomic-2.patch: I successfully 
installed yt. yt depends on numpy & Cython: good news, numpy & Cython were 
compiled correctly. These two libraries are well known users of the Python C 
API. It's not enough to check if this change breaks modules on PyPI, but it's 
still a good news :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Stefan Behnel]

Stefan Behnel added the comment:

Would there be a way to expose these internals rather than hiding them?

--
nosy: +scoder

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

STINNER Victor added the comment:

> Would there be a way to expose these internals rather than hiding them?

Here is the issue is that pyatomic.h cannot be compiled on OpenMP. We had the 
same issue with C++. In fact, it doesn't make sense to compile pyatomic.h 
differently to access an atomic variable from an extension module. We must 
always use exactly the same implementation, otherwise bad things will happen.

A solution for that is to hide the implementation details and only expose high 
level APIs.

For example, pyatomic.h must be completly hidden.

A consequence is that the _PyThreadState_Current variable must be hidden to. 
_PyThreadState_Current is an implementation detail, you must not access it 
directly.

The PyThreadState_GET() macro uses directly the _PyThreadState_Current 
variable. So the solution to expose the "PyThreadState_GET" symbol (not 
necessary as a macro) is to define it as an alias to the PyThreadState_Get() 
*function*.

The advantage of using a function is that we don't expose implementation 
details to third-party extensions, it avoids the risk of ABI incompatibilies.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Stefan Behnel]

Stefan Behnel added the comment:

Understood and agreed. Second patch looks good to me.

Cython calls PyThreadState_GET() in pretty much every helper function that 
deals with exceptions, but I doubt that the potential speed difference is going 
to be relevant in the real world. And we target CPython's API level anyway, not 
the ABI, so the C code will just adapt at compile time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

Changes by STINNER Victor :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Roundup Robot]

Roundup Robot added the comment:

New changeset d4fcb362f7c6 by Victor Stinner in branch '3.5':
Issue #25150: Hide the private _Py_atomic_xxx symbols from the public
https://hg.python.org/cpython/rev/d4fcb362f7c6

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

STINNER Victor added the comment:

I pushed my fix pyatomic-2.patch to Python 3.5 and 3.6.

Thanks for the bug report Alexander Heger!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

STINNER Victor added the comment:

"When I just comment out the << #include "pyatomic.h"  >> line, python 3.5.0 
will no longer compile"

Sure. My idea is to "disable" the header with we are not building Python 
itself. There is a nice define for that: Py_BUILD_CORE. See attached patch.

Since all symbols in pyatomic.h are prefixed by _Py, this header is fully part 
of the Python private API and so it's fine to modify it in a bugfix release 
(3.5.0 => 3.5.1).

--
keywords: +patch
Added file: http://bugs.python.org/file40493/pyatomic.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Alexander Heger]

Alexander Heger added the comment:

When I just comment out the 

#include "pyatomic.h" 

line, python 3.5.0 will no longer compile

gcc -pthread -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG 
-g -fwrapv -O3 -Wall -Wstrict-prototypes-Werror=declaration-after-statement 
  -I. -IInclude -I./Include-DPy_BUILD_CORE -o Programs/python.o 
./Programs/python.c
In file included from Include/traceback.h:8:0,
 from Include/Python.h:97,
 from ./Programs/python.c:3:
Include/pystate.h:186:8: error: unknown type name ‘_Py_atomic_address’
 PyAPI_DATA(_Py_atomic_address) _PyThreadState_Current;
^
Makefile:747: recipe for target 'Programs/python.o' failed
make: *** [Programs/python.o] Error 1

pystate.h:

/* Assuming the current thread holds the GIL, this is the
   PyThreadState for the current thread.

   Issue #23644: pyatomic.h is incompatible with C++ (yet). Disable
   PyThreadState_GET() optimization: declare it as an alias to
   PyThreadState_Get(), as done for limited API. */
#if !defined(Py_LIMITED_API) && !defined(__cplusplus)
PyAPI_DATA(_Py_atomic_address) _PyThreadState_Current;
#endif

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[STINNER Victor]

Changes by STINNER Victor :


--
title: yt package pip compile/install error -> 3.5: Include/pyatomic.h is 
incompatible with OpenMP (compilation of the third-party yt module fails on 
Python 3.5)
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Brett Cannon]

Brett Cannon added the comment:

If everything in the header is a _Py definition then I agree we should hide it 
in Python.h from the public.

--
nosy: +brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

[Alexander Heger]

Alexander Heger added the comment:

if I just include this patch, some modules don't build:

(...)

gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code 
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes 
-Werror=declaration-after-statement 
-Ibuild/temp.linux-x86_64-3.5/libffi/include 
-Ibuild/temp.linux-x86_64-3.5/libffi 
-I/home/alex/Python-3.5.0/Modules/_ctypes/libffi/src -I./Include 
-I/home/alex/Python/include -I. -IInclude -I/usr/local/include 
-I/home/alex/Python-3.5.0/Include -I/home/alex/Python-3.5.0 -c 
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c -o 
build/temp.linux-x86_64-3.5/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.o 
-Wall -fexceptions
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c: In function 
‘PyCSimpleType_from_param’:
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c:2062:15: error: 
‘_PyThreadState_Current’ undeclared (first use in this function)
 if (Py_EnterRecursiveCall("while processing _as_parameter_")) {
   ^
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c:2062:15: note: each 
undeclared identifier is reported only once for each function it appears in
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c:2062:28: error: 
‘__atomic_load_ptr’ undeclared (first use in this function)
 if (Py_EnterRecursiveCall("while processing _as_parameter_")) {
^
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c:2062:66: error: argument 1 of 
‘__atomic_load’ must be a non-void pointer type
 if (Py_EnterRecursiveCall("while processing _as_parameter_")) {
  ^
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c:2062:19: warning: cast to 
pointer from integer of different size [-Wint-to-pointer-cast]
 if (Py_EnterRecursiveCall("while processing _as_parameter_")) {
   ^
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c:2067:62: error: argument 1 of 
‘__atomic_load’ must be a non-void pointer type
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c:2067:21: warning: cast to 
pointer from integer of different size [-Wint-to-pointer-cast]
 Py_LeaveRecursiveCall();
 ^
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c:2067:62: error: argument 1 of 
‘__atomic_load’ must be a non-void pointer type
/home/alex/Python-3.5.0/Modules/_ctypes/_ctypes.c:2067:139: warning: cast to 
pointer from integer of different size [-Wint-to-pointer-cast]

Failed to build these modules:
_ctypes   _decimal  _json  
_pickle

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com