[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by STINNER Victor : -- resolution: wont fix -> fixed versions: -Python 2.7, Python 3.6, Python 3.9 ___ Python tracker ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Gregory P. Smith added the comment: New changeset 8766cb74e186d3820db0a855ccd780d6d84461f7 by Gregory P. Smith (Inada Naoki) in branch '3.7': [3.7] bpo-27987: align PyGC_Head to alignof(long double) (GH-13335) (GH-13581)

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Inada Naoki : -- pull_requests: +13488 pull_request: https://github.com/python/cpython/pull/13581 ___ Python tracker ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Gregory P. Smith : -- versions: -Python 2.7, Python 3.8, Python 3.9 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Gregory P. Smith added the comment: if someone runs into an actual need for this on 32-bit builds, please provide details and feel free to reopen the issue. closing as i don't believe there is any more for us to do. -- resolution: -> wont fix stage: needs patch -> resolved status:

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

miss-islington added the comment: New changeset 1b85f4ec45a5d63188ee3866bd55eb29fdec7fbf by Miss Islington (bot) in branch '3.7': bpo-27987: pymalloc: align by 16bytes on 64bit platform (GH-12850) https://github.com/python/cpython/commit/1b85f4ec45a5d63188ee3866bd55eb29fdec7fbf --

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Antoine Pitrou added the comment: Here is what I've found for (32-bit) ARM: - "long double" is 8 bytes long, so it's probably the same as "double" (see http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dai0274b/index.html) - the standard alignment for "double" is 8 bytes (see

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Gregory P. Smith : -- pull_requests: +13478 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/13569 ___ Python tracker ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Antoine Pitrou added the comment: And of course, someone who has this issue can at worse recompile Python without pymalloc. -- ___ Python tracker ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Gregory P. Smith added the comment: I'm not marking this bug as "Fixed" as the original complaint about obmalloc'd structs with a long double not being aligned is still going to be true on 32-bit platforms for 2.7 - 3.7. We've merely increased the obmalloc alignment to 16-bytes on 64-bit

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Gregory P. Smith added the comment: commit reverted in https://github.com/python/cpython/commit/2156fec1f7a8f9972e90cdbaf404e3fd9eaccb35 -- ___ Python tracker ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Gregory P. Smith added the comment: long double was changed to double seven years ago to avoid a different kind of undefined behavior... https://github.com/python/cpython/commit/e348c8d154cf6342c79d627ebfe89dfe9de23817#diff-fb41bdaf12f733cf6ab8a82677d03adc We are going in circles here.

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Inada Naoki added the comment: New changeset ea2b76bdc5f97f49701213d105b8ec2387ea2fa5 by Inada Naoki in branch '3.7': bpo-27987: align PyGC_Head to alignof(long double) (GH-13335) https://github.com/python/cpython/commit/ea2b76bdc5f97f49701213d105b8ec2387ea2fa5 --

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Inada Naoki : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Antoine Pitrou added the comment: Neil, I don't see the point of having this discussion here. -- ___ Python tracker ___ ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Neil Schemenauer added the comment: We now have a concrete use case. ;-) My idea was that we can introduce a new, CPython internal API that aligns on 8-byte boundaries (or takes alignment as a parameter). The API would be a stop-gap measure. We can use the API to reduce the overhead for

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

STINNER Victor added the comment: > It seems to me that we don't need to have the PyObject structure containing a > Python float to be 16-byte aligned. If so, could we introduce a new obmalloc > API that returns memory with 8-byte alignment, for use by objects that know > they don't

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Neil Schemenauer added the comment: > sys.getsizeof(3.14) is 24. And it becomes 32 byte in 16byte aligned > pymalloc. (+33%) I've been doing some reading and trying to understand this issue. My understanding is that malloc() needs to return pointers that are 16-byte aligned on AMD64 but,

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

STINNER Victor added the comment: New changeset f24a9f3bf42709fb97b954b6dd6f90853967712e by Victor Stinner in branch '2.7': bpo-27987: pymalloc: align by 16bytes on 64bit platform (GH-12850) (GH-13319) https://github.com/python/cpython/commit/f24a9f3bf42709fb97b954b6dd6f90853967712e

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Inada Naoki : -- pull_requests: +13248 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Inada Naoki : -- pull_requests: +13247 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by STINNER Victor : -- pull_requests: +13230 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by miss-islington : -- pull_requests: +13229 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Inada Naoki added the comment: yes. sys.getsizeof(3.14) is 24. And it becomes 32 byte in 16byte aligned pymalloc. (+33%) FYI, jemalloc has 8, 16, 32 size classes, but no 24 too. http://jemalloc.net/jemalloc.3.html#size_classes -- ___ Python

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Stefan Krah added the comment: +16% for float seems pretty high though. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Inada Naoki added the comment: New changeset f0be4bbb9b3cee876249c23f2ae6f38f43fa7495 by Inada Naoki in branch 'master': bpo-27987: pymalloc: align by 16bytes on 64bit platform (GH-12850) https://github.com/python/cpython/commit/f0be4bbb9b3cee876249c23f2ae6f38f43fa7495 --

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Inada Naoki added the comment: $ ./python -m perf compare_to master.json align16.json -G --min-speed=1 Slower (13): - pickle_list: 4.40 us +- 0.03 us -> 4.59 us +- 0.04 us: 1.04x slower (+4%) - xml_etree_iterparse: 129 ms +- 2 ms -> 133 ms +- 2 ms: 1.04x slower (+4%) - regex_dna: 201 ms +- 2

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Inada Naoki added the comment: $ ./python -m perf compare_to master-mem.json align16-mem.json -G --min-speed=2 Slower (30): - float: 20.6 MB +- 12.6 kB -> 23.8 MB +- 30.3 kB: 1.16x slower (+16%) - mako: 14.3 MB +- 760.5 kB -> 15.1 MB +- 54.1 kB: 1.06x slower (+6%) - xml_etree_iterparse: 11.1

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

STINNER Victor added the comment: > I wonder if we can add -fmax-type-align=8 for extension types... No, we cannot: it's a temporary fix. The flag causes compilation error if it's added to old version of clang or to a C compiler different than clang. > Any ideas about reduce impact for

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Inada Naoki added the comment: > In 3.8, the union used to ensure alignment on a C double is gone. Note that two uintptr_t is aligned 16bytes on 64bit platforms and 8bytes on 32bit platforms. Python 3.7 is worse than 3.8. It used "double dummy" to align by 8 bytes, not 16 bytes. We should

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Inada Naoki : -- pull_requests: +12775 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Florian Weimer added the comment: Minor correction: glibc malloc follows ABI on x86-64 and always returns a 16-byte-aligned pointer, independently of allocation size. However, other mallocs (such as jemalloc and tcmalloc) may return pointers with less alignment for allocation sizes less

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

STINNER Victor added the comment: The x86-64 ABI requires that memory allocated on the heap is aligned to 16 bytes. On x86-64, glibc malloc(size) aligns on 16 bytes for size >= 16, otherwise align to 8 bytes. So the glibc doesn't respect exactly the ABI. I understand that a compiler will

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Inada Naoki added the comment: I had not noticed bpo-33374 changed Python 2.7. I don't know why it caused segv only for Python 2.7. -- ___ Python tracker ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

STINNER Victor added the comment: PyGC_Head structure size depends on the Python version, sizes of 64-bit: * 2.7: 32 bytes * 3.6, 3.7: 24 bytes * 3.8 (master): 16 bytes bpo-36618 "clang expects memory aligned on 16 bytes, but pymalloc aligns to 8 bytes" should be even worse on 3.7: 24 is

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Inada Naoki added the comment: Now PyGC_Head is 16byte on 64bit platform. Maybe, should we just change obmalloc in Python 3.8? How about 32bit platforms? What can we do for Python 3.7 and 2.7? -- ___ Python tracker

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Inada Naoki : -- nosy: +inada.naoki ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Gregory P. Smith : -- versions: +Python 3.8, Python 3.9 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

STINNER Victor added the comment: While this issue looked purely theorical to me 3 years ago, it is now very concrete: bpo-36618 "clang expects memory aligned on 16 bytes, but pymalloc aligns to 8 bytes". -- ___ Python tracker

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Tom Grigg : -- nosy: +tgrigg ___ Python tracker ___ ___ Python-bugs-list

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Florian Weimer added the comment: This bug causes miscompilation of Python 2.7 by GCC 8 on x86-64 (with no sanitizers enabled, just compiler optimization). I think this is a fairly conservative way for papering over the issue:

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Florian Weimer : -- versions: +Python 2.7 ___ Python tracker ___ ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Gregory P. Smith : -- nosy: +gregory.p.smith, twouters ___ Python tracker ___ ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Neil Schemenauer added the comment: FYI, this would seem to be an incentive to get my "bitmaps for small GC objects" idea implemented. I.e. https://mail.python.org/pipermail/python-dev/2017-September/149307.html If implemented, the extra size of the PyGC_Head would

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Benjamin Peterson added the comment: On Wed, Nov 8, 2017, at 06:14, STINNER Victor wrote: > > STINNER Victor added the comment: > > alignment.patch: +long double dummy; /* force worst-case alignment > */ > > Would it be possible to use

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

STINNER Victor added the comment: alignment.patch: +long double dummy; /* force worst-case alignment */ Would it be possible to use max_align_t mentioned by Stefan, at least when this type is available? What is the impact of the patch on objects size?

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Benjamin Peterson added the comment: Yes, we could memcpy things around to obtain the desired alignment. It would be nicer to have a builtin solution, though. -- ___ Python tracker

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Stefan Krah added the comment: Since we have "#define PYMEM_FUNCS PYOBJ_FUNCS", I think extensions that use PyMem_Malloc() also won't get the glibc max_align_t alignment. But guess technically they should. -- nosy: +skrah ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Antoine Pitrou added the comment: > Can we use memcpy()? Hmm, perhaps. Do you want to try it out (and measure any performance degradation)? -- ___ Python tracker

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

STINNER Victor added the comment: Change by Antoine Pitrou: "versions: -Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6" The undefined behaviour exists and should be fixed in Python 2.7 and 3.6, no? Can we use memcpy()? --

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

STINNER Victor added the comment: What matters when a Python object is allocated? The start of the PyObject structure, or the start of the PyGC_Head structure? Would it be possible to align the PyObject start? The simplest option is to store data which needs to be

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Change by Antoine Pitrou : -- versions: -Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6 ___ Python tracker ___

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Antoine Pitrou added the comment: > My suggestion would be to pass alignof(type) into the allocator via macro. Do you mean using some new PyMem_ function? Or as as new tp_ field on the type declaration? -- ___ Python tracker

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Benjamin Peterson added the comment: My suggestion would be to pass alignof(type) into the allocator via macro. Then the allocator could at least assert it's providing good enough alignment if not provide the correct alignment. I believe 16-byte alignment is special

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

STINNER Victor added the comment: > ubsan complains about unaligned access when structs include "long double". An > example error: > runtime error: member access within misaligned address 0x7f77dbba9798 for > type 'struct CDataObject', which requires 16 byte

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

Antoine Pitrou added the comment: What do we do if at some point a C type requires a larger alignment (for example a vector type readable using AVX512 instructions)? -- nosy: +pitrou ___ Python tracker

[issue27987] obmalloc's 8-byte alignment causes undefined behavior

New submission from Benjamin Peterson: ubsan complains about unaligned access when structs include "long double". An example error: runtime error: member access within misaligned address 0x7f77dbba9798 for type 'struct CDataObject', which requires 16 byte alignment This is because (on x86