[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Łukasz Langa]

Łukasz Langa  added the comment:

Given that:
- this is only surfacing in test_decimal on macOS;
- the real solution to modify malloc has potential performance implications; and
- silencing the spew isn't possible from within the test module

I elected to accept Jack's patch to notify macOS users running regression tests 
of the harmless malloc spew. I don't think there's anything else to do here.

Thanks all for your effort in investigating this!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[miss-islington]

miss-islington  added the comment:


New changeset a5d99632766b458b42f327e8bd0f82b0345c9a63 by Miss Islington (bot) 
in branch '3.10':
bpo-40928: notify users running test_decimal on macOS of malloc warnings 
(GH-26783)
https://github.com/python/cpython/commit/a5d99632766b458b42f327e8bd0f82b0345c9a63


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 693a661478036dc2c5b48e138dd09320a972c25a by Łukasz Langa in 
branch '3.9':
[3.9] bpo-40928: notify users running test_decimal on macOS of malloc warnings 
(GH-26783) (GH-27629)
https://github.com/python/cpython/commit/693a661478036dc2c5b48e138dd09320a972c25a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Łukasz Langa]

Change by Łukasz Langa :


--
pull_requests: +26123
pull_request: https://github.com/python/cpython/pull/27629

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 6.0 -> 7.0
pull_requests: +26122
pull_request: https://github.com/python/cpython/pull/27628

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 15d3c14df32a35ac69898a7852115722e30d7857 by Jack DeVries in 
branch 'main':
bpo-40928: notify users running test_decimal on macOS of malloc warnings 
(GH-26783)
https://github.com/python/cpython/commit/15d3c14df32a35ac69898a7852115722e30d7857


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Jack DeVries]

Jack DeVries  added the comment:

Alternatively, my latest PR implements what @ronaldoussoren suggested: capping 
OS X memory allocations at 4TB.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Jack DeVries]

Change by Jack DeVries :


--
pull_requests: +25371
pull_request: https://github.com/python/cpython/pull/26786

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Jack DeVries]

Jack DeVries  added the comment:

>From my pr, "This is a bandaid to help people save time debugging this 
>non-bug. I don't think this solution is very strong; I'm just hoping to move 
>the discussion forward on the bpo, and hoping for a hint on how to suppress 
>these warnings all together or come up with some other optimal solution.

I applied the function in test_decimal because that test consistently generates 
these malloc warnings on my system."

I'm still trying to find a true solution. Can anyone help to point me in a 
better direction?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Jack DeVries]

Change by Jack DeVries :


--
keywords: +patch
nosy: +jack__d
nosy_count: 4.0 -> 5.0
pull_requests: +25368
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/26783

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Ronald Oussoren]

Change by Ronald Oussoren :


--
components: +macOS
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Stefan Krah]

Stefan Krah  added the comment:

> The first warning line in the initial message of this bug report tries to 
> allocate 872 petabyte.

Yes, that's for systems like Linux where you never know what overallocation is 
going to permit and subsequently freeze the system.


Special casing malloc() for OS X/large allocations sounds like a good plan.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I've filed a report with Apple about this (FB7731971), even though I expect 
that this behaviour will not change. That's something we should have done years 
ago.

BTW. The link to malloc_print_configure that Victor shared seems to indicate 
that the default for MallocDebugReport should be "none", instead of "stderr". 
The code on opensource.apple.com conforms this 
(https://opensource.apple.com/source/libmalloc/libmalloc-283.60.1/src/malloc_printf.c.auto.html).
  Apparently Apple uses a slightly different code when building macOS :-(

Note that this annoying message is only printed when the proces runs out of 
address space, which requires allocating an insane amount of memory (the first 
warning line in the initial message of this bug report tries to allocate 872 
petabyte).  

If this is something we want to avoid its probably easier to switch to a 
allocator that won't avoids calling malloc(3) on large enough allocations (for 
example return NULL for any allocation attempt above 4TB, which is comfortably 
above the max amount of memory in a Mac Pro).  This theoretically limits memory 
allocations, but in practice the system will be unusable with far smaller 
allocations (assuming the allocated memory is actually used).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Stefan Krah]

Stefan Krah  added the comment:

And indeed, with the new decimal MAX_PREC feature, you should see
the misguided diagnostic as well on OS X:

>>> from decimal import *
>>> 
>>> c = getcontext()
>>> c.prec = MAX_PREC
>>> Decimal(4).sqrt()
Decimal('2')
>>> 


So it is real code, but extremely rarely used.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Stefan Krah]

Stefan Krah  added the comment:

But if malloc_print_configure() is not a public API, it should probably not be 
used.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Stefan Krah]

Stefan Krah  added the comment:

No, it does not cause real issues. Adding this feature would just suppress 
chatty diagnostics like this one, which are a bit unfriendly: 

>>> [0] * 1
python3(36633,0x110c08dc0) malloc: can't allocate region
:*** mach_vm_map(size=84096, flags: 100) failed (error code=3)
python3(36633,0x110c08dc0) malloc: *** set a breakpoint in malloc_error_break 
to debug
Traceback (most recent call last):
  File "", line 1, in 
MemoryError


So yes, users do occasionally see it outside the test suite.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I'm not sure if it is worthwhile to tweak CPython to work around this 
misfeature of libc on macOS (and likely iOS).  Is this something that causes 
issues in real code, or just in the test suite?  Looking at this particular 
test is seems to use a context that is not likely to occur in real code (due to 
using a lot of memory)


Note that malloc_print_configure is not a public API, using that can cause 
problems for folks that embed Python in apps that are shipped in the app store.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

> But is it really an use case to log any memory allocation failure? Python is 
> very likely to raise a MemoryError exception in this case. It's well defined, 
> no?

Yes, we are not trying to log the allocation failure, we are just trying to 
suppress a dubious error message that occurs specifically on MacOS.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[STINNER Victor]

STINNER Victor  added the comment:

> Would it be possible to wrap malloc_print_configure() 
> (https://github.com/PureDarwin/libmalloc/blob/e37056265821cd6e014ab911d9fe3b9d0da88e22/src/malloc_printf.c#L59)
>  in a context manager that we put in test.support? or to override 
> https://github.com/PureDarwin/libmalloc/blob/e37056265821cd6e014ab911d9fe3b9d0da88e22/src/malloc_printf.c#L36?

I don't see much benefit of having a feature which only works on macOS.

You can put a hook on memory allocations using PyMem_SetAllocator() to trigger 
any action you want when any memory allocation done by Python fails.

But is it really an use case to log any memory allocation failure? Python is 
very likely to raise a MemoryError exception in this case. It's well defined, 
no?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[STINNER Victor]

STINNER Victor  added the comment:

> Ok, I will try to do this in the coming days. Is adding this kind of super 
> ugly, super specific code to pymain_init() ok?

Maybe _PyPreConfig_Write() would be a better place.

There is a very little risk that memory allocation done before 
_PyPreConfig_Write() (i.e. in _PyPreConfig_Read()) fail with out of memory.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] OS X: malloc(): set default diagnostics to DEBUG_WRITE_ON_CRASH

[Stefan Krah]

Stefan Krah  added the comment:

Changing the title to get more input from others.

--
title: test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: 
can't allocate region" on MacOS -> OS X: malloc(): set default diagnostics to 
DEBUG_WRITE_ON_CRASH

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com