[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Stefan Krah]

Stefan Krah  added the comment:

I would probably put all that code in a separate function like
init_darwin_libc(), and call init_darwin_libc() in pymain_init().

The real fun will start when we figure out which OS X versions
support this feature. I hope we have a buildbot for each. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

Ok, I will try to do this in the coming days. Is adding this kind of super 
ugly, super specific code to pymain_init() ok?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Stefan Krah]

Stefan Krah  added the comment:

> What about overriding the default to DEBUG_WRITE_ON_CRASH if 
> MallocDebugReport is not set?


That sounds like a very good compromise!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

> I have two observations:
> 
> First, I think that the default DEBUG_WRITE_ALWAYS, which is the cause
> of this issue, is not C standard conforming -- The standard does not
> permit implementation defined diagnostics (global side effects!) on a
> completely normal allocation failure.

That makes sense

> So I would feel justified to make DEBUG_WRITE_ON_CRASH the default
> on Python startup, even if that means that we force that on extension
> modules.
> 
> That would also address annoyances like in msg327446. Speaking of
> which, do you also get that diagnostic in cases like these?
> 
> 
> >>> [0] * 1
> Traceback (most recent call last):
>   File "", line 1, in 
> MemoryError
>

It gives:

>>> [0] * 1
python3(36633,0x110c08dc0) malloc: can't allocate region
:*** mach_vm_map(size=84096, flags: 100) failed (error code=3)
python3(36633,0x110c08dc0) malloc: *** set a breakpoint in malloc_error_break 
to debug
Traceback (most recent call last):
  File "", line 1, in 
MemoryError

Not very nice.

> 
> FreeBSD has a similar reporting mechanism, but the default is not
> DEBUG_WRITE_ALWAYS.  I suspect DEBUG_WRITE_ALWAYS is there for the
> convenience of Apple developers dealing with user reports.
> 
> 
> Second, we *could* write such a context manager and it would probably
> be okay *for the test suite*, but in general changing a static variable
> that is not thread safe at runtime in a library should be avoided.
> 

What about overriding the default to DEBUG_WRITE_ON_CRASH if MallocDebugReport
is not set? This would be more standard compliant and still let the user 
override it if he whishes so.

> 
> I don't have OS X at all though. Ronald, what do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Stefan Krah]

Stefan Krah  added the comment:

I have two observations:

First, I think that the default DEBUG_WRITE_ALWAYS, which is the cause
of this issue, is not C standard conforming -- The standard does not
permit implementation defined diagnostics (global side effects!) on a
completely normal allocation failure.

So I would feel justified to make DEBUG_WRITE_ON_CRASH the default
on Python startup, even if that means that we force that on extension
modules.

That would also address annoyances like in msg327446. Speaking of
which, do you also get that diagnostic in cases like these?


>>> [0] * 1
Traceback (most recent call last):
  File "", line 1, in 
MemoryError


FreeBSD has a similar reporting mechanism, but the default is not
DEBUG_WRITE_ALWAYS.  I suspect DEBUG_WRITE_ALWAYS is there for the
convenience of Apple developers dealing with user reports.


Second, we *could* write such a context manager and it would probably
be okay *for the test suite*, but in general changing a static variable
that is not thread safe at runtime in a library should be avoided.


I don't have OS X at all though. Ronald, what do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

Would it be possible to wrap malloc_print_configure() 
(https://github.com/PureDarwin/libmalloc/blob/e37056265821cd6e014ab911d9fe3b9d0da88e22/src/malloc_printf.c#L59)
 in a context manager that we put in test.support? or to override 
https://github.com/PureDarwin/libmalloc/blob/e37056265821cd6e014ab911d9fe3b9d0da88e22/src/malloc_printf.c#L36?


This would let us change this behaviour locally without masking all other 
errors.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Stefan Krah]

Stefan Krah  added the comment:

Thanks for checking, it's a pity that os.putenv() does not work.

There's a global variable malloc_logger that is set from the
environment on startup:

https://opensource.apple.com/source/libmalloc/libmalloc-53.1.1/src/malloc.c


It seems to be always checked for NULL, but I can't find any
documentation that says it is guaranteed.

A nuclear option would be to set malloc_logger to NULL on Python
startup.  Or write a custom malloc_logger, which *does* seem to
be supported.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

I forgot to say that MallocLogFile=/dev/null did not work, that's why I tried 
MallocDebugReport.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

Thanks, I found "MallocDebugReport" in `man malloc` (I could have thought to 
look into there...) that seem to do the trick:

$ MallocDebugReport=none ./python -m test test_decimal -m 
test_maxcontext_exact_arith

0:00:00 load avg: 1.27 Run tests sequentially
0:00:00 load avg: 1.27 [1/1] test_decimal

== Tests result: SUCCESS ==

1 test OK.

Total duration: 694 ms
Tests result: SUCCESS


There is also MallocDebugReport=crash that will "write messages to standard 
error only for a condition that is about to cause a crash." and effectively 
suppress this one.


Setting it in the test with os.putenv() does not seem to work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Ned Deily]

Change by Ned Deily :


--
nosy: +ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Stefan Krah]

Stefan Krah  added the comment:

"squashing the remark" means suppressing the message from malloc().
I would really like it if the test could be executed by default though.


Looking at:

https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/malloc.3.html


Could you try what happens if you set this environment variable?

   export MallocLogFile=/dev/null


If it works, we could build a solution around that.






.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

> The annoying "error" looks the same as #5614, which was closed as "won't fix".

Yes, it's a similar message, test_io does not display it on Catalina anymore 
thought.

> Is this the only place in the test suite now? 

As far as I know yes.


> If it is, I'm okay with squashing the remark.

I'm not sure to what this means.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Stefan Krah]

Stefan Krah  added the comment:

The annoying "error" looks the same as #5614, which was closed as "won't fix".

Is this the only place in the test suite now? If it is, I'm okay with
squashing the remark.

--
nosy: +skrah
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40928] test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: can't allocate region" on MacOS

[Rémi Lapeyre]

New submission from Rémi Lapeyre :

Here's the result of "./python -m test test_decimal -m 
test_maxcontext_exact_arith":

0:00:00 load avg: 1.33 Run tests sequentially
0:00:00 load avg: 1.33 [1/1] test_decimal
python(7835,0x11a218dc0) malloc: can't allocate region
:*** mach_vm_map(size=842105263157895168, flags: 100) failed (error code=3)
python(7835,0x11a218dc0) malloc: *** set a breakpoint in malloc_error_break to 
debug
python(7835,0x11a218dc0) malloc: can't allocate region
:*** mach_vm_map(size=842105263157895168, flags: 100) failed (error code=3)
python(7835,0x11a218dc0) malloc: *** set a breakpoint in malloc_error_break to 
debug
python(7835,0x11a218dc0) malloc: can't allocate region
:*** mach_vm_map(size=421052631578947584, flags: 100) failed (error code=3)
python(7835,0x11a218dc0) malloc: *** set a breakpoint in malloc_error_break to 
debug
python(7835,0x11a218dc0) malloc: can't allocate region
:*** mach_vm_map(size=421052631578947584, flags: 100) failed (error code=3)
python(7835,0x11a218dc0) malloc: *** set a breakpoint in malloc_error_break to 
debug

== Tests result: SUCCESS ==

1 test OK.

Total duration: 553 ms
Tests result: SUCCESS




I spent quite a time to find where this error was coming from and it's actually 
not an error but a dubious message from OSX. Others will surely lose time on 
this too, I think the best way to handle it is to use bigmemtest() with a small 
value on MacOS so that it's only run when '-M' is given on this platform, but 
always on the others.

--
components: Tests
messages: 371107
nosy: remi.lapeyre
priority: normal
severity: normal
status: open
title: test_decimal.CWhitebox.test_maxcontext_exact_arith() shows "malloc: 
can't allocate region" on MacOS
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com