[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:

Well, I misunderstood this sentence then, so it's just for testing. :)

> Our customers should use it or use ulimit -d.


One will hit this issue also when following the MAX_PREC section
in the FAQ, but that is a rare case:

https://docs.python.org/3.10/library/decimal.html#decimal-faq

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[David Edelsohn]

David Edelsohn  added the comment:

> About PSALLOC=early , I confirm that it perfectly fixes the issue.

> I'm surprised, because it is unspeakably slow on this machine,

These statements are not contradictory.  No one is suggesting that Python 
always should run with PSALLOC=early.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:

> That's probably due to a very small size of the Paging Space of the AIX 
> machine you used for testing.

That is the case, the machine has 160GB of memory and 1GB of paging space. I 
guess it is configured specifically for not freezing.


> About PSALLOC=early , I confirm that it perfectly fixes the issue.

I'm surprised, because it is unspeakably slow on this machine,
even with the skips in place:

PSALLOC=early time ./python -m test -uall test_decimal

I hit Ctrl-C after 10min, so it takes even longer:

Real   622.11
User   11.63
System 350.12  (!)



The -bmaxdata approach has no speed penalty. Note that you can
use 10 petabytes for the value, it should still prevent this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Tony Reix]

Tony Reix  added the comment:

Hi Stefan,
In your message https://bugs.python.org/issue41540#msg375462 , you said:
 "However, instead of freezing the machine, the process gets a proper SIGKILL 
almost instantly."
That's probably due to a very small size of the Paging Space of the AIX machine 
you used for testing. With very small PS, the OS quickly reaches the step where 
PS and memory are full and it tries to kill possible culprits (but often 
killing innocent processes, like my bash shell). However, with a large PS (size 
of the Memory, or half), it takes some time for the OS to consume the PS, and, 
during this time (many seconds if not minutes), the OS looks like frozen and it 
takes many seconds or minutes for a "kill -9 PID" to take effect.

About -bmaxdata, I always used it for extending default memory of a 32bit 
process, but I never used it for reducing the possible memory of a 64bit 
process since some users may want to use python with hundreds of GigaBytes of 
memory. And the python executable used for tests is the same one that is 
delivered to users.

About PSALLOC=early , I confirm that it perfectly fixes the issue. So, we'll 
use it when testing Python.
Our customers should use it or use ulimit -d .
But using -bmaxdata for building python process in 64bit would reduce the 
possibilities of the python process.
In the future, we'll probably improve the compatibility with Linux so that this 
(rare) case no more appear.

BTW, on AIX, we have only 12 test cases failing out of about 32,471 test cases 
run in 64bit, with probably only 5 remaining serious failures. Both with GCC 
and XLC. Not bad. Less in 32bit. Now studying these few remaining issues and 
the still skipped tests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Change by Stefan Krah :


--
assignee:  -> skrah
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:


New changeset 28bf82661ac9dfaf1b2d0fd0ac98fc0b31cd95bb by Miss Islington (bot) 
in branch '3.9':
bpo-41540: AIX: skip test that is flaky with a default ulimit. (GH-21890) 
(#21893)
https://github.com/python/cpython/commit/28bf82661ac9dfaf1b2d0fd0ac98fc0b31cd95bb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:


New changeset 39dab24621122338d01c1219bb0acc46ba9c9956 by Stefan Krah in branch 
'master':
bpo-41540: AIX: skip test that is flaky with a default ulimit. (#21890)
https://github.com/python/cpython/commit/39dab24621122338d01c1219bb0acc46ba9c9956


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 4.0 -> 5.0
pull_requests: +21012
pull_request: https://github.com/python/cpython/pull/21893

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Change by Stefan Krah :


--
keywords: +patch
pull_requests: +21009
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/21890

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:

To recap for people who find this: The problem occurs because of AIX's
extreme over-allocation and is specific to the 64-bit build.

Workarounds:

  1) Something like  ulimit -d 800.

  2) xlc: LDFLAGS="-L/usr/lib64 -q64 -bmaxdata:0x8" or
 LDFLAGS="-L/usr/lib64 -q64 -Wl,-bmaxdata:0x8".

 The first version seems more natural for xlc.

  3) gcc: LDFLAGS="-L/usr/lib64 -Wl,-bmaxdata:0x8 -maix64"



PSALLOC=early works really well for the libmpdec tests but is extremely
slow with the Python interpreter. Also, setting the environment in the
tests does not work.  It looks like it needs to be set before main()
starts.



So I'll just skip that test on AIX. It is not that important, and
the libmpdec maxprec tests, which are way more thorough, all pass
with PSALLOC=early.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:

> -qmaxmem affects the compiler optimization.

I know, that's just from the Python README.AIX.  I didn't expect it to have any 
influence.


> -Wl,-bmaxdata:0x8 is a GCC command line option.

That is indeed surprising. Linking is prepared by a script:

$ ./Modules/makexp_aix Modules/python.exp . libpython3.10.a;  xlc -L/usr/lib64 
-q64 -Wl,-bmaxdata:0x8-Wl,-bE:Modules/python.exp -lld -o python 
Programs/python.o libpython3.10.a -lintl -ldl  -lpthread -lm   -lm

The xlc command runs without warnings or errors.



Without Wl,-bmaxdata:0x8


$ ./python -m test -uall test_decimal
0:00:00 Run tests sequentially
0:00:00 [1/1] test_decimal
Killed


With Wl,-bmaxdata:0x8
=

$ ./python -m test -uall test_decimal
0:00:00 Run tests sequentially
0:00:00 [1/1] test_decimal

== Tests result: SUCCESS ==

1 test OK.

Total duration: 17.3 sec
Tests result: SUCCESS


> On the other hand, -bmaxdata behaves differently in 32 bit mode and 64 bit 
> mode.  In 32 bit mode, it increases the heap size from the default 256MB.  In 
> 64 bit mode, it sets a guaranteed maximum size for the heap.

Yes, that's what I expected. The test only allocates that much memory
for 64-bit builds.  The workaround only needs to be enabled for 64-bit.

So a memory softlimit, same as e.g. djb uses for qmail with his
softlimit program, is exactly what I was looking for.


> I sort of think that the solution desired for the testcase is PSALLOC=early 
> to match traditional Unix/Linux malloc() behavior.

I can try that, but our test suite might complain about the environment
being modified.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[David Edelsohn]

David Edelsohn  added the comment:

AIX uses a "late" memory allocation scheme by default.  If the test wants to 
malloc(52631578947368422ULL) and intends it to fail, it should run with the AIX

$ export PSALLOC=early

environment variable.  More than all of the other maxdata changes.

Separate from all of this, you are configuring as 64 bit (-q64).  -qmaxmem 
affects the compiler optimization.

-Wl,-bmaxdata:0x8 is a GCC command line option.  "-Wl," passes the 
appended flag to the linker.  So somehow you're using GCC to invoke the linker, 
although building with XLC.  XLC knows about -bmaxdata directly.

On the other hand, -bmaxdata behaves differently in 32 bit mode and 64 bit 
mode.  In 32 bit mode, it increases the heap size from the default 256MB.  In 
64 bit mode, it sets a guaranteed maximum size for the heap.

So I think that -bmaxdata may be helping, but not for the reason that you 
believe.  -Wl,-bmaxdata:0x8000 may work, although I don't understand how 
that correctly interacts with XLC.  If you truly are running in 64 bit mode, 
then -bmaxdata has an effect like PSALLOC.

I sort of think that the solution desired for the testcase is PSALLOC=early to 
match traditional Unix/Linux malloc() behavior.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:

Thank you, David!
  
Now that I can test on AIX, I can confirm that the data limit is the
culprit:

libmpdec deliberately calls malloc(52631578947368422ULL) in the
maxprec tests, which is supposed to fail, but succeeds.

However, instead of freezing the machine, the process gets a proper
SIGKILL almost instantly.



As I suggested earlier, using -bmaxdata prevents this from happening
and the test passes.

./configure CC=xlc AR="ar -X64" CFLAGS="-q64 -qmaxmem=7 -qlanglvl=extc99 
-qcpluscmt -qkeyword=inline -qalias=ansi -qthreaded -D_THREAD_SAFE 
-D__VACPP_MULTI__" LDFLAGS="-L/usr/lib64 -q64 -Wl,-bmaxdata:0x8"


I have not figured out a similar gcc option yet, but only searched for 5 min.



The question now is: Since this is expected behavior on AIX, and xlc (and 
probably
gcc as well) have limit command line switches, do we need to disable the test?



I tried to set rlimit in the test, which works on Linux but is broken on AIX:

test test_decimal failed -- Traceback (most recent call last):
  File "/home/skrah/cpython/Lib/test/test_decimal.py", line 5684, in 
test_maxcontext_exact_arith
resource.setrlimit(resource.RLIMIT_DATA, (800, hardlimit))
OSError: [Errno 14] Bad address


So disabling the test would be the best option, unless we want to educate users
about -bmaxdata.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[David Edelsohn]

David Edelsohn  added the comment:

AIX systems at OSUOSL have been part of the GNU Compile Farm for a decade. It 
also is the system on which I have been running the Python Buildbot.  Any 
Compile Farm user has access to AIX.

https://cfarm.tetaneutral.net/users/new/

Also, IBM is in the process of creating a dedicated VM at OSUOSL for the Python 
Buildbot to support more builders. We probably can provide access to Python 
core members to log in to that system as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:

> So, this issue with v3.10 (master) appeared to me as a regression.

I understand that from your point of view it appears as a regression.

However, quoting the C standard, 7.20.3 Memory management functions:

"The pointer returned points to the start (lowest byte
address) of the allocated space. If the space cannot be allocated, a null 
pointer is
returned."


So, for a system that is not currently officially supported, I don't
consider it my problem if AIX thinks the space can be allocated.

But as I said, I can disable that test on AIX.  If I get AIX access,
I can look at this more.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:

> Core developers have full access to AIX system for the asking.  Back to you, 
> Stefan.

That sounds great. Can we contact you directly, or have I missed an earlier 
announcement from someone else giving out AIX access?

Or are you working on it and I misunderstand the idiomatic English? :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Tony Reix]

Tony Reix  added the comment:

I forgot to say that this behavior was not present in stable version 3.8.5 . 
Sorry.

On 2 machines AIX 7.2, testing Python 3.8.5 with:
+ cd /opt/freeware/src/packages/BUILD/Python-3.8.5
+ ulimit -d unlimited
+ ulimit -m unlimited
+ ulimit -s unlimited
+ export 
LIBPATH=/opt/freeware/src/packages/BUILD/Python-3.8.5/64bit:/usr/lib64:/usr/lib:/opt/lib
+ export PYTHONPATH=/opt/freeware/src/packages/BUILD/Python-3.8.5/64bit/Modules
+ ./python Lib/test/regrtest.py -v test_decimal
...
gave:

507 tests in 227 items.
507 passed and 0 failed.
Test passed.

So, this issue with v3.10 (master) appeared to me as a regression. However, 
after hours debugging the issue, I forgot to say it in this defect, sorry.

(Previously, I was using limits for -d -m and -s : max 4GB. However, that 
appeared to be an issue when running tests with Python test option -M12Gb, 
which requires up and maybe more than 12GB of my 16GB memory machine in order 
to be able to run a large part of the Python Big Memory tests. And thus I 
unlimited these 3 resources, with no problem at all with version 3.8.5 .)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[David Edelsohn]

David Edelsohn  added the comment:

Core developers have full access to AIX system for the asking.  Back to you, 
Stefan.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:

The test (size > (size_t)PY_SSIZE_T_MAX)) has nothing to do with it. Within 
Python, most sizes are ssize_t, so a value larger than SSIZE_MAX is suspicious.


AIX is an unsupported platform.

Realistically, if people want AIX to be supported, someone has to give core 
devs full ssh access to an AIX system.

Perhaps I'll just skip this test on AIX.

--
components: +Extension Modules -C API
nosy: +David.Edelsohn
priority: normal -> low
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Tony Reix]

Tony Reix  added the comment:

Is it a 64bit AIX ? Yes, AIX is 64bit by default and only since ages, but it 
manages 32bit applications as well as 64bit applications.

The experiments were done with 64bit Python executables on both AIX and Linux.

The AIX machine has 16GB Memory and 16GB Paging Space.

The Linux Fdora32/x86_64 machine has 16GB Memory and 8269820 Paging Space 
(swapon -s).

Yes, I agree that the behavior of AIX malloc() under "ulimit -d unlimited" 
is... surprising. And the manual of malloc() does not talk about this.

Anyway, does the test:   if (size > (size_t)PY_SSIZE_T_MAX)  was aimed to 
prevent calling malloc() with such a huge size? If yes, that does not work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:

Also, perhaps build Python with -bmaxdata?

https://www.enterprisedb.com/edb-docs/d/postgresql/reference/manual/11.1/installation-platform-notes.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Stefan Krah]

Stefan Krah  added the comment:

We need more information.

Is this 64-bit AIX?

How much physical memory does the machine have?

Linux also has over-allocation and the default for ulimit is unlimited.
But it does not attempt to over-allocate such an outrageous amount of
memory.

Neither do FreeBSD, Windows, etc.

--
nosy: +skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Sanket Rathi]

Change by Sanket Rathi :


--
nosy: +sanket

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Tony Reix]

Tony Reix  added the comment:

Hi Pablo,
I'm only surprised that the maximum size generated in the test is always lower 
than the PY_SSIZE_T_MAX. And this appears both on AIX and on Linux, which both 
compute the same values.

On AIX, it appears (I've just discovered this now) that malloc() does not 
ALWAYS check that there is enough memory to allocate before starting to claim 
memory (and thus paging space). This appears when Data Segment size is 
unlimited.

On Linux/Fedora, I had no limit too. But it behaves differently and malloc() 
always checks that the size is correct.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Tony Reix]

Tony Reix  added the comment:

Some more explanations.

On AIX, the memory is controlled by the ulimit command.
"Global memory" comprises the physical memory and the paging space, associated 
with the Data Segment.

By default, both Memory and Data Segment are limited:
# ulimit -a
data seg size   (kbytes, -d) 131072
max memory size (kbytes, -m) 32768
...

However, it is possible to remove the limit, like:
# ulimit -d unlimited

Now, when the "data seg size" is limited, the malloc() routine checks if enough 
memory/paging-space are available, and it immediately returns a NULL pointer.

But, when the "data seg size" is unlimited, the malloc() routine first tries to 
allocate and quickly consumes the paging space, which is much slower than 
acquiring memory since it consumes disk space. And it nearly hangs the OS. 
Thus, in that case, it does NOT check if enough memory of data segments are 
available. Bad.

So, this issue appears on AIX only if we have:
# ulimit -d unlimited

Anyway, the test:
if (size > (size_t)PY_SSIZE_T_MAX)
in:
Objects/obmalloc.c: PyMem_RawMalloc()
seems weird to me since the max of size is always lower than PY_SSIZE_T_MAX .

--
nosy:  -facundobatista, mark.dickinson, pablogsal, rhettinger, skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Unfortunately, I do not understand why you conclude that the problem is in 
PyMem_RawMalloc. That code seems correct. Could you provide evidence that the 
value of PY_SSIZE_T_MAX is miscalculated in AIX? Alternatively, could you 
elaborate on what makes you believe that the problem is in PyMem_RawMalloc?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Could be a duplicate/related to the problem described in 
https://bugs.python.org/issue39576

--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +facundobatista, mark.dickinson, rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Tony Reix]

New submission from Tony Reix :

Python master of 2020/08/11

Test test_maxcontext_exact_arith (test.test_decimal.CWhitebox) checks that 
Python correctly handles a case where an object of size 421052631578947376 is 
created.

maxcontext = Context(prec=C.MAX_PREC, Emin=C.MIN_EMIN, Emax=C.MAX_EMAX)

Both on Linux and AIX, we have:
Context(prec=99,
rounding=ROUND_HALF_EVEN,
Emin=-99,
 Emax=99, capitals=1, clamp=0, flags=[], 
traps=[InvalidOperation, DivisionByZero, Overflow])

The test appears in:
  Lib/test/test_decimal.py
5665 def test_maxcontext_exact_arith(self):
and the issue (on AIX) exactly appears at:
self.assertEqual(Decimal(4) / 2, 2)

The issue is due to code in: Objects/obmalloc.c :
void *
PyMem_RawMalloc(size_t size)
{
/*
 * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
 * Most python internals blindly use a signed Py_ssize_t to track
 * things without checking for overflows or negatives.
 * As size_t is unsigned, checking for size < 0 is not required.
 */
if (size > (size_t)PY_SSIZE_T_MAX)
return NULL;
return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);

Both on Fedora/x86_64 and AIX, we have:
 size:421052631578947376
 PY_SSIZE_T_MAX: 9223372036854775807
thus: size < PY_SSIZE_T_MAX and _PyMem_Raw.malloc() is called.

However, on Linux, the malloc() returns a NULL pointer in that case, and then 
Python handles this and correctly runs the test.
However, on AIX, the malloc() tries to allocate the requested memory, and the 
OS gets stucked till the Python process is killed by the OS.

Either size is too small, or PY_SSIZE_T_MAX is not correctly computed:
./Include/pyport.h :
  /* Largest positive value of type Py_ssize_t. */
  #define PY_SSIZE_T_MAX ((Py_ssize_t)(((size_t)-1)>>1))

Anyway, the following code added in PyMem_RawMalloc() before the call to 
_PyMem_Raw.malloc() , which in turns calls malloc() :
if (size == 421052631578947376)
{
printf("TONY: 421052631578947376: --> PY_SSIZE_T_MAX: %ld \n", 
PY_SSIZE_T_MAX);
return NULL;
}
does fix the issue on AIX.
However, it is simply a way to show where the issue can be fixed.
Another solution (fix size < PY_SSIZE_T_MAX) is needed.

--
components: C API
messages: 375302
nosy: T.Rex
priority: normal
severity: normal
status: open
title: Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX
type: crash
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com