[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: Well, I misunderstood this sentence then, so it's just for testing. :) > Our customers should use it or use ulimit -d. One will hit this issue also when following the MAX_PREC section in the FAQ, but that is a rare case:

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

David Edelsohn added the comment: > About PSALLOC=early , I confirm that it perfectly fixes the issue. > I'm surprised, because it is unspeakably slow on this machine, These statements are not contradictory. No one is suggesting that Python always should run with PSALLOC=early. --

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: > That's probably due to a very small size of the Paging Space of the AIX > machine you used for testing. That is the case, the machine has 160GB of memory and 1GB of paging space. I guess it is configured specifically for not freezing. > About PSALLOC=early

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Tony Reix added the comment: Hi Stefan, In your message https://bugs.python.org/issue41540#msg375462 , you said: "However, instead of freezing the machine, the process gets a proper SIGKILL almost instantly." That's probably due to a very small size of the Paging Space of the AIX machine

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Change by Stefan Krah : -- assignee: -> skrah resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: New changeset 28bf82661ac9dfaf1b2d0fd0ac98fc0b31cd95bb by Miss Islington (bot) in branch '3.9': bpo-41540: AIX: skip test that is flaky with a default ulimit. (GH-21890) (#21893) https://github.com/python/cpython/commit/28bf82661ac9dfaf1b2d0fd0ac98fc0b31cd95bb

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: New changeset 39dab24621122338d01c1219bb0acc46ba9c9956 by Stefan Krah in branch 'master': bpo-41540: AIX: skip test that is flaky with a default ulimit. (#21890) https://github.com/python/cpython/commit/39dab24621122338d01c1219bb0acc46ba9c9956 --

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Change by miss-islington : -- nosy: +miss-islington nosy_count: 4.0 -> 5.0 pull_requests: +21012 pull_request: https://github.com/python/cpython/pull/21893 ___ Python tracker

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Change by Stefan Krah : -- keywords: +patch pull_requests: +21009 stage: -> patch review pull_request: https://github.com/python/cpython/pull/21890 ___ Python tracker ___

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: To recap for people who find this: The problem occurs because of AIX's extreme over-allocation and is specific to the 64-bit build. Workarounds: 1) Something like ulimit -d 800. 2) xlc: LDFLAGS="-L/usr/lib64 -q64 -bmaxdata:0x8" or

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: > -qmaxmem affects the compiler optimization. I know, that's just from the Python README.AIX. I didn't expect it to have any influence. > -Wl,-bmaxdata:0x8 is a GCC command line option. That is indeed surprising. Linking is prepared by a script: $

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

David Edelsohn added the comment: AIX uses a "late" memory allocation scheme by default. If the test wants to malloc(52631578947368422ULL) and intends it to fail, it should run with the AIX $ export PSALLOC=early environment variable. More than all of the other maxdata changes. Separate

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: Thank you, David! Now that I can test on AIX, I can confirm that the data limit is the culprit: libmpdec deliberately calls malloc(52631578947368422ULL) in the maxprec tests, which is supposed to fail, but succeeds. However, instead of freezing the machine,

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

David Edelsohn added the comment: AIX systems at OSUOSL have been part of the GNU Compile Farm for a decade. It also is the system on which I have been running the Python Buildbot. Any Compile Farm user has access to AIX. https://cfarm.tetaneutral.net/users/new/ Also, IBM is in the

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: > So, this issue with v3.10 (master) appeared to me as a regression. I understand that from your point of view it appears as a regression. However, quoting the C standard, 7.20.3 Memory management functions: "The pointer returned points to the start (lowest

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: > Core developers have full access to AIX system for the asking. Back to you, > Stefan. That sounds great. Can we contact you directly, or have I missed an earlier announcement from someone else giving out AIX access? Or are you working on it and I

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Tony Reix added the comment: I forgot to say that this behavior was not present in stable version 3.8.5 . Sorry. On 2 machines AIX 7.2, testing Python 3.8.5 with: + cd /opt/freeware/src/packages/BUILD/Python-3.8.5 + ulimit -d unlimited + ulimit -m unlimited + ulimit -s unlimited + export

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

David Edelsohn added the comment: Core developers have full access to AIX system for the asking. Back to you, Stefan. -- ___ Python tracker ___

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: The test (size > (size_t)PY_SSIZE_T_MAX)) has nothing to do with it. Within Python, most sizes are ssize_t, so a value larger than SSIZE_MAX is suspicious. AIX is an unsupported platform. Realistically, if people want AIX to be supported, someone has to give

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Tony Reix added the comment: Is it a 64bit AIX ? Yes, AIX is 64bit by default and only since ages, but it manages 32bit applications as well as 64bit applications. The experiments were done with 64bit Python executables on both AIX and Linux. The AIX machine has 16GB Memory and 16GB Paging

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: Also, perhaps build Python with -bmaxdata? https://www.enterprisedb.com/edb-docs/d/postgresql/reference/manual/11.1/installation-platform-notes.html -- ___ Python tracker

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Stefan Krah added the comment: We need more information. Is this 64-bit AIX? How much physical memory does the machine have? Linux also has over-allocation and the default for ulimit is unlimited. But it does not attempt to over-allocate such an outrageous amount of memory. Neither do

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Change by Sanket Rathi : -- nosy: +sanket ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Tony Reix added the comment: Hi Pablo, I'm only surprised that the maximum size generated in the test is always lower than the PY_SSIZE_T_MAX. And this appears both on AIX and on Linux, which both compute the same values. On AIX, it appears (I've just discovered this now) that malloc() does

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Tony Reix added the comment: Some more explanations. On AIX, the memory is controlled by the ulimit command. "Global memory" comprises the physical memory and the paging space, associated with the Data Segment. By default, both Memory and Data Segment are limited: # ulimit -a data seg size

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Pablo Galindo Salgado added the comment: Unfortunately, I do not understand why you conclude that the problem is in PyMem_RawMalloc. That code seems correct. Could you provide evidence that the value of PY_SSIZE_T_MAX is miscalculated in AIX? Alternatively, could you elaborate on what makes

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Pablo Galindo Salgado added the comment: Could be a duplicate/related to the problem described in https://bugs.python.org/issue39576 -- nosy: +pablogsal ___ Python tracker

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Change by STINNER Victor : -- nosy: +facundobatista, mark.dickinson, rhettinger ___ Python tracker ___ ___ Python-bugs-list mailing

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

Change by STINNER Victor : -- nosy: +skrah ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

New submission from Tony Reix : Python master of 2020/08/11 Test test_maxcontext_exact_arith (test.test_decimal.CWhitebox) checks that Python correctly handles a case where an object of size 421052631578947376 is created. maxcontext = Context(prec=C.MAX_PREC, Emin=C.MIN_EMIN,