[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: Thanks to David Edelsohn I have AIX access now. The issue reported by Pablo is the same as #41540, for a summary see msg375480. It is a trivial issue that requires that ulimits are in place due to the fact that AIX over-allocates petabytes even when the physical

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: The 3.9 change (see #40874) works successfully on all buildbots, except for the 32-bit xlc bot which should use c99_r. Additionally, it has been tested with the latest gcc/clang/icc/cl.exe, static analyzers and clang-tidy. It survives brute force allocation

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset 0f5a28f834bdac2da8a04597dc0fc5b71e50da9d by Stefan Krah in branch '3.8': [3.8] Revert bpo-39576: Prevent memory error for overly optimistic precisions (GH-20747) https://github.com/python/cpython/commit/0f5a28f834bdac2da8a04597dc0fc5b71e50da9d

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset 22faf6ad3bcc0ae478a9a3e2d8e35888d88d6ce8 by Stefan Krah in branch '3.7': [3.7] Revert bpo-39576: Prevent memory error for overly optimistic precisions (GH-20748) https://github.com/python/cpython/commit/22faf6ad3bcc0ae478a9a3e2d8e35888d88d6ce8

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- pull_requests: +19953 pull_request: https://github.com/python/cpython/pull/20748 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- pull_requests: +19952 pull_request: https://github.com/python/cpython/pull/20747 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset 9bd891920a5186b7d02281ea9966225efa0ceba1 by Stefan Krah in branch '3.7': [3.7] Revert bpo-39576: docs: set context for decimal arbitrary precision arithmetic (GH-20746)

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset 32c1fb07e6f2ded90e5dd24d4b46b7aa7a795d2e by Stefan Krah in branch '3.8': [3.8] Revert bpo-39576: docs: set context for decimal arbitrary precision arithmetic (GH-20745)

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- pull_requests: +19951 pull_request: https://github.com/python/cpython/pull/20746 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- pull_requests: +19950 pull_request: https://github.com/python/cpython/pull/20745 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset c0b79450bc9e93105799528151c48d25af8240a3 by Stefan Krah in branch '3.7': [3.7] Revert bpo-39576: Clarify the word size for the 32-bit build. (GH-20744) https://github.com/python/cpython/commit/c0b79450bc9e93105799528151c48d25af8240a3 --

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset 706de4e5a4b21880c67f6b90e3a2147a258d6fc5 by Stefan Krah in branch '3.8': [3.8] Revert bpo-39576: Clarify the word size for the 32-bit build. (GH-20743) https://github.com/python/cpython/commit/706de4e5a4b21880c67f6b90e3a2147a258d6fc5 --

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- pull_requests: +19949 pull_request: https://github.com/python/cpython/pull/20744 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- pull_requests: +19948 pull_request: https://github.com/python/cpython/pull/20743 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: I think I'm going to revert this for 3.7 and 3.8 -- not because of xlc (it is almost certainly a compiler or missing flag error), but because coordination with the Linux distributions is a mess, see #40874. I really want the system libmpdec to be the same as

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: > Traceback, does it work --without-pymalloc? No, it does also not work with `--without-pymalloc?`. > Has anything emerged xlc-wise? Not for now, I keep investigating and I may try to contact IBM about this, but at this stage, I am getting more

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: Has anything emerged xlc-wise? Traceback, does it work --without-pymalloc? I have the feeling that (in many OSS projects) the more complex xlc issues never get resolved after the initial report. So I'm contemplating to do the same as

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: The lower MAX_PREC for 32-bit could be the reason. On the other hand, historically, suncc and xlc have had a lot of problems with the 64-bit build. The winner is suncc, which seriously miscompiled libmpdec without a whole litany of flags:

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by STINNER Victor : -- nosy: +vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: I will try to investigate that but it think this is because without OBJECT_MODE=64 the value of decimal.MAX_PREC is much lower (42500 instead 99) so whatever is happening downstream with the higher does not happen with the lower

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: Since xlc has elementary bugs like https://github.com/openssl/openssl/issues/10624 it may be worth checking out if this is an optimizer bug with -q64. -- ___ Python tracker

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: After some debugging, I discovered that the new test succeeds if I configure and compile CPython without 'OBJECT_MODE=64' set. -- ___ Python tracker

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: Sorry, I'm reacting like Granlund now and close. These discussions lead nowhere. -- status: open -> closed ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: Just to be clear: I am saying that the *exact* regression is manifested with the new test because that is the behavioural difference that I experienced and how I found this problem. > but just ran the 3.7.6 tests that do not include any tests with

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: > In other words, contrary to your earlier dismissal, you did NOT run _decimal on AIX with MAX_PREC but just ran the 3.7.6 tests that do not include any tests with MAX_PREC. I did, and it raises MemoryError: ❯ uname -a AIX 1 7 powerpc 00CCAD974C00 AIX

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: "The **exact** regression is that I could run the test suite without any crash or freeze on this AIX system with 3.7.6 and I cannot in 3.7.7. At the very least this is due to the fact that there is a new test that crashes/hangs." In other words, contrary to

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: > Acknowledge that gcc builds work on the AIX buildbots (a fact that has been > entirely ignored so far). I do acknowledge that. I am not saying that I am sure there is a bug in the code. For what I know at this point it may be something with xlc,

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: It is also instructive how Granlund views xlc: https://gmplib.org/list-archives/gmp-bugs/2010-November/002119.html -- ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: A focused issue report would include at least the following: 1) Acknowledge that gcc builds work on the AIX buildbots (a fact that has been entirely ignored so far). 2) State the exact regression: msg364373 (which was also ignored, are you using

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- assignee: -> skrah keywords: -3.7regression, patch ___ Python tracker ___ ___ Python-bugs-list mailing

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: > I mean, it's not okay to complain about a regression and then mention xlc about 10 mails later. How is this related? Or is not ok to report a behaviour change in a stable release even if the platform is "best-effort"? A regression by definition is

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- assignee: -> skrah ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: BTW, if you are compiling with xlc and there"s no xlc buildbot, perhaps a company-that-shall-not-be-named should provide one. I mean, it's not okay to complain about a regression and then mention xlc about 10 mails later. --

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: Btw, this is AIX 7.1.0.0 with xlc in case that is relevant. -- ___ Python tracker ___ ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: These flags worked for xlc when snakebite was still up: ./configure CC=xlc_r AR="ar -X64" CFLAGS="-q64 -qmaxmem=7" LDFLAGS="-q64" -qmaxmem was always finicky, I remember segfaults too (AIX problem, not libmpdec problem). --

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: > If you are fortunate enough to have access to an AIX system, I guess you have to find out why POWER6 AIX 3.8 and PPC64 AIX 3.8 apparently work on https://buildbot.python.org/ but your 3.7 does not. I am working on trying to debug where the problem

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: Hi Michael, in case you have built 3.7.7 on AIX, have you observed any problems with test_decimal? -- nosy: +Michael.Felt ___ Python tracker

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: > Well, I just did and I can confirm that reverting the 3.7 backport fixes the > problem. If you are fortunate enough to have access to an AIX system, I guess you have to find out why POWER6 AIX 3.8 and PPC64 AIX 3.8 apparently work on

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: > Thanks for the lecture. This is an esoteric case between bugfix and feature that only occurs with very large context precisions. Nobody is lecturing anyone. I am just asking why this was backported. > If Bloomberg isn't happy with _decimal, they can

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: > This code has introduced a regression in AIX in Python 3.7.7 Also this is a rather bold statement since probably no one has ever run _decimal on AIX with MAX_PREC. -- ___ Python tracker

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: > This looks like a "new feature/improvement". Why was this code backported to > a stable version? Thanks for the lecture. This is an esoteric case between bugfix and feature that only occurs with very large context precisions. If Bloomberg isn't happy with

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Pablo Galindo Salgado : -- nosy: +ned.deily ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: This looks like a "new feature/improvement". Why was this code backported to a stable version? -- ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Pablo Galindo Salgado added the comment: This code has introduced a regression in AIX in Python 3.7.7 as the new "test_maxcontext_exact_arith" test hangs indefinitely or just segfaults. -- keywords: +3.7regression nosy: +pablogsal resolution: fixed -> status: closed -> open

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset c6ecd9c14081a787959e13df33e250102a658154 by Miss Islington (bot) in branch '3.8': bpo-39576: Clarify the word size for the 32-bit build. (GH-18616) (#18618) https://github.com/python/cpython/commit/c6ecd9c14081a787959e13df33e250102a658154

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset 24c570bbb82a7cb70576c253a73390accfa7ed78 by Miss Islington (bot) in branch '3.7': bpo-39576: Clarify the word size for the 32-bit build. (GH-18616) (#18617) https://github.com/python/cpython/commit/24c570bbb82a7cb70576c253a73390accfa7ed78

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by miss-islington : -- pull_requests: +17983 pull_request: https://github.com/python/cpython/pull/18618 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset b76518d43fb82ed9e5d27025d18c90a23d525c90 by Stefan Krah in branch 'master': bpo-39576: Clarify the word size for the 32-bit build. (#18616) https://github.com/python/cpython/commit/b76518d43fb82ed9e5d27025d18c90a23d525c90 --

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by miss-islington : -- nosy: +miss-islington nosy_count: 5.0 -> 6.0 pull_requests: +17982 pull_request: https://github.com/python/cpython/pull/18617 ___ Python tracker

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- pull_requests: +17981 pull_request: https://github.com/python/cpython/pull/18616 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Tim Peters added the comment: Thanks, Stefan! This turned out better than I expected ;-) I'm closing the report, under the likely assumption that nobody cares enough about obscure inelegancies in the Python version to bother. -- status: open -> closed

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- stage: patch review -> resolved ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: libmpdec and the docs are done, the question remains what to do with decimal.py. It has the same behavior, but I don't think users are running decimal.py with very large precisions. Anyway, unassigning myself in case anyone else wants to work on a patch.

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset d6965ff026f35498e554bc964ef2be8f4d80eb7f by Miss Islington (bot) in branch '3.8': bpo-39576: docs: set context for decimal arbitrary precision arithmetic (GH-18594) (#18597)

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset 00e45877e33d32bb61aa13a2033e3bba370bda4d by Miss Islington (bot) in branch '3.7': bpo-39576: docs: set context for decimal arbitrary precision arithmetic (GH-18594) (#18596)

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by miss-islington : -- pull_requests: +17964 pull_request: https://github.com/python/cpython/pull/18597 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by miss-islington : -- pull_requests: +17963 pull_request: https://github.com/python/cpython/pull/18596 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset a025d4ca99fb4c652465368e0b4eb03cf4b316b9 by Stefan Krah in branch 'master': bpo-39576: docs: set context for decimal arbitrary precision arithmetic (#18594) https://github.com/python/cpython/commit/a025d4ca99fb4c652465368e0b4eb03cf4b316b9

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: Updated docs: https://github.com/python/cpython/pull/18594 The PR uses some of Tim's suggestions while also explaining how to calculate the amount of memory used in a single large decimal. Hopefully it isn't too much information. --

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- pull_requests: +17961 pull_request: https://github.com/python/cpython/pull/18594 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: "So non-integer powers are left out" in isolation would indeed be wrong, but actual sentence is unambiguously qualified with: "... since _decimal has no notion of exact non-integer powers yet.", which clearly states that exact non-integer powers exist and

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Vedran Čačić added the comment: Well, it all depends on how do you intend to read it. To me, the closing quote and "So non-integer powers are left out" after it suggested that the non-integer powers being left out is somehow a consequence of the above paragraph. When in fact instead of "so"

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: Vedran, msg362365 is meant to say: "This PR implements $SOMEWHAT_MATHEMATICAL_SPEC except for inexact power." Had I put the caveat inside the statement as well, the message would have been: "This PR implements

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Vedran Čačić added the comment: Hm... is "exact result" a technical term that's defined somewhere? Because to me it seems that this > "If there exists an exact result at a lower precision, this result should also be returned at MAX_PREC (without MemoryError)". is a mathematical statement,

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset b6271025c640c228505dc9f194362a0c2ab81c61 by Miss Islington (bot) in branch '3.8': bpo-39576: Prevent memory error for overly optimistic precisions (GH-18581) (#18584)

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset c6f95543b4832c3f0170179da39bcf99b40a7aa8 by Miss Islington (bot) in branch '3.7': bpo-39576: Prevent memory error for overly optimistic precisions (GH-18581) (#18585)

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by miss-islington : -- pull_requests: +17954 pull_request: https://github.com/python/cpython/pull/18584 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by miss-islington : -- pull_requests: +17955 pull_request: https://github.com/python/cpython/pull/18585 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: New changeset 90930e65455f60216f09d175586139242dbba260 by Stefan Krah in branch 'master': bpo-39576: Prevent memory error for overly optimistic precisions (GH-18581) https://github.com/python/cpython/commit/90930e65455f60216f09d175586139242dbba260 --

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: BTW, this PR implements the invariant: "If there exists an exact result at a lower precision, this result should also be returned at MAX_PREC (without MemoryError)". So non-integer powers are left out, since _decimal has no notion of exact non-integer powers

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: Fortunately libmpdec raises MemoryError almost instantaneously, so the PR retries the affected operations with estimated upper bounds for exact results without slowing down the common case. The docs still need updating because people will still wonder why 1 /

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Stefan Krah : -- keywords: +patch pull_requests: +17951 stage: -> patch review pull_request: https://github.com/python/cpython/pull/18581 ___ Python tracker ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Tim Peters added the comment: Formulas based on physical RAM probably work well on Linux, but not so well on Windows: Windows has no "overcommit". Whether a virtual memory request succeeds on Windows depends on how much RAM (+ swap space, if any) has already been requested across all

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: > This isn't purely academic. The `decimal` docs, at the end: Yes, that is a good point. I think for libmpdec I'll just do a trial divmod if prec > BIGNUM_THRESHOLD. > Perhaps the only thing to be done is to add words to the part of the docs >

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Tim Peters added the comment: Vedran, as Mark said, the result is defined to have no trailing zeroes. In general the module strives to return results "as if" infinite precision were used internally, not to actually _use_ infinite precision internally ;-) Given the same setup, e.g., >>> i

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: The feature would be nice to have; however, if you choose the precision to match the amount of available RAM things work (I have 8GB here, one word in the coefficient has 19 digits for the 4 bit version): >>> from decimal import * >>> c = getcontext() >>>

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: MAX_PREC is chosen so that 5*MAX_PREC does not overflow 32-bit or 64-bit signed integers. This eliminates many overflow checks for the exponent. Updating the exponent is (perhaps surprisingly) quite performance sensitive, that's why the 32-bit build does not

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Stefan Krah added the comment: With _pydecimal the memory also grows very slowly (I didn't have the patience to wait for MemoryError). I'm pretty sure decNumber also does the same, it's just easier to implement and does not slow down division for small numbers. libmpdec always favors

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Vedran Čačić added the comment: Yeah, I should have said "represent" instead of "write". :-) -- ___ Python tracker ___ ___

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Mark Dickinson added the comment: Agreed that this seems surprising. @Vedran: Trailing zeros in a Decimal object are significant, so `Decimal("2.0")` and `Decimal("2.00")` are different (equal, but different). The rules about the "ideal exponent" of the result of an arithmetic operation

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Vedran Čačić added the comment: > Of course the result is exactly 2. Which I have enough RAM to hold ;-) You might think so, but if you write it as 2.00...0 with >>> decimal.MAX_PREC 99 zeros, I think you're overestimating your RAM capacity. :-P Now, what is the exact

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

Change by Batuhan Taskaya : -- nosy: +BTaskaya ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39576] Surprising MemoryError in `decimal` with MAX_PREC

New submission from Tim Peters : Here under Python 3.8.1 on 64-bit Windows: >>> import decimal >>> c = decimal.getcontext() >>> c.prec = decimal.MAX_PREC >>> i = decimal.Decimal(4) >>> i / 2 Traceback (most recent call last): File "", line 1, in MemoryError Of course the result is exactly