[issue37168] Decimal divisions sometimes 10x or 100x too large

Change by Facundo Batista : -- nosy: -facundobatista ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37168] Decimal divisions sometimes 10x or 100x too large

Change by Steven D'Aprano : -- nosy: +steven.daprano ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37168] Decimal divisions sometimes 10x or 100x too large

Phil Frost added the comment: Not yet. I had a hell of a time getting gdb to follow gunicorn forks so it would watch the workers but not also every other thing using popen or os.system(). And the lldb package on Alpine doesn't seem to work. So we're currently in the process of testing a

[issue37168] Decimal divisions sometimes 10x or 100x too large

Mark Dickinson added the comment: @Phil: Did you ever get to the bottom of this? -- ___ Python tracker ___ ___ Python-bugs-list

[issue37168] Decimal divisions sometimes 10x or 100x too large

STINNER Victor added the comment: Can you try to reproduce your workflow with Python compiled in debug mode (./configure --with-pydebug)? With Python 3.8, you can now use a debug Python without having to recompile your C extensions :-) -- ___

[issue37168] Decimal divisions sometimes 10x or 100x too large

Phil Frost added the comment: I'm afraid that won't be of much use since the object is statically allocated. Besides, the trouble isn't finding what created the object, but what mutated an int object which should be immutable. -- ___ Python

[issue37168] Decimal divisions sometimes 10x or 100x too large

STINNER Victor added the comment: On Python 3, you can use tracemalloc.get_object_traceback() to find where an object was allocated ;-) -- ___ Python tracker ___

[issue37168] Decimal divisions sometimes 10x or 100x too large

Phil Frost added the comment: Probably too many C extensions to feasibly audit them all. Also we can't rule out something busted in Alpine. So I'm going to set a watchpoint on small_ints[1]->ob_ival with a little gdb script to dump core right when it changes; hopefully inspecting that

[issue37168] Decimal divisions sometimes 10x or 100x too large

Mark Dickinson added the comment: This smells a bit as though someone's doing something like: myobject = PyInt_FromLong(some_long); and then assuming that they hold the only reference to `myobject`, so that it's safe to mutate in place. --

[issue37168] Decimal divisions sometimes 10x or 100x too large

Mark Dickinson added the comment: Interesting indeed. What 3rd party extension are installed in your Python environment? Is it possible that one of them is messing with the CPython internals (perhaps in an overeager attempt to optimize)? Any uses of ctypes? --

[issue37168] Decimal divisions sometimes 10x or 100x too large

Phil Frost added the comment: So this is uminteresting (gdb) p small_ints[0] $2 = (PyIntObject *) 0x558ce65df528 (gdb) p *small_ints[0] $3 = {ob_refcnt = 18, ob_type = 0x7fe019b694c0 , ob_ival = -5} (gdb) p *small_ints[1] $4 = {ob_refcnt = 65, ob_type = 0x7fe019b694c0 , ob_ival = -3}

[issue37168] Decimal divisions sometimes 10x or 100x too large

Change by Jeffrey Kintscher : -- nosy: +Jeffrey.Kintscher ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37168] Decimal divisions sometimes 10x or 100x too large

Stefan Krah added the comment: Yes, I'd try Debian first, then potential libc issues can also be eliminated in a single experiment. -- ___ Python tracker ___

[issue37168] Decimal divisions sometimes 10x or 100x too large

Phil Frost added the comment: > Alpine Linux apparently uses musl. Is that well supported and tested? Heh. Not really. Switching to a Debian container is probably just about as much work as building Python from source. Would that be preferable? --

[issue37168] Decimal divisions sometimes 10x or 100x too large

Stefan Krah added the comment: There are lots of ideas still. :) As I said, on a Linux other than RedHat or Debian or such I'd first try to blame the shipped Python, then the libc (I had a glibc memmove issue once with similar inexplicable results). Alpine Linux apparently uses musl. Is

[issue37168] Decimal divisions sometimes 10x or 100x too large

Phil Frost added the comment: skrah: Yes, that's correct. Since I can only produce this bug in production it will take me some days to build and validate a source build. But absent any better ideas, I will try. tim.peters: I've observed this bug across hundreds of EC2 hosts, in dozens of

[issue37168] Decimal divisions sometimes 10x or 100x too large

Change by STINNER Victor : -- nosy: +vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37168] Decimal divisions sometimes 10x or 100x too large

Tim Peters added the comment: Also basic: run hardware CPU and memory stress diagnostics, and/or try running the same thing on a different machine. Hardware isn't infallible, and can fail in nearly arbitrary ways. For example, perhaps a smidgen of silicon has gone flaky, so that one time

[issue37168] Decimal divisions sometimes 10x or 100x too large

Stefan Krah added the comment: Thank you for the detailed report. So, in an attempt to summarize, this is the crucial passage: >>> ideal_exp = -1 >>> exp = -29 >>> coeff = 15 >>> division_counter = 0 >>> while exp < ideal_exp and coeff % 10 == 0: ... coeff //=

[issue37168] Decimal divisions sometimes 10x or 100x too large

Phil Frost added the comment: I should note that while we do observe this issue with Decimal, my troubleshooting so far indicates the logic in decimal.py is fine. It seems instead the instructions in decimal.py aren't being correctly executed. I'm not sure if the problem is limited to only

[issue37168] Decimal divisions sometimes 10x or 100x too large

Change by Zachary Ware : -- nosy: +facundobatista, mark.dickinson, rhettinger, skrah ___ Python tracker ___ ___ Python-bugs-list

[issue37168] Decimal divisions sometimes 10x or 100x too large

New submission from Phil Frost : We've observed instances of Decimal divisions being incorrect a few times per day in a web application serving millions of requests daily. I've been unable to reproduce the issue but have been investigating core dumps which suggest either some issue in the C