[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

STINNER Victor added the comment: With my additional GH-30988 fix, msg411075 example no longer leaks :-) --- Loop #1: 2 refs Loop #2: 2 refs Loop #3: 2 refs ... Loop #98: 2 refs Loop #99: 2 refs Loop #100: 2 refs --- I close the issue. Thanks Christian and Kumar for the fix! --

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

STINNER Victor added the comment: New changeset 5a9e423473bf2c4eb32a0982e8d73420875db1da by Kumar Aditya in branch 'main': bpo-46449: deepfreeze get_code() now returns strong ref (GH-30987) https://github.com/python/cpython/commit/5a9e423473bf2c4eb32a0982e8d73420875db1da --

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Change by Kumar Aditya : -- pull_requests: +29166 pull_request: https://github.com/python/cpython/pull/30987 ___ Python tracker ___

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Change by Kumar Aditya : -- pull_requests: +29164 pull_request: https://github.com/python/cpython/pull/30985 ___ Python tracker ___

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Change by Christian Heimes : -- pull_requests: +29163 pull_request: https://github.com/python/cpython/pull/30984 ___ Python tracker ___

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Kumar Aditya added the comment: Christian's solution seems better to me so I'll close my PR. Christian would you like to create a PR for it ? -- ___ Python tracker ___

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Kumar Aditya added the comment: I created a PR https://github.com/python/cpython/pull/30976 which adjusts _Py_RefTotal and refcnt of immortal codeobjects to account for Py_INCREF/Py_DECREF on codeobjects. With that patch refcnt is 8 and increases by 8 with each initialization of Python.

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Change by Kumar Aditya : -- keywords: +patch pull_requests: +29155 stage: -> patch review pull_request: https://github.com/python/cpython/pull/30976 ___ Python tracker ___

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Christian Heimes added the comment: The problem is in PyImport_ImportFrozenModuleObject -> unmarshal_frozen_code() -> frozen_info.get_code() -> _Py_get_importlib__bootstrap_external_toplevel() call chain. PyImport_ImportFrozenModuleObject() expects unmarshal_frozen_code() to return a

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

STINNER Victor added the comment: The bpo-46476 added _Py_Deepfreeze_Fini() and _PyStaticCode_Dealloc() functions: commit c7f810b34d91a5c2fbe0a8385562015d2dd961f2. If we need to ajust _Py_RefTotal manually, *maybe* it can be done there? I don't understand well how static/immortal code

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Guido van Rossum added the comment: I tried to make the 'FROZEN' variable in freeze_modules.py empty, but it has a bunch of places where this is unexpected. Maybe someone can fix that? -- ___ Python tracker

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

STINNER Victor added the comment: I don't want to change the default. Keeping fast startup time is a nice goal! I'm asking to make it configurable for my own development workflow: build Python as fast as possible. It is easy to hack Makefile.am.in and Python/frozen.c to freeze less modules.

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Christian Heimes added the comment: Do you have an alternative suggestion how to build Python with minimal set of required deepfrozen modules or without any deepfrozen modules at all? A minimal set is not only helpful for Victor's use case. I would also like to use it in WebAssembly builds

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Kumar Aditya added the comment: > @Kumar do you want to tackle this? I don't like this approach as it is opposite to what we did to reduce the size of deep-frozen modules to merge them in one file and this approach requires to split it again. --

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Guido van Rossum added the comment: @Kumar do you want to tackle this? -- nosy: +kumaraditya303 ___ Python tracker ___ ___

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Christian Heimes added the comment: If you modify the freeze_modules script to generate code like https://github.com/python/cpython/compare/main...tiran:split_frozen?expand=1 , then I can add a build option to compile Python with only required frozen modules. --

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Guido van Rossum added the comment: > Is there a way to disable deepfreeze when building Python? It looks like this isn't easy, sorry. :-( Adding Christian Heimes in case he has a suggestion. -- nosy: +christian.heimes ___ Python tracker

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

STINNER Victor added the comment: See also bpo-46476: "Not all memory allocated by _Py_Quicken() is released at Python exit". -- ___ Python tracker ___

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

STINNER Victor added the comment: Is there a way to disable deepfreeze when building Python? It makes the Python build way slower. For example, a full build (after "make clean") of Python 3.10 takes 14.9 seconds on my laptop, whereas Python 3.11 takes 24.6 seconds (1.6x slower). It makes my

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Eric Snow added the comment: > the deep-frozen objects also reference the small ints directly, as well as > the singleton for b"". > Is this even safe across Py_Finalize()/Py_Initialize()? If not, we'll need to > roll that back as well. The small ints and the empty bytes object each have

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Guido van Rossum added the comment: I was hoping @eric.snow could tell me. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

STINNER Victor added the comment: > This reminds me, since https://github.com/python/cpython/pull/30715 (which I > merged yesterday) the deep-frozen objects also reference the small ints > directly, as well as the singleton for b"". Is this even safe across > Py_Finalize()/Py_Initialize()?

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

STINNER Victor added the comment: > Hm, the deep-frozen objects are statically initialized with a very large > refcount that isn't accounted for (they are intended to be immortal). It > seems that Py_Finalize() somehow decrefs those objects. I guess this means we > need some kind of flag

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Guido van Rossum added the comment: Hm, the deep-frozen objects are statically initialized with a very large refcount that isn't accounted for (they are intended to be immortal). It seems that Py_Finalize() somehow decrefs those objects. I guess this means we need some kind of flag

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

Change by STINNER Victor : -- nosy: +eric.snow, gvanrossum ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

New submission from STINNER Victor : Using the C program below, I see that _Py_RefTotal is decreasing at each iteration: --- #include int main(int argc, char *argv[]) { for (int i=1; i <= 100; i++) { Py_SetProgramName(L"./_testembed"); Py_Initialize();