[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[STINNER Victor]

STINNER Victor  added the comment:

With my additional GH-30988 fix, msg411075 example no longer leaks :-)
---
Loop #1: 2 refs
Loop #2: 2 refs
Loop #3: 2 refs
...
Loop #98: 2 refs
Loop #99: 2 refs
Loop #100: 2 refs
---

I close the issue.

Thanks Christian and Kumar for the fix!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 5a9e423473bf2c4eb32a0982e8d73420875db1da by Kumar Aditya in 
branch 'main':
bpo-46449: deepfreeze get_code() now returns strong ref  (GH-30987)
https://github.com/python/cpython/commit/5a9e423473bf2c4eb32a0982e8d73420875db1da


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Kumar Aditya]

Change by Kumar Aditya :


--
pull_requests: +29166
pull_request: https://github.com/python/cpython/pull/30987

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Kumar Aditya]

Change by Kumar Aditya :


--
pull_requests: +29164
pull_request: https://github.com/python/cpython/pull/30985

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +29163
pull_request: https://github.com/python/cpython/pull/30984

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Kumar Aditya]

Kumar Aditya  added the comment:

Christian's solution seems better to me so I'll close my PR. Christian would 
you like to create a PR for it ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Kumar Aditya]

Kumar Aditya  added the comment:

I created a PR https://github.com/python/cpython/pull/30976 which adjusts 
_Py_RefTotal and refcnt of immortal codeobjects to account for 
Py_INCREF/Py_DECREF on codeobjects. With that patch refcnt is 8 and increases 
by 8 with each initialization of Python.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Kumar Aditya]

Change by Kumar Aditya :


--
keywords: +patch
pull_requests: +29155
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/30976

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Christian Heimes]

Christian Heimes  added the comment:

The problem is in PyImport_ImportFrozenModuleObject -> unmarshal_frozen_code() 
-> frozen_info.get_code() -> _Py_get_importlib__bootstrap_external_toplevel() 
call chain.

PyImport_ImportFrozenModuleObject() expects unmarshal_frozen_code() to return a 
strong reference to the code object. However a frozen_info struct with a 
get_code() function returns a borrowed reference from deepfreeze.c's toplevel 
code object.

# --- test.c
#include 
int main(int argc, char *argv[])
{
for (int i=1; i <= 100; i++) {
Py_SetProgramName(L"./_testembed");
Py_Initialize();
Py_Finalize();
printf("Loop #%d: %zd refs, bootstrap refs: %zd\n", i, _Py_RefTotal, 
Py_REFCNT(_Py_get_importlib__bootstrap_external_toplevel()));
}
}
# ---

$ gcc -IInclude -I. -o test test.c libpython3.11d.a -lm && ./test
Loop #1: -3 refs, bootstrap refs: 8
Loop #2: -8 refs, bootstrap refs: 7
Loop #3: -13 refs, bootstrap refs: 6
Loop #4: -18 refs, bootstrap refs: 5
Loop #5: -23 refs, bootstrap refs: 4
Loop #6: -28 refs, bootstrap refs: 3
Loop #7: -33 refs, bootstrap refs: 2
Loop #8: -38 refs, bootstrap refs: 1
Loop #9: -43 refs, bootstrap refs: 0
Loop #10: -48 refs, bootstrap refs: 99989


After I changed unmarshal_frozen_code() to "return Py_NewRef(code);", the 
reference count of the frozen bootstrap module stays stable, but total refcount 
increases over time:

Loop #1: 10 refs, bootstrap refs: 9
Loop #2: 18 refs, bootstrap refs: 9
Loop #3: 26 refs, bootstrap refs: 9
Loop #4: 34 refs, bootstrap refs: 9
Loop #5: 42 refs, bootstrap refs: 9
Loop #6: 50 refs, bootstrap refs: 9
Loop #7: 58 refs, bootstrap refs: 9
Loop #8: 66 refs, bootstrap refs: 9
Loop #9: 74 refs, bootstrap refs: 9

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[STINNER Victor]

STINNER Victor  added the comment:

The bpo-46476 added _Py_Deepfreeze_Fini() and _PyStaticCode_Dealloc() 
functions: commit c7f810b34d91a5c2fbe0a8385562015d2dd961f2. If we need to ajust 
_Py_RefTotal manually, *maybe* it can be done there?

I don't understand well how static/immortal code object lead to negative 
_Py_RefTotal. For me, Py_INCREF() and Py_DECREF() should still be used on these 
objects, no?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Guido van Rossum]

Guido van Rossum  added the comment:

I tried to make the 'FROZEN' variable in freeze_modules.py empty, but it has a 
bunch of places where this is unexpected. Maybe someone can fix that?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[STINNER Victor]

STINNER Victor  added the comment:

I don't want to change the default. Keeping fast startup time is a nice goal!

I'm asking to make it configurable for my own development workflow: build 
Python as fast as possible.

It is easy to hack Makefile.am.in and Python/frozen.c to freeze less modules. 
If you want, I can try to work on a patch to make it configurable. Maybe some 
people want to freeze... *more* modules, rather than less ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Christian Heimes]

Christian Heimes  added the comment:

Do you have an alternative suggestion how to build Python with minimal set of 
required deepfrozen modules or without any deepfrozen modules at all? A minimal 
set is not only helpful for Victor's use case. I would also like to use it in 
WebAssembly builds to reduce the overall file size.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Kumar Aditya]

Kumar Aditya  added the comment:

> @Kumar do you want to tackle this?

I don't like this approach as it is opposite to what we did to reduce the size 
of deep-frozen modules to merge them in one file and this approach requires to 
split it again.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Guido van Rossum]

Guido van Rossum  added the comment:

@Kumar do you want to tackle this?

--
nosy: +kumaraditya303

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Christian Heimes]

Christian Heimes  added the comment:

If you modify the freeze_modules script to generate code like 
https://github.com/python/cpython/compare/main...tiran:split_frozen?expand=1 , 
then I can add a build option to compile Python with only required frozen 
modules.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Guido van Rossum]

Guido van Rossum  added the comment:

> Is there a way to disable deepfreeze when building Python?

It looks like this isn't easy, sorry. :-( Adding Christian Heimes in case he 
has a suggestion.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-46476: "Not all memory allocated by _Py_Quicken() is released at 
Python exit".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[STINNER Victor]

STINNER Victor  added the comment:

Is there a way to disable deepfreeze when building Python?

It makes the Python build way slower. For example, a full build (after "make 
clean") of Python 3.10 takes 14.9 seconds on my laptop, whereas Python 3.11 
takes 24.6 seconds (1.6x slower). It makes my workflow (trial-and-error based 
;-)) less efficient.

Moreover, I would like to disable it to investigate why _Py_RefTotal is now 
negative at Python exit:
https://bugs.python.org/issue46417#msg411307

Note: I pushed many changes in bpo-46417 to clear static types and a few 
"static" objects at Python exit (in Py_Finalize()).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Eric Snow]

Eric Snow  added the comment:

> the deep-frozen objects also reference the small ints directly, as well as 
> the singleton for b"".
> Is this even safe across Py_Finalize()/Py_Initialize()? If not, we'll need to 
> roll that back as well.

The small ints and the empty bytes object each have "immortal" refcounts too 
(9, just like you did in deepfreeze).  So they would cause a similar 
behavior to what Victor reported.  Otherwise I wouldn't expect any problems 
across Py_Finalize()/Py_Initialize().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Guido van Rossum]

Guido van Rossum  added the comment:

I was hoping @eric.snow could tell me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[STINNER Victor]

STINNER Victor  added the comment:

> This reminds me, since https://github.com/python/cpython/pull/30715 (which I 
> merged yesterday) the deep-frozen objects also reference the small ints 
> directly, as well as the singleton for b"". Is this even safe across 
> Py_Finalize()/Py_Initialize()? If not, we'll need to roll that back as well.

I don't know.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[STINNER Victor]

STINNER Victor  added the comment:

> Hm, the deep-frozen objects are statically initialized with a very large 
> refcount that isn't accounted for (they are intended to be immortal). It 
> seems that Py_Finalize() somehow decrefs those objects. I guess this means we 
> need some kind of flag indicating certain objects are immortal (Eric has 
> proposed several schemes), then we could just mark these objects as immortal.

The problem is only _Py_RefTotal. Maybe frozen_only_do_patchups() should 
increment _Py_RefTotal when Python it build with Py_DEBUG macro defined, so it 
can be safely decremented in Py_Finalize()?

Adding a flag in PyObject/PyTypeObject and modifying Py_DECREF() sounds more 
controversial. I suggest to do that later ;-) (I am not convinced that it's the 
best approach.) I would suggest to write a PEP for immortal objects.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[Guido van Rossum]

Guido van Rossum  added the comment:

Hm, the deep-frozen objects are statically initialized with a very large 
refcount that isn't accounted for (they are intended to be immortal). It seems 
that Py_Finalize() somehow decrefs those objects. I guess this means we need 
some kind of flag indicating certain objects are immortal (Eric has proposed 
several schemes), then we could just mark these objects as immortal.

This reminds me, since https://github.com/python/cpython/pull/30715 (which I 
merged yesterday) the deep-frozen objects also reference the small ints 
directly, as well as the singleton for b"". Is this even safe across 
Py_Finalize()/Py_Initialize()? If not, we'll need to roll that back as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +eric.snow, gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)

[STINNER Victor]

New submission from STINNER Victor :

Using the C program below, I see that _Py_RefTotal is decreasing at each 
iteration:
---
#include 
int main(int argc, char *argv[])
{
for (int i=1; i <= 100; i++) {
Py_SetProgramName(L"./_testembed");
Py_Initialize();
Py_Finalize();
printf("Loop #%d: %zd refs\n", i, _Py_RefTotal);
}
}
---

Example of output:
---
...
Loop #96: 9557 refs
Loop #97: 9544 refs
Loop #98: 9531 refs
Loop #99: 9518 refs
Loop #100: 9505 refs
---

It seems to be a regression caused by this change:

commit 1cbaa505d007e11c4a1f0d2073d72b6c02c7147c
Author: Guido van Rossum 
Date:   Wed Nov 10 18:01:53 2021 -0800

bpo-45696: Deep-freeze selected modules (GH-29118)

This gains 10% or more in startup time for `python -c pass` on UNIX-ish 
systems.

The Makefile.pre.in generating code builds on Eric's work for bpo-45020, 
but the .c file generator is new.

Windows version TBD.


Before the change, _Py_RefTotal was stable: 
---
...
Loop #97: 10805 refs
Loop #98: 10805 refs
Loop #99: 10805 refs
Loop #100: 10805 refs
---

I found this issue while working on bpo-46417 which is related to bpo-1635741.

--
components: Interpreter Core
messages: 411075
nosy: vstinner
priority: normal
severity: normal
status: open
title: Deep-freezed modules create inconsistency in sys.gettotalrefcount() 
(_Py_Reftotal)
versions: Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com