[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

STINNER Victor  added the comment:

I still consider that these functions must be moved to the internal C API. But 
I failed to find time to design a *public* C API for that, or help projects 
using these functions to avoid it. I prefer to close the isseu for now.

--
resolution:  -> rejected
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

STINNER Victor  added the comment:

See also "Removal of _Py_ForgetReference from public header in 3.9 issue" 
thread on python-dev list:
https://mail.python.org/archives/list/python-...@python.org/thread/CQYVR7TZZITURBZKVWIEOBGF343GI52W/#CQYVR7TZZITURBZKVWIEOBGF343GI52W

And "Re: [Python-Dev] Removal of _Py_ForgetReference from public header in 3.9 
issue" thread on capi-sig list:
https://mail.python.org/archives/list/capi-...@python.org/thread/4EOCN7P4HI56GQ74FY3TMIKDBIPGKL2G/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 15edaecd97a3f8e82895046462342d8ddd8b9f1b by Victor Stinner in 
branch 'master':
bpo-40989: Fix compiler warning in winreg.c (GH-21722)
https://github.com/python/cpython/commit/15edaecd97a3f8e82895046462342d8ddd8b9f1b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +20865
pull_request: https://github.com/python/cpython/pull/21722

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[Stefan Behnel]

Stefan Behnel  added the comment:

I looked into the freelists a bit more and (as always) it's not quite that 
simple. Calling _Py_NewReference() allows keeping the "ob_type" pointer in 
place, whereas PyObject_INIT() requires a blank object in order to set the type 
and properly ref-count heap types.

I think what that means is that there are at least two different cases for 
freelists: those that only keep the bare object memory alive (and can also 
support subtypes of the same size), and those that keep the (cleared) object 
alive, including its type. For the first, PyObject_INIT() works. For the 
latter, _Py_NewReference() seems the right helper function.

The advantage of keeping the object as it is is the much simpler freelist code 
in tp_dealloc(). All it needs to do is 1) clear the ref-counted object fields 
(if any) and 2) put it in the freelist. No other C-API interaction is needed. 
If we only want to keep the object memory, then we need C-API support in both 
tp_new() and tp_dealloc().

If _Py_NewReference() is removed/hidden, then it would be nice if there was a 
replacement for the use case of initialising an object from a freelist that 
already knows its type.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[Stefan Behnel]

Stefan Behnel  added the comment:

To add to the list, Cython also calls _Py_NewReference() in its async generator 
code, which uses a free-list. That code was mostly copied from the 
CPython-internal implementation.

Other freelist implementations in Cython call PyObject_INIT() instead, so I 
guess calling _Py_NewReference() directly here isn't actually required and 
could be avoided.

--
nosy: +scoder

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset fcc60e40bbfe8a229b8b83f1d1ee77fd4bf870d1 by Victor Stinner in 
branch 'master':
bpo-40989: Make _PyTraceMalloc_NewReference() internal (GH-20915)
https://github.com/python/cpython/commit/fcc60e40bbfe8a229b8b83f1d1ee77fd4bf870d1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +20094
pull_request: https://github.com/python/cpython/pull/20915

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> In the CPython code base, _Py_NewReference() is used:
> * to implement the free list optimization

I found a similar code pattern outside CPython code base.

Like Nuitka generic macros for free lists:
https://github.com/Nuitka/Nuitka/blob/8e2357ee8e9a93d835e98d5a88ca0181cc34dabc/nuitka/build/include/nuitka/freelists.h#L22

Another example in the old unmaintained gmpy project (last release in 2013):
https://github.com/LogicalKnight/gmpy/blob/5d758abc9fcb44b08dd000145afe48160bd802f1/src/gmpy2_cache.c#L93-L111

These functions can opt-in for the internal C API to continue to get access to 
_Py_NewReference().

Or maybe free lists should be implemented differently? (is it possible?)


> * in _PyBytes_Resize() and unicode_resize() (resize_compact() to be precise)

Third-party code can use the public PyUnicode_Resize() function and the private 
_PyBytes_Resize() function. Later, if needed, we can consider to expose 
_PyBytes_Resize() in the limited C API.


> * by PyObject_CallFinalizerFromDealloc() to resurrect an object

I found a few projects which basically reimplement the PEP 442 logic in their 
tp_dealloc function. Example with pyuv resurrect_object() function:
https://github.com/saghul/pyuv/blob/9d226dd61162a998745681eb87ef34e1a7d8586a/src/handle.c#L9

For these, using PEP 442 tp_finalizer here would avoid relying on private 
functions like _Py_NewReference(), make the code simpler and more reliable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

STINNER Victor  added the comment:

See "Removal of _Py_ForgetReference from public header in 3.9 issue" discussion 
on capi-sig:
https://mail.python.org/archives/list/capi-...@python.org/thread/4EOCN7P4HI56GQ74FY3TMIKDBIPGKL2G/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +20087
pull_request: https://github.com/python/cpython/pull/20904

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 04fc4f2a46b2fd083639deb872c3a3037fdb47d6 by Victor Stinner in 
branch 'master':
bpo-40989: PyObject_INIT() becomes an alias to PyObject_Init() (GH-20901)
https://github.com/python/cpython/commit/04fc4f2a46b2fd083639deb872c3a3037fdb47d6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

STINNER Victor  added the comment:

Work on this issue started in Python 3.9 with the following change of bpo-39542:

commit f58bd7c1693fe041f7296a5778d0a11287895648
Author: Victor Stinner 
Date:   Wed Feb 5 13:12:19 2020 +0100

bpo-39542: Make PyObject_INIT() opaque in limited C API (GH-18363)

In the limited C API, PyObject_INIT() and PyObject_INIT_VAR() are now
defined as aliases to PyObject_Init() and PyObject_InitVar() to make
their implementation opaque. It avoids to leak implementation details
in the limited C API.

Exclude the following functions from the limited C API, move them
from object.h to cpython/object.h:

* _Py_NewReference()
* _Py_ForgetReference()
* _PyTraceMalloc_NewReference()
* _Py_GetRefTotal()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +20084
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/20901

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40989] [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the public C API

[STINNER Victor]

New submission from STINNER Victor :

The _Py_NewReference() and _Py_ForgetReference() functions are tightly coupled 
to CPython internals.

_Py_NewReference() is only exposed because it used by the PyObject_INIT() macro 
which is the fast inlined flavor of PyObject_Init(). If we make PyObject_INIT() 
as alias to PyObject_Init(), as already done for the limited C API, 
_Py_NewReference() can be removed from the public C API (moved to the internal 
C API).

_Py_ForgetReference() function is only defined if Py_TRACE_REFS macro is 
defined. I propose to also removed it from the public C API (move it to the 
internal C API).

In the CPython code base, _Py_NewReference() is used:

* to implement the free list optimization
* in _PyBytes_Resize() and unicode_resize() (resize_compact() to be precise)
* by PyObject_CallFinalizerFromDealloc() to resurrect an object

These are corner cases which can be avoided in third party C extension modules.

--
components: C API
messages: 371597
nosy: vstinner
priority: normal
severity: normal
status: open
title: [C API] Remove _Py_NewReference() and _Py_ForgetReference() from the 
public C API
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com