[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

Changes already done:

* Macros converted to regular functions:
 
  * PyObject_GET_WEAKREFS_LISTPTR(); add internal inline 
_PyObject_GET_WEAKREFS_LISTPTR()
  * PyType_SUPPORTS_WEAKREFS(); add internal inline _PyType_SUPPORTS_WEAKREFS()
  * PyObject_CheckBuffer(); no fast internal API
  * PyObject_IS_GC(); no fast internal API
  * PyDescr_IsData(); no fast internal API

* Always implemented as a function, remove macro optimization in the 
non-limited API:

  * PyIter_Check(); no fast internal API
  * PyIndex_Check(); add internal inline _PyIndex_Check()
  * PyExceptionClass_Name(); no fast internal API

* Py_TRASHCAN_BEGIN() macro now calls _PyTrash_cond() function: no longer read 
directly tp_dealloc member
* PyObject_NEW() macro becomes an alias to PyObject_New()
* Remove PyHeapType_GET_MEMBERS() (move it to the internal C API)

PyType_HasFeature() is left unchanged since the change caused performance issue 
on macOS, whereas Python is not built with LTO.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 18ea973c21ee4a6adc26be41027881043fa498eb by Victor Stinner in 
branch 'main':
bpo-40170: Remove PyHeapType_GET_MEMBERS() macro (GH-30942)
https://github.com/python/cpython/commit/18ea973c21ee4a6adc26be41027881043fa498eb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 0575551f69ba9c999835bfb176a543d468083c03 by Victor Stinner in 
branch 'main':
bpo-40170: Move _Py_GetAllocatedBlocks() to pycore_pymem.h (GH-30943)
https://github.com/python/cpython/commit/0575551f69ba9c999835bfb176a543d468083c03


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +29122
pull_request: https://github.com/python/cpython/pull/30943

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 6b491b9dc0b0fdfd1f07ea4e2151236186d8e7e6 by Victor Stinner in 
branch 'main':
bpo-40170: Remove _Py_GetAllocatedBlocks() function (GH-30940)
https://github.com/python/cpython/commit/6b491b9dc0b0fdfd1f07ea4e2151236186d8e7e6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset af32b3ef1fbad3c2242627a14398320960a0cb45 by Victor Stinner in 
branch 'main':
bpo-40170: PyType_SUPPORTS_WEAKREFS() becomes a regular function (GH-30938)
https://github.com/python/cpython/commit/af32b3ef1fbad3c2242627a14398320960a0cb45


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +29121
pull_request: https://github.com/python/cpython/pull/30942

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +29119
pull_request: https://github.com/python/cpython/pull/30940

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +29117
pull_request: https://github.com/python/cpython/pull/30938

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

In the top 5000 PyPI projects, the _PyObject_SIZE() and _PyObject_VAR_SIZE() 
functions are used by 7 projects:

* Cython-0.29.26
* frozendict-2.2.0: implement "sizeof" function, found in copies of 
Objects/dictobject.c file
* JPype1-1.3.0
* numpy-1.22.1: gentype_alloc() in numpy/core/src/multiarray/scalartypes.c.src, 
used as type tp_alloc functions
* pickle5-0.0.12
* pyobjc-core-8.2
* recordclass-0.17.1

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

I searched for "_PyObject_DebugMallocStats" in top 5000 PyPI projects. There is 
a single project using it: guppy3.

Extract of guppy3 src/heapy/xmemstats.c:

...
dlptr__PyObject_DebugMallocStats = 
addr_of_symbol("_PyObject_DebugMallocStats");
...
static PyObject *
hp_xmemstats(PyObject *self, PyObject *args)
{
if (dlptr__PyObject_DebugMallocStats) {
fprintf(stderr, 
"==\n");
fprintf(stderr, "Output from _PyObject_DebugMallocStats()\n\n");
dlptr__PyObject_DebugMallocStats(stderr);
}
...
}

addr_of_symbol() is implemented with dlsym() or 
GetModuleHandle(NULL)+GetProcAddress(): it searchs for the symbol in the 
current process.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

I searched for "_PyGC_FINALIZED" in top 5000 PyPI projects. It seems like only 
Cython is impacted.

ddtrace and guppy3 use directly the internal C API.

== Cython 0.29.26 ==

* Cython/Compiler/ModuleNode.py: finalised_check = '!_PyGC_FINALIZED(o)'
* Cython/Compiler/ModuleNode.py: '(!PyType_IS_GC(Py_TYPE(o)) || 
!_PyGC_FINALIZED(o))')

== ddtrace 0.57.3 ==

In ddtrace/profiling/collector/stack.pyx:

IF PY_MINOR_VERSION >= 9:
# Needed for accessing _PyGC_FINALIZED when we build with 
-DPy_BUILD_CORE
cdef extern from "":
pass

== guppy3-3.1.2 ==

In src/heapy/hv.c:

#if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION >= 9
# define Py_BUILD_CORE
/* PyGC_Head */
#  undef _PyGC_FINALIZED
#  include 
# undef Py_BUILD_CORE
#endif

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

TODO:

* Macros still accessing directly PyTypeObject members:

  * PyHeapType_GET_MEMBERS()
  * PySequence_ITEM()
  * _PyObject_SIZE()
  * _PyObject_VAR_SIZE()
  * PyType_SUPPORTS_WEAKREFS()

* Try again to apply "bpo-40170: PyType_HasFeature() now always calls 
PyType_GetFlags() (GH-19378)" which has been reverted?
* Py_TYPE(obj)->tp_name is still commonly used to format error messages: see 
rejected bpo-34595

Other TODO tasks which can be addressed in follow-up issues:

* Add buffer protocol to PyType_FromSpec()
* Stdlib C extensions still define static types: see bpo-40077
* 3rd party C extensions still define static types: PEP 630 and others propose 
heap types

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 80580f5ab85e3c45c8c5e863a891598d10bf by Miss Islington (bot) 
in branch '3.9':
bpo-40170: Update What's New in Python 3.9 (GH-29470) (GH-29472)
https://github.com/python/cpython/commit/80580f5ab85e3c45c8c5e863a891598d10bf


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[miss-islington]

miss-islington  added the comment:


New changeset 69b3de65adc691cc5ad66e70e5c7caf61c202c79 by Miss Islington (bot) 
in branch '3.10':
bpo-40170: Update What's New in Python 3.9 (GH-29470)
https://github.com/python/cpython/commit/69b3de65adc691cc5ad66e70e5c7caf61c202c79


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[miss-islington]

Change by miss-islington :


--
pull_requests: +27724
pull_request: https://github.com/python/cpython/pull/29471

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[miss-islington]

Change by miss-islington :


--
pull_requests: +27725
pull_request: https://github.com/python/cpython/pull/29472

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 99c7e9853fa13af414168f179213e3d2fae03a45 by Victor Stinner in 
branch 'main':
bpo-40170: Update What's New in Python 3.9 (GH-29470)
https://github.com/python/cpython/commit/99c7e9853fa13af414168f179213e3d2fae03a45


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +27723
pull_request: https://github.com/python/cpython/pull/29470

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 630264a152115f9671d6b793455ef5c2cea09a97 by Erlend Egeberg 
Aasland in branch 'master':
 bpo-40170: Move 3 NEWS entries to the C API section (GH-24555)
https://github.com/python/cpython/commit/630264a152115f9671d6b793455ef5c2cea09a97


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Erlend Egeberg Aasland]

Change by Erlend Egeberg Aasland :


--
pull_requests: +23341
pull_request: https://github.com/python/cpython/pull/24555

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Erlend Egeberg Aasland]

Erlend Egeberg Aasland  added the comment:

> This can introduce a performance slowdown and so should wait until the PEP 
> 620 is accepted.

Noted.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> For PySequence_ITEM, I guess adding a private C version (for example 
> _PySequence_Item) and redirecting the macro to the C version would be 
> acceptable.

This can introduce a performance slowdown and so should wait until the PEP 620 
is accepted.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset cd80f430daa7dfe7feeb431ed34f88db5f64aa30 by Erlend Egeberg 
Aasland in branch 'master':
bpo-40170: Always define PyExceptionClass_Name() as a function (GH-24553)
https://github.com/python/cpython/commit/cd80f430daa7dfe7feeb431ed34f88db5f64aa30


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Erlend Egeberg Aasland]

Erlend Egeberg Aasland  added the comment:

Thanks, Victor.

For PySequence_ITEM, I guess adding a private C version (for example 
_PySequence_Item) and redirecting the macro to the C version would be 
acceptable.

Ditto for PyHeapType_GET_MEMBERS and PyType_SUPPORTS_WEAKREFS.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Erlend Egeberg Aasland]

Change by Erlend Egeberg Aasland :


--
pull_requests: +23335
pull_request: https://github.com/python/cpython/pull/24553

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> For PyExceptionClass_Name: Is it ok to just remove the macro version (like 
> with GH-24548)?

Yes, I think so.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Erlend Egeberg Aasland]

Erlend Egeberg Aasland  added the comment:

For PyExceptionClass_Name: Is it ok to just remove the macro version (like with 
GH-24548)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> New changeset cc54001c2eb3b14320c1667b22602d69c90d5865 by Erlend Egeberg 
> Aasland in branch 'master':
> bpo-40170: Always define PyIter_Check() as a function (GH-24548)

For macOS which doesn't use LTO compiler optimization, we added private static 
inline functions of some "Check" functions. But I don't think that it's worth 
it here (I don't think that the function is commonly called in "hot code").

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset cc54001c2eb3b14320c1667b22602d69c90d5865 by Erlend Egeberg 
Aasland in branch 'master':
bpo-40170: Always define PyIter_Check() as a function (GH-24548)
https://github.com/python/cpython/commit/cc54001c2eb3b14320c1667b22602d69c90d5865


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Erlend Egeberg Aasland]

Change by Erlend Egeberg Aasland :


--
pull_requests: +23331
pull_request: https://github.com/python/cpython/pull/24548

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 871eb4237b9be95263ca13ba8856e78344eb9eba by Erlend Egeberg 
Aasland in branch 'master':
bpo-40170: Convert PyDescr_IsData() to static inline function (GH-24535)
https://github.com/python/cpython/commit/871eb4237b9be95263ca13ba8856e78344eb9eba


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> Should we strive to fix the cases in Include/internal as well?

No. The internal C API access directly to structure members on purpose, for 
best performances.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Erlend Egeberg Aasland]

Erlend Egeberg Aasland  added the comment:

After GH-24533 and GH-24535, there's (AFAICS) only a handful of cases left.
$ grep -r "\->tp_" Include/:

Include/internal/pycore_abstract.h:PyNumberMethods *tp_as_number = 
Py_TYPE(obj)->tp_as_number;
Include/internal/pycore_interp.h:unsigned int version;  // initialized from 
type->tp_version_tag
Include/internal/pycore_object.h:return ((type->tp_flags & feature) != 0);
Include/internal/pycore_object.h:Py_ssize_t offset = 
Py_TYPE(op)->tp_weaklistoffset;
Include/internal/pycore_object.h:&& (Py_TYPE(obj)->tp_is_gc == NULL
Include/internal/pycore_object.h:|| 
Py_TYPE(obj)->tp_is_gc(obj)));
Include/cpython/pyerrors.h:#define PyExceptionClass_Name(x)  
(((PyTypeObject*)(x))->tp_name)
Include/cpython/abstract.h:offset = tp->tp_vectorcall_offset;
Include/cpython/abstract.h:( Py_TYPE(o)->tp_as_sequence->sq_item(o, i) )
Include/cpython/object.h:((PyMemberDef *)(((char *)etype) + 
Py_TYPE(etype)->tp_basicsize))
Include/cpython/objimpl.h:#define _PyObject_SIZE(typeobj) ( 
(typeobj)->tp_basicsize )
Include/cpython/objimpl.h:_Py_SIZE_ROUND_UP((typeobj)->tp_basicsize + \
Include/cpython/objimpl.h:(nitems)*(typeobj)->tp_itemsize,\
Include/cpython/objimpl.h:#define PyType_SUPPORTS_WEAKREFS(t) 
((t)->tp_weaklistoffset > 0)
Include/object.h:flags = type->tp_flags;

Should we strive to fix the cases in Include/internal as well?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Erlend Egeberg Aasland]

Change by Erlend Egeberg Aasland :


--
nosy: +erlendaasland
nosy_count: 7.0 -> 8.0
pull_requests: +23322
pull_request: https://github.com/python/cpython/pull/24535

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-41618: [C API] How many slots of static types should be exposed in 
PyType_GetSlot().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset ed1a5a5baca8f61e9a99c5be3adc16b1801514fe by Hai Shi in branch 
'master':
bpo-40170: Hide impl detail of Py_TRASHCAN_BEGIN macro (GH-23235)
https://github.com/python/cpython/commit/ed1a5a5baca8f61e9a99c5be3adc16b1801514fe


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset ba2958ed40d284228836735cbed4a155190e0998 by Victor Stinner in 
branch 'master':
bpo-40170: Fix PyType_Ready() refleak on static type (GH-23236)
https://github.com/python/cpython/commit/ba2958ed40d284228836735cbed4a155190e0998


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22133
pull_request: https://github.com/python/cpython/pull/23236

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[hai shi]

Change by hai shi :


--
pull_requests: +22132
pull_request: https://github.com/python/cpython/pull/23235

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22065
pull_request: https://github.com/python/cpython/pull/23153

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 97d15ae1d8411b49b1fcdc0c67c51849dccce9c9 by Victor Stinner in 
branch 'master':
bpo-40170: Use inline _PyType_HasFeature() function (GH-22375)
https://github.com/python/cpython/commit/97d15ae1d8411b49b1fcdc0c67c51849dccce9c9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +21414
pull_request: https://github.com/python/cpython/pull/22375

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Andy Lester]

Change by Andy Lester :


--
nosy:  -petdance

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> Thanks for doing this.  I can confirm the performance regression is fixed and 
> that clean code is being generated for PyTuple_Check().

Thanks for checking!

> BTW, I support your efforts — just wanted to make sure we didn't 
> unintentionally take a step backwards.

I tried to avoid changes which could affect performances.

I wrote PEP 620 for such changes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Thanks for doing this.  I can confirm the performance regression is fixed and 
that clean code is being generated for PyTuple_Check().

BTW, I support your efforts — just wanted to make sure we didn't 
unintentionally take a step backwards.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[miss-islington]

miss-islington  added the comment:


New changeset a0a6f1167834c87f12e2eca11dd77143103e7691 by Miss Islington (bot) 
in branch '3.9':
Revert "bpo-40170: PyType_HasFeature() now always calls PyType_GetFlags() 
(GH-19378)" (GH-21390)
https://github.com/python/cpython/commit/a0a6f1167834c87f12e2eca11dd77143103e7691


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> New changeset 45ec5b99aefa54552947049086e87ec01bc2fc9a by Victor Stinner in 
> branch 'master':
> bpo-40170: PyType_HasFeature() now always calls PyType_GetFlags() (GH-19378)

This change causes performance issues on macOS, see discussion starting at:
https://bugs.python.org/issue39542#msg372962

So I reverted the change. I will wait until my PEP 620 is accepted before 
considering to reapply it.

If it's reapplied, we have to make sure that Python internals currently using 
PyTuple_Check() still access directly PyTypeObject.tp_flags member. For 
example, a new _PyTuple_Check() function could be added and uses the internal 
_PyType_HasFeature() function.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b26a0db8ea2de3a8a8e4b40e69fc8642c7d7cb68 by Victor Stinner in 
branch 'master':
Revert "bpo-40170: PyType_HasFeature() now always calls PyType_GetFlags() 
(GH-19378)" (GH-21390)
https://github.com/python/cpython/commit/b26a0db8ea2de3a8a8e4b40e69fc8642c7d7cb68


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 6.0 -> 7.0
pull_requests: +20537
pull_request: https://github.com/python/cpython/pull/21391

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +20536
pull_request: https://github.com/python/cpython/pull/21390

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> PyIter_Check() and PySequence_ITEM() macros access directly PyTypeObject 
> members and must be converted to opaque functions: (...)

PyIter_Check() and PySequence_ITEM() are declared as functions in the limited C 
API, but overriden with macros in the CPython C API.

I suggest to simply remove the macros to always declare them as functions.

See bpo-33738 "PyIndex_Check conflicts with PEP 384" which added the functions.

See also Tools/scripts/pep384_macrocheck.py script.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

PyIter_Check() and PySequence_ITEM() macros access directly PyTypeObject 
members and must be converted to opaque functions:

#define PyIter_Check(obj) \
(Py_TYPE(obj)->tp_iternext != NULL && \
 Py_TYPE(obj)->tp_iternext != &_PyObject_NextNotImplemented)

#define PySequence_ITEM(o, i)\
( Py_TYPE(o)->tp_as_sequence->sq_item(o, i) )

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> Py_TRASHCAN_BEGIN() access directly PyTypeObject.tp_dealloc (...) currently, 
> PyType_GetSlot() can only be used on a heap allocated types

I created bpo-41073: [C API] PyType_GetSlot() should accept static types.

--
versions: +Python 3.10 -Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[hai shi]

hai shi  added the comment:

> It never prevented anyone to use a function of the C API :-)
Got it. If possible someone uses it, I will try to add a function to repalce 
it(MAYBE udpate this docs too;) )

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[hai shi]

Change by hai shi :


--
pull_requests: +18889
pull_request: https://github.com/python/cpython/pull/19541

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> Looks like this macro not recorded in docs.

It never prevented anyone to use a function of the C API :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[hai shi]

hai shi  added the comment:

> Py_TRASHCAN_BEGIN() access directly PyTypeObject.tp_dealloc

Looks like this macro not recorded in docs. Do we need using function to 
replace this macro?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Ronald Oussoren]

Ronald Oussoren  added the comment:

Something else that probably needs attention with the TypeSpec API is 
subclassing type in an extension when that subclass adds fields to the type 
object.

I use this in PyObjC to (dynamically) create types that have some additional 
data.  I could probably work around this issue by adding a level of indirection 
(basically storing the extra data in a WeakKeyDictionary), but haven't looked 
into this yet.

--
nosy: +ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

On python-dev, Ronald Oussoren asked to add support for the buffer protocol in 
PyType_FromSpec() and PyTypeSpec API:

Ronald:
> BTW. This will require growing the PyTypeSpec ABI a little, there are 
> features you cannot implement using that API for example the buffer protocol.

https://mail.python.org/archives/list/python-...@python.org/message/PGKRW7S2IUOWVRX6F7RT6VAWD3ZPUDYS/

See also PyType_FromSpec() issue with opaque PyObject:
https://bugs.python.org/issue39573#msg366473

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 675d9a3d7afc767a2818c84da7ba4bf4181dcf26 by Hai Shi in branch 
'master':
bpo-40170: Convert PyObject_IS_GC() macro to a function (GH-19464)
https://github.com/python/cpython/commit/675d9a3d7afc767a2818c84da7ba4bf4181dcf26


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[hai shi]

Change by hai shi :


--
pull_requests: +18819
pull_request: https://github.com/python/cpython/pull/19464

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[hai shi]

hai shi  added the comment:

> This macro should be converted to an opaque function
Can I try it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

/* Test if an object has a GC head */
#define PyObject_IS_GC(o) \
(PyType_IS_GC(Py_TYPE(o)) \
 && (Py_TYPE(o)->tp_is_gc == NULL || Py_TYPE(o)->tp_is_gc(o)))

This macro should be converted to an opaque function.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 307b9d0144e719b016a47fcc43397c070615e01e by Victor Stinner in 
branch 'master':
bpo-40170: Remove PyIndex_Check() macro (GH-19428)
https://github.com/python/cpython/commit/307b9d0144e719b016a47fcc43397c070615e01e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18786
pull_request: https://github.com/python/cpython/pull/19428

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

Hum, once most changes will land, maybe it would be worth it to document them 
at:
https://docs.python.org/dev/whatsnew/3.9.html#build-and-c-api-changes

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset a15e260b708a98edaba86a2aa663c3f6b2abc964 by Victor Stinner in 
branch 'master':
bpo-40170: Add _PyIndex_Check() internal function (GH-19426)
https://github.com/python/cpython/commit/a15e260b708a98edaba86a2aa663c3f6b2abc964


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 45ec5b99aefa54552947049086e87ec01bc2fc9a by Victor Stinner in 
branch 'master':
bpo-40170: PyType_HasFeature() now always calls PyType_GetFlags() (GH-19378)
https://github.com/python/cpython/commit/45ec5b99aefa54552947049086e87ec01bc2fc9a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18784
pull_request: https://github.com/python/cpython/pull/19426

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset ef5c615f5ae72c4f6979159c94da46afefbfab9a by Victor Stinner in 
branch 'master':
bpo-40170: Convert PyObject_CheckBuffer() macro to a function (GH-19376)
https://github.com/python/cpython/commit/ef5c615f5ae72c4f6979159c94da46afefbfab9a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 9205520d8c43488696d66cbdd9aefbb21871c508 by Victor Stinner in 
branch 'master':
bpo-40170: PyObject_NEW() becomes an alias to PyObject_New() (GH-19379)
https://github.com/python/cpython/commit/9205520d8c43488696d66cbdd9aefbb21871c508


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> Just checked - seems to be SPECIFIC to xlc-v16 as neither xlv-v11 nor xlc-v13 
> have any issues building.

That sounds like an AIX specific issue. Please open a separated issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

> It should use PyType_GetSlot()

Oh. It seems like currently, PyType_GetSlot() can only be used on a heap 
allocated types :-( The function starts with:

if (!PyType_HasFeature(type, Py_TPFLAGS_HEAPTYPE) || slot < 0) {
PyErr_BadInternalCall();
return NULL;
}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

Py_TRASHCAN_BEGIN() access directly PyTypeObject.tp_dealloc:

#define Py_TRASHCAN_BEGIN(op, dealloc) \
Py_TRASHCAN_BEGIN_CONDITION(op, \
Py_TYPE(op)->tp_dealloc == (destructor)(dealloc))

It should use PyType_GetSlot() or a new getter function (to read 
PyTypeObject.tp_dealloc) should be added.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Michael Felt]

Michael Felt  added the comment:

Just checked - seems to be SPECIFIC to xlc-v16 as neither xlv-v11 nor xlc-v13 
have any issues building.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Michael Felt]

Michael Felt  added the comment:

Just manually verified that PR19377, when compiled against xlc - crashes during 
make:

rm -f libpython3.9d.a
ar rcs libpython3.9d.a Modules/getbuildinfo.o  Parser/acceler.o  
Parser/grammar1.o  Parser/listnode.o  Parser/node.o  Parser/parser.o  
Parser/token.o   Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o  
Objects/abstract.o  Objects/accu.o  Objects/boolobject.o  
Objects/bytes_methods.o  Objects/bytearrayobject.o  Objects/bytesobject.o  
Objects/call.o  Objects/capsule.o  Objects/cellobject.o  Objects/classobject.o  
Objects/codeobject.o  Objects/complexobject.o  Objects/descrobject.o  
Objects/enumobject.o  Objects/exceptions.o  Objects/genobject.o  
Objects/fileobject.o  Objects/floatobject.o  Objects/frameobject.o  
Objects/funcobject.o  Objects/interpreteridobject.o  Objects/iterobject.o  
Objects/listobject.o  Objects/longobject.o  Objects/dictobject.o  
Objects/odictobject.o  Objects/memoryobject.o  Objects/methodobject.o  
Objects/moduleobject.o  Objects/namespaceobject.o  Objects/object.o  
Objects/obmalloc.o  Objects/picklebufobject.o  Objects/rangeobject.o  
Objects/setobject.o 
  Objects/sliceobject.o  Objects/structseq.o  Objects/tupleobject.o  
Objects/typeobject.o  Objects/unicodeobject.o  Objects/unicodectype.o  
Objects/weakrefobject.o  Python/_warnings.o  Python/Python-ast.o  Python/asdl.o 
 Python/ast.o  Python/ast_opt.o  Python/ast_unparse.o  Python/bltinmodule.o  
Python/ceval.o  Python/codecs.o  Python/compile.o  Python/context.o  
Python/dynamic_annotations.o  Python/errors.o  Python/frozenmain.o  
Python/future.o  Python/getargs.o  Python/getcompiler.o  Python/getcopyright.o  
Python/getplatform.o  Python/getversion.o  Python/graminit.o  Python/hamt.o  
Python/import.o  Python/importdl.o  Python/initconfig.o  Python/marshal.o  
Python/modsupport.o  Python/mysnprintf.o  Python/mystrtoul.o  
Python/pathconfig.o  Python/peephole.o  Python/preconfig.o  Python/pyarena.o  
Python/pyctype.o  Python/pyfpe.o  Python/pyhash.o  Python/pylifecycle.o  
Python/pymath.o  Python/pystate.o  Python/pythonrun.o  Python/pytime.o  
Python/bootstrap_hash.o  Python/structmember.o 
  Python/symtable.o  Python/sysmodule.o  Python/thread
.o  Python/traceback.o  Python/getopt.o  Python/pystrcmp.o  Python/pystrtod.o  
Python/pystrhex.o  Python/dtoa.o  Python/formatter_unicode.o  
Python/fileutils.o  Python/dynload_shlib.oModules/config.o  
Modules/getpath.o  Modules/main.o  Modules/gcmodule.o  Modules/posixmodule.o  
Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  
Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  
Modules/_operator.o  Modules/_collectionsmodule.o  Modules/_abc.o  
Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  
Modules/_stat.o  Modules/timemodule.o  Modules/_threadmodule.o  
Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o 
Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  
Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o  
Modules/symtablemodule.o  Modules/xxsubtype.o  Python/frozen.o
./Modules/makexp_aix Modules/python.exp . libpython3.9d.a;  xlc_r 
-Wl,-bE:Modules/python.exp -lld -o python Programs/python.o libpython3.9d.a 
-lintl -ldl  -lm   -lm 
 ./python -E -S -m sysconfig --generate-posix-vars ; if test $? -ne 0 ; 
then  echo "generate-posix-vars failed" ;  rm -f ./pybuilddir.txt ;  exit 1 ;  
fi
Objects/genobject.c:127: _PyObject_GC_TRACK: Assertion "!(((PyGC_Head 
*)(op)-1)->_gc_next != 0)" failed: object already tracked by the garbage 
collector
Enable tracemalloc to get the memory block allocation traceback
object address  : 30084150
object refcount : 0
object type : 20013aa8
object type name: generator
object repr : 
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: core initialized
Current thread 0x0001 (most recent call first):
  File "", line 1593 in _setup
  File "", line 1634 in _install
  File "", line 1189 in _install_external_importers
/bin/sh: 24117648 IOT/Abort trap(coredump)
make: 1254-004 The error code from the last command is 134.
Stop.

FYI: about two hours ago I verified that xlc and 
08050e959e6c40839cd2c9e5f6a4fd1513e3d605 : bpo-40147: Fix a compiler warning on 
Windows in Python/compile.c (GH-19389)

all was green.

--
nosy: +Michael.Felt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 38aefc585f60a77d66f4fbe5a37594a488b53474 by Victor Stinner in 
branch 'master':
bpo-40170: PyObject_GET_WEAKREFS_LISTPTR() becomes a function (GH-19377)
https://github.com/python/cpython/commit/38aefc585f60a77d66f4fbe5a37594a488b53474


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18741
pull_request: https://github.com/python/cpython/pull/19379

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18740
pull_request: https://github.com/python/cpython/pull/19378

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18739
pull_request: https://github.com/python/cpython/pull/19377

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18738
pull_request: https://github.com/python/cpython/pull/19376

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +18737
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19375

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:

Macros and static inline functions of the public C API which access directly 
PyTypeObject fields. There may be more.

#define _PyObject_SIZE(typeobj) ( (typeobj)->tp_basicsize )

static inline vectorcallfunc
PyVectorcall_Function(PyObject *callable)
{
...
tp = Py_TYPE(callable);
offset = tp->tp_vectorcall_offset;
...
}

#define PyObject_CheckBuffer(obj) \
((Py_TYPE(obj)->tp_as_buffer != NULL) &&  \
 (Py_TYPE(obj)->tp_as_buffer->bf_getbuffer != NULL))

#define PyIndex_Check(obj)  \
(Py_TYPE(obj)->tp_as_number != NULL &&\
 Py_TYPE(obj)->tp_as_number->nb_index != NULL)

#define PyObject_GET_WEAKREFS_LISTPTR(o) \
((PyObject **) (((char *) (o)) + Py_TYPE(o)->tp_weaklistoffset))

static inline int
PyType_HasFeature(PyTypeObject *type, unsigned long feature) {
#ifdef Py_LIMITED_API
return ((PyType_GetFlags(type) & feature) != 0);
#else
return ((type->tp_flags & feature) != 0);
#endif
}

#define _PyObject_SIZE(typeobj) ( (typeobj)->tp_basicsize )

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[hai shi]

Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Andy Lester]

Change by Andy Lester :


--
nosy: +petdance

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[Dong-hee Na]

Change by Dong-hee Na :


--
nosy: +corona10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

New submission from STINNER Victor :

Leaking the PyTypeObject structure in the C API indirectly causes ABI issue 
(especially for statically allocated types), cause practical issues when old 
fields are removed and new fields are added (ex: tp_vectorcall addition and 
tp_print removal caused a lot of troubles with C code generated by Cython: see 
bpo-37250), prevents us to add feature and experiment optimization.

I don't expect that we will be able to make PyTypeObject opaque soon. The 
purpose of this issue is to track the work done towards this goal.

I propose to slowly prepare the Python code base, the C API and third party 
code (especially Cython) to make PyTypeObject structure opaque.

We have to identify most common code patterns which access directly 
PyTypeObject fields, provide helper functions, and ease the migration to 
solutions which don't access directly PyTypeObject.

See also bpo-39573: "Make PyObject an opaque structure in the limited C API" 
and bpo-39947 "Make the PyThreadState structure opaque (move it to the internal 
C API)".

Longer rationale about making structures of the C API opaque:

* https://pythoncapi.readthedocs.io/
* https://pythoncapi.readthedocs.io/bad_api.html
* https://pythoncapi.readthedocs.io/optimization_ideas.html

--

Multiple practical issues are preventing us to make PyTypeObject opaque right 
now.


(*) Many C extension modules are still using statically allocated types: there 
is an on-going effort in bpo-40077 to convert C extension modules one by one to 
PyType_FromSpec().


(*) Py_TYPE(obj)->tp_name is commonly accessed to format an error message. 
Example:

PyErr_Format(PyExc_TypeError, "exec() globals must be a dict, not 
%.100s",
 Py_TYPE(globals)->tp_name);

I worked on bpo-34595 and started a discussion on python-dev to propose to add 
a new %T formatter to PyUnicode_FromFormatV() and so indirectly to 
PyUnicode_FromFormat() and PyErr_Format():
https://mail.python.org/archives/list/python-...@python.org/thread/HKYUMTVHNBVB5LJNRMZ7TPUQKGKAERCJ/#3UAMHYG6UF4MPLXBZORHO4JVKUBRUZ53

Sadly, we failed to reach a consensus and I gave up on this idea. We should 
reconsider this idea.

We need to agree on how types should be formatted:

* just the name without any dot "type_name",
* qualified name "something.type_name",
* fully qualified name "module.something.type_name"

There is also the question of breaking applications which rely on the current 
exact error message. And the question of removing legacy "%.100s" which was 
used before Python was able to allocate a buffer large enough to arbitrary 
string length. When an error is formatted in pure Python, names are never 
truncated.


(*) Call the function of the parent type when a method is overriden in a 
subclass. Example with PyTypeObject.tp_free called in a deallocator:

static void
abc_data_dealloc(_abc_data *self)
{
PyTypeObject *tp = Py_TYPE(self);
...
tp->tp_free(self);
Py_DECREF(tp);
}


(*) The PEP 384 provides the most generic PyType_GetSlot() but it's not 
convenient to use: need to handle error (NULL), need to cast the void* into the 
expected type (error prone cast), etc.

We should slowly add more and more helper functions for most common use cases. 
We can try to convert a few C extension modules of the Python stdlib to see 
which use cases are the most common.

Hopefully, many use cases are already abstracted by widely used functions like 
PyNumber_Add(), PySequence_Size(), etc.


(*) Likely other issues that I forgot.

--
components: C API
messages: 365689
nosy: vstinner
priority: normal
severity: normal
status: open
title: [C API] Make PyTypeObject structure an opaque structure in the public C 
API
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com