[issue40601] [C API] Hide static types from the limited C API

[Erlend E. Aasland]

Change by Erlend E. Aasland :


--
nosy: +erlendaasland

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[h-vetinari]

Change by h-vetinari :


--
nosy: +h-vetinari

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[Guido van Rossum]

Guido van Rossum  added the comment:

FWIW I have an idea that would allow code using e.g. _Type to continue 
to work, and even ABI compatible (though only in the main interpreter).

// In some header file

PyAPI_FUNC(PyHeapTypeObject *) PyList_GetType();

#define PyList_Type (PyList_GetType()->ht_type)

For the main interpreter we could make this return the address of PyList_Type.

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[STINNER Victor]

STINNER Victor  added the comment:

I plan to write such PEP soon.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[STINNER Victor]

STINNER Victor  added the comment:

The Steering Council asked for a PEP to explain why static types should be 
converted to heap types.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[junyixie]

junyixie  added the comment:

It seems that there is no continued progress for move static type in heap.This 
will make it impossible to continue to achieve sub interpreters parallel. Are 
there any plans to try other solutions to the problem?

In my project, i try to slove this problem, It can work, we verify on millions 
of devices.

1. In typeobject.c add lock to ensure that some functions that modification 
type are thread-safe
2. and make the PyCFunction and descri object of the Type will never be 
released. （Frequently used when load method/attributed, locking affects 
performance）

Can this change be submitted to cpython？

--
nosy: +JunyiXie

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[Petr Viktorin]

Petr Viktorin  added the comment:

Sorry, I lost this bug in my TODO list :(

> > I don't think it's necessary here.
> 
> Did you read my rationale (first message)? Do you mean that per-interpreter 
> GIL is not worth it?

Right, I mean that it it is not worth breaking the C-API for all existing 
modules.
Instead, I think that it can be done as an addition: only modules that don't 
use things like these static types would be allowed in subinterpreters that 
have their own GIL.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +23227
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/24146

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[STINNER Victor]

STINNER Victor  added the comment:

PC/python3dll.c exports 66 types in the stable ABI:

Py_GenericAliasType
PyObject_Type
_PyWeakref_CallableProxyType
_PyWeakref_ProxyType
_PyWeakref_RefType
PyBaseObject_Type
PyBool_Type
PyByteArray_Type
PyByteArrayIter_Type
PyBytes_Type
PyBytesIter_Type
PyCallIter_Type
PyCapsule_Type
PyCFunction_Type
PyClassMethodDescr_Type
PyComplex_Type
PyDict_Type
PyDictItems_Type
PyDictIterItem_Type
PyDictIterKey_Type
PyDictIterValue_Type
PyDictKeys_Type
PyDictProxy_Type
PyDictValues_Type
PyEllipsis_Type
PyEnum_Type
PyExc_TypeError
PyFilter_Type
PyFloat_Type
PyFrozenSet_Type
PyGetSetDescr_Type
PyList_Type
PyListIter_Type
PyListRevIter_Type
PyLong_Type
PyLongRangeIter_Type
PyMap_Type
PyMemberDescr_Type
PyMemoryView_Type
PyMethodDescr_Type
PyModule_Type
PyModuleDef_Type
PyNullImporter_Type
PyODict_Type
PyODictItems_Type
PyODictIter_Type
PyODictKeys_Type
PyODictValues_Type
PyProperty_Type
PyRange_Type
PyRangeIter_Type
PyReversed_Type
PySeqIter_Type
PySet_Type
PySetIter_Type
PySlice_Type
PySortWrapper_Type
PySuper_Type
PyTraceBack_Type
PyTuple_Type
PyTupleIter_Type
PyType_Type
PyUnicode_Type
PyUnicodeIter_Type
PyWrapperDescr_Type
PyZip_Type

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[STINNER Victor]

STINNER Victor  added the comment:

> Technically, it is not, see 
> https://www.python.org/dev/peps/pep-0384/#structures
> Structures like PyLong_Type are *not* part of the limited API.

The symbol is exported by libpython:

$ objdump -T /lib64/libpython3.8.so.1.0|grep PyLong_Type
0030de00 gDO .data  01a0  BasePyLong_Type

A C extension can use a reference to PyLong_Type.

> I don't think it's necessary here.

Did you read my rationale (first message)? Do you mean that per-interpreter GIL 
is not worth it?

--

A first step would be to expose "CheckExact" macros as function calls in the 
limited C API.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[STINNER Victor]

Change by STINNER Victor :


--
components: +Subinterpreters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[hai shi]

Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[Petr Viktorin]

Petr Viktorin  added the comment:

> For example, Objects/longobject.c defines "PyTypeObject PyLong_Type = 
> {...};". This type is exposed in the limited C API (!)

Technically, it is not, see https://www.python.org/dev/peps/pep-0384/#structures
Structures like PyLong_Type are *not* part of the limited API.

> I propose to break the limited C API backward compatibility on purpose by 
> removing these type definitions form the limited C API.

That could only be done in Python 4.0, or if we started C-API 4.0. But I don't 
think it's necessary here.

--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[STINNER Victor]

STINNER Victor  added the comment:

> I propose to break the limited C API backward compatibility on purpose by 
> removing these type definitions form the limited C API.

Hum. How would a C extension subclass the Python int type (PyLong_Type) if it's 
no longer exposed? One option is to add one function per type, like:

PyObject* Py_GetLongType(void);

It would return a *strong reference* to the type (PyLong_Type).

Another option is to get the type from builtins module or builtins dictionary 
(PyInterpreterState.builtins). But there is no simple C function to get a 
builtin object. It requires many calls, handle errors, etc. Maybe a generic 
helper like the following function would help:

PyObject *Py_GetBuiltin(const char *name);

Note: PyEval_GetBuiltins() exposes the builtins of the *current frame* which 
maybe not be what you may expect.

Currently, Py_GetBuiltin(name) is not needed since basically *all* Python 
builtins are *directly* exposed in the C API...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-40077: "Convert static types to PyType_FromSpec()".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

[STINNER Victor]

New submission from STINNER Victor :

"Statically allocated types" prevents to get per-interpreter GIL: bpo-40512. 
These types are currently shared by all interpreters.

Eric Snow proposed the idea of creating a heap allocated type in 
subintepreters. But we should take care of direct usage of the statically 
allocated type.

For example, Objects/longobject.c defines "PyTypeObject PyLong_Type = {...};". 
This type is exposed in the limited C API (!) in Include/longobject.c:

PyAPI_DATA(PyTypeObject) PyLong_Type;

It's used but such macro:

#define PyLong_CheckExact(op) Py_IS_TYPE(op, _Type)

I don't think that these types are directly accessed in C extensions built with 
the limited C API. My expectation is that the type is only exposed for 
"CheckExact" macros.

Currently, 100 statically allocated types are declared in Python header files:

$ grep -F '(PyTypeObject)' Include/ -R
Include/cpython/fileobject.h:PyAPI_DATA(PyTypeObject) PyStdPrinter_Type;
(...)
Include/object.h:PyAPI_DATA(PyTypeObject) PySuper_Type; /* built-in 'super' */
Include/methodobject.h:PyAPI_DATA(PyTypeObject) PyCFunction_Type;

Most of them seem to be exposed in the limited C API.

I propose to break the limited C API backward compatibility on purpose by 
removing these type definitions form the limited C API.

For "CheckExact" macros, we can continue to provide them in the limited C API 
but as function calls. So a built C extension would no longer access directly 
the type, but only do function calls.

--
components: C API
messages: 368667
nosy: vstinner
priority: normal
severity: normal
status: open
title: [C API] Hide static types from the limited C API
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com