[issue5872] New C API for declaring Python types
Martin v. Löwis mar...@v.loewis.de added the comment: With PEP 384, this is now obsolete, so closing it as such. If you think that selected features of this patch should still be added, please submit them as separate patches. -- nosy: +loewis resolution: - out of date status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Changes by Reid Kleckner r...@mit.edu: -- nosy: +rnk versions: +Python 3.2 -Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Changes by Gerhard Häring g...@ghaering.de: -- nosy: +ghaering ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Antoine Pitrou pit...@free.fr added the comment: This patch is huge. Some things: - you should provide an example of using the new API. Your description is not very, well, descriptive :) - all PyTypeObjects must now be *pointers* rather than static instances: is it mandatory, or did you add this out of purity? It would be nice if we could minimize disruption (that is, provide the new API as a feature, but not force ourselves and other people to rewrite huge chunks of code) - same for the private declaration of PyTypeObject: is it really necessary to hide it from outside code? Lastly, since your patch is not ready for consumption, I would advocate creating a branch somewhere (on the SVN sandbox if you have access rights, or in a separate e.g. bitbucket repository by cloning the existing Mercurial mirror). -- nosy: +nnorwitz, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Larry Hastings la...@hastings.org added the comment: Antoine: As the patch matured I would obviously provide documentation and examples and such. The point of submitting the patch in this form was a) so it was alive somewhere besides my hard drive, and b) to get some public review from the core team to ensure I was going in a valid direction. As for mandatory vs purity: The whole purpose of the patch was to make PyTypeObject a private type; see THE PROBLEM / THE SOLUTION. This requires that all the public interfaces take pointers. So within the context of what the patch is trying to accomplish, it's mandatory. The patch attempts to mitigate this as much as possible with the backwards-compatibility p.s. By huge I suspect you mean large, though on first reading I thought you meant fabulous. That's what I get for working with a big crew of 20-somethings. p.p.s. Did this patch get mentioned recently or something? After months of inactivity, there have been two nosy+ this week. p.p.p.s. I have not touched this patch since submitting it. The tribulations of working at a startup. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Larry Hastings la...@hastings.org added the comment: Whoops! I think I'll finish that unfinished sentence. The patch attempts to mitigate this as much as possible with the backwards-compatibility header file; it minimizes as much as possible the changes you need to perform to get your code up and working again. But I fear I've taken that concept about as far as it makes sense to go. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Antoine Pitrou pit...@free.fr added the comment: As the patch matured I would obviously provide documentation and examples and such. What I meant is that it's difficult for me (and perhaps others) to assess how much more practical your patch makes it to create C types. This requires that all the public interfaces take pointers. So within the context of what the patch is trying to accomplish, it's mandatory. Hmm. That public interfaces take pointers is one thing, but that doesn't mean the PyTypeObject structure itself must be concealed. It could be exposed as an implementation detail. Please note that your approach will make it difficult for third-party C extensions to remain compatible accross several Python versions (those before and after the API switch). While we sometimes change or deprecate APIs, we never do it as massively as that (even the 2.x - 3.x transition is quite gentle). p.s. By huge I suspect you mean large, though on first reading I thought you meant fabulous. That's what I get for working with a big crew of 20-somethings. I meant large indeed :) p.p.s. Did this patch get mentioned recently or something? After months of inactivity, there have been two nosy+ this week. Well, a nosy+ bumps up the issue at the top of the recently modified issues, which means other people notice it as well. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Changes by Alexander Belopolsky belopol...@users.sourceforge.net: -- nosy: +belopolsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Changes by Gregory P. Smith g...@krypto.org: -- nosy: +gregory.p.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Changes by Eric Smith e...@trueblade.com: -- nosy: +eric.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
Eric Smith e...@trueblade.com added the comment: I think this is great stuff, Larry. It's a definite improvement. Unfortunately with my workload of other Python issues, I'm not going to be able to review this before the 3.1 beta. I don't use many PyTypeObject's, so my review would be mostly theoretical, anyway. But I do want to encourage you to get this included in Python, at least in concept. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5872] New C API for declaring Python types
New submission from Larry Hastings la...@hastings.org:
EXECUTIVE SUMMARY
I've written a patch against py3k trunk creating a new function-based
API for creating extension types in C. This allows PyTypeObject to
become a (mostly) private structure.
THE PROBLEM
Here's how you create an extension type using the current API.
* First, find some code that already has a working type declaration.
Copy and paste their fifty-line PyTypeObject declaration, then
hack it up until it looks like what you need.
* Next--hey! There *is* no next, you're done. You can immediately
create an object using your type and pass it into the Python
interpreter and it would work fine. You are encouraged to call
PyType_Ready(), but this isn't required and it's often skipped.
This approach causes two problems.
1) The Python interpreter *must support* and *cannot change*
the PyTypeObject structure, forever. Any meaningful change to
the structure will break every extension. This has many
consequences:
a) Fields that are no longer used must be left in place,
forever, as ignored placeholders if need be. Py3k cleaned
up a lot of these, but it's already picked up a new one
(tp_compare is now tp_reserved).
b) Internal implementation details of the type system must
be public.
c) The interpreter can't even use a different structure
internally, because extensions are free to pass in objects
using PyTypeObjects the interpreter has never seen before.
2) As a programming interface this lacks a certain gentility. It
clearly *works*, but it requires programmers to copy and paste
with a large structure mostly containing NULLs, which they must
pick carefully through to change just a few fields.
THE SOLUTION
My patch creates a new function-based extension type definition API.
You create a type by calling PyType_New(), then call various accessor
functions on the type (PyType_SetString and the like), and when your
type has been completely populated you must call PyType_Activate()
to enable it for use.
With this API available, extension authors no longer need to directly
see the innards of the PyTypeObject structure. Well, most of the
fields anyway. There are a few shortcut macros in CPython that need
to continue working for performance reasons, so the tp_flags and
tp_dealloc fields need to remain publically visible.
One feature worth mentioning is that the API is type-safe. Many such
APIs would have had one generic PyType_SetPointer, taking an
identifier for the field and a void * for its value, but this would
have lost type safety. Another approach would have been to have one
accessor per field (PyType_SetAddFunction), but this would have
exploded the number of functions in the API. My API splits the
difference: each distinct *type* has its own set of accessors
(PyType_GetSSizeT) which takes an identifier specifying which
field you wish to get or set.
SIDE-EFFECTS OF THE API
The major change resulting from this API: all PyTypeObjects must now
be *pointers* rather than static instances. For example, the external
declaration of PyType_Type itself changes from this:
PyAPI_DATA(PyTypeObject) PyType_Type;
to this:
PyAPI_DATA(PyTypeObject *) PyType_Type;
This gives rise to the first headache caused by the API: type casts
on type objects. It took me a day and a half to realize that this,
from Modules/_weakref.c:
PyModule_AddObject(m, ref,
(PyObject *) _PyWeakref_RefType);
really needed to be this:
PyModule_AddObject(m, ref,
(PyObject *) _PyWeakref_RefType);
Hopefully I've already found most of these in CPython itself, but
this sort of code surely lurks in extensions yet to be touched.
(Pro-tip: if you're working with this patch, and you see a crash,
and gdb shows you something like this at the top of the stack:
#0 0x081056d8 in visit_decref (op=0x8247aa0, data=0x0)
at Modules/gcmodule.c:323
323 if (PyObject_IS_GC(op)) {
your problem is an errant , likely on a type object you're passing
in to the interpreter. Think--what did you touch recently? Or debug
it by salting your code with calls to collect(NUM_GENERATIONS-1).)
Another irksome side-effect of the API: because of tp_flags and
tp_dealloc, I now have two declarations of PyTypeObject. There's
the externally-visible one in Include/object.h, which lets external
parties see tp_dealloc and tp_flags. Then there's the internal
one in Objects/typeprivate.h which is the real structure. Since
declaring a type twice is a no-no, the external one is gated on
#ifndef PY_TYPEPRIVATE
If you're a normal Python extension programmer, you'd include Python.h
as normal:
#include Python.h
Python implementation files that need to see the real PyTypeObject
structure now look like this:
#define
[issue5872] New C API for declaring Python types
Changes by Brett Cannon br...@python.org: -- nosy: +brett.cannon ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5872 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
