[issue15870] PyType_FromSpec should take metaclass as an argument

[Petr Viktorin]

Petr Viktorin  added the comment:

The new issue is bpo-45383.

There's a sprint newt week; I'll probably get to it then.


> But, I am trying to understand the preference for static better.  There is 
> probably something to learn or even important I am missing.

The original motivating use case is not "create types dynamically on demand", 
but "create types without details of the C structure being compiled into the 
extension". That is, make it possible for modules to be compatible with several 
versions of Python.
The spec is designed to be extensible; the type struct is designed to be fast 
at runtime.


> And I have some plans to make static specs useful in type checking

This is quickly getting out of scope here, but if you're interested, I put down 
some notes here: https://github.com/encukou/abi3/issues/19

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Sebastian Berg]

Sebastian Berg  added the comment:

Yeah, I will try and have a look.  I had posted the patch, because the test 
looked like a bit of a larger chunk of work ;).

> And I'm surprised that you're surprised :)

:).  I am coming from a completely different angle, probably.  Just if you are 
curious, I use a from-spec like API (not public yet) in NumPy and dynamic use 
seemed natural/logical in that use-case, e.g.: 
https://github.com/numpy/numpy/blob/c92864091e5928d92bc109d1505febe35f3909f1/numpy/core/src/multiarray/convert_datatype.c#L2434

But, I am trying to understand the preference for static better.  There is 
probably something to learn or even important I am missing.


> And I have some plans to make static specs useful in type checking, since we 
> can assume that types made from the same spec share the memory layout.

(I don't understand how it is useful, unless you reuse slot structs?)
It sounds interesting, but even static does not guarantee constant unless the 
user indicates it through a flag?
Maybe you could achieve this by figuring it out by inspecting/comparing content 
rather than using the spec pointer(s)? (More complex, but also more powerful?)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Petr Viktorin]

Petr Viktorin  added the comment:

> I am slightly surprised we actually care about static C-definitions?

And I'm surprised that you're surprised :)
AFAIK, supporting dynamically allocated specs/slots was an afterthought. I do 
agree they should be supported, though! Thanks for filing bpo-45315, and please 
file any more bugs your find.

But I'd still say that best practice is to make specs static if possible. (And 
I have some plans to make static specs useful in type checking, since we can 
assume that types made from the same spec share the memory layout.)


> My patch tries to address the first (the class creator has to take care that 
> this is reasonable for the metaclass).  I had hoped the `slot` mechanism can 
> avoid the API discussion for the second one, but I guess not.

Whoa, I missed the patch completely --  2021 looks too much like 2012, I'm used 
to patches being old since we use pull requests now, and the conversation 
turned to slots too quickly... but missing that you mentioned it is completely 
on me. Sorry!

Do you want to go through with the patch yourself, or should I take over? It 
still needs:
- opening a PR on GitHub
- tests
- documentation & a What's New entry
- probably separate bug as well, since it doesn't fix this one

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Sebastian Berg]

Sebastian Berg  added the comment:

> But if tp_name is created dynamically, it could lead to a dangling pointer.

I will guess this is probably just an oversight/bug since the main aim was to 
move towards heap-types, and opened an issue: https://bugs.python.org/issue45315

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

> Everything is copied by `_FromSpec` after all.

One thing I noticed isn't copied is the string pointed to by tp_name: 
https://github.com/python/cpython/blob/0c50b8c0b8274d54d6b71ed7bd21057d3642f138/Objects/typeobject.c#L3427

This isn't an issue if tp_name is initialized from a string literal.  But if 
tp_name is created dynamically, it could lead to a dangling pointer.  If the 
general message is that "everything is copied by _FromSpec", it might make 
sense to copy the tp_name string too.

> However, I suppose that would replace a safe-by-design API with a "best 
> practice" to never define the spec/slots statically (a best practice that is 
> probably not generally followed or even advertised currently, I guess).

Yes that seems reasonable.  I generally prefer static declarations, since they 
will end up in .data instead of .text and will avoid a copy to the stack at 
runtime.  But these are very minor differences, especially for code that only 
runs once at startup, and a safe-by-default recommendation of always 
initializing PyType_* on the stack makes sense.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Sebastian Berg]

Sebastian Berg  added the comment:

Just to note, that there are two – somewhat distinct – issues here in my 
opinion:

1. `FromSpec` does not scan `bases` for the correct metaclass, which it could; 
this could even be considered a bug?
2. You cannot pass in a desired metaclass, which may require a new API function.

My patch tries to address the first (the class creator has to take care that 
this is reasonable for the metaclass).  I had hoped the `slot` mechanism can 
avoid the API discussion for the second one, but I guess not.


On the discussion on `tp_type/meta` being incorrect usage:
I am slightly surprised we actually care about static C-definitions?

There is no reason that a `spec` should be declared static (aside that you have 
to move it into the function otherwise)?  Everything is copied by `_FromSpec` 
after all.
However, I suppose that would replace a safe-by-design API with a "best 
practice" to never define the spec/slots statically (a best practice that is 
probably not generally followed or even advertised currently, I guess).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

> Windows/MSVC defines DLLs as separate programs, with their own lifetime and 
> entry point (e.g. you can reload a DLL multiple times and it will be 
> reinitialised each time).

All of this is true of so's in ELF also.  It doesn't mean that the 
implementation needs to reject standards-conforming programs.

I still think the Python documentation is incorrect on this point.  I filed 
https://bugs.python.org/issue45306 to track this separately.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Steve Dower]

Steve Dower  added the comment:

> MSVC rejects this standard-conforming TU when __declspec(dllimport) is added: 
> https://godbolt.org/z/GYrfTqaGn  I am pretty sure this is out of compliance 
> with C99.

Windows/MSVC defines DLLs as separate programs, with their own lifetime and 
entry point (e.g. you can reload a DLL multiple times and it will be 
reinitialised each time). So there's no conflict with the standard here, and 
certainly nothing that affects the real discussion.

If you'd like to continue this sideline, feel free to take it elsewhere - I'm 
done with it. Let's keep the focus on making sure the added feature is useful 
for users.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

This behavior is covered by the standard.  The following C translation unit is 
valid according to C99:

  struct PyTypeObject;
  extern struct PyTypeObject Foo_Type;
  struct PyTypeObject *ptr = _Type;

Specifically, _Type is an "address constant" per the standard because it is 
a pointer to an object of static storage duration (6.6p9).

The Python docs contradict this with the following incorrect statement:

> However, the unary ‘&’ operator applied to a non-static variable like 
> PyBaseObject_Type() is not required to produce an address constant.

This statement is incorrect:

1. PyBaseObject_Type is an object of static storage duration.  (Note, this is 
true even though it does not use the "static" keyword -- the "static" 
storage-class specifier and "static storage duration" are separate concepts).

2. It follows that _Type is required to produce an address 
constant. because it is a pointer to an object of static storage duration.

MSVC rejects this standard-conforming TU when __declspec(dllimport) is added: 
https://godbolt.org/z/GYrfTqaGn  I am pretty sure this is out of compliance 
with C99.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Steve Dower]

Steve Dower  added the comment:

The section of documentation you reference explains that this behaviour is  not 
covered by the standard ("applied to a non-static variable like 
PyBaseObject_Type() is not required to produce an address constant"), and so 
static addresses of exported symbols do not have to be supported.

It also says that gcc supports it (I assume by generating dynamic code for 
getting the address) while MSVC does not (requiring you to write your own 
dynamic code). 

The conclusion, "tp_base should be set in the extension module’s init 
function," is exactly the right conclusion if you want your code to work across 
all the supported compilers. Invoking the C standard to explain why this looks 
similar to standard code but actually is not is totally fine.

Though I do note that the text can obviously be clearer. I assume it was 
written this way because of a discussion that started "but the C standard says 
..." and so it was clarified to point out that this isn't actually the part of 
the spec that someone thought it was. If we can make it clearer, happy to, but 
it's certainly not incorrect as it stands.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

> On ELF/Mach-O...

nvm, I just realized that you were speaking about Windows specifically here.  I 
believe you that on Windows "static" makes no difference in this case.

The second point stands: if you consider LoadLibrary()/dlopen() to be outside 
the bounds of what the C standard speaks to, then the docs shouldn't invoke the 
C standard to explain the behavior.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

> "static" anything in C is completely irrelevant to how symbols are looked up 
> and resolved between modules

That is not true.  On ELF/Mach-O the "static" storage-class specifier in C will 
prevent a symbol from being added to the dynamic symbol table, which will make 
it unavailable for use across modules.

> I wasn't aware the C standard covered dynamic symbol resolution?

Well the Python docs invoke the C standard to justify the behavior of DLL 
symbol resolution on Windows, using incorrect arguments about what the standard 
says: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_base

Fixing those docs would be a good first step.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Steve Dower]

Steve Dower  added the comment:

> As a tangential point, I think that the DLL case on Windows may be a case 
> where Windows is not compliant with the C standard

I wasn't aware the C standard covered dynamic symbol resolution? "static" 
anything in C is completely irrelevant to how symbols are looked up and 
resolved between modules, and how addresses are relocated in PE binaries, by 
the Windows loader.

So this point isn't even tangential - it's nowhere near the surface ;)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

> I consider Py_tp_bases to be a mistake: it's an extra way of doing things 
> that doesn't add any extra functionality

I think it does add one extra bit of functionality: Py_tp_bases allows the 
bases to be retrieved with PyType_GetSlot().

This isn't quite as applicable to the metaclass, since that can easily be 
retrieved with Py_TYPE(type).

> but is sometimes not correct (and it might not be obvious when it's not 
> correct).

Yes I guess that most all slots are ok to share across sub-interpreters.  I can 
see the argument for aiming to keep slots sub-interpreter-agnostic.

As a tangential point, I think that the DLL case on Windows may be a case where 
Windows is not compliant with the C standard: 
https://mail.python.org/archives/list/python-...@python.org/thread/2WUFTVQA7SLEDEDYSRJ75XFIR3EUTKKO/

Practically speaking this doesn't change anything (extensions that want to be 
compatible with Windows DLLs will still want to avoid this kind of 
initialization) but I think the docs may be incorrect on this point when they 
describe Windows as "strictly standard conforming in this particular behavior."

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Petr Viktorin]

Petr Viktorin  added the comment:

> But at least if it's available as a slot then a module is *able* to use it 
> with limited ABI going backwards. A new function doesn't allow that.

I think you're confusing PyType_Slot with the tp_* members of type structures.  
If a Py_tp_meta is added, it won't appear in past versions. See the end of 
Include/typeslots.h.
(Sadly, they're both called "slots".)

> Bases are available both as a slot (Py_tp_bases) and as an argument 
> (PyType_FromSpecWithBases).  I don't see why this has to be an either/or 
> proposition.  Both can be useful.

I consider Py_tp_bases to be a mistake: it's an extra way of doing things that 
doesn't add any extra functionality, but is sometimes not correct (and it might 
not be obvious when it's not correct).

> Either would satisfy my use case.

So let's go for the one that isn't a trap in the other use cases :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

> It's better to pass the metaclass as a function argument, as with bases. I'd 
> prefer adding a new function that using a slot.

Bases are available both as a slot (Py_tp_bases) and as an argument 
(PyType_FromSpecWithBases).  I don't see why this has to be an either/or 
proposition.  Both can be useful.

Either would satisfy my use case.  I'm constructing N such classes, so the spec 
won't be statically initialized anyway and the initialization issues on Windows 
don't apply.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Steve Dower]

Steve Dower  added the comment:

But at least if it's available as a slot then a module is *able* to use it with 
limited ABI going backwards. A new function doesn't allow that.

And yeah, it means they have to write more complex code than just a static 
array definition, but people are willing and able to do it.

I don't think there's anything about the current FromSpec that suggests all the 
parameters have to be interpreter-agnostic. And the design does suggest that 
we'll add more later without having to break the ABI (otherwise it would be a 
regular struct).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Petr Viktorin]

Petr Viktorin  added the comment:

- specs/slots are (usually) constant & static; pointers to types are not always 
C constant expressions (on Windows, if they come from a different DLL)
- specs/slots are (usually) shared across all subinterpreters; types are 
specific to a single interpreter

It's better to pass the metaclass as a function argument, as with bases. I'd 
prefer adding a new function that using a slot.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

> Passing the metaclass as a slot seems like the right idea for this API, 
> though I recall there being some concern about the API (IIRC, mixing function 
> pointers and data pointers doesn't work on some platforms?)

PyType_Slot is defined as a void* (not a function pointer): 
https://github.com/python/cpython/blob/8492b729ae97737d22544f2102559b2b8dd03a03/Include/object.h#L223-L226

So putting a PyTypeObject* into a slot would appear to be more kosher than 
function pointers.

Overall, a slot seems like a great first approach.  It doesn't require any new 
functions, which seems like a plus.  If the any linking issues a la tp_base are 
seen, a new function could be added later.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Sebastian Berg]

Sebastian Berg  added the comment:

I can make a PR from the patch (and add the `Py_tp_metaclass` slot if desired) 
with a basic test here, if that is what is blocking things.

Fixing the type and size of the allocation (as the patch does) would allow me 
to give people a way to create a new NumPy DType dynamically.  I only need the 
user to call my initialization function as soon as the type was created (with 
`PyType_FromSpec` or otherwise).
(And I can avoid any internal acrobatics creating the type for the user; this 
stuff tends to be super confusing even if the principles are fairly straight 
forward...)

Happy to pursue other avenues, but I am not clear which...


> IIRC, mixing function pointers and data pointers doesn't work on some 
> platforms?

... I guess it is too late to do some weird thing like (not sure it would be 
reasonable or is valid anyway though):

typedef union {
void *pdata;
void (*pfunc)(void);
} slot_value;

I am a bit interested in it, because I want to use a `FromSpec` API in NumPy 
and it would be nice to be sure I can grow it to include data without too much 
hassle.  But the easier thing may just be to add one or two `void *reserved` 
slot to the spec struct that must be NULL for now...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Steve Dower]

Steve Dower  added the comment:

Passing the metaclass as a slot seems like the right idea for this API, though 
I recall there being some concern about the API (IIRC, mixing function pointers 
and data pointers doesn't work on some platforms?) that mean it'll be 
deprecated in the future.

--
nosy: +steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Sebastian Berg]

Sebastian Berg  added the comment:

I am still fighting with this (and the issues surrounding it) for NumPy.  The 
main point is that my new DTypes in NumPy are metaclasses that extend the 
(heap)type struct.
That just feels right and matches the structure perfectly, but seems to get 
awkward when you want (users) to dynamically create new MetaClass instances.


It also works fine: I could allow creating new instances from Python (although 
I don't need it now) and allow users to create static types (for this I have to 
fix the size of the additional fields for a stable ABI, but that is fine, I can 
allocate a new opaque slot).

But cython or heaptypes (stable API) seem not accessible without annoying 
hacks...


For this concrete issue, would it be acceptable to use the base classes to 
correctly find the metaclass and use its alloc?
I personally don't see why a new signature would be needed even if the 
metaclass was to be passed (I do not need this), it seems like you could pass a 
`Py_tp_meta` slot and not add a new function?


I have attached a potential patch (it is a bit large because it needs to move 
the `bases` discovery code to before the `res` allocation).


Doing this would allow to provide a `FromSpec` function which internally calls 
`PyType_FromSpec`.
That may not make things much neater for cython, but it feels like this is 
really just a bug fix?  Unless you want to consider any extension of the type 
struct an unofficial hack to begin with :).


(Python metaclasses won't get their `__new__` called, but that is already the 
case, presumably a metaclass author will provide their own `InitFromSpec` 
function that must be called immediately after type creation, or just create 
the type completely.  I can do the second even now probably, but this tiny 
change would make it a lot cleaner.)


Trying more desperate angles, I was even wondering today if I should instead 
petition for Python to "donate" a `tp_metaslot` type slot...  A single `void *` 
unused by `PyType_Type` but available to metatypes to use as they wish.

While a bit strange, that might even simplify some ABI concerns or cython 
compatibility ;).

--
nosy: +seberg
Added file: https://bugs.python.org/file50285/typeobject.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Zachary Ware]

Change by Zachary Ware :


Removed file: 
https://bugs.python.org/file50281/MicrosoftOnlineServicesTerms(WW)(English)(February2021)(CR).docx

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Zachary Ware]

Change by Zachary Ware :


--
components:  -2to3 (2.x to 3.x conversion tool), Argument Clinic, Build, C API, 
Cross-Build, Demos and Tools, Distutils, Documentation, Extension Modules, 
FreeBSD, IDLE, IO, Installation, Library (Lib), Parser, Regular Expressions, 
SSL, Subinterpreters, Tests, Tkinter, Unicode, Windows, XML, asyncio, ctypes, 
email, macOS
nosy:  -Alex.Willmer, asvetlov, barry, dstufft, eric.araujo, ezio.melotti, 
koobs, larry, lys.nikolaou, mrabarnett, ned.deily, pablogsal, paul.moore, 
r.david.murray, ronaldoussoren, stephaniegilbert944, steve.dower, terry.reedy, 
tim.golden, vstinner, yselivanov, zach.ware
type: crash -> enhancement
versions:  -Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Stephanie Gilbert]

Change by Stephanie Gilbert :


--
components: +2to3 (2.x to 3.x conversion tool), Argument Clinic, Build, C API, 
Cross-Build, Demos and Tools, Distutils, Documentation, Extension Modules, 
FreeBSD, IDLE, IO, Installation, Library (Lib), Parser, Regular Expressions, 
SSL, Subinterpreters, Tests, Tkinter, Unicode, Windows, XML, asyncio, ctypes, 
email, macOS
nosy: +Alex.Willmer, asvetlov, barry, dstufft, eric.araujo, ezio.melotti, 
koobs, larry, lys.nikolaou, mrabarnett, ned.deily, pablogsal, paul.moore, 
r.david.murray, ronaldoussoren, stephaniegilbert944, steve.dower, terry.reedy, 
tim.golden, vstinner, yselivanov, zach.ware
type: enhancement -> crash
versions: +Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9
Added file: 
https://bugs.python.org/file50281/MicrosoftOnlineServicesTerms(WW)(English)(February2021)(CR).docx

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

I found a way to use metaclasses with the limited API.

I found that I can access PyType_Type.tp_new by creating a heap type derived 
from PyType_Type:

  static PyType_Slot dummy_slots[] = {
{0, NULL}
  };

  static PyType_Spec dummy_spec = {
  "module.DummyClass", 0, 0, Py_TPFLAGS_DEFAULT, dummy_slots,
  };

  PyObject *bases = Py_BuildValue("(O)", _Type);
  PyObject *type = PyType_FromSpecWithBases(_spec, bases);
  Py_DECREF(bases);

  type_new = PyType_GetSlot((PyTypeObject*)type, Py_tp_new);
  Py_DECREF(type);

  #ifndef Py_LIMITED_API
assert(type_new == PyType_Type.tp_new);
  #endif

  // Creates a type using a metaclass.
  PyObject *uses_metaclass = type_new(metaclass, args, NULL);

PyType_GetSlot() can't be used on PyType_Type directly, since it is not a heap 
type.  But a heap type derived from PyType_Type will inherit tp_new, and we can 
call PyType_GetSlot() on that.

Once we have PyType_Type.tp_new, we can use it to create a new type using a 
metaclass. This avoids any of the class-switching tricks I was trying before.  
We can also get other slots of PyType_Type like tp_getattro to do the 
equivalent of super().

The PyType_FromSpecEx() function proposed in this bug would still be a nicer 
solution to my problem.  Calling type_new() doesn't let you specify object size 
or slots.  To work around this, I derive from a type I created with 
PyType_FromSpec(), relying on the fact that the size and slots will be 
inherited.  This works, but it introduces an extra class into the hierarchy 
that ideally could be avoided.

But I do have a workaround that appears to work, and avoids the problems 
associated with setting ob_type directly (like PyPy incompatibility).

--
nosy: +haberman2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Christian Tismer]

Christian Tismer  added the comment:

Since PyPy does not use the Limited API, PySide can quite easily work around 
the limitations by directly working with the type object. 

But the usage of PyType_Modified() would make very much sense for PySide‘s new 
switchable features. That would work immediately without any change, because we 
already use that function to invalidate Python 3.10‘s type cache.

--
nosy: +Christian.Tismer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[mattip]

mattip  added the comment:

>> I wouldn't recommend [setting ob_type] after PyType_Ready is called.

> Why not?  What bad things will happen?  It seems to be working so far.

It breaks the unwritten contract that "once PyType_Ready is called, the C 
struct will not be modified". This is implemented in PyPy, since calling 
PyType_Ready creates the PyPy object in the interpreter based on the C 
structure. Any further changes will not be reflected in the PyPy interpreter 
object, so now the python-level and c-level objects do not agree what type(obj) 
is.

We have discussed this in the PyPy team, and would like to propose relaxing the 
contract to state that "if the c-level contents of an object are modified, 
PyType_Modified must be called to re-synce the python level and c-level objects"

--
nosy: +mattip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

> You can also call (PyObject_Call*) the metaclass with (name, bases, 
> namespace);

But won't that just call my metaclass's tp_new?  I'm trying to do this from my 
metaclass's tp_new, so I can customize the class creation process. Then Python 
code can use my metaclass to construct classes normally.

> I wouldn't recommend [setting ob_type] after PyType_Ready is called.

Why not?  What bad things will happen?  It seems to be working so far.

Setting ob_type directly actually solves another problem that I had been having 
with the limited API.  I want to implement tp_getattro on the metaclass, but I 
want to first delegate to PyType.tp_getattro to return any entry that may be 
present in the type's tp_dict.  With the full API I could call 
self->ob_type->tp_base->tp_getattro() do to the equivalent of super(), but with 
the limited API I can't access type->tp_getattro (and PyType_GetSlot() can't be 
used on non-heap types).

I find that this does what I want:

  PyTypeObject *saved_type = self->ob_type;
  self->ob_type = _Type;
  PyObject *ret = PyObject_GetAttr(self, name);
  self->ob_type = saved_type;

Previously I had tried:

   PyObject *super = PyObject_CallFunction((PyObject *)_Type, "OO",
   self->ob_type, self);
   PyObject *ret = PyObject_GetAttr(super, name);
   Py_DECREF(super);

But for some reason this didn't work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Petr Viktorin]

Petr Viktorin  added the comment:

> 2. Use eval from C to create the class with a metaclass, eg.
>  class Foo(metaclass=MessageMeta)

You can also call (PyObject_Call*) the metaclass with (name, bases, namespace); 
this should produce a class. Or not:

>>> class Foo(metaclass=print):
... def foo(self): pass
... 
Foo () {'__module__': '__main__', '__qualname__': 'Foo', 'foo': }

PyType_FromSpecEx will surely need to limit the metaclass to subtypes of type. 
What other limitations are there? How closely can we approach the behavior of 
the `class` statement in Python?


> 3. Manually set FooType->ob_type = 

I wouldn't recommend doing that after PyType_Ready is called. Including 
indirectly, which the type-creation functions in the stable ABI do.

--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Alexander Belopolsky]

Alexander Belopolsky  added the comment:

I'll reopen this issue to resume the discussion.  The motivating case - PEP 
3121 refactoring of the ctypes module - is still open.  See bpo-15884.

--
resolution: rejected -> 
stage: resolved -> patch review
status: closed -> open
versions: +Python 3.11 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Josh Haberman]

Josh Haberman  added the comment:

I know this is quite an old bug that was closed almost 10 years ago.  But I am 
wishing this had been accepted; it would have been quite useful for my case.

I'm working on a new iteration of the protobuf extension for Python.  At 
runtime we create types dynamically, one for each message defined in a .proto 
file, eg. from "message Foo" we dynamically construct a "class Foo".

I need to support class variables like Foo.BAR_FIELD_NUMBER, but I don't want 
to put all these class variables into tp_dict because there are a lot of them 
and they are rarely used.  So I want to implement __getattr__ for the class, 
which requires having a metaclass.  This is where the proposed 
PyType_FromSpecEx() would have come in very handy.

The existing protobuf extension gets around this by directly calling 
PyType_Type.tp_new() to create a type with a given metaclass:

https://github.com/protocolbuffers/protobuf/blob/53365065d9b8549a5c7b7ef1e7e0fd22926dbd07/python/google/protobuf/pyext/message.cc#L278-L279

It's unclear to me if PyType_Type.tp_new() is intended to be a supported/public 
API.  But in any case, it's not available in the limited API, and I am trying 
to restrict myself to the limited API.  (I also can't use 
PyType_GetSlot(PyType_Type, Py_tp_new) because PyType_Type is not a heap type.)

Put more succinctly, I do not see any way to use a metaclass from the limited C 
API.

Possible solutions I see:

1. Add PyType_FromSpecEx() (or similar with a better name) to allow a metaclass 
to be specified.  But I want to support back to at least Python 3.6, so even if 
this were merged today it wouldn't be viable for a while.

2. Use eval from C to create the class with a metaclass, eg.
  class Foo(metaclass=MessageMeta)

3. Manually set FooType->ob_type = , as recommended here: 
https://stackoverflow.com/a/52957978/77070 .  Since PyObject.ob_type is part of 
the limited API, I think this might be possible!

--
nosy: +jhaberman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Alexander Belopolsky]

Changes by Alexander Belopolsky alexander.belopol...@gmail.com:


--
resolution:  - rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Berker Peksag]

Changes by Berker Peksag berker.pek...@gmail.com:


--
stage: test needed - resolved

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[lekma]

Changes by lekma lekma...@gmail.com:


--
nosy: +lekma

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Jesús Cea Avión]

Changes by Jesús Cea Avión j...@jcea.es:


--
nosy: +jcea

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Antoine Pitrou]

Antoine Pitrou added the comment:

 If filling out a type with all slots one-by-one is considered too
 tedious, and patching ob_type too hacky - here is another approach:
 Use FromSpec to create a type with all slots filled out, then call the
 metatype to create a subtype of that. I.e. the type which is based on
 a metatype would actually be a derived class of the type which has the
 slots defined.

As a matter of fact, this is what the io module is doing (except that
the derived type is written in Python). It does feel like pointless
complication, though.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Alexander Belopolsky]

New submission from Alexander Belopolsky:

PyType_FromSpec() is a convenient function to create types dynamically in C 
extension modules, but its usefulness is limited by the fact that it creates 
new types using the default metaclass.

I suggest adding a new C API function

PyObject *PyType_FromSpecEx(PyObject *meta, PyType_Spec *spec)

and redefine PyType_FromSpec() as

PyType_FromSpecEx((PyObject *)PyType_Type, spec)


This functionality cannot be implemented by user because PyType_FromSpec 
requires access to private slotoffsets table.

A (trivial) patch attached.

--
assignee: belopolsky
components: Interpreter Core
files: typeobject.diff
keywords: needs review, patch
messages: 169925
nosy: belopolsky, loewis
priority: normal
severity: normal
stage: test needed
status: open
title: PyType_FromSpec should take metaclass as an argument
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file27137/typeobject.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc added the comment:

The patch is a bit light: see how type_new also computes the metaclass from the 
base classes.

--
nosy: +amaury.forgeotdarc

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Alexander Belopolsky]

Alexander Belopolsky added the comment:

On Thu, Sep 6, 2012 at 12:44 PM, Amaury Forgeot d'Arc
rep...@bugs.python.org wrote:
 The patch is a bit light: see how type_new also computes the metaclass from 
 the base classes.

This was intentional.  I was looking for a lightweight facility to
create heap types.  I know metaclass when I call PyType_FromSpec.  If
i wanted to invoke the metaclass from the base classes logic, I
would just specify an appropriate base class in the spec.  This would
still leave an open problem of specifying the metatype for the most
basic class.   This is the problem I am trying to solve.

--
nosy: +Alexander.Belopolsky

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Alexander Belopolsky]

Changes by Alexander Belopolsky alexander.belopol...@gmail.com:


--
nosy: +Robin.Schreiber

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Alexander Belopolsky]

Alexander Belopolsky added the comment:

 see how type_new also computes the metaclass from the base classes.

As you can see from my first message, I originally considered 
PyType_FromSpecEx(PyObject *meta, PyType_Spec *spec) without bases.  (In fact I 
was unaware of the recent addition of PyType_FromSpecWithBases.)  Maybe the 
original signature makes more sense than the one in the patch.  Explicitly 
setting a metaclass is most useful for the most basic type. On the other hand, 
a fully general function may eventually replace both PyType_FromSpec and 
PyType_FromSpecWithBases for most uses.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Martin v . Löwis]

Martin v. Löwis added the comment:

What is your use case for this API?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Alexander Belopolsky]

Alexander Belopolsky added the comment:

On Sep 6, 2012, at 5:10 PM, Martin v. Löwis rep...@bugs.python.org wrote:

 
 What is your use case for this API?
 

I can describe my use case, but it is somewhat similar to ctypes.   I searched 
the tracker for a PEP 3121 refactoring applied to ctypes and could not find 
any.   I'll try to come up with a PEP 3121 patch for ctypes using the proposed 
API.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Martin v . Löwis]

Martin v. Löwis added the comment:

If it's very special, I'm -0 on this addition. This sounds like this is 
something very few people would ever need, and they can learn to write more 
complicated code to achieve the same effect. Convenience API exists to make the 
common case convenient.

I'm -1 on calling it PyType_FromSpecEx.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Alexander Belopolsky]

Alexander Belopolsky added the comment:

On Sep 6, 2012, at 6:25 PM, Martin v. Löwis rep...@bugs.python.org wrote:

 I'm -1 on calling it PyType_FromSpecEx.

I find it encouraging that you commented on the choice of name. :-) I can live 
with PyType_FromMetatypeAndSpec and leave out bases.  PyType_FromTypeAndSpec is 
fine too. 

On the substance, I don't think this API is just convenience.  In my 
application I have to replace meta type after my type is created with 
PyType_FromSpec. This is fragile and works only for very simple metatypes.

Let's get back to this discussion once I have a ctypes patch.  I there will be 
a work-around for ctypes it will probably work for my case. (My case is a 
little bit more complicated because I extend the size of my type objects to 
store custom metadata.  Ctypes fudge this issue by hiding extra data in a 
custom tp_dict. )

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Antoine Pitrou]

Antoine Pitrou added the comment:

This API may make it easier to declare ABCs in C.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15870] PyType_FromSpec should take metaclass as an argument

[Martin v . Löwis]

Martin v. Löwis added the comment:

As for declaring ABCs: I don't think the API is necessary, or even helps. An 
ABC is best created by *calling* ABCMeta, with the appropriate name, a 
possibly-empty bases tuple, and a dict. What FromSpec could do is to fill out 
slots with custom functions, which won't be necessary or desirable for ABCs. 
The really tedious part may be to put all the abstract methods into the ABC, 
for which having a TypeSpec doesn't help at all. (But I would certainly agree 
that simplifying creation of ABCs in extension modules is a worthwhile reason 
for an API addition)

For the case that Alexander apparently envisions, i.e. metaclasses where the 
resulting type objects extend the layout of heap types: it should be possible 
for an extension module to fill out the entire type from scratch. This will 
require knowledge of the layout of heap types, so it can't use just the stable 
ABI - however, doing this through the stable ABI won't be possible, anyway, 
since the extended layout needs to know how large a HeapType structure is.

If filling out a type with all slots one-by-one is considered too tedious, and 
patching ob_type too hacky - here is another approach: Use FromSpec to create a 
type with all slots filled out, then call the metatype to create a subtype of 
that. I.e. the type which is based on a metatype would actually be a derived 
class of the type which has the slots defined.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15870
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com