Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
On Fri, Oct 13, 2017, at 02:30, Nick Coghlan wrote: > Metaclasses currently tend to serve two distinct purposes: > > 1. Actually altering the runtime behaviour of a class and its children >in non-standard ways (e.g. enums, ABCs, ORMs) > 2. Boilerplate reduction in class definitions, reducing the amount of >code you need to write as the author of that class > > Nobody has a problem with using metaclasses for the first purpose - > that's what they're for. > > It's the second use case where they're problematic, as the fact that > they're preserved on the class becomes a leaky implementation detail, > and the lack of a JIT in CPython means they can also end up being > expensive from a runtime performance perspective. What about a metaclass that isn't a metaclass? A metaclass can be any callable and can return any object, e.g. a normal type. def AutoSlotMeta(name, bases, dct, real_metaclass=type): """turn all class variables into slots""" dct['__slots__'] = list(dct) return real_metaclass(name, bases, dct) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
> > Date: Fri, 13 Oct 2017 08:57:00 -0700 > From: Guido van Rossum> To: Martin Teichmann > Cc: Python-Dev > Subject: Re: [Python-Dev] What is the design purpose of metaclasses vs > code generating decorators? (was Re: PEP 557: Data Classes) > Message-ID: > < > cap7+vjkbvudqf09ztwdauvq-ccnm+cf82c22s2njoj+a9k7...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > This is food for thought. I'll have to let it sink in a bit, but you may be > on to something. > > Since the question was asked at some point, yes, metaclasses are much older > than class decorators. At some point I found the book Putting Metaclasses > to Work by Ira Forman and Scott Danforth ( > https://www.amazon.com/Putting-Metaclasses-Work-Ira-Forman/dp/0201433052) > and translated the book's ideas from C++ to Python, except for the > automatic merging of multiple inherited metaclasses. > > But in many cases class decorators are more useful. > > I do worry that things like your autoslots decorator example might be > problematic because they create a new class, throwing away a lot of work > that was already done. But perhaps the right way to address this would be > to move the decision about the instance layout to a later phase? (Not sure > if that makes sense though.) > > --Guido > Just FYI, recreating the class with slots runs into problems with regards to PEP 3135 (New Super). In attrs we resort to black magic to update the __class__ cell in existing methods. Being able to add slotness later would be good, but not that useful for us since we have to support down to 2.7. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
On 10/13/2017 02:35 AM, Martin Teichmann wrote: Metaclasses currently tend to serve two distinct purposes: 1. Actually altering the runtime behaviour of a class and its children in non-standard ways (e.g. enums, ABCs, ORMs) 2. Boilerplate reduction in class definitions, reducing the amount of code you need to write as the author of that class Nobody has a problem with using metaclasses for the first purpose - that's what they're for. I am that nobody. The examples you give would be much nicer solved with decorators. The same holds for enums. Inheriting from enums is possible, but weird, given that you cannot add new enums to it. So, especially when comparing to the dataclasses, the following looks appealing to me: @enum class Breakfast: spam = 0 ham = 1 Things that will not work if Enum does not have a metaclass: list(EnumClass) -> list of enum members dir(EnumClass) -> custom list of "interesting" items len(EnumClass) -> number of members member in EnumClass -> True or False - protection from adding, deleting, and changing members - guards against reusing the same name twice - possible to have properties and members with the same name (i.e. "value" and "name") -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2017-10-06 - 2017-10-13) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open6251 (+26) closed 37280 (+37) total 43531 (+63) Open issues with patches: 2410 Issues opened (47) == #27172: Undeprecate inspect.getfullargspec() https://bugs.python.org/issue27172 reopened by larry #31718: some methods of uninitialized io.IncrementalNewlineDecoder obj https://bugs.python.org/issue31718 opened by Oren Milman #31721: assertion failure in FutureObj_finalize() after setting _log_t https://bugs.python.org/issue31721 opened by Oren Milman #31722: _io.IncrementalNewlineDecoder doesn't inherit codecs.Increment https://bugs.python.org/issue31722 opened by serhiy.storchaka #31724: test_xmlrpc_net should use something other than buildbot.pytho https://bugs.python.org/issue31724 opened by zach.ware #31725: Turtle/tkinter: NameError crashes ipython with "Tcl_AsyncDelet https://bugs.python.org/issue31725 opened by Rick J. Pelleg #31726: Missing token.COMMENT https://bugs.python.org/issue31726 opened by fpom #31727: FTP_TLS errors when https://bugs.python.org/issue31727 opened by jonathan-lp #31729: multiprocesssing.pool.AsyncResult undocumented field https://bugs.python.org/issue31729 opened by g...@nlc.co.nz #31731: [2.7] test_io hangs on x86 Gentoo Refleaks 2.7 https://bugs.python.org/issue31731 opened by haypo #31733: [2.7] Add PYTHONSHOWREFCOUNT environment variable to Python 2. https://bugs.python.org/issue31733 opened by haypo #31734: crash or SystemError in sqlite3.Cache in case it is uninitiali https://bugs.python.org/issue31734 opened by Oren Milman #31735: Documentation incorrectly states how descriptors are invoked https://bugs.python.org/issue31735 opened by Paul Pinterits #31737: Documentation renders incorrectly https://bugs.python.org/issue31737 opened by gibson042 #31738: Lib/site.py: method `abs_paths` is not documented https://bugs.python.org/issue31738 opened by lielfr #31739: socket.close recommended but not demonstrated in same-page exa https://bugs.python.org/issue31739 opened by Nathaniel Manista #31742: Default to emitting FutureWarning for provisional APIs https://bugs.python.org/issue31742 opened by ncoghlan #31743: Proportional Width Font on Generated Python Docs PDFs https://bugs.python.org/issue31743 opened by synthmeat #31745: Overloading "Py_GetPath" does not work https://bugs.python.org/issue31745 opened by kayhayen #31746: crashes in sqlite3.Connection in case it is uninitialized or p https://bugs.python.org/issue31746 opened by Oren Milman #31748: configure fails to detect fchdir() using CFLAGS="-Werror -Wall https://bugs.python.org/issue31748 opened by dilyan.palauzov #31749: Request: Human readable byte amounts in the standard library https://bugs.python.org/issue31749 opened by miserlou2 #31750: expose PyCell_Type in types module https://bugs.python.org/issue31750 opened by akvadrako #31751: Support for C++ 11 and/or C++ 14 in python.org installer https://bugs.python.org/issue31751 opened by hardkrash #31752: Assertion failure in timedelta() in case of bad __divmod__ https://bugs.python.org/issue31752 opened by serhiy.storchaka #31753: Unnecessary closure in ast.literal_eval https://bugs.python.org/issue31753 opened by Aaron Hall #31754: Documented type of parameter 'itemsize' to PyBuffer_FillContig https://bugs.python.org/issue31754 opened by snoeberger #31756: subprocess.run should alias universal_newlines to text https://bugs.python.org/issue31756 opened by andrewclegg #31757: Tutorial: Fibonacci numbers start with 1, 1 https://bugs.python.org/issue31757 opened by skyhein #31758: various refleaks in _elementtree https://bugs.python.org/issue31758 opened by Oren Milman #31760: Re-definition of _POSIX_C_SOURCE with Fedora 26. https://bugs.python.org/issue31760 opened by matrixise #31763: Add NOTICE level to the logging module https://bugs.python.org/issue31763 opened by mp5023 #31764: sqlite3.Cursor.close() crashes in case the Cursor object is un https://bugs.python.org/issue31764 opened by Oren Milman #31765: BUG: System deadlocks performing big loop operations in python https://bugs.python.org/issue31765 opened by Nik101 #31767: Windows Installer fails with error 0x80091007 when trying to i https://bugs.python.org/issue31767 opened by Igor.Skochinsky #31768: argparse drops '|'s when the arguments are too long https://bugs.python.org/issue31768 opened by caveman #31769: configure includes user CFLAGS when detecting pthreads support https://bugs.python.org/issue31769 opened by floppymaster #31770: crash and refleaks when calling sqlite3.Cursor.__init__() more https://bugs.python.org/issue31770 opened by Oren Milman #31771: tkinter geometry string +- when window ovelaps left or top of https://bugs.python.org/issue31771 opened by
Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
This is food for thought. I'll have to let it sink in a bit, but you may be on to something. Since the question was asked at some point, yes, metaclasses are much older than class decorators. At some point I found the book Putting Metaclasses to Work by Ira Forman and Scott Danforth ( https://www.amazon.com/Putting-Metaclasses-Work-Ira-Forman/dp/0201433052) and translated the book's ideas from C++ to Python, except for the automatic merging of multiple inherited metaclasses. But in many cases class decorators are more useful. I do worry that things like your autoslots decorator example might be problematic because they create a new class, throwing away a lot of work that was already done. But perhaps the right way to address this would be to move the decision about the instance layout to a later phase? (Not sure if that makes sense though.) --Guido On Fri, Oct 13, 2017 at 2:35 AM, Martin Teichmannwrote: > > Metaclasses currently tend to serve two distinct purposes: > > > > 1. Actually altering the runtime behaviour of a class and its children in > > non-standard ways (e.g. enums, ABCs, ORMs) > > 2. Boilerplate reduction in class definitions, reducing the amount of > code > > you need to write as the author of that class > > > > Nobody has a problem with using metaclasses for the first purpose - > that's > > what they're for. > > I am that nobody. The examples you give would be much nicer solved > with decorators. Especially for ABCs it would be much better, because > the fact that a class is an ABC is explicitly not inherited - its > entire reason of existence is to be inherited with all the > abstractness going away. And yet, currently the concrete class will > still inherit from ABC. The following way of writing ABCs looks much > nicer to me: > > @abstractclass > class Spam: > @abstractmethod > def ham(self): > ... > > The same holds for enums. Inheriting from enums is possible, but > weird, given that you cannot add new enums to it. So, especially when > comparing to the dataclasses, the following looks appealing to me: > > @enum > class Breakfast: > spam = 0 > ham = 1 > > I'm not an expert on ORMs, but they seem to me a lot like data classes > in the context of this discussion. > > I am aware that currently some things are easier done with > metaclasses. But given that decorators can create an entirely new > class if they like, they have all the power to do what they want, and > even in a way much easier understood by people. > > As an example, here the autoslot decorator: > > def autoslot(cls): > """turn all class variables into slots""" > cls.__slots__ = list(cls.__dict__) > return type(cls.__name__, cls.__bases__, class.__dict__) > > So I am personally more and more leaning towards a metaclass-free future. > > Cheers > > Martin > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scrapy Question
Hello. This mailing list is to work on developing Python (adding new features to Python itself and fixing bugs); if you're having problems learning, understanding or using Python, please find another forum. Probably python-list/comp.lang.python mailing list/news group is the best place; there are Python developers who participate in it; you may get a faster, and probably more complete, answer there. See http://www.python.org/community/ for other lists/news groups/fora. Thank you for understanding. On Fri, Oct 13, 2017 at 08:28:29PM +0530, Kishore Kumar Alajangiwrote: > Hi Experts, > > Could someone guide me how to use the code in below question(Link). > https://stackoverflow.com/questions/46711909/extract-urls-recursively-from-website-archives-in-scrapy > > Thanks, > KK. Oleg. -- Oleg Broytmanhttp://phdru.name/p...@phdru.name Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Scrapy Question
Hi Experts, Could someone guide me how to use the code in below question(Link). https://stackoverflow.com/questions/46711909/extract-urls-recursively-from-website-archives-in-scrapy Thanks, KK. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
On 13 October 2017 at 19:35, Martin Teichmannwrote: > > Metaclasses currently tend to serve two distinct purposes: > > > > 1. Actually altering the runtime behaviour of a class and its children in > > non-standard ways (e.g. enums, ABCs, ORMs) > > 2. Boilerplate reduction in class definitions, reducing the amount of > code > > you need to write as the author of that class > > > > Nobody has a problem with using metaclasses for the first purpose - > that's > > what they're for. > > I am that nobody. The examples you give would be much nicer solved > with decorators. Especially for ABCs it would be much better, because > the fact that a class is an ABC is explicitly not inherited - its > entire reason of existence is to be inherited with all the > abstractness going away. And yet, currently the concrete class will > still inherit from ABC. Aye, ABCs are definitely a case where I think it would be valuable to have a class decorator that: 1. Transplants any concrete method implementations from the ABC 2. Ensures that the class being defined actually implements all the required abstract methods The register method doesn't do either of those things, while inheritance has the unwanted side-effect of changing the metaclass of even concrete subclasses. As a handwavey concept, something like: @abc.implements(collections.Mapping) class MyMapping: ... # Just implement the abstract methods, get the rest injected So I am personally more and more leaning towards a metaclass-free future. > I agree with this view in the sense that I'd like the number of use cases that *require* a custom metaclass to get ever smaller (replacing them with regular inheritance + definition time method injection), but I also think they'll always have a place as a way for folks to explore the design space of what's possible given full control over the class definition process. That way, proposals like __init_subclass__ and __set_name__ can be based on pattern extraction from cases where people have decided the feature was valuable enough to be worth the hassle of maintaining a custom metaclass. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
While I really can't continue to be active in this discussion now [*], here are some thoughts based on observations I made: These three PEPs are all essentially solving an occurrence of the same problem: PEP 549 Instance descriptors PEP 562 Module __getattr__ PEP 560 Core support for generic types (the __class_getitem__ part) https://www.python.org/dev/peps/pep-0549/ https://www.python.org/dev/peps/pep-0562/ https://www.python.org/dev/peps/pep-0560/ PEPs 549 and 562 want an instance of ModuleType (i.e. modules) to define something on itself that looks like there was something defined on the class. For PEP 549 this is a property and for 562 it's a __getattr__ method. PEP 560 wants a __class_getitem__ method, defined on a class (instance of a metaclass), to look like there was a __getitem__ on the metaclass. PEP 560 is thus an attempt at a more fine-grained definition of a metaclass-like feature, where conflicts are less likely or can potentially be better dealth with. While PEPs 549 and 562 are doing a very similar thing to PEP 560 in theory, these use cases do not fall nicely into Nick's classification of uses for metaclasses in the email below. PEP 560 is trying to avoid a metaclass (a subclass of type) as an additional base class specifically for one class object). PEPs 549 and 562 are trying to avoid an additional class (a subclass of ModuleType) as an additional base class specifically for this one module. Whether or not fine-grainedness is the answer, it might make sense to list more different related use cases. Probably even the peps repo has more examples than the three I listed above. ––Koos [*] I'll try to be able to do what's nee½ded for the PEP 555 discussion — no1/3w still on python-ideas. On Fri, Oct 13, 2017 at 9:30 AM, Nick Coghlanwrote: > On 13 October 2017 at 04:21, Martin Teichmann > wrote: > >> For me, the dataclasses were a typical example for inheritance, to be >> more precise, for metaclasses. I was astonished to see them >> implemented using decorators, and I was not the only one, citing >> Guido: >> >> > I think it would be useful to write 1-2 sentences about the problem with >> > inheritance -- in that case you pretty much have to use a metaclass, >> and the >> > use of a metaclass makes life harder for people who want to use their >> own >> > metaclass (since metaclasses don't combine without some manual >> > intervention). >> >> Python is at a weird point here. At about every new release of Python, >> a new idea shows up that could be easily solved using metaclasses, yet >> every time we hesitate to use them, because of said necessary manual >> intervention for metaclass combination. >> > > Metaclasses currently tend to serve two distinct purposes: > > 1. Actually altering the runtime behaviour of a class and its children in > non-standard ways (e.g. enums, ABCs, ORMs) > 2. Boilerplate reduction in class definitions, reducing the amount of code > you need to write as the author of that class > > Nobody has a problem with using metaclasses for the first purpose - that's > what they're for. > > It's the second use case where they're problematic, as the fact that > they're preserved on the class becomes a leaky implementation detail, and > the lack of a JIT in CPython means they can also end up being expensive > from a runtime performance perspective. > > Mixin classes have the same problem: something that the author may want to > handle as an internal implementation detail leaks through to the runtime > state of the class object. > > Code generating decorators like functools.total_ordering and > dataclasses.dataclass (aka attr.s) instead aim at the boilerplate reduction > problem directly: they let you declare in the class body the parts that you > need to specify as the class designer, and then fill in at class definition > time the parts that can be inferred from that base. > > If all you have access to is the runtime class, it behaves almost exactly > as if you had written out all the autogenerated methods by hand (there may > be subtle differences in the method metadata, such as the values of > `__qualname__` and `__globals__`). > > Such decorators also do more work at class definition time in order to > reduce the amount of runtime overhead introduced by reliance on chained > method calls in a non-JITted Python runtime. > > As such, the code generating decorators have a clear domain of > applicability: boilerplate reduction for class definitions without > impacting the way instances behave (other than attribute and method > injection), and without implicitly impacting subclass definitions (other > than through regular inheritance behaviour). > > As far as the dataclass interaction with `__slots__` goes, that's a > problem largely specific to slots (and `__metaclass__` before it), in that > they're the only characteristics of a class definition that affect how > CPython allocates memory for the class object itself
Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
> Metaclasses currently tend to serve two distinct purposes: > > 1. Actually altering the runtime behaviour of a class and its children in > non-standard ways (e.g. enums, ABCs, ORMs) > 2. Boilerplate reduction in class definitions, reducing the amount of code > you need to write as the author of that class > > Nobody has a problem with using metaclasses for the first purpose - that's > what they're for. I am that nobody. The examples you give would be much nicer solved with decorators. Especially for ABCs it would be much better, because the fact that a class is an ABC is explicitly not inherited - its entire reason of existence is to be inherited with all the abstractness going away. And yet, currently the concrete class will still inherit from ABC. The following way of writing ABCs looks much nicer to me: @abstractclass class Spam: @abstractmethod def ham(self): ... The same holds for enums. Inheriting from enums is possible, but weird, given that you cannot add new enums to it. So, especially when comparing to the dataclasses, the following looks appealing to me: @enum class Breakfast: spam = 0 ham = 1 I'm not an expert on ORMs, but they seem to me a lot like data classes in the context of this discussion. I am aware that currently some things are easier done with metaclasses. But given that decorators can create an entirely new class if they like, they have all the power to do what they want, and even in a way much easier understood by people. As an example, here the autoslot decorator: def autoslot(cls): """turn all class variables into slots""" cls.__slots__ = list(cls.__dict__) return type(cls.__name__, cls.__bases__, class.__dict__) So I am personally more and more leaning towards a metaclass-free future. Cheers Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
On 13 October 2017 at 04:21, Martin Teichmannwrote: > For me, the dataclasses were a typical example for inheritance, to be > more precise, for metaclasses. I was astonished to see them > implemented using decorators, and I was not the only one, citing > Guido: > > > I think it would be useful to write 1-2 sentences about the problem with > > inheritance -- in that case you pretty much have to use a metaclass, and > the > > use of a metaclass makes life harder for people who want to use their own > > metaclass (since metaclasses don't combine without some manual > > intervention). > > Python is at a weird point here. At about every new release of Python, > a new idea shows up that could be easily solved using metaclasses, yet > every time we hesitate to use them, because of said necessary manual > intervention for metaclass combination. > Metaclasses currently tend to serve two distinct purposes: 1. Actually altering the runtime behaviour of a class and its children in non-standard ways (e.g. enums, ABCs, ORMs) 2. Boilerplate reduction in class definitions, reducing the amount of code you need to write as the author of that class Nobody has a problem with using metaclasses for the first purpose - that's what they're for. It's the second use case where they're problematic, as the fact that they're preserved on the class becomes a leaky implementation detail, and the lack of a JIT in CPython means they can also end up being expensive from a runtime performance perspective. Mixin classes have the same problem: something that the author may want to handle as an internal implementation detail leaks through to the runtime state of the class object. Code generating decorators like functools.total_ordering and dataclasses.dataclass (aka attr.s) instead aim at the boilerplate reduction problem directly: they let you declare in the class body the parts that you need to specify as the class designer, and then fill in at class definition time the parts that can be inferred from that base. If all you have access to is the runtime class, it behaves almost exactly as if you had written out all the autogenerated methods by hand (there may be subtle differences in the method metadata, such as the values of `__qualname__` and `__globals__`). Such decorators also do more work at class definition time in order to reduce the amount of runtime overhead introduced by reliance on chained method calls in a non-JITted Python runtime. As such, the code generating decorators have a clear domain of applicability: boilerplate reduction for class definitions without impacting the way instances behave (other than attribute and method injection), and without implicitly impacting subclass definitions (other than through regular inheritance behaviour). As far as the dataclass interaction with `__slots__` goes, that's a problem largely specific to slots (and `__metaclass__` before it), in that they're the only characteristics of a class definition that affect how CPython allocates memory for the class object itself (the descriptors for the slots are stored as a pointer array after the class struct, rather than only in the class dict). Given PEP 526 variable annotations, __slots__ could potentially benefit from a __metaclass__ style makeover, allowing an "infer_slots=True" keyword argument to type.__new__ to request that the list of slots be inferred from __annotations__ (Slot inference would conflict with setting class level default values, but that's a real conflict, as you'd be trying to use the same name on the class object for both the slot descriptor and the default value) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com