[Python-Dev] Re: PEP 642 v3: Explicit patterns for structured pattern matching
On Sat, 9 Jan 2021, 7:07 am Joseph Martinot-Lagarde, wrote: > Paul Sokolovsky wrote: > > Hello, > > On Tue, 5 Jan 2021 20:37:27 +1000 > > Nick Coghlan ncogh...@gmail.com wrote: > > > object(host=as host, port=as port}:", but that > > > couldn't ever be > > > I'd like to point out the weirdness of the "as" syntax when applied > > to > > positional arguments, e.g.: > > case [as x, as y]: > > case Cls(as x, as y): > > That feels unnatural, and the fact behind that intuitive feeling is > > that "as" never used like that in Python so far. Where it's used > > now, there's explicit "what" is available before "as": > > import org_name as alias_name > > with expr as var: > > So again, standalone "as" feels weird. > > It's a matter of taste, I like the compacity of the standalone as. It > describe clearly the variable name, and the order matches the positional > arguments so I don't find it surprising. If it's not the positional value, > what else can it be ? What would be clearer by using an underscore, which > itself corresponds to nothing ? > The fact that it was never used like this it not an argument per se, > because that's the point of new syntax... It's also a pure syntactic shortcut, so anyone that really dislikes it could favour the "__ as name" form. The extra leading "__" doesn't convey any information that the leading "as" doesn't convey on its own, but YMMV. The key difference relative to PEP 634 is that even when the code author uses the shorthand form, *readers* will still get at least the "as" keyword as a prompt, rather than having to just know that "name" appearing in a pattern means "__ as name", unless it's a class or attribute name in a class pattern, or an attribute name in a value pattern. While we know from experience that it's feasible to keep that in mind for iterable unpacking, it's far from clear that it will work as well for the more complex destructuring involved in class and mapping patterns. Cheers, Nick. > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Y3XRDKT7PUQ6EZSOAXUCIAALOILHCFP3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Heap types (PyType_FromSpec) must fully implement the GC protocol
Hi,
In the Python stdlib, many heap types currently don't "properly"
(fully?) implement the GC protocol which can prevent to destroy these
types at Python exit. As a side effect, some other Python objects can
also remain alive, and so are not destroyed neither.
There is an on-going effect to destroy all Python objects at exit
(bpo-1635741). This problem is getting worse when subinterpreters are
involved: Refleaks buildbots failures which prevent to spot other
regressions, and so these "leaks" / "GC bugs" must be fixed as soon as
possible. In my experience, many leaks spotted by tests using
subinterpreters were quite old, it's just that they were ignored
previously.
It's an hard problem and I don't see any simple/obvious solution right
now, except of *workarounds* that I dislike. Maybe the only good
solution is to fix all heap types, one by one.
== Only the Python stdlib should be affected ==
PyType_FromSpec() was added to Python 3.2 by the PEP 384 to define
"heap types" in C, but I'm not sure if it's popular in practice (ex:
Cython doesn't use it, but defines static types). I expect that most
types to still be defined the old style (static types) in a vas
majority of third party extension modules.
To be clear, static types are not affected by this email.
Third party extension modules using the limited C API (to use the
stable ABI) and PyType_FromSpec() can be affected (if they don't fully
implement the GC protocol).
== Heap type instances now stores a strong reference to their type ==
In March 2019, the PyObject_Init() function was modified in bpo-35810
to keep a strong reference (INCREF) to the type if the type is a heap
type. The fixed problem was that heap types could be destroyed before
the last instance is destroyed.
== GC and heap types ==
The new problem is that most heap types don't collaborate well with
the garbage collector. The garbage collector doesn't know anything
about Python objects, types, reference counting or anything. It only
uses the PyGC_Head header and the traverse functions. If an object
holds a strong reference to an object but its type does not define a
traverse function, the GC cannot guess/infer this reference.
A heap type must respect the following 3 conditions to collaborate with the GC:
* Have the Py_TPFLAGS_HAVE_GC flag;
* Define a traverse function (tp_traverse) which visits the type:
Py_VISIT(Py_TYPE(self));
* Instances must be tracked by the GC.
If one of these conditions is not met, the GC can fail to destroy a
type during a GC collection. If an instance is kept alive late while a
Python interpreter is being deleted, it's possible that the type is
never deleted, which can keep indirectly *many* objects alive and so
don't delete them neither.
In practice, when a type is not deleted, a test using subinterpreter
starts to fail on Refleaks buildbot since it leaks references. Without
subinterpreters, such leak is simply ignored, whereas this is an
on-going effect to delete Python objects at exit (bpo-1635741).
== Boring traverse functions ==
Currently, there is no default traverse implementation which visits the type.
For example, I had the implement the following function for _thread.LockType:
static int
lock_traverse(lockobject *self, visitproc visit, void *arg)
{
Py_VISIT(Py_TYPE(self));
return 0;
}
It's a little bit annoying to have to implement the GC protocol
whereas a lock cannot contain other Python objects, it's not a
container. It's just a thin wrapper to a C lock.
There is exactly one strong reference: to the type.
== Workaround: loop on gc.collect() ==
A workaround is to run gc.collect() in a loop until it returns 0 (no
object was collected).
== Traverse automatically? Nope. ==
Pablo Galindo attempts to automatically visit the type in the traverse function:
https://bugs.python.org/issue40217
https://github.com/python/cpython/commit/0169d3003be3d072751dd14a5c84748ab63a249f
Moreover, What's New in Python 3.9 contains a long section suggesting
to implement a traverse function for this problem, but it doesn't
suggest to track instances:
https://docs.python.org/dev/whatsnew/3.9.html#changes-in-the-c-api
This solution causes too many troubles, and so instead, traverse
functions were defined on heap types to visit the type.
Currently in the master branch, 89 types are defined as heap types on
a total of 206 types (117 types are defined statically). I don't think
that these 89 heap types respect the 3 conditions to collaborate with
the GC.
== How should we address this issue? ==
I'm not sure what should be done. Working around the issue by
triggering multiple GC collections? Emit a warning in development mode
if a heap type doesn't collaborate well with the GC?
If core developers miss these bugs and have troubles to debug them, I
expect that extension module authors would suffer even more.
== GC+heap type bugs became common ==
I'm fixing such GC issue for 1 year as part as the work on cleaning
Python objects at exit, and
[Python-Dev] 3.10 change (?) for __bool__
Someone reported a testsuite break on stuff I work on (scons) with 3.10a4, and it looks similar to this which appears in the changelog at https://docs.python.org/3.10/whatsnew/changelog.html#changelog bpo-23898: Fix inspect.classify_class_attrs() to support attributes with overloaded __eq__ and __bool__. Patch by Mike Bayer. Except when I go look at that BPO issue, it's old - closed 2015. Is this appearing in the 3.10 changelog in error? Sorry - confused !!! The test in question does indeed touch a class which overrides __bool_ in order to raise an exception (to say "don't do that"), and in the test run the (expected) exception is not raised. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VTK4DT4M26DKAPIAK6WYNWN4K45JH7IT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Story behind vars() vs .__dict__
On 9/01/21 9:12 am, Chris Barker wrote: (though I notice that if you create __slots__ in pure Python, its names show up in dict anyway -- so clearly I'm confused...) Descriptors for the slots get added to the *class* dict. But that's not the dict that vars() looks at. -- Greg ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IVGKEFTRXTSS7MHLH6G4RMJ4INM4LYZT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Unification of the Mac builds?
On Jan 8, 2021, at 14:38, Chris Barker via Python-Dev wrote: > Sorry if I'm out of the loop here, but with Apple's new chip coming out, we > need new a build configuration (which I think has already been started, if > not done). > > Perhaps we could take this opportunity to better modularize / unify the build > setup? [...] Chris: Since the topics you are touching on are fairly arcane macOS-only related issues and some of the issues you bring up will require clarification, I suggest you bring this up on the Python Mac sig mailing list (pythonmac-...@python.org) and let's discuss it there rather than here in python-dev. We can always summarize any recommendations back here if necessary. Thanks! --Ned -- Ned Deily n...@python.org -- [] ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H3RUV7SOCQ2QSWG3DTAMXDEQEBJQZRME/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 642 v3: Explicit patterns for structured pattern matching
Paul Sokolovsky wrote: > Hello, > On Tue, 5 Jan 2021 20:37:27 +1000 > Nick Coghlan ncogh...@gmail.com wrote: > > object(host=as host, port=as port}:", but that > > couldn't ever be > > I'd like to point out the weirdness of the "as" syntax when applied > to > positional arguments, e.g.: > case [as x, as y]: > case Cls(as x, as y): > That feels unnatural, and the fact behind that intuitive feeling is > that "as" never used like that in Python so far. Where it's used > now, there's explicit "what" is available before "as": > import org_name as alias_name > with expr as var: > So again, standalone "as" feels weird. It's a matter of taste, I like the compacity of the standalone as. It describe clearly the variable name, and the order matches the positional arguments so I don't find it surprising. If it's not the positional value, what else can it be ? What would be clearer by using an underscore, which itself corresponds to nothing ? The fact that it was never used like this it not an argument per se, because that's the point of new syntax... ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AM7X2UBI56YHVYFKWWODPJRDRSG5X3G7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Story behind vars() vs .__dict__
This was discussed a bit over on python-ideas recently, so a note from me, and one from that thread: Or for that matter, not the reason to provide > object's internal storage via object's attribute: obj.__dict__. > Well, it IS an implementation detail that it's a dictionary, but having a dunder that gives you the object's namespace seems like a specification to me -- would you be happier to call it obj.__names__ ? And as for having vars() be the way to get that instead -- it's pretty much universal in Python that dunders provide the implementation, while functions (such as len()) provide a universal and clean interface to the functionality -- so vars() seems quite consistent in that sense. However, as pointed out on python-ideas, some objects have a .__dict__ and/or .__slots__ -- and THAT distinction seems to be an implementation issue leaking out. Maybe it would be too backward incompatible, but it makes sense to me that vars() should return the union of .__dict__ and .__slots__ (though I notice that if you create __slots__ in pure Python, its names show up in dict anyway -- so clearly I'm confused...) But the question was about (thru-the-time) relationship of vars() > vs .__dict__ ... > I believe Guido's recollection is that vars() was intended to provide easy access, particularly in a REPL. And that accessing .__dict__ directly in code is perfectly reasonable, since you only need to do that when you are metaprogramming anyway -- which inherently requires some knowledge of internal structure of objects anyway. I'd be interested to hear if any other folks that have been around a long time have other thoughts. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MG3BSQR7ZTVQV6OGJWUDHE4NBQVFTCSK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: unittest of sequence equality
On Wed, Dec 23, 2020 at 1:06 AM Steven D'Aprano wrote: > We're not obligated to take heroic > measures to integrate numpy arrays with unittest methods. If we can do > so easily, sure, let's fix it. > > I think Ivan's suggestion that the assertSequenceEqual method fall back > on element-by-element comparisons has some merit. > If there are other common types this helps with, sure. But for numpy, as pointed out elsewhere in this thread, it would still fail for numpy arrays of > 1 dimension. Personally I think this is really an issue with the structure of unitest -- having a custom assertion for every possibility is intractable. If you want to test numpy arrays, use the utilities provided by numpy. - CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NSZ7GLILVVITHOXIARWPVCMYHAEI7PE2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Unification of the Mac builds?
Sorry if I'm out of the loop here, but with Apple's new chip coming out, we need new a build configuration (which I think has already been started, if not done). Perhaps we could take this opportunity to better modularize / unify the build setup? As it was last I checked, you really had only two options: 1) Ignore anything mac specific, and get a "unix" build. 2) Get a full "Framework" build, optionally with Universal support. It would be nice to keep the Framework structure independent of the Mac-specific features, if possible. In particular, I'd love to get be able to get the "pythonw" executable wrapper in an otherwise standard unix build [*]. It would also be nice if it were possible to get universal binaries in a "unix style" build. (option 3 would be to simply abandon the Framework Build altogether -- it's still not clear to me what this really buys mac users) Any chance of this happening? I'm afraid I know nothing of autoconf, so can't be much help, but I'd be willing to help out with testing, or documenting, or anything else that I have the skills to do. Thanks, -Chris [*] The pythonw issue has been a thorn in the side of conda for years. conda uses a standard unix build on the Mac, for consistency with other unix systems. But there is no configuration set up to build the pythonw wrapper outside of a "Framework" build. So instead, conda has creates its own "pythonw" wrapper -- but that is a bash script that re-directs to a different executable. This works fine on the command line (or #! line), but it does not work with setuptools' entry_points. And the setuptools community hasn't shown any interest in hacking around it. Anyway, few enough people develop Desktop apps (particularly with conda) so this has lingered, but it would be nice to fix. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S27ZRILE4QNVG7OZ3ROQW3TB7GRQHFVL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2021-01-01 - 2021-01-08) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open7549 ( -7) closed 47055 (+71) total 54604 (+64) Open issues with patches: 3026 Issues opened (39) == #42807: smtplib send_message should gives more clear exception if the https://bugs.python.org/issue42807 opened by oon #42808: Add PyType_Type.tp_vectorcall for type(obj) performance https://bugs.python.org/issue42808 opened by Dennis Sweeney #42809: Improve pickle tests for recursive data https://bugs.python.org/issue42809 opened by serhiy.storchaka #42812: @overload-ing method of parent class without actual implementa https://bugs.python.org/issue42812 opened by chaim422 #42813: Extra spaces cause unexpected EOF error in "compile" function https://bugs.python.org/issue42813 opened by xxm #42815: new thread doesn't copy context of the parent thread https://bugs.python.org/issue42815 opened by uriyyo #42819: readline 8.1 bracketed paste https://bugs.python.org/issue42819 opened by dtrodrigues #42820: Sphinx conf.py needs_version entry is outdated https://bugs.python.org/issue42820 opened by timhoffm #42823: Incorrect frame.f_lineno when frame.f_trace is set https://bugs.python.org/issue42823 opened by Mark.Shannon #42825: Build libraries with "/OPT:REF" linker optimization on Windows https://bugs.python.org/issue42825 opened by Austin-Lamb #42827: pegen parser: Multiline eval with assignment to function call: https://bugs.python.org/issue42827 opened by mdk #42830: tempfile mkstemp() leaks file descriptors if os.close() is not https://bugs.python.org/issue42830 opened by mieczyslaw.torchala #42831: IDLE fix colours for MacOS dark mode https://bugs.python.org/issue42831 opened by epaine #42832: classmethod change in 3.9 needs better documentation https://bugs.python.org/issue42832 opened by iritkatriel #42833: Lib/urllib/request.py: digest algorithm should be case insensi https://bugs.python.org/issue42833 opened by Pierre.Tardy #42834: [subinterpreters] Convert "global" static variable caches in _ https://bugs.python.org/issue42834 opened by kj #42836: docs: struct: clarify struct caching behaviour https://bugs.python.org/issue42836 opened by uniq10 #42837: Symbol table incorrectly identifies code as a generator, when https://bugs.python.org/issue42837 opened by Mark.Shannon #42838: Wait for cleanup coroutines before event loop is closed. https://bugs.python.org/issue42838 opened by xloem #42839: SourceFileLoader does not (fully) accept path-like objects https://bugs.python.org/issue42839 opened by favonia #42840: `type` takes **kwargs for __init_subclass__ https://bugs.python.org/issue42840 opened by esoma #42843: What min_sphinx for Python 3.10 https://bugs.python.org/issue42843 opened by mdk #42844: Turtle Module -- "onclick" arguments enchancement https://bugs.python.org/issue42844 opened by sc1341 #42848: asyncio produces an unexpected traceback with recursive __geta https://bugs.python.org/issue42848 opened by asleep-cult #42849: pool worker can't be terminated https://bugs.python.org/issue42849 opened by huangzhesi #42850: Process hangs when calling urllib.request in a multiprocessing https://bugs.python.org/issue42850 opened by fakufaku #42851: Subclassing Enum with ipaddress.IPv4Network/IPv6Network raises https://bugs.python.org/issue42851 opened by felixxm #42853: `OverflowError: signed integer is greater than maximum` in ssl https://bugs.python.org/issue42853 opened by amacd31 #42854: OpenSSL 1.1.1: use SSL_write_ex() and SSL_read_ex() https://bugs.python.org/issue42854 opened by christian.heimes #42855: pathlib.exists on Windows raises an exception on URL like/bad https://bugs.python.org/issue42855 opened by gaborjbernat #42856: ensurepip: add configure --with-wheel-pkg-dir=PATH to get whee https://bugs.python.org/issue42856 opened by vstinner #42859: smtplib: recipe for certifying message delivery https://bugs.python.org/issue42859 opened by dgoldenberg123 #42861: ipaddress - add ability to get next closet subnet of any prefi https://bugs.python.org/issue42861 opened by fasial.mahmood94 #42862: Use functools.lru_cache iso. _sqlite.Cache in sqlite3 module https://bugs.python.org/issue42862 opened by erlendaasland #42863: Python venv inconsistent folder structure on windows https://bugs.python.org/issue42863 opened by jmoguill2 #42864: Improve error messages regarding unclosed parentheses https://bugs.python.org/issue42864 opened by pablogsal #42865: sysconfig appends CFLAGS to LD https://bugs.python.org/issue42865 opened by Greentwip #42867: Entry Widget not editable on Windows 10, but is on Linux Ubunt https://bugs.python.org/issue42867 opened by jmccabe #42868: SpooledTemporaryFile.__iter__ is not transparent to rollover https://bugs.python.org/issue42868
