Re: making your own DirEntry.
Antoon,
On 12/23/23 01:00, Antoon Pardon via Python-list wrote:
I am writing a program that goes through file hierarchies and I am mostly
using scandir for that which produces DirEntry instances.
At times it would be usefull if I could make my own DirEntry for a specific
path, however when I try, I get the following diagnostic:
os.DirEntry('snap')
Traceback (most recent call last):
File "", line 1, in
TypeError: cannot create 'posix.DirEntry' instances
Does anyone have an idea for why this limitation and how to go around it.
At this moment I don't consider pathlib very usefull, it lacks the
follow_symlinks parameter in the is_dir, is_file, ... methods.
Can't recall ever trying this.
The manual (https://docs.python.org/3/library/os.html#os.DirEntry)
suggests that a DirEntry is one of those Python data-constructs which it
creates, but we may only use: "cannot create".
Secondly, that a DirEntry object consists of a lot more than the
directory-name, eg its path.
Thirdly, that os.scandir() deals (only) with concrete directories -
unlike pathlib's ability to work with both the real thing and abstract
files/dirs.
Why create a DirEntry? Why not go directly to os.mkdir() or whatever?
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list
Re: How/where to store calibration values - written by program A, read by program B
Apologies: neglected suggested web.refs: https://datagy.io/python-environment-variables/ https://pypi.org/project/json_environ/ -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: How/where to store calibration values - written by program A, read by program B
On 12/6/23 03:37, Chris Green via Python-list wrote: Is there a neat, pythonic way to store values which are 'sometimes' changed? My particular case at the moment is calibration values for ADC inputs which are set by running a calibration program and used by lots of programs which display the values or do calculations with them. From the program readability point of view it would be good to have a Python module with the values in it but using a Python program to write/update a Python module sounds a bit odd somehow. I could simply write the values to a file (or a database) and I suspect that this may be the best answer but it does make retrieving the values different from getting all other (nearly) constant values. Are there any Python modules aimed specifically at this sort of requirement? Another programming-term for these might be "environment variables". However, be aware that such also has a specific meaning at the Operating System level. 1 Sysops Environment Variables exist completely outside the code. Python interrogates the Sysops to fetch/set them. Can be problematic because only apply on single machine and are not part of change-control, VS, etc. 2 A .con file (in my tradition, likely still .uni type in MSFT) or similar, which contains the key:value pairs recommended elsewhere. There are formal .con and .uni (etc) formats. Most of the teams I've worked-with recently seem to settle on .JSON files which are very-conveniently structured as (read into/written from) a Python dictionary, and a single function-call interaction. Word of warning/voice of [bitter] experience: ALWAYS *trumpet* any changes in these values AND output them (as you would any "assumptions" for a calculation) at the top of output reports. The trouble is that humans assume continuity but such an arrangement is NOT idempotent - which leads to complaints: "I ran it on Monday and got these results, but when I ran it again on Tuesday, the results changed"... Yes there are Python libraries. Choose your method/format first, and then search - either Duckboards or straight from Pepi. I have systems which use an DBMS for environment variables, but (a) only when there's a significant number, and (b) when the application is already connecting to the DBMS for processing [and maybe (c) because I know my way around such tools so they're 'easy']. Not recommended! -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Newline (NuBe Question)
Avi, On 11/27/2023 4:15 PM, avi.e.gr...@gmail.com wrote: Dave, Back on a hopefully more serious note, I want to make a bit of an analogy with what happens when you save data in a format like a .CSV file. Often you have a choice of including a header line giving names to the resulting columns, or not. If you read in the data to some structure, often to some variation I would loosely call a data.frame or perhaps something like a matrix, then without headers you have to specify what you want positionally or create your own names for columns to use. If names are already there, your program can manipulate things by using the names and if they are well chosen, with no studs among them, the resulting code can be quite readable. More importantly, if the data being read changes and includes additional columns or in a different order, your original program may run fine as long as the names of the columns you care about remain the same. Positional programs can be positioned to fail in quite subtle ways if the positions no longer apply. Must admit to avoiding .csv files, if possible, and working directly with the .xls? original (cf expecting the user to export the .csv - and NOT change the worksheet thereafter). However, have recently been using the .csv format (as described) as a placeholder or introduction to formatting data for an RDBMS. In a tabular structure, the expectation is that every field (column/row intersection) will contain a value. In the RDBMS-world, if the value is not-known then it will be recorded as NULL (equivalent of Python's None). Accordingly, two points: 1 the special case of missing/unavailable data can be handled with ease, 2 most 'connector' interfaces will give the choice of retrieving data into a tuple or a dictionary (where the keys are the column-names). The latter easing data-identification issues (as described) both in terms of improving over relational-positioning and name-continuity (or column changes/expansions). The point about data 'appearing' without headings should be considered carefully. The phrase "create your own names for columns" only vaguely accesses the problem. If someone else has created/provided the data, then we need to know the exact design (schema = rules). What is the characteristic of each component? Not only column-names, but also what is the metric (eg the infamous confusion between feet and meters)... As I see it, many situations where some aspects are variable are not ideal for naming. A dictionary is an example that is useful when you have no idea how many items with unknown keys may be present. You can iterate over the names that are there, or use techniques that detect and deal with keys from your list that are not present. Not using names/keys here might involve a longer list with lots of empty slots to designate missing items, This clearly is not great when the data present is sparse or when the number of items is not known in advance or cannot be maintained in the right order. Agreed, and this is the draw-back incurred by folk who wish to take advantage of the schema-less (possibility) NoSQL DBs. The DB enjoys flexibility, but the downstream-coder has to contort and flex to cope. In this case, JSON files are an easy place-holder/intro for NoSQL DBs - in fact, Python dicts and MongoDB go hand-in-glove. The next issue raised is sparseness. In a table, the assumption is that all fields, or at least most of them, will be filled with values. However, a sparse matrix would make such very 'expensive' in terms of storage-space (efficacy). Accordingly, there are other ways of doing things. All of these involve labeling each data-item (thus, the data expressed as a table needs to be at least 50% empty to justify the structural change). In this case, one might consider a tree-type of structure - and if we have to continue the pattern, we might look at a Network Database methodology (as distinct from a DB on a network!) There are many other situations with assorted tradeoffs and to insist on using lists/tuples exclusively would be silly but at the same time, if you are using a list to hold the real and imaginary parts of a complex number, or the X/Y[/Z] coordinates of a point where the order is almost universally accepted, then maybe it is not worth using a data structure more complex or derived as the use may be obvious. No argument (in case anyone thought I might...) See @Peter's earlier advice. Much of the consideration (apart from mutable/immutable) is likely to be ease of coding. Getting down 'into the weeds' is probably pointless unless questions are being asked about (execution-time) performance... Isn't the word "obvious" where this discussion started? Whereas "studs" might be an "obvious" abbreviation for "students" to some, it is not to others (quite aside from the abbreviation being unnecessary in this day-and-age). Curiously, whereas I DO happen to think a point as ( x, y, ) or
Re: Newline (NuBe Question)
On 11/27/2023 10:04 AM, Peter J. Holzer via Python-list wrote: On 2023-11-25 08:32:24 -0600, Michael F. Stemper via Python-list wrote: On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote: Of course, for serious work, some might suggest avoiding constructs like a list of lists and switch to using modules and data structures [...] Those who would recommend that approach do not appear to include Mr. Rossum, who said: Avoid overengineering data structures. ^^^ The key point here is *over*engineering. Don't make things more complicated than they need to be. But also don't make them simpler than necessary. Tuples are better than objects (try namedtuple too though). If Guido thought that tuples would always be better than objects, then Python wouldn't have objects. Why would he add such a complicated feature to the language if he thought it was useless? The (unspoken?) context here is "if tuples are sufficient, then ..." At recent PUG-meetings I've listened to a colleague asking questions and conducting research on Python data-structures*, eg lists-of-lists cf lists-of-tuples, etc, etc. The "etc, etc" goes on for some time! Respecting the effort, even as it becomes boringly-detailed, am encouraging him to publish his findings. * sadly, he is resistant to OOP and included only a cursory look at custom-objects, and early in the process. His 'new thinking' has been to look at in-core databases and the speed-ups SQL (or other) might offer... However, his motivation came from a particular application, and to create a naming-system so that he could distinguish a list-of-lists structure from some other tabular abstraction. The latter enables the code to change data-format to speed the next process, without the coder losing-track of the data-type/format. The trouble is, whereas the research reveals which is faster (in-isolation, and (only) on his 'platform'), my suspicion is that he loses all gains by reformatting the data between 'the most efficient' structure for each step. A problem of only looking at the 'micro', whilst ignoring wider/macro concerns. Accordingly, as to the word "engineering" (above), a reminder that we work in two domains: code and data. The short 'toy examples' in training courses discourage us from a design-stage for the former - until we enter 'the real world' and meet a problem/solution too large to fit in a single human-brain. Sadly, too many of us are pre-disposed to be math/algorithmically-oriented, and thus data-design is rarely-considered (in the macro!). Yet, here we are... -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Newline (NuBe Question)
On 11/27/2023 1:08 AM, Roel Schroeven via Python-list wrote: I prefer namedtuples or dataclasses over tuples. They allow you to refer to their fields by name instead of index: student.gpa is much clearer than student[2], and makes it less likely to accidentally refer to the wrong field. +1 readability/comprehension! -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Newline (NuBe Question)
On 11/27/2023 12:48 AM, Chris Angelico via Python-list wrote: On Sun, 26 Nov 2023 at 21:08, Michael F. Stemper via Python-list wrote: On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote: Grizz[l]y, I think the point is not about a sorted list or sorting in general It is about reasons why maintaining a data structure such as a list in a program can be useful beyond printing things once. There are many possible examples such as having a list of lists containing a record where the third item is a GPA for the student and writing a little list comprehension that selects a smaller list containing only students who are Magna Cum Laude or Summa Cum Laude. studs = [ ["Peter", 82, 3.53], ["Paul", 77, 2.83], ["Mary", 103, 3.82] ] I've seen Mary, and she didn't look like a "stud" to me. That's what happens when you abbreviate "student" though :) Don't worry, there's far FAR worse around the place, and juvenile brains will always find things to snigger at, usually in mathematical libraries with "cumulative" functions. The OP used an abbreviation: "studs". Why? Too lazy to type the full word? Abbreviation has full-meaning in the (narrow) domain? Was wanting something funny, or to snigger over? Was the respondent sniggering? Perhaps he, like the OP, was also saving typing-time by making a joke, hoping that the OP would see the implicit-error in expecting others to understand that "studs" meant "students"? Actually, Peter, Paul, and Mary were a band (https://www.peterpaulandmary.com/), so "studs" is even less expressive when the data also tells a story... Working with "trainees", I avoid the word "student" even though some might see them as synonyms. In my mind, the abbreviation did not readily expand to the full word (mea culpa). Accordingly, would not pass Code Review! For the want of a few characters... (https://en.wikipedia.org/wiki/For_Want_of_a_Nail) -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Printing dict value for possibly undefined key
On 11/25/2023 3:31 AM, Loris Bennett via Python-list wrote:
Hi,
I want to print some records from a database table where one of the
fields contains a JSON string which is read into a dict. I am doing
something like
print(f"{id} {d['foo']} {d['bar']}")
However, the dict does not always have the same keys, so d['foo'] or
d['bar'] may be undefined. I can obviously do something like
if not 'foo' in d:
d['foo']="NULL"
if not 'bar' in d:
d['bar']="NULL"
print(f"{id} {d['foo']} {d['bar']}")
Is there any more compact way of achieving the same thing?
What does "the dict does not always have the same keys" mean?
a) there are two (or...) keys, but some records don't include both;
b) there may be keys other than 'foo' and 'bar' which not-known in-advance;
c) something else.
As mentioned, dict.get() solves one of these.
Otherwise, there are dict methods which collect/reveal all the keys, all
the values, or both - dict.keys(), .values(), .items(), resp.
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list
[Python-announce] PyConZA 2023 - Second Call for Submissions
This is a second call for submissions to PyConZA 2023. PyConZA 2023 will take place on the 5th & 6th of October, 2023. This year, PyConZA will be a hybrid conference (with in-person and online access) hosted at the Premier Splendid Inn in Umhlanga, Durban. We are looking for the following presentations: - Keynotes (45 minute long talks on a subject of general interest) - Talks (30 minute long talks on more specific topics) - Remote Talks (30 minute long talks that will be delivered remotely - note that the number of remote talk slots is more limited due to technical constraints). If you would like to give a presentation, please register at https://za.pycon.org/ and submit your proposal, following the instructions at https://za.pycon.org/talks/how-to-submit-a-talk/ . We have a number of tracks available, including: Data Science, Teaching and Learning with Python, Web, Scientific Computing, Testing and Other (which includes all talks that don't fall under the mentioned tracks). We hope to notify accepted presenters by no later than the 31st of August 2023. Speakers will be expected to be available after the presentation for a short Q session. Shared sessions are also possible. The presentations will be in English. PyConZA offers a mentorship program for inexperienced speakers. If you would like assistance preparing your submission, email t...@za.pycon.org with a rough draft of your talk proposal and we'll find a suitable experienced speaker to act as a mentor. If you want to present something that doesn't fit into the standard talk categories at PyConZA, please contact the organising committee at t...@za.pycon.org so we can discuss whether that will be feasible -- Neil Muller On behalf of the PyConZA organising committee ___ Python-announce-list mailing list -- python-announce-list@python.org To unsubscribe send an email to python-announce-list-le...@python.org https://mail.python.org/mailman3/lists/python-announce-list.python.org/ Member address: arch...@mail-archive.com
[Python-announce] PyConZA 2023 - Call for Submissions
PyConZA 2023 will take place on the 5th & 6th of October, 2023. This year, PyConZA will be a hybrid conference (with in-person and online access) hosted at the Premier Splendid Inn in Umhlanga, Durban. We are looking for the following presentations: - Keynotes (45 minute long talks on a subject of general interest) - Talks (30 minute long talks on more specific topics) - Remote Talks (30 minute long talks that will be delivered remotely - note that the number of remote talk slots is more limited due to technical constraints). If you would like to give a presentation, please register at https://za.pycon.org/ and submit your proposal, following the instructions at https://za.pycon.org/talks/how-to-submit-a-talk/ . We have a number of tracks available, including: Data Science, Teaching and Learning with Python, Web, Scientific Computing, Testing and Other (which includes all talks that don't fall under the mentioned tracks). We hope to notify accepted presenters by no later than the 31st of August 2023. Speakers will be expected to be available after the presentation for a short Q session. Shared sessions are also possible. The presentations will be in English. PyConZA offers a mentorship program for inexperienced speakers. If you would like assistance preparing your submission, email t...@za.pycon.org with a rough draft of your talk proposal and we'll find a suitable experienced speaker to act as a mentor. If you want to present something that doesn't fit into the standard talk categories at PyConZA, please contact the organising committee at t...@za.pycon.org so we can discuss whether that will be feasible. -- Neil Muller On behalf of the PyConZA organising committee ___ Python-announce-list mailing list -- python-announce-list@python.org To unsubscribe send an email to python-announce-list-le...@python.org https://mail.python.org/mailman3/lists/python-announce-list.python.org/ Member address: arch...@mail-archive.com
[Python-announce] PyConZA 2022 - Second Call for Submissions
This is a second call for submissions to PyConZA 2022. PyConZA 2022 will take place on the 13th & 14th of October, 2022. This year, PyConZA will be a hybrid conference (with in-person and online access) hosted at the Premier Splendid Inn in Umhlanga, Durban. To accommodate speakers who are unable to travel to Durban, we will be accepting a small number of talks to be given remotely. We are looking for the following presentations: - Keynotes (45 minute long talks on a subject of general interest) - Talks (30 minute long talks on more specific topics) - Remote Talks: (30 minute talks to be delivered remotely - note that the number of remote submissions we can accommodate is limited). We are accepting submissions for tutorials, which will run on the 12th of October. Tutorials can either be half-day (4 hours) or full-day (8 hours). If you would like to give a presentation, please register at https://za.pycon.org/ and submit your proposal, following the instructions at https://za.pycon.org/talks/submit-talk/ . We have a number of tracks available, including: Data Science, Teaching and Learning with Python, Web, Scientific Computing, Testing and Other (which includes all talks that don't fall under the mentioned tracks). We hope to notify accepted presenters by no later than the 14th of September 2022. Speakers will be expected to be available after the presentation for a short Q session. Shared sessions are also possible. The presentations will be in English. PyConZA offers a mentorship program for inexperienced speakers. If you would like assistance preparing your submission, email t...@za.pycon.org with a rough draft of your talk proposal and we'll find a suitable experienced speaker to act as a mentor. If you want to present something that doesn't fit into the standard talk categories at PyConZA, please contact the organising committee at t...@za.pycon.org so we can discuss whether that will be feasible. -- Neil Muller On behalf of the PyConZA organising committee ___ Python-announce-list mailing list -- python-announce-list@python.org To unsubscribe send an email to python-announce-list-le...@python.org https://mail.python.org/mailman3/lists/python-announce-list.python.org/ Member address: arch...@mail-archive.com
[Python-announce] PyConZA 2022 - Call for Submissions
PyConZA 2022 will take place on the 13th & 14th of October, 2022. This year, PyConZA will be a hybrid conference (with in-person and online access) hosted at the Premier Splendid Inn in Umhlanga, Durban. We are looking for the following presentations: - Keynotes (45 minute long talks on a subject of general interest) - Talks (30 minute long talks on more specific topics) We are accepting submissions for tutorials, which will run on the 12th of October. Tutorials can either be half-day (4 hours) or full-day (8 hours). Currently, we are only accepting talks to be delivered in person at the venue. If you would like to give a presentation, please register at https://za.pycon.org/ and submit your proposal, following the instructions at https://za.pycon.org/talks/submit-talk/ . We have a number of tracks available, including: Data Science, Teaching and Learning with Python, Web, Scientific Computing, Testing and Other (which includes all talks that don't fall under the mentioned tracks). We hope to notify accepted presenters by no later than the 31st of August 2022. Speakers will be expected to be available after the presentation for a short Q session. Shared sessions are also possible. The presentations will be in English. PyConZA offers a mentorship program for inexperienced speakers. If you would like assistance preparing your submission, email t...@za.pycon.org with a rough draft of your talk proposal and we'll find a suitable experienced speaker to act as a mentor. If you want to present something that doesn't fit into the standard talk categories at PyConZA, please contact the organising committee at t...@za.pycon.org so we can discuss whether that will be feasible. -- Neil Muller On behalf of the PyConZA organising committee ___ Python-announce-list mailing list -- python-announce-list@python.org To unsubscribe send an email to python-announce-list-le...@python.org https://mail.python.org/mailman3/lists/python-announce-list.python.org/ Member address: arch...@mail-archive.com
Re: C is it always faster than nump?
Dan Stromberg wrote: > On Fri, Feb 25, 2022 at 8:12 AM BELAHCENE Abdelkader < > abdelkader.belahc...@enst.dz> wrote: > >> Hi, >> a lot of people think that C (or C++) is faster than python, yes I agree, >> but I think that's not the case with numpy, I believe numpy is faster than >> C, at least in some cases. >> > > This is all "last time I heard". > > numpy is written, in significant part, in Fortran. > > Fortran, especially for matrix math with variable dimensions, can be faster > than C. > > Fortran, (still last I heard) did not support pointers, which gives Fortran > compilers the chance to exploit a very nice class of optimizations you > can't use nearly as well in languages with pointers. > > I used to code C to be built with the "noalias" optimization, to get much > of the speed of Fortran in C. But it required using an error prone subset > of C without good error detection. Pointers were introduced in Fortran 90. Neil. -- https://mail.python.org/mailman/listinfo/python-list
[issue46757] dataclasses should define an empty __post_init__
Neil Girdhar added the comment: @eric Good thinking. Would it make sense to add to the documentation as well that the __post_init__ methods aren't collected, and you should call super's __post_init__ if there is one using something like if hasattr(super(), "__post_init__"): super().__post_init__() Noting this will make it easier to point to the docs if someone wonders when they see code like this. -- ___ Python tracker <https://bugs.python.org/issue46757> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46828] math.prod can return integers (contradicts doc)
New submission from Neil Webber : The math module documentation says: Except when explicitly noted otherwise, all return values are floats. But this code returns an integer: from math import prod; prod((1,2,3)) Doc should "explicitly note otherwise" here, I imagine. The issue being wanting to know that the result on all-integer input will be an exact (integer) value not a floating value. -- assignee: docs@python components: Documentation messages: 413741 nosy: docs@python, neilwebber priority: normal severity: normal status: open title: math.prod can return integers (contradicts doc) type: behavior ___ Python tracker <https://bugs.python.org/issue46828> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46757] dataclasses should define an empty __post_init__
Neil Girdhar added the comment: @Raymond yeah I've been thinking about this some more, and there's no way to have a "top level" method with the dataclass decorator. I think I will make a case to have a class version of dataclasses that works with inheritance. Class versions of dataclasses are used in some places like here: https://github.com/google/flax/blob/main/flax/struct.py#L184 They just call dataclass on the class in __init_subclass__. If we had something like this in the standard library, then you could put your empty __post_init__ in that class. You could also make __init__ call super so that mixins would be initialized (right now the collider pattern you showed also breaks if B is not a dataclass, and has a non-trivial __init__). -- ___ Python tracker <https://bugs.python.org/issue46757> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46757] dataclasses should define an empty __post_init__
Neil Girdhar added the comment: > How would an arbitrary derived class know how to call this? It can't. There > has to be knowledge of the base class's requirements already. Surely knowing > "__post_init__ must be called with some_arg" isn't too different from "I know > __post_init__ doesn't exist". This is exactly the same problem you have with all other "augmenting methods" that have arbitrary parameters (e.g., __init__). When calling super on a non-final class you could simply forward keyword arguments. @dataclass class X: def __post_init__(self, **kwargs): super().__post_init__(**kwargs) ... @dataclass class Y(X): def __post_init__(self, **kwargs): super().__post_init__(**kwargs) ... > I'm still unconvinced, but I'll hold off on making a decision to see if > there's more support. Maybe taking it to python-ideas would be worthwhile. Sounds good, done: https://groups.google.com/g/python-ideas/c/-gctNaSqgew -- ___ Python tracker <https://bugs.python.org/issue46757> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46757] dataclasses should define an empty __post_init__
Neil Girdhar added the comment: > I'm not crazy about adding a method to every dataclass just for the 0.1% of > the times it's needed. I'm not sure I totally understand the cost here. You can have a single shared global function that you set on each dataclass. So the only cost would be an entry in each dataclass type's dict. Even if a user creates a thousand dataclasses, that should only be tens of killobytes in pointers. > I think using hasattr or catching the exception is a better way to go. If you choose this, then I think this should be documented under the __post_init__ saying that any time you define __post_init__, you should either be a final class, or else call super. If you call super, you musteither use hasattr(super().__post_init__) or catch the exception. I have to admit, I find this quite ugly from a user perspective. -- ___ Python tracker <https://bugs.python.org/issue46757> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46757] dataclasses should define an empty __post_init__
Neil Girdhar added the comment: On Sat, Feb 19, 2022 at 2:51 AM Vedran Čačić wrote: > > Vedran Čačić added the comment: > > That "except AttributeError" approach is a powerful bug magnet, since it > can very easily mask real attribute errors stemming from misspelled > attribute names in the __post_init__ call itself. What you should _really_ > do is > > def __post_init__(self): > with contextlib.suppress(AttributeError): > post_init = super().__post_init__ > post_init() > > But of course, nobody will ever write that. > > Great point! Raymond in his "super considered super" video ( > https://youtu.be/xKgELVmrqfs?t=2068) says the right thing to do is to > make your own root which knows exactly what classes it manages, and drops > the supercalls from them (after possibly verifying that all kwargs have > actually been used and so on). > > But in case of dataclasses, usually any class can serve as such a root, > and the main reason people use dataclasses is to avoid boilerplate code. So > I think it would be a nice compromise. > I'm not sure I understand what you're saying here. Anyone can inherit from a class C with the special root and some other class D, and then your introduced root won't catch D's super calls. > > -- > nosy: +veky > > ___ > Python tracker > <https://bugs.python.org/issue46757> > ___ > -- ___ Python tracker <https://bugs.python.org/issue46757> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46757] dataclasses should define an empty __post_init__
New submission from Neil Girdhar : When defining a dataclass, it's possible to define a post-init (__post_init__) method to, for example, verify contracts. Sometimes, when you inherit from another dataclass, that dataclass has its own post-init method. If you want that method to also do its checks, you need to explicitly call it with super. However, if that method doesn't exist calling it with super will crash. Since you don't know whether your superclasses implement post-init or not, you're forced to check if the superclass has one or not, and call it if it does. Essentially, post-init implements an "augmenting pattern" like __init__, ___enter__, __exit__, __array_finalize__, etc. All such methods define an empty method at the top level so that child classes can safely call super. Please consider adding such an empty method to dataclasses so that children who implement __post_init__ can safely call super. -- components: Library (Lib) messages: 413283 nosy: NeilGirdhar priority: normal severity: normal status: open title: dataclasses should define an empty __post_init__ ___ Python tracker <https://bugs.python.org/issue46757> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46730] Please consider mentioning property without setter when an attribute can't be set
Neil Girdhar added the comment: Thank you, this would have saved me a lot of time! On Sat, Feb 12, 2022 at 8:37 PM Alexander wrote: > > Alexander added the comment: > > Indeed, the error message does not help to identify the problem. Moreover, > it collides with similar errors in namedtuple and DynamicClassAttribute > which may lead to even more confusion. > > I made a draft patch that could help with it ( > https://github.com/Alex-Blade/cpython/commit/06df3a72dfe462c8fe4eac60dce0ef059b1738f8), > but I have a concern related to backwards compatibility (that's why no PR). > I don't really understand if according to PEP 387 a change in an exception > message should be considered compatibility breaking? > > -- > nosy: +Alex-Blade > > ___ > Python tracker > <https://bugs.python.org/issue46730> > ___ > -- ___ Python tracker <https://bugs.python.org/issue46730> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46730] Please consider mentioning property without setter when an attribute can't be set
New submission from Neil Girdhar : class C: @property def f(self) -> int: return 2 class D(C): pass D().f = 2 Gives: Traceback (most recent call last): File "/home/neil/src/cmm/a.py", line 10, in D().f = 2 AttributeError: can't set attribute 'f' This can be a pain to debug when the property is buried in a base class. Would it make sense to mention the reason why the attribute can't be set, namely that it's on a property without a setter? -- components: Interpreter Core messages: 413122 nosy: NeilGirdhar priority: normal severity: normal status: open title: Please consider mentioning property without setter when an attribute can't be set versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46730> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46657] Add mimalloc memory allocator
Neil Schemenauer added the comment: My preference would be for --with-mimalloc=yes in an upcoming release. For platforms without the required stdatomic.h stuff, they can manually specify --with-mimalloc=no. That will make them aware that a future release of Python might no longer build (if mimalloc is no longer optional). A soft-landing for merging nogil is not a good enough reason to merge mimalloc, IMHO. nogil may never be merged. There should be some concrete and immediate advantage to switch to mimalloc. The idea of using the "heap walking" to improve is cyclic GC is not concrete enough. It's just an idea at this point. I think the (small) performance win could be enough of a reason to merge. This seems to be the most recent benchmark: https://gist.github.com/pablogsal/8027937b71cd30f175ef7c885d3e There is also the long-term maintenance issue. So far, mimalloc upstream has been responsive. The mimalloc code is not so huge or complicated that we couldn't maintain it (if for some reason it gets abandoned upstream). However, I think we would prefer to maintain obmalloc rather than mimalloc, all else being equal. Abandonment by the upstream seems fairly unlikely. So, I'm not too concerned about maintenance. -- ___ Python tracker <https://bugs.python.org/issue46657> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46657] Add mimalloc memory allocator
Neil Schemenauer added the comment: Thanks, I'm indeed interested. Most credit goes to Christian for advancing this. For the missing stdatomic.h, would it be appropriate to have an autoconfig check for it? Can just disable mimalloc if it doesn't exist. -- ___ Python tracker <https://bugs.python.org/issue46657> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13475] Add '--mainpath'/'--nomainpath' command line options to override sys.path[0] initialisation
Change by Neil Isaac : -- nosy: +nisaac ___ Python tracker <https://bugs.python.org/issue13475> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45561] Run smelly.py and multissltests.py from $(srcdir)
Change by Neil Schemenauer : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue45561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Getting Directory of Command Line Entry Point For Packages
On 13/11/2021 10.51, Abdur-Rahmaan Janhangeer wrote: > Greetings list, > > Let's say i created a package named miaw > > miaw also has a cli command called miaw > > miaw prints files and folders in the directory it is called in > > except that when miaw is used, it prints the files and folders in > site-packages > > This is an analogy for a package i have. > > Well forgetting about the lines above, how do i get the path from > which miaw the command is called from? try: file_path = __file__ print( file_path ) and process file_path as-required. -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
Neil Schemenauer added the comment: Closing since I believe the bug is fixed and there is an appropriate unit test. -- assignee: -> nascheme resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue38006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: The task is to invent names for things
On 27/10/2021 12.29, Stefan Ram wrote: > dn writes: >> On 27/10/2021 11.16, Stefan Ram wrote: >>> The Mental Game of Python - Raymond Hettinger (PyBay 2019) >>> | "The computer gives us words that do ### things. > ... >> Alternately, if your question was to identify the mumbled word, it is >> (seemed to me to be) "does". > > Thanks! > > Yes, my question was about the word at the place of the > "###". It does not even seem mumbled to me, but pronounced > with certainty and intention. That's why it makes me wonder. > As if there was a term "does things". That is a colloquialism: - my computer does things - my program[me] does stuff The "stuff" is something of a euphemism. In our profession, I would suggest it is used to avoid detail, eg as a 'signal' to a non-IT person that a more detailed answer would likely bore, or 'go over your head'. In PM-circles we identify the beginning and end of a project - the rest of the project plan 'stuff', is known as 'the miracle that happens in the middle'. Want more detail? Do we have more detail? What do I know? If a dog owner said: "my dog does things" it would again be a euphemism, but in this case employed to avoid saying something distasteful, ie that the puppy is not (yet) house-trained. That said, I suspect if you tried to use it in an English (language/literature) essay, the teacher/prof would take exception to such informality, and demand a 'better' noun! Believe it or not, my second trainee-discussion of the day included a question similarly-worded: 'why does the computer/interpreter/run-time do these things?'. Rather than 'literature', I taught this guy one of my favorite excursions into the world of poetry (the specific type of poetic stuff is "doggerel"): I really hate this dumb* machine, I wish that they would sell it. It never does quite what I want, but only what I tell it! * you might regard this word as a euphemism for another Upon which note, and your observation that I am no English-major, it's probably time we went to do things... -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
[issue45561] Run smelly.py and multissltests.py from $(srcdir)
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +27415 pull_request: https://github.com/python/cpython/pull/29138 ___ Python tracker <https://bugs.python.org/issue45561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45561] Run smelly.py and multissltests.py from $(srcdir)
New submission from Neil Schemenauer : Some makefile rules don't work if you build in a separate folder. -- messages: 404671 nosy: nascheme priority: normal severity: normal stage: patch review status: open title: Run smelly.py and multissltests.py from $(srcdir) type: behavior ___ Python tracker <https://bugs.python.org/issue45561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
Change by Neil Schemenauer : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
Neil Schemenauer added the comment: New changeset 1cdac61065e72db60d26e03ef9286d2743d7000e by Miss Islington (bot) in branch '3.10': bpo-45521: Fix a bug in the obmalloc radix tree code. (GH-29051) (GH-29122) https://github.com/python/cpython/commit/1cdac61065e72db60d26e03ef9286d2743d7000e -- ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
Neil Schemenauer added the comment: New changeset 311910b31a4bd94dc79298388b7cb65ca5546438 by Neil Schemenauer in branch 'main': bpo-45521: Fix a bug in the obmalloc radix tree code. (GH-29051) https://github.com/python/cpython/commit/311910b31a4bd94dc79298388b7cb65ca5546438 -- ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
Neil Schemenauer added the comment: I have not yet been able to reproduce methane's crash. My guess it it's not related. An explanation of what I think the impact of this bug is: The radix tree is used to determine if memory is from obmalloc or from the system malloc (i.e return value from address_in_range()). WIth ADDRESS_BITS set to 48, we ignore the top 16 bits of addresses. The next 10 bits are supposed to be the index into the top level node array for the radix tree. Due to the bug, we mask those and only use the bottom 8 of those 10. So, if you have virtual addresses that span more than that 8 bit range, we will index into the wrong node. That means address_in_range() could give the wrong answer. Which means you might try to free memory with the wrong malloc. I think this is likely to be triggered only if you allocate a massive amount of memory, like 70 TB. However, triggering it would depend on how the kernel maps virtual memory to the Python process. I.e. there might be a wierd OS that gives pages at 0x7f00 and then right after pages at 0x3f00. -- ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45526] Set ADDRESS_BITS to 64 for obmalloc radix tree
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +27329 pull_request: https://github.com/python/cpython/pull/29062 ___ Python tracker <https://bugs.python.org/issue45526> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45526] Set ADDRESS_BITS to 64 for obmalloc radix tree
New submission from Neil Schemenauer : Given this feedback: https://github.com/python/cpython/pull/14474/files#r725488766 it is perhaps not so safe to assume that only the lower 48 bits of virtual addresses are significant. I had the idea that Go made similar assumptions but now I'm not sure it does. There is also a comment in this article about 5-level page tables in Linux: https://lwn.net/Articles/717293/ https://lwn.net/Articles/717300/ I.e. that Linux does not allocate virtual address space above 47-bit by default. Setting ADDRESS_BITS to 64 is safer and the performance impact seems small. The virtual memory size of a small Python process goes up a little. Resident set size doesn't significantly change. I think the pyperformance changes are just noise. The pyperformance attached file shows the 3.10 branch with ADDRESS_BITS set to 48 and to 64. -- assignee: methane files: pyperf-compare-addr-64.txt messages: 404317 nosy: methane, nascheme priority: normal severity: normal stage: patch review status: open title: Set ADDRESS_BITS to 64 for obmalloc radix tree Added file: https://bugs.python.org/file50370/pyperf-compare-addr-64.txt ___ Python tracker <https://bugs.python.org/issue45526> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +27322 pull_request: https://github.com/python/cpython/pull/29051 ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
New submission from Neil Schemenauer : There is a typo in the radix tree obmalloc code, spotted by Inada Naoki. -#define MAP_TOP_MASK (MAP_BOT_LENGTH - 1) +#define MAP_TOP_MASK (MAP_TOP_LENGTH - 1) This should be fixed both in the main branch and in 3.10.x. -- assignee: methane components: Interpreter Core messages: 404268 nosy: methane, nascheme priority: high severity: normal stage: patch review status: open title: obmalloc radix tree typo in code versions: Python 3.10, Python 3.11 ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44036] asyncio SSL server can be DOSed, event loop gets blocked: busy loops and uses 100% CPU
Change by Neil Booth : -- nosy: +kyuupichan ___ Python tracker <https://bugs.python.org/issue44036> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: XML Considered Harmful
On 25/09/2021 11.00, Chris Angelico wrote: > Invented because there weren't enough markup languages, so we needed another? Anything You Can Do I Can Do Better https://www.youtube.com/watch?v=_UB1YAsPD6U -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Help me split a string into elements
DFS wrote:
> Typical cases:
> lines = [('one\ntwo\nthree\n')]
> print(str(lines[0]).splitlines())
> ['one', 'two', 'three']
>
> lines = [('one two three\n')]
> print(str(lines[0]).split())
> ['one', 'two', 'three']
>
>
> That's the result I'm wanting, but I get data in a slightly different
> format:
>
> lines = [('one\ntwo\nthree\n',)]
>
> Note the comma after the string data, but inside the paren.
> splitlines() doesn't work on it:
>
> print(str(lines[0]).splitlines())
> ["('one\\ntwo\\nthree\\n',)"]
>
>
> I've banged my head enough - can someone spot an easy fix?
>
> Thanks
lines[0][0].splitlines()
(You have a list containing a tuple containing the string you want to
split up.)
--
https://mail.python.org/mailman/listinfo/python-list
[issue45020] Freeze all modules imported during startup.
Neil Schemenauer added the comment: [Larry] > The one thing I guess I never mentioned is that building and working with the > prototype was frightful; it had both Python code and C code, and it was > fragile and hard to get working. I took Larry's PR and did a fair amount of cleanup on it to make the build less painful and fragile. My branch is fairly easy to re-build. The major downsides remaining are that you couldn't update .py files and have them used (static ones take priority) and the generated C code is quite large. I didn't make any attempt to work on the serializer, other than to make it work with an alpha version of Python 3.10. https://github.com/nascheme/cpython/tree/static_frozen It was good enough to pass nearly(?) all tests and I did some profiling. It helped reduce startup time quite a bit. -- ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of PyObject_GC_UnTrack() with the trashcan C API
Neil Schemenauer added the comment: Another small correction, _PyType_IS_GC() checks only the type flag (Py_TPFLAGS_HAVE_GC). _PyObject_IS_GC() checks both the type flag and calls tp_is_gc(), if it exists. tp_is_gc() is wart, IMHO, and it would be nice if we can kill it off so only the type flag is needed. That's a topic for a different issue though. -- ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
Change by Neil Schemenauer : -- nosy: +pablogsal, tim.peters, vstinner ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
Neil Schemenauer added the comment: Based on some testing, I think an extra type slot is not worth the extra complication. I made some small improvements to _Py_Dealloc and now the performance seems a bit better. Basically, I expanded _PyObject_IS_GC() to put the most common branches first. See pypref-trashcan2.txt for benchmark. Overall I think might be a bit slower (less than 1%?) but we have gained trashcan protection for the dealloc of all GC objects. If we are okay with the performance, here are other questions to answer: A) is it safe to untrack GC objects before calling tp_dealloc? B) is it safe to have PyObject_CallFinalizerFromDealloc() track objects that are resurrected? We might have to look at 3rd party extensions to decide on those. I.e. maybe in theory it is not safe but in practice there is no extension code that would actually break. -- Added file: https://bugs.python.org/file50223/pyperf-trashcan2.txt ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
Change by Neil Schemenauer : Added file: https://bugs.python.org/file50220/perf-annotate-trash.txt ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
Neil Schemenauer added the comment: As I suspected, the performance impact is significant (although pretty small). Based on Linux perf, it looks like the extra test+branch in _Py_Dealloc adds about 1% overhead. pyperformance shows something similar, see attached reports (pypref-trashcan.txt and perf-annotate-trash.txt). An idea I've been trying is to add another type slot, e.g. tp_call_dealloc. It could be set by PyType_Ready(). It would be called by the Py_DECREF macro on refcnt going to zero. If the object is non-GC and Py_TRACE_REFS is off, can make tp_call_dealloc actually be the tp_dealloc pointer. If the type has the GC flag, point tp_call_dealloc to a _Py_Dealloc version that checks tp_is_gc and does the trashcan stuff. Unfortunately, all types don't have PyType_Ready() called on them. So, we cannot rely on tp_call_dealloc being set correctly. -- Added file: https://bugs.python.org/file50219/pyperf-trashcan.txt ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of PyObject_GC_UnTrack() with the trashcan C API
Neil Schemenauer added the comment: > I think in any case we should benchmark this because this will affect *all* > GC types if I understand correctly and the TS mechanism had shown slowdowns > before We definitely need to benchmark. I would guess that adding trashcan protection to all GC types would not be a performance issue. We already had the macros for some performance critical ones (frame, tuple, list, dict). The performance hit will likely come as a result of adding a branch that uses _PyType_IS_GC() to the DECREF code path. It means any time an object hits zero refcount, we call _PyType_IS_GC() on it. Previously, we would just call tp_dealloc. Because of PEP 442, _PyType_IS_GC() checks not only a type flag but might also call tp_is_gc. So, benchmarks will need to be done. We might get a small win because the trashcan logic can be better optimized now that it's in a single complied unit. Small correction for my explaination above: if tp_dealloc gets called mutiple times because of the trashcan, it is the code before the BEGIN macro that gets called mutiple times. -- ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of PyObject_GC_UnTrack() with the trashcan C API
Neil Schemenauer added the comment:
> The problem of PyObject_GC_UnTrack() is just the most visible effect of the
> trashcan mecanism: tp_dealloc can be called twice, and this is *not* expected
> by the tp_dealloc API.
The fact that Py_TRASHCAN_BEGIN and Py_TRASHCAN_END can cause tp_dealloc to be
called more than once is not a huge issue, IMHO. The statements executed more
than once are the ones before or after the trashcan macros. If you use them
properly, the multiple calls doesn't cause issues. The big trap is
to use _PyObject_GC_UNTRACK before Py_TRASHCAN_BEGIN. It is not safe to call
that macro multiple times. Anywhere you see Py_TRASHCAN_BEGIN and
_PyObject_GC_UNTRACK before it, that's a bug.
The point of this PR is to make the macros harder to use incorrectly. If there
is no need to call untrack, Py_TRASHCAN_BEGIN and Py_TRASHCAN_END can normally
be the first and last lines of the tp_dealloc function (declarations can be
inside). Then, the fact that tp_dealloc is actually called more than once by
the
trashcan doesn't cause an issue.
tp_dealloc can get called more than once for another reason. If the object is
resurrected by tp_finalize or tp_del, then tp_dealloc can be called again.
It's pretty mind bending to try to understand all of implications of that. Most
of the ugliness is inside PyObject_CallFinalizerFromDealloc(). The type with
the finalizer has to be careful not to put the object in an invalid state
before it gets resurrected.
> Putting trashcan mecanism outside tp_dealloc can allow to make sure that
> tp_dealloc is called exactly once. _Py_Dealloc() sounds like a good
> candidate, but I didn't check if it's the only way to call tp_dealloc. Are
> there other places where tp_dealloc is called *directly*?
tp_dealloc is called from a few more places, as you found. As for providing
C-stack exhaustion protection via the trashcan, I think we are only interested
in places were tp_dealloc is called as a result of a DECREF. And I think that
only goes through _Py_Dealloc().
I suppose it would be possible to blow up the C stack via a call chain like:
subtype_dealloc -> basedealloc -> subtype_dealloc -> basedealloc -> ...
I think you would have to make a long chain of subclasses. Seems unlikely in
real code so I'm not worried about trying to fix that.
> Using military grade regex, I found the following functions calling
> tp_dealloc:
>
> grep 'dealloc[) ]*([a-zA-Z]\+)' */*.c
>
> * _Py_Dealloc() obviously
> * _PyTrash_thread_destroy_chain()
> * subtype_dealloc()
> * Modules/_testcapimodule.c: check_pyobject_freed_is_freed() <= unit test, it
> can be ignored
>
> So if we move the trashcan usage inside functions, 3 functions must be
> modified:
>
> * _Py_Dealloc()
> * _PyTrash_thread_destroy_chain()
> * subtype_dealloc()
Take a look at bpo-44897. It implements the _Py_Dealloc approach. You can see
what needs to be modified. I removed the Py_TRASHCAN_BEGIN/Py_TRASHCAN_END
macros as well, just because they are not needed anymore. _PyObject_GC_UNTRACK
calls inside tp_dealloc methods need to be removed because _Py_Dealloc already
does the untrack. For external extensions, they must be using
PyObject_GC_UnTrack() and so that's safe (object is already untracked and so it
does nothing).
I think the big change is calling tp_dealloc methods with the object already
untracked. If an extension did something like:
mytype_dealloc(PyObject *self)
{
...
assert(PyObject_GC_IsTracked(self));
...
}
That would break after PR 27738, at least as it's currently written.
The other issue I'm not quite sure about is that
PyObject_CallFinalizerFromDealloc insists that GC objects are tracked when that
function is called. I think it is okay to just call _PyObject_GC_TRACK on the
ones that are not tracked, to ensure that. Maybe Tim Peters will jump in and
correct the errors of my thinking.
--
nosy: +tim.peters
___
Python tracker
<https://bugs.python.org/issue44881>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of PyObject_GC_UnTrack() with the trashcan C API
Neil Schemenauer added the comment: I wrote a proof-of-concept as bpo-44897. PR 27718 (this issue) might a slightly better performance but I like the other one better because it doesn't expose so much implementation detail to extension types. Either of them require careful review before merging. -- ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +26218 pull_request: https://github.com/python/cpython/pull/27738 ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
New submission from Neil Schemenauer : This is a WIP/proof-of-concept of doing away with Py_TRASHCAN_BEGIN and Py_TRASHCAN_END and instead integrating the functionality into _Py_Dealloc. There are a few advantages: - all container objects have the risk of overflowing the C stack if a long reference chain of them is created and then deallocated. So, to be safe, the tp_dealloc methods for those objects should be protected from overflowing the stack. - the Py_TRASHCAN_BEGIN and Py_TRASHCAN_END macros are hard to understand and a bit hard to use correctly. Making the mechanism internal avoids confusion. The code can be slightly simplified as well. This proof-of-concept seems to pass tests but it will need some careful review. The exact rules related to calling GC Track/Untrack are subtle and this changes things a bit. I.e. tp_dealloc is called with GC objects already untracked. For 3rd party extensions, they are calling PyObject_GC_UnTrack() and so I believe they should still work. The fact that PyObject_CallFinalizerFromDealloc() wants GC objects to definitely be tracked is a bit of a mystery to me (there is an assert to check that). I changed the code to track objects if they are not already tracked but I'm not sure that's correct. There could be a performance hit, due to the _PyType_IS_GC() test that was added to the _Py_Dealloc() function. For non-GC objects, that's going to be a new branch and I'm worried it might hurt a bit. OTOH, maybe it's just in the noise. Profiling will need to be done. -- components: Interpreter Core messages: 399433 nosy: nascheme priority: normal severity: normal stage: patch review status: open title: Integrate trashcan mechanism into _Py_Dealloc type: enhancement ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of PyObject_GC_UnTrack() with the trashcan C API
Neil Schemenauer added the comment: I was thinking about this more today and I think the better fix is to actually build the trashcan mechanism into _Py_Dealloc(). Requiring that types opt-in to the trashcan mechanism by using the trashcan macros is not ideal. First, the trashcan macros are a bit tricky to use correctly. Second, every "container" type is potentially a part of a long ref chain and could blow up the stack on deallocation (i.e. triggered from DECREF). So, for correctness/robustness, every type that supports cyclic GC should get trashcan-style deallocation. We would have to find a way to do this without incurring a (significant) performance hit. Also, it would have to be done without breaking C extensions. Ideally they would get trashcan-style deallocation without any source code changes. I'm not sure if that can be done but I'm hopeful it's possible. -- ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of GC_UNTRACK with TRASHCAN
Neil Schemenauer added the comment:
Since C99 is now allowed, perhaps we should suggest that declarations go after
Py_TRASHCAN_BEGIN, e.g.
mytype_dealloc(mytype *p)
{
Py_TRASHCAN_BEGIN(p, mytype_dealloc)
... declarations go here ...
... The body of the deallocator goes here, including all calls ...
... to Py_DECREF on contained objects. ...
Py_TRASHCAN_END// there should be no code after this
}
The only dealloc method that doesn't fit this pattern is subtype_dealloc() and
that's because the object might not be a GC object.
Given the pattern, it occurs to me that that _Py_Dealloc() could do the
trashcan begin/end work. However, the authors of the dealloc methods would
have to realize the special rules of the trashcan (e.g. no returns from the
dealloc method body). Also, there would be some overhead added to
_Py_Dealloc() to check if the trashcan is supported. E.g. checking a type flag.
Another idea would be to add a new slot for the method, e.g. tp_dealloc_trash.
Then, _Py_Dealloc() could check if it exists and if so do the trashcan
begin/end dance around it. That would still add some overhead to _Py_Dealloc()
so I think the current macros are the best approach.
--
___
Python tracker
<https://bugs.python.org/issue44881>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of GC_UNTRACK with TRASHCAN
Neil Schemenauer added the comment: For examples of bugs caused by forgetting the untrack calls, see bpo-31095. -- ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of GC_UNTRACK with TRASHCAN
Neil Schemenauer added the comment: Extensions that call PyObject_GC_UnTrack before calling Py_TRASHCAN_BEGIN will still work, they will just take a very minor performance hit. I don't think it is worth the trouble to introduce new macros for that reason. Extensions that really care about performance can wrap the call in a Python version ifdef. There is an issue if someone writes and tests their extension with the new API, i.e. without having the explicit PyObject_GC_UnTrack() call in their dealloc method. If they compile with an older Python, they likely get a crash. If they compile with asserts enable, they would get an assert fail in _PyTrash_thread_deposit_object, i.e.: _PyObject_ASSERT(op, !_PyObject_GC_IS_TRACKED(op)); I guess that's an argument for new macros. -- stage: patch review -> needs patch ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of GC_UNTRACK with TRASHCAN
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +26201 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/27718 ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33930] Segfault with deep recursion into object().__dir__
Neil Schemenauer added the comment: I'm thinking that the explicit call to PyObject_GC_UnTrack should be made unnecessary by integrating it into the trashcan code. That way, we avoid someone else running into this kind of bug in the future. See bpo-44881. -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue33930> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of GC_UNTRACK with TRASHCAN
New submission from Neil Schemenauer : The fix for bpo-33930 includes a somewhat mysterious comment: // The Py_TRASHCAN mechanism requires that we be able to // call PyObject_GC_UnTrack twice on an object. I wonder if we can just integrate the untrack into the body of the trashcan code. Then, the explicit call to untrack in the dealloc function body can be removed. That removes the risk of incorrectly using the macro version. I suspect the reason this was not done originally is because the original trashcan mechanism did not use the GC header pointers to store objects. Now, any object that uses the trashcan *must* be a GC object. -- messages: 399356 nosy: nascheme priority: normal severity: normal stage: needs patch status: open title: Consider integration of GC_UNTRACK with TRASHCAN type: enhancement ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Ask for help on using re
Jach Feng wrote:
> I want to distinguish between numbers with/without a dot attached:
>
text = 'ch 1. is\nch 23. is\nch 4 is\nch 56 is\n'
re.compile(r'ch \d{1,}[.]').findall(text)
> ['ch 1.', 'ch 23.']
re.compile(r'ch \d{1,}[^.]').findall(text)
> ['ch 23', 'ch 4 ', 'ch 56 ']
>
> I can guess why the 'ch 23' appears in the second list. But how to get rid of
> it?
>
> --Jach
Does
>>> re.findall(r'ch\s+\d+(?![.\d])',text)
do what you want? This matches "ch", then any nonzero number of
whitespaces, then any nonzero number of digits, provided this is not
followed by a dot or another digit.
--
https://mail.python.org/mailman/listinfo/python-list
PyConZA 2021 - Second call for submissions
This is a second call for talk proposals for PyConZA 2021. While we've already received several great talk proposals, we still have a number of open talk slots and are thus keen to see further talk submissions. PyCon ZA 2021 will take place on the 7th & 8th of October, 2021. This year, due to the ongoing pandemic, PyCon ZA will again be an online event. We will be streaming the talks and using discord for discussion around the talks and other conference activities. We are looking for the following presentations: - Keynotes (45 minute long talks on a subject of general interest) - Talks (30 minute long talks on more specific topics) If you would like to give a presentation, please register at https://za.pycon.org/ and submit your proposal, following the instructions at https://za.pycon.org/talks/submit-talk/ . We have a number of tracks available, including: Data Science, Teaching and Learning with Python, Web, Scientific Computing, Testing and Other (which includes all talks that don't fall under the mentioned tracks). We hope to notify accepted presenters by no later than the 31st of August 2021. Speakers will be expected to be available after the presentation for a short Q session. Shared sessions are also possible. The presentations will be in English. PyCon ZA offers a mentorship program for inexperienced speakers. If you would like assistance preparing your submission, email t...@za.pycon.org with a rough description of what you would like to talk about and we'll find a suitable experienced speaker to act as a mentor. If you want to present something that doesn't fit into the two talk categories at PyCon ZA, please contact the organising committee at t...@za.pycon.org so we can discuss whether that will be feasible. -- Neil Muller On behalf of the PyConZA organising committee ___ Python-announce-list mailing list -- python-announce-list@python.org To unsubscribe send an email to python-announce-list-le...@python.org https://mail.python.org/mailman3/lists/python-announce-list.python.org/ Member address: arch...@mail-archive.com
[issue43384] Include regen-stdlib-module-names in regen-all
Change by Neil Schemenauer : -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue43384> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44523] weakref.proxy documentation might be outdated
Neil Schemenauer added the comment: It seems to me the old behaviour (don't forward hash) was done for good reasons. If the referent might go away, it is not valid to use it as a dict key since the hash and eq result changes. If it can't go away, what reason is there to use a weakref and not a direct reference? -- ___ Python tracker <https://bugs.python.org/issue44523> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23316] Incorrect evaluation order of function arguments with *args
Neil Girdhar added the comment: FYI: https://github.com/PyCQA/pylint/issues/4586 -- ___ Python tracker <https://bugs.python.org/issue23316> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44331] Generate static PyCodeObjects for faster startup
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +25161 pull_request: https://github.com/python/cpython/pull/26571 ___ Python tracker <https://bugs.python.org/issue44331> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44331] Generate static PyCodeObjects for faster startup
New submission from Neil Schemenauer : Note: This is a proof of concept and not ready for merging as is. This is based on 'frozen_modules' from Jeethu Rao , via Larry Hastings. Larry's git branch was: g...@github.com:larryhastings/cpython.git not_another_use_of_the_word_frozen Usage: - Compile Python as normal - Run "make regen-freeze-startup" to re-generate Python/frozenmodules_code.c - Compile Python a second time Changes from Larry's branch: - Move static C code generation tool to Tools/freeze2 - Move _serializer to Modules - Rebase on Python 3.10.0b1 - determine startup modules by running sys.executable - use importlib.util.find_spec() to get code objects - fix ref-counting when setting __path__ - put static frozen modules in frozen_code_objects dict - reduce set of "bad" modules as it seems only _collections_abc needs exclusion - fix the is_frozen_package() and is_frozen() functions to find static frozen code It's not passing all unit tests yet but I'm somewhat hopeful there are no deep problems. Porting the changes from 3.6 to 3.8 and then to 3.10 was not too horrible. There was a few changes to PyGC_Head, to the PyCodeObject structure and to the runtime initialization process. That gives me some hope that it wouldn't be too burdensome to maintain this in the long-term. Mostly it would be updating _serialize.c to keep up with PyCodeObject changes. Based on benchmarking with 3.8, this gives a decent speedup for startup of a trival program, e.g. python -c "True". I measure it as taking 76% of the time. The savings are mostly in marshal.c but there is also some importlib/filesystem overhead that's removed. -- messages: 395243 nosy: nascheme priority: low severity: normal stage: patch review status: open title: Generate static PyCodeObjects for faster startup type: performance ___ Python tracker <https://bugs.python.org/issue44331> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42972] [C API] Heap types (PyType_FromSpec) must fully implement the GC protocol
Change by Neil Schemenauer : -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue42972> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
PyCon ZA 2021 - Call for Submissions
PyCon ZA 2021 will take place on the 7th & 8th of October, 2021. This year, due to the ongoing pandemic, PyCon ZA will again be an online event. We are looking for the following presentations: - Keynotes (45 minute long talks on a subject of general interest) - Talks (30 minute long talks on more specific topics) If you would like to give a presentation, please register at https://za.pycon.org/ and submit your proposal, following the instructions at https://za.pycon.org/talks/submit-talk/ . We have a number of tracks available, including: Data Science, Teaching and Learning with Python, Web, Scientific Computing, Testing and Other (which includes all talks that don't fall under the mentioned tracks). We hope to notify accepted presenters by no later than the 31st of August 2021. Speakers will be expected to be available after the presentation for a short Q session. Shared sessions are also possible. The presentations will be in English. PyCon ZA offers a mentorship program for inexperienced speakers. If you would like assistance preparing your submission, email t...@za.pycon.org with a rough draft of your talk proposal and we'll find a suitable experienced speaker to act as a mentor. If you want to present something that doesn't fit into the two talk categories at PyCon ZA, please contact the organising committee at t...@za.pycon.org so we can discuss whether that will be feasible. -- Neil Muller On behalf of the PyCon ZA organising committee ___ Python-announce-list mailing list -- python-announce-list@python.org To unsubscribe send an email to python-announce-list-le...@python.org https://mail.python.org/mailman3/lists/python-announce-list.python.org/ Member address: arch...@mail-archive.com
Re: Horrible abuse of __init_subclass__, or elegant hack?
Officially April-Fools Day is over (here), but... On 01/04/2021 19.25, Chris Angelico wrote: > On Thu, Apr 1, 2021 at 3:36 PM dn via Python-list > wrote: >> >> On 01/04/2021 13.54, Chris Angelico wrote: >>> Real and imaginary are the same thing, just rotated a quarter turn >> >> In which dimension(s)? > > Cartesian. Isn't this water you get out of the ground and then deliver using an horse-pulled vehicle? Without looking into the details/context: surely there's a more straightforward approach? >>> >>> Perhaps, but there are potentially a LOT of recipes, and I needed to >>> be able to cleanly edit those, even if the code at the top was a mess. >>> (The goal here is to map out production patterns in the game >>> "Satisfactory", for the curious. It's easy to add other things, like >>> computer manufacturing or bauxite processing, simply by adding more >>> recipes.) >>> >>> My original plan was basically pairwise tuple summing (deriving a set >>> of "oil in, water in, rubber out, fuel out" for each set of recipes, >>> where some might be zero), but it turned out that that wasn't flexible >>> enough, and it really needed more options than that. >> >> Which was where my mind was going*. Why not a dict of inputs, processes, >> and outputs**? Each dict having variable length, from None, and >> key:values assigned at declaration/init. In the case of process, the >> contained objects could be Python-functions. With "compact >> representation" (3.6+) the functions could also be relied upon to >> represent a 'production line' or pipeline of functions. > > Perhaps, but the key here is the input method. It wouldn't look nearly as > clean. Not sure about that. For example, is it immediately-obvious which of "Crude", "Residue", and "Plastic" are inputs; and which outputs? >>> I already have certificates from Rutledge's Asylum and MaayaInsane's >>> (unnamed) asylum, so that seems pretty likely. >> >> Noted you on the list of lauded alumni at the latter. > > Hmm, where do you see that list? I'm curious. Appeared to be some sort of 'leader board' for the game. Recognised your handle there. Think it was https://www.twitch.tv/maayainsane. Perhaps I was looking at the same time as you logged-in? Daren't disappear down that rabbit-hole again... >> When you left the former, did they allow you to keep the t-shirt, or did >> you have to buy your own memorabilia? >> (https://mysterious.americanmcgee.com/products/rutledge-asylum-mug) > > I beg your pardon? What is this "left"? I'm still there! Actually I > pay my membership on a monthly basis, and in return, my walls are > lavishly decorated in Alice art. Whereas us lateral-thinkers don't like the feeling of being fenced-in. >> The latter's treatment list sounds remarkably like .mil training. I know >> of plenty with that t-shirt - but can't think of a one sporting a mug... >> Should you have one, kindly bring it (with appropriate contents) come >> ANZAC Day at the end of this month... > > https://streamlabs.com/maayainsane/merch/1053635 > This is what I'd bring. They're the standard mugs that I offer to > guests. Well, I would if ever I had guests, but hermitism is a > thing... Is that the same as (anti-) club-ism (cue Groucho: https://www.goodreads.com/quotes/6517787-i-wouldn-t-want-to-belong-to-a-club-that-would) After suffering the RSA's ?quaint tradition of rum-in-milk (that BEFORE Dawn Parade, and thereafter marching on a less-than flat surface) I have learned to guard against the pre-dawn cold by heading for the Hot Chocolate counter - can't say the mugs look like that though. >> Magic, you ask? Well, maybe more "sinister". We did manage to find a >> loose floor-board, but a sad life-lesson was learned, when certain ones >> (un-named*) took it upon themselves to eat all of the contraband >> secreted there. > > Uh oh. How old was the contraband? Was between 'tuck days' (when we were allowed to acquire such goodies - once?twice per week). The theory being that we would all contribute, in anticipation of some later "midnight feast") - such gyre and gimble-ing being more Jabberwock than Alice! -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
[issue43593] pymalloc is not aware of Memory Tagging Extension (MTE) and crashes
Neil Schemenauer added the comment: > If MTE is actually being used, system software assigns "random" values to 4 > of the higher-order bits. Oh, interesting. Two ideas about handling that: we could change our assertion check to be different on ARM platforms that we know have a certain size physical address space. Probably just turn off that high-bits check. Second idea, we could change the radix tree to not assume high address bits are unused. That's trickier to do without performance or memory usage degradations. I have a work-in-progress patch that adds a cache on top of the radix tree lookup. It looks like that cache can be made to have a pretty high hit rate. Based on a small amount of testing, the radix tree lookup for address_in_range() only happens about 1% of the time. If that approach works, we could add another layer to the tree and handle the full 64-bit address space. Based on my wip testing, my benchmark was showing about equal performance with the cache to without. So, no benefit to offset the increase in code complexity. Handling the MTE high bits tricks might enough to justify the cache addition. -- versions: +Python 3.9 -Python 3.8 ___ Python tracker <https://bugs.python.org/issue43593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37368] test_asyncio: test_create_server_ssl_match_failed() failed on s390x SLES 3.x and logged an unraisable exception
Neil Schemenauer added the comment:
It seems to not be specific to S390, same kind of failure on the PPC64LE RHEL8
LTO + PGE 3.x tester:
Exception ignored in:
Traceback (most recent call last):
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/sslproto.py",
line 321, in __del__
self.close()
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/sslproto.py",
line 316, in close
self._ssl_protocol._start_shutdown()
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/sslproto.py",
line 590, in _start_shutdown
self._abort()
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/sslproto.py",
line 731, in _abort
self._transport.abort()
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/selector_events.py",
line 680, in abort
self._force_close(None)
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/selector_events.py",
line 731, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/base_events.py",
line 745, in call_soon
self._check_closed()
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/base_events.py",
line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
--
___
Python tracker
<https://bugs.python.org/issue37368>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37368] test_asyncio: test_create_server_ssl_match_failed() failed on s390x SLES 3.x and logged an unraisable exception
Neil Schemenauer added the comment:
Seems this failure might be back. At least, the traceback looks quite similar
to me.
The buildbot failed with this:
heads/master:85b6b70589, Mar 29 2021, 22:53:15
0:05:26 load avg: 3.95 [426/427] test_tokenize passed (56.0 sec) -- running:
test_asyncio (44.8 sec)
0:05:35 load avg: 3.34 [427/427/1] test_asyncio failed (env changed) (54.1 sec)
Warning -- Unraisable exception
Exception ignored in:
Traceback (most recent call last):
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py",
line 321, in __del__
self.close()
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py",
line 316, in close
self._ssl_protocol._start_shutdown()
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py",
line 590, in _start_shutdown
self._abort()
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py",
line 731, in _abort
self._transport.abort()
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/selector_events.py",
line 680, in abort
self._force_close(None)
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/selector_events.py",
line 731, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/base_events.py",
line 745, in call_soon
self._check_closed()
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/base_events.py",
line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
--
nosy: +nascheme
resolution: out of date ->
stage: resolved ->
status: closed -> open
versions: +Python 3.10 -Python 3.9
___
Python tracker
<https://bugs.python.org/issue37368>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43593] pymalloc is not aware of Memory Tagging Extension (MTE) and crashes
Neil Schemenauer added the comment: I've merged PR 14474 so you can just test with an up-to-date "main" branch and see if that fixes the problem. I would expect it should fix the problem since with the radix tree arena tracking, no memory unsanitary behaviour remains. -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue43593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37448] obmalloc: radix tree for tracking arena address ranges
Change by Neil Schemenauer : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue37448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37448] obmalloc: radix tree for tracking arena address ranges
Neil Schemenauer added the comment: New changeset 85b6b70589c187639aeebc560d67e9cc04abb4d8 by Neil Schemenauer in branch 'master': bpo-37448: Use radix tree for pymalloc address_in_range(). (GH-14474) https://github.com/python/cpython/commit/85b6b70589c187639aeebc560d67e9cc04abb4d8 -- ___ Python tracker <https://bugs.python.org/issue37448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43445] Add frozen modules to sys.stdlib_module_names
Neil Schemenauer added the comment: Not sure the proper place to discuss this but I wonder if putting this stdlib module names list in the executable is the best idea. The list of available stdlib modules could change after compiling Python. I understand you don't want to dynamically crawl the library path the build the list. That's too slow. However, is there a really strong reason to embed it in the Python executable? Did you consider generating a .py module, containing the list. E.g. "_stdlib_modules.py" inside the lib folder. Then, you can have site.py or some similar startup logic import that module and assign it to sys.stdlib_module_names. -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue43445> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42246] Implement PEP 626 -- Precise line numbers for debugging
Neil Schemenauer added the comment: New changeset 87ec26b812e9c4095c017dc60f246eda37b83ab2 by Neil Schemenauer in branch 'master': bpo-43372: Use _freeze_importlib for regen-frozen. (GH-24759) https://github.com/python/cpython/commit/87ec26b812e9c4095c017dc60f246eda37b83ab2 -- ___ Python tracker <https://bugs.python.org/issue42246> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Change by Neil Schemenauer : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Neil Schemenauer added the comment: New changeset 87ec26b812e9c4095c017dc60f246eda37b83ab2 by Neil Schemenauer in branch 'master': bpo-43372: Use _freeze_importlib for regen-frozen. (GH-24759) https://github.com/python/cpython/commit/87ec26b812e9c4095c017dc60f246eda37b83ab2 -- ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Change by Neil Schemenauer : -- assignee: -> nascheme ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42246] Implement PEP 626 -- Precise line numbers for debugging
Change by Neil Schemenauer : -- pull_requests: +23532 pull_request: https://github.com/python/cpython/pull/24759 ___ Python tracker <https://bugs.python.org/issue42246> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Change by Neil Schemenauer : -- pull_requests: +23531 pull_request: https://github.com/python/cpython/pull/24759 ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43382] github CI blocked by the Ubuntu CI with an SSL error
Neil Schemenauer added the comment: It seems it is enough to make a new commit. The CI seems to re-base and re-run the PR. At least, it worked on two of my PRs. -- ___ Python tracker <https://bugs.python.org/issue43382> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43384] Include regen-stdlib-module-names in regen-all
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +23494 pull_request: https://github.com/python/cpython/pull/24713 ___ Python tracker <https://bugs.python.org/issue43384> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43384] Include regen-stdlib-module-names in regen-all
New submission from Neil Schemenauer : While I was fixing the regen-frozen issue, I noticed it seems unnecessary to have regen-stdlib-module-names separate from regen-all. Maybe Victor knows why it needs to be separate. If it doesn't need to be separate, the CI scripts can be slightly simplified. -- components: Build messages: 388003 nosy: nascheme priority: normal severity: normal stage: patch review status: open title: Include regen-stdlib-module-names in regen-all type: enhancement ___ Python tracker <https://bugs.python.org/issue43384> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43382] github CI blocked by the Ubuntu CI with an SSL error
Neil Schemenauer added the comment: I think it may be related to bpo-41561. There is a bug in the Ubuntu tracker as well: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1899878 I agree with the temporary fix to use "ubuntu-18.04" for CI testing. -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue43382> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42246] Implement PEP 626 -- Precise line numbers for debugging
Change by Neil Schemenauer : -- pull_requests: +23492 pull_request: https://github.com/python/cpython/pull/24714 ___ Python tracker <https://bugs.python.org/issue42246> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Change by Neil Schemenauer : -- pull_requests: +23491 pull_request: https://github.com/python/cpython/pull/24714 ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43381] add small test for frozen module line number table
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +23490 pull_request: https://github.com/python/cpython/pull/24712 ___ Python tracker <https://bugs.python.org/issue43381> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43381] add small test for frozen module line number table
New submission from Neil Schemenauer : In bug #43372, we didn't notice that the code for the __hello__ module was not re-generated. Things seems to be okay but the line number table was corrupted. It seems a good idea to add a small test to ensure that doesn't happen again. I marked the test as CPython implementation specific. -- components: Tests messages: 387989 nosy: nascheme priority: low severity: normal stage: patch review status: open title: add small test for frozen module line number table type: enhancement ___ Python tracker <https://bugs.python.org/issue43381> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +23484 stage: -> patch review pull_request: https://github.com/python/cpython/pull/24708 ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42246] Implement PEP 626 -- Precise line numbers for debugging
Change by Neil Schemenauer : -- nosy: +nascheme nosy_count: 7.0 -> 8.0 pull_requests: +23485 pull_request: https://github.com/python/cpython/pull/24708 ___ Python tracker <https://bugs.python.org/issue42246> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Neil Schemenauer added the comment:
I believe the line table format got changed but the frozen code didn't get
re-generated. If you try to call co_lines() on the __hello__ code, Python
crashes.
>>> import __hello__
Hello world!
>>> co = __hello__.__spec__.loader.get_code('__hello__')
>>> co.co_linetable
b'\x04\x01'
>>> list(co.co_lines())
python: ../Objects/codeobject.c:1185: PyLineTable_NextAddressRange: Assertion
`!at_end(range)' failed.
My PR re-generates the code and fixes the test. Perhaps I should also add a
test to exercise co_lines() on the frozen code object.
--
___
Python tracker
<https://bugs.python.org/issue43372>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42212] Check if generated files are up-to-date in Github Actions
Change by Neil Schemenauer : -- nosy: +nascheme nosy_count: 3.0 -> 4.0 pull_requests: +23486 pull_request: https://github.com/python/cpython/pull/24708 ___ Python tracker <https://bugs.python.org/issue42212> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37448] obmalloc: radix tree for tracking arena address ranges
Change by Neil Schemenauer : Added file: https://bugs.python.org/file49834/perf_compare_radix4x.txt ___ Python tracker <https://bugs.python.org/issue37448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37448] obmalloc: radix tree for tracking arena address ranges
Change by Neil Schemenauer : Added file: https://bugs.python.org/file49833/perf_compare_noradix.txt ___ Python tracker <https://bugs.python.org/issue37448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43288] test_importlib failure due to missing skip() method
Neil Schemenauer added the comment: New changeset 44fe32061d60f4bd9c4fa48c24e3e4ba26033dba by Neil Schemenauer in branch '3.9': [3.9] bpo-43288: Fix bug in test_importlib test. (GH-24616) https://github.com/python/cpython/commit/44fe32061d60f4bd9c4fa48c24e3e4ba26033dba -- ___ Python tracker <https://bugs.python.org/issue43288> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43288] test_importlib failure due to missing skip() method
Change by Neil Schemenauer : -- pull_requests: +23397 pull_request: https://github.com/python/cpython/pull/24616 ___ Python tracker <https://bugs.python.org/issue43288> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43288] test_importlib failure due to missing skip() method
Neil Schemenauer added the comment: New changeset 84f7afe65c29330f3ff8e318e054b96554a2dede by Neil Schemenauer in branch 'master': Fix failed merge of bpo-43288. (GH-24614) https://github.com/python/cpython/commit/84f7afe65c29330f3ff8e318e054b96554a2dede -- ___ Python tracker <https://bugs.python.org/issue43288> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
