Re: Question about garbage collection
On 2024-01-17 3:01 AM, Greg Ewing via Python-list wrote: On 17/01/24 1:01 am, Frank Millman wrote: I sometimes need to keep a reference from a transient object to a more permanent structure in my app. To save myself the extra step of removing all these references when the transient object is deleted, I make them weak references. I don't see how weak references help here at all. If the transient object goes away, all references from it to the permanent objects also go away. A weak reference would only be of use if the reference went the other way, i.e. from the permanent object to the transient object. You are right. I got my description above back-to-front. It is a pub/sub scenario. A transient object makes a request to the permanent object to be notified of any changes. The permanent object stores a reference to the transient object and executes a callback on each change. When the transient object goes away, the reference must be removed. Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 17/01/24 1:01 am, Frank Millman wrote: I sometimes need to keep a reference from a transient object to a more permanent structure in my app. To save myself the extra step of removing all these references when the transient object is deleted, I make them weak references. I don't see how weak references help here at all. If the transient object goes away, all references from it to the permanent objects also go away. A weak reference would only be of use if the reference went the other way, i.e. from the permanent object to the transient object. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 17/01/24 4:00 am, Chris Angelico wrote: class Form: def __init__(self): self.elements = [] class Element: def __init__(self, form): self.form = form form.elements.append(self) If you make the reference from Element to Form a weak reference, it won't keep the Form alive after it's been closed. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more
> On 16 Jan 2024, at 17:11, Sibylle Koczian via Python-list > wrote: > > while the new Windows 11 machine finds the Microsoft stub You can turn off the stub in windows settings. The magic windows jargon is “App Execution Aliases”. Once you find it in settings you can turn off the python and python3 aliases. Barry -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
> On 16 Jan 2024, at 12:10, Frank Millman via Python-list > wrote: > > My problem is that my app is quite complex, and it is easy to leave a > reference dangling somewhere which prevents an object from being gc'd. What I do to track these problems down is use gc.get_objects() then summerize the number of each type. Part 2 is to print the delta after an interval of a 2nd summary. Leaks of objects show up as the count of a type increasing every time you sample. Barry -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
> On 16 Jan 2024, at 13:17, Thomas Passin via Python-list > wrote: > > The usual advice is to call deleteLater() on objects derived from PyQt > classes. I don't know enough about PyQt to know if this takes care of all > dangling reference problems, though. It works well and robustly. Barry -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more
Am 15.01.2024 um 23:55 schrieb Mats Wichmann via Python-list: On 1/15/24 12:01, Thomas Passin via Python-list wrote: On 1/15/2024 1:26 PM, Mats Wichmann via Python-list wrote: Python from the App Store is not the same as Python from python.org: yes. this question is about the python.org distribution. but, Windows natively has something called python.exe and python3.exe which is interfering here, IF the python.org install isn't directed to put itself into the path, AND if the "#!/usr/bin/env python3" form is used, causing a search in PATH, which is the setup Sibylle has described, unless I've misunderstood details. No, you didn't misunderstand any detail. It's exactly right. My Windows 10 box doesn't find anything for "where python", "where python3", while the new Windows 11 machine finds the Microsoft stub. "Irritating" is a very friendly attribute for that thing. Why must it be called "python.exe" and not something else like the installation files from python.org? I'll stop using "/env" - hopefully that won't create problems with the scripts I use in the Linux VM. But in that case I'll know what's up. Thank you very much! Sibylle -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On Wed, 17 Jan 2024 at 01:45, Frank Millman via Python-list wrote: > > On 2024-01-16 2:15 PM, Chris Angelico via Python-list wrote: > > > > Where do you tend to "leave a reference dangling somewhere"? How is > > this occurring? Is it a result of an incomplete transaction (like an > > HTTP request that never finishes), or a regular part of the operation > > of the server? > > > > I have a class that represents a database table, and another class that > represents a database column. There is a one-to-many relationship and > they maintain references to each other. > > In another part of the app, there is a class that represents a form, and > another class that represents the gui elements on the form. Again there > is a one-to-many relationship. I don't know when you'd be "done" with the table, so I won't try to give an example, but I'll try this one and maybe it'll give some ideas that could apply to both. When you open the form, you initialize it, display it, etc, etc. This presumably includes something broadly like this: class Form: def __init__(self): self.elements = [] class Element: def __init__(self, form): self.form = form form.elements.append(self) frm = Form(...) Element(frm, ...) # as many as needed frm.show() # present it to the user This is a pretty classic refloop. I don't know exactly what your setup is, but most likely it's going to look something like this. Feel free to correct me if it doesn't. The solution here would be to trap the "form is no longer being displayed" moment. That'll be some sort of GUI event like a "close" or "delete" signal. When that comes through (and maybe after doing other processing), you no longer need the form, and can dispose of it. The simplest solution here is: Empty out frm.elements. That immediately leaves the form itself as a leaf (no references to anything relevant), and the elements still refer back to it, but once nothing ELSE refers to the form, everything can be disposed of. > A gui element that represents a piece of data has to maintain a link to > its database column object. There can be a many-to-one relationship, as > there could be more than one gui element referring to the same column. Okay, so the Element also refers to the corresponding Column. If the Form and Element aren't in a refloop, this shouldn't be a problem. However, if this is the same Table and Column that you referred to above, that might be the answer to my question. Are you "done" with the Table at the same time that the form is no longer visible? If so, you would probably have something similar where the Form refers to the Table, and the Table and Columns refer to each other... so the same solution hopefully should work: wipe out the Table's list of columns. > There are added complications which I won't go into here. The bottom > line is that on some occasions a form which has been closed does not get > gc'd. > > I have been trying to reproduce the problem in my toy app, but I cannot > get it to fail. There is a clue there! I think I have just > over-complicated things. Definitely possible. > I will start with a fresh approach tomorrow. If you don't hear from me > again, you will know that I have solved it! > > Thanks for the input, it definitely helped. Cool cool, happy to help. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 2024-01-16 2:15 PM, Chris Angelico via Python-list wrote: Where do you tend to "leave a reference dangling somewhere"? How is this occurring? Is it a result of an incomplete transaction (like an HTTP request that never finishes), or a regular part of the operation of the server? I have a class that represents a database table, and another class that represents a database column. There is a one-to-many relationship and they maintain references to each other. In another part of the app, there is a class that represents a form, and another class that represents the gui elements on the form. Again there is a one-to-many relationship. A gui element that represents a piece of data has to maintain a link to its database column object. There can be a many-to-one relationship, as there could be more than one gui element referring to the same column. There are added complications which I won't go into here. The bottom line is that on some occasions a form which has been closed does not get gc'd. I have been trying to reproduce the problem in my toy app, but I cannot get it to fail. There is a clue there! I think I have just over-complicated things. I will start with a fresh approach tomorrow. If you don't hear from me again, you will know that I have solved it! Thanks for the input, it definitely helped. Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 1/16/2024 4:17 AM, Barry wrote: On 16 Jan 2024, at 03:49, Thomas Passin via Python-list wrote: This kind of thing can happen with PyQt, also. There are ways to minimize it but I don't know if you can ever be sure all Qt C++ objects will get deleted. It depends on the type of object and the circumstances. When this has been seen in the past it has been promptly fixed by the maintainer. The usual advice is to call deleteLater() on objects derived from PyQt classes. I don't know enough about PyQt to know if this takes care of all dangling reference problems, though. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On Tue, 16 Jan 2024 at 23:08, Frank Millman via Python-list wrote: > > On 2024-01-15 3:51 PM, Frank Millman via Python-list wrote: > > Hi all > > > > I have read that one should not have to worry about garbage collection > > in modern versions of Python - it 'just works'. > > > > I don't want to rely on that. My app is a long-running server, with > > multiple clients logging on, doing stuff, and logging off. They can > > create many objects, some of them long-lasting. I want to be sure that > > all objects created are gc'd when the session ends. > > > > I did not explain myself very well. Sorry about that. > > My problem is that my app is quite complex, and it is easy to leave a > reference dangling somewhere which prevents an object from being gc'd. > > This can create (at least) two problems. The obvious one is a memory > leak. The second is that I sometimes need to keep a reference from a > transient object to a more permanent structure in my app. To save myself > the extra step of removing all these references when the transient > object is deleted, I make them weak references. This works, unless the > transient object is kept alive by mistake and the weak ref is never removed. > > I feel it is important to find these dangling references and fix them, > rather than wait for problems to appear in production. The only method I > can come up with is to use the 'delwatcher' class that I used in my toy > program in my original post. > > I am surprised that this issue does not crop up more often. Does nobody > else have these problems? > It really depends on how big those dangling objects are. My personal habit is to not worry about a few loose objects, by virtue of ensuring that everything either has its reference loops deliberately broken at some point in time, or by keeping things small. An example of deliberately breaking a refloop would be when I track websockets. Usually I'll tag the socket object itself with some kind of back-reference to my own state, but I also need to be able to iterate over all of my own state objects (let's say they're dictionaries for simplicity) and send a message to each socket. So there'll be a reference loop between the socket and the state. But at some point, I will be notified that the socket has been disconnected, and that's when I go to its state object and wipe out its back-reference. It can then be disposed of promptly, since there's no loop. It takes a bit of care, but in general, large state objects won't have these kinds of loops, and dangling references haven't caused me any sort of major issues in production. Where do you tend to "leave a reference dangling somewhere"? How is this occurring? Is it a result of an incomplete transaction (like an HTTP request that never finishes), or a regular part of the operation of the server? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 2024-01-15 3:51 PM, Frank Millman via Python-list wrote: Hi all I have read that one should not have to worry about garbage collection in modern versions of Python - it 'just works'. I don't want to rely on that. My app is a long-running server, with multiple clients logging on, doing stuff, and logging off. They can create many objects, some of them long-lasting. I want to be sure that all objects created are gc'd when the session ends. I did not explain myself very well. Sorry about that. My problem is that my app is quite complex, and it is easy to leave a reference dangling somewhere which prevents an object from being gc'd. This can create (at least) two problems. The obvious one is a memory leak. The second is that I sometimes need to keep a reference from a transient object to a more permanent structure in my app. To save myself the extra step of removing all these references when the transient object is deleted, I make them weak references. This works, unless the transient object is kept alive by mistake and the weak ref is never removed. I feel it is important to find these dangling references and fix them, rather than wait for problems to appear in production. The only method I can come up with is to use the 'delwatcher' class that I used in my toy program in my original post. I am surprised that this issue does not crop up more often. Does nobody else have these problems? Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
> On 16 Jan 2024, at 03:49, Thomas Passin via Python-list > wrote: > > This kind of thing can happen with PyQt, also. There are ways to minimize it > but I don't know if you can ever be sure all Qt C++ objects will get deleted. > It depends on the type of object and the circumstances. When this has been seen in the past it has been promptly fixed by the maintainer. Barry -- https://mail.python.org/mailman/listinfo/python-list
