Re: [Gimp-developer] memory manage in python-fu
Hi, On Wed, 2007-03-14 at 09:01 -0700, William Skaggs wrote: It might help if I explain a little more about how GIMP handles memory. GIMP does not rely on the operating system for swapping. When you start GIMP, it creates a file, of fixed size which you designate in the Preferences, to serve as a swap area. Sorry, but I have to correct you. The size of the swap file is of course not fixed. It will grow and may even shrink with the amount of swap space that is needed. I said that it may shrink because there is currently no code to deal with holes in the swap file. So it will only shrink if the tiles stored at its end are released. What is happening to you is that the swap file is filling up, because data is accumulating there that ought to be cleared away. You could delay the failure by increasing the size of the swap file (called the tile cache) in Preferences The tile-cache-size in the Preferences is the amount of memory GIMP will use before starting to swap tiles out to the swap file. It does not specify the size of the swap file. Sven ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
Hi, On Tue, 2007-03-13 at 22:18 -0600, D. Stimits wrote: There could be a mix of a gimp bug, and definitely some portion of my plug-in. Although I know for a fact that I had a memory leak that was my fault, the reaction to the leak was unexpected. The system in question has only a /boot and /, with well in excess of 30 GB unused (lots of inodes as well). For a moment I thought that perhaps it was a max open file descriptor limit...which it could still be, but the system itself has a default setting of a max of around 76,618. The thing I did not expect was that it had well over a gig of ram left unused, no swap used, 30 gig of unused drive, and it still thought the hard drive was full. On the other hand, it might not be worth pursuing when it requires a memory leak for it to show up. How is that unexpected behavior? That's exactly what I would expect if you configure the GIMP tile-cache-size to be smaller than the amount of physical memory. GIMP will then start to swap tiles out even before it runs out of memory. And if you don't delete images, you will end up with your harddisk filled up. Sven ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
[...] The thing I did not expect was that it had well over a gig of ram left unused, no swap used, 30 gig of unused drive, and it still thought the hard drive was full. [...] It might help if I explain a little more about how GIMP handles memory. GIMP does not rely on the operating system for swapping. When you start GIMP, it creates a file, of fixed size which you designate in the Preferences, to serve as a swap area. You also tell GIMP, in the Preferences, how much RAM it is allowed to use. If GIMP needs more memory, it moves some of its data from RAM into the swap file. What is happening to you is that the swap file is filling up, because data is accumulating there that ought to be cleared away. You could delay the failure by increasing the size of the swap file (called the tile cache) in Preferences, but really, in the work flow you described, it ought not to be filling up, so there is a bug either in your code or in GIMP -- and as I said earlier, I think there is probably a bug in GIMP. (I am not 100% certain that I got all the details right here. Presumably Sven or Mitch will correct any mis-statement.) -- Bill __ __ __ __ Sent via the CNPRC Email system at primate.ucdavis.edu ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
On 3/14/07, William Skaggs wrote: What is happening to you is that the swap file is filling up, because data is accumulating there that ought to be cleared away. btw, is it possible to provide some feedback to users that available resources are getting pretty close to zero? Alexandre ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
On Wednesday 14 March 2007 17:01, William Skaggs wrote: It might help if I explain a little more about how GIMP handles memory. GIMP does not rely on the operating system for swapping. When you start GIMP, it creates a file, of fixed size which you designate in the Preferences, to serve as a swap area. You also tell GIMP, in the Preferences, how much RAM it is allowed to use. If GIMP needs more memory, it moves some of its data from RAM into the swap file. Strange... I have 1GB of RAM, and 1GB of swap. My tile cache is set to 512MB, and undo history to 256MB. I recently tried to edit a big pano. After one color correction, The RAM get full, and the system swap too! Then, all stopped working. According to your explanation, it should not do that. Do I missed something? -- Frédéric http://www.gbiloba.org pgpXVfWcUvltG.pgp Description: PGP signature ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
Hi, On Mon, 2007-03-12 at 06:51 -0800, William Skaggs wrote: I think you have hit an actual bug, and that it doesn't have anything to do with python, because I have encountered a similar thing in a modified version of gimpressionist that I worked up, written purely in C. I believe that there is some sort of memory leak that causes gimp in some situations to maintain reference to tiles that are no longer being used in any way, and it is something that shows up when you create and delete layers over and over again, in a certain way. So it would probably be useful for you to file a bug report about this, if you would. It would only be useful if that bug report included a way to reproduce the problem. And it would probably be helpful to discuss that code example beforehand. I am not saying that there are no memory leaks in GIMP. But we are frequently testing the software for this and I am not aware of any major memory leaks such as the one that Bill describes here. The memory that is being exhausted, by the way, is the swap area that gimp allocates on the hard disk each time you run it. The tile manager moves tile data there if space is needed. Actually, it doesn't do that when space is needed, but when the memory allocated for tiles exceeds the limit configured as 'tile-cache-size'. Sven ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
... I think you have hit an actual bug, and that it doesn't have anything to do with python, because I have encountered a similar thing in a modified version of gimpressionist that I worked up, written purely in C. I believe that there is some sort of memory leak that causes gimp in some situations to maintain reference to tiles that are no longer being used in any way, and it is something that shows up when you create and delete layers over and over again, in a certain way. So it would probably be useful for you to file a bug report about this, if you would. It would only be useful if that bug report included a way to reproduce the problem. And it would probably be helpful to discuss that code example beforehand. I am not saying that there are no memory leaks in GIMP. But we are frequently testing the software for this and I am not aware of any major memory leaks such as the one that Bill describes here. The memory that is being exhausted, by the way, is the swap area that gimp allocates on the hard disk each time you run it. The tile manager moves tile data there if space is needed. Actually, it doesn't do that when space is needed, but when the memory allocated for tiles exceeds the limit configured as 'tile-cache-size'. There could be a mix of a gimp bug, and definitely some portion of my plug-in. Although I know for a fact that I had a memory leak that was my fault, the reaction to the leak was unexpected. The system in question has only a /boot and /, with well in excess of 30 GB unused (lots of inodes as well). For a moment I thought that perhaps it was a max open file descriptor limit...which it could still be, but the system itself has a default setting of a max of around 76,618. The thing I did not expect was that it had well over a gig of ram left unused, no swap used, 30 gig of unused drive, and it still thought the hard drive was full. On the other hand, it might not be worth pursuing when it requires a memory leak for it to show up. I do believe that after using the delete function in the loop, that it still used up memory which it did not release. The amount was barely noticable though, and after running all night to create 45,000 800x800 color jpegs, I'd say the amount was barely distinguishable as above the amount it started with (I didn't take exact measures, I was going by a mix of top and a graphical tool). Without a breakdown though of where gimp was using the memory, it'd be a bit of a stretch though to say it truly was a leak...it could simply be doing something like caching of something, e.g., it was using the text generation and changing fonts on each of the 45,000 images. I know gimp has some ability to put out debug code, but I'm curious if it has any sort of way of dumping statistics on how it's using memory? In the future I'll have to work on several plug-ins to do mass image handling, probably less than 45,000 per batch most of the time. I'll watch and see if I can find anything useful to report. Otherwise, just have a loop intentionally create 800x800 images without deleting, expect it to run out of resources, but see if it does it in the way expected...mine ran out of resources LONG before the system was even stressed slightly. Again, I suspect the slow laptop hard drive compared to fast core2 and tons of ram made it a race the hd could not win. D. Stimits, stimits AT comcast DOT net ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
I am hereforth asking you to apologise, if possible, both in private and on the list for this message. We are runing a project needing I cannot sincerely do so, in this instance. I do not apologize for expressing my amusement, unless I judge that the situation was genuinely worsened by my behaviour. It's certainly true that my message can be misinterpreted -- on the net, it's probably wisest to just assume no hidden meanings -- if it looks like a fish, assume it is a fish (and nothing more). much of new developers and contributors, and are in no position of theating anyone like this. Please perceive I am no police - I am asking this personally, but I also have promised the people organizing the LGM I'd tlak to gimp developers to try to make the enviroment more frindly to newcomers. In my observation, friendly sometimes is not polite -- for instance, answering as I did rather than giving a detached commentary free of any emotional context at all. This is why I view the way I replied as better (and certainly more friendly) than most previous mails of mine on this list. (btw, having to explicitely delete a python object __is__ a bug - objects that are seeing in python should be garbage collected when they are no longer referecenced. ) I avoided commenting on that because it wasn't clear whether they actually were no longer referenced. I certainly agree that there is something wrong with the gimp.delete convention here -- it only behaves how I'd expect sometimes. I'd definitely like to see more PyGimp developers myself -- hardly anyone seems to make as extensive use of it as I do. Perhaps a standard module path for pygimp plugins could be worked out (arguably, the PDB removes this need, but the PDB is cranky to use compared to Python modules) ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
On Mon, 12 Mar 2007 00:54:56 +0100, D. Stimits [EMAIL PROTECTED] wrote: Speculating about why it claimed it was out of memory, I'm wondering if the hard drive simply was too slow responding. It's a laptop, running with core duo, entirely in ram. The drive is thus much slower than the ability to consume ram. How does gimp determine that there is no space left on the device...is it a timeout in addition to other means? I saw something a little like this recently. I cant document it thoroughly but here's a resumé in case it's related. I tried to open about eight multi-megabyte jpeg images in one open operation not wanting to go through the labourious open dialogue eight times and not noticing quite how big they were. I was half expecting some error after a few were opened but expected error trapping to handle it correctly. What I ended up with was a completely unstable mess which I had to clean up with xkill , restart the xserver and a bit of ps and kill to clean up the residue. Now this could be regarded as extreme conditions and represents a good stress test. It seems running out of RAM and/or swap is not properly error trapped. I just throw this info in in case it relates the this script issue posted. gg ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
Von: David Gowers [EMAIL PROTECTED] I am hereforth asking you to apologise, if possible, both in private and on the list for this message. We are runing a project needing I cannot sincerely do so, in this instance. I do not apologize for expressing my amusement, unless I judge that the situation was genuinely worsened by my behaviour. We could make it the policy that anyone who does shout RTFM! has to: - make sure that there is a FM - make sure that it is up-to-date - provide the URL In this case, this would be like: RTFM! http://www.gimp.org/docs/python/index.html; I'll leave the check whether this is up-to-date for David as an exercise :) HTH, Michael -- Feel free - 5 GB Mailbox, 50 FreeSMS/Monat ... Jetzt GMX ProMail testen: www.gmx.net/de/go/mailfooter/promail-out ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
I think you have hit an actual bug, and that it doesn't have anything to do with python, because I have encountered a similar thing in a modified version of gimpressionist that I worked up, written purely in C. I believe that there is some sort of memory leak that causes gimp in some situations to maintain reference to tiles that are no longer being used in any way, and it is something that shows up when you create and delete layers over and over again, in a certain way. So it would probably be useful for you to file a bug report about this, if you would. The memory that is being exhausted, by the way, is the swap area that gimp allocates on the hard disk each time you run it. The tile manager moves tile data there if space is needed. -- Bill __ __ __ __ Sent via the CNPRC Email system at primate.ucdavis.edu ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
[Gimp-developer] memory manage in python-fu
I've created a simple python-fu plugin to generate test images for use in another project (all on linux, gimp 2.2.13, fedora core 5). It works, but terminates before finished. What the plugin does is to create images of a given specification, in a loop. Each pass of the loop alters the image slightly. It creates the image, it creates a layer, it does some painting (basically bucket fills and text), then it saves the file. It does not do any interactive display of the image. After about 4000 images, it fails, claiming no space left on device. However, there's about another 30 GB left. Furthermore, I monitor ram use, and there is still about 1.25 GB of ram left (it has swap turned off, it's all physical ram). Monitoring ram, I do see ram use continually climb, and never drop (actual use, not cache). The amount roughly appears to corresponds to the image sizes. I need to generate approximately 45000 files per batch. I have not found any python-fu way to close a file, or to reclaim memory after creating an image or layer. Is there such a thing? Is there instead some sort of garbage collection? Speculating about why it claimed it was out of memory, I'm wondering if the hard drive simply was too slow responding. It's a laptop, running with core duo, entirely in ram. The drive is thus much slower than the ability to consume ram. How does gimp determine that there is no space left on the device...is it a timeout in addition to other means? D. Stimits, stimits AT comcast DOT net ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
On 3/12/07, D. Stimits [EMAIL PROTECTED] wrote: I have not found any python-fu way to close a file, or to reclaim memory after creating an image or layer. Is there such a thing? Is there instead some sort of garbage collection? hahahaha! gimp.delete(image) or gimp.delete(layer) #only do this if the layer isn't attached to an image. If it is attached to an image, then deleting the image will delete all the layers and other things -- no need to delete them directly. This *is* in the pygimp documentation (plug-ins/pygimp/doc/pygimp.html in the source tree)! RTFM! ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
Re: [Gimp-developer] memory manage in python-fu
On Sunday 11 March 2007 20:54, D. Stimits wrote: I have not found any python-fu way to close a file, or to reclaim memory after creating an image or layer. Is there such a thing? Is there instead some sort of garbage collection? Speculating about why it claimed it was out of memory, I'm wondering if the hard drive simply was too slow responding. It's a laptop, running with core duo, entirely in ram. The drive is thus much slower than the ability to consume ram. How does gimp determine that there is no space left on the device...is it a timeout in addition to other means? The normal python way would be to set the object for destruction after it is no longer used. However, it may well be possible that the GIMP bindings to python are not deleting GIMP images creted from within python, as it should be done. I had, myself, never thought of using a script o create so many images. As you rerro r mesage is no space left on device - please try to cehck gimp preferences for tmp dirs, maybe it is using another partition that might be getting filled. The increaseof RAM usage for itself however shows there is something wrong going on - could you please fill a bug report about this at bugzilla.gnome.org ? If possible, please attach a minimalistic version of your script taht would trigger the problem. Thank you, js -- ___ Gimp-developer mailing list Gimp-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
