Re: [Gimp-developer] memory manage in python-fu

2007-03-15 Thread Sven Neumann
Hi,

On Wed, 2007-03-14 at 09:01 -0700, William Skaggs wrote:

 It might help if I explain a little more about how GIMP handles
 memory.  GIMP does not rely on the operating system for swapping.
 When you start GIMP, it creates a file, of fixed size which you
 designate in the Preferences, to serve as a swap area.

Sorry, but I have to correct you. The size of the swap file is of course
not fixed. It will grow and may even shrink with the amount of swap
space that is needed. I said that it may shrink because there is
currently no code to deal with holes in the swap file. So it will only
shrink if the tiles stored at its end are released.

 What is happening to you is that the swap file is filling up,
 because data is accumulating there that ought to be cleared
 away.  You could delay the failure by increasing the size of
 the swap file (called the tile cache) in Preferences

The tile-cache-size in the Preferences is the amount of memory GIMP will
use before starting to swap tiles out to the swap file. It does not
specify the size of the swap file.


Sven


___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-14 Thread Sven Neumann
Hi,

On Tue, 2007-03-13 at 22:18 -0600, D. Stimits wrote:

 There could be a mix of a gimp bug, and definitely some portion of my 
 plug-in. Although I know for a fact that I had a memory leak that was my 
 fault, the reaction to the leak was unexpected. The system in question 
 has only a /boot and /, with well in excess of 30 GB unused (lots of 
 inodes as well). For a moment I thought that perhaps it was a max open 
 file descriptor limit...which it could still be, but the system itself 
 has a default setting of a max of around 76,618. The thing I did not 
 expect was that it had well over a gig of ram left unused, no swap used, 
 30 gig of unused drive, and it still thought the hard drive was full. On 
 the other hand, it might not be worth pursuing when it requires a memory 
 leak for it to show up.

How is that unexpected behavior? That's exactly what I would expect if
you configure the GIMP tile-cache-size to be smaller than the amount of
physical memory. GIMP will then start to swap tiles out even before it
runs out of memory. And if you don't delete images, you will end up with
your harddisk filled up.


Sven


___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-14 Thread William Skaggs

 [...] The thing I did not 
expect was that it had well over a gig of ram left unused, no swap used, 
30 gig of unused drive, and it still thought the hard drive was full. 
 [...]

It might help if I explain a little more about how GIMP handles
memory.  GIMP does not rely on the operating system for swapping.
When you start GIMP, it creates a file, of fixed size which you
designate in the Preferences, to serve as a swap area.  You also
tell GIMP, in the Preferences, how much RAM it is allowed to use.
If GIMP needs more memory, it moves some of its data from RAM
into the swap file.

What is happening to you is that the swap file is filling up,
because data is accumulating there that ought to be cleared
away.  You could delay the failure by increasing the size of
the swap file (called the tile cache) in Preferences, but
really, in the work flow you described, it ought not to be
filling up, so there is a bug either in your code or in GIMP --
and as I said earlier, I think there is probably a bug in
GIMP.

(I am not 100% certain that I got all the details right here.
Presumably Sven or Mitch will correct any mis-statement.)

  -- Bill
 

 
__ __ __ __
Sent via the CNPRC Email system at primate.ucdavis.edu


 
   
___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-14 Thread Alexandre Prokoudine
On 3/14/07, William Skaggs wrote:

 What is happening to you is that the swap file is filling up,
 because data is accumulating there that ought to be cleared
 away.

btw, is it possible to provide some feedback to users that available
resources are getting pretty close to zero?

Alexandre
___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-14 Thread Frédéric
On Wednesday 14 March 2007 17:01, William Skaggs wrote:

 It might help if I explain a little more about how GIMP handles
 memory.  GIMP does not rely on the operating system for swapping.
 When you start GIMP, it creates a file, of fixed size which you
 designate in the Preferences, to serve as a swap area.  You also
 tell GIMP, in the Preferences, how much RAM it is allowed to use.
 If GIMP needs more memory, it moves some of its data from RAM
 into the swap file.

Strange... I have 1GB of RAM, and 1GB of swap. My tile cache is set to 
512MB, and undo history to 256MB. I recently tried to edit a big pano. 
After one color correction, The RAM get full, and the system swap too! 
Then, all stopped working.

According to your explanation, it should not do that. Do I missed 
something?

-- 
   Frédéric

   http://www.gbiloba.org


pgpXVfWcUvltG.pgp
Description: PGP signature
___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-13 Thread Sven Neumann
Hi,

On Mon, 2007-03-12 at 06:51 -0800, William Skaggs wrote:
 I think you have hit an actual bug, and that it doesn't have anything
 to do with python, because I have encountered a similar thing in a
 modified version of gimpressionist that I worked up, written purely
 in C.  I believe that there is some sort of memory leak that causes
 gimp in some situations to maintain reference to tiles that are no
 longer being used in any way, and it is something that shows up when
 you create and delete layers over and over again, in a certain way.  
 So it would probably be useful for you to file a bug report about this, 
 if you would.

It would only be useful if that bug report included a way to reproduce
the problem. And it would probably be helpful to discuss that code
example beforehand.

I am not saying that there are no memory leaks in GIMP. But we are
frequently testing the software for this and I am not aware of any major
memory leaks such as the one that Bill describes here.

 The memory that is being exhausted, by the way, is the swap area
 that gimp allocates on the hard disk each time you run it.  The
 tile manager moves tile data there if space is needed.

Actually, it doesn't do that when space is needed, but when the memory
allocated for tiles exceeds the limit configured as 'tile-cache-size'.


Sven


___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-13 Thread D. Stimits
...

I think you have hit an actual bug, and that it doesn't have anything
to do with python, because I have encountered a similar thing in a
modified version of gimpressionist that I worked up, written purely
in C.  I believe that there is some sort of memory leak that causes
gimp in some situations to maintain reference to tiles that are no
longer being used in any way, and it is something that shows up when
you create and delete layers over and over again, in a certain way.  
So it would probably be useful for you to file a bug report about this, 
if you would.



It would only be useful if that bug report included a way to reproduce
the problem. And it would probably be helpful to discuss that code
example beforehand.

I am not saying that there are no memory leaks in GIMP. But we are
frequently testing the software for this and I am not aware of any major
memory leaks such as the one that Bill describes here.

  

The memory that is being exhausted, by the way, is the swap area
that gimp allocates on the hard disk each time you run it.  The
tile manager moves tile data there if space is needed.



Actually, it doesn't do that when space is needed, but when the memory
allocated for tiles exceeds the limit configured as 'tile-cache-size'.
  


There could be a mix of a gimp bug, and definitely some portion of my 
plug-in. Although I know for a fact that I had a memory leak that was my 
fault, the reaction to the leak was unexpected. The system in question 
has only a /boot and /, with well in excess of 30 GB unused (lots of 
inodes as well). For a moment I thought that perhaps it was a max open 
file descriptor limit...which it could still be, but the system itself 
has a default setting of a max of around 76,618. The thing I did not 
expect was that it had well over a gig of ram left unused, no swap used, 
30 gig of unused drive, and it still thought the hard drive was full. On 
the other hand, it might not be worth pursuing when it requires a memory 
leak for it to show up.

I do believe that after using the delete function in the loop, that it 
still used up memory which it did not release. The amount was barely 
noticable though, and after running all night to create 45,000 800x800 
color jpegs, I'd say the amount was barely distinguishable as above the 
amount it started with (I didn't take exact measures, I was going by a 
mix of top and a graphical tool). Without a breakdown though of where 
gimp was using the memory, it'd be a bit of a stretch though to say it 
truly was a leak...it could simply be doing something like caching of 
something, e.g., it was using the text generation and changing fonts on 
each of the 45,000 images.

I know gimp has some ability to put out debug code, but I'm curious if 
it has any sort of way of dumping statistics on how it's using memory?

In the future I'll have to work on several plug-ins to do mass image 
handling, probably less than 45,000 per batch most of the time. I'll 
watch and see if I can find anything useful to report. Otherwise, just 
have a loop intentionally create 800x800 images without deleting, expect 
it to run out of resources, but see if it does it in the way 
expected...mine ran out of resources LONG before the system was even 
stressed slightly. Again, I suspect the slow laptop hard drive compared 
to fast core2 and tons of ram made it a race the hd could not win.

D. Stimits, stimits AT comcast DOT net
___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-12 Thread David Gowers

I am hereforth asking you to apologise, if possible, both in private

and on the list for this message. We are runing a project needing



I cannot sincerely do so, in this instance. I do not apologize for
expressing my amusement, unless I judge that the situation was genuinely
worsened by my behaviour.

It's certainly true that my message can be misinterpreted -- on the net,
it's probably wisest to just assume no hidden meanings -- if it looks like a
fish, assume it is a fish (and nothing more).

much of new developers and contributors, and are in no position of

theating anyone like this. Please perceive I am no  police  - I am
asking this personally, but I also have promised the people
organizing the LGM I'd tlak to gimp developers to try to make the
enviroment more frindly to newcomers.



In my observation, friendly sometimes is not  polite -- for instance,
answering as I did rather than giving a detached commentary free of any
emotional context at all.
This is why I view the way I replied as better (and certainly more friendly)
than most previous mails of mine on this list.

(btw, having to  explicitely delete  a python object __is__ a bug -

objects that are seeing in python should be garbage collected when
they are no longer referecenced. )



I avoided commenting on that because it wasn't clear whether they actually
were no longer referenced. I certainly agree that there is something wrong
with the gimp.delete convention here -- it only behaves how I'd expect
sometimes.

I'd definitely like to see more PyGimp developers myself -- hardly anyone
seems to make as extensive use of it as I do. Perhaps a standard module path
for pygimp plugins could be worked out (arguably, the PDB removes this need,
but the PDB is cranky to use compared to Python modules)
___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-12 Thread gg
On Mon, 12 Mar 2007 00:54:56 +0100, D. Stimits [EMAIL PROTECTED] wrote:

 Speculating about why it claimed it was out of memory, I'm wondering if
 the hard drive simply was too slow responding. It's a laptop, running
 with core duo, entirely in ram. The drive is thus much slower than the
 ability to consume ram. How does gimp determine that there is no space
 left on the device...is it a timeout in addition to other means?


I saw something a little like this recently. I cant document it thoroughly  
but here's a resumé in case it's related.

I tried to open about eight multi-megabyte jpeg images in one open  
operation not wanting to go through the labourious open dialogue eight  
times and not noticing quite how big they were. I was half expecting some  
error after a few were opened but expected error trapping to handle it  
correctly.

What I ended up with was a completely unstable mess which I had to clean  
up with xkill , restart the xserver and a bit of ps and kill to clean up  
the residue.

Now this could be regarded as extreme conditions and represents a good  
stress test. It seems running out of RAM and/or swap is not properly error  
trapped.

I just throw this info in in case it relates the this script issue posted.


gg
___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-12 Thread Michael Schumacher
Von: David Gowers [EMAIL PROTECTED]

 I am hereforth asking you to apologise, if possible, both in private
  and on the list for this message. We are runing a project needing
 
 I cannot sincerely do so, in this instance. I do not apologize for
 expressing my amusement, unless I judge that the situation was genuinely
 worsened by my behaviour.

We could make it the policy that anyone who does shout RTFM! has to:

- make sure that there is a FM
- make sure that it is up-to-date
- provide the URL

In this case, this would be like:

RTFM! http://www.gimp.org/docs/python/index.html;

I'll leave the check whether this is up-to-date for David as an exercise :)


HTH,
Michael
-- 
Feel free - 5 GB Mailbox, 50 FreeSMS/Monat ...
Jetzt GMX ProMail testen: www.gmx.net/de/go/mailfooter/promail-out
___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-12 Thread William Skaggs

I think you have hit an actual bug, and that it doesn't have anything
to do with python, because I have encountered a similar thing in a
modified version of gimpressionist that I worked up, written purely
in C.  I believe that there is some sort of memory leak that causes
gimp in some situations to maintain reference to tiles that are no
longer being used in any way, and it is something that shows up when
you create and delete layers over and over again, in a certain way.  
So it would probably be useful for you to file a bug report about this, 
if you would.

The memory that is being exhausted, by the way, is the swap area
that gimp allocates on the hard disk each time you run it.  The
tile manager moves tile data there if space is needed.

  -- Bill
 

 
__ __ __ __
Sent via the CNPRC Email system at primate.ucdavis.edu


 
   
___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-11 Thread David Gowers

On 3/12/07, D. Stimits [EMAIL PROTECTED] wrote:



I have not found any python-fu way to close a file, or to reclaim
memory after creating an image or layer. Is there such a thing? Is there
instead some sort of garbage collection?



hahahaha!

gimp.delete(image)

or
gimp.delete(layer) #only do this if the layer isn't attached to an image. If
it is attached to an image, then deleting the image will delete all the
layers and other things -- no need to delete them directly.

This *is* in the pygimp documentation (plug-ins/pygimp/doc/pygimp.html in
the source tree)! RTFM!
___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer


Re: [Gimp-developer] memory manage in python-fu

2007-03-11 Thread Joao S. O. Bueno Calligaris
On Sunday 11 March 2007 20:54, D. Stimits wrote:
 I have not found any python-fu way to close a file, or to reclaim
 memory after creating an image or layer. Is there such a thing? Is
 there instead some sort of garbage collection?

 Speculating about why it claimed it was out of memory, I'm
 wondering if the hard drive simply was too slow responding. It's a
 laptop, running with core duo, entirely in ram. The drive is thus
 much slower than the ability to consume ram. How does gimp
 determine that there is no space left on the device...is it a
 timeout in addition to other means?


The normal python way would be to set the object for destruction after 
it is no longer used.

However, it may well be possible that the GIMP bindings to python are 
not deleting GIMP images creted from within python, as it should be 
done.

I had, myself,  never thought of using a script o create so many 
images. 

As you rerro r mesage is no space left on device - please try to 
cehck gimp preferences for tmp dirs, maybe it is using another 
partition that might be getting filled.

The increaseof RAM usage for itself however shows there is something 
wrong going on - could you please fill a bug report about this at
bugzilla.gnome.org ? 

If possible, please attach a minimalistic version of your script taht 
would trigger the problem.

Thank you,

js
-- 
___
Gimp-developer mailing list
Gimp-developer@lists.XCF.Berkeley.EDU
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer