Re: Tile Cache Size
On Sat, Nov 13, 1999 at 09:42:33PM +0100, Marc Lehmann <[EMAIL PROTECTED]> wrote: > On Sat, Nov 13, 1999 at 03:32:45AM +, Nick Lamb <[EMAIL PROTECTED]> >wrote: Nick, it's a shame that we feel forced to write such mails. This is not not funny for both of us. How about, if we both tried very hard getting along with each other? I know you don't like gimp-perl, but maybe you could then just ignore it. I will try not to needlessly kick script-fu or you all the time myself, while trying to improve gimp-perl to a point where you no longer have to worry about it? -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Tile Cache Size
On Sat, Nov 13, 1999 at 03:32:45AM +, Nick Lamb <[EMAIL PROTECTED]> wrote: > On Fri, Nov 12, 1999 at 05:32:10PM +0100, Marc Lehmann wrote: > > You saying that the tile system in Gimp is faster is not useful. > ^^^ > > I didn't. Please don't leap into every discussion just to bait me Marc, I never did this! > it's very annoying and I somehow doubt that others are reading this > list just to see whether you can bait me enough for me to really bite. I also don't think others do this, but this is just my opinion (rather than fact). However, if you really think that way then this explains the many problems you seem to have. In any case, I don't think I _ever_ started spreading fud and bitching around, like you seem to enjoy to do so much. (And which(1) is robbing my sleep). PS: This list is public. if you can't cope with this consider going away. That will definitely improve my live, and will also leave more time to write patches for both of us. Alternatively, try not to think badly of others for no reason other than yourself. (1) You might find this refreshing, but this is a sad fact. -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Tile Cache Size
On Fri, Nov 12, 1999 at 02:51:04PM +0100, Ewald R. de Wit wrote: > > This has optimal performance when the image data is on disk, and there is > > precisely enough memory spare to store a duplicate image in memory, and > > assuming that your disk is very slow and your memory is very fast, which > > is probably true on a typical PC architecture. > > Right. This would be the typical case that I'm interested in. Great. For a specific operation you can make really great speed-ups like this, and I wish you luck with it (though I really wonder if there isn't already a perfectly good free solution somewhere). Tile behaviour won't win much (anything?) in this case, though some image formats like TIFF provide for tiling on disk... Gimp doesn't often do this, so perhaps Gimp wasn't the place to look for examples in doing this kind of operation. [Both images in memory] > I don't follow this. L1/L2 cache is being trashed if you use non-tiling > images?? Yeah, think about it, or wait for someone who does this stuff for real to step forward. I am tired and not in the mood for drawing ASCII Art diagrams of memory access. No offense. [Too little memory example] > One doesn't want to encounter this case in an interactive application > whether you use tiling or non-tiling images. Yes, but in one (tiling) it would suck, and in the other it either won't work or will suck much worse. So all other things being equal this is a vote in favour of tiling. [Sub image example] > Right, for out-of-mem images this is an example were the tiling > comes in handy. Yes, this example supports tiling too. > > So Ewald, unless your critique of the Gimp tile system is going to > > get a bit more sophisticated I'd prefer to see time spent on fixing > > the thing to not leak tiles onto disk, rather than trying to take > > us back down to the level of Paint Shop Pro 4.0 > > Well I'm writing an image processing library right now and I was > wondering whether it would be better of with or without a tile system. > That's why I was studying the way Gimp is doing things. No need for > getting upset about me wondering about the workings of it. That's understood now, I was more defending tiling as a practise than Gimp's implementation. I thought you were suggesting a return to 0.54s amateurish "malloc(width * height * depth)" approach. For your app this might be the appropriate thing to do of course... > Anyway, I won't be using tiling myself as images that don't fit into > memory are of little concern to me. Like you said earlier, photoshop > doesn't handle insanely huge images at all so it's just not an > immediate need for the vast majority of people. This is a matter of > setting priorities (what to optimize and what not). Um, no, I said Photoshop won't load OUR huge images, these images are not just huge because we have too much disk - they're Tiled JPEG TIFF arranged in pyramid format for tile-at-a-time network access via a thing called the Internet Imaging Protocol. Gimp only loads this stuff because I (re)wrote Gimp's TIFF loader (and recompiled my libtiff) If I dig out a 5Kx5K cartoon, and load it into a similarly specified PotatoShop I can do some comparisons, but I'd guess it will kick Gimp's arse. However, it will do so while using tiles, COW, and all the same tricks as Gimp, just better. Tiling isn't a crazy Gimp hack as you may have thought -- it's The Right Thing To Do in an app of this kind. Now, perhaps we can steal some of your nice fast algorithms for our nasty complicated application :) Nick.
Re: Tile Cache Size
On Fri, Nov 12, 1999 at 05:32:10PM +0100, Marc Lehmann wrote: > You saying that the tile system in Gimp is faster is not useful. ^^^ I didn't. Please don't leap into every discussion just to bait me Marc, it's very annoying and I somehow doubt that others are reading this list just to see whether you can bait me enough for me to really bite. Nick.
Re: Tile Cache Size
Marc Lehmann ([EMAIL PROTECTED]) wrote: > Nick, he claims that his software can do the same operation faster on the > same machine with a linear memory representation. > > So either he lies, he does not know how to measure, his software _is_ twice > as fast since the tile system in gimp is crap, his software does do something > completely different or... A part of the software I'm now working on is over at http://pobox.leidenuniv.nl/~ewit/kde/fastpixmap It's the rendering part of what will be a general image processing library. The test program that comes with it is completely different from the Gimp, but it does contrast & brighness & gamma adjustments 18x (not 2x) faster as the Gimp does it (this is with the Gimp enhancing on the fly, so no undo information is being saved). For me it is then natural to question where this speed differential comes from. Initially I thought it was the tile system but I could be wrong about that. That is all there is to it... -- -- Ewald
Re: Tile Cache Size
On Fri, Nov 12, 1999 at 12:44:14PM +, Nick Lamb <[EMAIL PROTECTED]> wrote: > case which can again cost us 2x -- 10x performance without a thought > on mid-sized or large images. > > So Ewald, unless your critique of the Gimp tile system is going to > get a bit more sophisticated I'd prefer to see time spent on fixing Nick, he claims that his software can do the same operation faster on the same machine with a linear memory representation. So either he lies, he does not know how to measure, his software _is_ twice as fast since the tile system in gimp is crap, his software does do something completely different or... You saying that the tile system in Gimp is faster is not useful. -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Tile Cache Size
Nick Lamb ([EMAIL PROTECTED]) wrote: > On Fri, Nov 12, 1999 at 11:38:57AM +0100, Ewald R. de Wit wrote: > > You can always read in the linear file in row order (and write the > > results in column order to memory in the case of 90 degree rotation). > > This will go at full disk speed. > > This has optimal performance when the image data is on disk, and there is > precisely enough memory spare to store a duplicate image in memory, and > assuming that your disk is very slow and your memory is very fast, which > is probably true on a typical PC architecture. Right. This would be the typical case that I'm interested in. > For almost all the interesting cases this has abysmal performance. > Example: > > Both images fit in memory => Since this isn't a ZX Spectrum we have a > cache thrash problem, if we don't use tiles then this will cost us 2x > -- 10x performance over a mid-sized image. I don't follow this. L1/L2 cache is being trashed if you use non-tiling images?? > Example: > > Neither image fits in memory => We hit OS swap, worst case scenario > flags start going off in the swapper heuristics and we wait until > doomsday while "memory" is paged on and off the disk multiple times > for each row/ column we process. One doesn't want to encounter this case in an interactive application whether you use tiling or non-tiling images. > Example: > > Sub-image => The process you describe touchs whole strips in this > case which can again cost us 2x -- 10x performance without a thought > on mid-sized or large images. Right, for out-of-mem images this is an example were the tiling comes in handy. > So Ewald, unless your critique of the Gimp tile system is going to > get a bit more sophisticated I'd prefer to see time spent on fixing > the thing to not leak tiles onto disk, rather than trying to take > us back down to the level of Paint Shop Pro 4.0 Well I'm writing an image processing library right now and I was wondering whether it would be better of with or without a tile system. That's why I was studying the way Gimp is doing things. No need for getting upset about me wondering about the workings of it. Anyway, I won't be using tiling myself as images that don't fit into memory are of little concern to me. Like you said earlier, photoshop doesn't handle insanely huge images at all so it's just not an immediate need for the vast majority of people. This is a matter of setting priorities (what to optimize and what not). -- -- Ewald
Re: Tile Cache Size
On Fri, Nov 12, 1999 at 11:38:57AM +0100, Ewald R. de Wit wrote: > You can always read in the linear file in row order (and write the > results in column order to memory in the case of 90 degree rotation). > This will go at full disk speed. This has optimal performance when the image data is on disk, and there is precisely enough memory spare to store a duplicate image in memory, and assuming that your disk is very slow and your memory is very fast, which is probably true on a typical PC architecture. For almost all the interesting cases this has abysmal performance. Example: Both images fit in memory => Since this isn't a ZX Spectrum we have a cache thrash problem, if we don't use tiles then this will cost us 2x -- 10x performance over a mid-sized image. Example: Neither image fits in memory => We hit OS swap, worst case scenario flags start going off in the swapper heuristics and we wait until doomsday while "memory" is paged on and off the disk multiple times for each row/ column we process. Example: Sub-image => The process you describe touchs whole strips in this case which can again cost us 2x -- 10x performance without a thought on mid-sized or large images. So Ewald, unless your critique of the Gimp tile system is going to get a bit more sophisticated I'd prefer to see time spent on fixing the thing to not leak tiles onto disk, rather than trying to take us back down to the level of Paint Shop Pro 4.0 Nick.
Re: Tile Cache Size
Eduardo Perez ([EMAIL PROTECTED]) wrote: > Performing 'linear' operations on an image (such as color correction) is > not much slower on the tiled architecture. But 'non-linear' access to > the image (such as 90 degree rotations) is terribly slow when the image > has been swapped out. As far as you read the data on the same order it > was stored (row order, usually), the disk heads will not have to move > too much, because you are reading data from adjacent positions. But if > you try to read the data in another order (column order, on the 90 > degree rotation example), the heads of the disk must go forward and > backwards to find the data. Storing the data using a tiled order solves > this probblem, as you can read a whole tile into memory each time. You can always read in the linear file in row order (and write the results in column order to memory in the case of 90 degree rotation). This will go at full disk speed. The main uses of the tile system, as far as I can see it, is for undo's, and for the case that if you have one huge image that doesn't fit into memory and you're going to change small portions of it (like brushstroking). Btw, for the storing of undo data to disk, the Gimp is doing it at a fraction of full disk speed. -- -- Ewald
Re: Tile Cache Size
"Ewald R. de Wit" wrote: > > Anyway, today I went over the Gimp sources and noticed how complicated > the tile architecture makes things and I couldn't help wondering why > the heck it was put in. All it seems to do is to give you an order of > magnitude slower speed when dealing with large images. And large > images were supposed to be the very reason for a tiling architecture. Performing 'linear' operations on an image (such as color correction) is not much slower on the tiled architecture. But 'non-linear' access to the image (such as 90 degree rotations) is terribly slow when the image has been swapped out. As far as you read the data on the same order it was stored (row order, usually), the disk heads will not have to move too much, because you are reading data from adjacent positions. But if you try to read the data in another order (column order, on the 90 degree rotation example), the heads of the disk must go forward and backwards to find the data. Storing the data using a tiled order solves this probblem, as you can read a whole tile into memory each time. Edu.
Re: Re: Tile Cache Size
On Thu, Nov 11, 1999 at 12:56:58PM +0100, "Ewald R. de Wit" <[EMAIL PROTECTED]> wrote: > Well the algorithm involved is a simple 256 byte lookup table (or 3 of > them for each of the RGB channels). There is not much one can screw up > about it, both performance and precision wise. The only different to the gimp is that it is signal driven, i.e. the work is done in the gtk idle loop and the screen is updated as necessary. Maybe the cycles are burnt here. Yes... anybody who wants to do a profile? ;() > I consider any image that doesn't fit into RAM to be a pathological > case and I don't really care about this sort of oddball performance. Magazine covers rarely if ever fit into memory (unless you give up on undo). > image as a large file into memory. Chances are that the linear layout > of this large file give a much more efficient disk access pattern then > the Gimp's current swap file. In the case of brightness_contrast and similar operations, the layout is of no consequence to performance (the pixels are not connected and can be processed in any order). -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Tile Cache Size
On Thu, Nov 11, 1999 at 01:40:28AM +0100, Uwe Koloska wrote: > Nich Lamb wrote on Die, 09 Nov 1999: > >Why does my 7274 x 9985 RGB image (212743Kb of data by my calculations) > >result in the creation of a gimpswap which is up to 500Mb in size? > > > Where do you think can the undo information reside??? I would guess that when I actually DO something there will an appropriate number of additional tiles allocated in the swapfile. Experience agrees. Give or take tile leaks, which now tigert has mentioned them, explain this problem very well... Why? Where do YOU think the undo information goes? Nick.
Re: Tile Cache Size
Nich Lamb wrote on Die, 09 Nov 1999: >Why does my 7274 x 9985 RGB image (212743Kb of data by my calculations) >result in the creation of a gimpswap which is up to 500Mb in size? > Where do you think can the undo information reside??? Uwe -- mailto:[EMAIL PROTECTED] http://rcswww.urz.tu-dresden.de/~koloska/ ---- right now the web page is in german only but this will change as time goes by ;-)
Re: Re: Tile Cache Size
Marc Lehmann ([EMAIL PROTECTED]) wrote: > I really don't agree with you on the speed issue. data is most often > processed by tile, in which case the program will find an almost ideal > situation, memory and cache-wise. If it's not the tile architecture that's causing the inefficiencies, then what is? I'm just curious as to what the Gimp is doing with all the CPU cycles. I tried some brightness/contrast changes on a largish image and the speed was one update per 6 seconds (everything was done in memory, no disk activity). My own software gives 3 updates per seconds (18x faster) and I believe Photoshop gives a similar speed. So this sort of basic functionality seems to be implemented quite inefficient on the Gimp. For the less basic functionality like the unsharp mask plugin, the story is much worse. -- -- Ewald
Re: Re: Tile Cache Size
On Tue, Nov 09, 1999 at 03:53:11AM +, Nick Lamb wrote: > > Further to my last post (and possibly related to Ewald's complaints too) > > Why does my 7274 x 9985 RGB image (212743Kb of data by my calculations) > result in the creation of a gimpswap which is up to 500Mb in size? At least earlier, before some mem-leak plugging, gimp leaked tiles. So when I was editin a large image (3000x4000 or something, a scanned slide), I started to hear a "tick... tick.. tick.. tick.. "-sound from the harddisk.. Further investigation showed that while Gimp was basically idle and I was doing nothing with the image.. Hm.. It happens still: tigert@tigert ~/.gimp-1.1 >date Wed Nov 10 21:35:29 EET 1999 tigert@tigert ~/.gimp-1.1 >ls -la gimpswap.1108 -rw---1 tigert tigert 104087552 Nov 10 21:35 gimpswap.1108 tigert@tigert ~/.gimp-1.1 >date Wed Nov 10 21:35:36 EET 1999 tigert@tigert ~/.gimp-1.1 >ls -la gimpswap.1108 -rw---1 tigert tigert 104382464 Nov 10 21:35 gimpswap.1108 tigert@tigert ~/.gimp-1.1 >date Wed Nov 10 21:35:42 EET 1999 tigert@tigert ~/.gimp-1.1 >ls -la gimpswap.1108 -rw---1 tigert tigert 104656896 Nov 10 21:35 gimpswap.1108 tigert@tigert ~/.gimp-1.1 >date Wed Nov 10 21:35:46 EET 1999 tigert@tigert ~/.gimp-1.1 >ls -la gimpswap.1108 -rw---1 tigert tigert 104845312 Nov 10 21:35 gimpswap.1108 tigert@tigert ~/.gimp-1.1 >date Wed Nov 10 21:35:49 EET 1999 tigert@tigert ~/.gimp-1.1 >ls -la gimpswap.1108 -rw---1 tigert tigert 105070592 Nov 10 21:35 gimpswap.1108 tigert@tigert ~/.gimp-1.1 > However, the GOOD part is it doesnt eat RAM anymore, so it doesnt crash X when I go to lunch :) But 1MB in 20 seconds is pretty weird to me.. And like I said, gimp is sitting idle. I'll check once more.. Wed Nov 10 21:35:49 EET 1999 tigert@tigert ~/.gimp-1.1 >ls -la gimpswap.1108 -rw---1 tigert tigert 105070592 Nov 10 21:35 gimpswap.1108 tigert@tigert ~/.gimp-1.1 >date Wed Nov 10 21:38:52 EET 1999 tigert@tigert ~/.gimp-1.1 >ls -la gimpswap.1108 -rw---1 tigert tigert 116310016 Nov 10 21:38 gimpswap.1108 tigert@tigert ~/.gimp-1.1 > Ok, while I was writing this, during 3 minutes we got 11MB of more swapfile. Any clue? Tuomas -- .---( t i g e r t @ g i m p . o r g )---. | some stuff at http://tigert.gimp.org/ | `---'
Re: Re: Tile Cache Size
On Tue, Nov 09, 1999 at 11:59:58AM +1000, David Bonnell wrote: > On Tue, 9 Nov 1999, Ewald R. de Wit wrote: > > > Anyway, today I went over the Gimp sources and noticed how complicated > > the tile architecture makes things and I couldn't help wondering why > > the heck it was put in. All it seems to do is to give you an order of > > magnitude slower speed when dealing with large images. And large > > images were supposed to be the very reason for a tiling architecture. > > > I'm afraid I have to agree with you on the performance WRT large images. > I tried editing a couple of large images yesterday (10MB/600dpi) and it > was painfully slow (Dual 300MHz PII, 128MB RAM). I've got a 20MB/1200dpi > one I want to edit and I'm not looking forward to it! Getting more than 128MB ram does help. Actually, the more the better. I cant think of too much. I have 380MB at work and manipulating large scans is _considerably_ faster than with similar cpu but less ram (which is kinda obvious anyway) Also, like someone pointed out, put as much tile-cache as you have free ram for gimp. Sucks to have earthquakes in Taiwan just now kicking ram-prices up.. :( Tuomas -- .---( t i g e r t @ g i m p . o r g )---. | some stuff at http://tigert.gimp.org/ | `---'
Re: Re: Tile Cache Size
On Tue, Nov 09, 1999 at 01:27:29AM +0100, "Ewald R. de Wit" <[EMAIL PROTECTED]> wrote: > 2. The fragmentation of tiles within the swap file. The sound of Gimp >trashing the harddisk suggests that this is a very big issue. For which spatial indexing would be solution. > Anyway, today I went over the Gimp sources and noticed how complicated > the tile architecture makes things and I couldn't help wondering why > the heck it was put in. All it seems to do is to give you an order of > magnitude slower speed when dealing with large images. And large > images were supposed to be the very reason for a tiling architecture. I really don't agree with you on the speed issue. data is most often processed by tile, in which case the program will find an almost ideal situation, memory and cache-wise. -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Re: Tile Cache Size
Nick Lamb ([EMAIL PROTECTED]) wrote: > Loading a large image (*): Wait about 2 mins, loader finishes and now > after a further couple of minutes the image is drawn, however later > performance is slightly faster than in the default case above. > > (*) A large image here is one which genuinely WILL NOT fit in memory, > by any stretch of the imagination. It is a JPEG tiled TIFF (nasty!) of > dimensions 7274x9985 and in full 24-bit colour. Hi Nick, That is one large image you have there. I'm using filmscans and PhotoCD (base 5) images which are of the order of 2500x4000. On a 128 MB machine with a tile-cache-size of 80M everything is going quite slow. Adjusting the tile-cache-size doesn't make things go faster for me. If the image doesn't fit into memory then operating on it will take a long time anyway. IMHO it is silly to optimize for this case at the (very considerable) expense of the much more common image dimensions. I'm doing my own image processing using non tiling images and not only is it much more pleasant to program for, it is also much faster. -- -- Ewald
Re: Re: Tile Cache Size
On Mon, 8 Nov 1999, Andrew Kieschnick wrote: > Hmm. Are you setting the tile cache size to something reasonable? It will > definitely suck with the default 10mb tile cache... > Bumped it up to 60MB and it's better. Something as simple has hiding one of the layers takes about 10 seconds to complete. (Took about 20-30secs before I think). Or moving a rectangular region of one layer (which should affect only the union of the pixels under the src and dest rectangles?) takes about 10 seconds too. (Again, was probably almost 20 seconds before increasing the tile cache size). -Dave
Re: Re: Tile Cache Size
Further to my last post (and possibly related to Ewald's complaints too) Why does my 7274 x 9985 RGB image (212743Kb of data by my calculations) result in the creation of a gimpswap which is up to 500Mb in size? The performance for such images seems adequate to me (can't compare PotatoShop because it refuses to load any of our larger images at all) but this is wasting a lot of active disk space :( Nick.
Re: Re: Tile Cache Size
On Tue, Nov 09, 1999 at 01:27:29AM +0100, Ewald R. de Wit wrote: > Anyway, today I went over the Gimp sources and noticed how complicated > the tile architecture makes things and I couldn't help wondering why > the heck it was put in. All it seems to do is to give you an order of > magnitude slower speed when dealing with large images. And large > images were supposed to be the very reason for a tiling architecture. I have no idea where this came from, Ewald did you actually do any benchmarking, or just a few thought experiments? Computers do not behave in practise as it may seem they should ideally. Here are some practical results from my real Gimp machine, a PII 300MHz with 64Mb of memory and ~64Mb of swap. This is with CVS Gimp. If I configure Gimp to believe that it has as much memory as one might conceivably need, then the results are as follows: Loading a large image (*): Wait 10 mins, get bored, try to kill, but the machine is in a swap death loop, after 5 more minutes, just as I log in as root from a serial console, X experiences resource starvations and so Gimp, Gnome, xterms and everything go into the drink. If I configure Gimp with a large but not improbable tile cache (64Mb): Loading a large image (*): Wait 5 mins, TIFF loader finishes, after a further 10 mins the image has been drawn at 10:1 reduction. Now with the defaults as supplied (12Mb ISTR): Loading a large image (*): Wait about 2 mins, loader finishes and now after a further 2 or 3 mins the image has been drawn. And finally with my preferred settings (20Mb): Loading a large image (*): Wait about 2 mins, loader finishes and now after a further couple of minutes the image is drawn, however later performance is slightly faster than in the default case above. (*) A large image here is one which genuinely WILL NOT fit in memory, by any stretch of the imagination. It is a JPEG tiled TIFF (nasty!) of dimensions 7274x9985 and in full 24-bit colour. Nick.
Re: Re: Tile Cache Size
On Tue, 9 Nov 1999, David Bonnell wrote: > On Tue, 9 Nov 1999, Ewald R. de Wit wrote: > > > Anyway, today I went over the Gimp sources and noticed how complicated > > the tile architecture makes things and I couldn't help wondering why > > the heck it was put in. All it seems to do is to give you an order of > > magnitude slower speed when dealing with large images. And large > > images were supposed to be the very reason for a tiling architecture. > > > I'm afraid I have to agree with you on the performance WRT large images. > I tried editing a couple of large images yesterday (10MB/600dpi) and it > was painfully slow (Dual 300MHz PII, 128MB RAM). I've got a 20MB/1200dpi > one I want to edit and I'm not looking forward to it! Hmm. Are you setting the tile cache size to something reasonable? It will definitely suck with the default 10mb tile cache... later, Andrew Kieschnick
Re: Re: Tile Cache Size
On Tue, 9 Nov 1999, Ewald R. de Wit wrote: > Anyway, today I went over the Gimp sources and noticed how complicated > the tile architecture makes things and I couldn't help wondering why > the heck it was put in. All it seems to do is to give you an order of > magnitude slower speed when dealing with large images. And large > images were supposed to be the very reason for a tiling architecture. > I'm afraid I have to agree with you on the performance WRT large images. I tried editing a couple of large images yesterday (10MB/600dpi) and it was painfully slow (Dual 300MHz PII, 128MB RAM). I've got a 20MB/1200dpi one I want to edit and I'm not looking forward to it! -Dave
Re: Re: Tile Cache Size
Marc Lehmann ([EMAIL PROTECTED]) wrote: > But this is not at all a problem. For example, on my 8GB main (i.e. /usr, > /home) partition that I already use since two years ans that is 95% full > (too full for the file system in question) I have 0.5% fragmentation. Only > two files have fragmented chunks smaller than 8MB (the maximum). I think 2 things need to be clearly seperated here: 1. The fragmentation of the swap file on the harddisk. I agree that this is a bit of a non issue with the ext2 filesystem (even if the swapfile gets fragmented a bit it's no big deal); 2. The fragmentation of tiles within the swap file. The sound of Gimp trashing the harddisk suggests that this is a very big issue. Anyway, today I went over the Gimp sources and noticed how complicated the tile architecture makes things and I couldn't help wondering why the heck it was put in. All it seems to do is to give you an order of magnitude slower speed when dealing with large images. And large images were supposed to be the very reason for a tiling architecture. -- -- Ewald
Re: Re: Tile Cache Size
On Tue, 9 Nov 1999, Guillermo S. Romero / Familia Romero wrote: > So Gimp could use it, instead of using OS things (swap or filesystem). I > guess everybody will agree that a partition handled by one process (with > high performance in mind) is a good solution. > What's wrong with using mmap? You can even map a raw partition if you want. (According to the man page, I've never tried it). -Dave -- http://www.flamingtext.com/
Re: Re: Tile Cache Size
>> About HD: is there a way to do swap on demand to a partition? A daemon, lib >> or something? >is this gimp-related (?) or do you want something like swapd? or swap >priorities? I know what swapd and swap priorities are (I think I do, OS thing and how partitions are used). I am speaking about something that "owns" a partition (or filesystem or warp generator) and accepts request from apps like "hey you! move this are of my RAM to your storage" and "quickly! give me back the data I give you some minutes ago". So Gimp could use it, instead of using OS things (swap or filesystem). I guess everybody will agree that a partition handled by one process (with high performance in mind) is a good solution. Well, now you can point why my suggestion is 100% wrong. ;] >> I think that is what high end DBs do, handle their own data physical >these high end db's also do not recommend this technique ;-> Really? Maybe time to get a book about DB implementation. Damn, so many thing, so few time. GSR
Re: Re: Tile Cache Size
On Tue, Nov 09, 1999 at 12:04:14AM +0100, "Guillermo S. Romero / Familia Romero" <[EMAIL PROTECTED]> wrote: > About HD: is there a way to do swap on demand to a partition? A daemon, lib > or something? is this gimp-related (?) or do you want something like swapd? or swap priorities? > I think that is what high end DBs do, handle their own data physical these high end db's also do not recommend this technique ;-> -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Re: Tile Cache Size
>I rather think _you_ are missing the point (which is disk layout and >minimizing seeks, and _not_ a better memory layout. The tile based scheme >leads itself naturally to spatial indexing, in fatc it's already half the >way to go). About HD: is there a way to do swap on demand to a partition? A daemon, lib or something? I think that is what high end DBs do, handle their own data physical space. Just and idea based in what I have heard. Discard at will. GSR
Re: Re: Tile Cache Size
On Thu, Nov 04, 1999 at 09:30:56AM +, Austin Donnelly <[EMAIL PROTECTED]> wrote: > Look, you're all missing the point. No, since we are not talking about the current situation... > Gimp does it's own tile swapping not because it wants to control the > layout on disk. As some of you have pointed out, this is futile. ... which is probably why this technique is soo common. "Algorithms" by John Sedgewick has a nice chapter on why using virtual memory for objects larger than it is a very bad idea in general (exceptions possible). > The only reason to swap a tile at a time is to do with controlling how > far apart in memory neighbouring pixels are. Which does not mean that the situation on disk cannot be improved? Most databases actively use spatial indexing _on disk_. > Consider a very wide image. (which will not fit into memory) > memory, there seems little point in converting it into a linear buffer > before writing it to disk. This would certainly take more time than > it would to just hand the data to the OS. certainly... but the current situation does a very suboptimal layout on disk, which could be improved (in some distant future version ;) > And gentlemen, this is not rocket science. It's what undergraduates > are normally taught in their basic "OS Functions" lectures. The gimp > is a good example of why application-specific paging control can be a > performance boost. Now can we drop this silly subject, please? If it's so basic, why do you describe it as silly, and why should we NOT talk about implementing it in gimp? I rather think _you_ are missing the point (which is disk layout and minimizing seeks, and _not_ a better memory layout. The tile based scheme leads itself naturally to spatial indexing, in fatc it's already half the way to go). -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Re: Tile Cache Size
On Wed, Nov 03, 1999 at 05:40:03PM -0600, Tim Mooney <[EMAIL PROTECTED]> wrote: > As far as I know, most Unix and Unix-like OSes will generally try give you > the space you're requesting as a contiguous chunk. In the case of files like > a (e.g.) 40 Meg swap-file for the gimp, that may not be possible, even for > a filesystem that is much less than 50% full. All it takes is "average" > fragmentation to ruin the OS' ability to give you a contiguous chunk. Then you have to convince me (you don't have, but I won't believe you) that this is a problem... surely, each ~8MB chunk of a large file is fragmented on an ext2 file system _by definition_, but these are very large chunks. > This means, I think, that all bets *are* off, at least regarding the gimp's > ability to keep tiles "close" on-disk. I think that on an OS that encourages fragmentation you are right. However, I can't think of a common os in use that allows to run gimp and has such a bad filesystem. > The bigger the swap file, the less likely it is that the gimp will be > able to do this. That's why I asked my original question. But this is not at all a problem. For example, on my 8GB main (i.e. /usr, /home) partition that I already use since two years ans that is 95% full (too full for the file system in question) I have 0.5% fragmentation. Only two files have fragmented chunks smaller than 8MB (the maximum). In any case, since you think this is bad (and I think this is good), only hard data would clear this problem. However, as a matter of fact, linux' swapping (the os with the ext2 fs) is _far_ worse than _any_ application-side swapping. -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Re: Tile Cache Size
In regard to: Re: Re: Tile Cache Size, Marc Lehmann said (at 1:05am on Nov...: >On Mon, Nov 01, 1999 at 08:04:39PM -0600, Tim Mooney ><[EMAIL PROTECTED]> wrote: >> Wouldn't the situation be even worse, then, if we're going through the >> filesystem and there's "average" fragmentation? You seem to be assuming that >> the filesystem allocation will be contiguous (or at least close) on disk, >> but can you really make that assumption? > >I don´t care for windows, if you wanted to hear that ;-> If the OS does >not make this assumption a fact (in most cases) then all bets are off >anyway. I wasn't talking about windoze, and I don't care for it either. ;-) All filesystems have the notion of fragmentation. Some of them *encourage* it (UFS/BFFS, for example), and many of them *discourage* it, providing tools to defragment, coalescing the pockets of free space into large chunks of free space. As far as I know, most Unix and Unix-like OSes will generally try give you the space you're requesting as a contiguous chunk. In the case of files like a (e.g.) 40 Meg swap-file for the gimp, that may not be possible, even for a filesystem that is much less than 50% full. All it takes is "average" fragmentation to ruin the OS' ability to give you a contiguous chunk. This means, I think, that all bets *are* off, at least regarding the gimp's ability to keep tiles "close" on-disk. The bigger the swap file, the less likely it is that the gimp will be able to do this. That's why I asked my original question. Tim -- Tim Mooney [EMAIL PROTECTED] Information Technology Services (701) 231-1076 (Voice) Room 242-J1, IACC Building (701) 231-8541 (Fax) North Dakota State University, Fargo, ND 58105-5164
Re: Re: Tile Cache Size
On Mon, Nov 01, 1999 at 08:04:39PM -0600, Tim Mooney <[EMAIL PROTECTED]> wrote: > Wouldn't the situation be even worse, then, if we're going through the > filesystem and there's "average" fragmentation? You seem to be assuming that > the filesystem allocation will be contiguous (or at least close) on disk, > but can you really make that assumption? I don´t care for windows, if you wanted to hear that ;-> If the OS does not make this assumption a fact (in most cases) then all bets are off anyway. -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Re: Tile Cache Size
In regard to: Re: Re: Tile Cache Size, Marc Lehmann said (at 10:35pm on Nov...: >On Mon, Nov 01, 1999 at 10:22:08PM +0100, "Guillermo S. Romero / Familia Romero" ><[EMAIL PROTECTED]> wrote: >> Yes, but Gimp swaps to files, while system normally swaps to partition, and >> if the admin is smart, to a fast disk which main (unique?) task is swapping, >> maybe even sharing swap among a group of disks. Kernels swap is optimized (I >> hope it is, otherwise... argh!), we dunno about Gimp. > >The point is that the kernel keyes the swap by memory address (physical or >virtual does not matter). Which means the keys are basically random. > >Gimp can use optimized ordering (e.g. group tiles that are near eahc other >near on the medium) that no kernel can use. > >Once you start to seek your performance is gone, _no matter_ how fast your >physical swap may be (for linear r/w). Wouldn't the situation be even worse, then, if we're going through the filesystem and there's "average" fragmentation? You seem to be assuming that the filesystem allocation will be contiguous (or at least close) on disk, but can you really make that assumption? Tim -- Tim Mooney [EMAIL PROTECTED] Information Technology Services (701) 231-1076 (Voice) Room 242-J1, IACC Building (701) 231-8541 (Fax) North Dakota State University, Fargo, ND 58105-5164
Re: Tile Cache Size
>This is totally wrong in the case of Linux (ok, not unix, but even more >common). Hehehe, then how will you describe my experiences with other non-unix systems? Do not waste your time trying: pathetic and noisy just to start. >With a better layout, gimp swapping should be able to succeed virtual >memory in all cases (of if partition writes are faster). Thanks for the info. Then the bad performace of Gimp must be due the fact that Gimp and kernel were swapping at the same time (so hd heads move from one point to another constantly). >(Ok, 2.4 will fix most of linux´ swpaping mess and use a better layout on >disk, but at the moment what I say holds). Thanks kernel developers. Question: Could Gimp use a file with few holes? In otherwords, reserve space in advance, say in chunks of some MB (so OS tries to make each big block in an acceptabe layout). And what about partitions, like OS or some CDR apps? GSR
Re: Re: Tile Cache Size
On Mon, Nov 01, 1999 at 10:22:08PM +0100, "Guillermo S. Romero / Familia Romero" <[EMAIL PROTECTED]> wrote: > Yes, but Gimp swaps to files, while system normally swaps to partition, and > if the admin is smart, to a fast disk which main (unique?) task is swapping, > maybe even sharing swap among a group of disks. Kernels swap is optimized (I > hope it is, otherwise... argh!), we dunno about Gimp. The point is that the kernel keyes the swap by memory address (physical or virtual does not matter). Which means the keys are basically random. Gimp can use optimized ordering (e.g. group tiles that are near eahc other near on the medium) that no kernel can use. Once you start to seek your performance is gone, _no matter_ how fast your physical swap may be (for linear r/w). There are obvious exceptions, like nfs, but I think we shouldn´t optimize for the broken case. -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Re: Tile Cache Size
>This is not necissarily true. The System-Swap routine is optimized for >arbitrary data. Gimp organizes its image-data in tiles and may perform >better in swapping those tiles, since they are a very special data-structure. Nor false. >So the swapping routines could be optimized specially for those data >(I have no idea if this is done currently) and perform better than the >systems routine. Yes, but Gimp swaps to files, while system normally swaps to partition, and if the admin is smart, to a fast disk which main (unique?) task is swapping, maybe even sharing swap among a group of disks. Kernels swap is optimized (I hope it is, otherwise... argh!), we dunno about Gimp. This is more a per-site than theoric thing, I know. In the machines I used, the best was to use kernel fuctions instead of Gimp (specially if it fills your home dir up to quota limit). Is there a way to get a valid performance measurement? >The NFS problem should be adressed. Can we detect somehow if the >configured swap-directory is a NFS-Direcrtory and issue a warning? Via mount command. I think all Unix can do that. GSR
Re: [gimp-devel] Re: Tile Cache Size
On Mon, Nov 01, 1999 at 10:02:29PM +0100, Simon Budig <[EMAIL PROTECTED]> wrote: > So the swapping routines could be optimized specially for those data > (I have no idea if this is done currently) and perform better than the > systems routine. Last time I looked, swap space in gimp is managed on a first come first server basis, i.e. no special layouting. It should not perform worse than linux´ swapping, which is much worse in some respects. -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: Tile Cache Size
On Mon, Nov 01, 1999 at 09:41:17PM +0100, "Guillermo S. Romero / Familia Romero" <[EMAIL PROTECTED]> wrote: > Why I say that? Becuase Unix swap (I suppose to a partition, not to a file) > will be better than Gimp swap This is totally wrong in the case of Linux (ok, not unix, but even more common). With a better layout, gimp swapping should be able to succeed virtual memory in all cases (of if partition writes are faster). (Ok, 2.4 will fix most of linux´ swpaping mess and use a better layout on disk, but at the moment what I say holds). -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: [gimp-devel] Re: tile cache size
On Mon, Nov 01, 1999 at 12:05:26PM -0800, Tuomas Kuosmanen <[EMAIL PROTECTED]> wrote: > > But at least tell me what _I_ should use to avoid excess swapping and even It´s easy... try to detect the pysical memory (on common platforms). Then use getrlimit to find out how much virtual memory we are to use. Then take a bit less than the minimum of both values. If none exist, use 10MB. In any case, process limits need to be set administratively on multi-user machines anyway, so they should be a good guideline. > Someone mentioned the tile-cache should be something like half of the ram > available, does that make sense? This lacks a definition of "available". The best value is clearly 100% of the ram that is "available" ;) -- -==- | ==-- _ | ---==---(_)__ __ __ Marc Lehmann +-- --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| -=/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ The choice of a GNU generation | |
Re: [gimp-devel] Re: Tile Cache Size
Guillermo S. Romero / Familia Romero ([EMAIL PROTECTED]) wrote: > Why I say that? Becuase Unix swap (I suppose to a partition, not to a file) > will be better than Gimp swap. So if you need to swap, use first what the OS > provides, and then Gimp system (and if NFS, remember to use local /tmp for > Gimp). This is not necissarily true. The System-Swap routine is optimized for arbitrary data. Gimp organizes its image-data in tiles and may perform better in swapping those tiles, since they are a very special data-structure. So the swapping routines could be optimized specially for those data (I have no idea if this is done currently) and perform better than the systems routine. The NFS problem should be adressed. Can we detect somehow if the configured swap-directory is a NFS-Direcrtory and issue a warning? Bye, Simon -- [EMAIL PROTECTED] http://www.home.unix-ag.org/simon/
Re: Tile Cache Size
>And, Tigert, in your case I'd propose to set it to something like 200MB >assuming you don't do too much other stuff when working on large images. >From my experience, tile cache size should be calculated (in a stand alone workstation, aka system where processes are mainly owned by the guy with the mouse) as RAM plus real swap minus kernel, X, wm, terminals, etc memory. Why I say that? Becuase Unix swap (I suppose to a partition, not to a file) will be better than Gimp swap. So if you need to swap, use first what the OS provides, and then Gimp system (and if NFS, remember to use local /tmp for Gimp). Disclaimer: personal experience based on the speed I experienced and the noise I heard (HD making "music" vs pure mad noise). I have not read the source, measured stats with top or anything technical. GSR
Re: [gimp-devel] Re: tile cache size
On Mon, Nov 01, 1999 at 06:58:09PM +0100, Simon Budig wrote: > Austin Donnelly ([EMAIL PROTECTED]) wrote: > > Idea: if the size is set to 0, make it mean "guess something good". > > Out of the box gimp can come with it set to 0, and we just make the > > algorithm pick something appropriate. That's the hard part. > > Just to start a discussion: What about trying to detect if it is a > "private" machine with less than 5 regular users and then using > 80% of the physical memory? What? All our users have login names starting from nobody000 to nobody999! :) Also, 5x80% = 400%.. a hypothetical situation of all 5 users using Gimp at the same time :) Just joking, this might have a point but it is not trivial. But at least tell me what _I_ should use to avoid excess swapping and even crashing X.. Like I said, I have 256MB. X eats like 90MB (high res, high depth), Netscape bloats too (60MB is just a 'normal' case) etc.. so say, I have something like 100MB for Gimp, sometimes a bit more (Netscape will get swapped out when I work on gimp) Someone mentioned the tile-cache should be something like half of the ram available, does that make sense? Thanks, Tuomas -- .---( t i g e r t @ g i m p . o r g )---. | some stuff at http://tigert.gimp.org/ | `---'
Re: [gimp-devel] Re: tile cache size
Austin Donnelly ([EMAIL PROTECTED]) wrote: > Idea: if the size is set to 0, make it mean "guess something good". > Out of the box gimp can come with it set to 0, and we just make the > algorithm pick something appropriate. That's the hard part. Just to start a discussion: What about trying to detect if it is a "private" machine with less than 5 regular users and then using 80% of the physical memory? Bye, Simon -- [EMAIL PROTECTED] http://www.home.unix-ag.org/simon/
Re: tile cache size
Idea: if the size is set to 0, make it mean "guess something good". Out of the box gimp can come with it set to 0, and we just make the algorithm pick something appropriate. That's the hard part. Austin
Re: Tile Cache Size
Austin Donnelly wrote: > [Lots of people writing barking mad things about tile swapping] > > Look, you're all missing the point. > > Gimp does it's own tile swapping not because it wants to control the > layout on disk. As some of you have pointed out, this is futile. > > The only reason to swap a tile at a time is to do with controlling how > far apart in memory neighbouring pixels are. > > Consider a very wide image. If it is stored as a large linear array > in memory (possibly paged by the OS to an OS-managed swap file), then > the common operation of consulting a pixel above or below the current > one results in needing to skip a large number of bytes through the > linear array. This results in poor CPU cache performance. So, we use > a tiled memory layout. Once the data is in a tiled representation in > memory, there seems little point in converting it into a linear buffer > before writing it to disk. This would certainly take more time than > it would to just hand the data to the OS. > > Now, the size of a tile cache (ie number of tiles we'd like to be able > to access as speedily as possible) should be a little over the number > of tiles it takes to cover the width of the image. This is so that > filters which iterate over every single pixel from left to right, top > to bottom, perform better on the horizontal boundary between adjacent > strips of tiles. Consider a 3x3 convolution (let's say a blur > matrix). When the center of the matrix is at the top of the second > row of tiles, the top of the matrix needs to reference the first row > of tiles. It is helpful for performance to have this top row > available. Which means caching ceil(img_width / tile_width) + 1 > tiles. > > And gentlemen, this is not rocket science. It's what undergraduates > are normally taught in their basic "OS Functions" lectures. The gimp > is a good example of why application-specific paging control can be a > performance boost. > > Now can we drop this silly subject, please? > > Austin okay... now what is the best size to set it to? /me runs *grin* -- Garrett LeSage - Linux.com Art Director [EMAIL PROTECTED] - http://linux.com/
Re: Re: Tile Cache Size
[Lots of people writing barking mad things about tile swapping] Look, you're all missing the point. Gimp does it's own tile swapping not because it wants to control the layout on disk. As some of you have pointed out, this is futile. The only reason to swap a tile at a time is to do with controlling how far apart in memory neighbouring pixels are. Consider a very wide image. If it is stored as a large linear array in memory (possibly paged by the OS to an OS-managed swap file), then the common operation of consulting a pixel above or below the current one results in needing to skip a large number of bytes through the linear array. This results in poor CPU cache performance. So, we use a tiled memory layout. Once the data is in a tiled representation in memory, there seems little point in converting it into a linear buffer before writing it to disk. This would certainly take more time than it would to just hand the data to the OS. Now, the size of a tile cache (ie number of tiles we'd like to be able to access as speedily as possible) should be a little over the number of tiles it takes to cover the width of the image. This is so that filters which iterate over every single pixel from left to right, top to bottom, perform better on the horizontal boundary between adjacent strips of tiles. Consider a 3x3 convolution (let's say a blur matrix). When the center of the matrix is at the top of the second row of tiles, the top of the matrix needs to reference the first row of tiles. It is helpful for performance to have this top row available. Which means caching ceil(img_width / tile_width) + 1 tiles. And gentlemen, this is not rocket science. It's what undergraduates are normally taught in their basic "OS Functions" lectures. The gimp is a good example of why application-specific paging control can be a performance boost. Now can we drop this silly subject, please? Austin
