On Tue, Apr 8, 2014 at 6:15 PM, Pedro Côrte-Real <pe...@pedrocr.net> wrote: > Went ahead and built this: > > https://github.com/pedrocr/darktable/tree/diskcache
Just pushed some minor fixes to that branch. It no longer leaks memory and the debugging fprintf's are commented out. The code can still be improved quite easily to not do any memory allocation at all but it should be enough for testing. Also redid the benchmarks with the exact same code with and without the file backed cache enabled. My results all for 1000 images over 3 consecutive runs: Without file backing: thumbnails from raw: 1583.022s cold, 1585.638s hot, 1584.993s hotter thumbnails from jpeg: 43.958s cold, 8.715s hot, 9.224 hotter With file backing: thumbnails from raw: 1603.648s cold, 9.664s hot, 1.082s hotter thumbnails from jpeg: 43.432s cold, 1.445s hot, 1.056s hotter Basically the cold case doesn't make much difference (the writes are too small, chances are the kernel is buffering them). The hot cases are 5-10x faster with JPEG and 150-1500x faster with RAW. The JPEG case would probably show a higher improvement at 10000 images, since at 1000 it shows an improvement in the hot case without the patch. That's probably the kernel keeping those files in cache. Compared to the cold case the improvement is 30-40x faster. As would be expected calculating thumbnails from RAW is particularly costly. While testing this out I noticed that as far as I can tell right now if you ask for the same image at mip 0 and mip 1 you'll be doing two different conversions, thus duplicating a lot of work. Strangely lighttable seems to ask for different mip levels even without changing zoom levels. A second optimization that could be useful would be to generate all the thumbnail MIP levels (0 through 3) when any of them is requested as sort of a prefetch. Then you could either: 1) Just save the extra levels to disk to not occupy memory slots 2) Push them all to memory as well. This will almost always make all the caches have the same contents (unless there are a lot of concurrent requests) so for this we might as well do 3) 3) Have only a single thumbnail cache where each slot holds all the various resolutions. dt_mipmap would abstract that away so the rest of darktable would still ask for the different mip levels A simpler solution could be to just have a single thumbnail mip level (the highest level) and use all the memory for that. Selecting the best balance of size vs quality for that highest level would be hard though. Cheers, Pedro ------------------------------------------------------------------------------ Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test & Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees _______________________________________________ darktable-devel mailing list darktable-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/darktable-devel
