True but then shouldn't it release memory allocated to thumbnails for a given BatchImportJob once that job is finished? By the looks of it, it doesn't always do that as the memory usage only goes up.
The way the F-Spot import was (re-)designed it creates a series of small BatchImportJob's rather than a single massive one to enable the core BatchImport class to clean up after each job. The idea was that even with a massive database to import, it should never really spike but should rather plateau with processed jobs being removed from the head of queue while new ones are added to the tail. Note that this is not necessarily the most efficient design in terms of processor performance because it results in quite a few threads but the size of each individual jobs could be quite easily adjusted. Bruno On 25/09/12 19:25, Jim Nelson wrote: > Just to jump in here, I'll add that what Bruno's seeing is completely > in line with the design of the BatchImport class. The whole import > process is a kind of bucket brigade, moving files from one stage to > the next. The last stage (creating the thumbnails) is a bottleneck > because encoding and writing JPEGs takes a bit of time. So in import > memory usage will spike due to the number of Pixbufs sitting in memory > waiting to be committed to disk. > > -- Jim > > On Tue, Sep 25, 2012 at 8:29 AM, Bruno Girin <[email protected]> wrote: >> On 24/09/12 18:22, Lorenzo Milesi wrote: >> >> Are you sure you uploaded the file correctly? It claims it's 0 bytes >> >> when I download it. The massif output doesn't have to be huge but it >> >> should be more than 0 bytes :-) >> > ups, something went wrong during the upload. >> > this should work! >> > http://dl.dropbox.com/u/706934/massif.out.13520.gz >> >> Lorenzo, >> >> If you run ms_print on this file, you get a nice graph and some detailed >> memory snapshot. What we get out of this is that Shotwell was using >> 386MB of memory by the time the software finished running, which in >> itself is not a massive concern. What is more concerning is that the >> memory usage seems to only go up and rarely down. >> >> Most of the memory (~94% by the end of the run) seems to be allocated to >> pixbuf structures (i.e. images), which is sort of understandable for a >> photo management application. 80% in gdk_pixbuf_new and the remainder in >> gdk_pixbuf_copy. However, the fact that it happens during an import >> probably means they're all thumbnails as Shotwell creates them at >> that time. >> >> Having said this, I'm not convinced memory usage is the root cause of >> your problem. The output you got would still warrant opening a defect in >> redmine though as it would be good to understand why all those pixbufs >> are not released. >> >> One thing I didn't ask you, which I should have: when it hangs, what is >> the CPU usage? Is Shotwell using 100% CPU or not that much? If it uses >> 100% CPU, it's probably stuck in an infinite loop somewhere. If not, it >> may be a threading issue. >> >> The next thing to do would be to run it through valgrind again but this >> time using the helgrind or DRD tools to confirm what is happening in >> terms of thread management and see if it locks itself out. Using the >> callgrind tool could produce interesting output too. I haven't used any >> of those so can't really help on how to use them or interpret the >> results but if you have data, we can always try to have a look see if it >> tells us something. >> >> Cheers, >> >> Bruno >> _______________________________________________ >> Shotwell mailing list >> [email protected] >> http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell > _______________________________________________ Shotwell mailing list [email protected] http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
