Re: [Evolution-hackers] Memory consumption and virtual machines
On Wed, 2006-07-19 at 10:58 +0530, Parthasarathi Susarla wrote: Hi all, I second fejjs thoughts. Note that fejj recently improved the patch by removing some ugly pieces of it. Fejj and me discussed (thinks like) the patch on IRC. Note that the patch also fixes a lot non-mmap related memory problems. Also i have been testing the patch for sometime now. Heres the inference: * The patch works in reducing the memory consumed during the initial startup of evolution. And it does a wonderful job of that. * The patch intends to fix the overall consumption of memory during the usage of evolution, and it does *not* do that. I kept evo running with this patch in for more than 8 hours and using Evo as i would use it regularly. The memory that gets saved by the mmap technique is never reclaimed by another (existing nor new) technique. Data that is made available using mmap, is effectively never again made available using another technique. This basically means that it's technically not possible that somehow the patch stops saving the memory that it saves. What it saves from the initial startup, it will always save. It puzzles me a bit how you blame the increase of memory Evolution shows you after eight hours, on the patch. How does it do that then? Evolution consumes a lot real memory when it receives new messages. What you are seeing after eight hours is the real memory being consumed for this new information. The exact same real memory is also being consumed without the patch, there's no difference here. There's no blame on the patch for that. The patch still (for existing information) saves real memory. However. When you'll restart, that new information will be converted to something that can be mmap()ed. Evolution will, after that restart, not consume as much real memory for the same information (that is now on disk and mmap()ed by Evolution). Me, the current Camel maintainer Varadhan and old Camel warrior Fejj are very aware of this. We have a few solutions in mind and will be working on implementing these ideas. The solutions will not require a restart of Evolution. The current patch requires this. This is not a technical impossibility. It's just something that isn't technically implemented by the patch yet. Mostly because tinymail doesn't need this restart (and the patch was initially engineered for the purpose of tinymail). (Reading of new mails, changing foldersblah! blah!) and more than a couple of hundred new mails later and modifying the summary file as many times we lose the memory gain we got initially. I *dont* have fancy graphs to display this information though - but i surely have the data and the necessary information i need. * mmap is *not* the solution to this problem - it helps to a certain extent. * Generating message list each we get a new message is bad enough - we dont want to reload the summary file each time. There's solutions for this. I still have not tested philips latest patches. I gather he has improvised on the older patches. Would report on the patches after i test them enough. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-25 at 21:03 +0200, Philip Van Hoof wrote: Me, the current Camel maintainer Varadhan and old Camel warrior Fejj are very aware of this. We have a few solutions in mind and will be working on implementing these ideas. The solutions will not require a restart of Evolution. The current patch requires this. This is not a technical impossibility. It's just something that isn't technically implemented by the patch yet. Mostly because tinymail doesn't need this restart (and the patch was initially engineered for the purpose of tinymail). Also note that cause of a few reasons involving stress on the side of the Evolution team (feature freeze and stuff like that), and me relaxing a little bit now, I've put the work on the patch temporarily on hold. The current agreement with the Evolution team and idea is to make a branch for evolution-data-server where the experiment will continue. The camel-lite-builder tool* will, however, build you a camel with the summary patch. You can even use the resulting camel with Evolution if you overwrite the so's with the so's that will be installed if you compile a normale evolution-data-server. You can of course also simply apply the patch on evolution-data-server (which is what camel-lite- builder will do for you) https://svn.tinymail.org/svn/camel-lite-builder/trunk -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 23:05 +, Philip Van Hoof wrote: If Novell wants me to implement unit tests (or other tests) for this, I will ask for payment. I am afraid that you won't get paid as Camel already has a neat-test-suite and can be used/extended, IMO. ;-) V. Varadhan Novell, Inc. Software for the Open Enterprise™ http://www.novell.com ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Wed, 2006-07-19 at 17:40 +0200, Philip Van Hoof wrote: ... On the other hand, Philip, next time we meet in person I'll happily buy you dinner :) oh ... what about Boston? :) I'll check with my daytime employer whether it's okay if I can visit the Summit. I don't know for sure whether I'll be at the Summit. If not, next GUADEC, maybe? [Where do you live? I'm in Boston this week and the next.] Federico ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Wed, 2006-07-19 at 01:10 -0600, Veerapuram Varadhan wrote: On Tue, 2006-07-18 at 23:05 +, Philip Van Hoof wrote: If Novell wants me to implement unit tests (or other tests) for this, I will ask for payment. I am afraid that you won't get paid as Camel already has a neat-test-suite and can be used/extended, IMO. ;-) ... On the other hand, Philip, next time we meet in person I'll happily buy you dinner :) Federico ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Wed, 2006-07-19 at 09:53 -0500, Federico Mena Quintero wrote: On Wed, 2006-07-19 at 01:10 -0600, Veerapuram Varadhan wrote: On Tue, 2006-07-18 at 23:05 +, Philip Van Hoof wrote: If Novell wants me to implement unit tests (or other tests) for this, I will ask for payment. I am afraid that you won't get paid as Camel already has a neat-test-suite and can be used/extended, IMO. ;-) Aaaah, damn :) ... On the other hand, Philip, next time we meet in person I'll happily buy you dinner :) oh ... what about Boston? :) I'll check with my daytime employer whether it's okay if I can visit the Summit. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
I have to wonder if it's even worth ever merging the mmap hack into Evolution at all. If the plan is to finish Zucchi's disk-summary branch, which also solves the memory problems (afaik) as well as: 1. introducing an API for using cursors to get at message infos 2. better designed on-disk format that uses B-Trees the problem with philip's mmap file format is that the strings that will be hit for sorting/viewing/etc are all spread out over a huge number of pages. I just see this being re-examined later to try and design the format to better optimise it by compacting all the strings into a strtab type thing. I also don't like how it has to reload the summary anytime new messages arrive. I just don't get the feeling this is really all that well thought out and it scares me. I'd just hate to see a rush job come out of this Jeff On Tue, 2006-07-18 at 13:26 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 18:29 +0200, Philip Van Hoof wrote: I'm waiting for the decision (yours) of making this optional using a compilation flag or at run-time. Let's do this in the usual manner: 0. Polish the patch in the usual way: make sure it follows the indentation and naming conventions of the surrounding code, etc. 1. Branch evolution-data-server into HEAD (development, with Philip's patch), and the stable branch (without the patch). 2. Make the patch *mandatory* in HEAD, so that it gets a good amount of testing. 3. ??? 4. Profit!!! I'd suggest that (3) become write a good stress-test suite for Camel, independent of Evolution. We need that anyway. Novell already has a bunch of LDTP stuff to test the Evo mailer from the user's viewpooint - run those tests on the patched version to see how well they work. [Varadhan, those tests are already part of our QA process, aren't they?] Federico ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers -- Jeffrey Stedfast Evolution Hacker - Novell, Inc. [EMAIL PROTECTED] - www.novell.com ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
Hi all, I second fejjs thoughts. Also i have been testing the patch for sometime now. Heres the inference: * The patch works in reducing the memory consumed during the initial startup of evolution. And it does a wonderful job of that. * The patch intends to fix the overall consumption of memory during the usage of evolution, and it does *not* do that. I kept evo running with this patch in for more than 8 hours and using Evo as i would use it regularly. (Reading of new mails, changing foldersblah! blah!) and more than a couple of hundred new mails later and modifying the summary file as many times we lose the memory gain we got initially. I *dont* have fancy graphs to display this information though - but i surely have the data and the necessary information i need. * mmap is *not* the solution to this problem - it helps to a certain extent. * Generating message list each we get a new message is bad enough - we dont want to reload the summary file each time. I still have not tested philips latest patches. I gather he has improvised on the older patches. Would report on the patches after i test them enough. Cheers, partha On Tue, 2006-07-18 at 14:46 -0400, Jeffrey Stedfast wrote: I have to wonder if it's even worth ever merging the mmap hack into Evolution at all. If the plan is to finish Zucchi's disk-summary branch, which also solves the memory problems (afaik) as well as: 1. introducing an API for using cursors to get at message infos 2. better designed on-disk format that uses B-Trees the problem with philip's mmap file format is that the strings that will be hit for sorting/viewing/etc are all spread out over a huge number of pages. I just see this being re-examined later to try and design the format to better optimise it by compacting all the strings into a strtab type thing. I also don't like how it has to reload the summary anytime new messages arrive. I just don't get the feeling this is really all that well thought out and it scares me. I'd just hate to see a rush job come out of this Jeff On Tue, 2006-07-18 at 13:26 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 18:29 +0200, Philip Van Hoof wrote: I'm waiting for the decision (yours) of making this optional using a compilation flag or at run-time. Let's do this in the usual manner: 0. Polish the patch in the usual way: make sure it follows the indentation and naming conventions of the surrounding code, etc. 1. Branch evolution-data-server into HEAD (development, with Philip's patch), and the stable branch (without the patch). 2. Make the patch *mandatory* in HEAD, so that it gets a good amount of testing. 3. ??? 4. Profit!!! I'd suggest that (3) become write a good stress-test suite for Camel, independent of Evolution. We need that anyway. Novell already has a bunch of LDTP stuff to test the Evo mailer from the user's viewpooint - run those tests on the patched version to see how well they work. [Varadhan, those tests are already part of our QA process, aren't they?] Federico ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers -- Parthasarathi Susarla [EMAIL PROTECTED] Novell Software Development (I) Ltd. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 13:26 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 18:29 +0200, Philip Van Hoof wrote: I agree with 1,2,..3 and 4. I will make sure 1 will be finished soon. Probably this evening with a compile-time option (--enable-mmap) I'm waiting for the decision (yours) of making this optional using a compilation flag or at run-time. Let's do this in the usual manner: 0. Polish the patch in the usual way: make sure it follows the indentation and naming conventions of the surrounding code, etc. 1. Branch evolution-data-server into HEAD (development, with Philip's patch), and the stable branch (without the patch). 2. Make the patch *mandatory* in HEAD, so that it gets a good amount of testing. 3. ??? 4. Profit!!! I'd suggest that (3) become write a good stress-test suite for Camel, independent of Evolution. We need that anyway. Novell already has a bunch of LDTP stuff to test the Evo mailer from the user's viewpooint - run those tests on the patched version to see how well they work. [Varadhan, those tests are already part of our QA process, aren't they?] -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 14:46 -0400, Jeffrey Stedfast wrote: I have to wonder if it's even worth ever merging the mmap hack into Evolution at all. If the plan is to finish Zucchi's disk-summary branch, which also solves the memory problems (afaik) as well as: 1. introducing an API for using cursors to get at message infos 2. better designed on-disk format that uses B-Trees When? I'm also very interested in this for tinymail. the problem with philip's mmap file format is that the strings that will be hit for sorting/viewing/etc are all spread out over a huge number of pages. My own tests indicated that it's as-fast as the old implementation. Some test (like sorting) where even faster. I'm guessing mostly because qsort doesn't make large jumps afaik. I mostly fear NFS shared $HOME folders. I just see this being re-examined later to try and design the format to better optimise it by compacting all the strings into a strtab type thing. That would be an excellent idea. I also don't like how it has to reload the summary anytime new messages arrive. This is exactly the same as the current implementation. The current implementation does exactly the same and isn't changed at all. Look at the patch. It doesn't change anything to reloading the summary. I just don't get the feeling this is really all that well thought out and it scares me. So lets test it then?! I'd just hate to see a rush job come out of this Jeff On Tue, 2006-07-18 at 13:26 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 18:29 +0200, Philip Van Hoof wrote: I'm waiting for the decision (yours) of making this optional using a compilation flag or at run-time. Let's do this in the usual manner: 0. Polish the patch in the usual way: make sure it follows the indentation and naming conventions of the surrounding code, etc. 1. Branch evolution-data-server into HEAD (development, with Philip's patch), and the stable branch (without the patch). 2. Make the patch *mandatory* in HEAD, so that it gets a good amount of testing. 3. ??? 4. Profit!!! I'd suggest that (3) become write a good stress-test suite for Camel, independent of Evolution. We need that anyway. Novell already has a bunch of LDTP stuff to test the Evo mailer from the user's viewpooint - run those tests on the patched version to see how well they work. [Varadhan, those tests are already part of our QA process, aren't they?] Federico ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 16:05 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 14:46 -0400, Jeffrey Stedfast wrote: I have to wonder if it's even worth ever merging the mmap hack into Evolution at all. If the plan is to finish Zucchi's disk-summary branch, which also solves the memory problems (afaik) as well as: 1. introducing an API for using cursors to get at message infos 2. better designed on-disk format that uses B-Trees Let's put that discussion on Pause until someone actually starts resurrecting the disk-summary branch. If Philip's patch turns out to work well for daily use, it may be a good stopgap measure. The disk-summary branch will need way more QA time than the mmap() stuff, and a lot more performance tuning work as well :) the problem with philip's mmap file format is that the strings that will be hit for sorting/viewing/etc are all spread out over a huge number of pages. I just see this being re-examined later to try and design the format to better optimise it by compacting all the strings into a strtab type thing. I've been talking to Philip on IRC, and gave him these requirements for his patch: 1. Don't change the external ABI of Camel, so that Evo needs no changes, *OR* also submit a patch to update Evo for the changed API. The external API nor ABI has changed by the patches. 2. Make sure the summary format on disk works with older Evos without making *them* rewrite the summaries. This is for deployments which have machines with old and new versions of GNOME, but NFS homedirs accessible from any machine. What about making an internal little tool that converts any summary file it detects to a summary.mmap file, and after that never touches the original summary file. This will allow the format of the mmaped summary file to change, and only an evolution with the mmap patches will only the first time have to perform things. We can also simply change the file-name and let the older evolutions continue using the summary file, and the mmaped evolutions the new file. The older evolutions will not accept any changes to the summary files (for example version changes). They will reload the summary file if it has defects it can't handle, however. 3. Keep the coding style, variable naming convention, indentation, etc. Of course. It may be possible to change the summary format by *just* adding a nul-terminator to strings; that may work with older Evos if we are lucky enough that they'll just ignore the nul byte at the end. This needs testing. I'd say that (1) and (2) are hard requirements. (3) is the usual stuff. I just don't get the feeling this is really all that well thought out and it scares me. I'd just hate to see a rush job come out of this Yeah, it needs good testing. Philip says he'll cook a patch so that I can use it with my system's e-d-s RPM for daily use. Then I can test it with my normal mailbox. It might take me a few days as in fact I was planning to give it a rest for a few days. I've been caring to much about it. I don't know, it might also be finished in a few hours. Oh, no .. it's to late now. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
