Re: [Evolution-hackers] Memory consumption and virtual machines
Hi Philip, -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of ext Philip Van Hoof [] For you, attached and on a plate: o. The patch for evolution-data-server o. The patch for evolution-exchange Trying to get this upstream is, for me, saying thank you. Looking at the patch technically AND testing it (and if it doesn't perform, giving me numbers that compare it with the original implement- ation) is all I'm asking for. Great work -- thanks! If you did some amazing things there; most of all putting a bit of much needed focus on the performance issues in camel. The tinymail stuff makes it possible to use libcamel on the 770; the mmap-stuff might/will enable us to use insane amounts of headers as well. Best wishes, Dirk. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 18:32 +, Federico Mena Quintero wrote: Novell already has a bunch of LDTP stuff to test the Evo mailer from the user's viewpooint - run those tests on the patched version to see how well they work. [Varadhan, those tests are already part of our QA process, aren't they?] Yes and Yes, they are part of our QA process and with the SoC thing, we are getting quite a good number of automation scripts that are compatible with Evolution 2.6. V. Varadhan Novell, Inc. Software for the Open Enterprise™ http://www.novell.com ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 23:05 +, Philip Van Hoof wrote: If Novell wants me to implement unit tests (or other tests) for this, I will ask for payment. I am afraid that you won't get paid as Camel already has a neat-test-suite and can be used/extended, IMO. ;-) V. Varadhan Novell, Inc. Software for the Open Enterprise™ http://www.novell.com ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 18:32 +, Federico Mena Quintero wrote: 1. Branch evolution-data-server into HEAD (development, with Philip's patch), and the stable branch (without the patch). I have created a branch exclusively for the camel mmap summary work, viz., mmapped-camel-summary-branch which will help Phillip to continue his research further. Phillip: Make sure you don't change the format-of-summary-file too drastically, the \0 is acceptable though. 2. Make the patch *mandatory* in HEAD, so that it gets a good amount of testing. Federico, I would like it to be maintained and well tested in the branch rather then rushing it up in HEAD. IMO, a branch gives more freedom to do such research work than HEAD. Correct me if I am wrong. :-) V. Varadhan Novell, Inc. Software for the Open Enterprise™ http://www.novell.com ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
I'm going to attempt to conclude this mini-thread that got extended to other mailing lists. On Wed, 2006-07-19 at 02:45 -0600, Veerapuram Varadhan wrote: I have created a branch exclusively for the camel mmap summary work, viz., mmapped-camel-summary-branch which will help Phillip to continue his research further. I'm going to let the patch rest for a few days so that you can take a good look at it and decide what needs improvement before it goes into that branch. Phillip: Make sure you don't change the format-of-summary-file too drastically, the \0 is acceptable though. We have the discuss this :). But lets do that in focused mailing lists like evolution-hackers. Federico, I would like it to be maintained and well tested in the branch rather then rushing it up in HEAD. IMO, a branch gives more freedom to do such research work than HEAD. Correct me if I am wrong. :-) I agree here. Bringing it to HEAD might be to early. Nevertheless would I be indeed very pleased if a lot people test this on a lot config- urations. Thanks for considering the patch. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Wed, 2006-07-19 at 09:53 -0500, Federico Mena Quintero wrote: On Wed, 2006-07-19 at 01:10 -0600, Veerapuram Varadhan wrote: On Tue, 2006-07-18 at 23:05 +, Philip Van Hoof wrote: If Novell wants me to implement unit tests (or other tests) for this, I will ask for payment. I am afraid that you won't get paid as Camel already has a neat-test-suite and can be used/extended, IMO. ;-) Aaaah, damn :) ... On the other hand, Philip, next time we meet in person I'll happily buy you dinner :) oh ... what about Boston? :) I'll check with my daytime employer whether it's okay if I can visit the Summit. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Wed, 2006-07-19 at 17:40 +0200, Philip Van Hoof wrote: ... On the other hand, Philip, next time we meet in person I'll happily buy you dinner :) oh ... what about Boston? :) I'll check with my daytime employer whether it's okay if I can visit the Summit. I don't know for sure whether I'll be at the Summit. If not, next GUADEC, maybe? [Where do you live? I'm in Boston this week and the next.] Federico ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Mon, 2006-07-17 at 08:57 +, Philip Van Hoof wrote: The lovely smell of programming environment flame wars! Part one As the developer of an application that has an extremely high focus on reduced memory consumption and as the author of a patch for Camel that reduced Evolutions memory footprint with ~40 MB (maybe more, but that number I'm certain of) . . . Take for example Evolution. Using ONE WEEK of hacking, I managed to reduce its memory footprint with at least 40 MB of ram. I don't know how many times I need to repeat, because, this keeps coming in lot different threads and I see no progress to make the patch complete. Yes, I agree, the patch does reduce *STARTUP-MEMORY-FOOT PRINT* of Evolution as mentioned by Federico in his blog, however, it is *as of now* just-a-hack that the Evolution team cannot take it *as is*. Phillip, as you keep saying the patch needs rework before considering upstream, when are we going to get it? Will the final-patch addresses all the concerns raised by me and Fejj? Guess, you are aware of the GNOME release cycle and API freeze dates. V. Varadhan Novell, Inc. Software for the Open Enterprise™ http://www.novell.com ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 18:29 +0200, Philip Van Hoof wrote: I'm waiting for the decision (yours) of making this optional using a compilation flag or at run-time. Let's do this in the usual manner: 0. Polish the patch in the usual way: make sure it follows the indentation and naming conventions of the surrounding code, etc. 1. Branch evolution-data-server into HEAD (development, with Philip's patch), and the stable branch (without the patch). 2. Make the patch *mandatory* in HEAD, so that it gets a good amount of testing. 3. ??? 4. Profit!!! I'd suggest that (3) become write a good stress-test suite for Camel, independent of Evolution. We need that anyway. Novell already has a bunch of LDTP stuff to test the Evo mailer from the user's viewpooint - run those tests on the patched version to see how well they work. [Varadhan, those tests are already part of our QA process, aren't they?] Federico ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 13:26 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 18:29 +0200, Philip Van Hoof wrote: I agree with 1,2,..3 and 4. I will make sure 1 will be finished soon. Probably this evening with a compile-time option (--enable-mmap) I'm waiting for the decision (yours) of making this optional using a compilation flag or at run-time. Let's do this in the usual manner: 0. Polish the patch in the usual way: make sure it follows the indentation and naming conventions of the surrounding code, etc. 1. Branch evolution-data-server into HEAD (development, with Philip's patch), and the stable branch (without the patch). 2. Make the patch *mandatory* in HEAD, so that it gets a good amount of testing. 3. ??? 4. Profit!!! I'd suggest that (3) become write a good stress-test suite for Camel, independent of Evolution. We need that anyway. Novell already has a bunch of LDTP stuff to test the Evo mailer from the user's viewpooint - run those tests on the patched version to see how well they work. [Varadhan, those tests are already part of our QA process, aren't they?] -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 14:46 -0400, Jeffrey Stedfast wrote: I have to wonder if it's even worth ever merging the mmap hack into Evolution at all. If the plan is to finish Zucchi's disk-summary branch, which also solves the memory problems (afaik) as well as: 1. introducing an API for using cursors to get at message infos 2. better designed on-disk format that uses B-Trees When? I'm also very interested in this for tinymail. the problem with philip's mmap file format is that the strings that will be hit for sorting/viewing/etc are all spread out over a huge number of pages. My own tests indicated that it's as-fast as the old implementation. Some test (like sorting) where even faster. I'm guessing mostly because qsort doesn't make large jumps afaik. I mostly fear NFS shared $HOME folders. I just see this being re-examined later to try and design the format to better optimise it by compacting all the strings into a strtab type thing. That would be an excellent idea. I also don't like how it has to reload the summary anytime new messages arrive. This is exactly the same as the current implementation. The current implementation does exactly the same and isn't changed at all. Look at the patch. It doesn't change anything to reloading the summary. I just don't get the feeling this is really all that well thought out and it scares me. So lets test it then?! I'd just hate to see a rush job come out of this Jeff On Tue, 2006-07-18 at 13:26 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 18:29 +0200, Philip Van Hoof wrote: I'm waiting for the decision (yours) of making this optional using a compilation flag or at run-time. Let's do this in the usual manner: 0. Polish the patch in the usual way: make sure it follows the indentation and naming conventions of the surrounding code, etc. 1. Branch evolution-data-server into HEAD (development, with Philip's patch), and the stable branch (without the patch). 2. Make the patch *mandatory* in HEAD, so that it gets a good amount of testing. 3. ??? 4. Profit!!! I'd suggest that (3) become write a good stress-test suite for Camel, independent of Evolution. We need that anyway. Novell already has a bunch of LDTP stuff to test the Evo mailer from the user's viewpooint - run those tests on the patched version to see how well they work. [Varadhan, those tests are already part of our QA process, aren't they?] Federico ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 14:46 -0400, Jeffrey Stedfast wrote: I have to wonder if it's even worth ever merging the mmap hack into Evolution at all. If the plan is to finish Zucchi's disk-summary branch, which also solves the memory problems (afaik) as well as: 1. introducing an API for using cursors to get at message infos 2. better designed on-disk format that uses B-Trees Let's put that discussion on Pause until someone actually starts resurrecting the disk-summary branch. If Philip's patch turns out to work well for daily use, it may be a good stopgap measure. The disk-summary branch will need way more QA time than the mmap() stuff, and a lot more performance tuning work as well :) the problem with philip's mmap file format is that the strings that will be hit for sorting/viewing/etc are all spread out over a huge number of pages. I just see this being re-examined later to try and design the format to better optimise it by compacting all the strings into a strtab type thing. I've been talking to Philip on IRC, and gave him these requirements for his patch: 1. Don't change the external ABI of Camel, so that Evo needs no changes, *OR* also submit a patch to update Evo for the changed API. 2. Make sure the summary format on disk works with older Evos without making *them* rewrite the summaries. This is for deployments which have machines with old and new versions of GNOME, but NFS homedirs accessible from any machine. 3. Keep the coding style, variable naming convention, indentation, etc. It may be possible to change the summary format by *just* adding a nul-terminator to strings; that may work with older Evos if we are lucky enough that they'll just ignore the nul byte at the end. This needs testing. I'd say that (1) and (2) are hard requirements. (3) is the usual stuff. I just don't get the feeling this is really all that well thought out and it scares me. I'd just hate to see a rush job come out of this Yeah, it needs good testing. Philip says he'll cook a patch so that I can use it with my system's e-d-s RPM for daily use. Then I can test it with my normal mailbox. Federico ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 16:05 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 14:46 -0400, Jeffrey Stedfast wrote: I've been talking to Philip on IRC, and gave him these requirements for his patch: 1. Don't change the external ABI of Camel, so that Evo needs no changes, *OR* also submit a patch to update Evo for the changed API. Achieved 2. Make sure the summary format on disk works with older Evos without making *them* rewrite the summaries. This is for deployments which have machines with old and new versions of GNOME, but NFS homedirs accessible from any machine. Achieved my renaming all the summary filenames 3. Keep the coding style, variable naming convention, indentation, etc. Done For you, attached and on a plate: o. The patch for evolution-data-server o. The patch for evolution-exchange Trying to get this upstream is, for me, saying thank you. Looking at the patch technically AND testing it (and if it doesn't perform, giving me numbers that compare it with the original implement- ation) is all I'm asking for. If Novell wants me to implement unit tests (or other tests) for this, I will ask for payment. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be evolution_data_server__mmap_summary.diff.gz Description: GNU Zip compressed data evolution_exchange__mmap_summary.diff.gz Description: GNU Zip compressed data ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
