Re: [Evolution-hackers] Introduction and Questions
I'm preserving the exchange for context; my responses are sprinkled below. On Sun, 2007-06-10 at 23:47 -0400, Jeffrey Stedfast wrote: On Sun, 2007-06-10 at 15:55 -0700, Ross Boylan wrote: On Fri, 2007-06-08 at 20:22 -0400, Jeffrey Stedfast wrote: Second question: even if it creates a folder, does it need to stick around for the folder creation to finish? I think I remember seeing that camel was single-threaded not true... I should have been clearer here: yes, it needs to wait for the folder object instantiation to finish (the function returns a pointer to the instantiated object, afterall). And your proposal with open on that object is designed to decouple creation of the object from creation of the summary, right? When I said folder creation I was thinking of the whole get headers and make index process. not true that camel is single-threaded - it has a few of its own threads going - most notably is the thread for filtering incoming mail for providers like imap and maildir (they share the same thread). , relying on the client app to do threading. Would there be a way to multi-thread this somewhere (either in camel or in the app)? Obviously doing so would complicate things, because at some point one might need to block (e.g., if I move a message from folder A to B and then switch the UI to look at B). okay, I think you need to familiarize yourself with Camel's API before we discuss anything further :) http://www.go-evolution.org/Mail_Threading begins The Camel API is blocking and synchronous although it then goes on to qualify this: Custom Threads Two tasks inside Camel use threads internally in order to implement a cancellable api ontop of a non-cancellable one. Further Mail-OPS mail-ops.h contains asynchronous versions of varous Camel api's It sounds as if mail-ops is outside of Camel, however. it is external to camel, yes. So it sounds as if Camel could (in principle) respond to a move request by issuing the appropriate IMAP command and then, starting a thread to do the other activities (indexing the target folder and deleting the the message from the source folder) and return. it could, yes, but it'd need a way to report an error unless it waited for the operations to finish before returning. For moving mail, you typically want to know that it succeeded :) all of the current camel APIs are designed such that the caller expects that when the function returns, the operation has either succeeded or failed (where the failure would be set on the CamelException argument). It would then block on operations that attempted to access the target folder until the other operations completed. yes, this is true... well, the way folders are implemented at this time anyway... I think this could be called a syncronous API, though perhaps that's a stretch. it is indeed a synchronous API :) Syncronous, but it fails the you know if you've succeeded when the function returns test. On the other hand, http://www.go-evolution.org/Camel.Operation does not sound like a bocking syncronous API at all, so maybe the statement quoted at the top is just obsolete? Camel code is written under the assumption that it is likely it is being used in a thread, so it has CamelOperation as a means to: 1. report the progress 2. cancel operations (typically i/o) what happens is Evolution has a thread-pool with a CamelOperation object for each of the threads the mailer creates so that it can cancel operations and/or get progress updates. This part of the camel API is not syncronous, right? So, first of all I'm confused about the nature of Camel's API and operation as far as threading and syncronicity. Second, I don't have a sense of whether its features are historical accidents (camel was implemented in a simple way and evo then used it as it was) evolution mailer was originally not multi-threaded if my recollection serves correctly, although around the time of the first 0.0 release threading was added (about 7 years ago). camel and evolution were developed together, so their designs evolved together. Why was camel separated out at all? Side note: when I first was trying to figure out what was going on, I looked in the evo sources and discovered the code wasn't there, but was in the evo-data-server. This was confusing because, at least on debian, the latter was described as providing calendar and address book functions. I think the Debian split mirrors the what I see in your repository. My current guess is that some of the code in evolution-data-server (the directory) is really for evolution, and not for the evo-data-server application. or the result of some deliberate design decisions. Blocking syncronous operations are simpler to implement, yep to use, yep to debug, yep, altho with threads thrown
Re: [Evolution-hackers] Introduction and Questions
On Mon, 2007-06-11 at 09:46 -0700, Ross Boylan wrote: Speaking of complexity, evolution/camel (and maybe Gnome?) do a kind of objects with C. Why not just use C++? Was the C++ toolchain too unreliable at the start, or was there some other reason? I didn't mean to exclude other alternatives. The general question was if you're doing objects anyway, why not use a language that already supports them? -- Ross Boylan wk: (415) 514-8146 185 Berry St #5700 [EMAIL PROTECTED] Dept of Epidemiology and Biostatistics fax: (415) 514-8150 University of California, San Francisco San Francisco, CA 94107-1739 hm: (415) 550-1062 ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Mon, 2007-06-11 at 09:46 -0700, Ross Boylan wrote: I'm preserving the exchange for context; my responses are sprinkled below. On Sun, 2007-06-10 at 23:47 -0400, Jeffrey Stedfast wrote: On Sun, 2007-06-10 at 15:55 -0700, Ross Boylan wrote: On Fri, 2007-06-08 at 20:22 -0400, Jeffrey Stedfast wrote: Second question: even if it creates a folder, does it need to stick around for the folder creation to finish? I think I remember seeing that camel was single-threaded not true... I should have been clearer here: yes, it needs to wait for the folder object instantiation to finish (the function returns a pointer to the instantiated object, afterall). And your proposal with open on that object is designed to decouple creation of the object from creation of the summary, right? correct, the ::open() would load the summary from disk and (if the store is network based), sync state with the server (find out about messages which were deleted since last session, flag changes, new messages, etc) When I said folder creation I was thinking of the whole get headers and make index process. ok not true that camel is single-threaded - it has a few of its own threads going - most notably is the thread for filtering incoming mail for providers like imap and maildir (they share the same thread). , relying on the client app to do threading. Would there be a way to multi-thread this somewhere (either in camel or in the app)? Obviously doing so would complicate things, because at some point one might need to block (e.g., if I move a message from folder A to B and then switch the UI to look at B). okay, I think you need to familiarize yourself with Camel's API before we discuss anything further :) http://www.go-evolution.org/Mail_Threading begins The Camel API is blocking and synchronous although it then goes on to qualify this: Custom Threads Two tasks inside Camel use threads internally in order to implement a cancellable api ontop of a non-cancellable one. Further Mail-OPS mail-ops.h contains asynchronous versions of varous Camel api's It sounds as if mail-ops is outside of Camel, however. it is external to camel, yes. So it sounds as if Camel could (in principle) respond to a move request by issuing the appropriate IMAP command and then, starting a thread to do the other activities (indexing the target folder and deleting the the message from the source folder) and return. it could, yes, but it'd need a way to report an error unless it waited for the operations to finish before returning. For moving mail, you typically want to know that it succeeded :) all of the current camel APIs are designed such that the caller expects that when the function returns, the operation has either succeeded or failed (where the failure would be set on the CamelException argument). It would then block on operations that attempted to access the target folder until the other operations completed. yes, this is true... well, the way folders are implemented at this time anyway... I think this could be called a syncronous API, though perhaps that's a stretch. it is indeed a synchronous API :) Syncronous, but it fails the you know if you've succeeded when the function returns test. most of the camel APIs don't fail that test On the other hand, http://www.go-evolution.org/Camel.Operation does not sound like a bocking syncronous API at all, so maybe the statement quoted at the top is just obsolete? Camel code is written under the assumption that it is likely it is being used in a thread, so it has CamelOperation as a means to: 1. report the progress 2. cancel operations (typically i/o) what happens is Evolution has a thread-pool with a CamelOperation object for each of the threads the mailer creates so that it can cancel operations and/or get progress updates. This part of the camel API is not syncronous, right? in so much as doing camel_operation_cancel() returning doesn't mean that the other thread has stopped, correct. Basically it just means that you've sent a command along a pipe to the thread to cancel at its earliest convenience. So, first of all I'm confused about the nature of Camel's API and operation as far as threading and syncronicity. Second, I don't have a sense of whether its features are historical accidents (camel was implemented in a simple way and evo then used it as it was) evolution mailer was originally not multi-threaded if my recollection serves correctly, although around the time of the first 0.0 release threading was added (about 7 years ago). camel and evolution were developed together, so their designs evolved together. Why was camel separated out at all? good model/view split :) you
Re: [Evolution-hackers] Introduction and Questions
On Mon, 2007-06-11 at 14:39 -0400, Jeffrey Stedfast wrote: So it sounds as if Camel could (in principle) respond to a move request by issuing the appropriate IMAP command and then, starting a thread to do the other activities (indexing the target folder and deleting the the message from the source folder) and return. it could, yes, but it'd need a way to report an error unless it waited for the operations to finish before returning. For moving mail, you typically want to know that it succeeded :) all of the current camel APIs are designed such that the caller expects that when the function returns, the operation has either succeeded or failed (where the failure would be set on the CamelException argument). It would then block on operations that attempted to access the target folder until the other operations completed. yes, this is true... well, the way folders are implemented at this time anyway... I think this could be called a syncronous API, though perhaps that's a stretch. it is indeed a synchronous API :) Syncronous, but it fails the you know if you've succeeded when the function returns test. most of the camel APIs don't fail that test This was in the context of something I was thinking of (So it sounds as if Camel could ). The feature/change I was contemplating sounds as if it would be a bad idea because it violates the usual model for camel calls. Probably either a smaller change (keeping the blocking/syncronous/useful return values in place) or a bigger one (general asyncronicity) would be a better idea--at least in a perfect world of infinite coding time.:) In fact, your idea with folder open preserves the expected features of the camel API. -- Ross Boylan wk: (415) 514-8146 185 Berry St #5700 [EMAIL PROTECTED] Dept of Epidemiology and Biostatistics fax: (415) 514-8150 University of California, San Francisco San Francisco, CA 94107-1739 hm: (415) 550-1062 ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Sun, 2007-06-10 at 11:39 +0300, Philip Van Hoof wrote: On Sat, 2007-06-09 at 08:14 -0400, Jeffrey Stedfast wrote: server says: * 1 EXPUNGE camel-imap-summary does: g_ptr_array_remove_index (messages, seqid - 1); In imap_rescan, for example in case a message got removed by another E-mail client while this E-mail client was not online. A hash table won't solve this... Jeff ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Fri, 2007-06-08 at 20:22 -0400, Jeffrey Stedfast wrote: [Ross wrote] Second question: even if it creates a folder, does it need to stick around for the folder creation to finish? I think I remember seeing that camel was single-threaded not true... , relying on the client app to do threading. Would there be a way to multi-thread this somewhere (either in camel or in the app)? Obviously doing so would complicate things, because at some point one might need to block (e.g., if I move a message from folder A to B and then switch the UI to look at B). okay, I think you need to familiarize yourself with Camel's API before we discuss anything further :) http://www.go-evolution.org/Mail_Threading begins The Camel API is blocking and synchronous although it then goes on to qualify this: Custom Threads Two tasks inside Camel use threads internally in order to implement a cancellable api ontop of a non-cancellable one. Further Mail-OPS mail-ops.h contains asynchronous versions of varous Camel api's It sounds as if mail-ops is outside of Camel, however. So it sounds as if Camel could (in principle) respond to a move request by issuing the appropriate IMAP command and then, starting a thread to do the other activities (indexing the target folder and deleting the the message from the source folder) and return. It would then block on operations that attempted to access the target folder until the other operations completed. I think this could be called a syncronous API, though perhaps that's a stretch. On the other hand, http://www.go-evolution.org/Camel.Operation does not sound like a bocking syncronous API at all, so maybe the statement quoted at the top is just obsolete? So, first of all I'm confused about the nature of Camel's API and operation as far as threading and syncronicity. Second, I don't have a sense of whether its features are historical accidents (camel was implemented in a simple way and evo then used it as it was) or the result of some deliberate design decisions. Blocking syncronous operations are simpler to implement, to use, to debug, and to understand, so they clearly have some advantages. But it seems that the entire application (evolution) does not have that character, so the benefits of that simplicity end up lost anyway. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Fri, 2007-06-08 at 20:45 -0400, Jeffrey Stedfast wrote: On Fri, 2007-06-08 at 18:34 +0300, Philip Van Hoof wrote: The imap4 project is making things look better, though all in all it's still much of the same (blocking and waiting for results, in stead of letting the server do most the work and do pipelining). adding pipelining support would not be difficult to do, actually. I noticed that, indeed. Last time I was reading it, I did notice a few things in its code that would make it difficult. All in all, the ideas behind the 'imap4' way of working (with the IMAP server) are indeed a lot better than the 'imap' code. Maybe it would be a good idea to bring features like condstore and idle to it? If there are some more people interested in this, I would probably help out too. Unlike the stuff that I have put in the 'imap' code, this should be done in a proper non-hackish way then, imo. What I disliked most about Camel's 'imap' code, though, is the fact that the sequences have to correspond to the array indexes of the CamelFolderSummary. It sounds like it would have been more easy if that was a key in a hashtable. For example if a message got expunged that had a sequence value in the beginning of the folder, it right now more or less means that the entire folder has to be re-synchronised. While the majority of the local data can be perfectly corrected in stead too. Luckily very few people often remove things in the beginning of their mail folders (since those E-mails are usually the older emails). I have some ideas on having blocks of mmap data for CamelFolderSummary, that for example contain ~ 1000 items per block, and having reference counting per block. That way I could create a vfolder-like feature that keeps only the blocks alive that contain summary items that matched the query. In stead of having to keep all the memory of each folder in the vfolder alive. I have a prototype of this somewhat working, although done extremely hackish too. So at some point I'm going to rewrite this first. Another benefit is that the blocks will never have to be rewritten: removals are simply marked until 50% of the block is removed (and then the entire block is rewritten), flag changes are stored in a different file and on push-email events or a summary download, the latest headers are kept in normal memory until there are enough of them to start a new block. Rewriting the summary file truly is a hit on performance on some devices. On Flash it also introduces some level wearing. Which is why it's another benefit. -- Philip Van Hoof, software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://www.pvanhoof.be/blog ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Sat, 2007-06-09 at 13:00 +0300, Philip Van Hoof wrote: What I disliked most about Camel's 'imap' code, though, is the fact that the sequences have to correspond to the array indexes of the CamelFolderSummary. It sounds like it would have been more easy if that was a key in a hashtable. ...??? are you serious? For example if a message got expunged that had a sequence value in the beginning of the folder, it right now more or less means that the entire folder has to be re-synchronised. While the majority of the local data can be perfectly corrected in stead too. huh? it doesn't re-fetch anything, it simply removes the item from the array at the index given in the untagged EXPUNGE notification. server says: * 1 EXPUNGE camel-imap-summary does: g_ptr_array_remove_index (messages, seqid - 1); (seqid in this case would be '1', but keep in mind that IMAP seqids start at 1 while in c, array indexes start at 0) How would a hash table simplify this? It'd only serve to complicate things because you'd have to re-key the entire hash table after each EXPUNGE notification. Not to mention it would consume a fair bit more memory... Jeff ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-06-07 at 09:25 -0400, Jeffrey Stedfast wrote: On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote: The best way is to ask for the ENVELOPE and the remaining info using the normal BODY.PEEK method. Have you actually ever tested this theory? or did you just pull this out of your proverbial? I have. The ENVELOPE-way is slightly better because words like From: and To: are not repeated each time. Although with COMPRESS and/or support for deflate in the TLS/SSL library will probably compress that difference away mostly. Note I didn't test this with compression. I saw a difference of 5% to 10% depending on several factors in bandwidth saving. I usually test all this with wireshark, which doesn't help a lot with compressed data. It would be nice, indeed, to get some analysis of how data is being compressed when compression by either COMPRESS or the SSL library is being done. Note that I haven't yet implemented COMPRESS in camel-lite, but I will do this very soon (I have a CamelStreamGzip thingy ready for this). On top of bandwidth, on pure speed a lot IMAP servers will return your result faster if you only use ENVELOPE. That's because they optimise on the fields that are in ENVELOPE (indexes, etc etc). But then again, there's not enough in ENVELOPE to support all of Evolution's features, so you'll still have to use BODY.PEEK which will most likely slowdown everything back again. You should note that Thunderbird does not use ENVELOPE, probably because they found it was LESS efficient to fetch the ENVELOPE + BODY.PEEK[HEADER.FIELDS (REFERENCES CONTENT-TYPE)] on many existing IMAP servers in the wild. Probably, indeed. Having myself tested this in the past, I don't recall it making much difference one way or the other. On bandwidth, it does. Especially when you have a lot of messages in some folders and the server is not compressing things (no COMPRESS nor TLS/SSL compressing things for you). Sadly using ENVELOPE breaks on at least one proprietary IMAP server which occasionally likes to send back responses with NIL as each and every item, at least until a client has issued a BODY.PEEK fetch request. Sounds like a bug in that server implementation. It should be possible to specify per folder which of the headers are to be fetched, to make it really efficient. how do you propose calculating the list of headers each folder should request? which headers would you eliminate conditionally? I'd probably start with the references in case the user doesn't want his headers-view to be threaded, and fetch only the sequence number plus the references header the first time threaded is enabled on that folder and after that (while its enabled) for each summary-item update too. A lot mobile devices's screens are simply not suitable for threaded view anyway: introduces way to much wasted white space. Why fetch this information and waste the user's GPRS bandwidth and therefore often his money too? (for a lot people, GPRS bandwidth costs money per downloaded megabyte) In case the From is not displayed, there's no real need to fetch it until the first time the column is added. Same for the dates (date sent and date received, usually only one of both is used by the user). Etc etc etc. The imap4 implementation seems to have an ENVELOPE parser, so I guess either it does it already or it will use ENVELOPE in future. it fetches: ENVELOPE BODY.PEEK[HEADER.FIELDS (Content-Type References In-Reply-To)] and optionally the mailing-list headers I believe it fetches IN-REPLY-TO (which is included in ENVELOPE) explicitly because some IMAP servers tried to be smart and parse the IN-REPLY-TO header for us and didn't handle embedded phrases which certain broken free-software mailers like to send. For a mobile device, that works over GPRS, you for example are usually not interested (not at all) in the mailinglist specific headers. this is mobile-device specific tho, when I was implementing imap4 and people started trying to use it, it was their #1 complaint (which is why imap4 conditionally requests the mlist headers... tho that flag may be hard-coded to TRUE now, I'm not sure) But some people don't use it. Although people more often complain about missing features and not so much about that their E-mail client is wasting their money on GPRS bandwidth, I still believe it's important to support low bandwidth modes too. Also for Evolution, as Evolution is often used with laptops. I do agree, though, that Evolution has a focus difference compared to what I'm doing with Camel. In my opinion is this why splitting Camel from Evolution might help: we could then more easily introduce features for letting Camel switch from low bandwidth mode, to Evolution mode and back. Right now any change that isn't absolutely 100% super pro and for Evolution and Evolution only, gets ignored by you guys at Novell. Usually the explanation goes like this: Yeah but Evolution
Re: [Evolution-hackers] Introduction and Questions
On Fri, 2007-06-08 at 14:39 +0300, Philip Van Hoof wrote: On Thu, 2007-06-07 at 09:25 -0400, Jeffrey Stedfast wrote: On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote: The best way is to ask for the ENVELOPE and the remaining info using the normal BODY.PEEK method. Have you actually ever tested this theory? or did you just pull this out of your proverbial? I have. The ENVELOPE-way is slightly better because words like From: and To: are not repeated each time. Although with COMPRESS and/or support for deflate in the TLS/SSL library will probably compress that difference away mostly. ENVELOPE alone is no doubt as fast or faster (hopefully faster) than explicit header fetching (I expect most servers probably cache ENVELOPE info, whereas explicit header fetching would have to be done by re-parsing the message) and (probably) less processor intensive server-side. Note I didn't test this with compression. I saw a difference of 5% to 10% depending on several factors in bandwidth saving. I usually test all this with wireshark, which doesn't help a lot with compressed data. It would be nice, indeed, to get some analysis of how data is being compressed when compression by either COMPRESS or the SSL library is being done. Note that I haven't yet implemented COMPRESS in camel-lite, but I will do this very soon (I have a CamelStreamGzip thingy ready for this). On top of bandwidth, on pure speed a lot IMAP servers will return your result faster if you only use ENVELOPE. That's because they optimise on the fields that are in ENVELOPE (indexes, etc etc). nod, which is why imap4 used ENVELOPE. I later had to add explicit fetching of other headers, though, to maintain feature compat with imap. But then again, there's not enough in ENVELOPE to support all of Evolution's features, so you'll still have to use BODY.PEEK which will most likely slowdown everything back again. indeed, and it doesn't appear to be faster trying to use ENVELOPE + explicit header fetching, /probably/ because the server still has to spend time parsing the actual message off disk, so the performance gain of using ENVELOPE for the (typically? cached) summary info is negated - possibly made even slower (server has to hit 2 code-paths instead of just the 1), but bandwidth is still slightly decreased, possibly making up for the increased processing time. You should note that Thunderbird does not use ENVELOPE, probably because they found it was LESS efficient to fetch the ENVELOPE + BODY.PEEK[HEADER.FIELDS (REFERENCES CONTENT-TYPE)] on many existing IMAP servers in the wild. Probably, indeed. Having myself tested this in the past, I don't recall it making much difference one way or the other. On bandwidth, it does. Especially when you have a lot of messages in some folders and the server is not compressing things (no COMPRESS nor TLS/SSL compressing things for you). ok, fair enough... not sure how much you really cut out tho, address lists are likely more bandwidth intensive in the ENVELOPE response than in raw format. To: Jeffrey Stedfast [EMAIL PROTECTED], Miguel de Icaza [EMAIL PROTECTED]\r\n ENVELOPE equiv: ((Jeffrey Stedfast NIL fejj novell.com) (Miguel de Icaza NIL miguel ximian.com)) You can see that the ENVELOPE response is more verbose, so the savings really depends on individual messages. You do save on the command echoing from the server, tho... all in all, yea, probably you save some bandwidth. For most users, ENVELOPE is probably a win as far as bytes-sent-across-the-wire. The biggest win for ENVELOPE, in my opinion, is not bandwidth but server processing (measured in both time and resources). Unfortunately, if you have to explicitly request headers anyway, this is completely negated :( Sadly using ENVELOPE breaks on at least one proprietary IMAP server which occasionally likes to send back responses with NIL as each and every item, at least until a client has issued a BODY.PEEK fetch request. Sounds like a bug in that server implementation. yes, it is. It should be possible to specify per folder which of the headers are to be fetched, to make it really efficient. how do you propose calculating the list of headers each folder should request? which headers would you eliminate conditionally? I'd probably start with the references in case the user doesn't want his headers-view to be threaded, and fetch only the sequence number plus the references header the first time threaded is enabled on that folder and after that (while its enabled) for each summary-item update too. A lot mobile devices's screens are simply not suitable for threaded view anyway: introduces way to much wasted white space. Why fetch this information and waste the user's GPRS bandwidth and therefore often his money too? (for a lot people, GPRS bandwidth costs money per downloaded megabyte) In case the From is not displayed, there's no
Re: [Evolution-hackers] Introduction and Questions
On Fri, 2007-06-08 at 09:48 -0400, Jeffrey Stedfast wrote: I think the laptop problem is solved with the basic headers feature, at least as far as collecting new summary info is concerned. Syncing flags is another story, and where the real slowness/user-frustration lies. I'm sure David Woodhouse would agree here. ps. The condstore work in camel-lite is the most easy part to migrate. Right now any change that isn't absolutely 100% super pro and for Evolution and Evolution only, gets ignored by you guys at Novell. I would argue that not to be the case. I would instead argue that the reason many of your patches haven't been incorporated into Camel is that they are hackish and/or tailored explicitly for /your/ needs, and not for the good of the many. I agree that a lot of the things are hackish, some of the changes are this way because there are some design problems in Camel that make certain things not possible without such hacks. For example the fact that non-blocking read()s are not possible by the default CamelStream things, and that none of the existing code really copes very well with non-blocking reads. The imap4 project is making things look better, though all in all it's still much of the same (blocking and waiting for results, in stead of letting the server do most the work and do pipelining). That's not the one I was thinking of, actually... I was more thinking of the BODY(STRUCTURE) stuff. Camel-lite doesn't use BODYSTRUCTURE, it only uses body.peek[0] for doing a partial retrieval. I'm, however, planning to let the CamelMimePart late fetch not-yet retrieved parts when needed (in stead of always downloading the full, or only the body part of the message). That would conflict with IMAP servers that don't do mime parsing right. Then don't be so negative. I'd hardly say that imap4 has a majority of its design wrong. Try some constructive criticism for a change... You are right, let's try to criticise constructive. My apologises for the negative way of putting that I don't like how imap4 is designed. -- Philip Van Hoof, software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://www.pvanhoof.be/blog ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Fri, 2007-06-08 at 12:23 -0700, Ross Boylan wrote: On Thu, 2007-06-07 at 09:25 -0400, Jeffrey Stedfast wrote: it's not possible to do better w/o dropping features like message threading. In fact, the above minimalizing of header fetching already breaks the quick context-menu vfolder on mailing-list and filter on mailing-list features as well as making vfoldering on mailing-list oodles slower (if it even still works after disabling the headers) What about letting the server do the threading, if it can? 1. not all servers support it, so we need to implement threading client-side anyway 2. not all servers do it /correctly/, so we'd need to have a lookup table of server idents so we knew when to fallback (note that every mail client I know of that tried to be smart and let the server do the threading has lived to regret making this decision) 3. if threading behavior is different from other accounts, it will feel weird to the user. I realize not all can, and probably all will seem not as smart as they could be (or just different from how threading would be done in evolution). But otherwise, there's no way around getting lots of headers, and that is a huge hit with big folders. That's the theory; the practice seems uglier. I tried mulberry, which is supposed to make very good use of IMAP. In many ways it does, but its use of server-side threading doesn't work too well. It seemed to get the response back quickly from the server, but then take minutes processing and parsing this response. Furthermore, every time one scrolls the window, it does it all over again. The other problem with server-side threading is that it threads everything. In principle it might make more sense to retrieve a window of headers and thread them. actually, I think this is what Mulberry is doing - or at least that would make sense if it has to re-fetch thread info when you scroll. The window might not include all items in the thread, but that seems a reasonable trade-off to me in return for decent performance. One could do a more thorough job in the background and update when it finished. but then you have items in the list moving around as more thread info becomes available which would be annoying to users, imho. Jeff ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Fri, 2007-06-08 at 17:11 -0400, Jeffrey Stedfast wrote: On Fri, 2007-06-08 at 12:23 -0700, Ross Boylan wrote: On Thu, 2007-06-07 at 09:25 -0400, Jeffrey Stedfast wrote: it's not possible to do better w/o dropping features like message threading. In fact, the above minimalizing of header fetching already breaks the quick context-menu vfolder on mailing-list and filter on mailing-list features as well as making vfoldering on mailing-list oodles slower (if it even still works after disabling the headers) What about letting the server do the threading, if it can? 1. not all servers support it, so we need to implement threading client-side anyway 2. not all servers do it /correctly/, so we'd need to have a lookup table of server idents so we knew when to fallback (note that every mail client I know of that tried to be smart and let the server do the threading has lived to regret making this decision) The parenthetical remark is pretty interesting. 3. if threading behavior is different from other accounts, it will feel weird to the user. All good points. I should explain I'm thinking of a mode that might kick in only on large mailboxes, or as a user option or something like that. My main concern is that with very large folders most mail cients, including evolution, become sluggish to the point of being unusable. For many of the clients life is much uglier during the initial contact with the folders than later. For example, when I move a message into a folder I haven't opened or operated on in evolution, I'm noticing long delays (like 1 minute) while it does something (fetch headers and create an index? whatever it is uses some, but not all the CPU). In this context an responsive but imperfect solution is better than an unresponsive perfect one. For example, if evo relies on the server's threads then it will obviously inherit their behavior, warts and all. In general I agree with the point, made recently on this list, that it's desirable for clients to shield users from server problems (and most users don't know and don't care that the the real problem is with the server). But if doing so takes forever, some may be willing to give up on that. Whether relaying on server side threading, which currently only operates on the whole mailbox, can solve the performance problems is another issue. I realize not all can, and probably all will seem not as smart as they could be (or just different from how threading would be done in evolution). But otherwise, there's no way around getting lots of headers, and that is a huge hit with big folders. That's the theory; the practice seems uglier. I tried mulberry, which is supposed to make very good use of IMAP. In many ways it does, but its use of server-side threading doesn't work too well. It seemed to get the response back quickly from the server, but then take minutes processing and parsing this response. Furthermore, every time one scrolls the window, it does it all over again. The other problem with server-side threading is that it threads everything. In principle it might make more sense to retrieve a window of headers and thread them. actually, I think this is what Mulberry is doing - or at least that would make sense if it has to re-fetch thread info when you scroll. My impression, based partly on what a (the?) Mulberry developer said, is that Mulberry gets the entire threaded list from the server. The time seems to go into the post-processing, and apparently this gets redone every time you scroll. (The developer thought it was time parsing the response string from the server, though I would think building up a virtual message list window is a more likely culprit). BTW, in case people aren't aware, Mulberry was recently open-sourced. The window might not include all items in the thread, but that seems a reasonable trade-off to me in return for decent performance. One could do a more thorough job in the background and update when it finished. but then you have items in the list moving around as more thread info becomes available which would be annoying to users, imho. Worst case the display of the results could be deferred until something else, like a user hitting a scrollbar, causes a display update. But I don't see how this case differs from other things that might change the display, such as the arrival of new mail. Ross ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Fri, 2007-06-08 at 15:13 -0700, Ross Boylan wrote: On Fri, 2007-06-08 at 17:11 -0400, Jeffrey Stedfast wrote: On Fri, 2007-06-08 at 12:23 -0700, Ross Boylan wrote: On Thu, 2007-06-07 at 09:25 -0400, Jeffrey Stedfast wrote: it's not possible to do better w/o dropping features like message threading. In fact, the above minimalizing of header fetching already breaks the quick context-menu vfolder on mailing-list and filter on mailing-list features as well as making vfoldering on mailing-list oodles slower (if it even still works after disabling the headers) What about letting the server do the threading, if it can? 1. not all servers support it, so we need to implement threading client-side anyway 2. not all servers do it /correctly/, so we'd need to have a lookup table of server idents so we knew when to fallback (note that every mail client I know of that tried to be smart and let the server do the threading has lived to regret making this decision) The parenthetical remark is pretty interesting. (Balsa is one such client, if you are interested) 3. if threading behavior is different from other accounts, it will feel weird to the user. All good points. I should explain I'm thinking of a mode that might kick in only on large mailboxes, or as a user option or something like that. My main concern is that with very large folders most mail cients, including evolution, become sluggish to the point of being unusable. For many of the clients life is much uglier during the initial contact with the folders than later. For example, when I move a message into a folder I haven't opened or operated on in evolution, I'm noticing long delays (like 1 minute) while it does something (fetch headers and create an index? whatever it is uses some, but not all the CPU). yea, it's creating a summary. IMHO, the design needs to change such that a CamelFolder object need not load the summary when instantiated. If that gets implemented, then there's no need to hit the server to sync summary info before the object gets returned to the caller who requested it, thus allowing appends w/o this particular slowness. This is all part of my camel architecture redesign ideas that I had started implementing in libspruce as a testbed. I think Michael Zucchi even started to implement this in the disk-summary branch (it basically boiled down to ::open() and ::close() methods on a CamelFolder object) In this context an responsive but imperfect solution is better than an unresponsive perfect one. I would argue that it's not a perfect solution if it's unresponsive :) For example, if evo relies on the server's threads then it will obviously inherit their behavior, warts and all. In general I agree with the point, made recently on this list, that it's desirable for clients to shield users from server problems (and most users don't know and don't care that the the real problem is with the server). But if doing so takes forever, some may be willing to give up on that. Whether relaying on server side threading, which currently only operates on the whole mailbox, can solve the performance problems is another issue. it doesn't operate on the whole mailbox, it operates on a seqid or uid set afaik. been a while since I've read the specs ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Fri, 2007-06-08 at 18:27 -0400, Jeffrey Stedfast wrote: On Fri, 2007-06-08 at 15:13 -0700, Ross Boylan wrote: On Fri, 2007-06-08 at 17:11 -0400, Jeffrey Stedfast wrote: On Fri, 2007-06-08 at 12:23 -0700, Ross Boylan wrote: . All good points. I should explain I'm thinking of a mode that might kick in only on large mailboxes, or as a user option or something like that. My main concern is that with very large folders most mail cients, including evolution, become sluggish to the point of being unusable. For many of the clients life is much uglier during the initial contact with the folders than later. For example, when I move a message into a folder I haven't opened or operated on in evolution, I'm noticing long delays (like 1 minute) while it does something (fetch headers and create an index? whatever it is uses some, but not all the CPU). yea, it's creating a summary. IMHO, the design needs to change such that a CamelFolder object need not load the summary when instantiated. If that gets implemented, then there's no need to hit the server to sync summary info before the object gets returned to the caller who requested it, thus allowing appends w/o this particular slowness. This is all part of my camel architecture redesign ideas that I had started implementing in libspruce as a testbed. I think Michael Zucchi even started to implement this in the disk-summary branch (it basically boiled down to ::open() and ::close() methods on a CamelFolder object) Why does it need to create a CamelFolder for the destination at all, assuming I keep the focus on the source folder? Actually, I guess I don't know if IMAP copy/delete is happening behind the scenes, or if camel is creating the message from scratch in the target folder. The latter is a lot more work (though probably easier to implement). Second question: even if it creates a folder, does it need to stick around for the folder creation to finish? I think I remember seeing that camel was single-threaded, relying on the client app to do threading. Would there be a way to multi-thread this somewhere (either in camel or in the app)? Obviously doing so would complicate things, because at some point one might need to block (e.g., if I move a message from folder A to B and then switch the UI to look at B). In this context an responsive but imperfect solution is better than an unresponsive perfect one. I would argue that it's not a perfect solution if it's unresponsive :) For sure. For example, if evo relies on the server's threads then it will obviously inherit their behavior, warts and all. In general I agree with the point, made recently on this list, that it's desirable for clients to shield users from server problems (and most users don't know and don't care that the the real problem is with the server). But if doing so takes forever, some may be willing to give up on that. Whether relaying on server side threading, which currently only operates on the whole mailbox, can solve the performance problems is another issue. it doesn't operate on the whole mailbox, it operates on a seqid or uid set afaik. been a while since I've read the specs I was mistaken about threading being only for the whole box. I just looked at http://www.ietf.org/internet-drafts/draft-ietf-imapext-sort-18.txt and see you can add additional search specifications to limit your thread request. Of course, clients might still ask for all the threads. Though I've mostly focused on responsiveness, the index the whole folder strategy also impacts bandwidth (as Philip pointed out) and local disk useage. Ross ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Fri, 2007-06-08 at 16:02 -0700, Ross Boylan wrote: On Fri, 2007-06-08 at 18:27 -0400, Jeffrey Stedfast wrote: On Fri, 2007-06-08 at 15:13 -0700, Ross Boylan wrote: On Fri, 2007-06-08 at 17:11 -0400, Jeffrey Stedfast wrote: On Fri, 2007-06-08 at 12:23 -0700, Ross Boylan wrote: . All good points. I should explain I'm thinking of a mode that might kick in only on large mailboxes, or as a user option or something like that. My main concern is that with very large folders most mail cients, including evolution, become sluggish to the point of being unusable. For many of the clients life is much uglier during the initial contact with the folders than later. For example, when I move a message into a folder I haven't opened or operated on in evolution, I'm noticing long delays (like 1 minute) while it does something (fetch headers and create an index? whatever it is uses some, but not all the CPU). yea, it's creating a summary. IMHO, the design needs to change such that a CamelFolder object need not load the summary when instantiated. If that gets implemented, then there's no need to hit the server to sync summary info before the object gets returned to the caller who requested it, thus allowing appends w/o this particular slowness. This is all part of my camel architecture redesign ideas that I had started implementing in libspruce as a testbed. I think Michael Zucchi even started to implement this in the disk-summary branch (it basically boiled down to ::open() and ::close() methods on a CamelFolder object) Why does it need to create a CamelFolder for the destination at all, assuming I keep the focus on the source folder? because you need both a source and a destination folder to move the message(s) to? kinda hard to move messages to a folder if you don't know which folder to move them to, don't you agree? :) Actually, I guess I don't know if IMAP copy/delete is happening behind the scenes, or if camel is creating the message from scratch in the target folder. The latter is a lot more work (though probably easier to implement). it uses imap COPY and mark-as-deleted Second question: even if it creates a folder, does it need to stick around for the folder creation to finish? I think I remember seeing that camel was single-threaded not true... , relying on the client app to do threading. Would there be a way to multi-thread this somewhere (either in camel or in the app)? Obviously doing so would complicate things, because at some point one might need to block (e.g., if I move a message from folder A to B and then switch the UI to look at B). okay, I think you need to familiarize yourself with Camel's API before we discuss anything further :) In this context an responsive but imperfect solution is better than an unresponsive perfect one. I would argue that it's not a perfect solution if it's unresponsive :) For sure. For example, if evo relies on the server's threads then it will obviously inherit their behavior, warts and all. In general I agree with the point, made recently on this list, that it's desirable for clients to shield users from server problems (and most users don't know and don't care that the the real problem is with the server). But if doing so takes forever, some may be willing to give up on that. Whether relaying on server side threading, which currently only operates on the whole mailbox, can solve the performance problems is another issue. it doesn't operate on the whole mailbox, it operates on a seqid or uid set afaik. been a while since I've read the specs I was mistaken about threading being only for the whole box. I just looked at http://www.ietf.org/internet-drafts/draft-ietf-imapext-sort-18.txt and see you can add additional search specifications to limit your thread request. Of course, clients might still ask for all the threads. when you think about it, it's really the only way to do it. The only way to figure out which UIDs to display in the message-list 'window' in threaded mode depends on being able to calculate which rows to show. To do this, you need the whole thread model since you aren't able to request a partial thread-model based on view row indexes (which would be ideal) Though I've mostly focused on responsiveness, the index the whole folder strategy also impacts bandwidth (as Philip pointed out) and local disk useage. disk usage isn't that much, it's something like 2% of the size of the mailbox. also, unless you want to repeatedly request the summary info from the server each session, then you need to cache it. You also need to cache it for disconnected operation. Mobile-device users might not care to have this, but a desktop client requires it. That said, Camel is pluggable... someone could easily write a mobile-imap provider that works the way tnymail needs it to work. Jeff
Re: [Evolution-hackers] Introduction and Questions
On Fri, 2007-06-08 at 20:22 -0400, Jeffrey Stedfast wrote: Why does it need to create a CamelFolder for the destination at all, assuming I keep the focus on the source folder? because you need both a source and a destination folder to move the message(s) to? kinda hard to move messages to a folder if you don't know which folder to move them to, don't you agree? :) All you need for IMAP is the name of the target folder, not the fuller representation that CamelFolder provides. In particular, there's no need to fetch the headers from the destination folder, index them, etc. As you observe, I'm not too familiar with camel's internals: just a quick look at the docs and a little debugging. So I don't know if such a minimalist approach would play very well in that framework. Ross ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
Le jeudi 07 juin 2007 à 01:53 +0300, Philip Van Hoof a écrit : Without immediately writing to disk work, the download by itself will consume around 120 MB of RAM, and will most likely fail due to network timeouts and other such problems (it'll take a while, since Evolution fetches a ridiculous large amount of headers for each message for building its summary). isn't the imap-features plugin's goal to reduce/customize the amount of downloaded headers ? Or is it that it's still not enough ? -- Gilles Dartiguelongue [EMAIL PROTECTED] EE/CS last year student, ESIEE signature.asc Description: Ceci est une partie de message numériquement signée ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-06-07 at 08:25 +0200, Gilles Dartiguelongue wrote: Le jeudi 07 juin 2007 à 01:53 +0300, Philip Van Hoof a écrit : Without immediately writing to disk work, the download by itself will consume around 120 MB of RAM, and will most likely fail due to network timeouts and other such problems (it'll take a while, since Evolution fetches a ridiculous large amount of headers for each message for building its summary). isn't the imap-features plugin's goal to reduce/customize the amount of downloaded headers ? Or is it that it's still not enough ? It improves the situation by setting your url-string to have the basic_headers option. In the imap code of Camel, it will then ask for less headers (but still way too much). A major improvement it certainly isn't. The best way is to ask for the ENVELOPE and the remaining info using the normal BODY.PEEK method. It should be possible to specify per folder which of the headers are to be fetched, to make it really efficient. The imap4 implementation seems to have an ENVELOPE parser, so I guess either it does it already or it will use ENVELOPE in future. For a mobile device, that works over GPRS, you for example are usually not interested (not at all) in the mailinglist specific headers. It would also be possible to do it in multiple passes. For example get a modest list of headers, and after that get more and more. In any case, none of the current Evolution code implements consuming the CONDSTORE capabilities of some modern IMAP servers (like MBox and Cyrus). CONDSTORE is really going to make an enormous difference in bandwidth consumption with large folders. That's because you only need to fetch the flags of the changed messages to synchronise flags. Note that Tinymail's camel-lite implements all the needed solutions to this bandwidth consumption problem. And that its code is, although a lot of work because the Evolution maintainers didn't seem interested at the time it was happening, definitely portable to upstream. Its implementation include solutions for the headers, it immediately saves the headers to disk and unlike Evolution's Camel it can resume partial summary downloads, it wont peek memory allocation during downloading, it implements Push E-mail using IMAP IDLE and it implements CONDSTORE. Although I'm definitely not satisfied with any of either Camel nor camel-lite's IMAP code. The thing is that I'd much prefer a client-side implementation that does proper pipelining. For example Infotrope, Telomer and Polymer does this on an experimental basis (Infotrope is the library). ps. In my opinion is also the imap4 project getting the majority of its design wrong. Although it looks better than the imap one, it's not the best way to use an IMAP server. If a project is going to write an IMAP client implementation 'today': they better do it right. A lot is changing in IMAP today (Lemonade, MORG, etc). It's important to get it right this time (the vast majority of E-mail clients totally suck at how they use the IMAP server, including Evolution indeed). -- Philip Van Hoof, software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://www.pvanhoof.be/blog ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote: On Thu, 2007-06-07 at 08:25 +0200, Gilles Dartiguelongue wrote: Le jeudi 07 juin 2007 à 01:53 +0300, Philip Van Hoof a écrit : Without immediately writing to disk work, the download by itself will consume around 120 MB of RAM, and will most likely fail due to network timeouts and other such problems (it'll take a while, since Evolution fetches a ridiculous large amount of headers for each message for building its summary). isn't the imap-features plugin's goal to reduce/customize the amount of downloaded headers ? Or is it that it's still not enough ? It improves the situation by setting your url-string to have the basic_headers option. In the imap code of Camel, it will then ask for less headers (but still way too much). May be way too much for a mobile client; but not for a desktop email client. Sure you dont want your desktop mail client to have threading or show mail-size or have such basic things ? A major improvement it certainly isn't. Try comparing the performance of fetching all headers and basic_headers only. Some crude data which I collected a year back is at http://psankar.blogspot.com/2006/05/imap-performance.html The best way is to ask for the ENVELOPE and the remaining info using the normal BODY.PEEK method. It should be possible to specify per folder which of the headers are to be fetched, to make it really efficient. Do you really think any user on earth will really be interested in configuring what headers to fetch on a folder-basis ? The imap4 implementation seems to have an ENVELOPE parser, so I guess either it does it already or it will use ENVELOPE in future. For a mobile device, that works over GPRS, you for example are usually not interested (not at all) in the mailinglist specific headers. None of the mailing list headers will be fetched if you have chosen the basic_headers option. It would also be possible to do it in multiple passes. For example get a modest list of headers, and after that get more and more. In any case, none of the current Evolution code implements consuming the CONDSTORE capabilities of some modern IMAP servers (like MBox and Cyrus). CONDSTORE is really going to make an enormous difference in bandwidth consumption with large folders. That's because you only need to fetch the flags of the changed messages to synchronise flags. Note that Tinymail's camel-lite implements all the needed solutions to this bandwidth consumption problem. And that its code is, although a lot of work because the Evolution maintainers didn't seem interested at the time it was happening, definitely portable to upstream. Its implementation include solutions for the headers, it immediately saves the headers to disk and unlike Evolution's Camel it can resume partial summary downloads, it wont peek memory allocation during downloading, it implements Push E-mail using IMAP IDLE and it implements CONDSTORE. Although I'm definitely not satisfied with any of either Camel nor camel-lite's IMAP code. The thing is that I'd much prefer a client-side implementation that does proper pipelining. For example Infotrope, Telomer and Polymer does this on an experimental basis (Infotrope is the library). ps. In my opinion is also the imap4 project getting the majority of its design wrong. Although it looks better than the imap one, it's not the best way to use an IMAP server. If a project is going to write an IMAP client implementation 'today': they better do it right. A lot is changing in IMAP today (Lemonade, MORG, etc). It's important to get it right this time (the vast majority of E-mail clients totally suck at how they use the IMAP server, including Evolution indeed). Such a sweeping generalization !? As you mentioned, IMAP (like any other technology) grows at a very high-speed. What you consider as the right client implementation today may become obsolete tomorrow. When the initial IMAP provider was written there may not be a standard spec. for the PUSH/IDLE etc. No application can have an intelligent design that will remain forever satisfying all new needs/changes. Application designs should just evolve, adapting to the new changes. IIRC, you had a branch on Evolution's Camel to work on whatever changes you wanted to make, so that you don't have to wait for the delay due to review/maintainer. Would love to see some of the improvements that you have done ported there. So that it becomes easy for the maintainer to approve them and bring onto trunk. -- Sankar Novell, Inc. Software for the Open Enterprise* http://www.novell.com ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-06-07 at 03:05 -0600, Sankar P wrote: On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote: It improves the situation by setting your url-string to have the basic_headers option. In the imap code of Camel, it will then ask for less headers (but still way too much). May be way too much for a mobile client; but not for a desktop email client. Sure you dont want your desktop mail client to have threading or show mail-size or have such basic things ? When I'm using Evolution on a laptop, I feel very mobile too. btw. This is a quote that I have from Federico, I think (remember) he said that to me last GUADEC. Although Very often I'm indeed using my laptop at an airport or something, using GPRS. Why isn't Evolution suitable for this use-case? I agree, though, that Tinymail's camel-lite memory improvements are less of a priority for Evolution. It's bandwidth work, though, is important in my opinion. Same for the Push E-mail capabilities. A major improvement it certainly isn't. Try comparing the performance of fetching all headers and basic_headers only. Some crude data which I collected a year back is at http://psankar.blogspot.com/2006/05/imap-performance.html Of course, you can look at the code to see what differs between the basic_headers mode and the normal one. And that will obviously make a big difference (the selection of the query is simply a lot smaller). But if Evolution is causing 900% of its bandwidth to be unnecessary, and the basic_headers mode saves 400% of that, there is still 500% of it that is too much. Which is the situation with basic_headers. The basic_headers option also doesn't implement condstore, which is quite important to safe bandwidth (if supported by the IMAP server). The best way is to ask for the ENVELOPE and the remaining info using the normal BODY.PEEK method. It should be possible to specify per folder which of the headers are to be fetched, to make it really efficient. Do you really think any user on earth will really be interested in configuring what headers to fetch on a folder-basis ? The application should figure that list out automatically, obviously. ps. In my opinion is also the imap4 project getting the majority of its design wrong. Although it looks better than the imap one, it's not the best way to use an IMAP server. If a project is going to write an IMAP client implementation 'today': they better do it right. A lot is changing in IMAP today (Lemonade, MORG, etc). It's important to get it right this time (the vast majority of E-mail clients totally suck at how they use the IMAP server, including Evolution indeed). Such a sweeping generalization !? Maybe, but I've seen quite a lot of their code, and they are indeed very badly programmed. Most of the custom Camel providers are in a very bad shape too, by the way. And ... camel-lite, camel upstream and its standard providers are also in a bad shape :( As you mentioned, IMAP (like any other technology) grows at a very high-speed. What you consider as the right client implementation today may become obsolete tomorrow. When the initial IMAP provider was written there may not be a standard spec. for the PUSH/IDLE etc. This is true. I agree. No application can have an intelligent design that will remain forever satisfying all new needs/changes. Application designs should just evolve, adapting to the new changes. I also agree with this. IIRC, you had a branch on Evolution's Camel to work on whatever changes you wanted to make, so that you don't have to wait for the delay due to review/maintainer. Would love to see some of the improvements that you have done ported there. So that it becomes easy for the maintainer to approve them and bring onto trunk. I think the best way forward is what Matthew and I share an opinion on: To split Camel away from Evolution, and start getting as much of camel-lite back into that upstream version by making the new function- ality more and more pluggable. It would probably take a few releases to get things in upstream, though. To do this adventure on my own, I have too much other priorities right now. If I see somebody being interested, I will definitely help this person and join his efforts as much as possible. -- Philip Van Hoof, software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://www.pvanhoof.be/blog ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-06-07 at 12:28 +0300, Philip Van Hoof wrote: On Thu, 2007-06-07 at 03:05 -0600, Sankar P wrote: On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote: It improves the situation by setting your url-string to have the basic_headers option. In the imap code of Camel, it will then ask for less headers (but still way too much). May be way too much for a mobile client; but not for a desktop email client. Sure you dont want your desktop mail client to have threading or show mail-size or have such basic things ? When I'm using Evolution on a laptop, I feel very mobile too. btw. This is a quote that I have from Federico, I think (remember) he said that to me last GUADEC. Although Very often I'm indeed using my laptop at an airport or something, using GPRS. Why isn't Evolution suitable for this use-case? I spend more than 99% of my internet time in a non-GPRS connection. Just because, I might want to check mail from an airport a few times in a year, I don't want my mail client to be a minimalistic bare-bones application all the time. I would rather use a web-based client at these times. And if I want to use Evolution even in such scenario, then there is an option to get basic_headers alone as well. (Such option was not there in last GUADEC though) I agree, though, that Tinymail's camel-lite memory improvements are less of a priority for Evolution. It's bandwidth work, though, is important in my opinion. Same for the Push E-mail capabilities. A major improvement it certainly isn't. Try comparing the performance of fetching all headers and basic_headers only. Some crude data which I collected a year back is at http://psankar.blogspot.com/2006/05/imap-performance.html Of course, you can look at the code to see what differs between the basic_headers mode and the normal one. And that will obviously make a big difference (the selection of the query is simply a lot smaller). But if Evolution is causing 900% of its bandwidth to be unnecessary, and the basic_headers mode saves 400% of that, there is still 500% of it that is too much. Which is the situation with basic_headers. The basic_headers option also doesn't implement condstore, which is quite important to safe bandwidth (if supported by the IMAP server). The band-width used by Evolution cannot be considered unnecessary, by comparing with a specific style that is designed for mobiles and intended to reduce bandwidth. It will be good to have CONDSTORE and ENVELOPE (and whatever-new) but it is not something that makes a user to say, Damn. I need this and cant use Evo unless I have it. But there are many other such things that needs attention and resources. There are more than 5500 bugs in Evolution and EDS in bgo alone and among these there is just only one bug that complains about bandwidth. This bug is filed by you and will eventually benefit the evolution project once you complete the patch that you are attempting there and being reviewed by Fejj. If Evolution (and hence Gnome as well) has to be successful in an enterprise level it needs more things like: Better Exchange support, Nicer and Richer UI, and more importantly Stability. So the Evolution project's roadmap should be driven based on these, instead of a feature that is not yet implemented by all the servers. I am not denying that it is good to have all these bandwidth savings but Feeping Creaturism(1) should be kept in check. However, if you feel this is the biggest blocker for a mail-application, patches are always welcomed :) The best way is to ask for the ENVELOPE and the remaining info using the normal BODY.PEEK method. It should be possible to specify per folder which of the headers are to be fetched, to make it really efficient. Do you really think any user on earth will really be interested in configuring what headers to fetch on a folder-basis ? The application should figure that list out automatically, obviously. I still don't understand why you need to get different headers for different folders. ps. In my opinion is also the imap4 project getting the majority of its design wrong. Although it looks better than the imap one, it's not the best way to use an IMAP server. If a project is going to write an IMAP client implementation 'today': they better do it right. A lot is changing in IMAP today (Lemonade, MORG, etc). It's important to get it right this time (the vast majority of E-mail clients totally suck at how they use the IMAP server, including Evolution indeed). Such a sweeping generalization !? Maybe, but I've seen quite a lot of their code, and they are indeed very badly programmed. Most of the custom Camel providers are in a very bad shape too, by the way. And ... camel-lite, camel upstream and its standard providers are also in a bad shape :( As you mentioned, IMAP (like any other technology) grows at a very high-speed. What you
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-06-07 at 04:56 -0600, Sankar P wrote: On Thu, 2007-06-07 at 12:28 +0300, Philip Van Hoof wrote: On Thu, 2007-06-07 at 03:05 -0600, Sankar P wrote: On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote: It improves the situation by setting your url-string to have the basic_headers option. In the imap code of Camel, it will then ask for less headers (but still way too much). May be way too much for a mobile client; but not for a desktop email client. Sure you dont want your desktop mail client to have threading or show mail-size or have such basic things ? When I'm using Evolution on a laptop, I feel very mobile too. btw. This is a quote that I have from Federico, I think (remember) he said that to me last GUADEC. Although Very often I'm indeed using my laptop at an airport or something, using GPRS. Why isn't Evolution suitable for this use-case? I spend more than 99% of my internet time in a non-GPRS connection. Just because, I might want to check mail from an airport a few times in a year, I don't want my mail client to be a minimalistic bare-bones application all the time. I would rather use a web-based client at these times. And if I want to use Evolution even in such scenario, then there is an option to get basic_headers alone as well. (Such option was not there in last GUADEC though) On Evoluion 2.10, i see a option for basic headers . Edit - preferences - mail account - select your imap account - edit - IMAP headers - Basic headers I agree, though, that Tinymail's camel-lite memory improvements are less of a priority for Evolution. It's bandwidth work, though, is important in my opinion. Same for the Push E-mail capabilities. A major improvement it certainly isn't. Try comparing the performance of fetching all headers and basic_headers only. Some crude data which I collected a year back is at http://psankar.blogspot.com/2006/05/imap-performance.html Of course, you can look at the code to see what differs between the basic_headers mode and the normal one. And that will obviously make a big difference (the selection of the query is simply a lot smaller). But if Evolution is causing 900% of its bandwidth to be unnecessary, and the basic_headers mode saves 400% of that, there is still 500% of it that is too much. Which is the situation with basic_headers. The basic_headers option also doesn't implement condstore, which is quite important to safe bandwidth (if supported by the IMAP server). The band-width used by Evolution cannot be considered unnecessary, by comparing with a specific style that is designed for mobiles and intended to reduce bandwidth. It will be good to have CONDSTORE and ENVELOPE (and whatever-new) but it is not something that makes a user to say, Damn. I need this and cant use Evo unless I have it. But there are many other such things that needs attention and resources. There are more than 5500 bugs in Evolution and EDS in bgo alone and among these there is just only one bug that complains about bandwidth. This bug is filed by you and will eventually benefit the evolution project once you complete the patch that you are attempting there and being reviewed by Fejj. If Evolution (and hence Gnome as well) has to be successful in an enterprise level it needs more things like: Better Exchange support, Nicer and Richer UI, and more importantly Stability. So the Evolution project's roadmap should be driven based on these, instead of a feature that is not yet implemented by all the servers. I am not denying that it is good to have all these bandwidth savings but Feeping Creaturism(1) should be kept in check. However, if you feel this is the biggest blocker for a mail-application, patches are always welcomed :) The best way is to ask for the ENVELOPE and the remaining info using the normal BODY.PEEK method. It should be possible to specify per folder which of the headers are to be fetched, to make it really efficient. Do you really think any user on earth will really be interested in configuring what headers to fetch on a folder-basis ? The application should figure that list out automatically, obviously. I still don't understand why you need to get different headers for different folders. ps. In my opinion is also the imap4 project getting the majority of its design wrong. Although it looks better than the imap one, it's not the best way to use an IMAP server. If a project is going to write an IMAP client implementation 'today': they better do it right. A lot is changing in IMAP today (Lemonade, MORG, etc). It's important to get it right this time (the vast majority of E-mail clients totally suck at how they use the IMAP server, including Evolution indeed). Such a sweeping generalization !? Maybe, but I've seen quite a lot of their
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote: On Thu, 2007-06-07 at 08:25 +0200, Gilles Dartiguelongue wrote: Le jeudi 07 juin 2007 à 01:53 +0300, Philip Van Hoof a écrit : Without immediately writing to disk work, the download by itself will consume around 120 MB of RAM, and will most likely fail due to network timeouts and other such problems (it'll take a while, since Evolution fetches a ridiculous large amount of headers for each message for building its summary). isn't the imap-features plugin's goal to reduce/customize the amount of downloaded headers ? Or is it that it's still not enough ? It improves the situation by setting your url-string to have the basic_headers option. In the imap code of Camel, it will then ask for less headers (but still way too much). if you set the option to basic headers, it will only fetch the minimal set of headers required to show the message-list: DATE FROM TO CC SUBJECT REFERENCES IN-REPLY-TO MESSAGE-ID MIME-VERSION CONTENT-TYPE (actually, why is MIME-VERSION in there? it's useless) A major improvement it certainly isn't. it's not possible to do better w/o dropping features like message threading. In fact, the above minimalizing of header fetching already breaks the quick context-menu vfolder on mailing-list and filter on mailing-list features as well as making vfoldering on mailing-list oodles slower (if it even still works after disabling the headers) The best way is to ask for the ENVELOPE and the remaining info using the normal BODY.PEEK method. Have you actually ever tested this theory? or did you just pull this out of your proverbial? You should note that Thunderbird does not use ENVELOPE, probably because they found it was LESS efficient to fetch the ENVELOPE + BODY.PEEK[HEADER.FIELDS (REFERENCES CONTENT-TYPE)] on many existing IMAP servers in the wild. Having myself tested this in the past, I don't recall it making much difference one way or the other. Sadly using ENVELOPE breaks on at least one proprietary IMAP server which occasionally likes to send back responses with NIL as each and every item, at least until a client has issued a BODY.PEEK fetch request. It should be possible to specify per folder which of the headers are to be fetched, to make it really efficient. how do you propose calculating the list of headers each folder should request? which headers would you eliminate conditionally? The imap4 implementation seems to have an ENVELOPE parser, so I guess either it does it already or it will use ENVELOPE in future. it fetches: ENVELOPE BODY.PEEK[HEADER.FIELDS (Content-Type References In-Reply-To)] and optionally the mailing-list headers I believe it fetches IN-REPLY-TO (which is included in ENVELOPE) explicitly because some IMAP servers tried to be smart and parse the IN-REPLY-TO header for us and didn't handle embedded phrases which certain broken free-software mailers like to send. For a mobile device, that works over GPRS, you for example are usually not interested (not at all) in the mailinglist specific headers. this is mobile-device specific tho, when I was implementing imap4 and people started trying to use it, it was their #1 complaint (which is why imap4 conditionally requests the mlist headers... tho that flag may be hard-coded to TRUE now, I'm not sure) It would also be possible to do it in multiple passes. For example get a modest list of headers, and after that get more and more. but then you introduce a lot more complexity (e.g. scheduling such a fetch) slowness for the user (e.g. if he expects to be able to right-click and select vfolder on mailing-list, he'd have to wait if the info had not yet been fetched... or even just selecting a mailing-list vfolder) keep in mind also that this is a one-time fetch, so while opening a folder for the first time might be slow (if you have 1000's of messages), future folder selections would take a much more reasonable amount of time to open (especially with CONDSTORE support which reduces the FLAGS sync requirements, which is where the real slowness is when opening a folder for most users) In any case, none of the current Evolution code implements consuming the CONDSTORE capabilities of some modern IMAP servers (like MBox and Cyrus). CONDSTORE is really going to make an enormous difference in bandwidth consumption with large folders. That's because you only need to fetch the flags of the changed messages to synchronise flags. yes, this would be nice to have... but as far as I'm aware, it is still only a draft standard (e.g. it could change). I would probably not be opposed to a CONDSTORE implementation making it into upstream camel, tho. Note that Tinymail's camel-lite implements all the needed solutions to this bandwidth consumption problem. And that its code is, although a lot of work because the Evolution maintainers didn't seem interested at the time it was happening, definitely
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 14:10 -0400, Jeffrey Stedfast wrote: On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: Hi. I've been getting into the code of evolution recently, and am thinking of doing a bit more to see if I can get it working OK for my situation. I have an IMAP mailbox which is very large, both in terms of folders (over 100) and messages (the largest folder has 300,000 messages; my INBOX has about 22,000). the largest INBOX I've ever used was about ~100,000 messages, so you may get to have some fun :) Without CONDSTORE support, Evolution needs ridiculous amounts of both time and bandwidth to synchronise such a folder's flags. When online, this happens each time you click on a folder in Evolution. Without immediately writing to disk work, the download by itself will consume around 120 MB of RAM, and will most likely fail due to network timeouts and other such problems (it'll take a while, since Evolution fetches a ridiculous large amount of headers for each message for building its summary). -- Philip Van Hoof, software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://www.pvanhoof.be/blog ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thursday 31 May 2007 16:09, Jeffrey Stedfast wrote: On Thu, 2007-05-31 at 17:18 -0400, Jeffrey Stedfast wrote: [snip] Single namespace. It's all INBOX.folder.subfolder. The one wrinkle is that in some cases 'folder' exists in the namespace, but is not an actual box or folder (whatever the right term is) on the server: INBOX.folder.subfolder is a real folder; INBOX.folder is not. I guess just a bug I don't have a build environment anymore so I can't easily test this patch, but perhaps i will fix your missing folder troubles. I'm not sure that the names that aren't folders problem is causing any of the problems I'm seeing. While such names don't appear in the folder list on the left pane, I also have plenty of ordinary folders that aren't appearing. And the ghost folders, along with their children, do show up in the Folder| subscription dialogue. evolution seems to insist on showing only subscribed folders, even when I disable the show only subscribed folders option. Since the ghost folders aren't folders, there is no way I can subscribe to them. So if I wanted to see only subscribed folders, that might create a problem. I was able to start up evo under the debugger on the system that's having problems, and this time it worked. Yay! I put future breakpoints on camel_store_supports_subscriptions, and (following the location of the patches you provided) imap4_build_folder_info. I also put one on imap_build_folder_info (a wild guess). Unfortunately none of these were hit when evo started up and was deciding what to show. I did hit the camel_store_support_subscriptions when I opened the Folders | subscriptions dialogue. I tried creating an IMAP type account. This did hit imap_build_folder_info, and it did show all my folders (including ghosts). Performance seems pretty good. I'm going to stop for now, while I'm ahead. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
Hi, On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: Hi. I've been getting into the code of evolution recently, and am thinking of doing a bit more to see if I can get it working OK for my situation. I have an IMAP mailbox which is very large, both in terms of folders (over 100) and messages (the largest folder has 300,000 messages; my INBOX has about 22,000). None of the email clients I've tried cope with this very well. Since I've been using evo at work in a similar setup (cyrus server, though not quite as big), I thought it might be the best to try to tweak. The problems I've had so far involve setup activities rather than core functionaility. First, evolution couldn't create the account (solved); second I've had problems getting it to show all of my subfolders. My narrow question is how to debug evolution. When I launched evolution in gdb evolution (the GUI) came up, and then the debugger told me the process had exited (though the GUI was up and running). I believe the initial process does some kind of activation of the real process and then exits. Hmm, doesn't it give you a point to do 'bt' to get the traces? I know I can attach to the second process, but I think the stuff I need to see happens at startup. So how can I get a debugger on the startup process of the real evolution? I tried to search the archive, but the search function seems broken: it searches everything, even though it says search this list only. Also, evolution seems to have two presences on the web: a web site and a wiki. The developer stuff on the web site is old, and has no pointers to the newer stuff that I could find. It would be good if it did. We are in the process of revamping the wiki with more relevant in information. We would also rebuild the site with latest informations. But that would take a while. I have some broader questions too, if anyone has any comments on them. What version to start with? I'm on Debian GNU/Linux, which currently has evo 2.6. I notice that's a bit dated (although I did see that a few months ago some of the Debian packagers were interested in making a more recent version). I've been working from the Debian version. Does that version, the last stable release (from evo, not Debian), or svn head make the most sense to work from? (BTW, the one bug I fixed was one that was already fixed post-2.6). I would prefer, if you can try it with Evolution 2.10.2. It is the most recent stable release. -Srini. Mission Impossible? Am I likely to get anywhere without spending lots of time? I'm a professional software developer, but I'm not familiar with GNOME, and this is clearly a complex application. Mission Advisable? If I get past the setup hurdles, is evolution likely to be able to handle the mail store I described? How do I find out which of the imap store's code I'm actually using? Thanks. Ross Boylan ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: Hi. I've been getting into the code of evolution recently, and am thinking of doing a bit more to see if I can get it working OK for my situation. I have an IMAP mailbox which is very large, both in terms of folders (over 100) and messages (the largest folder has 300,000 messages; my INBOX has about 22,000). the largest INBOX I've ever used was about ~100,000 messages, so you may get to have some fun :) None of the email clients I've tried cope with this very well. Since I've been using evo at work in a similar setup (cyrus server, though not quite as big), I thought it might be the best to try to tweak. The problems I've had so far involve setup activities rather than core functionaility. First, evolution couldn't create the account (solved); second I've had problems getting it to show all of my subfolders. are they in different namespaces? Current evo IMAP provider doesn't handle multiple namespaces :( My narrow question is how to debug evolution. When I launched evolution in gdb evolution (the GUI) came up, and then the debugger told me the process had exited (though the GUI was up and running). I believe the initial process does some kind of activation of the real process and then exits. sounds like you already had an instance of Evolution running (on another virtual desktop perhaps?) Evolution only allows one instance of itself to be running for the same user account... if you try to start a second instance, it signals the original instance to open a new window and then exits. I know I can attach to the second process, but I think the stuff I need to see happens at startup. So how can I get a debugger on the startup process of the real evolution? should just be able to do gdb evolution (Note: I haven't debugged evolution in over a year since I've moved onto other projects, but that's how I always debugged it) I tried to search the archive, but the search function seems broken: it searches everything, even though it says search this list only. Also, evolution seems to have two presences on the web: a web site and a wiki. The developer stuff on the web site is old, and has no pointers to the newer stuff that I could find. It would be good if it did. I have some broader questions too, if anyone has any comments on them. What version to start with? I'm on Debian GNU/Linux, which currently has evo 2.6. I notice that's a bit dated (although I did see that a few months ago some of the Debian packagers were interested in making a more recent version). I've been working from the Debian version. Does that version, the last stable release (from evo, not Debian), or svn head make the most sense to work from? (BTW, the one bug I fixed was one that was already fixed post-2.6). probably best to start with 2.10(.2) (or svn if you are brave) so as to avoid spending time fixing things that have already been fixed. Mission Impossible? Am I likely to get anywhere without spending lots of time? I guess the answer to this is all relative... :) I'm a professional software developer, but I'm not familiar with GNOME, and this is clearly a complex application. Mission Advisable? If I get past the setup hurdles, is evolution likely to be able to handle the mail store I described? I think it'll be possible, there's lots of improvement that can be made to the current imap code :) How do I find out which of the imap store's code I'm actually using? you are probably using the provider in evolution-data-server/camel/providers/imap - imap4 is a project I started to replace the current imap provider and works fairly ok, but isn't quite complete (I forget all what it is missing since it's been a few years since I actively hacked on it - I think the main thing is cached search results?). This provider, like the current imap backend, suffers from being synchronous, but it is far better designed and much cleaner code to read. This backend also supports multiplenamespaces (tho it'd be better if the Camel API included multiple-namespace support, the way it works in imap4 is because all folders are listed from a toplevel namespace rather than from individual namespaces like it should). - imapp is an old attempt at making a pipelined imap provider, tho it is basically a dead-end at this point. - imapx is yet another attempt at replacing the current imap backend but depends on a lot of unfinished stuff in a development branch that has been abandoned for about 2 years now I think (the guy that had been working on that quit) Jeff ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 23:14 +0530, Srinivasa Ragavan wrote: Hi, On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: Hi. I've been getting into the code of evolution recently, and am thinking of doing a bit more to see if I can get it working OK for my situation. I have an IMAP mailbox which is very large, both in terms of folders (over 100) and messages (the largest folder has 300,000 messages; my INBOX has about 22,000). None of the email clients I've tried cope with this very well. Since I've been using evo at work in a similar setup (cyrus server, though not quite as big), I thought it might be the best to try to tweak. The problems I've had so far involve setup activities rather than core functionaility. First, evolution couldn't create the account (solved); second I've had problems getting it to show all of my subfolders. My narrow question is how to debug evolution. When I launched evolution in gdb evolution (the GUI) came up, and then the debugger told me the process had exited (though the GUI was up and running). I believe the initial process does some kind of activation of the real process and then exits. Hmm, doesn't it give you a point to do 'bt' to get the traces? I don't think there's anything to trace, since the action is happening in another process. When I begin I am at the gdb command line. I was able to insert a breakpoint at what seemed like a good spot (this was a future breakpoint, since the relevant library hadn't loaded). Then when I did 'run' the GUI came up, and the program I was debugging exited. There was no error, and I didn't hit the breakpoint, so I don't think bt would be meaningful. Or perhaps I'm misunderstanding and it is only one thread that has exited. But I see in main.c a lot of IDL code, and I know GNOME apps are supposed to be able to discover if they've already launched. I guessed (mostly by analogy with MS COM) that the initial program simply requests a service, and that is either conveyed to an existing instance of the app or a new instance is created. I know I can attach to the second process, but I think the stuff I need to see happens at startup. So how can I get a debugger on the startup process of the real evolution? -- Ross Boylan wk: (415) 514-8146 185 Berry St #5700 [EMAIL PROTECTED] Dept of Epidemiology and Biostatistics fax: (415) 514-8150 University of California, San Francisco San Francisco, CA 94107-1739 hm: (415) 550-1062 ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 14:10 -0400, Jeffrey Stedfast wrote: On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: Hi. I've been getting into the code of evolution recently, and am thinking of doing a bit more to see if I can get it working OK for my situation. I have an IMAP mailbox which is very large, both in terms of folders (over 100) and messages (the largest folder has 300,000 messages; my INBOX has about 22,000). the largest INBOX I've ever used was about ~100,000 messages, so you may get to have some fun :) None of the email clients I've tried cope with this very well. Since I've been using evo at work in a similar setup (cyrus server, though not quite as big), I thought it might be the best to try to tweak. The problems I've had so far involve setup activities rather than core functionaility. First, evolution couldn't create the account (solved); second I've had problems getting it to show all of my subfolders. are they in different namespaces? Current evo IMAP provider doesn't handle multiple namespaces :( Single namespace. It's all INBOX.folder.subfolder. The one wrinkle is that in some cases 'folder' exists in the namespace, but is not an actual box or folder (whatever the right term is) on the server: INBOX.folder.subfolder is a real folder; INBOX.folder is not. My narrow question is how to debug evolution. When I launched evolution in gdb evolution (the GUI) came up, and then the debugger told me the process had exited (though the GUI was up and running). I believe the initial process does some kind of activation of the real process and then exits. sounds like you already had an instance of Evolution running (on another virtual desktop perhaps?) I'm pretty sure I shut everything down. Hmm, I just tried again and this time I'm still in the debugger. There was another instance of evolution running, but for a different user. That instance is in a funny state, because it is not visible in the other user's screen. Running under KDE in all cases. Mission Advisable? If I get past the setup hurdles, is evolution likely to be able to handle the mail store I described? I think it'll be possible, there's lots of improvement that can be made to the current imap code :) I'm hoping it will be tolerable without improvements :) I think one of the main factors will be whether evo tries to get all the headers from each folder it operates on. Even if it does, if it only does it once and is otherwise responsive I'll be doing better than what I have now. At work, I think my biggest folder has about 85k messages, and evo is working fine. How do I find out which of the imap store's code I'm actually using? you are probably using the provider in evolution-data-server/camel/providers/imap Is the other not loaded, or does it depend on which option I picked for the server (I said IMAP4rev1). Thanks to everyone for the quick replies. Not having a functional email client is a big problem. I used to archive off my mail when it got to be too much. I thought it would be more convenient to have it consolidated under IMAP since, as a practical matter, once I archived the stuff it became invisible. So far, it's not so convenient! So far, I have tried evolution kmail (using it for now) thunderbird/icedove mulberry Apple's mail app claws mutt -- Ross Boylan wk: (415) 514-8146 185 Berry St #5700 [EMAIL PROTECTED] Dept of Epidemiology and Biostatistics fax: (415) 514-8150 University of California, San Francisco San Francisco, CA 94107-1739 hm: (415) 550-1062 ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 14:10 -0400, Jeffrey Stedfast wrote: On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: Hi. I've been getting into the code of evolution recently, and am thinking of doing a bit more to see if I can get it working OK for my situation. I have an IMAP mailbox which is very large, both in terms of folders (over 100) and messages (the largest folder has 300,000 messages; my INBOX has about 22,000). the largest INBOX I've ever used was about ~100,000 messages, so you may get to have some fun :) None of the email clients I've tried cope with this very well. Since I've been using evo at work in a similar setup (cyrus server, though not quite as big), I thought it might be the best to try to tweak. The problems I've had so far involve setup activities rather than core functionaility. First, evolution couldn't create the account (solved); second I've had problems getting it to show all of my subfolders. are they in different namespaces? Current evo IMAP provider doesn't handle multiple namespaces :( Maybe it's time to plan this on the wiki ? My narrow question is how to debug evolution. When I launched evolution in gdb evolution (the GUI) came up, and then the debugger told me the process had exited (though the GUI was up and running). I believe the initial process does some kind of activation of the real process and then exits. sounds like you already had an instance of Evolution running (on another virtual desktop perhaps?) Evolution only allows one instance of itself to be running for the same user account... if you try to start a second instance, it signals the original instance to open a new window and then exits. I know I can attach to the second process, but I think the stuff I need to see happens at startup. So how can I get a debugger on the startup process of the real evolution? should just be able to do gdb evolution This is one of the reasons we independent instances across mail, calendar, contacts, etc would be good. Easier debugging, isolation of problems, performance tune up, all by component in a far easier way, IMHO. (Note: I haven't debugged evolution in over a year since I've moved onto other projects, but that's how I always debugged it) I tried to search the archive, but the search function seems broken: it searches everything, even though it says search this list only. Also, evolution seems to have two presences on the web: a web site and a wiki. The developer stuff on the web site is old, and has no pointers to the newer stuff that I could find. It would be good if it did. I have some broader questions too, if anyone has any comments on them. What version to start with? I'm on Debian GNU/Linux, which currently has evo 2.6. I notice that's a bit dated (although I did see that a few months ago some of the Debian packagers were interested in making a more recent version). I've been working from the Debian version. Does that version, the last stable release (from evo, not Debian), or svn head make the most sense to work from? (BTW, the one bug I fixed was one that was already fixed post-2.6). probably best to start with 2.10(.2) (or svn if you are brave) so as to avoid spending time fixing things that have already been fixed. Mission Impossible? Am I likely to get anywhere without spending lots of time? I guess the answer to this is all relative... :) I'm a professional software developer, but I'm not familiar with GNOME, and this is clearly a complex application. Mission Advisable? If I get past the setup hurdles, is evolution likely to be able to handle the mail store I described? I think it'll be possible, there's lots of improvement that can be made to the current imap code :) How do I find out which of the imap store's code I'm actually using? you are probably using the provider in evolution-data-server/camel/providers/imap - imap4 is a project I started to replace the current imap provider and works fairly ok, but isn't quite complete (I forget all what it is missing since it's been a few years since I actively hacked on it - I think the main thing is cached search results?). This provider, like the current imap backend, suffers from being synchronous, but it is far better designed and much cleaner code to read. This backend also supports multiplenamespaces (tho it'd be better if the Camel API included multiple-namespace support, the way it works in imap4 is because all folders are listed from a toplevel namespace rather than from individual namespaces like it should). - imapp is an old attempt at making a pipelined imap provider, tho it is basically a dead-end at this point. - imapx is yet another attempt at replacing the current imap backend but depends on a lot of unfinished stuff in a development branch
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 23:14 +0530, Srinivasa Ragavan wrote: [Ross] What version to start with? I would prefer, if you can try it with Evolution 2.10.2. It is the most recent stable release. Is EVOLUTION_DATA_SERVER_1_10_2 the corresponding tag to use for evolution-data-server? -- Ross Boylan wk: (415) 514-8146 185 Berry St #5700 [EMAIL PROTECTED] Dept of Epidemiology and Biostatistics fax: (415) 514-8150 University of California, San Francisco San Francisco, CA 94107-1739 hm: (415) 550-1062 ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 12:22 -0700, Ross Boylan wrote: On Thu, 2007-05-31 at 23:14 +0530, Srinivasa Ragavan wrote: [Ross] What version to start with? I would prefer, if you can try it with Evolution 2.10.2. It is the most recent stable release. Is EVOLUTION_DATA_SERVER_1_10_2 the corresponding tag to use for evolution-data-server? Yes. You are right. -Srini. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: What version to start with? I'm on Debian GNU/Linux, which currently has evo 2.6. I notice that's a bit dated (although I did see that a few months ago some of the Debian packagers were interested in making a more recent version). I've been working from the Debian version. Does that version, the last stable release (from evo, not Debian), or svn head make the most sense to work from? (BTW, the one bug I fixed was one that was already fixed post-2.6). FYI, Debian Unstable has Evolution 2.10. Might be easier to grab at least the 2.10 dependencies from there. You'll need to upgrade gtkhtml and likely also your GTK+ library stack to get 2.10 to build. Matthew Barnes ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 13:38 -0700, Ross Boylan wrote: On Thu, 2007-05-31 at 16:10 -0400, Matthew Barnes wrote: On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: What version to start with? I'm on Debian GNU/Linux, which currently has evo 2.6. I notice that's a bit dated (although I did see that a few months ago some of the Debian packagers were interested in making a more recent version). I've been working from the Debian version. Does that version, the last stable release (from evo, not Debian), or svn head make the most sense to work from? (BTW, the one bug I fixed was one that was already fixed post-2.6). FYI, Debian Unstable has Evolution 2.10. Might be easier to grab at least the 2.10 dependencies from there. You'll need to upgrade gtkhtml and likely also your GTK+ library stack to get 2.10 to build. Matthew Barnes If I update GTK+ is that going to break other apps that use that library? no, gtk maintains binary compat with older versions of the same major revision (2.x in this case) Jeff ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 11:38 -0700, Ross Boylan wrote: On Thu, 2007-05-31 at 14:10 -0400, Jeffrey Stedfast wrote: On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: Hi. I've been getting into the code of evolution recently, and am thinking of doing a bit more to see if I can get it working OK for my situation. I have an IMAP mailbox which is very large, both in terms of folders (over 100) and messages (the largest folder has 300,000 messages; my INBOX has about 22,000). the largest INBOX I've ever used was about ~100,000 messages, so you may get to have some fun :) None of the email clients I've tried cope with this very well. Since I've been using evo at work in a similar setup (cyrus server, though not quite as big), I thought it might be the best to try to tweak. The problems I've had so far involve setup activities rather than core functionaility. First, evolution couldn't create the account (solved); second I've had problems getting it to show all of my subfolders. are they in different namespaces? Current evo IMAP provider doesn't handle multiple namespaces :( Single namespace. It's all INBOX.folder.subfolder. The one wrinkle is that in some cases 'folder' exists in the namespace, but is not an actual box or folder (whatever the right term is) on the server: INBOX.folder.subfolder is a real folder; INBOX.folder is not. I guess just a bug [snip] How do I find out which of the imap store's code I'm actually using? you are probably using the provider in evolution-data-server/camel/providers/imap Is the other not loaded, or does it depend on which option I picked for the server (I said IMAP4rev1). ah, in that case you chose imap4 and not imap the imap provider is simply named IMAP in the drop-down menu in the account editor. Jeff ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 23:20 +0200, Øystein Gisnås wrote: 2007/5/31, Matthew Barnes [EMAIL PROTECTED]: On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: What version to start with? I'm on Debian GNU/Linux, which currently has evo 2.6. I notice that's a bit dated (although I did see that a few months ago some of the Debian packagers were interested in making a more recent version). I've been working from the Debian version. Does that version, the last stable release (from evo, not Debian), or svn head make the most sense to work from? (BTW, the one bug I fixed was one that was already fixed post-2.6). FYI, Debian Unstable has Evolution 2.10. Might be easier to grab at least the 2.10 dependencies from there. You'll need to upgrade gtkhtml and likely also your GTK+ library stack to get 2.10 to build. In case you're on a Debian-based distribution Yes; straight Debian. and not pulling from svn, I would recommend using pre-built packages, or even building the packages yourself. 2.10.2 is in the archive, and I will do 2.11.2 this weekend. Terrific. Is unstable the place to look, or experimental, or somewhere else? For my own development setup I use the 2.10.x packages plus custom build from svn for the module I'm hacking on. e-d-s for example, I install to /opt/evolution-data-server. Then I can start development e-d-s with 'LD_LIBRARY_PATH=/opt/evolution-data-server/lib /opt/evolution-data-server/libexec/evolution-data-server-1.12' and can also start the stable e-d-s with '/usr/lib/evolution/evolution-data-server-1.10' That's a great tip. I built evo from Debian source (with one fix) and it took 45 minutes. I clearly need a shorter route to trying out changes. Thanks for packing evo, and for making the -dbg files available. Ross -- Ross Boylan wk: (415) 514-8146 185 Berry St #5700 [EMAIL PROTECTED] Dept of Epidemiology and Biostatistics fax: (415) 514-8150 University of California, San Francisco San Francisco, CA 94107-1739 hm: (415) 550-1062 ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
On Thu, 2007-05-31 at 14:22 -0700, Ross Boylan wrote: That sounds as if no upgrade of other apps would be forced. Or did you mean the stuff the evolution and gtk+ depends on would all need to go to newer versions? That's probably a lot of stuff, but it's not so bad. Hard to predict. You can always cancel apt-get. :) Is there somewhere that has a useful introduction to bonobo-activation-server and related machinery? The references from http://developer.gnome.org/doc/guides/platform-overview/platform-overview.html#bonobo-corba? None of those look very friendly. Not to my knowledge. Bonobo itself isn't very friendly, and is on its way out. Matthew Barnes ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Introduction and Questions
2007/5/31, Ross Boylan [EMAIL PROTECTED]: On Thu, 2007-05-31 at 23:20 +0200, Øystein Gisnås wrote: 2007/5/31, Matthew Barnes [EMAIL PROTECTED]: On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote: What version to start with? I'm on Debian GNU/Linux, which currently has evo 2.6. I notice that's a bit dated (although I did see that a few months ago some of the Debian packagers were interested in making a more recent version). I've been working from the Debian version. Does that version, the last stable release (from evo, not Debian), or svn head make the most sense to work from? (BTW, the one bug I fixed was one that was already fixed post-2.6). FYI, Debian Unstable has Evolution 2.10. Might be easier to grab at least the 2.10 dependencies from there. You'll need to upgrade gtkhtml and likely also your GTK+ library stack to get 2.10 to build. In case you're on a Debian-based distribution Yes; straight Debian. and not pulling from svn, I would recommend using pre-built packages, or even building the packages yourself. 2.10.2 is in the archive, and I will do 2.11.2 this weekend. Terrific. Is unstable the place to look, or experimental, or somewhere else? 2.11.2 will end up in pkg-evolution's svn for sure. If we upload binary packages, they will go to experimental. For my own development setup I use the 2.10.x packages plus custom build from svn for the module I'm hacking on. e-d-s for example, I install to /opt/evolution-data-server. Then I can start development e-d-s with 'LD_LIBRARY_PATH=/opt/evolution-data-server/lib /opt/evolution-data-server/libexec/evolution-data-server-1.12' and can also start the stable e-d-s with '/usr/lib/evolution/evolution-data-server-1.10' That's a great tip. I built evo from Debian source (with one fix) and it took 45 minutes. I clearly need a shorter route to trying out changes. I don't think you get around the initial build; you can feature strip it to reduce the time. Once you've built it. make will figure out what has to be rebuilt. A make make install when I edit files in only one directory takes about 5 seconds on my system.. Thanks for packing evo, and for making the -dbg files available. You're welcome. Don't hesitate with questions on how you can use packages or package scripts in your development setup.. Cheers, Øystein ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
