Re: performance issue (imap spool on san)
As Mulberry is released freely now I tried it and I have to say, that neither configuration nor usage is intuitive in any way. I needed several tries to configure it for IMAPS and Auth'ed SMTPS. I really cannot use it, I have to search too long for every function and most are located that illogically, that I seem to be unable to remember them. And it misses a lot of features I use every day. Virtual folders, inline attachments (jpegs for example), forwarding emails attached, view attached emails, Drag and Drop support and so on and so forth. And a thunderbird with cached headers is multiple times faster in resorting and scrolling, not only over 3 MBit DSL Line, but even over LAN. It's fine that mulberry doesn't need to cache headers, but why isn't it able to do so? Loading on demand and then caching it would be the best of both worlds. So just implementing every IMAP feature available might be the best thing for the server and the protocol, but not for the user. You need a intuitive interface and nowadays it really has to look nice, too if a non-geek should use it. And to be honest, mulberry simply looks horrible... But it's nice, that everyone who doesn't care about looking and usability now has a suitable free IMAP client availiable. Best, Daniel Sebastian Hagedorn wrote: Hi, let me start by saying that I'm basically always on high-speed Internet connections and keep my mailboxes relatively small. So I haven't had bad experiences with Thunderbird in that regard. I still don't use it, but that's not the issue here. -- Daniel Eckl [EMAIL PROTECTED] is rumored to have mumbled on 26. Juli 2006 21:31:40 +0200 regarding Re: performance issue (imap spool on san): In a graphical client you want to be able to scroll through your whole message list without any delay. So I think there is no other chance but caching all header information of a folder. On low bandwith situations it seems to be impossible for me to do this on demand only. Well, you don't know Mulberry then. It has a GUI and is very adaptable to any kind of network situation and it doesn't cache anything! You do have to tweak it, however. It doesn't auto-configure, but there are *many* settings to play around with. Obviously that's both a blessing and a curse. I was searching for a full featured graphical IMAP client for a long time and tried everything and I have to say: Not only is thunderbird an IMAP client, it is the best graphical IMAP client I could find in the whole Linux and Windows world. Unfortunately that's not saying much ... it's just too bad that the only decent IMAP GUI client isn't developed further. I have hopes for Thunderbird 2, but I have a feeling that I will continue to use Mulberry for a long time ... -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum für angewandte Informatik - Universitätsweiter Service RRZK Universität zu Köln / Cologne University - Tel. +49-221-478-5587 Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
--On Monday, September 11, 2006 14:14 +0200 Daniel Eckl [EMAIL PROTECTED] wrote: But it's nice, that everyone who doesn't care about looking and usability now has a suitable free IMAP client availiable. It is possible to state your preference without insulting everyone else. Joseph Brennan Columbia University Information Technology Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
--On Monday, September 11, 2006 2:14 PM +0200 Daniel Eckl [EMAIL PROTECTED] wrote: Everyone is entitled to their opinion and yours is certainly welcome. However, beauty is in the eye of the beholder... I have heard comments the cover pretty much the whole spectrum with regards to its usability and looks, and really, I don't see any particular opinion winning over the other. For the most part, opinions are based on how well that particular user loved a favorite e-mail client they previously used. And it misses a lot of features I use every day. Virtual folders, inline attachments (jpegs for example), forwarding emails attached, view attached emails, Drag and Drop support and so on and so forth. I must admit, I would love to see Virtual Folder support... however, is this implemented in the majority of IMAP clients out there? I am not sure that it is... so, to single out Mulberry for that is unreasonable. Is it a wish-list item? Certainly. Inline attachments are a long heated topic of debate here... of course there are features left to implement... however, Cyrus is one man and he has to prioritize what order of features should get implemented in. When Mulberry was not a free product, the features that were requested most and/or payed for were the features to get implemented first. Now that Cyrus is a one-man show (currently), it may be awhile before we see any new features get added... Forwarding as attachments is a function that already exists. Viewing attached e-mails exists, but maybe not the way you would like... If it can be viewed inline, Mulberry allows it... otherwise, you right-click and click view and it opens an external viewer... Drag and drop support is also implemented... though, without context in your complaint, I am not sure what you are expecting... I certainly can drag and drop an attachment from my desktop to the attachment section of my draft and it works as expected. Admittedly, I haven't dragged and dropped things anywhere else... Your arguments that things aren't as intuitive and easy to find is a good argument. Some people don't have problems (like me), but other do. This may be why some of the features you mention above aren't known to you. There was a project from CMU (by students?) that had taken on the job of analyzing the Mulberry interface to make recommendations on how to improve it. They did surveys and usability testing, and had entered an agreement (contract?) with Cyrusosft (Isamet) to implement most if not all of the recommended changes. This was a great step forward and a great idea. However, the project seem to go slowly and then disbanded when Isamet fell apart. And a thunderbird with cached headers is multiple times faster in resorting and scrolling, not only over 3 MBit DSL Line, but even over LAN. It's fine that mulberry doesn't need to cache headers, but why isn't it able to do so? Loading on demand and then caching it would be the best of both worlds. The same could be said for IMSP preferences. It isn't as noticeable over high speed connections, but over dialup... *whoosh*. The first thing I see is Mulberry download all of my preferences (and I have a lot). Then I see it turn around and write all the preferences back (at least from what I could see). Only then can I start to use Mulberry. Shutting down is also just as hard... all of my preferences get written back. I am not sure if there is a good answer... especially since I don't REALLY know what is going on... Another popular problem people complained about is the multi-threaded nature of Mulberry. Some Mulberry actions freezes the whole application and prevents you from doing anything until those actions are done. The point it, there is lots of improvements that could still be made and were in the works. Cyrus acknowledged the problems it had and was very clear where he stood and where he was going. It is a tough job being the author of one of the most popular IMAP clients on the marget, as a lot more demand and expectation gets added to that... So just implementing every IMAP feature available might be the best thing for the server and the protocol, but not for the user. You need a intuitive interface and nowadays it really has to look nice, too if a non-geek should use it. And to be honest, mulberry simply looks horrible... I would not agree with the horrible philosophy. I personally like the look and feel of Mulberry. I think the CMU approach was a good one and it would be nice for something like that to be resurrected. There is something to be said for surveys and usability testing... it answers some of the issues you have brought up. But it's nice, that everyone who doesn't care about looking and usability now has a suitable free IMAP client availiable. Uh... okay :) Of course, we really don't know how big the everyone who doesn't care about looking at usability really is now, do we? As I said before, there are a lot of people on BOTH sides of the
Re: performance issue (imap spool on san)
--On 11. September 2006 14:14:21 +0200 Daniel Eckl [EMAIL PROTECTED] wrote: And it misses a lot of features I use every day. Virtual folders, inline attachments (jpegs for example), forwarding emails attached, view attached emails, Drag and Drop support and so on and so forth. Scott already answered many points, but I wanted to mention that the Linux version is *by far* less polished than the Windows and OS X versions. Drag and drop is merely one example of that. The Linux Mulberry 3 release was fine, but version 4 never really made it out of beta. You may not care, but I wanted to state it for the record ... -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. Zentrum für angewandte Informatik - Universitätsweiter Service RRZK .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. pgpmSkY302jsh.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
Hi Scott! Thanks for your thoughts! I really appreciate reading your perspective! I might not have stated clearly enough, whos opinion I speak about. It was the opinion of about 20 very different users I evaluated Mulberry with and my opinion, too. This of course is not representative in any imaginable way and might and will not apply to you and many other people. Above all I think it does not apply top most of the mailing list members here which all more or less do administrate a major professional grade IMAP server named cyrus-imapd. But it could easily be the opinion of most users without technical background being used to clients like Outlook Express or Thunderbird. Yes I fully believe that I might find some or many of my missing features if I read the manual thoroughly. So perhaps the only problem which is a real one for me might be the lack of intuitive usability. All others might be just an aftereffect of that. It would really be nice if the (near or far) future would provide an improvement here. Virtual folders I used and use in thunderbird, kmail, IMP Webmail (Horde framework) and opera mail. Without that, mulberry is not unreasonable for everyone, that's just for people using IMAP the way I do. I don't have any clue how many people this might be. Because I used vfolders until now, I have less real folders and changing to mulberry would mean to create a lot of folders, moving and copying mails around, and many of the new folders will contain copies of mails already in other folders. Or as an alternative, starting the same searches manually over and over again. Both is not practicable either for me or for my mailbox quota. :) Inline viewing of attachments surely is a subject worth of discussion, but you will never get a definite answer. You have to make that configurable and depending on your major type of users you have to choose what's the default. With mulberry which is mostly choosen by geeks I think, I would choose disabled inline viewing by default. Drag and Drop from konqueror to mulberry somehow didn't work for me, but after your writign I don't want to blame mulberry here without further investigating that. So I just accept that it should work and there might be something broken in my KDE DnD functionality. :) Regarding the visual candy thing, you are correct, there will be people loving mulberrys interface style, but I cannot believe that this is more than a homeopathic attenuation. Just look at all the actual major interfaces out there (qt, gtk, windows xp/vista) to get a clue what seems to be wanted by the majority of normal PC users. I'm really sure that Mulberry would benefit from being ported to a nowadays widget style and follow one of the major usability guidelines people are used to. Regarding the single-threaded nature of mulberry I didn't run into problems with that so far, but thinking of small dial-in lines like GSM I could easily imagine that it's disturbing to wait for one mail to be sent and not being able to start writing the next one until it's done... So thanks again and have a lot of fun! Best, Daniel Scott Adkins wrote: --On Monday, September 11, 2006 2:14 PM +0200 Daniel Eckl [EMAIL PROTECTED] wrote: Everyone is entitled to their opinion and yours is certainly welcome. However, beauty is in the eye of the beholder... I have heard comments the cover pretty much the whole spectrum with regards to its usability and looks, and really, I don't see any particular opinion winning over the other. For the most part, opinions are based on how well that particular user loved a favorite e-mail client they previously used. And it misses a lot of features I use every day. Virtual folders, inline attachments (jpegs for example), forwarding emails attached, view attached emails, Drag and Drop support and so on and so forth. I must admit, I would love to see Virtual Folder support... however, is this implemented in the majority of IMAP clients out there? I am not sure that it is... so, to single out Mulberry for that is unreasonable. Is it a wish-list item? Certainly. Inline attachments are a long heated topic of debate here... of course there are features left to implement... however, Cyrus is one man and he has to prioritize what order of features should get implemented in. When Mulberry was not a free product, the features that were requested most and/or payed for were the features to get implemented first. Now that Cyrus is a one-man show (currently), it may be awhile before we see any new features get added... Forwarding as attachments is a function that already exists. Viewing attached e-mails exists, but maybe not the way you would like... If it can be viewed inline, Mulberry allows it... otherwise, you right-click and click view and it opens an external viewer... Drag and drop support is also implemented... though, without context in your complaint, I am not sure what you are expecting... I certainly can drag and
Re: performance issue (imap spool on san)
Hi Sebastian! That's a good point! I just looked at the linux version and didn't thought that there could be major differences! Thanks a lot for this info! Best, Daniel Sebastian Hagedorn wrote: --On 11. September 2006 14:14:21 +0200 Daniel Eckl [EMAIL PROTECTED] wrote: And it misses a lot of features I use every day. Virtual folders, inline attachments (jpegs for example), forwarding emails attached, view attached emails, Drag and Drop support and so on and so forth. Scott already answered many points, but I wanted to mention that the Linux version is *by far* less polished than the Windows and OS X versions. Drag and drop is merely one example of that. The Linux Mulberry 3 release was fine, but version 4 never really made it out of beta. You may not care, but I wanted to state it for the record Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
Well, I really don't know why you should feel insulted if that applies to you. If you don't take care of polished, skinnable widget style with bells and whistles and spinning activity indicators and shaded buttons and so on, because your preference is more to features and speed and such things than this is fully legitimate and it's the reason why we have such a big variety of software in the world. I call that superior, not insulting. Best, Daniel Joseph Brennan wrote: --On Monday, September 11, 2006 14:14 +0200 Daniel Eckl [EMAIL PROTECTED] wrote: But it's nice, that everyone who doesn't care about looking and usability now has a suitable free IMAP client availiable. It is possible to state your preference without insulting everyone else. Joseph Brennan Columbia University Information Technology Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
Thank you all for your responses. This was insightful. So, perhaps we could state that the desired behavior of any IMAP client would be to fetch only those message headers it nedds to and perhaps a bit more. In case of TB, that would transalte to fetching only headers that would be visible to the user and perhaps screenful of header up and down. Caching would also be advised only for the session or perhaps make it configurable. Nix. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
On Fri, Jul 28, 2006 at 12:18:12AM -0700, Nikola Milutinovic wrote: So, perhaps we could state that the desired behavior of any IMAP client would be to fetch only those message headers it nedds to and perhaps a bit more. In case of TB, that would transalte to fetching only headers that would be visible to the user and perhaps screenful of header up and down. It also helps if the clients ask for a limited set of headers from each message of interest. For example, Squirrelmail asks for this lot at mailbox opening time: (FLAGS UID RFC822.SIZE INTERNALDATE BODY.PEEK[HEADER.FIELDS (Date To Cc From Subject X-Priority Importance Priority Content-Type)]) Cyrus caches some headers in the index file, but unfortunately does not include Importance or Content-Type so it has to open and parse every message file in the mailbox to satisfy the request! This is defined in imapd/mailbox.c in struct mailbox_header_cache mailbox_cache_headers. I am considering expanding the list in that definition to improve efficiency: does anyone know of any problems this might cause? (This is for a new deployment, so there are no existing index files to worry about). Andrew -- --- | From Andrew Findlay, Skills 1st Ltd | | Consultant in large-scale systems, networks, and directory services | | http://www.skills-1st.co.uk/+44 1628 782565 | --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
Andrew Findlay schrieb: On Fri, Jul 28, 2006 at 12:18:12AM -0700, Nikola Milutinovic wrote: So, perhaps we could state that the desired behavior of any IMAP client would be to fetch only those message headers it nedds to and perhaps a bit more. In case of TB, that would transalte to fetching only headers that would be visible to the user and perhaps screenful of header up and down. It also helps if the clients ask for a limited set of headers from each message of interest. Yes, I want to second all these statements. That would be my prefered solution, too. By the way: I checked kmails behavior. It fetches the headers of all mails in a folder, too, but I think it uses just the limited set of headers and that's why it is so incredible fast compared to thunderbird. Disadvantage: It cannot show me if a mail has attachments, before I click on it and load it this way. But that might be no problem in most cases. So I think TB already would get a huge speed improvement by just implementing a limited header set query. Best, Daniel Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
On Fri, Jul 28, 2006 at 10:41:19AM +0200, Daniel Eckl wrote: Andrew Findlay schrieb: On Fri, Jul 28, 2006 at 12:18:12AM -0700, Nikola Milutinovic wrote: So, perhaps we could state that the desired behavior of any IMAP client would be to fetch only those message headers it nedds to and perhaps a bit more. In case of TB, that would transalte to fetching only headers that would be visible to the user and perhaps screenful of header up and down. It also helps if the clients ask for a limited set of headers from each message of interest. Yes, I want to second all these statements. That would be my prefered solution, too. By the way: I checked kmails behavior. It fetches the headers of all mails in a folder, too, but I think it uses just the limited set of headers and that's why it is so incredible fast compared to thunderbird. Might it not be better to have Cyrus 'learn' what header lines are needed, rather than just bloating the list with more headers. The set of headers would needed to be dynamically changable. The points are: 1) different IMAP clients want different sets of headers. The same IMAP client at different releases might change the set requested. 2) most individual sites run only 2 or 3 different IMAP clients, why get Cyrus to collect the headers that the IMAP clients at that site don't want. 3) most system admins don't have the skills/inclination/... to optimise the set of headers cached. -- Alain Williams Parliament Hill Computers Ltd. Linux Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer. +44 (0) 787 668 0256 http://www.phcomp.co.uk/ #include std_disclaimer.h Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
On Fri, Jul 28, 2006 at 10:37:43AM +0100, Alain Williams wrote: Might it not be better to have Cyrus 'learn' what header lines are needed, rather than just bloating the list with more headers. The set of headers would needed to be dynamically changable. The points are: 1) different IMAP clients want different sets of headers. The same IMAP client at different releases might change the set requested. 2) most individual sites run only 2 or 3 different IMAP clients, why get Cyrus to collect the headers that the IMAP clients at that site don't want. 3) most system admins don't have the skills/inclination/... to optimise the set of headers cached. The problem with the learning approach is that the index is no use at all unless it contains the same headers from every message, and it is built at message delivery time rather than when the message is read. I suppose some sort of 'lazy indexing' could add headers to the index on demand, but the code changes might be significant. Headers are not usually very large, so I would be more inclined to the idea that the index should store every header (perhaps with a blocklist to avoid things like Received:) Andrew -- --- | From Andrew Findlay, Skills 1st Ltd | | Consultant in large-scale systems, networks, and directory services | | http://www.skills-1st.co.uk/+44 1628 782565 | --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
On 2006-07-28 at 11:05 +0100, Andrew Findlay wrote: Headers are not usually very large, so I would be more inclined to the idea that the index should store every header (perhaps with a blocklist to avoid things like Received:) Those fine folk at Cambridge who introduced replication have a number of other patches; what you suggest is in fact done: URL:http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/performance.html and there's a lot more interesting stuff on David Carter's cyrus pages. Top-level, predictably: URL:http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/index.html http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/index.html -- Everything has three factors: politics, money, and the right way to do it. In that order. -- Gary Donahue Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
The first time you open a large IMAP folder is not very fast, I have to admit, but I didn't find any other comparable IMAP client without this problem. Perhaps there are some, but I didn't try them because of the lack of other basic email features. This is why they aren't IMAP clients. IMAP servers make all manner of searching, sorting, retrieval, and storage options completely available to the client, without having to download even all the headers. This is why Mulberry, Pine, Mutt, and Kmail are so much faster. If TBird would just do that instead of insisting on blindly attempting to download all the headers and performing all sorting and searching on the client. TBird and most of the others have their roots and brains seated back in the POP3 dark ages near as I can tell and that's how they treat all mail stores. IMAP allows the clients to easily ask for threaded views (unless you turn the index options off or something like that) from the server, as well as partial sets of headers in batch. This massively speeds things up when you're on a modem, or working with large mailboxes, or mailboxes you only occasionally open. I must say I'm a bit lost here. I have been using and administering Cyrus IMAP for years now, so I'm not really a new kid on the block. Say you have a GUI IMAP client XxX. Say you start it up and click on the INBOX. What would you desire/expect XxX to do? I would expect to see all mail headers in my inbox. So, how do you do that in XxX, other then fetching all mail headers? How do you do that in Pine and Mutt? Now, fetching it EVERY time, well, that would make me switch to another client. So, caching comes in. Of course, that introduces the issue of synchronization, not just for new mails, but also for mails deleted outside that client. I'm not sure how TB handles that. What is your modus operandi in Mutt or Pine for this? If I understand correctly, you just make IMAP queries to the server and display what you desire. Hmm, call me lazy, but I would mark such a feature in TB unwelcome. When I click on the INBOX, I expect to see it's contents. That is the semantics I'm used to, when I click on a folder. Is this semantic somehow confronted to the root philosophy of IMAP client/server? Furthermore, if TB is implementing those semantics in IMAP protocol in an undesirable fashion, what should it do to become desirable again? And don't get me wrong, there were a couple of features in TB that made me reach into my dictionary of less than favourable epitets. Like connection caching, which made my installation crash (it could have been my mistake in not applying some patches). For that matter, OE has a similar feature, which when turned on, made my IMAPD processes die (something like include this folder/account in sync on startup or something like that). That looked like a bug in the server, but never mind. I didn't quite understand the need for connection caching on a single IMAP account. It just opened up to 5 connections for each folder I opened. Well, at least it can be turned off. Nix. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
Hi Nikola! Say you have a GUI IMAP client XxX. Say you start it up and click on the INBOX. What would you desire/expect XxX to do? I would expect to see all mail headers in my inbox. Well, you cannot see _all_ of your headers when you have a big mailbox full of mails. You can only see a few of them, perhaps 20, or 50 or so. The client theoretically would be able to clarify, how many emails it can show to the user, depending on the height of the message list window. Then it could just fetch only the headers of the messages, which it can show you now. As soon as a user scrolls, it can fetch the next few headers. My opinion: That's fine in an Webmail Client or in a console based client, where you only can scroll line by line or page by page. I don't know what a graphical client should do, when a user drags the view with the vertical window slider and scrolls up to down to up in between 3 seconds. The client would have to try to get all headers in 3 seconds? Or should he just show n/a to the user until he stops? Very bad idea in my eyes. I often scroll through my whole mailbox searching for a specific date range or similar. I don't want to issue a search query because of that task, that would be very non-economical. So in my opinion: you cannot use this behavior in a graphical client. Well, I have to believe that mulberry can do that somehow, I cannot check that, this program and its vendor are non-existant anymore. But I know: The smaller your bandwith to the server is, the nastier would be the delay you have when you scroll through your folder searching for something. Best, Daniel Nikola Milutinovic schrieb: The first time you open a large IMAP folder is not very fast, I have to admit, but I didn't find any other comparable IMAP client without this problem. Perhaps there are some, but I didn't try them because of the lack of other basic email features. This is why they aren't IMAP clients. IMAP servers make all manner of searching, sorting, retrieval, and storage options completely available to the client, without having to download even all the headers. This is why Mulberry, Pine, Mutt, and Kmail are so much faster. If TBird would just do that instead of insisting on blindly attempting to download all the headers and performing all sorting and searching on the client. TBird and most of the others have their roots and brains seated back in the POP3 dark ages near as I can tell and that's how they treat all mail stores. IMAP allows the clients to easily ask for threaded views (unless you turn the index options off or something like that) from the server, as well as partial sets of headers in batch. This massively speeds things up when you're on a modem, or working with large mailboxes, or mailboxes you only occasionally open. I must say I'm a bit lost here. I have been using and administering Cyrus IMAP for years now, so I'm not really a new kid on the block. Say you have a GUI IMAP client XxX. Say you start it up and click on the INBOX. What would you desire/expect XxX to do? I would expect to see all mail headers in my inbox. So, how do you do that in XxX, other then fetching all mail headers? How do you do that in Pine and Mutt? Now, fetching it EVERY time, well, that would make me switch to another client. So, caching comes in. Of course, that introduces the issue of synchronization, not just for new mails, but also for mails deleted outside that client. I'm not sure how TB handles that. What is your modus operandi in Mutt or Pine for this? If I understand correctly, you just make IMAP queries to the server and display what you desire. Hmm, call me lazy, but I would mark such a feature in TB unwelcome. When I click on the INBOX, I expect to see it's contents. That is the semantics I'm used to, when I click on a folder. Is this semantic somehow confronted to the root philosophy of IMAP client/server? Furthermore, if TB is implementing those semantics in IMAP protocol in an undesirable fashion, what should it do to become desirable again? And don't get me wrong, there were a couple of features in TB that made me reach into my dictionary of less than favourable epitets. Like connection caching, which made my installation crash (it could have been my mistake in not applying some patches). For that matter, OE has a similar feature, which when turned on, made my IMAPD processes die (something like include this folder/account in sync on startup or something like that). That looked like a bug in the server, but never mind. I didn't quite understand the need for connection caching on a single IMAP account. It just opened up to 5 connections for each folder I opened. Well, at least it can be turned off. Nix. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ:
Re: performance issue (imap spool on san)
--On 27. Juli 2006 02:58:15 -0700 Nikola Milutinovic [EMAIL PROTECTED] wrote: Say you have a GUI IMAP client XxX. Say you start it up and click on the INBOX. What would you desire/expect XxX to do? That depends on the situation. I would expect to see all mail headers in my inbox. Really? Even if there are 20,000 messages and you are connected via modem? So, how do you do that in XxX, other then fetching all mail headers? How do you do that in Pine and Mutt? Being console-based, they have it easy: they only load those headers that they are currently displaying. Mulberry lets *you* decide what to do, with separate values for initial download and increment. You can specify limits of headers to get if you want to. When you scroll through the mailbox, it may need to load additional headers. So maybe you have to wait at that point, but the *initial* wait is much shorter. Now, fetching it EVERY time, well, that would make me switch to another client. So, caching comes in. Caching sucks. It's bad on public computers, e.g. in pools, because of privacy issues. In general it's flaky if you work on more than one computer. Mulberry doesn't cache anything and still it's very fast. What is your modus operandi in Mutt or Pine for this? If I understand correctly, you just make IMAP queries to the server and display what you desire. Hmm, call me lazy, but I would mark such a feature in TB unwelcome. When I click on the INBOX, I expect to see it's contents. That is the semantics I'm used to, when I click on a folder. Is this semantic somehow confronted to the root philosophy of IMAP client/server? It's tailored to exactly *one* use scenario: high-speed connection to a fast server. Good luck when that assumption isn't valid! Furthermore, if TB is implementing those semantics in IMAP protocol in an undesirable fashion, what should it do to become desirable again? Huh? -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. Zentrum für angewandte Informatik - Universitätsweiter Service RRZK .:.Universität zu Köln / Cologne University - Tel. +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. pgpTZF6EpfM7k.pgp Description: PGP signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
--On Thursday, July 27, 2006 2:58 -0700 Nikola Milutinovic [EMAIL PROTECTED] wrote: Say you have a GUI IMAP client XxX. Say you start it up and click on the INBOX. What would you desire/expect XxX to do? First I'd like it not to open inbox unless I do click on inbox (or unless I configured it to open inbox). That's a pop holdover. As if, what else would I be running a mail client for? Well, with imap I might be wanting to check some other folder. When opening inbox, a client is usually configured to list new messages. If so it really needs only to fetch headers of new messages. The rest do not matter unless the user wants to scroll back, and even then it might fetch only the next N messages back. Some clients actually open and cache headers of all subscribed folders. That does not scale on a system where users are advised to subscribe to many shared folders. My telnet client could do 'ls -lR /' when I log in, and cache it, but it doesn't :-) Joseph Brennan Columbia University Information Technology Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
My telnet client could do 'ls -lR /' when I log in, and cache it, but it doesn't :-) If your telnet client would need several minutes to show you all files in case you are searching for a file you don't exactly know its name, then you would definitely prefer that it would do so. Best, Daniel Am Donnerstag, 27. Juli 2006 14:45 schrieb Joseph Brennan: --On Thursday, July 27, 2006 2:58 -0700 Nikola Milutinovic [EMAIL PROTECTED] wrote: Say you have a GUI IMAP client XxX. Say you start it up and click on the INBOX. What would you desire/expect XxX to do? First I'd like it not to open inbox unless I do click on inbox (or unless I configured it to open inbox). That's a pop holdover. As if, what else would I be running a mail client for? Well, with imap I might be wanting to check some other folder. When opening inbox, a client is usually configured to list new messages. If so it really needs only to fetch headers of new messages. The rest do not matter unless the user wants to scroll back, and even then it might fetch only the next N messages back. Some clients actually open and cache headers of all subscribed folders. That does not scale on a system where users are advised to subscribe to many shared folders. My telnet client could do 'ls -lR /' when I log in, and cache it, but it doesn't :-) Joseph Brennan Columbia University Information Technology Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
performance issue (imap spool on san)
Hi,
I've installed the latest cyrus release and I'm having trouble with
large mailboxes.
I was going to try if the 4Gig limit is gone and I'm filling up a
mailbox with mails.
If I open the mailbox trough mutt it gets loaded at a acceptable
(lighting fast) speed. However when using thunderbird it gets very
slow, I haven't been able to open the mailbox of 294M.
Attaching strace to the pid I see only the following messages:
open(/mail/mail2/imap/domain/m/mail.ugent.be/r/user/rudy^gevaert2/203750.,
O_RDONLY) = 15
fstat64(15, {st_mode=S_IFREG|0600, st_size=23898, ...}) = 0
mmap2(NULL, 23898, PROT_READ, MAP_SHARED, 15, 0) = 0xb5749000
close(15) = 0
munmap(0xb5749000, 23898) = 0
open(/mail/mail2/imap/domain/m/mail.ugent.be/r/user/rudy^gevaert2/203751.,
O_RDONLY) = 15
fstat64(15, {st_mode=S_IFREG|0600, st_size=23898, ...}) = 0
mmap2(NULL, 23898, PROT_READ, MAP_SHARED, 15, 0) = 0xb5749000
close(15) = 0
munmap(0xb5749000, 23898) = 0
open(/mail/mail2/imap/domain/m/mail.ugent.be/r/user/rudy^gevaert2/203752.,
O_RDONLY) = 15
fstat64(15, {st_mode=S_IFREG|0600, st_size=23898, ...}) = 0
mmap2(NULL, 23898, PROT_READ, MAP_SHARED, 15, 0) = 0xb5749000
The above logs are only an extract of the output. These messages are
going in slow bursts.
The imap spool and configdirectory are on a SAN. How can I further
debug this?
Thanks in advance,
--
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Rudy Gevaert [EMAIL PROTECTED] tel:+32 9 264 4734
Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept.
Groep Systemen Systems group
Universiteit Gent Ghent University
Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
Hi Michael! Thunderbird is NOT an IMAP client. I don't want to start a flamewar, so I tell you that I write this email with a very polite temper in mind. Just to not set you up. But you have to admit, your saying is extremely provocative. Please can you explain your saying? Pine and Mutt both are console only applications and I think you simply cannot compare them to a graphical client like Thunderbird. An unix-unexperienced user will even simply fail to use them at all, because the text interface is so limited by console possibilities, that it cannot be intuitive in any way. In a graphical client you want to be able to scroll through your whole message list without any delay. So I think there is no other chance but caching all header information of a folder. On low bandwith situations it seems to be impossible for me to do this on demand only. I was searching for a full featured graphical IMAP client for a long time and tried everything and I have to say: Not only is thunderbird an IMAP client, it is the best graphical IMAP client I could find in the whole Linux and Windows world. It has the most complete feature set and the most intuitive interface for both Linux and Windows users. And it is way faster than every IMAP client with a comparable feature set. Kmail is extremely near to my wishes, it only lacks IMAP IDLE support. The first time you open a large IMAP folder is not very fast, I have to admit, but I didn't find any other comparable IMAP client without this problem. Perhaps there are some, but I didn't try them because of the lack of other basic email features. Anyway: I'd happily listen to other suggestions for full featured graphical IMAP clients which could be better than thunderbird. There surely are things in thunderbird which could be a lot better! I just need an alternative which I was not able to find yet. So regarding the performance problem of the original poster: When I'm on my LAN, I can initially open a folder with 6000 mails in it in about 30 seconds. I have a network throughput of about 900 KB/s. So network just isn't the bottleneck. (see below) After Thunderbird has this initial header download done, the folder opens without any delay. It just downloads headers of new emails, so if you don't get thousands of mails between two folder checks, there is no problem at all. I think that your storage backend might get into iowait issues. I had a much worse issue as you are describing. I had that with an extraordinary hardware (Dual Xeon and SCSI RAID-5 direct attached with an 64-bit ICP host raid adapter) but the wrong file system. With ext3 I had incredible iowait whenever a user's client downloaded all headers of a big folder. And after some seconds of high iowait, the machine even freezed up for several (up to 30!) seconds while having a lot harddisk write access. Load in this periods raised up to over 35!! Then with a much smaller temporary machine (single P4 with a SATA Hardware Raid-1 mirror) and reiserfs I had no lockups anymore, but still high iowait when a user downloaded all headers of a big folder and load increases to nearly 20. The download started rather fast but as iowait (and load) increased, it slowed down and all users got slow performance. But luckily the machine did not freeze anymore... Way better than before even though this hardware was a not nearly as good as the one before. Now I switched back to the first hardware platform (SCSI RAID-5) but now running on XFS. Now I have no iowait and performance issues anymore. Load is ridiculous low, most time below 0,1. IOwait is mostly zero and sometimes increases up to 3% and peaks to 5% for only one or two seconds. So I'd say you should check the load and iowait of your mail server when you try to open a big folder with thunderbird. I'd bet both is increasing fast and far until the machine nearly stalls and so you cannot get the whole folder. It's very good that you can prevent the problem with kmail. This shows that Thunderbird really has problems here, but you have to think of normal users. On the user's side there is a lot of Outlook, Outlook Express and Thunderbird which all will trigger the problem. I think it will be better to eliminate this problem instead of working around. You might find yourself in a situation where the server is in high usage by your users and it's very hard to replace and migrate mailboxes without long downtimes. Best, Daniel Michael Loftis schrieb: --On July 26, 2006 12:02:41 PM +0200 Rudy Gevaert [EMAIL PROTECTED] wrote: Hi, I've installed the latest cyrus release and I'm having trouble with large mailboxes. I was going to try if the 4Gig limit is gone and I'm filling up a mailbox with mails. If I open the mailbox trough mutt it gets loaded at a acceptable (lighting fast) speed. However when using thunderbird it gets very slow, I haven't been able to open the mailbox of 294M. This is a thunderbird problem, not a Cyrus problem. Thunderbird is NOT an
Re: performance issue (imap spool on san)
Hi David! All your points are fully correct. But changes nothing to my basic saying: It's plain wrong to say thunderbird is NOT an IMAP client. It might be a very bad IMAP client regarding this one feature, but it IS an IMAP client. It's fully RFC compliant, it does not matter if it uses all IMAP features or not. And for nearly all users, only graphical clients are usable at all. So for nearly all users, Thunderbird is the best availiable IMAP client. Only some very few console based IMAP clients are better in at least one feature: Fetching headers on demand. I just want to have this stated. To be provocative and exaggerating just isn't netiquette and if the said statement is plain wrong, it's really bad and has to be corrected. Even lynx is a web browser, although it cannot do some very basic things everyone expects a web browser to be capable of. So I hope everyone can agree with my statement and this subject can be closed now. Best, Daniel David Lang schrieb: On Wed, 26 Jul 2006, Daniel Eckl wrote: Hi Michael! Thunderbird is NOT an IMAP client. I don't want to start a flamewar, so I tell you that I write this email with a very polite temper in mind. Just to not set you up. But you have to admit, your saying is extremely provocative. Please can you explain your saying? Pine and Mutt both are console only applications and I think you simply cannot compare them to a graphical client like Thunderbird. An unix-unexperienced user will even simply fail to use them at all, because the text interface is so limited by console possibilities, that it cannot be intuitive in any way. In a graphical client you want to be able to scroll through your whole message list without any delay. So I think there is no other chance but caching all header information of a folder. On low bandwith situations it seems to be impossible for me to do this on demand only. it has nothing to do with being graphical or not. the key is how it manages the mail. a 'true' imap client realizes that the server holds all the info and it can query individual pieces as needed (headers, flags, including sorting, searching, threading, etc) a pop3 client that knows some IMAP uses the IMAP commands to fetch the data, but it stores it locally, just like it stores pop3 messages, and all actions (including searching, threading, etc) take place against the local copy of the messages. since pop3 is such a simple protocol many mail clients start with support for it and then add IMAP support by substatuting IMAP commands for the POP3 commands, but otherwise operates exactly the same. This is what Thunderbird does. unfortunantly there isn't a true IMAP client that's graphical available to my knowledge (which is why I still use pine) I was searching for a full featured graphical IMAP client for a long time and tried everything and I have to say: Not only is thunderbird an IMAP client, it is the best graphical IMAP client I could find in the whole Linux and Windows world. It has the most complete feature set and the most intuitive interface for both Linux and Windows users. And it is way faster than every IMAP client with a comparable feature set. Kmail is extremely near to my wishes, it only lacks IMAP IDLE support. there are a lot of IMAP features that it doesn't use, not just IDLE. The first time you open a large IMAP folder is not very fast, I have to admit, but I didn't find any other comparable IMAP client without this problem. Perhaps there are some, but I didn't try them because of the lack of other basic email features. with pine it takes the same time for me to open a folder with 10 messages in it as one with 10,000 messages in it, becouse it doesn't care how many messages are there and only fetches what it needs to, when it needs to. it does take longer to sort or thread a large mailbox, but that's server-side processing, which can be greatly assisted by caches, etc on the server (depending on your IMAP server software) Anyway: I'd happily listen to other suggestions for full featured graphical IMAP clients which could be better than thunderbird. There surely are things in thunderbird which could be a lot better! I just need an alternative which I was not able to find yet. nobody is saying that there's an available graphical IMAP client that's better then thunderbird, they are saying that there isn't a real graphical IMAP client period. all of them that are available, including thunderbird, are not makeing proper use of IMAP, and as a result they all suffer the same performance problems when you open a folder with lots of new messages. So regarding the performance problem of the original poster: When I'm on my LAN, I can initially open a folder with 6000 mails in it in about 30 seconds. I have a network throughput of about 900 KB/s. So network just isn't the bottleneck. (see below) After Thunderbird has this initial header download done, the
Re: performance issue (imap spool on san)
Hi, let me start by saying that I'm basically always on high-speed Internet connections and keep my mailboxes relatively small. So I haven't had bad experiences with Thunderbird in that regard. I still don't use it, but that's not the issue here. -- Daniel Eckl [EMAIL PROTECTED] is rumored to have mumbled on 26. Juli 2006 21:31:40 +0200 regarding Re: performance issue (imap spool on san): In a graphical client you want to be able to scroll through your whole message list without any delay. So I think there is no other chance but caching all header information of a folder. On low bandwith situations it seems to be impossible for me to do this on demand only. Well, you don't know Mulberry then. It has a GUI and is very adaptable to any kind of network situation and it doesn't cache anything! You do have to tweak it, however. It doesn't auto-configure, but there are *many* settings to play around with. Obviously that's both a blessing and a curse. I was searching for a full featured graphical IMAP client for a long time and tried everything and I have to say: Not only is thunderbird an IMAP client, it is the best graphical IMAP client I could find in the whole Linux and Windows world. Unfortunately that's not saying much ... it's just too bad that the only decent IMAP GUI client isn't developed further. I have hopes for Thunderbird 2, but I have a feeling that I will continue to use Mulberry for a long time ... -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum für angewandte Informatik - Universitätsweiter Service RRZK Universität zu Köln / Cologne University - Tel. +49-221-478-5587 pgpReV1h3xAs4.pgp Description: PGP signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
Hi Sebastian! I just can (and so I will) believe your statements about mulberry, because it's dead and my users and I cannot get or use it legally anywhere. So it has disqualified itself. Sadly, I want to add. I heared a lot good things about this one, and I could imagine that this one could have become my alternative to thunderbird. Best, Daniel Sebastian Hagedorn schrieb: Hi, let me start by saying that I'm basically always on high-speed Internet connections and keep my mailboxes relatively small. So I haven't had bad experiences with Thunderbird in that regard. I still don't use it, but that's not the issue here. -- Daniel Eckl [EMAIL PROTECTED] is rumored to have mumbled on 26. Juli 2006 21:31:40 +0200 regarding Re: performance issue (imap spool on san): In a graphical client you want to be able to scroll through your whole message list without any delay. So I think there is no other chance but caching all header information of a folder. On low bandwith situations it seems to be impossible for me to do this on demand only. Well, you don't know Mulberry then. It has a GUI and is very adaptable to any kind of network situation and it doesn't cache anything! You do have to tweak it, however. It doesn't auto-configure, but there are *many* settings to play around with. Obviously that's both a blessing and a curse. I was searching for a full featured graphical IMAP client for a long time and tried everything and I have to say: Not only is thunderbird an IMAP client, it is the best graphical IMAP client I could find in the whole Linux and Windows world. Unfortunately that's not saying much ... it's just too bad that the only decent IMAP GUI client isn't developed further. I have hopes for Thunderbird 2, but I have a feeling that I will continue to use Mulberry for a long time ... -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum für angewandte Informatik - Universitätsweiter Service RRZK Universität zu Köln / Cologne University - Tel. +49-221-478-5587 Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
--On July 26, 2006 9:31:40 PM +0200 Daniel Eckl [EMAIL PROTECTED] wrote: Hi Michael! Thunderbird is NOT an IMAP client. ... The first time you open a large IMAP folder is not very fast, I have to admit, but I didn't find any other comparable IMAP client without this problem. Perhaps there are some, but I didn't try them because of the lack of other basic email features. This is why they aren't IMAP clients. IMAP servers make all manner of searching, sorting, retrieval, and storage options completely available to the client, without having to download even all the headers. This is why Mulberry, Pine, Mutt, and Kmail are so much faster. If TBird would just do that instead of insisting on blindly attempting to download all the headers and performing all sorting and searching on the client. TBird and most of the others have their roots and brains seated back in the POP3 dark ages near as I can tell and that's how they treat all mail stores. IMAP allows the clients to easily ask for threaded views (unless you turn the index options off or something like that) from the server, as well as partial sets of headers in batch. This massively speeds things up when you're on a modem, or working with large mailboxes, or mailboxes you only occasionally open. I'm not trying to start a flamewar either, I'm stating the observed behavior. They're not IMAP clients. They speak IMAP but they make no real use of the protocol. I really do wish there were more better clients out there, there aren't. I totally agree with you there that Pine and Mutt are not a replacement for a GUI client. I've never used Kmail extensively though. Anyway: I'd happily listen to other suggestions for full featured graphical IMAP clients which could be better than thunderbird. There surely are things in thunderbird which could be a lot better! I just need an alternative which I was not able to find yet. I haven't found one other than Mulberry either. It seems developers widely assume you're not on a modem anymore, which for me is all to often not the case. It's faster for me to use SquirrelMail, IMP, or Horde than to use TBird when I don't have access to Mulberry. The size in MB of the folder has little to do with IMAP client speed, it's mostly the number of files. Older versions of EXT3 (before they added directory hashing support) had pretty terrible performance in this regard. I don't use Ext3 much of anywhere anymore but I know there are some documents on how to enable that in l later model Linux kernels. It may or may not help your mail spool performance. It's doubtful TBird/etc will ever load a mailbox with 20-30k+ messages in a very fast manner on first open unless they start to implement and make use of the IMAP extensions for partial loading combined with a local header cache as the view is scrolled. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
