Re: [Imap-uw] Re: moving mail between folders is intermittently failing
On Thu, 6 May 2010, Timo Sirainen wrote: I think that's a different issue. They're unhappy when an idling device gets woken up (constantly). The Hang in there messages are sent only when client has requested some command that takes 15 seconds. Most of the users/clients never see those messages at all. You're right. It isn't quite the same, and mobile devices are less likely to run afoul of server head-pats every 15 seconds during long-running commands. But there still are issues, even if the device is already quite awake. On mobile devices where you pay per packet (rather than per KB or MB), head-pats add packets on top of IMAP's already excessive chattiness. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: moving mail between folders is intermittently failing
On Thu, 6 May 2010, Dan White wrote: And what do you do if the server takes longer that you think it should to respond to a query? Do you assume that it's a networking issue or a slow server? Crapware assumes that it is a network issue that somehow is utterly irrecoverable in TCP, yet magically goes away if you tear down the TCP connection and establish a new one. Guess what happens when thousands of pieces of crapware are all doing the same thing at the same time. It gets ever more entertaining as additional crapware (with longer timeouts but not long enough) join in the festivities. Typically, the original event that triggered the problem has long since resolved itself. It's very much like a minor traffic slowdown that escalates into a traffic jam that escalates into a mass freeway collision. And no matter how many times you attempt to educate people about maintaining a safe following distance and safe lane changes, they still do the same bad things. Treating an IMAP *session* like a stateless http request is doomed to repeat history. RFC 2683 covers some of this ground. Yup. The stateless religion was absolute dogma in CS classes starting in the late 1970s/early 1980s. As a result, many of the young'uns simply have no clue how to think otherwise. Yet, over and over again, the same young'uns end up beating their heads against a wall in an attempt to re-implement state. When they talk about the problem they are trying to solve, they confuse it with cache. When a young'un talks about keeping the cache synchronized, I listen carefully. More times than not, s/he's trying to keep state but doesn't know how to it. The whole basis of the stateless dogma 30 years ago was the belief that it is less inefficent to jump through hoops to attain state in a stateless world than to maintain a stateful world if you didn't care about state. I first became aware of it with PARC's Woodstock File System which was the ideological predecessor of NFS. The WFS paper became the manifesto of the stateless ideology; yet only a few read it and even fewer noticed that it rather coyly defined out of the problem space all the cases in which statelessness did less well. Basically, statelessness requires accepting the following on faith: . State is unimportant. . State is expensive to acquire and maintain. . If state is important, it can be easily and cheaply acquired on top of a stateless infrastructure. . If state can not be easily and cheaply acquired in a stateless infrastructure, you can get the equivalent effect easily and cheaply through synchronization. . Synchronization is magically atomic, so you don't have to worry about the fact that it is stateless. . If synchronization is not atomic, and things change in the middle of the synchronization, it is alright since you'll notice the issue the next time you synchronize. There is more to the faith of statelessness, but this ought to be enough to see the overall picture. The bottom line is that IMAP is a stateful, not a stateless, protocol; and that treating an IMAP session like a stateless HTTP request is doomed to failure. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] moving mail between folders is intermittently failing
On Thu, 6 May 2010, Oswald Buddenhagen wrote: you may rail about the stupidity of those developers as much as you want. apparently unlike you, they live in the real world where the only guarantee which tcp provides is that the data stream is intact - *if* it arrives. Mr. Newton, your notions about gravity are of no use to those of us in the real world where heavier objects fall faster than lighter objects. An earmarks of a sham argument is we live in the real world. timeouts as a workaround for shortcomings of the tcp/ip stack under real-world conditions are a perfectly reasonable approach, Tailgating and cutting across multiple lanes of traffic as a workaround for the shortcomings of highways under real-world conditions are a perfectly reasonable approach. It is important to grasp that an action that may seem to be beneficial in empirical testing may in fact be quite harmful, not just to other agents but also ultimately to oneself; and furthermore that the benefits are more illusory than real. In the highway example, there are agents called police officers whose function includes issuing traffic tickets, thus inflicting lesser harm (a fine and points on one's driving license) as a pedagogical attempt to avoid the greater harm of a mass collision. The benefit -- and demerit -- of open standards is that there are no police officers. Open standards depend upon voluntary compliance. Closed standards, through their patents and licenses, can (and do!) enforce compliance. It's also important to grasp that what one things are shortcomings may in fact be there for a reason; and that when you do something bad to workaround a shortcoming you may be sabotaging something important. just like keep-alive signals in higher-level stateful protocols to prevent timeouts. The difference is that in higher level stateful protocols, both the timeouts and the keepalives are defined in the specification, along with the rules that both must follow. The problem is with ad-hoc actions based upon incomplete or misunderstood empirical evidence. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: moving mail between folders is intermittently failing
On Fri, 7 May 2010, Oswald Buddenhagen wrote: loss of the state we are talking about here. it's all based on mark's postulation that a tcp connection is reliable. TCP connections are reliable. Run, don't walk, to your nearest technical bookstore and read about network layering. Crappy software implementations may be unreliable. In my own review of several IMAP RFCs, it's clear that connection problems have been anticipated and several options have been standardized, i suggest you verify the chronology of events. and compare the names on particular rfcs. Ah, sophomores who read a little and think that they understand all. If you actually read through the history of IMAP RFCs, you would have read RFC 1733 and learned the purpose of IMAP synchronization. You would have also noticed when UIDs were introduced, and by whom. Next, you would have learned the purpose of CONDSTORE. But it's so much easier to jump to conclusions. Before you mention QRESYNC, you should first see if anyone actually uses it. The mobile device world barely stifled a yawn over the entire LEMONADE effort, and no major commerical implementations are doing anything about it. It turned out, as predicted years earlier, to be a solution in search of a problem. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: moving mail between folders is intermittently failing
On Thu, 6 May 2010, Brian Hayden wrote: TCP connections are more reliable than UDP; that does not mean they are reliable, full stop. You are using the wrong definition of reliable. Reliable does not mean does not fail. UDP has no provision for reliability, ordering or data integrity. TCP does all of these. Regardless of what happens in the link layer, TCP delivers a sequential octet stream without duplication or missing data. The only thing that terminates that stream is a session disconnect. There is no such thing as failure in TCP. What you think of as being failure are all application layer concept: [1] The application received a session disconnect (FIN) from TCP. This is completely an application concept; TCP considers this to be a completely normal shutdown of the session. [2] The application received a session reset (RST) from TCP. This indicates that the application attempted to communicate with a TCP peer that does not exist. This is what most people (mistakenly) call a TCP failure. [3] The application unilaterally decides that a failure has occurred. Now, [1] and [2] generally indicate the demise of the peer, with [1] being the normal and expected result of a mutually-agreed upon demise. [2] is not supposed to happen with debugged implementations, except when a link-level disconnect outlasts a FIN-wait. But neither of these are what the discussion is about. That is [3]: TCP connections often do either fail completely or stall for so long as to constitute a failure Now we come into myth vs. reality. Many people have observed that web browsers seem to fail or stall, and that hitting the refresh button seems to fix it. Because the web browser uses HTTP over TCP, they falsely conclude that this is an attribute of TCP and thus the equivalent of the refresh button is appropriate for other protocols. This conclusion is completely and utterly false; and is where stateful vs. stateless comes in. A web browser typically has multiple HTTP sessions in progress, each one of which is charged with resolving a different URI as these are encountered in the page. The whole idea is not to serialize the rendering of the web page; the fate of a JPEG being loaded is independent of any other piece. A web server, in turn, is obliged to turn around requests as quickly as possible. It poots data to the session and expects steady progress; otherwise it abandons the session. The browser is somewhat more patient but it too abandons the session if steady progress is not forthcoming. Now comes the important part: The server routinely abandons sessions, or refuses to initiate them, for load based reasons. This is alright, because HTTP is stateless and drops are expected in HTTP. There is no reason to believe that immediate retry will not succeed. IMAP, on the other hand, is a stateful protocol. A server does not drop IMAP sessions except under specification defined conditions: [a] The server received a TCP-level FIN or RST from the client. [b] Negotiated session disconnect (LOGOUT command). [c] Server crash. [d] 30 minute client inactivity timeout. If your IMAP connection stalls, there is no reason to believe that disconnecting, creating a new connection, and retrying the operation will in any way help the situation. At best, it is a waste of effort; you destroyed your session state that now has to be rebuilt to get you back to where you were before. More typically, it is futile; the underlying problem impacts your new session just as your old session was impacted. As in the best case, you wasted effort; and now are worse off because you gave up all the data in your state which you could have used. In the worst case, it is harmful; not only is it futile, but it has also made a bad situation (e.g., server overload) worse. But, but, you protest, what if some router went out and came back up? TCP recovers from that; or would recover if you let it. Please read up on the subject of TCP retransmissions and their algorithms including backoffs. These old guys who came up with TCP 30+ years ago knew what they were doing. Now, you may feel that the standard for TCP retransmission algorithms may need adjusting to reflect modern-day networking. You may be surprised to learn that I agree; and that the backoffs to avoid swamping the 56KB links on ARPAnet need updating for modern Ethernet and wireless link layers. So hop to it. Get involved with the standards development process. Do the experiments to work out what are suitable retransmission and backoff algorithms in the modern world. RFC 2988 is nearly 10 years old; and in particular sections 2.4 and 2.5 are probably completely obsolete and should be changed to something quite different. Don't duplicate TCP's functionality in the application layer (in a FAR less efficient and effective manner). Simple solutions to complex problems backfire. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for
Re: [Imap-uw] Re: moving mail between folders is intermittently failing
On Thu, 6 May 2010, Brian Hayden wrote: Reliable does not mean does not fail. Coincidentally, nobody said it did. Interesting! If that was not your meaning in claiming that TCP is not reliable, then you don't know what you are talking about. TCP is most certainly reliable. This is another fun historical dissertation, at whose core is: change the RFCs, and until then maintain some righteous anger. I go to the trouble to teach you how things actually work, and you respond with a typical nihilistic Gen-X retort. The righteous thing is to follow the specifications; and if you think that the specifications are incorrect then work to get them changed. You're the one who seems to be angrily insisting that the specifications shouldn't be followed. And then to make stupid statements such as TCP is not reliable. That is quite a sleight of hand there. It makes your pats on the heads of the young'ns look even sillier. You've oversimplified [2[ to the point where it edges from oversimplified to misleading. Pshaw. So you want to bring up Linux's half-duplex close behavior, eh? That's irrelevant to RST in IMAP sessions. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: moving mail between folders is intermittently failing
On Wed, 5 May 2010, Paul Vixie wrote: of course. but my primary mail interface is emacs mh-e and i'm not going to abandon it, nor the many filters and cronjobs and tools i've based on MH, just to support my secondary need to open attachment-containing messages in an IMAP client. i fully understand that i do not represent a growing segment of the mail market. as before, i'm thankful that uw-imap supports MH at all. Have you thought about making mh be an IMAP client? You might need to have some sort of proxy daemon to keep state, since IIRC mh is actually a set of programs invoked from the shell. I don't know if your other tools use the mh programs, or if they separately know about the mh layout. With mh-e, you ought to be able to make it babble IMAP protocol and keep your interface. Doing that would make it possible to move your mail to any IMAP provider; which you may not want to do but at least you have the option. whatever sold the most iron was the corporate mantra of that moment. the people who say just use Exchange today are cut from that same cloth. Yeah. There's a lot of that! ;) i may try to add MH support to Dovecot so that i won't have to maintain a fork of the uw-imap abandonware nor use a non-open codebase (Panda). Another alternative is to join the re-alpine project on sourceforge. UW IMAP is part of re-alpine, so technically there already is an open codebase fork. I don't think that the re-alpine people have done much, if anything, in the IMAP part; so they would welcome your contributions. I haven't decided about opening Panda IMAP. The issue is, as it always has been, funding to support any work other than my own personal use. There's only one task remaining that I personally need in Panda IMAP; anything else that I do is at someone else's request. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: moving mail between folders is intermittently failing
On Thu, 6 May 2010, Yiorgos Adamopoulos wrote: That is why people like you and me can donate to Mark (I did last week) in order for him to continue working on stuff that makes our systems tick. Enough people can form a Panda-IMAP empire :) And thank you! The donations do help keep Panda IMAP alive, and keeps me still on this mailing list; I would have dropped out a while ago otherwise. And I am, slowly, chugging away at some feature additions (most notably fast mix delivery). Sadly, though, it's a long way from forming an empire... ;) -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] moving mail between folders is intermittently failing
On Wed, 5 May 2010, UCTC Sysadmin wrote: I don't know what other formats are available, moreover the mbox format I thought was actually the canonical format of a file containing multiple email messages and is used as well by POP. It depends upon which POP server you use. The POP server in the IMAP package (UW or Panda) uses the same internal library as the IMAP server, and thus supports all the alternative formats. There's a file called formats.txt in the documentation that talks a little bit about it. After reading about the Cyrus IMAP server I decided not to allow it to determine to me what I could and couldn't do without it; uw-imap plays with others. UW and Panda are probably the only servers that do this strictly, although Dovecot comes a lot closers than most. Just about every IMAP server wants to use its own preferred format which is never mbox format. Not even UW. The default format is mbox, but that never was preferred. If they can understand the difference between TCP and UDP, they can understand state. I wouldn't be so certain. If you look at a lot of applications these days, they blat out a query, and read an answer. If they don't get an answer in what they consider to be a suitable time, they drop the connection and try again. Put another way, they treat TCP just like UDP, only with this strange and annoying connection stuff that they don't understand why it's there. Ironically, the original design for IMAP was for it to be UDP based. I was talked into making it be stateful and TCP based. Well, with Thunderbird at least the avenue is (still?) open to muck with the source. Perhaps when I've nothing else to do, try to rewrite their IMAP support to behave properly and send them the code, or (ha ha) offer my IMAP client as a plugin for Thunderbird (ha ha.) I would, if I were paid to do so. The problem is, it doesn't seem as if there is any market for email clients (and hence funding) these days. I'm averse to storing user mail in a noncompatible format (meaning the POP server can't play) but may have to, for these users at least (I'll put them in their own sandbox.) If POP supports the mix format, then that objection vanishes. Which POP server? If it's qpopper, then it probably does not support mix. If it's ipop3d (bundled with imapd), then it does. Argh, obviously the question becomes how does one convert existing mail folders to mix format ... or is there a magic cookie such that the mix handler knows to bow out, and or is there any automatic (blind) way that upon first use the program can do a conversion in the background? All I'd need is a filter to convert mbox to mix, and stop the users getting in until that is done. There's a program called mixcvt that will do that. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: moving mail between folders is intermittently failing
On Wed, 5 May 2010, Paul Vixie wrote: i must have spoken improperly. if i say scan and learn thereby about messages 1,3,5,20 and then one month later after every computer has been power cycled four times i say show 5 i want the same message. Oh. In that case, what you want are UIDs. imap's statefulness isn't nearly persistent enough for me. Actually, it would be if the mh code could implement UIDs correctly. The problem is that the the mh code uses the filename numbers as the UID; but then has to account for the mh compact command, which renumbers all the files. If you never use the compact command, then IMAP UIDs would be just what you need. Otherwise, you need to have some other means to tie a permanent UID to a particular message, while preserving the IMAP requirement of being strictly ascending in the mailbox. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: moving mail between folders is intermittently failing
On Tue, 4 May 2010, Timo Sirainen wrote: This mtime flushing actually works pretty easily in all modern OSes: just open and close the file and then stat/fstat. Yup. You just said it: modern OSes... I remember an OS which would not update mtime if ANY local agent had the file open. So, no matter how many times you did an open/close/stat, you would get the same out of data data. That, interacting with NFS attribute caching, made things quite painful. Perhaps this is now just a sad memory and no longer needs to be worried about. Yeah, I actually also fallback to MD5 of a few specific headers if X-UID: headers haven't been written to disk yet. That may work as long as Received: headers are included. Also, these days, MD5 is not patent encumbered nor is it under any export restrictions. That wasn't the case back then... UW IMAP has no such thing as X-UID headers haven't been written to disk yet for existing messages. That state only exists with new mail, and the first thing UW IMAP does is write those X-UID headers. Safer, but slower. The annoying thing with Dovecot's mbox optimizations is that they're pretty complex and I'm sure there are bugs there. Also it's difficult to sometimes figure out if something is a bug or just a side effect of some other software modifying the mbox, possibly with incompatible locking rules, etc.. Yeah, and when I was supporting 80,000 people using that format over NFS I did not want to take that risk. So I'm kind of hoping people would stop using mbox. :) You and me both! ;) -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Implementing mix on FreeBSD
On Sun, 2 May 2010, Chris Boyd wrote: Anyone got pointers to some good guideline or docs for implementing mix mailboxes on FreeBSD? I've been googling about a bit, but can't seem to find the key part for getting the Sendmail LDA set up correctly. The solution seems to be using procmail, but I'm not (and don't really want to become) a procmail hacker. There's no need for procmail. The old UW IMAP FAQ hasn't been taken down yet. Read this topic: http://www.washington.edu/imap/IMAP-FAQs/index.html#4.5 Substitute mix for mbx and it's mostly still accurate. The step for sendmail is the very last step, and the answer is tmail, not procmail. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: moving mail between folders is intermittently failing
Dovecot is a good server. It is one of only two (the other being Panda IMAP) that fully passes IMAP compliance testing: http://imapwiki.org/ImapTest/ServerStatus [UW IMAP flunks two of the tests...it hasn't been updated in 2 years.] The main concern that I have with using Dovecot for traditional UNIX mailbox files (mbox) is that Dovecot gives up some of the aggressive compatibility with ancient/stupid mbox practices for performance. UW (and Panda) try damn hard to be compatible with even the most ancient stupid things that people do with mbox files; and take a considerable performance hit for doing so. UW and Panda assume the worst about mbox. It assumes that NFS is probably in the picture, that you may well be farting around in some ancient 1980s mbox tool at the same time that the IMAP server is trying to do something; and thus it has to go through extreme checks to make sure that your mbox file doesn't get trashed. This aggressive support for worst case was there for a reason. That worst case actually existed once upon a time. I hope that it is forever extinct, but people tend to do crazy things... Oh well, mankind will probably survive even though it refuses to take my advice... :) The issues to be aware of in Dovecot are: [1] Access to the mbox files via NFS; a true idiocy but nonetheless one that UW itself insisted upon doing for years (over my repeated and vigorous objections). I don't think that Dovecot tries to make an NFS back end work right. I wouldn't blame him for not doing so; most of the UW IMAP performance slowdown with mbox files is code to make NFS work (for a half-assed sort of definition of work). [2] Dovecot runs multi-threaded (which itself requires OS support), and the threads exchange state information. Among other things, this allows significant performance benefits and multiple read-write access to the same mbox format mailbox...as long as Dovecot is the only consumer of the mbox file. Once again, UW IMAP did not have the luxury of being able to assume that. The nice thing about mix format is that there was no need to be compatible with ancient idiocies; and as a result mix is so much faster. Even without the threading, mix is probably faster than Dovecot on mbox because even Dovecot has to do some mbox operations the hard way. Nonetheless, if you really need mbox format, and are sure that you won't be running dinoware and/or doing stupid things like access via NFS, then Dovecot is definitely an option to consuder. If you want to use maildir format, I would go further and say that Dovecot is the ONLY choice; do not use an unsupported third-party driver in UW and especially do not use Courier. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: moving mail between folders is intermittently failing
On Sun, 2 May 2010, Linda Walsh wrote: Seems to be fine with me using the 'mbox.lock' locking files to gain exclusive access. I believe was a compatibility setting somewhere. The .lock file is a delivery lock; to prevent more than one agent from writing (= appending) to the mbox at the same time. It doesn't synchronize between agents which hold state on the mbox. The main issue is if any other mail reading program is consuming the mbox. If Dovecot is the only consumer you will be OK. But if you have other consumers (including Pine, Alpine, elm, /usr/ucb/mail, UW IMAP, etc.) accessing the mbox while Dovecot is doing its thing there may be a problem. I think the current version of dovecot does suport 'mix' (folders and messages in same?), but I didn't test it -- didn't want to screw up my working mail store. Maybe I'll eventually feel braver, out of curiosity. I wasn't aware of Dovecot supporting mix. As far as I know, Dovecot only supports maildir (its preferred format) and mbox. Doesn't Cyrus use maildir format, or is Cyrus=Courier? No to both. Cyrus format is a completely different format, with more in common with netnews than maildir. What may have confused you is that both Cyrus and maildir put each message into a separate file. However, Cyrus does extra stuff to make it scale a bit better. Maildir, in turn, does extra stuff to be NFS-safe at the cost of not being at all ameniable for IMAP. Dovecot actually implements a modified version of maildir which is not NFS-safe... One dir, many little files? just seemed likea mess to me. But my 80+ active mailboxes might seem a mess to some. Some people have many more mailboxes than that. No reason to NFS -- the IMAP server should be where the source files were and use it to mitigate access...using NFS and IMAP... two means to access same read/write share would almost inevitably lead to a mess. Well, then, you are more sensible than a great many people! ;) I'm still trying synchronize everything between smb and local views of regular files, and end up with observable quirks. Hey, if you really want fun and laughter, try synchronizing SMB, NFS, and local files. Simultaneously! ;) One of the reasons that drew me to Dovecot was that my OS does support threads, so I wanted to use use things that provide multi-thread usage to better parallelize my workload -- it's the only way I'll ever do a better job of processor utilization. I think that you may have mistaken what Dovecot's multi-threading does. The multi-threading allows multiple simutaneous read/write access to an mbox format mailbox, as long as Dovecot is the only consumer of the mbox file (and you don't want to violate that assumption). It does this by exchanging semaphores between the threads, which run in the same process; otherwise there are no such semaphore with mbox format. You get the same level of service in UW IMAP and Panda IMAP using mix format. mix has its own equivalent semaphores and does not need to be multi-threaded. My new IMAP server at Messaging Architects is in fact also multi-threaded, but it doesn't need the threading for semaphore exchange. It uses an expanded form of mix that has metadata and stubbing (which I call virtual mailboxes and am quite happy with/proud of). Right now, we're just using the stubbing for user quarantines, in which the per-user quarantine mailbox has stubbing pointers into the global quarantine which contains the actual messages. The other extension in mix is that it is clustered(!). Thanks for the appraisal -- makes me feel like I wasn't crazy for moving the direction I did, given my hardware/software setup. Yes, Dovecot is a reasonable server; and as I said in a previous message Dovecot and Panda IMAP are the only two servers which are tested to be fully compliant. I haven't yet had my new MA server tested yet, mostly because there are some known issues in the underlying storage architecture that need to be resolved first. I expect that it will eventually test fully-compliant as well. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] moving mail between folders is intermittently failing
Comments on your message: [1] There is no such operation as move. What you think is a move is actually implemented by two separate operations: copy from source to destination delete from source and optionally a third: expunge (permanently remove deleted messages from) source [2] Your report of sometimes the move is partly honors in that the item appears in the target but is not deleted from the source is explained as the first operation completing successfully, but not the second operation. [3] Your report of sometimes the move is not honored at all is explained as the first operation never taking place. [4] UW IMAP uses /tmp, not /usr/tmp. [5] It sounds like you are using traditional UNIX mailbox format, a.k.a. mbox format; a flat-file containing all the messages, one after another, with each message preceded with a line starting with From . If this is the case: traditional UNIX mailbox format was designed in a time when a very large mailbox was between 100 KB and 800 KB in size. It does not scale well to sizes of 100 MB to 800 MB. In particular, doing things with mailboxes in the hundreds of MB in that format takes a while. The authors of Outlook and Thunderbird are victims of a computer science course mindset which, starting in the 1980s, taught their pupils that all protocols are (or should be) stateless. Thus, they believe that IMAP is like HTTP; that when a server fails to respond immediately, that means that the correct remedial action is to disconnect and try again, or just disconnect and assume that everything happened anyway. These developers simply can not grasp the concept of a stateful protocol like IMAP (HTTP is stateless) in which certain operations (such as manipulations of an 800MB flat file) might take a while; and that being a stateful protocol IMAP is guaranteed to respond if you wait long enough. Unfortunately, years of repeated attempts to educate these developers proved to be utterly futile. When you talk about state, you get vacant stares. Thus, there is no hope of a working version of crapware such as Outlook or Thunderbird. The only thing that you can do is see what you can do on the server to not unduly stress the limitations of the crapware. One way to do accomplish this is to switch mailbox formats to mix format. I can't imagine how anyone can tolerate traditional UNIX mailbox format with a 100 MB mailbox, much less larger. mix format was designed to accomodate such large mailboxes; and does in a matter of a second or two what can take minutes with traditional UNIX mailbox format. [6] What you describe is typical of scaling problems. It seems to work until you reach a critical point. Very likely, the mailboxes got bit enough that operations now tend to take long enough for Thunderbird to decide to give up. [7] Both Thunderbird and Outlook have a way to record protocol negotiations. The c-client library also has a mechanism to record protocol negotiations, at the client (not server) level. You want to do client-based telemetry, not server-based, due to the size of the protocol logs. IMAP protocol logs are HUGE, and on anything beyond a small test system you will quickly be buried under the weight of the logs. On a small test system, you can easily set up a script for server logs using tee, at the cost of giving up SSL capability. You quickly find out that the server logs are much less useful than you thought. I went through this exercise on my new IMAP server. On Fri, 30 Apr 2010, support wrote: We have users whose folders are between 100 MB and 800 MB in size. Most of those users are using Outlook but some are using Thunderbird. Lately (and seemingly suddenly) the users are encountering trouble in that they will move one or more items from the inbox to another folder and get inconsistent results: * sometimes the move is honored, with the item appearing in the target folder and disappearing from the source folder. * sometimes the move is partly honored in that the item appears in the target folder, but is NOT deleted from the source folder. * and sometimes the move is not honored at all, despite the client email program thinking for a moment that it was (before seeing in moments after that it wasn't.) The server is FreeBSD 5.4 using imap2006e, I think. I'll upgrade to imap2007, whatever's current in the FreeBSD ports tree, to see if it helps. (1) I insured that /var/mail and /usr/tmp were permission 1777 as suggested, but just now so I don't know if that will prove to help. Things used to work just fine without these changes; the only things that may have changed are (a) the users' mail folders have gotten larger, (b) the versions of Outlook they are using are new. Thunderbird (the latest in general) also suffers the failed transfers, though, and only suddenly now for no apparent reason. (2) I see no reference to debugging on the website. I thought I could perhaps turn on a flag for imap in inetd to be able to track
Re: [Imap-uw] tmail_quota() for other folders?
On Thu, 22 Apr 2010, Yiorgos Adamopoulos wrote: I've been using tmail_quota() in order to set a maximum size for the INBOX with no problems. Thank you for providing the hook Mark! Now where should I dive into and use a similar function if I want to set a maximum size for other folders too? This is a much more difficult problem. tmail is a consumer of the c-client append routines, used specifically for mail delivery. To review the status quo: The c-client append routines attempt to handle kernel-enforced hard quotas by catching an append-write exception and rolling back the mailbox to its previous state. In some cases, c-client calculates the size needed, and appends a sufficient amount of NULs to the file so that it occupies that space. If that gets an exception, it truncates the file back to its size prior to the NULs. Otherwise, it assumes that all is well and does the real write. Sadly, this doesn't always work. Certain systems, including that from a recently-deceased major UNIX workstation vendor (one that I regularly criticized and complained about here), keep the over-quota error status hot even after the truncation. This stymies c-client's effort to roll back; and c-client gets quite dismayed at finding that it is forbidden to overwrite space on the disk that it already occupies (as opposed to being forbidden from occupying more). So, most people have learned not to rely upon hard quotas except as a last-ditch measure against abuse. Hitting a hard quota leads to corrupted mailboxes when c-client is unable to undo things due to the issue in the previous paragraph (it's one thing to say don't box yourself into a corner; it's quite enough to spring a trap that closes the exit...). tmail's tmail_quota() hook, and the associated hook in dmail, offers a way to query a policy module from mail delivery. Note that this is specific to mail delivery, not INBOX. tmail_quota() sits on top of the c-client append reoutines within tmail; it does not in any way alter the behavior of any other consumer of c-client. The idea here is that rather than using some hard disk quota number, your policy enforcement is of the form given such-and-such mailbox with such-and-such messages totaling such-and-such bytes, is it alright to add a message of X bytes rather than is it alright to add X+Y bytes, where X is the size of the message and Y is the overhead space for that form, to an existing mailbox of N+M bytes where N is the size of all the messages and M is the overhead space. Hard quotas do the latter; you really want the former. So, there's your problem. One way is to create a similar task to the tmail_quota() hook for the other consumers to c-client that you care about; at a minimum, imapd (ipop3d does not matter). If you have a server on which imapd and ipop3d are the only c-client consumers, this may be your best best. What you need to do is to insert some sort of policy hook akin to tmail_quota() before mail_copy() and mail_append() are called. On the other hand, if you have lots of direct consumers of c-client (e.g., Pine and Alpine doing local access) then you need to go into the c-client library itself. Again, modifying mail_copy() and mail_append() based upon gross policy (as I discussed above) is probably better than having to dig into each driver to include overhead in the enforcement. It's a non-trivial task in any case. I hope that it's just a matter of imapd hacking for you. Good luck! -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re[4]: [Imap-uw] getting UID of a message copied to another mailbox?
On Sat, 17 Apr 2010, Vadim Zeitlin wrote: MC However I do like trash model for other MC mailboxes because it allows me to keep the deleted messages if I ever need MC them again. Think about trash as being archive in this case. MC Why not create archive mailboxes for that purpose? Sorry, what's the difference? By calling something Trash, you are inviting its destruction without notice to you. Trash. n. things that you throw away because you no longer want or need them. Archive. n. a collection of historical documents or records; the place where these records are stored. v. to put or store a document or other material in an archive. Well, this seems like something that I thought about, i.e. use client-side filtering to exclude the deleted messages. But while this could work I don't really see what advantage would this provide while this clearly will have many problems (e.g. interoperability with the other clients which wouldn't hide the messages marked as archived in the mailbox). Yet, in the name of interoperability, you are storing archive data in a place where you have granted implicit permission for its destruction! If you assume that the server management can delete your mailboxes without your consent, there is nothing I (or anybody else) can do to help. Not just server management. ANYTHING. A trash can is not a file cabinet. You can not blame the janitor from emptying it while you are out of the office. But it's not just the janitor. It may be your co-worker (or, to make the metaphor clearer, it may be an application that, unknown to you, automatically empties trash). MC The main hangup that people have is the difficulty of visually displaying MC a trashcan icon that covers multiple mailboxes without having all those MC multiple mailboxes open. Do you mean users or developers by people here? In any case I don't think I ever heard anybody complaining about it... That is the only excuse for implementing the client trash can visual metaphor as a mailbox called Trash. It is otherwise completely unnatural and inappropriate for the IMAP model. What does it mean to check for trash? I personally do have dozens of mailboxes which use trash (and dozens more which do not) but I don't check them. You evidentally are a novice in this. I strongly suggest that you do some research before inflicting yet another bad client on the world. The issue that I refer to is whether or not to show the trash as being empty if there is no Trash mailbox, but instead simply \Deleted status on messages. If you fret about the possibility of a message being deleted in some mailbox that is not normally opened, and want that reflected in the Trash status, then you have to check that mailbox to see if it has deleted messages before showing the Trash status. The point that I am making is that for the vast majority of users this is a silly argument. But it is the only argument to justify a Trash mailbox, which in every other way is inferior to the IMAP delete-expunge model. 1. You should never ever call expunge or you lose your entire archive. [a] Don't delete messages unless you are willing to have them vanish at any time without notice to you. [Don't throw things in your office trash can unless you are willing to have them vanish while you take a restroom break because the janitor came by at that time.] [b] Use a proper archive mechanism. [c] IMAP UIDPLUS-capable servers have UID EXPUNGE 2. Interoperability suffers. If you have to use another client you will see all the deleted messages in it. Worse, it might decide to expunge them, see (1). Nothing whatsoever prevents the same fate from happening to a trash mailbox. 3. Performance will suffer as well. My active mailboxes have from a few hundred to a couple of thousands messages in them and this, of course, is not a problem at all. Having several dozens of thousands of messages might though. In fact if I open my trash right now, it takes 1 minute for the server (Dovecot 1.2) to thread it. Panda IMAP is not that slow. It can thread 50,000 messages in under a second. And mostly I just really don't see what are the _advantages_ of doing it. Of course, undeleting becomes easy/trivial, but this is a rare operation and with UIDPLUS support it's not difficult with the trash neither. You can't put a message back if it has been moved of the mailbox. All you can do is create a new copy. I should also note that Gmail does exactly what I say should be done by IMAP clients. The problem with Gmail is that it forces it on the back end, then layers IMAP on top on that and does it wrong. In Gmail, everything is in one mailbox. Everything else is a view in that mailbox. This is exactly what IMAP keywords are good for, but instead Gmail kludges the IMAP mailbox interface for it (and there are problems because it isn't possible to get that completely right). A much better way to layer IMAP on top of a Gmail type store is a
Re[2]: [Imap-uw] getting UID of a message copied to another mailbox?
On Fri, 16 Apr 2010, Vadim Zeitlin wrote: Ah, thanks, this is indeed exactly what I needed and I've somehow failed to find it it while looking for a solution. Unfortunately in practice it doesn't help that much because IMAP servers without support for UIDPLUS seem to be quite widespread and I'm not sure if it's really user-friendly to tell the user Message can't be undeleted because your server doesn't implement UIDPLUS. So I'll still need a fallback approach. How about writing a proper IMAP client that uses the delete-expunge model instead? The world already has plenty of crappy clients that use the unnatural Trash mailbox kludge. Trash is a user interface artifact, not something that should be inflicted upon the back end. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] RE: Stale imapd process hanging around
On Fri, 16 Apr 2010, Moray Henderson wrote: Anyone? Is there a build option I could try? Should I have posted to another list? Was it a silly question? Shall I give up trying to fix it and just write a script to kill old imapds? Have you tried to see what is specific to that user? If you are using traditional UNIX mailbox format and the user has a ridiculously large mailbox, perhaps you should look into a different mailbox format. You should also consider trying a different kernel (e.g., Linux) in just this is a specific problem to CentOS. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] using shared mail folders
On Mon, 12 Apr 2010, Andrew Daviel wrote:
- question: if using the MIX format, what happens to the ownership and
permissions of new folder fragments (whatever .mixnnn files are called)
in shared folders ? Do they get set with mode 600 hence become unreadable
by other group members ?
Mailboxes in #shared are created with mode 660 for files and 770 for
directories.
If I create a regular account with appropriate file permissions and group
memberships I can access it easily in Alpine
as {mail.example.com/user=jane}~dick/INBOX
But in Thunderbird I can't see how to easily subscribe.
I wish that I could help you on this, but I can't.
Thunderbird's implementation of the IMAP subscription mechanism is broken
by design, and has been for 15+ years. I wrote, years ago:
9. Thou shalt not abuse subscriptions, for verily the
LIST command is the proper way to discover mailboxes
on the server. Thou shalt not subscribe names to the
user's subscription list without explicit instructions
from the user; nor shalt thou assume that only
subscribed names are valid. Rather, thou shalt treat
subscribed names as akin to bookmarks, or perhaps akin
to how Windows shows the My Documents folder -- a
set of names that are separate from the hierarchy, for
they are such.
Thunderbird (and its predecessors such as Netscape Messenger) is the worst
violator, followed closely by Outlook.
Regrettably, the simple answer is that Thunderbird users are screwed.
I also get a complaint from sendmail that ~dick is group-writable, and
sendmail refuses to read ~dick/.forward. Not perhaps a problem since
forwarding a group account is unlikely.
~dick needs to be group-writeable only if you need to allow other users to
create new mailboxes in ~dick. If you do, then you are probably best off
using the mailsubdir mechanism to put mailboxes in a subdirectory; that
way you can make the subdirectory group-writeable without making the home
directory group-writeable and then sendmail won't have a hissy-fit.
-- Mark --
http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
___
Imap-uw mailing list
Imap-uw@u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Did someone build imap c-client on iPhone OS?
On Tue, 13 Apr 2010, Hanjie XU wrote: I have iPhone OS3.0 SDK on my mac mini，can I use the Apple's SDK to build c-client? I don't know. c-client does not need any special Apple SDK includes or libraries, but it does need standard C/UNIX ones (including openssl and pam). I never used the Apple SDK because I object to its licensing terms. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote.___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] slow IMAP connection
Although I agree with Bob's conclusion (use mix format), I think that it overstates the issue to say that mbx format is only marginally better than mbox. mbx is a major jump up from mbox, and is handled much better than mbox. mbx scaled very well for the demands of 1996 when it was designed; it's just that we no longer live in that world. mix was designed in 2006 to be as much of a jump from mbx as mbx is a jump from mbox. mix is the ONLY choice for a mailbox 2GB in size; mbx and mbox will not work at all. Although mbx will work with a mailbox of 40,000+ message, it would be painful; and mbox would be unusable. I think that a way of looking at it is that if mbox is a 1, then mbx is a 10 and mix is a 100. On Tue, 16 Mar 2010, Bob Atkins wrote: The only way to fix the slow I/O problem, especially with large mail boxes is to convert to mix format. mbx format is only marginally better than mbox from a performance standpoint. This is from my personal experience of having mail boxes with 40,000+ messages and 2GB in size. Today's users have no idea how much space their mail is consuming, nor would they care if they did. Our only option as administrators is to implement the fastest and most scalable solution that we can. We converted our systems to mix format about a year ago and oh what a different it made. Much happier users and even happier admins ;-) -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] slow IMAP connection
On Tue, 16 Mar 2010, Tim Mooney wrote: This brings up a question I've had: both traditional format and MBX run into issues when they cross the 2 GiB boundary, because of signedness issues in the UW code. Correct. It's not the code so much as it is the system calls; there are separate system calls that have to be used if you want to cross 2GB filesize on a 32-bit OS. It doesn't make much sense to fix because flat files perform so poorly long before you get to that size. I realize a MIX folder is composed of multiple files, but does that mean that the 2 GiB is not a problem for MIX? Not until/unless you start dealing with single messages that are larger than 2GB. It will be a while before people start wanting to do that. :) -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] slow IMAP connection
On Tue, 16 Mar 2010, Tim Mooney wrote: Not until/unless you start dealing with single messages that are larger than 2GB. It will be a while before people start wanting to do that. :) We're still not allowing individual messages larger than 25 MiB, so we're safe for now... Yeah, even Gmail limits to 35MB. Even if the promise of universal 50Mbit/s broadband comes to pass, a 2GB Godzillagram would take a minimum of several minutes to transmit at (unobtainable) optimum performance with no other traffic. If SMTP is a suitable mechanism to transmit 2GB hunks of data, that begs the question of why we need bittorrent... ;) -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
RE: [Imap-uw] slow IMAP connection
On Tue, 16 Mar 2010, Troy Campbell wrote: I'll take a look at the size of the INBOXES to see if that's the issue. What would be the tool/process for converting from mbx to mix? mixcvt is the best tool to use. There is an updated Panda version of mixcvt, but UW mixcvt should be alright for most purposes. The tool that you want to avoid is UW mixrbld. UW's code is badly broken and will lose messages. Panda mixrbld fixes these issues. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] slow IMAP connection
On Tue, 16 Mar 2010, Bob Atkins wrote: As those who went down the maildir path (with another imap server) have painfully discovered - directories with too many inodes are very slow to search. mix is a great balance between not too many inodes and keeping file sizes and linear search runs to a minimum. Interesting. I have had acolytes of the maildir church scream that I don't know what I am talking about when I said that there was a scaling issue with directories with too many inodes. FWIW, the same issue happens with mh and Cyrus. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] slow IMAP connection
On Tue, 16 Mar 2010, David B Funk wrote: It's also kinda fun to talk to the Exchange admins and watch them cringe when I talk about our users who have 5GB inboxes with 30K messages in them. ;) Hah! I suppose that you know that UW jumped on the Exchange bandwagon. That is, for the privileged few, such as their million-dollar plus president; plebes, proles, and students are to be outsourced to the cloud. Then again, I guess that Exchange is an upgrade from GroupWise... -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] slow IMAP connection
On Tue, 16 Mar 2010, Jim O'Leary wrote: The mix format dug us out of an approaching performance catastrophe. Three cheers to Mark for developing mix in the nick of time to save our necks! Thanks everybody for the nice comments. It means a lot to me to know that the efforts were appreciated, especially in light of subsequent events! Not everybody at UW was happy that I fixed the IMAP servers... If you like mix, and are considering using commercial products, I'd like to mention what I've been doing at Messaging Architects. We recently released version 2009.1 of M+Guardian: http://www.messagingarchitects.com/products/m-guardian-email-security.html 2009.1 was a major update, nearly a complete rewrite. It is now the first of our products to use the foundation technology that will presently underlie our other products. One of my big efforts for foundation's was the mail store, which uses a greatly expanded version of mix; adding clustering (no more requirement that the IMAP server and mailbox files be located on the same CPU), mailbox metadata, message metadata, and...stubbing!! Currently, we use stubbing (my term for it is virtual mailboxes) to implement the quarantine in M+Guardian; all quarantined messages go to a mailbox in the global quarantine. The per-user quarantines are virtual mailboxes containing pointers to the data in the global quarantine. However, the stubbing facility is far more general than this; among other things it can be nested (a virtual mailbox can have a pointer into a virtual mailbox). I'll leave it to your imagination how this will be used in our other products... The clustering is also way cool. Between the IMAP server and the mix store is a stateless URI-based protocol (albeit with features to support stateful protocols such as IMAP and POP) and a clustered filesystem. I had my paws in the design of the stateless protocol and was the first to beat(!) hard(!!) on the clustered filesystem. I also wrote a completely new IMAP server that (unlike UW and Panda IMAP) supports ACL. Obviously, this is not just for M+Guardian (an anti-malware, anti-spam, content filtering, and data leak prevention appliance). If you look at our web site, it's not hard to guess how we'll be using foundation in those products. This is all stuff that our customers are using now. There's a lot more neat stuff coming up in the future. My teammates have been quite busy... -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
RE: [Imap-uw] slow IMAP connection
I would have to see the dumps in question to comment intelligently on what they contain and whether or not there is a problem. Unfortunately I don't have the time to do that. However, I can tell you that a STATUS command internally opens the destination mailbox. Certain broken clients think that they should do a LIST of all possible names, SUBSCRIBE all those names, and then issue a STATUS of those names at each synchronization. If you use traditional UNIX mailbox format (mbox format) then you'll certainly have performance problems with clients that do that. mix format will help you a lot with that. On Tue, 16 Mar 2010, Troy Campbell wrote: My other concern is that there are certain folders showing up empty. When I do a tcpdump from the server watching the client just as they click on the folder and capture the traffic I'm seeing IMAP STATUS commands of other folders. It's like the pointers are screwed up somehow or I'm seeing delayed data from her scrolling down (kind of like seeing the ls data in a directory prior to the cat command of the file). -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
RE: [Imap-uw] slow IMAP connection
On Tue, 16 Mar 2010, Troy Campbell wrote: Thanks Mabry, it sounds like we need to move to a newer file format. Is mix supported by other imap servers as well i.e., is it a standard format or something UW specific? mix is specific to UW IMAP (imap-2006 and later) and Panda IMAP. Any application which uses the c-client library (e.g., Alpine) is also mix-capable. The new Messaging Architects mail store uses a superset of mix. It is possible to take mailbox data from UW IMAP or Panda IMAP and place it into an MA mail store mailbox. In fact, that is how I created my test data. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] BAD Missing or invalid argument to APPEND
On Mon, 11 Jan 2010, Robert Hardy wrote: I get the error message BAD Missing or invalid argument to APPEND from my mail clients (Profimail 3.25 and I think alpine 2.00 too) when sending a message and it's saving in sent-mail. I doubt very much that this is happening with Alpine 2.00. I would have to see proof before I believe such a thing. Proof would be a .pine-debug# file resulting a session started with the command alpine -d imap=4 in which the problem was made to happen. I'm not saying that it's impossible. I am saying that I don't believe it; and I won't believe it, unless/until I see a .pine-debug# file that shows Alpine making it happen. I wrote both the APPEND command generation code in Alpine 2.00, and the APPEND command parsing code in UW IMAP. Those two segments of code have interoperated successfully with each other for nearly two decades without ever generating that error message. I would be extremely surprised if there was a bug after all these years. The error message is from UW IMAP. It is a catch-all for a syntax error in the APPEND command from the client. I would have to see the command in question to determine what, precisely, is wrong with it. Can someone tell me if this is a server bug or a client bug, what is wrong and ideally how to fix it? It is almost certainly a client bug. If, as I suspect, you can not make the problem happen in Alpine 2.00 after all, then delete the word almost from the previous sentence. If that is the case, then see if there is some option in Profimail 3.25 that takes a telemetry transcript of the IMAP protocol negotiations. If so, I can look at it for you, and tell you what Profimail 3.25 is doing wrong. It will then be between you and the vendor of Profimail 3.25. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] BAD Missing or invalid argument to APPEND
Although my analysis is limited due to the log being damaged (by your own
admission, you changed names/addresses), I see the cause of the error.
Profimail 3.25 is sending the message data incorrectly. After exactly 990
message text bytes (the stated count in your message) the IMAP protocol
expects either a CR/LF sequence (to terminate the append) or a space
followed by the arguments to define another message to append.
Unfortunately, because you have edited your trace I can not tell for
certain if it sent a short message size count, or if it sent a spurious
space after the message. The message text, as you have it, is either 989
bytes with UNIX-style LF-only newlines (a bug if the client is doing that)
or 1016 bytes with proper CR/LF newlines. However, because you changed
things, those counts can not be relied upon.
In IMAP, counts must be exact; even the slightest variation completely
changes the interpretation of the protocol.
I also can't tell what the client means by the following in the log:
: 3 BAD Missing or invalid argument to APPEND
The : is apparently its prefix for something that it is sending; but
the server sent the 3 BAD Missing or invalid argument to APPEND message.
I would expect to see something like : followed by a newline.
In any case, that should be enough to get you started in producing a bug
report to them.
On Mon, 11 Jan 2010, Robert Hardy wrote:
On Mon, 11 Jan 2010, Robert Hardy wrote:
I get the error message BAD Missing or invalid argument to APPEND from my
mail clients (Profimail 3.25 and I think alpine 2.00 too) when sending a
message and it's saving in sent-mail. I tried upgrading to imap-2007e from
imap-2007b but this hasn't helped.
Can someone tell me if this is a server bug or a client bug, what is wrong
and ideally how to fix it?
To clarify this only happens rarely with certain messages. Once it happens
with a particular message I can't ever save that message to sent-mail.
I tried to include an attached IMAP trace from Profimail but it seemed to get
striped by the mailing list.
Here is the text inline with the names/addresses changed to protect the
innocent:
Connect
Connection already active, using it
Socket opened
Connect to host mail.mydomain.ca:993
Connect to IP mail IP addr
Connected: mail.mydomain.ca:993
Init SSL
SSL inited
SSL handshake done
* OK [CAPABILITY IMAP4REV1 I18NLEVEL=1 LITERAL+ SASL-IR LOGIN-REFERRALS
AUTH=PLAIN AUTH=LOGIN] mail.mydomain.ca IMAP4rev1 2007b.404 at Mon, 11 Jan
2010 15:34:59 -0500 (EST)
: 1 CAPABILITY
* CAPABILITY IMAP4REV1 I18NLEVEL=1 LITERAL+ IDLE UIDPLUS NAMESPACE CHILDREN
MAILBOX-REFERRALS BINARY UNSELECT ESEARCH WITHIN SCAN SORT THREAD=REFERENCES
THREAD=ORDEREDSUBJECT MULTIAPPEND SASL-IR LOGIN-REFERRALS AUTH=PLAIN
AUTH=LOGIN
1 OK CAPABILITY completed
: login command sent
2 OK [CAPABILITY IMAP4REV1 I18NLEVEL=1 LITERAL+ IDLE UIDPLUS NAMESPACE
CHILDREN MAILBOX-REFERRALS BINARY UNSELECT ESEARCH WITHIN SCAN SORT
THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND] User rhardy
authenticated
: 3 APPEND mail/sent-mail_mobile (\Seen) {990}
+ Ready for argument
: Subject:
=?utf-8?q?Re:_Reply_to_your_3_Toilets_for_sale_from_renovation_Ad_on_Kijiji?=
Date: Mon, 11 Jan 2010 13:56:57 -0500
From: Robert Hardy rha...@mydomain.ca
To: pine...@theirdomain.com
MIME-Version: 1.0
X-Mailer: LCG ProfiMail
Message-ID: 2703167228.947685...@mydomain.ca
In-Reply-To:
377624194.1174331263223621741.javamail.r...@kj-classy018.intern.kijiji.com
Content-Type: text/plain; charset=utf-8; format=flowed
: Yes they are. There is someone interested in 3 coming on Weds but it is
first come first serve.
Rob
--- Original message ---
From: Kijiji Reply (from pine...@theirdomain.com) p...@kijiji.ca
To: rha...@mydomain.ca
Sent: '10-01-11, 10:27
Hello! The following is a reply to your 3 Toilets for sale from
renovation Ad on Kijiji:
From: pine...@theirdomain.com
Hi I am interested in one ... please let me know if they are still
available.
Martha
You can respond to pine...@theirdomain.com by replying to this email.
: 3 BAD Missing or invalid argument to APPEND
Destroying socket
Socket cancel: OK
SSLsock close: OK
Socket close
Ok
~finished
___
Imap-uw mailing list
Imap-uw@u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
-- Mark --
http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
___
Imap-uw mailing list
Imap-uw@u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] saving email in imap inbox
I don't know what you mean by a general IMAP user name and password. Each IMAP user has his own user name and password. Unless you set up a mail administrator userid (which can log in as any other user, hence has the equivalent of root access), an IMAP user can generally only save to the mailboxes that are owned by that user, and not to some other user's INBOX. If you need to write to someone else's INBOX, you probably need to use SMTP to send the message as mail. On Fri, 8 Jan 2010, chamila piyasena wrote: I have a requirement of saving an email in the inbox of the receivers mailbox. As the in c-client I have to open the mailbox prior to appending, can I use the genaral imap user name and password for it. Or else is there any other way to do it? (I dont have the recivers email password at that time). I would be greatful if somebody can help me on this. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] public and shared folder identifiers are not displayed as directories
The simple answer is that it is a bug in Mozilla. #public is not a folder. It is a namespace. Think of a namespace as like an A: drive in Windows. If, and only if, there is a floppy inserted on your PC, you can reference names in a separate hierarchy starting with A: A namespace similiarly identifies a separate hierarchy that may exist. If, and only if, that hierarchy actually exists, you can references names that start with that namespace name. In UW IMAP, the namespace #public identifies a hierarchy whose root is the home directory of user imappublic. Thus, #public/foo is exactly equivalent to ~imappublic/foo. If the IMAP client LISTs #public/*, one of two things will happen: [1] No results. The user imappublic does not exist. [2] The contents of the home directory of user imappublic. In the case of [2], you will see #public/ returned as a \NoSelect name, as this is the root of the #public space. #public/#public is simply nonsense. Something concatenated #public twice. UW IMAP will never do that, so it must have been Mozilla. Perhaps Mozilla was attempting to apply a prefix; if so, it did it incorrectly. The correct way to apply a prefix is to use the reference field of the LIST command. That is, given a prefix of #public/ and a request to open #public#: Wrong: concatenate #public/ and #public, forming #public/#public Correct: in IMAP, do the command: tag LIST #public/ #public and use the name that is returned from LIST. If no name is returned, there is no such name. In conclusion, this is completely a bug in Mozilla. I should also mention that this comment in bug 497096 that claims a server bug is also completely wrong: 5 list Trash/emu2000/* * LIST (\HasNoChildren) / Trash/emu2000/ (B) This response means folder named null under emu2000 under Trash. Folder name in response has to be correct name which can be specified in SELECT commant etc. So this is apparently server side bug. There is no such thing as a folder named null. The LIST response here simply confirms that Trash/emu2000 exists as a directory. Otherwise, there would be no way to distinguish directory empty from directory does not exist. The bottom line is that if a client developer claims that there is a bug in UW IMAP, he is almost always wrong; and is confused by a subtle aspect of the IMAP protocol. On Sat, 2 Jan 2010, Juergen Edner wrote: Hello, I'm using uw-imapd 2007b on my mail server. For quite a while I'm now using public/shared folders to share information between different users or provide a central spam mailbox. This is e.g. my folder/mailbox structure: #public - folder + Spam-Mail - mailbox Sometimes I'm erroneously selecting the #public (or #shared) folder in my Thunderbird client which causes the following error message: Can't open mailbox #public/#public: no such mailbox I raised a bug request for TB but based on the following discussion of the developers this problem seems to be a uw-imapd problem because the server doesn't return a \NoSelect if such a folder is selected. Here you can find further debug information and the discussion of the problem: https://bugzilla.mozilla.org/show_bug.cgi?id=517054 Now I wonder if this is really a problem of uw-imapd and if so, how it can be solved? Thanks Juergen -- Mail: juer...@eisfair.org ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] major data corruption bug in UW mixrbld
I have discovered a rather nasty bug in mixrbld which occurs if you run it on a mailbox that was created/copied using mixcvt. The bug causes messages to be lost in the rebuilt index. Although a mixcvt created mailbox will almost certainly encounter the problem (causing the loss of all mixcvt-copied data), it can occur with other mailboxes. This problem is fixed in the Panda version of mixrbld. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] need an advice
On Mon, 28 Dec 2009, chamila piyasena wrote: can I only put the mail.c in my applicationand use those functions or is there other files to be used. mail.c is simply the external interface to the c-client library. You need to build the complete c-client library, and link with it. Look at the mtest.c sample program for a simple example. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Force IMAP server to refresh message flags?
On Mon, 21 Dec 2009, Ken Murchison wrote: I've added an option that can be enabled that forces \Seen state to be flushed immediately and exhibits proper IMAP behavior. The new dev branch of Cyrus will have a reworked \Seen state where it should work correctly by default. Cool! Thanks! By the way (given another conversation we had) the new POP server that I'm doing doesn't use any IMAP flags at all. I may change it to use the IMAP flags the way that UW/Panda ipop3d does, but for the nonce it turned out to be more convenient not to do that. This also means that I'm not bothering with the LAST command. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Force IMAP server to refresh message flags?
On Thu, 17 Dec 2009, Shawn Walker wrote: How does the client tell the server to refresh the message flags? The client is multithreaded and that we have multiple connections to the server (2). That is not a good client design. There is neither guarantee nor requirement in IMAP that the server supports multiple simultaneous read-write access to a mailbox. If the server is UW with traditional UNIX format (mbox) format, there is no simultaneous read-write access, and every time you open the mailbox read-write it forcibly closes the mailbox in the previous session. The mix and mbx formats support simultaneous read-write access, but they do immediate flag updates across sessions. We have a situation where the client mark a message as read in one thread, but when the synchronize the folder, the client send . UID FETCH 15402 FLAGS, the server send back * 5261 FETCH (FLAGS (\Answered) UID 15402) which mean that the message is unseen for that connection. If the server is UW with traditional UNIX format, this happened because the mailbox was forcibly closed. See above. Obviously the IMAP server is caching the flags for the connection, but I need it to dump the cache or refresh the message cache, but without logging out and log back in every time the client want to get the message flags. No server caches the flags as you describe. If the mail store in the server supports multiple simultaneous sessions to the same mailbox (e.g., mix and mbx formats in UW), then flags are updated immediately. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Force IMAP server to refresh message flags?
On Thu, 17 Dec 2009, Timo Sirainen wrote: On Thu, 2009-12-17 at 10:22 -0800, Mark Crispin wrote: No server caches the flags as you describe. If the mail store in the server supports multiple simultaneous sessions to the same mailbox (e.g., mix and mbx formats in UW), then flags are updated immediately. Well, not necessarily immediately, at least not all servers. A NOOP command should refresh flags and other things with most servers. A server that requires use of the NOOP command to synchronize is broken. A NOOP command MUST NOT do anything special. It's a no-op. Its only purpose is to cycle the server engine if otherwise has nothing to do. Any server refresh operations MUST be done on every command. A client that uses the IDLE command has no reason to use the NOOP command, ever. Hmm. Dovecot's current behavior is that doing a FETCH FLAGS just after its flags had changed in another session first returns one FETCH FLAGS with the old flags (the FETCH reply), then another FETCH FLAGS with the updated flags (after it noticed the changes). That's OK. It's in the same set of responses, isn't it? The later flags response overrides the former one. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] invalid remote specification
The most common cause of this problem is failure to do
#include linkage.c
early in the main() function of the application (and prior to any c-client
library calls).
The use of linkage.c is mandatory. Some programmers omit the linkage
entirely; others attempt to do the linkage manually. Either way, they
come to grief.
If you are certain that you have included linkage.c in your main()
function, then the only other cause is that the IMAP software was built
without SSL.
Finally: in your example, neither the :993, nor the /imap, nor the
INBOX are necessary although they won't hurt. /imap is the default,
:993 is implied by /ssl for IMAP, and INBOX is also the default.
Hence
{imap.gmail.com/ssl}
is exactly equivalent to
{imap.gmail.com:993/imap/ssl}INBOX
but is much less typing.
On Wed, 18 Nov 2009, Mike T wrote:
I am trying to write a simple test program for the c-client library based off
of mtest.c. When I am entering the mailbox string (eg.
{imap.gmail.com:993/imap/ssl}INBOX) I keep receiving an error saying Can't
open mailbox mailbox string: invalid remote specification. What could be
the causes to thsi error?
-- Mark --
http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
___
Imap-uw mailing list
Imap-uw@u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] problem/bug/limitation in uw-client-library
On Thu, 15 Oct 2009, Volker Schwicking wrote: Mark, neither of us is trying to get anything done FOR us. WE (together) are trying to fix a problem for US (notice the capitalized letters). So why, after being repeatedly told that the problem is ultimately due to an incorrect definition in a glibc include file, do you continue to discuss the problem here? Fix the incorrect definition in the include file, and file a bug with the maintainers of glibc so that it is fixed in subsequent glibc updates. That way, the problem gets fixed in ALL libraries and applications that use select(). There has been testimony that other libraries used by Apache, completely unrelated to c-client, are impacted by this glibc bug. There has been testimony that other applications, completely unrelated to IMAP, are impacted by this glibc bug. It should not require elite knowledge to conclude that the remedy is to be found by fixing the glibc bug. It just requires common sense. If you do not know how to fix the glibc bug -- a one-line edit in a system include file -- then find someone who does. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] problem/bug/limitation in uw-client-library
On Thu, 15 Oct 2009, Ico Doornekamp wrote: There were - and still are -- sites that run UW IMAP on platforms that do not have a select() call able to handle more then 1024 file descriptors. Name one. BSD and Linux systems are perfectly capable of handling more than 1024 file descriptors in the select() call. The problem is in glibc. A ONE LINE bugfix to two glibc include files fixes that limitation. Just ONE LINE. There were -- and still are -- sites that use UW IMAP on platforms that do not have a poll() call. Send in a patch to fix those platforms! The patch is to fix the broken glibc includes on Linux, and has already been discussed in more than enough detail for anyone to figure out what to do. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] problem/bug/limitation in uw-client-library
On Thu, 15 Oct 2009, Ico wrote: GNU/Linux There is no such thing. I don't care about Richard Stallman's megalomanic fantasies. particulary the versions using glibc as the C library. Which would be most if not all of the mainstream GNU/Linux distributions. Nonsense. The bug is in a C compiler include file. Many systems don't have the C compiler include files installed, as they do not have the C compiler installed. Neither a compiler, nor its include files, are part of an operating system. The operating system does not care what that value is set to. It is not an operating system constant or variable. It simply defines the size of the default bitmap, generated by the compiler, that is passed to a system call that takes the actual maximum size as an argument. A binary generated with a different bitmap size will work perfectly well on any system...INCLUDING a system which does the fix to that include file. Suppose that there was a bug that caused the string routines to fail if the string ended with .nl. Would people in Holland accept being told don't use the .nl domain, use .com instead? Yes, that's a stupid example, but no more stupid than don't use this system call because there's a bug in the compiler include files that we refuse to fix. It is a simple, one line fix which anyone can make. It does not change the effect of any system binaries. The only thing that it does is cause binaries built with the fixed version to work with larger file descriptor values. Those binaries can be ported to any other system and they will work. BSD and We're not talking BSD here. BSD serves as proof that it is technically possible to solve the problem, because they did. It has further been shown that it is technically possible to apply a similar solution to Linux. If you disagree, then the answer should be don't use Linux, use BSD instead. Linux systems are perfectly capable of handling more than 1024 file descriptors in the select() call. No, Linux 'systems' are not. The linux *kernel* is. The 'system' as a whole usually consist of a bit more then just a kernel, for example a C library. That is Richard Stallman's nonsense. It is to cover up the fact that he was not competent enough to create an operating system when he announced GNU nearly 30 years ago, and lacks such competence today. The definition in question does not impact any system libraries. It only impacts whatever applications and libraries use select(). By fixing the include file defintion on the system where the application/library is built, you fix the resulting binary on ALL systems that run that binary. The problem is in glibc. A ONE LINE bugfix to two glibc include files fixes that limitation. I never denied this, did I ? It is just a bit unfortunate that the main author and maintaner of glibc does not agree with you this is a feasible fix: http://sourceware.org/ml/libc-alpha/2003-05/msg00173.html Ulrich Drepper's opinions no longer matter. Everybody is switching to eglibc from Drepper's glibc, in part because Drepper refuses to maintain glibc properly. Funny details is that he recommends to use poll() instead. His binary compatibility argument is bogus. It doesn't matter even if you link an application with two libraries which were build with different definitions. The first argument to the select() call defines the maximum length of the fd_sets. The only thing that FD_SETSIZE does is ensure that the compiler generates an fd_set that is long enough. What it really comes down to is that Drepper didn't want to fix his bugs, and instead wants everybody else to break their code to accomodate his bugs. There is an excellent reason why I did not use poll(); c-client to this day is used on systems that do not have that system call. The problem with select() on Linux is not due to anything on Linux. The operating system is perfectly capable of correctly executing a binary that was built with that include file bug fixed...even if the bug is not fixed on the system running the resulting binary! Similarly, there is no reason to believe that a different compiler would have that bug. Since the problem is not in binary execution, but rather in a certain compiler on a certain system generating incorrect binary code, the correct fix is to fix the compiler. In this case, it is a one line change to an include file, not a compiler binary. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] problem/bug/limitation in uw-client-library
In BSD, you can define FD_SETSIZE to some higher value than 1024 prior to the inclusion of sys/types.h. The only thing that FD_SETSIZE does is define the default capacity of the FD_SET; the kernel is quite happy to access a larger fd_set. Doubling FD_SETSIZE to 2048 would increase the fd_set by a whole 128 bytes. Oh, the horror! How can we possibly afford it? [Excuse the venomous sarcasm, but this sort of baby-computer thinking has been an ongoing problem with UNIX type systems.] In Linux, it is messier. FD_SETSIZE is defined by __FD_SETSIZE, which in turn is defined in /usr/include/bits/typesizes.h and /usr/include/bits/posix_types.h The comments in posix_types.h are definitely worth reading, as they betray more baby-computer thinking. Worse, there is a __kernel_fd_set definition which implies that the Linux kernel is incapable of handling programs which are compiled with larger definitions of FD_SET. Even more annoying, Linux has a #define for the maximum file descriptor value -- NR_OPEN -- and there is no reason other than cranium-up-rectum disease for FD_SETSIZE to be any other value. I didn't investigate further. For some reason known only to the developers of Linux, the includes and their mechanisms are incredibly arcane. Try looking for the errno definitions -- you have to go through multiple levels of #include before you find them. So, I don't know if an application program on Linux can define a larger fd_set size that the kernel subsequently can use. If not, then the only fix is to change tcp_unix.c to use poll() instead of select(), and hope that you never have to build c-client on a system that has select() but not poll(). -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] problem/bug/limitation in uw-client-library
On Wed, 14 Oct 2009, Ico Doornekamp wrote: I think I have understood these two pretty well. Like I said, the linux FD_SETSIZE limit is well known and well documented. From 'man select' : : Executing FD_CLR() or FD_SET() with a value of fd that is negative or is : equal to or larger than FD_SETSIZE will result in undefined behavior. That's linux. We're not talking BSD here. You missed the point. The BSD man page is emphatic that an application can change the definition of FD_SETSIZE and obtain defined behavior. It states how to do it. There is nothing in the Linux man page which says that you can (or can not) change the definition of FD_SETSIZE. If, as some have said, this is a glibc issue on Linux, then some means should be possible for the application to change FD_SETSIZE. If, on the hand, it is a kernel restriction, only a change to the kernel will fix it. I do not know which of these is is; nor do I know the necessarily procedures either way. This is a research task for someone else. The linux (glibc) select() manual does say nothing about the solution you mention, because this is a peculiarity of the BSD select. We're not talking BSD here. Do you meant to say that Linux is hopelessly inferior to BSD and will never be as good as BSD? If not, then what are you saying? BSD demonstrably has a capability to make select() work with all valid fds. Does Linux have that capability or not? I don't know; and if you don't know you should find out instead of arguing with me. Although I'd not bet my head, I'm pretty damn sure this will not work. Anyway, according to linux's select() documentation, it's not ment to work this way. I see nothing in Linux's documentation that says that. There is no listed bug that says select() on Linux can't ever handle fds greater than 1024. The only reference to FD_SETSIZE is that it is the limit to the FD_SET and FD_CLEAR macros. There is nothing whatsoever stating whether that limit can be changed, nor if the kernel would respect that changed limit. Once again, instead of arguing with me, you should talk to someone who knows either way. Whatever the cause of this limitation: all I'm saying this is a *well known* limitation. So what? The select() system call itself is considered arcane and has more probles besides this limitation, it's only there bacause it always has been. Every kernel developer will simply tell you not to use it and to take a look at poll() instead. Every kernel developer?? Or just every 18-year-old who thought that he was a kernel developer? poll() can be a very poor interface, both from the application and the kernel's point of view. poll() does not scale at all for large numbers of events. Yes, doing select() well requires an understanding of how to work effectively with bitmasks; a topic that is generally deferred for advanced CS classes. Some people actually have mastered that concept. More to the point: NOWHERE in the Linux man page for either poll() or select() is select() declared to be arcane or has problems. The Linux man pages are not at all shy about making such declarations. I was not aware the project is not activily maintained. Probably my wrong for not looking this up elsewhere before asking anything on the mailing list involving the code ? The discussion came up with multiple courses of action: [1] Give up. [2] Determine that the Linux kernel doesn't have any wired knowledge of the length of an fd_set. Then: [2a] Determine how to redefine FD_SETSIZE in an application so that the fd_set struct and FD_SET/FD_CLEAR macros are defined to be a size large enough to accomodate any possible fd. [2b] Fix glibc to redefine FD_SETSIZE so that the fd_set struct and FD_SET/FD_CLEAR macros are defined to be a size large enough to accomodate any possible fd. [3] Determine that the Linux kernel has wired knowledge of the length of an fd_set. Change that wired knowledge so that the kernel can handle a fd_set of size large enough to accomodate any possible fd. Then do either [2a] or [2b]. [4] Kludge tcp_open() in c-client so that when it gets a socket it will get one in the range. [5] Kludge Apache so that it cleans up its fds and libraries (not just c-client) called by do not hit the limit. [6] Change c-client to use poll(). [7] Use some other library that still has a developer. If I were you, I would look into [2a], and would seek authoritative information from a real Linux kernel developer on the issues in [2] and [3]. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] imap-2007e on ubuntu SSL not starting, should make lnp IP=6
I can't speak intelligently about Ubuntu edgy. I started with feisty and am now on jaunty. As far as Panda IMAP (and UW IMAP 2007e) is concerned, modern versions (and possibly earlier ones) of Ubuntu are a type of Debian, and therefore the appropriate build command is make ldb You do not need to specify IP=6 since that is set unconditionally with Debian builds. You can ignore the rest of this message unless you are interested in technical details. lnp may build successfully, but the SSL paths will be incorrect. As with other Debian systems, Ubuntu expects the CA certificates to be in /etc/ssl/certs and the private keys to be in /etc/ssl/private. Sadly, consumers of OpenSSL (such as IMAP) are expected to know this, and to tell OpenSSL this. It would have made more sense if there was a systemwide setting that OpenSSL would use without asking the consumer to know; but there isn't and thus we need these multiple builds. The other way is what autoconf does; look at various well-known places to see where the right place is. But then if you have certificates or keys in multiple places (perhaps due to updates which changed the location), it can build to configure the wrong place. That, in turn, leads to all sorts of strange issues, because sometimes it may work (because the right key is in the wrong place) and sometimes not...oh dear, oh my... On Tue, 6 Oct 2009, Michel Henry de Generet wrote: Hello, After many trials, I was able to enable SSL on ubuntu making it with lnp IP=6 as saw in one mail. Could it be related to inetutils-inetd ? Could you confirm it is the correct make option for ubuntu edgy (at least) ? Maybe I didn't read well the FAQ. I use uw imap for about 10 years, this is the first time it take so much effort to launch it. Regards, Michel de Generet -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] CRLF
On Mon, 28 Sep 2009, Oswald Buddenhagen wrote: hmm, i think i interpolated that from the formats.txt and other files. also, with the unix driver (LF in the mailboxes), imapd still delivers nicely CRLF-terminated lines. The traditional UNIX format is incapable of preserving newlines. Thus its driver must convert newlines in data, both incoming and outgoing. mix and mbx do not have this problem. This is all c-client library. None of this has anything to do with imapd. imapd and IMAP protocol exist solely in a world where newline is CRLF, and a CR or LF by itself is not a newline. If your MDA really has LF-newlines, then something converted the CRLFs carried by SMTP to LF. You should fix whatever did that, instead of thinking that your MDA should convert LF-newline to CRLF. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] CRLF
On Sun, 27 Sep 2009, Oswald Buddenhagen wrote: my MDA uses mail_append() with a mail_string. the string contents are not pre-normalized to CRLF upon the call. Do you really mean an MDA (Message Delivery Agent, called by an incoming SMTP server)? SMTP transmits text newlines in CRLF format. If the text presented to an MDA is not in CRLF format, then something (most likely the SMTP server) normalized the SMTP CRLFs to (I assume) UNIX LF-only newlines. You need to tell the SMTP server not to do that. In sendmail, this is usually done with E=\r\n in the Mlocal line in your sendmail.cf (see the tmail man page). If you do not really mean an MDA, what do you mean? Do you mean MUA (Mail User Agent), as in Pine or Alpine? Please clarify. the mix driver writes the unix line endings literally into the mailbox, and when reading it, it delivers the contents literally. of course, this results in the imap server being non-compliant with the spec. I don't know why you think that this results in the IMAP server being non-compliant with the spec; IMAP has no such requirement upon the data it processes or transmits. IMAP does, indeed, require CRLF newlines in IMAP protocol, but protocol and data are two entirely different things. IMAP *data* is a sequence of zero or more octets from 0x01 to 0xff. now i wonder whether my MDA is expected to pre-normalize the data before building the mail_string or whether the mix driver is broken? mix is performing as designed and desired. mix can, among other things, be used to store binary content. So can IMAP. If you want to store an RFC 5322-compliant message in mix (and IMAP!) you must present an RFC 5322-compliant message in the mail_string presented to mail_append(). Whether this means your MDA is expected to pre-normalize the data or your MDA should not de-normalize the data from SMTP or blurdybloops are bombastic is for you to determine. I can't tell you which based upon the information you gave me. I can tell you the requirements of IMAP, mail_string(), and mix. I also made a suggestion as to what may be wrong if you really are talking about an MDA as opposed to something else. There is no reason for an MDA to pre-normalize since an MDA's data should be in RFC 5322 compliant format to begin with. If it is not, then something fixed it into non-compliance, and you would be better off telling it not to do that than to write additional broken code to fix a broken fix. Those who live by kludge towers ultimately are tossed from them. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Snow Leopard / openpam support
On Sun, 13 Sep 2009, Nicholas Cole wrote: Thank you for looking at this. I'm sure you've replicated, but here are the build errors, as requested. Is there any workaround? According to information from David Morsberger, it ought to work to remove the setting of MAC_OSX_KLUDGE from the oxp build in the top-level Makefile. But I don't have a clue as to why it would cause the build errors that you reported. It's as if either gcc, ar, or ranlib on your system is broken. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Snow Leopard / openpam support
I don't know. The build looks like a build from clean, but the errors from ranlib are totally bizarre. On Sun, 13 Sep 2009, Morsberger David wrote: I wonder if he did a make clean first? I also don't know what version he has. On Sep 13, 2009, at 4:08 PM, Mark Crispin wrote: On Sun, 13 Sep 2009, Nicholas Cole wrote: Thank you for looking at this. I'm sure you've replicated, but here are the build errors, as requested. Is there any workaround? According to information from David Morsberger, it ought to work to remove the setting of MAC_OSX_KLUDGE from the oxp build in the top-level Makefile. But I don't have a clue as to why it would cause the build errors that you reported. It's as if either gcc, ar, or ranlib on your system is broken. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Snow Leopard / openpam support
On Sun, 13 Sep 2009, Nicholas Cole wrote: 13/09/2009 21:51:32 imapd[13109]in pam_sm_authenticate(): Failed to determine Kerberos principal name. This message comes from the PAM library and not from imapd. Apparently, Apple's new PAM library thinks that you want to use Kerberos. I think that they can safely be disregarded unless you are using Kerberos. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] heimdal
On Wed, 2 Sep 2009, Paul Vixie wrote: i have a small patch for compiling uw-imap with heimdal (vs. mit kerberos). what's the process for queuing this up for integration? I'll be happy to look at your patch for inclusion in Panda IMAP. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] [PATCH] remove usage of tmpnam() in ssl_unix.c
On Fri, 14 Aug 2009, Bjoern Voigt wrote: I see the problem, that the existing tmpnam usage may result in an endless loop. This unlikely case can occur for instance if /tmp is read-only for any reason or 100% full. I fail to see why this is a problem. If /tmp (or /var/tmp) is unusable, the system is already crippled until the condition is fixed. If anything, having programs proceed after such an occurence is detected problem worse. That is not the problem that the link warning refers to. The link warning refers to a specific timing race which can be a security issue in some applications, but is of no consequence in this case. What's more, boys and girls, IT NEVER IS EXECUTED at all on any modern system. It is executed ONLY on ancient systems from 20+ years ago which do not have /dev/urandom. Stop fretting about non-problems. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] [PATCH] remove usage of tmpnam() in ssl_unix.c
On Fri, 14 Aug 2009, Andrew Laurence wrote: One ponders, how simpler could UW/Panda become if support for ancient systems were swept aside? Not much, compared to the vast simplicity that would be achieved if every SVR4 system -- particularly Solaris -- were to be exterminated. At least the death warrant was signed for Tru64 nee OSF/1. BSD and Linux are now close enough that their differences don't matter to most programmers. SVR4 is a true disaster of incompatibility. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] [PATCH] remove usage of tmpnam() in ssl_unix.c
On Fri, 14 Aug 2009, Andrew Laurence wrote: What about the mailbox formats? Are there things which could proceed much more smoothly if support for X were dropped? Nope. The drivers are autonomous. The only simplification would be if you got rid of all local file drivers except for mix. But it would be a major project to implement the simplification. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Tracing IMAP session
On Thu, 13 Aug 2009, Thomas Börkel wrote: OK, I thought access to / was restricted by default. It's not. By default, imapd is just like ssh or ftp. So, I will report to Palm, that they should not only search the #mh namespace, right? They should search all or only the default namespace? No. They should search all personal namespaces, not just the last one in the personal namespace list. It's OK to search #mh/, but not to the exclusion of . And provide them this answer from uw-imap: * NAMESPACE (( /)(#mhinbox NIL)(#mh/ /)) ((~ /)) ((#shared/ /)(#ftp/ /)(#news. .)(#public/ /)) Right. There are three personal namespaces there: , #mhinbox, and #mh/. A LIST of each of these on your system will show that only is useful. On other systems it may be a different story. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote.___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Tracing IMAP session
This is normally done from the client side, due to the security sensitivity of IMAP transactions. Check to see if there is some option in your smart phone to enable IMAP session logging. Otherwise, the normal way to do server-based logging is with tee (for non-encrypted sessions) and/or an SSL proxy that does logging. Sadly, most smart phone IMAP clients are pretty bad. Palm once had a good client, but I don't think it ever got built for the Pre. On Wed, 12 Aug 2009, Thomas Börkel wrote: I have some problems with the IMAP client of the Palm Pre phone and would like to debug this. How can I enable tracing all IMAP communication in uw-imap 2007e? I have seen trace files on the net, but cannot find how to enable that. -- Mark -- http://panda.com/mrc Science does not emerge from voting, party politics, or public debate. Si vis pacem, para bellum.___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Tracing IMAP session
On Wed, 12 Aug 2009, Thomas Börkel wrote: Otherwise, the normal way to do server-based logging is with tee (for non-encrypted sessions) and/or an SSL proxy that does logging. How do I do this with tee? I am starting imapd via xinetd and I can do non-encrypted for debugging. I have an executable script: #!/bin/sh tee /tmp/imapd-$$-in | /usr/local/sbin/imapd | tee /tmp/imapd-$$-out and have [x]inetd invoke this instead of the imapd binary. Sadly, most smart phone IMAP clients are pretty bad. Palm once had a good client, but I don't think it ever got built for the Pre. No, it's a totally new client. It works with some IMAP servers, but not very good with uw-imap, yet. So, I'd like to give Palm some feedback. Good luck with that. The fellow who was the developer there was extremely IMAP-clueful, but I have no idea what happened with the product after he left Palm about a year ago. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote.___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Tracing IMAP session
On Wed, 12 Aug 2009, Thomas Börkel wrote: Yes, I heard that. I was using his program on the old PalmOS and it only had one glitch with uw-imap. I don't know why he left Palm. I don't know either. Anyway, the first problem is, that the client cannot find the standard folders or any folder. This is what it happens during initial sync Well, although it is a bit strange that it seems to have listed the #mh/ namespace (which you don't have enabled) but not the default namespace, I don't see anything wrong in the transcript. There's nothing wrong with it trying the #mh/ namespace, but it should have listed the default namespace as well. The only bad thing that might have come from that is that it wouldn't show any mailboxes other than INBOX. The transcript shows that it looked up all messages that arrived since August 5, and it clearly did 30 other commands (probably fetching those messages) before entering IDLE mode. Once it's in IDLE mode, there's no particular reason for it to exit the IDLE unless a new message comes in. So, what is the client doing that you think is wrong? -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote.___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Tracing IMAP session
On Thu, 13 Aug 2009, Thomas Börkel wrote: The only bad thing that might have come from that is that it wouldn't show any mailboxes other than INBOX. Yes, that's exactly the problem for that case. It's not that the client does not work at all. It does only show my INBOX, no other folder (sent, trash or any custom folder). OK, then that's why; it's only attempting to list the #mh/ namespace and not the other personal namespaces. Perhaps it's just listing the last one. If it only lists one, the first one would be a better choice. So, since it insists on only querying the #mh namespace, I tried enabled #mh in uw-imap. What you did isn't sufficient, as you discovered; you actually have to create mailboxes in the mh format. I recommend that you don't do that, and that you delete your .mh_profile file as that will cause more problems than it solves. Instead, you need to figure out how to make your client list the first personal namespace (which is the normal default namespace for any IMAP server). So, what is the client doing that you think is wrong? In this case, it was still syncing. But initial sync should be done after the fetch of the messages of the last 7 days, not take forever. Well, that doesn't seem right. By doing an IDLE, it's clearly done with its interactions with the IMAP server and knows it. So it shouldn't be saying that it's still syncing. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote.___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Problem: mail_fetchtext() incomplete result
Turn on debugging telemetry and you will see the IMAP protocol interactions via mm_dlog(). Is the data being truncated from the server? Note that mm_dlog() will not show the data in literals, but you will see the byte counts and that should indicate if you are getting truncated data. If the data is truncated from the server, that's where you need to persue your further investigations. If it's coming from the server OK, then there is some problem in how you are calling c-client. Are you aware that the buffer used by mail_fetchtext() is used by other functions? This means that you can only use the return data from a mail_fetch***() function until you call another mail_fetch() function. A common mistake is to call mail_fetchheader() and then mail_fetchtext() and expect to use both pointers afterwards. In any case, be sure to verify that the server isn't doing this, especially if it is not one of the standard good ones such as UW/Panda, Cyrus, or Dovecot. -- Mark -- http://panda.com/mrc Science does not emerge from voting, party politics, or public debate. Si vis pacem, para bellum. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] [PATCH] remove usage of tmpnam() in ssl_unix.c
That patch is an EXTREMELY BAD IDEA.
It will prevent the c-client library from building.
There is no such function as mkstmp(). You may be thinking of mkstemp().
But even if you use mkstemp(), this patch will prevent the c-client
library from building on some platforms.
On most systems, the tmpnam() won't ever be executed, except on old
systems which don't have /dev/urandom (and won't have mkstemp() either).
Nevertheless, EVEN IF tmpnam() is executed, there is NO security problem.
The issue that the compiler warning identifies DOES NOT occur in the way
that tmpnam() is used in this code. It is a FALSE ALARM.
#ifdef is another BAD idea. Unless you have the #ifdef perfectly right
for all possible combinations of systems, there will be systems on which
you will either
[1] break the build
[2] create a REAL, critical security problem (and I do mean critical)
All because of a compiler warning that you have been told in the FAQ is a
FALSE alarm.
I understand your good intentions; but sometimes the way to hell is paved
with good intentions. This is one of those cases.
The people who put in that compiler warning also had good intentions.
That doesn't alter the fact that there are cases where their good
intentions are wrong-headed.
On Tue, 14 Jul 2009, David Sommerseth wrote:
The compiler complained about usage of the unsafe tmpnam() function in
ssl_unix.c. This patch changes this to make use of the safe mkstmp().
In the FAQ I see that it is claimed that this part of the code is not
executed on Linux, in particular. If this is the case, I am surprised
this part of the code is not in a removed in a #ifdef block.
The problem gets even a bit more complicated when other programs
linking in the the library statically also comes with the same compiler
warnings.
The code is, from a code perspective, available in the library in binary
form. In this perspective, it might be a possibility that this code is
executed by some software packages.
This was also the only place in the code where I could find tmpnam()
used in the c-client-2007e library.
kind regards,
David Sommerseth
--- a/src/osdep/unix/ssl_unix.c 2008-06-04 20:18:34.0 +0200
+++ b/src/osdep/unix/ssl_unix.c 2009-06-02 13:19:00.0 +0200
@@ -98,7 +98,9 @@
struct stat sbuf;
/* if system doesn't have /dev/urandom */
if (stat (/dev/urandom,sbuf)) {
- while ((fd = open (tmpnam (tmp),O_WRONLY|O_CREAT|O_EXCL,0600)) 0)
+ strncpy (tmp, tmpXX\0, MAILTMPLEN-1);
+ mkstmp (tmp);
+ while ((fd = open (tmp, O_WRONLY|O_CREAT|O_EXCL,0600)) 0)
sleep (1);
unlink (tmp);/* don't need the file */
fstat (fd,sbuf);/* get information about the file */
___
Imap-uw mailing list
Imap-uw@u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
-- Mark --
http://panda.com/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
___
Imap-uw mailing list
Imap-uw@u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] How to manage c-client memory management?
On Tue, 14 Jul 2009, Shawn Walker wrote: I'm trying OP_SHORTCACHE, but this seems to slow the download messages since now each message has to be fetched from the server. Indeed it does. If you want to use early 1990s memory models, you must suffer early 1990s performance problems. there are countries that have very low bandwidth (Australia has 180k) and to download huge files can take a long time. That is why you should cache. Aggressively. Memory is cheaper than bandwidth. The issue with calling mail_free_cache is that the nmsgs get reset back to 0 when I'm trying to get the header, envelope, body or attachment as the message is being downloaded or the user took some action that I need to get the text of the message or attachment. I did not see that you were using mail_free_cache(). That is an internal function that you should never call. The appropriate function, which is documented in internal.txt, is mail_gc(). The issue is how the users is storing their messages on the server, some users has 30,000, 100,000 or some ridiculous amount of messages in a folder. Caching all of those messages consume the memory. Memory is cheaper than bandwidth. I routinely deal with mailboxes of that size. I would not think of slowing things down to save a few pennies of bandwidth. And not everybody has a computer that has 4 GB of memory, some people in eastern Europe is still using computers from the mid nineties running Windows 95! 14 year old computers are suitable for tasks of 14 years ago. They are not suitable for modern tasks. At today's netbook prices, there is no excuse to continue using an obsolete, power-wasting, dinosaur. Try running Outlook on that 100,000 message mailbox on that Windows 95 machine and see how well it does. 4GB of memory costs less than an espresso at Starbucks. However, you should not need anywhere near this. I regularly play with large mailboxes on a Nokia N800 which only has 128MB. From your use of the word download, you are not using the c-client library effectively. If you were, nobody would ever download 30,000 messages in a session, much less 100,000 messages. Look at the Alpine source code for an example of how it is done properly. -- Mark -- http://panda.com/mrc Science does not emerge from voting, party politics, or public debate. Si vis pacem, para bellum. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] How to manage c-client memory management?
On Tue, 14 Jul 2009, Shawn Walker wrote: There are companies that will not upgrade the computer regardless how cheap memory is today. I can't force the customers to buy more memory. Their choice is stark and brutal. Either they buy modern equipment to handle modern requirements, or they pay much more for the software work that will do a mediocre job of getting dinosaur equipment to do a half-assed job of the requirements. A 486 with 16MB memory and Windows 95 is not a suitable platform for dealing with 100,000 message mailboxes. And no, Outlook doesn't handle 100,000 message mailboxes on Windows 95. 14 year old computers are suitable for tasks of 14 years ago. They are not suitable for modern tasks. At today's netbook prices, there is no excuse to continue using an obsolete, power-wasting, dinosaur. Try saying that to the customers that won't budge on upgrading. We are only forced to have to deal with what the customers are using. Nothing can change that. If they are stupid enough to expect 100,000 message mailbox support on a 16MB Windows 95 machine, then they are stupid enough to pay large sums of money forever for software that will never work. 4GB of memory costs less than an espresso at Starbucks. However, you should not need anywhere near this. I regularly play with large mailboxes on a Nokia N800 which only has 128MB. I would say that Nokia N800 does not store all of you message on your phone. It is not a phone, it is an Internet tablet; and this is using Alpine. I can access my 15,000 IMAP folder with my iPhone with 16 GB of memory, but the iPhone does not does download all 15,000 messages, just the first 100 or whatever I have it configured it for. I am not talking about the iPhone client, or any other client written by people who insist upon doing things their way because they don't understand how IMAP should be used. I can access my 35,000 message IMAP mailbox with a Nokia N800 with 128MB RAM, with full scrolling threaded view and access to all messages and not just 100 messages. Think about the story of Columbus' egg. From your use of the word download, you are not using the c-client library effectively. If you were, nobody would ever download 30,000 messages in a session, much less 100,000 messages. The users that we are dealing with will and always will download all 30,000, 100,000 1,000,000 messages to their computer. We cannot control how the user want to use the product. Unless we really cripple how the product work. Users don't think in terms of downloading messages. They think in terms of reading messages. Client authors think in terms of downloading, mostly because they don't understand how else to enable a user to read their mail. The only reason that users need to download is if the client author is unable to envision any other technique for enabling the user to read mail. Downloading is most suitable for saving content (such as attachments) on the local machine. There are other, more suitable, techniques for other tasks. If the only tool that you ever use is a screwdriver, it may seem that all tasks are solved by using screwdrivers. Look at the Alpine source code for an example of how it is done properly. Using Alpine source to how we need to use IMAP is comparing apples and oranges. If we could control how the application is running then we could model after Alpine. Unfortunately, you are making your life much harder for yourself. I have given you as many clues as I can. Unfortunately, there is a limit to my free advice. -- Mark -- http://panda.com/mrc Science does not emerge from voting, party politics, or public debate. Si vis pacem, para bellum. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] How to manage c-client memory management?
On Mon, 13 Jul 2009, Shawn Walker wrote: I have an issue that I would like to tell c-client to flush it's cache since I don't want it to keep headers, body of the messages, etc for 20,000 messages since it take up a lot of memory. Either mail_free_cache() or use short caching (OP_SHORTCACHE mode of open). mail_free_cache wouldn't be the ideal function to call since that wipe out the list of known messages on the server and cannot download any known messages that I want to download (headers only or just the body of the message or the attachments). I have no idea what you mean by this. I have repeatedly read what you wrote, and I can not make any sense of that combination of words. Would be ideal that once I have what I wanted downloaded I want c-client to free all of the memory for that message before I go on to the next message. That sounds like short caching, but unless you are running on MS-DOS in 640K, short caching is a very bad idea. UW stopped using short caching about 10 years ago since it is so horribly inefficient. If you want to download without going through caching, you can set up a mailgets function for that purpose. If you just want to download messages to a local copy, why are you using IMAP at all? POP would be better for download. If you want to do interactive access, then you want full caching. -- Mark -- http://panda.com/mrc Science does not emerge from voting, party politics, or public debate. Si vis pacem, para bellum. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] interface specifications or listening only on specific interfaces?
On Fri, 3 Jul 2009, Linda Walsh wrote: I don't see any options documented to do this -- and was wondering how to restrict what interfaces imap listens on. This is done by your Internet listener, which is xinetd on most modern systems. Check the xinetd documentation for how to handle your internal vs. external services. Is there still official development and bug fixing going on with imap? Not at UW. I have continued development/bug fixing as a hobby project called Panda IMAP, see http://panda.com/imap/ -- Mark -- http://panda.com/mrc Science does not emerge from voting, party politics, or public debate. Si vis pacem, para bellum. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] running out of disk space
On Thu, 2 Jul 2009, Andrew Daviel wrote: I ran out of disk space this morning, and dmail broke the index files on several of my inboxes. This surprises me. The code specifically expands the index and status files by the space needed, and makes sure that it can write that space, before any update. Look, for example, at the section of code with comment need to write additional space? in mix_index_update and mix_status_update. All subsequent writes in both routines are overwriting data which has already been successfully written. Subsequent access with dmail or Pine gave Oversize mix status record Status record would be the status file, not the index file. Did you look at the file in question? What was the oversize data? If it was a bunch of nulls, that would be from the expansion code. Ideally, dmail should fail gracefully. It's supposed to, yes. Note to self - make quite sure we don't run out of space. A good idea in any case. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] bug in imap c-client when using OP_PROTOTYPE
Hi Dave - This is not a bug. Rather, you have misunderstood some important points. There are three issues with your sample program. [1] You SHOULD include c-client.h, and not mail.h directly. mail.h has most, but not all, of the consumer API definitions and prototypes. [2] You MUST (repeat, MUST!!) include linkage.c at the start of your main() function instead of calling mail_link() directly. [3] A prototype stream is not something that can be given to mail_close_full(). Without knowing why you are opening a prototype stream, it appears to me that you do not understand what a prototype stream is and how/why it is used; especially since this is an IMAP prototype stream, something which is almost completely useless except for internal c-client purposes. A prototype stream is not a stream. The closest analog to a prototype stream would be a factory object or class definition. Internally, a prototype stream is simply a pointer to a static area of constant memory that has the dtb for that driver. Prototype streams have VERY limited use to API consumers. The primary consumer use is that of a local filesystem format prototype stream as an argument to mail_create() to force the created mailbox to be in that format. However, that use is deprecated in favor of the #driver.???/ prefix; e.g., mail_create (NIL,#driver.mix/newbox); is the preferred and more modern way of doing mail_create (mail_open (NIL,existingmixmailbox,OP_PROTOTYPE),newbox); The prototype stream method is the way to create a new local filesystem mailbox of the same format as an existing local filesystem mailbox, as opposed to a specified format via the #driver.???/ syntax . This is therefore the 99% reason why any API consumer would use a prototype stream. Your use may be in the remaining 1% (it would have to be, given that it's an IMAP prototype stream), but I suspect that it's really a case of your not understanding what you are doing. Getting back to the subject at hand; since it is a factory object (or class definition), it is inappropriate to call non-factory methods on it. For most API consumers, other than mail_create() with a local filesystem driver prototype, the remaining uses are power tools for master sorcerers. mail_close() and mail_close_full() are completely inappropriate methods to use with a prototype stream, even for a master sorcerer. A prototype stream is not a stream, and is not something that can be closed. When you are finished with a prototype stream, you just drop the pointer. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] empty messages in imap client, mix inbox
On Mon, 15 Jun 2009, Jimmy Dorff wrote: On 6/15/09 9:53 PM, Gary R. Schmidt wrote: What file system are the mix folders on? NFSv3 over tcp to CentOS 5.3 NFS server. NFS server is using XFS on disk. NFS is not supported by the mix format. NFS is a guaranteed Grade A Fancy sure-thing method to corrupt any mix format mailbox. Ditto for mbx format. Using NFS to access a mix or mbx format mailbox is tantamont to playing Russian roulette; you WILL lose. NFS is supported with traditional UNIX format (a.k.a. mbox format), but only in a crippled mode. This is not unique to UW IMAP and Panda IMAP. Cyrus has similar restrictions. Please do not allow yourself to be misled into believing that NFS is a real filesystem. It is not, it never was, and it never will be. Furthermore, NFS is architectually unsound as an NFS back end. NFS is a NAS. So it IMAP. It makes no sense to layer a NAS on top of a NAS. Sorry to be the bearer of bad news. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] speeding searches by pre-indexing
On Fri, 12 Jun 2009, Andrew Daviel wrote: 129,634 messages in 23 hours Yikes! What is that? Indexing time? Search time? Even with all the Unicode cruft, UW IMAP should search that many messages in a few seconds, with the bulk of the time being I/O. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] imapd aborts with header size inconsistent
On Wed, 3 Jun 2009, Andrew Daviel wrote: 2 select head1 * 1 EXISTS * BYE [ALERT] IMAP4rev1 server crashing: header size inconsistent Aborted This is a bug in imapd, or more accurately the traditional UNIX mailbox support module in the c-client library used by imapd. I thought that I had exterminated all possible causes of this problem while I was still at UW. Obviously, I missed one. I was wondering why imapd does not merely return BAD to the SELECT operation, and continue. Or is the thread too screwed up at that point to safely do so ? It is screwed at this point, and quite unsafe to attempt anything further. If you can provide me with a sample mailbox that exhibits the problem, I can look into fixing the problem in Panda IMAP. That might motivate you into becoming a Panda IMAP site... ;-) If you want to try fixing it yourself, have fun and good luck...(he says with an evil chortle). -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
RE: [Imap-uw] Possible buffer overflow in mailutil
On Sun, 31 May 2009, Chris Picciotto wrote: But Panda imap is simply not available. Several sites run Panda IMAP. It is available. Therefore, UW-Imap is effectively non-existent. The one statement does not follow from the other, but that doesn't matter. UW is indeed out of the IMAP business; and pretty much out of the open standards email business as well. I was a long time fan of UW-imap, but Dovecot (with maildir backend) is probably the way go. Dovecot is a reasonable choice. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Possible buffer overflow in mailutil
On Thu, 28 May 2009, Bjoern Voigt wrote: My question is: Are you planning to write a fix or do you plan to accept a fix for the buffer overflow bug? I have nothing to do with fixing bugs in UW IMAP, and have had nothing to do with that for over a year. There are more serious problems that have discovered in UW IMAP since development ended a year ago, yet remained unfixed: http://panda.com/imap Given this and the fact that UW IMAP is a dead project, I think that it's pretty silly to worry about this particular issue. As far as Linux distributions go, I think that they too have more important things to fix. For example: I just spent the past 12 hours repeatedly rebooting (via power cycle) a 64-bit Linux system because Ubuntu switched to a new whiz-bang audio driver that can lock up the CPU. Numerous applications became slower (in the case of one rendering engine, three orders of magnitude slower!) in newer versions of Linux because some pinhead improved glibc so that threading mutexes are in place even when the application is not built to do threading. If you're lucky, the man pages were updated to tell you about the xxx_unlocked() variants that don't do these useless mutexes. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Possible buffer overflow in mailutil
On Wed, 27 May 2009, Bjoern Voigt wrote: Anyway, I think, the bug should be fixed for the next release of UW Imap, Alpine etc. Who can do this? I can help with a patch and with some testing it this is helpful. I fail to see why this is important. The fix is obvious and trival, but I don't see why it is worth wasting time. You agree that this is not a security issue; by its nature getpass() is only usable in shell programs and the interactive prompt makes it unsuitable for scripts. It does not affect Alpine. It only affects mtest (which has other issues and is only a sample program) and mailutil. mailutil has a 1024 byte buffer. How many people have passwords that are that long? So the only crash will be if someone deliberately makes it crash, and to accomplish what purpose? -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Problem deleting folders with Thunderbird
On Fri, 22 May 2009, Andrew Daviel wrote: If in Thunderbird the option server supports folders which can contain both folders and messages is checked No client should ever have, or need, such a configuration option. This is handled by the IMAP protocol. With the option set, the GUI has a single option create subfolder. With it clear, it asks if you want a folder or a directory. The missing piece of your puzzle is that it is perfectly reasonable to have a directory-only object in a dual-use work where a name can be both a mailbox and a directory. Consider USENET newsgroups. The fact that comp.mail.pine and comp.mail.imap both exist does not mean that there is a mailbox called comp or comp.mail. Yet both comp and comp.mail are superior directories of comp.mail.imap and comp.mail.pine. The same thing happens in IMAP mailboxes. You can create directories to hold mailboxes without making them also be mailboxes. The only thing that changes with dual-use mailbox names is to allow a mailbox to also act as a directory. This also happens if you delete a mailbox that has children. This causes the mailbox to become a directory, since the children are not deleted. That is, if you have the following mailboxes: junk junk/crap junk/cruft and you then delete junk, then you still have a directory named junk containing mailboxes junk/crap and junk/cruft. As far as I can see, with MBX or Unix, create test will do open (test, O_RDWR) and create test/ will do mkdir (test). Yes. mbx and unix as single-use mailbox formats. In RFC 3501 you say If the mailbox name is suffixed with the .. hierarchy separator .. the client intends to create mailbox names under this name. Which implies that the client needs to know which to create. Correct. Which is why the server supports folders which can contain both folders and messages option is stupid. Since the server workings are (rightly) hidden from users, and the paradigm is that a folder on a GUI desktop contains both files and folders, users will assume that a folder can contain both subfolders and messages. Why? Is it because of the stupid name folder and what it meant on the Macintosh? I have yet to hear anyone saying that files should contain other files. Yet that is what these poorly-designed GUIs are doing. Maybe I am missing something, but I can't see anything in the protocol for a client to determine whether a server can create subfolders without testing it. That's because that's the wrong thing to test. It has NOTHING to do with what a server can do, and EVERYTHING to do with the underlying mail store. The client needs to declare what type of object to create. Either it creates a directory, or it creates a mailbox. In the case of creating a mailbox, it can then determine -- AFTER the mailbox is created -- whether it is also a directory. That client decision point is ALWAYS there. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Problem deleting folders with Thunderbird
On Thu, 21 May 2009, Andrew Daviel wrote: If in Thunderbird the option server supports folders which can contain both folders and messages is checked No client should ever have, or need, such a configuration option. This is handled by the IMAP protocol. (the default, which is true for MIX) then it creates a folder test with no suffix, listed as such in .mailboxlist. If the option is unchecked, it appends a / when it creates a directory and not when it creates a folder. This indicates that the developers of Thunderbird don't have a clue about IMAP. This has been a problem since the bad old days of Netscape Messenger. In IMAP, CREATE using the form with no hierarchy delimiter suffix means create a container that holds messages (a mailbox), and the form with hierarchy delimiter suffix means create a container that holds other containers (a directory). This has nothing to do with whether or not the server allows a mailbox to be a directory as well. There is no need for that configuration switch. 15 list Trash/test2/* * LIST (\HasNoChildren) / Trash/test2/ 15 OK LIST completed This is because Trash/test2/ matches the pattern but Trash/test2 does not. 16 delete Trash/test2/ 16 OK DELETE completed This is because the name Trash/test2/ resolves as a mailbox. 17 unsubscribe Trash/test2/ 17 NO Not subscribed to mailbox Trash/test2/ This is because Trash/test2/ is not in the mailbox list. The subscription list can best be thought of as being like a bookmarks file. The fact that http://www.example.com and http://example.com might take you to the same page does not mean that the two names match in the bookmarks. 71 create test3f/ 71 OK CREATE completed 72 subscribe test3f/ * NO CLIENT BUG DETECTED: subscribe of non-mailbox directory test3f/ 72 OK SUBSCRIBE completed As the server said, test3f/ is a non-mailbox directory and shouldn't be subscribed. 30 unsubscribe test3f/test4/ 30 NO Not subscribed to mailbox test3f/test4/ Once again, that is not the name that is in the subscription list. I do see an inconsistancy here, that when Thunderbird deletes the message folder it appends a /. However, imapd successfully deletes it even though it does not exactly match the name given while creating it. The name with the / appended works when accessing it as a mailbox. The case where it doesn't work is when you are trying to look up the name in the subscription list. Is it fair to say that Thunderbird has a bug because it uses a different folder name for creation and deletion, Yes. and/or that imapd is inconsistent in its treatment of deletion vs. unsubscription ? No. The behavior of the trailing / form is undefined in the specification for all commands other than CREATE. As a courtesy, imapd allows the trailing / form for command which actually access a mailbox. The subscription commands do not access a mailbox; they merely manipulate a list of names. The LIST command returns that form for the foo/* case in order to verify that the superior exists at all. Otherwise, there would be no difference between foo is empty and foo does not exist. Other servers may behave differently; but in any case any client that uses the suffix form for anything other than CREATE is broken. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Problem with mix folders...
If I had to guess, it would be that the DIR_SIZE macro is miscalculating the size of the direct struct that's assigned to p in line 53, leading to a buffer overflow in line 55. Try adding an assert that verifies that DIR_SIZE(d) is greater than, or equal to, ((d-d_name + strlen(d-d_name) + 1) - d). If that assert bites, that's the cause of the problem. The whole reason why a private Scandir is used is that, for the longest time, Solaris didn't have a scandir() call at all and when it did it was broken. I forget why. Maybe SUN finally figured out how to do a working scandir() call, but if they've broken DIR_SIZE() that is bad news. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Problem deleting folders with Thunderbird
On Thu, 14 May 2009, Brian Hayden wrote: Just to be explicit, the moral of the story is that using subscriptions is not a good idea because every client is stupid in their handling of subs, and worse, they're all stupid in slightly different ways. Correct. The ONLY valid use for subscriptions is if a server exports netnews groups or similar functionality. This allows the user to keep a list of which newsgroups he is subscribed to. Now that netnews is effectively dead, the purpose and use of IMAP subscriptions should die with it. Any client which do a LIST * and subscribe all the names returned is broken by design. This means you, Outlook and Thunderbird. That practice is completely unnecessary, and is based upon false folklore. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Multi threading support in UW IMAP toolkit
On Thu, 30 Apr 2009, Tamal Saha wrote: Hi,I am trying to use UW toolkit in a multi thread application. I am wondering whether the global variables or methods will cause any problem in multi threaded application? Those globals apply to all threads and are supposed to be globals. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: Imap-uw and mbox corruption (Mark Crispin)
On Mon, 6 Apr 2009, David Houlder wrote: I don't think longjmp() is async signal safe. There is safe and there is safe There is safe in the sense of being able to return back to what the program was doing. But in this case, the program has no intention of returning. It's reached an exception; it wants to take a specific action and then exit. In the case of imapd: imapd got some form of time to die signal: a hangup, a termination, a kiss of death. imapd has determined that whatever it was doing, it was NOT an update to the mailbox; it is perfectly alright to abort whatever it was. So, imapd has no intention of continuing what it is doing. imapd simply wants to do the following: (1) If the mailbox traditional UNIX format, it wants to save any unsaved changes. (2) it wants to syslog that it is exiting, and why. (3) it wants to exit. For 15 or so years, imapd simply did this in the signal handler. That worked well; and older versions of libc explicitly supported signal handlers doing this. You could screw up your context as long as you didn't try to go back. Let me emphasize: libc explicitly supported you doing this. Then glibc came along and applied mutexes. Suddenly in newer versions of Linux, imapd would be hanging in the syslog() because it may have been doing a printf() in the main line. And the answer from the glibc developers was that you couldn't do syslog() in a single handler. You have to continue what the program was doing, and somehow in gawdknowswhat code figure out that the signal happened and take the error path. The problem was, the server ended up getting hung, typically in TCP wait on a socket that was dead but somehow failing to fault the IOT on it. So going back to what the program was doing wasn't working out. Lo and behold, in looking at glibc code it appeared that longjmp() unwound the mutexes. And it seems to work. But now we have these wierd corruptions, which have nothing to do with anything since it isn't even writing the file at that point! It's almost as if glibc randomly picks a file descriptor, seeked to 0, and piddled some stuff there. At this point, it looks like the whole exercise is futile. Since glibc has broken how signal handlers used to work, the only way out is not to try to log why the server terminated. Just vanish without a trace. Similarly, don't even try to save updates in traditional UNIX mailbox format ...even though we KNOW that the server wasn't doing anything to the file at the time therefore that file descriptor is completely clean. It's a shame that Linux (and I guess BSD) does not have useful signals any longer. For nearly 40 years, it has been commonplace for a signal handler to take an abort action with logging without going back to what it was doing. That apparently has been improved into abolition. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] flock (fcntl) with QFS Shared FS
Ken - You're stepped on a hornet's nest... I understand what you want to do. At Messaging Architects (MA), we're busy at work on a new generation product that offers a clustered message store. But in the concept of UW imapd, you're setting yourself up for a world of hurt. [1] Almost all network filesystems are completely unsuitable for use with mbx format. With few exceptions -- one being the Cluster File System (CFS) that we developed at MA -- network filesystems do NOT, repeat, NOT!! maintain the data/inode synchronization semantics of a real filesystem. mbx is utterly dependent upon full filesystem synchronization semantics. mbx uses random access updates which MUST be in COMPLETE synchronization at all times. Using mbx with any network filesystem that does not have those semantics is like playing Russian roulette; you WILL eventually lose the game. mix format is less dependent -- in fact, we're working on a variant of mix that uses CAstor! -- but I still wouldn't want to risk a network filesystem that has not be specifically certified for mix. [2] Solaris. By using Solaris, and hence SVR4, you are stuck with the fcntl form of locking. fcntl locking is broken by design. To understand the insanity of using fcntl, consider the following statement from the BSD man pages on fcntl locking: This interface follows the completely stupid semantics of System V and IEEE Std 1003.1-1988 (``POSIX.1'') that require that all locks associated with a file for a given process are removed when ANY file descriptor for that file is closed by that process. This semantic means that applications must be aware of any files that a subroutine library may access. For example if an application for updating the password file locks the password file database while making the update, and then calls getpwname(3) to retrieve a record, the lock will be lost because getpwname(3) opens, reads, and closes the password database. The database close will release all locks that the process has associated with the database, even if the library routine never requested a lock on the database. Another minor semantic problem with this interface is that locks are not inherited by a child process created using the fork(2) function. The flock(2) interface has much more rational last close semantics and allows locks to be inherited by child processes. Flock(2) is recommended for applications that want to ensure the integrity of their locks when using library routines or wish to pass locks to their children. Note that flock(2) and fcntl(2) locks may be safely used concurrently. Not only must the application must be aware of any files that a subroutine library may access, but it must also be aware of file access interactions in multiple subroutine libraries and in multiple instances in the same subroutine library. To put it in a more vivid way: by using Solaris -- even with a local filesystem -- you are playing the same game of Russian roulette, with every cylinder in the revolver is loaded and the safety set. You're pulling strenuously and repeatedly on the trigger, all the time assuring yourself that the safety will hold. In this case, the safety is the flocksim code that I wrote, including its elaborate master/slave communication mechanism. I banged my head against the wall for years in an only partially-successful attempt to keep that code effective. I've seen the safety fail on that revolver many times, and I keep on fixing it; but every couple of years something new comes up to make it fail. I do not recommend Solaris or any other form of SVR4 -- including HP-UX and AIX -- for use with UW or Panda IMAP. Linux and BSD (including Mac OS X) are suitable operating systems. [3] IMAP is a NAS. Network filesystems are also a NAS. This is a fundamental collision. It usually does not make sense; and you're trying to do this in a multi-vendor environment. The new server that I've developed at MA layers IMAP not only over CFS, but over a stateless store protocol. This was a HUGE undertaking, and was accomplished only by divorcing mix from IMAP (the layering goes imap - store - mix - CFS). What's more, numerous changes were made in CFS specifically to accomodate mix; and store was built from the onset to accomodate IMAP. None of these lower-level technologies -- store, mix, CFS -- are intended to be consumed by end users. The consumers are our end-user facing products, such as the IMAP server. So it's less of a layer NAS on top of NAS as it is a how we implemented the NAS. And even with all those advantages, it took us MONTHS to get it working. You don't have any of those advantages. You are attempting to layer one end-user facing NAS on top of another end-user facing NAS. QFS, NFS, AFS, blurdybloopFS, etc. ad nauseum know nothing about what mix wants to do.
Re: [Imap-uw] Imap-uw and mbox corruption
I am aware of the problem, but only recently that it happened on other systems than Solaris. I am now aware that it happens on FreeBSD and Linux. I am astounded by the fact that the first three bytes of the corruption always seem to be CTRL/WCTRL/CCTRL/A, 0x17 0x03 0x01, even on different platforms. I have no idea what is significant about those three bytes, nor why it should suddenly have come up. The only thing that seems related is that it happens in association with the imapd process being killed, and the switch to using setjmp()/longjmp() in the signal handlers. Here is the underlying problem: glibc improved things so that there are numerous new mutexes to cover possible multi-threading, even for non-threaded applications such as imapd. There were other complications: e.g., putc() is now far slower. The impact extends to syslog(). imapd, when it receives a signal to terminate, wants to issue a log message announcing this fact. Thanks to the mutex, it no longer can do so in the signal handler...even when it has no intention of returning back to the interrupted code! Matters are futher complicated in traditional UNIX mailbox format; imapd would like to update the mailbox before it exits (to avoid the problem of lost flags) but once again runs afoul of the mutex. To work around this, I tried to use a setjmp()/longjmp() in the signal handler that would take imapd back to the main command loop and then to code to save and exit. Supposedly, longjmp() is supposed to unwind whatever context occurred since the setjmp(). The patch that I suggested in January 2008 removed the step of saving the mailbox updates after the longjmp(). What this all means is that it should still do the longjmp(), but not write anything further to the file and just syslog() and exit. The reports indicate that this doesn't seem to have fixed the problem. I am working with a FreeBSD site that has experienced the problem to try a more aggressive version of the patch that removes the longjmp() entirely. If it isn't the longjmp(), then I don't know what the hell is going on. If it is the longjmp(), then I'll develop some other way around the issue in Panda IMAP. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] mix reconstructor?
On Thu, 2 Apr 2009, Joe Pruett wrote: sorry, by recover, i meant recover from tape. but i didn't want to just recover on top of the inbox. i'm trying the theory of recovering to a new folder and then copying via imap rather than try to do any rebuild magic. but i'll look at the mixrbld and mixdfix stuff... I think that recovering to a new folder, and then letting the user decide what to do with it, is best. There is some black magic possible, but it's best left to sorcerers who can talk with the user and determine if invoking demons is safe... ;-) -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] reply-to/sender
On Sat, 21 Mar 2009, Oliver Block wrote: does anybody know if the c-client's function imap_rfc822_parse_headers() returns sender and reply-to, even if it is not set in the message!? There is no such function as imap_rfc822_parse_headers() If you are referring to the rfc822_parse_msg_full() function, an internal function that should never be called by applications directly, and to the ENVELOPE structure that is returned, the answer is yes. The ENVELOPE structure is what is returned in IMAP as the ENVELOPE, and that behavior is mandatory. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Unread flag always on after reindexing
On Fri, 13 Mar 2009, Dag Nygren wrote: Why did you run mixrbld and mixdfix? These are very powerful tools that should only be run in specific circumstances. It is too long ago I did this for me to remember the exact reason. I think it had something to do with kmail complaining about an attibute missing for certain mail when doing a search. The only reason to run mixrbld/mixdfix is if the mix driver reports errors and refuses to open the mailbox. You probably ran mixrbld/mixdfix unnecessarily, and that caused your problems. There aren't many imapd messages with the word attribute. If the error message was Bogus attribute list, that means that the IMAP client (kmail) sent faulty IMAP protocol. That would be a bug in kmail, and NOTHING wrong with imapd or the mix mailbox. [Or nothing wrong until the ill-advised usage of mixrbld/mixdfix.] However, since you paraphrased the message, this is only a guess. Please do not paraphrase messages. It wastes time. Sounded very much like a corruption in the mailbox to me, Especially since it happened only in my INBOX. I don't know how you came to that conclusion, but it is almost certainly incorrect. If the mailbox opens, it is not corrupt in any way that mixrbld/mixdfix can remedy. It's like getting heart surgery for a head cold; completely ineffective for the problem, and dangerous in other ways. I saw mixrbld as a means to fix this by rebuilding the mailbox. Ran mixrdfix after this as I saw nothing at all in the mailbox after the rebuild. Now the mails are there, but half of them always marked unread. And cannot be marked as read whatever I do. I don't know what you did with mixrbld/mixdfix, but neither tool does anything with the status file, which is where seen state is kept. I have no idea why seen state would behave that way. Just checked the source code and the actual message was from mixrbld and was the one on row 181, printf (Data file %s UID ran backwards. Isn't that out of sequence. The text out of sequence does not occur in Data file...ran backwards, and thus is a paraphrase. Please don't do that. If you got a Data file...ran backwards message, it is likely that some expunge operation did not complete properly. mixrbld will recover from this. I did check this up and found that you have indicated earlier that it is not a real problem. Also tried mixcvt to copy the mailbox as somewhere suggested, but even the result still had the same problem. mixcvt should have made a clean mailbox. I suggest that you try to find some expert who is local to you to look at it. There is nothing obvious that can be diagnosed from the other side of the world. It is obvious to me that there is more to this story than what you say, but unfortunately I do not have the time to investigate it. Unfortunately, I can't offer the same level of user assistance than I did when I was working at UW, and you seem to need much more assistance than I can offer. I'm sorry. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Best way to refresh the message numbers of a stale folder?
There is no refresh. The concept is meaningless in IMAP, particularly for number of messages and other mailbox state. Mailbox state is always pushed from the server to the client. You need to deal with the underlying cause. If the client state is stalled, then your code is in some way failing to update client state properly. Probably, either your implementation of IDLE is incorrect, or (as suggested in my previous message) you incorrectly expect state to cross from one session to the other without per-session notification. There is nothing that you can do to refresh client state. I understand your desire for a simple palliative. In this case the palliative is neither simple, nor effective, nor existent. I am not hiding some secret technique from you. You can continue searching for such, but will continue to be frustrated until you change course and go about it the right way. Last, but not least, IDLE is not push. In many servers IDLE causes worse battery consumption, and doesn't deliver instantaneous notification for all that (the delay can be up to 1 minute in UW and Panda). If your customers expect push, they will be very disappointed with a product that does IDLE. Apple does not do IDLE on iPhone and iPod Touch at all. RIM does IDLE between BIS and the IMAP server, but does real push (not IDLE or even IMAP at all) to the BlackBerry. Put another way, BIS runs interference and prevents the battery consumption problems caused by IDLE. I am sorry if this free advice is not to your liking. I tell people what they need to know to solve their problem, which is not necessarily they want to hear. On Fri, 13 Mar 2009, Shawn Walker wrote: Mark, c-client as a IMAP client now does support IDLE, multi-threaded (for IDLE and because the Windows application is a multi threaded application) and support asynchronous sessions to be able to handle IDLE. This was done so that the application can use c-client for the IMAP client communication. The code was modified to be able to handle that. I know you are not a fan of IDLE, but our customers wanted the push feature to get e-mails instantaneously instead of the client polling the server X minutes (or seconds if the client want to be aggressive about it). Everything is working great except for one small issue. The client has two IMAP sessions opened in two different threads (in a single process, multi-threaded). The reason is out of my control due to how the windows application was written (but that's another discussion for another day). From the process IDLE thread, the client got the untagged IMAP responses, the client end the IDLE with the DONE command and then the client sent a NOOP, but the message cache is still staled. I'm not here to debate what is wrong in your view with using IDLE, multi-threaded application, using more than one IMAP connections to the IMAP server. I just want a solution to how to get c-client to refresh it's message cache properly without having to disconnect and reconnect to the server. Regards, Shawn Mark Crispin wrote: On Fri, 13 Mar 2009, Shawn Walker wrote: The application has multiple threads with 2 connections to the IMAP server. One of them is for IDLE. This application does not use c-client to do IMAP client. c-client does not support client-end IDLE. Presumably, by thread, you mean threads in a process as opposed to message threads. UW imapd does not run multi-threaded; each IMAP session has its own process. Nor does the c-client library use threads. So, whatever is threading and using IDLE does not seem to have anything to do with c-client or imapd. When something happen on the IDLE thread, the server send a list of untagged IMAP commands to the client of what happened. The server sends untagged IMAP responses, not commands. The IDLE thread see that it need to update a folder, but the IDLE thread has two messages the UID of 100 and 101 (an example). But, UID 101 is doesn't exist anymore, but UID 102 is on the server. So, the IDLE thread request the message cache for UID 102 but c-client doesn't know about 102 in it's message cache due that it only know of UID 100 and 101 and return with a NIL. Hence, the message cache is stale. This makes no sense, so I have to guess what you are talking about. My guess is that client has two IMAP sessions open. One of those sessions did an IDLE command that notified the client of new messages. You expected that the other session would instantaneously know about those new messages, even though that session had not yet been notified. That is not the way IMAP works. Each IMAP session has its own independent state, and is notified of new messages independently. Why do you have two IMAP sessions open on the same mailbox? That, by itself, suggests that you are not using IMAP properly. No well-written application should need more than one IMAP session open to a mailbox at a time
Re: [Imap-uw] Unread flag always on after reindexing
On Fri, 13 Mar 2009, Dag Nygren wrote: In other words: The status file is messed up by my rebuild of the mailbox Comparing the erring .mixstatus with the newly created (by the Mark all read) shows that the last 8-digit hex code is the only difference (except from an occasional flag of course) I don't know why that would be, unless the old file had something like . Now my question is: What does the last number mean? Refer to imap-/docs/mixfmt.txt -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Unread flag always on after reindexing
On Fri, 13 Mar 2009, Dag Nygren wrote: Ok. Didn't see any warnings on them creating problems. Thought it was more like fsck. Just rebuilding the metadata from the raw messages. Good point. Thanks for bringing it up. Yes, these repair tools are not like fsck. Their task is a single-minded effort to create usable .mixindex (mixrbld) and .mix# (mixdfix) files that (by choosing to run these tools) you have said are unusable for some reason. In the course of performing that task they will happily disregard and destroy perfectly valid information. The equivalent to fsck's check functionality is mailutil check. If mailutil check does not snarl, then there is no reason to run mixrbld or mixdfix. An equivalent to the safe fixing functionality of fsck is actually built into the mix code of the c-client library. The mix code is designed to be self-healing; if it sees a problem that it can fix safely, it will do so without even asking you. So, the mixrbld and mixdfix programs are best thought as unsafe fixing, to be deployed when, for some reason, the self-healing safe fixing didn't work. If the mailbox opens, it is not corrupt in any way that mixrbld/mixdfix can remedy. So that is a good enough test? Will remember that in the future. Yes. Of course the current problem indicates tha opposite as it very well opens, but still doesn't work as expected... Whatever anomaly you found is not something that is addressed by mixrbld or mixdfix. I wish that I knew what that anomaly was. However, your solution of deleting the .mixstatus file and recreating is the current way of dealing with .mixstatus issues. Is ther any documentation on mix? docs/mixfmt.txt Just checked the source code and the actual message was from mixrbld and was the one on row 181, printf (Data file %s UID ran backwards. Isn't that out of sequence. The text out of sequence does not occur in Data file...ran backwards, and thus is a paraphrase. Please don't do that. OK. But is does mean the same thing :-) There are numerous things in mix (and IMAP) that are sequenced. There is also a specific datum that is called a sequence. Neither of these are relevant to this warning message. This warning message merely states that it found a message with a lower UID value than one it had seen before. Since mix tends to order data files in ascending order, this is an interesting event worth noting to an expert repairing the mailbox who wants to understand the nature and scope of the damage. But this situation in data files is harmless on its own, and mixrbld is supposed to allow for that. However, the UW version of mixrbld has a bug in handling this particular anomaly. That can cause more serious problems later on. If you got a Data file...ran backwards message, it is likely that some expunge operation did not complete properly. mixrbld will recover from this. Shouldn't a rerun of mixrbld be clean in that case? No. This is a situation in the data files. mixrbld does not change data files. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Best way to refresh the message numbers of a stale folder?
On Fri, 13 Mar 2009, Shawn Walker wrote: I'll just figure a way to get the message cache update properly. The way to figure it out is to fix the bug in your code. I identified the bug. In case you missed it: either your implementation of IDLE is incorrect, or (as suggested in my previous message) you incorrectly expect state to cross from one session to the other without per-session notification. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Unread flag always on after reindexing
On Sat, 14 Mar 2009, Dag Nygren wrote: Wrote a small program that sorts the .mixstatus file and found that things work better. The noticed that the recreated .mixindex also was out of order (on the UID:s) and ran the same sorting on that file. Now everything works perfect. The out of order .mixindex file is the ultimate cause of the problem. It was created by the broken UW version of mixrbld. The mix code itself won't do that. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Best way to refresh the message numbers of a stale folder?
On Thu, 12 Mar 2009, Shawn Walker wrote: What is the best way to refresh a stale folder state? I'm having a issue with one thread that contain a stale UID in it's cache. I don't know what you mean by a folder state, much less a stale folder state or the act of refreshing such; nor what message numbers of a stale folder may be; nor what a thread that contains a stale UID in its cache may be. I know that I could disconnect from the server and reconnect, but is rather expensive to have to wait for the server/client to connect. I'm even more bewildered reading this sentence than I am the previous one. Please explain what behavior that you are seeing, and what behavior you expect in its place. Maybe I'm going senile; I haven't a clue as to what you're talking about. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Unread flag always on after reindexing
I am very confused reading this message. Why did you run mixrbld and mixdfix? These are very powerful tools that should only be run in specific circumstances. What were the exact text of any complaint messages which you received? Those message are very specific. Paraphrasing them in a problem report destroys their usefulness. There are several messages in mixrbld, mixdfix, and the mix driver in c-client; but none contain the string out of sequence. I have no idea what kmail and horde/IMP webmail do. Have you tried access with Alpine? If so, what behavior do you see? On Thu, 12 Mar 2009, Dag Nygren wrote: After running mixrbld my mix-format inbox always shows about half of the box is unread, both in kmail and accessed from my horde/imp webmail. Trying to mark the mails as read doesn't do anything. Have also tried using mixdfix. Doesn't change anything though. Had complaints about mails being out of sequence from one of the tools, don't remember which. The mailbox has once been transformed from MH to mix format, but worked fine for months after that. Could this explain the numerous out of sequence messages from the rebuild process? Any hints on how to fix this. Now it is a bit hard to spot new mails... -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] LIST / hidden files
On Tue, 24 Feb 2009, Oliver Block wrote: The ls command of unix will not show those files or directories unless you use the -a option. On the other hand uw-imapd does show the hidden files. That is correct. If the IMAP server did not list those files, it would be impossible for the IMAP client to show them to the user under any of those circumstances. UW imapd once suppressed listing of dot files. I was made to remove that feature, for the reason stated in the previous paragraph. I look at the prospect of revisiting that decision with about the same as I view a colonoscopy. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Optimize way to retreive UIDs?
On Fri, 20 Feb 2009, Shawn Walker wrote: Sheesh, how slow is the link? Not everybody has a fast link. Yes, but there is slow and there is slow; just as there is fast and there is fast. Algorithms which may be suitable for 300 bps are dreadful for 2G wireless (EDGE or 1x). Algorithms which may be suitable for 2G wireless are less good for 3G (UMTS or EV-DO). I have found that people spend time worrying about bandwidth issues that are ultimately unimportant, and end up blundering into the hell of excessive RTTs. When you talk about fetching 25 UIDs at a time, that is a clue to me that you are about to blunder into RTT hell. That's a suitable number for a bandwidth of 1200 bps. Now, you may be in a time machine that took you back 20 years; but otherwise you are almost certainly have much faster bandwidth. RTTs are deadly. Just watch how long it takes Alpine to gobble a large attachment -- it's due to the RTTs introduced by breaking up a big download into smaller chunks to allow the user to stop the download. Mind you; the Alpine guys had a clue of what a good chunk size would be; yet if you have any kind of broadband it is almost always better to set Prevent Partial Fetching. I'm not sure why I didn't think of using message numbers instead of UIDs. Duh! I'll be using that. A lot of people fail to understand the important properties of message sequence numbers, including those who advocate a simplified version of IMAP that abolishes sequence numbers (while meanwhile adding all sorts of new fluff...witness all the silly new IMAP extensions defined in recent RFCs). It's the perception that to the end user that the product is downloading the messages instead of having the user think the application isn't doing anything for several minutes. I'll be playing around what the number would be for getting the uids in a batch. You need to do SUBSTANTIAL testing. Almost certainly you will want your program to be adaptive rather than wiring in numbers that you picked out of a hat. First, and foremost, keep firmly in mind that a single download is ALWAYS faster than multiple partial downloads. ALWAYS do the single download if there is any way you can get it past the user. If the need is to make the user think that the application has not frozen up, consider doing the download in background. Now, given that we have some cases where you MUST break up the download, here's what you should do: Start with defining the bandwidths that you can reasonably expect to encounter. For most purposes, something like 32 kbps is probably the worst case. Also, you ought to put a high cost on RTTs; at least 1/4 second but I recommend closer to 1/2 second. Oh yeah, also beware the occasional RTT From Hell when your TCP session goes into retransmission; this will really bite you in the ass in any kind of radio (Wi-Fi, mobile phone, etc.). We're talking 2-digit second delays here. Suppose you have 1000 UIDs to fetch. On our 32 kbps link, that will take about 2 seconds as a single download. Now, if we do it 25 at a time, we now find ourselves undergoing a task that may take 42 seconds! Perhaps it is better to do 500 at a time. Now consider the best case. We'll exclude wired broadband on the grounds that we assume that we will always do single download in that case. Wi-Fi and mobile phones (especially 3G) are much faster than 32 kbps, yet they still occasionally get the RTT From Hell. Your calculations become quite different now. There really is no single value or rule that works in both the dialup and the Wi-Fi/mobile phone cases. You have to have multiple tuning for both. This isn't quite rocket science, but it's not seat-of-the-pants either. I have seen entirely too many clients killed by RTTs from badly chosen download strategies, only to hear the developer blame IMAP instead of his poor use of IMAP. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] c-client and smtp
On Fri, 20 Feb 2009, Marian Sorin Nasoi wrote: I get an error stating that: TLS unavailable with this server:smtp.gmail.com Is this a Windows machine? If so, the problem is almost certainly due to anti-virus software on your system that sets itself up as a man-in-the-middle for outgoing mail filtering. Some of these programs have a list of programs which are allowed to evade the filter. Of course, any self-respecting virus writer knows this, and will make his virus masquerade as one of these programs. Disable the outgoing mail filtering in your anti-virus software. It's a useless feature, and it's harmful. Outgoing mail filtering is properly the responsibility of the mail submission server, not the user's PC. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] c-client and smtp
You will also get that error message if you built the c-client library without SSL/TLS support. I didn't mention that possibility because I thought that it would be obvious. On Fri, 20 Feb 2009, Marian Sorin Nasoi wrote: Thanks for the quick response, It is not a windows machine, it's Linux (no anti-virus or any other filtering in place). Any other suggestion? Thanks, Sorin On Fri, Feb 20, 2009 at 9:14 PM, Mark Crispin markrcris...@panda.com wrote: On Fri, 20 Feb 2009, Marian Sorin Nasoi wrote: I get an error stating that: TLS unavailable with this server:smtp.gmail.com Is this a Windows machine? If so, the problem is almost certainly due to anti-virus software on your system that sets itself up as a man-in-the-middle for outgoing mail filtering. Some of these programs have a list of programs which are allowed to evade the filter. Of course, any self-respecting virus writer knows this, and will make his virus masquerade as one of these programs. Disable the outgoing mail filtering in your anti-virus software. It's a useless feature, and it's harmful. Outgoing mail filtering is properly the responsibility of the mail submission server, not the user's PC. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] c-client and smtp
On Fri, 20 Feb 2009, Marian Sorin Nasoi wrote: Also everything works from Alpine (that uses UW IMAP) with the same settings so it can not be caused by filtering, right? Some of the anti-virus programs check for the name of the application and allow it through if it is one of the permitted names. Since you are on Linux, that is probably not the case. Another thing to check is to make sure that there isn't some entry in /etc/hosts that overrides the DNS. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] c-client and smtp
I don't think that the Alpine code is going to help you. The problem is either environmental or is caused by something wrong that you are doing. Do you have the required #include linkage.c at the start of your application's main() function? Forgetting to do that is a common mistake. On Fri, 20 Feb 2009, Marian Sorin Nasoi wrote: I have built the library with ssl support (make slx IP6=4) that should set the SSLTYPE=nopwd' according to the build docs. wrt. the DNS suggestion the log shows the resolved IP ( [Trying IP address [72.14.221.111]] ) so everything fine as far as the name resolving goes. Just in case I tried to set the hostlist to [72.14.221.111]:587/tls/user=gmail_user and the result is the same. Thanks for the suggestions: I think they answer part of my question (i.e. the problem does not seem to be caused by some misconfiguration of the UW IMAP library or the call to it). I guess the next step would be to look into the Alpine sources (the thing I wanted to avoid due to the size of the sources). If you have any other suggestions I would very much appreciate them :) Thanks, Sorin On Fri, Feb 20, 2009 at 9:33 PM, Mark Crispin markrcris...@panda.com wrote: On Fri, 20 Feb 2009, Marian Sorin Nasoi wrote: Also everything works from Alpine (that uses UW IMAP) with the same settings so it can not be caused by filtering, right? Some of the anti-virus programs check for the name of the application and allow it through if it is one of the permitted names. Since you are on Linux, that is probably not the case. Another thing to check is to make sure that there isn't some entry in /etc/hosts that overrides the DNS. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] c-client and smtp
On Fri, 20 Feb 2009, Marian Sorin Nasoi wrote: I have only included c-client.h because I thought that was enough... It isn't. You must do all the linkage I have added the #include linkage.c at the start of your application's main() function and I have to solve an error I get: undefined reference to `mail_versioncheck' Are you linking with the same version of c-client.a that is with the c-client sources (where you have c-client.h, linkage.c, etc.)? If you are using some RPM or other package from a third party distribution, delete it and install the tarball from the UW FTP server: ftp://ftp.cac.washington.edu/mail/imap.tar.Z -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] c-client and smtp
mail_versioncheck() is in the c-client.a library (it's in mail.c) so you should not be getting an undefined by referencing it. What is the EXACT text of the error message? On Fri, 20 Feb 2009, Marian Sorin Nasoi wrote: I am linking against c-client.a that is with the c-client sources, no RPM or other package from a third party because the application has to compile on Windows, Linux and also Mac OS. On Fri, Feb 20, 2009 at 10:21 PM, Mark Crispin markrcris...@panda.com wrote: On Fri, 20 Feb 2009, Marian Sorin Nasoi wrote: I have only included c-client.h because I thought that was enough... It isn't. You must do all the linkage I have added the #include linkage.c at the start of your application's main() function and I have to solve an error I get: undefined reference to `mail_versioncheck' Are you linking with the same version of c-client.a that is with the c-client sources (where you have c-client.h, linkage.c, etc.)? If you are using some RPM or other package from a third party distribution, delete it and install the tarball from the UW FTP server: ftp://ftp.cac.washington.edu/mail/imap.tar.Z -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
