Re: [VM] Info node External Messages
Alan Wehmann writes: I can emphasize the following conclusions from this evidence: 1) Thunderbird is synchronizing a file on my computer's hard drive, matching the contents of the IMAP inbox folder. Yes, indeed, it does! VM's IMAP folders work the same way too. But what shape the cache folders are in, before you force a full synchronization in some way, are internal decisions for Thunderbird as well as VM. They are not meant to be portable across different mail tools. (Note that Thunderbird also keeps essential information about the folder in .msf files. The format of those files is again internal to Thunderbird. If you make any changes to a Thunderbird folder inside VM, VM will delete the .msf file. For Local Folders, that is perfectly fine. Thunderbird can reconstruct the .msf file from the folder itself. For IMAP cache folders, deleting the .msf file can result in loss of information for Thunderbird. So messing with Thunderbird's cache folders inside VM is not recommended.) 2) The setting of 5 for vm-imap-max-message-size is working. Messages in excess of that size are having only their headers downloaded, when vm-get-new-mail is used. The full message is downloaded if I choose it in the Summary buffer. That is good. 3) It is possible to have VM parse the Thunderbird synchronization file (local on my hard drive), but it involves a bit of trickery. Covered in my response to (1). My experience on April 4 (when I was using the IMAP inbox folder as a spool file), of being asked a) if I wanted to download each large file (answer no), and then being asked b) if I wanted to delete it from the maildrop (answer yes . . . bad choice, bad choice, bad choice) has me wondering about the design of those questions. It seems to me rather odd to ask me if I want to delete a large message if I haven't even looked at it (with VM). That was Kyle Jones's design. I guess the idea must have been that, if you don't want to download a message from the server due to its size, then it would be sitting on the server for ever and ever, filling up disk space. So, there must be a way to get rid of it. Looking through the CHANGES file, this has been there probably since VM 6.14 (February 1997), first for POP then adapted to IMAP (July 1998). In those days, mail servers were meant for delivering email, and you were expected to download the email as quickly as possible and delete it from the server. So, it made sense. POP folders were implemented in December 2001. External messages (headers downloading) was planned but never done. Here is a thread on vm.info archives which indicates the plan: https://groups.google.com/forum/?fromgroups#!searchin/gnu.emacs.vm.info/pop$20max$20message$20size/gnu.emacs.vm.info/17qDHuWoIqc/HiuVa_FoIfgJ (Look for Terry W.'s message on 2/21/2002 and Kyle's responses.) Now that we have external messages implemented, that is the preferred way of reading IMAP mail. We should regard the idea of IMAP spool files as a historical relic that is out-of-date and to be avoided at all costs. Cheers, Uday
Re: [VM] Info node External Messages
moritzmaxei...@googlemail.com writes: So, evidence is practically useless! this is not a universal conclusion. If it were, then reverse engineering closed source software would be impossible, because there you do exaxtly that: You try to look for evidence (either by sniffing, disassembling, etc.) that the software does a certain thing. We are going off-topic, so I won't belabour this. Reverse-engineering works only as long as the original developers stick to a particular behaviour. If the behaviour is not documented and published, essentially being made into a contract, the developers can change the behaviour at any time and break interoperability. That was the foundation of the celebrated EU vs. Microsoft case. Even though Microsoft made its source code available, EU said that it wasn't enough. The specifications needed to be published. (One wonders whether Microsoft had any specifications at all in order to divulge them. They learnt the hard way that they had to write specifications, if only to help their own competition!) Coming to the issue at hand, I doubt if Thunderbird developers are any better at writing specifications than the Microsoft developers. The best you will get is stuff like this https://developer.mozilla.org/En/Folders. The reason we are able to inter-operate with Thunderbird is not that Thunderbird is well-documented, but rather that they have chosen to base their folder format on a standard RFC, the same one we use. (RFC 4155). Nothing stops Thunderbird from deviating from the RFC, but in all likelihood, they won't. On the other hand, for IMAP cache folders, the very first line is a violation of the RFC. So, in a way, they are warning us that this is not an RFC-client folder and we shouldn't mess with it. Cheers, Uday Even for well documented software the conclusion only applies if what you want to to with it is part of what is documented, which is not always the case. Granted, in the context of this discussion (viewmail/thunderbird) the conclusion does hold but only because both are well documented and what he wants to do is covered in it. Bye, Moritz
Re: [VM] Info node External Messages
I reduced the number of messages in my IMAP server inbox to six, using the server's web interface. I then started up Thunderbird and checked the Options setting that synchronizes such messages to a file on my computer. The result is a file on my local computer that looks as follows (from a Dired buffer): c:/Documents and Settings/Alan/Application Data/Thunderbird/Profiles/qn5qd3de.default/ImapMail/email.fnal.gov: -rw-rw-rw- 1 Alan None 43765 04-07 13:43 INBOX That INBOX file starts out as follows: From Received: from gateway01.fnal.gov (131.225.104.18) by MAIL05V-CAS02.fnal.gov (131.225.199.7) with Microsoft SMTP Server id 14.1.355.2; Wed, 7 Mar 2012 10:04:17 -0600 Received: from localhost (localhost.localdomain [127.0.0.1]) by gateway01.fnal.gov (Postfix) with ESMTP id 5BAE5D80F06; Wed, 7 Mar 2012 10:04:17 -0600 (CST) X-Virus-Scanned: amavisd-new at fnal.gov Received: from gateway01.fnal.gov ([127.0.0.1]) by localhost (gateway01.fnal.gov [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 8NhI+x9lwUmj; Wed, 7 Mar 2012 10:04:17 -0600 (CST) X-Mailgw-Auth: no Received: from [127.0.0.1] (adhq107679-lt.dhcp.fnal.gov [131.225.169.179]) by gateway01.fnal.gov (Postfix) with ESMTP id 786C6D80C93; Wed, 7 Mar 2012 10:04:16 -0600 (CST) Message-ID: 4f5786fe.9060...@fnal.gov Date: Wed, 7 Mar 2012 10:04:14 -0600 One reason to show how it starts off is to illustrate that the first line is not recognized by the value of vm-leading-message-separator-regexp-From_ inside the function vm-skip-past-folder-header, so this file cannot be parsed by function vm-build-message-list (I had hoped to show a Summary buffer for this file). I suppose that such files are also built when Thunderbird is made to go offline, but that is somewhat off-topic. In the place of a Summary buffer, I can show the following (from the use of function occur): 5 matches for ^Subject: in buffer: INBOX 57:Subject: Next quarterly lunch Wednesday, April 11 [retired scientist list] 127:Subject: LIBRARY - You borrowed this book 208:Subject: Next quarterly lunch Wednesday, April 11 [retired scientist list] 365:Subject: Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26) 551:Subject: Type 3X Faster with Your Voice - 50% Off Dragon (for a later comparison with the Summary buffer from ~/mail_rmail/imap-cache-d0925bcceba88ad7703984efcad1f482) I had a size limit of 50K imposed one of the six messages in my IMAP inbox was not therefore not displayed when I went offline in Thunderbird. In VM I made/have the settings illustrated below (copied from the *scratch* buffer, using function eval-print-last-sexp): (format %s\n%s\n %s\n%s\n%s\n vm-primary-inbox vm-imap-account-alist vm-enable-external-messages vm-imap-max-message-size vm-imap-sync-on-get) imap:localhost:143:inbox:login:wehmann:* ((imap:localhost:143:*:login:wehmann:* fermi_email)) (imap) 5 nil so that I can use VM to access the IMAP server in the manner that you favor. I used the command vm-visit-imap-folder and answered fermi_email:inbox. I also used vm-get-new-mail, as needed (e.g. to get asked the password). The Summary buffer that resulted I show below: - 1 UDick Carrigan Mar 7 37/5605 Next quarterly lunch Wednesday, April 11 [retired scientist list] 2 libr...@fnal.gov Mar 16 16/2025 LIBRARY - You borrowed this book 3 Dick Carrigan Apr 6 91/7806 Next quarterly lunch Wednesday, April 11 [retired scientist list] 4 Mark Messier Apr 7 137/10825 Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26) 5 UWinZip Computing Apr 5 293/17469 Type 3X Faster with Your Voice - 50% Off Dragon 6 Fermilab TodayApr 60/113632 Fermilab Today - Friday, April 6 Message #6 is the one that is greater than 50K (the summary buffer indicates that its size is 113632 bytes). For message #6 there is no indication that it was left on the server because of being over-sized. The buffer *Messages* said for it: Retrieving message 1 (of 1) from fermi_email:inbox, 69%... inbox: Decoding MIME message... inbox: Inlining text/html by emacs-w3m... WARNING: Could not load feature w3m. WARNING: Related functions may not work correctly! inbox: Inline text/html by emacs-w3m display failed: Symbol's value as variable is void: w3m-cid-retrieve-function-alist inbox: Recreating summary... done inbox: Decoding MIME message... done (Clearly something is wrong with my setup of w3m.) I was not asked any question about whether or not I wished to download message #6, nor was I asked any question about whether or not I wished to delete it from the maildrop. Cheers, Alan Uday Reddy writes: To: awehm...@sbcglobal.net Subject: Re: [VM] Info node External Messages Date: Fri, 6 Apr 2012 23:52:55 +0100 From: Uday Reddy usr.vm.ro...@gmail.com Alan Wehmann writes
Re: [VM] Info node External Messages
Alan Wehmann writes: One reason to show how it starts off is to illustrate that the first line is not recognized by the value of vm-leading-message-separator-regexp-From_ You can't read Thunderbird's IMAP folders in VM, only the Local Folders. (In fact, you can see from the size of the file that it is not a full folder anyway.) 6 Fermilab TodayApr 60/113632 Fermilab Today - Friday, April 6 For message #6 there is no indication that it was left on the server because of being over-sized. The 0 in the size indicator (presumably 0 lines?) tells you that the message wasn't downloaded. What happens if you try to read the message 6? I was not asked any question about whether or not I wished to download message #6, nor was I asked any question about whether or not I wished to delete it from the maildrop. Indeed, that is how it is supposed to work. It is designed to be painless. If you delete message 6, expunge and save the folder, then message 6 will get deleted on the server. Cheers, Uday
Re: [VM] Info node External Messages
customization value. In this case ~/INBOX should not be confused with INBOX above, since they are in different directories. File ~/INBOX is what I have been using with a Yahoo POP server. It was also used in a VM session on April 4, when I was using the IMAP inbox as a spool file. As for the size numbers being different on the Summary buffers for INBOX and inbox, that I assume is because inbox has attributes saved in it and INBOX does not. As a further test, I added another large message to the IMAP inbox folder. The Dired entry for INBOX became: -rw-rw-rw- 1 Alan None 387242 04-07 22:32 INBOX Its Summary buffer became: - 1 NDick Carrigan Mar 7 36/1573 Next quarterly lunch Wednesday, April 11 [retired scientist list] 2 Nlibr...@fnal.gov Mar 16 15/375LIBRARY - You borrowed this book 3 NDick Carrigan Apr 6 91/3729 Next quarterly lunch Wednesday, April 11 [retired scientist list] 4 NMark Messier Apr 7 137/6457 Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26) 5 NWinZip Computing Apr 5 293/15111 Type 3X Faster with Your Voice - 50% Off Dragon 6 NFermilab TodayApr 6 1811/113631 Fermilab Today - Friday, April 6 7 Fermilab TodayApr 5 3123/218957 Fermilab Today - Thursday, April 5 (after making sure that there was a blank line ahead of each From line--as before). In the inbox buffer I did the command vm-get-new-mail. After a significant delay it did so and the Summary buffer became: - 1 UDick Carrigan Mar 7 37/5605 Next quarterly lunch Wednesday, April 11 [retired scientist list] 2 libr...@fnal.gov Mar 16 16/2025 LIBRARY - You borrowed this book 3 Dick Carrigan Apr 6 91/7806 Next quarterly lunch Wednesday, April 11 [retired scientist list] 4 Mark Messier Apr 7 137/10825 Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26) 5 WinZip Computing Apr 5 293/17469 Type 3X Faster with Your Voice - 50% Off Dragon 6 Fermilab TodayApr 6 1812/113632 Fermilab Today - Friday, April 6 7 Fermilab TodayApr 50/225037 Fermilab Today - Thursday, April 5 when I stepped to message 7 in the Summary buffer, more was fetched from the IMAP server and the inbox Summary buffer became: 1 UDick Carrigan Mar 7 37/5605 Next quarterly lunch Wednesday, April 11 [retired scientist list] 2 libr...@fnal.gov Mar 16 16/2025 LIBRARY - You borrowed this book 3 Dick Carrigan Apr 6 91/7806 Next quarterly lunch Wednesday, April 11 [retired scientist list] 4 Mark Messier Apr 7 137/10825 Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26) 5 WinZip Computing Apr 5 293/17469 Type 3X Faster with Your Voice - 50% Off Dragon 6 Fermilab TodayApr 6 1812/113632 Fermilab Today - Friday, April 6 - 7 Fermilab TodayApr 5 3124/218959 Fermilab Today - Thursday, April 5 I can emphasize the following conclusions from this evidence: 1) Thunderbird is synchronizing a file on my computer's hard drive, matching the contents of the IMAP inbox folder. 2) The setting of 5 for vm-imap-max-message-size is working. Messages in excess of that size are having only their headers downloaded, when vm-get-new-mail is used. The full message is downloaded if I choose it in the Summary buffer. 3) It is possible to have VM parse the Thunderbird synchronization file (local on my hard drive), but it involves a bit of trickery. My experience on April 4 (when I was using the IMAP inbox folder as a spool file), of being asked a) if I wanted to download each large file (answer no), and then being asked b) if I wanted to delete it from the maildrop (answer yes . . . bad choice, bad choice, bad choice) has me wondering about the design of those questions. It seems to me rather odd to ask me if I want to delete a large message if I haven't even looked at it (with VM). Cheers, Alan Uday Reddy writes: To: alan.wehm...@gmail.com Subject: Re: [VM] Info node External Messages Date: Sat, 7 Apr 2012 21:26:22 +0100 From: Uday Reddy usr.vm.ro...@gmail.com Alan Wehmann writes: One reason to show how it starts off is to illustrate that the first line is not recognized by the value of vm-leading-message-separator-regexp-From_ You can't read Thunderbird's IMAP folders in VM, only the Local Folders. (In fact, you can see from the size of the file that it is not a full folder anyway.) 6 Fermilab TodayApr 60/113632 Fermilab Today - Friday, April 6 For message #6 there is no indication that it was left on the server because of being over-sized. The 0 in the size indicator (presumably 0 lines?) tells you that the message wasn't downloaded. What happens if you try to read
Re: [VM] Info node External Messages
Alan Wehmann writes: For me there is a mode of operation where I like to download selected IMAP messages into Local Folders have a local copy. As I indicated in one of my messages, I need to use the web interface to the IMAP server to clean out the IMAP server inbox. I would use the web interface for this, since then the limited DSL upload speed is not a factor when moving messages from one folder to another on the server. If you are really sure that you wan to use Local Folders (I still don't see why), then please be aware that you are using functionality that Thunderbird doesn't provide. So, you can't expect VM to behave like Thunderbird. As I said previously, I would prefer to get rid of Local Folders for IMAP because they are confusing and don't work well. Uday Reddy writes: On the other hand, the vm-get-new-mail was invoked automatically. So, VM should not have asked you yes/no questions. I will check why that is happening. It looks like this is happening because we have changed the automatic `vm-get-new-mail' to be regarded as an interactive use so that people can type in passwords when they do `M-x vm'. So, it now asks yes/no questions during `M-x vm' as well. I guess I will need to add another flag to say that it is ok to ask for password, but not ok to ask yes/no questions. Cheers, Uday
Re: [VM] Info node External Messages
Alan Wehmann writes: I tried out some of what is described in Info node External Messages yesterday. Hi Alan, I can't tell what you tried from this info node, but the rest of your message indicates to me that you were using Local Folders. External messages are only available for IMAP Folders. So, I would expect that whatever you tried from the info node would not have had any effect. Since you say that you chose to delete the downloaded messages from the server, the only copies you have now are in the Local Folder. So, please be very careful with what you do with them. They are gone from the server. My impression is that Thunderbird doesn't really have Local Folders for IMAP servers. They are a bad idea. If I had a choice, I would get rid of them. But Kyle Jones implemented them and a lot of people use them. So, it appears that my hands are tied. The variable `vm-imap-max-message-size' is also used for downloading messages into Local Folders but I notice that it wasn't documented in the info node. I will fix that. Cheers, Uday
Re: [VM] Info node External Messages
Alan Wehmann writes: IMAP server at work. Since I am used to how Thunderbird implements its size limitations I was surprised to find myself being asked yes or no for each over-sized message. The doc-string for `vm-imap-max-message-size' says that the yes/no question is asked only when `vm-get-new-mail' is invoked interactively. It is not supposed to be asked if you did `vm-visit-folder' and had `vm-auto-get-new-mail' set to t. Did that work correctly? Cheers, Uday
Re: [VM] Info node External Messages
Alan Wehmann writes: vm-spool-files '(imap:127.0.0.1:143:inbox:login:*:*) I used the command vm to initiate the downloading of messages from the IMAP server. Right, you have used an IMAP spool file to download messages into a local folder, the vm-primary-inbox. So, the external message stuff doesn't apply. On the other hand, the vm-get-new-mail was invoked automatically. So, VM should not have asked you yes/no questions. I will check why that is happening. There were 533 messages in the IMAP inbox. Roughly 35 of these were greater than 5 in size. For each of these large messages I was asked if I wished to download them. For each I responded no. I was then asked if I wanted to delete them from the maildrop. In a confused state, I answered yes (not a good answer). As a result they were completely gone. Were only the downloaded messages gone, or did the skipped messages go away too? That would be quite a serious bug if the skipped messages got deleted. But nobody reported anything like it all these years! Cheers, Uday
Re: [VM] Info node External Messages
On 4/5/2012 5:42 PM, Uday Reddy wrote: Alan Wehmann writes: vm-spool-files '(imap:127.0.0.1:143:inbox:login:*:*) I used the command vm to initiate the downloading of messages from the IMAP server. Right, you have used an IMAP spool file to download messages into a local folder, the vm-primary-inbox. So, the external message stuff doesn't apply. On the other hand, the vm-get-new-mail was invoked automatically. So, VM should not have asked you yes/no questions. I will check why that is happening. There were 533 messages in the IMAP inbox. Roughly 35 of these were greater than 5 in size. For each of these large messages I was asked if I wished to download them. For each I responded no. I was then asked if I wanted to delete them from the maildrop. In a confused state, I answered yes (not a good answer). As a result they were completely gone. Were only the downloaded messages gone, or did the skipped messages go away too? That would be quite a serious bug if the skipped messages got deleted. But nobody reported anything like it all these years! Cheers, Uday The messages that were under 5 bytes in size were downloaded to my local file ~/INBOX (and are still present on the IMAP server). For the ones that were greater than 5 bytes in size, I was asked (1) did I want them downloaded (I answered no), (2) did I want them deleted from the maildrop (I answered yes--a bad answer). In summary, the messages 50K are still on the IMAP server; the ones 50K are not. The messages 50K are also in my file ~/INBOX; the messages 50K are not. -- Alan Wehmann alan.wehm...@gmail.com
