Bug#433531: offlineimaps produces doublicate mails
On Tue, Jul 31, 2007 at 02:03:33PM -0500, John Goerzen wrote: Hi John, I have looked over the log and, looking at that first date you reported, I see it was in the INBOX.mls to start with with UID 79603. Later, it also appeared with UID 79608. But OfflineIMAP had not ever called APPEND to write a new message into that folder. The inescapable conclusion is that something else had duplicated it, or you are syncing to a local Maildir that is the physical mail store for the IMAP server (a broken and pointless exercise). Hm no. There is nothing else to doublicate it. Why this would have started with the newer OfflineIMAP version, I have no idea. Ok let's try to see it from another perspective again. When I open a folder localy before I sync again everything works as it should so let's brake it down to where the mail is and what happesn. Remote: New mail A in new, others in cur. Local: Nothing in new, others in cur. sync Remote: New mail A in new, others in cur. Local: Mail A in new, others in cur. Scenario 1: Now let's say I read my new mail A and a new mail arrives on the server. So I sync. Remote: New mail B in new, others in cur mail A get's moved to cur. Local: New mail B in new, mail A now in cur. Scenario 2: I'm lazy and don't read mail A still hanging arround in new. Now I sync again because I really want to get mail B. Current state: Remote: A and B in new, rest in cur. Local: A in new, rest in cur. Then I sync and get Remote: A and B in new, rest in cur. Local: A,A and B in new, rest in cur. So this is the condition where the sync process seems to be broken. Based on that pattern I'd guess that the sync process simply doesn't take in account that there are still mails in new. Mails for the cur part of the maildir are synced properly. Maybe that makes it a little clearer and can maybe shed some light on were to search for the root of the problem. I'll try to get my hands dirty on it in a few days anyway. Guess I'll first try to reproduce it on a second system ... Sven -- If you won't forgive me the rest of my life Let me apologize while I'm still alive I know it's time to face all of my past mistakes [Less than Jake - Rest Of My Life] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#433531: offlineimaps produces doublicate mails
On Thu July 19 2007 12:01:18 pm Sven Hoexter wrote: On Wed, Jul 18, 2007 at 04:27:08PM -0500, John Goerzen wrote: Try offlineimap -1 -d imap or -1 -d maildir. The other change that was made was switching to Python's stock imaplib.py, which may have contributed. I haven't seen the behavior myself, so I am somewhat suspicious of the IMAP side of things too. Ok I've attached a logfile to this message with -1 -d imap,maildir output. All password lines got removed (at least I hope so ;). I have looked over the log and, looking at that first date you reported, I see it was in the INBOX.mls to start with with UID 79603. Later, it also appeared with UID 79608. But OfflineIMAP had not ever called APPEND to write a new message into that folder. The inescapable conclusion is that something else had duplicated it, or you are syncing to a local Maildir that is the physical mail store for the IMAP server (a broken and pointless exercise). Why this would have started with the newer OfflineIMAP version, I have no idea. -- John -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#433531: offlineimaps produces doublicate mails
On Sat, Jul 21, 2007 at 12:58:14AM -0700, Asheesh Laroia wrote: Hi Asheesh, Sven, is that what it seems to you, too? Yes. Maybe we can try looking at a diff of the last 4.x to the first 5.x. I guess that I'll do something like this when I've some more free time in about two weeks. Cheers, Sven -- If you won't forgive me the rest of my life Let me apologize while I'm still alive I know it's time to face all of my past mistakes [Less than Jake - Rest Of My Life] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#433531: offlineimaps produces doublicate mails
John, it sounds like my patch was a red herring. That makes sense to me because I can't think of anything wrong with my beautiful change. (-; Sven, is that what it seems to you, too? Maybe we can try looking at a diff of the last 4.x to the first 5.x. -- Asheesh. -- Kleeneness is next to Godelness. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#433531: offlineimaps produces doublicate mails
On Tue, Jul 17, 2007 at 03:39:13PM -0500, John Goerzen wrote: On Tue July 17 2007 3:20:19 pm Sven Hoexter wrote: On Tue, Jul 17, 2007 at 02:17:20PM -0500, John Goerzen wrote: Moin, I am suspicious of one particular patch here. I think this is the only code path involved that has been modified recently. Can you try applying this to the OfflineIMAP tree with patch -p1 -R and see if it helps things for you? I haven't seen this myself so your assistance tracking it down will be helpful. Seems to work so far. Thanks for this fast reply. :) OK. If that holds, then I will back out your patch for 5.99.2, Asheesh. Could you debug it, and see if you can track down the problem? If you can, I'll be happy to re-include it in future versions. *sigh* It's happening again. I'm not yet sure if it has changed the conditions a little bit. I've now downloaded 4.0.16 and 5.99.0 and will try to find out this evening where exactly the problem started. Cheers, Sven -- If you won't forgive me the rest of my life Let me apologize while I'm still alive I know it's time to face all of my past mistakes [Less than Jake - Rest Of My Life] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#433531: offlineimaps produces doublicate mails
Hi, the problem has been introduced with the 5.x version so maybe there is a way to get complete debug log with every step that offline imap does where it might be possible to see why it thinks that some messages are still new and need to be fetched? I'll try my best to narrow something down in the -d maildir output but it's still a lot of information to sort before you can use it to find a special pattern in it. :( Cheers, Sven -- If you won't forgive me the rest of my life Let me apologize while I'm still alive I know it's time to face all of my past mistakes [Less than Jake - Rest Of My Life] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#433531: offlineimaps produces doublicate mails
On Wed July 18 2007 3:57:05 pm Sven Hoexter wrote: Hi, the problem has been introduced with the 5.x version so maybe there is a way to get complete debug log with every step that offline imap does where it might be possible to see why it thinks that some messages are still new and need to be fetched? Try offlineimap -1 -d imap or -1 -d maildir. The other change that was made was switching to Python's stock imaplib.py, which may have contributed. I haven't seen the behavior myself, so I am somewhat suspicious of the IMAP side of things too. I appreciate your willingness to help track this down! -- John -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#433531: offlineimaps produces doublicate mails
Package: offlineimap Version: 5.99.1 Severity: important Hi, I'm using offlineimap to checkout mails from a courier imap server with ssl and read my mails on the local system with mutt. Since the last update of offline imap I've constantly doublicated mails in my various mailboxes. I think that's mostly happening when I re-run offline imap without opening a mailbox with new mails and there are again new mails. Let's say I've got a bunch of new mails in my INBOX maildir and my INBOX.mls dir and only read the mails in INBOX I'll have doublicate emails after the next offlineimap run when there are new mails to fetch for INBOX.mls. If there are no new mails to sync for INBOX.mls everything is fine. In cases where I've got new mails in INBOX.mls maidir the old and still new mails are added again. I'm quite unsure how to properly debug the issue so that you've a chance to maybe find out why that's happening. So maybe you've a hint for me. Thanks for creating offlineimap anyway. Very helpfull tool so far. :) Cheers, Sven -- System Information: Debian Release: lenny/sid APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: i386 (i686) Kernel: Linux 2.6.22-rc4-git7 (PREEMPT) Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Versions of packages offlineimap depends on: ii python2.4.4-6An interactive high-level object-o ii python-support0.6.4 automated rebuilding support for p offlineimap recommends no packages. -- no debconf information -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#433531: offlineimaps produces doublicate mails
Hi Sven,
I am suspicious of one particular patch here. I think this is the only code
path involved that has been modified recently.
Can you try applying this to the OfflineIMAP tree with patch -p1 -R and see
if it helps things for you?
I haven't seen this myself so your assistance tracking it down will be
helpful.
Thanks,
- John
Tue Jun 12 17:42:15 CDT 2007 Asheesh Laroia [EMAIL PROTECTED]
* only-write-once-to-cur-or-new.patch
I have tested this and Dovecot no longer beats offlineimap
to the punch. (-:
I achieved that by keeping the renames in tmp/ until it finally does
one last rename in new/ or cur/.
diff -rN -u old-offlineimap/offlineimap/folder/Maildir.py new-offlineimap/offlineimap/folder/Maildir.py
--- old-offlineimap/offlineimap/folder/Maildir.py 2007-07-17 14:16:20.812752215 -0500
+++ new-offlineimap/offlineimap/folder/Maildir.py 2007-07-17 14:16:20.816752617 -0500
@@ -130,6 +130,8 @@
return st.st_mtime
def savemessage(self, uid, content, flags, rtime):
+# This function only ever saves to tmp/,
+# but it calls savemessageflags() to actually save to cur/ or new/.
ui = UIBase.getglobalui()
ui.debug('maildir', 'savemessage: called to write with flags %s and content %s' % \
(repr(flags), repr(content)))
@@ -140,12 +142,9 @@
# We already have it.
self.savemessageflags(uid, flags)
return uid
-if 'S' in flags:
-# If a message has been seen, it goes into the cur
-# directory. CR debian#152482, [complete.org #4]
-newdir = os.path.join(self.getfullname(), 'cur')
-else:
-newdir = os.path.join(self.getfullname(), 'new')
+
+# Otherwise, save the message in tmp/ and then call savemessageflags()
+# to give it a permanent home.
tmpdir = os.path.join(self.getfullname(), 'tmp')
messagename = None
attempts = 0
@@ -179,20 +178,21 @@
os.utime(os.path.join(tmpdir,tmpmessagename), (rtime,rtime))
ui.debug('maildir', 'savemessage: moving from %s to %s' % \
(tmpmessagename, messagename))
-os.link(os.path.join(tmpdir, tmpmessagename),
-os.path.join(newdir, messagename))
-os.unlink(os.path.join(tmpdir, tmpmessagename))
+if tmpmessagename != messagename: # then rename it
+os.link(os.path.join(tmpdir, tmpmessagename),
+os.path.join(tmpdir, messagename))
+os.unlink(os.path.join(tmpdir, tmpmessagename))
try:
# fsync the directory (safer semantics in Linux)
-fd = os.open(newdir, os.O_RDONLY)
+fd = os.open(tmpdir, os.O_RDONLY)
os.fsync(fd)
os.close(fd)
except:
pass
self.messagelist[uid] = {'uid': uid, 'flags': [],
- 'filename': os.path.join(newdir, messagename)}
+ 'filename': os.path.join(tmpdir, messagename)}
self.savemessageflags(uid, flags)
ui.debug('maildir', 'savemessage: returning uid %d' % uid)
return uid
@@ -203,6 +203,7 @@
def savemessageflags(self, uid, flags):
oldfilename = self.messagelist[uid]['filename']
newpath, newname = os.path.split(oldfilename)
+tmpdir = os.path.join(self.getfullname(), 'tmp')
if 'S' in flags:
# If a message has been seen, it goes into the cur
# directory. CR debian#152482, [complete.org #4]
@@ -225,6 +226,10 @@
self.messagelist[uid]['flags'] = flags
self.messagelist[uid]['filename'] = newfilename
+# By now, the message had better not be in tmp/ land!
+final_dir, final_name = os.path.split(self.messagelist[uid]['filename'])
+assert final_dir != tmpdir
+
def deletemessage(self, uid):
if not uid in self.messagelist:
return
Bug#433531: offlineimaps produces doublicate mails
On Tue, Jul 17, 2007 at 02:17:20PM -0500, John Goerzen wrote: Hi John, I am suspicious of one particular patch here. I think this is the only code path involved that has been modified recently. Can you try applying this to the OfflineIMAP tree with patch -p1 -R and see if it helps things for you? I haven't seen this myself so your assistance tracking it down will be helpful. Seems to work so far. Thanks for this fast reply. :) Cheers, Sven -- If you won't forgive me the rest of my life Let me apologize while I'm still alive I know it's time to face all of my past mistakes [Less than Jake - Rest Of My Life] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#433531: offlineimaps produces doublicate mails
On Tue July 17 2007 3:20:19 pm Sven Hoexter wrote: On Tue, Jul 17, 2007 at 02:17:20PM -0500, John Goerzen wrote: Hi John, I am suspicious of one particular patch here. I think this is the only code path involved that has been modified recently. Can you try applying this to the OfflineIMAP tree with patch -p1 -R and see if it helps things for you? I haven't seen this myself so your assistance tracking it down will be helpful. Seems to work so far. Thanks for this fast reply. :) OK. If that holds, then I will back out your patch for 5.99.2, Asheesh. Could you debug it, and see if you can track down the problem? If you can, I'll be happy to re-include it in future versions. Cheers, Sven -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
