Re: broken mdboxes
On 7/14/19 11:30 PM, Aki Tuomi via dovecot wrote: >> Try with one mailbox first. >> >> Delete dovecot.index.map files and run force-resync. >> >> --- >> Aki Tuomi > > Sorry I ment dovecot.map.index there Hey, that seemed to work! I'll need to do more testing because I still have those false mdbox files in the directories that need to be cleared out but I have a process for that too. Thanks again! Mike-
Re: broken mdboxes
On 15/07/2019 09:22 Aki Tuomi via dovecot wrote: On 15/07/2019 09:00 Mike via dovecot < dovecot@dovecot.org> wrote: Hello, I was forced to do an emergency server migration with about 3000 mailboxes and I botched it somewhat. Hoping to find some help here. My symptoms are, users showing apparently deleted mail messages, some replicated mail, and a time period of mail that is just 'missing'. The source server is centos 6.10 with dovecot 2.2.5, and the destination is ubuntu 16 / dovecot 2.2.22. User mailboxes are mdbox format and I did an rsync from the old to the new. My screw up however was that my rsync was not --delete files, and had to be run multiple times over the course of a few weeks due to to bandwidth disparities and the sheer size of the mail spool approaching 700GB (source server had 10mbps. Should have used sneakernet instead). The bottom line however is that I wound up having many extra m.* files in each user storage dir that were not actually present on the source when the final, final sync was made and I went live with the migration server. I know this is my fsckup but now I have to deal with it. I first tried to fix the symptoms with a force-resync on the boxes not realizing the above, and that seemed to make things worse by only showing new mail as of the server migration date. Ugh. New mail is being delivered and this does appear in user mail boxes. But as I said, there seems to be some users with a problem with old mail presumably deleted on the prior server suddenly re-appearing (probbly due to the extraneous m.* mdbox files present). A more pressing problem is that there is simply a time period of mail that is in user mailboxes (on the old server) from before the migration, which is not displaying now for these users on the new server. I have investigated this and hacked on "mdbox-recover.pl" somewhat so that it iterates over all mdbox files for a user and I can extract ALL messages, and see that, yes, the mail really is present in the mdbox files, but pop/imap never shows it. This is puzzling to me. I have a test user that when I check imap, it shows only 21 message in inbox and 1 in sent. But if I extract the messages from the mdbox files, then I actually have 1117 messages. I know I can't just rsync the source server again with the right --delete flag, there has been lots of mail deliveries to the migrated boxes and so I don't want to lose all of that and create yet more problems. But I need to find some way to make these present mail messages appear for my users again. Im thinking it's got to be a flag of some kind since the data is there it's just ignored. Wondering what you would reccomend? Thanks. Mike- Try with one mailbox first. Delete dovecot.index.map files and run force-resync. --- Aki Tuomi Sorry I ment dovecot.map.index there --- Aki Tuomi
Re: broken mdboxes
On 15/07/2019 09:00 Mike via dovecot < dovecot@dovecot.org> wrote: Hello, I was forced to do an emergency server migration with about 3000 mailboxes and I botched it somewhat. Hoping to find some help here. My symptoms are, users showing apparently deleted mail messages, some replicated mail, and a time period of mail that is just 'missing'. The source server is centos 6.10 with dovecot 2.2.5, and the destination is ubuntu 16 / dovecot 2.2.22. User mailboxes are mdbox format and I did an rsync from the old to the new. My screw up however was that my rsync was not --delete files, and had to be run multiple times over the course of a few weeks due to to bandwidth disparities and the sheer size of the mail spool approaching 700GB (source server had 10mbps. Should have used sneakernet instead). The bottom line however is that I wound up having many extra m.* files in each user storage dir that were not actually present on the source when the final, final sync was made and I went live with the migration server. I know this is my fsckup but now I have to deal with it. I first tried to fix the symptoms with a force-resync on the boxes not realizing the above, and that seemed to make things worse by only showing new mail as of the server migration date. Ugh. New mail is being delivered and this does appear in user mail boxes. But as I said, there seems to be some users with a problem with old mail presumably deleted on the prior server suddenly re-appearing (probbly due to the extraneous m.* mdbox files present). A more pressing problem is that there is simply a time period of mail that is in user mailboxes (on the old server) from before the migration, which is not displaying now for these users on the new server. I have investigated this and hacked on "mdbox-recover.pl" somewhat so that it iterates over all mdbox files for a user and I can extract ALL messages, and see that, yes, the mail really is present in the mdbox files, but pop/imap never shows it. This is puzzling to me. I have a test user that when I check imap, it shows only 21 message in inbox and 1 in sent. But if I extract the messages from the mdbox files, then I actually have 1117 messages. I know I can't just rsync the source server again with the right --delete flag, there has been lots of mail deliveries to the migrated boxes and so I don't want to lose all of that and create yet more problems. But I need to find some way to make these present mail messages appear for my users again. Im thinking it's got to be a flag of some kind since the data is there it's just ignored. Wondering what you would reccomend? Thanks. Mike- Try with one mailbox first. Delete dovecot.index.map files and run force-resync. --- Aki Tuomi
broken mdboxes
Hello, I was forced to do an emergency server migration with about 3000 mailboxes and I botched it somewhat. Hoping to find some help here. My symptoms are, users showing apparently deleted mail messages, some replicated mail, and a time period of mail that is just 'missing'. The source server is centos 6.10 with dovecot 2.2.5, and the destination is ubuntu 16 / dovecot 2.2.22. User mailboxes are mdbox format and I did an rsync from the old to the new. My screw up however was that my rsync was not --delete files, and had to be run multiple times over the course of a few weeks due to to bandwidth disparities and the sheer size of the mail spool approaching 700GB (source server had 10mbps. Should have used sneakernet instead). The bottom line however is that I wound up having many extra m.* files in each user storage dir that were not actually present on the source when the final, final sync was made and I went live with the migration server. I know this is my fsckup but now I have to deal with it. I first tried to fix the symptoms with a force-resync on the boxes not realizing the above, and that seemed to make things worse by only showing new mail as of the server migration date. Ugh. New mail is being delivered and this does appear in user mail boxes. But as I said, there seems to be some users with a problem with old mail presumably deleted on the prior server suddenly re-appearing (probbly due to the extraneous m.* mdbox files present). A more pressing problem is that there is simply a time period of mail that is in user mailboxes (on the old server) from before the migration, which is not displaying now for these users on the new server. I have investigated this and hacked on "mdbox-recover.pl" somewhat so that it iterates over all mdbox files for a user and I can extract ALL messages, and see that, yes, the mail really is present in the mdbox files, but pop/imap never shows it. This is puzzling to me. I have a test user that when I check imap, it shows only 21 message in inbox and 1 in sent. But if I extract the messages from the mdbox files, then I actually have 1117 messages. I know I can't just rsync the source server again with the right --delete flag, there has been lots of mail deliveries to the migrated boxes and so I don't want to lose all of that and create yet more problems. But I need to find some way to make these present mail messages appear for my users again. Im thinking it's got to be a flag of some kind since the data is there it's just ignored. Wondering what you would reccomend? Thanks. Mike-
