Re: pigeonhole question: filtering on delivered-to in case of fetchmail

[@lbutlr via dovecot]

On 15 Jul 2019, at 18:11, Trever L. Adams via dovecot  
wrote:
> So, one of the problems I am seeing is that people are trying to fake
> users into revealing information by sending from an outside domain but
> with an internal reply to address and claiming to be administration, IT
> or what not.

You should not accept external mail claiming to be from your domain unless that 
mail comes via authenticated submission. But if the reply to is going to an 
internal address… 

I’m puzzled by exactly what you mean here. Are you saying that users on your 
system are trying to phish other users on your system?

> I can set up something that will reject if from is outside the domain by
> reply to is internal. The problem is in some setups, there are fetchmail
> setups. I do not want to reject these with a message. Which is what I am
> currently doing for the others. Maybe I should discard them all without
> rejecting.

I haven’t used fetch mail in many many years, so I can’t answer anything 
specifically about it, but if you use it to allow external senders to send mail 
via your system in a way that is not authenticated then you should not do that.



-- 
NON-FLAMMABLE IS NOT A CHALLENGE Bart chalkboard Ep. BABF13

pigeonhole question: filtering on delivered-to in case of fetchmail

[Trever L. Adams via dovecot]

So, one of the problems I am seeing is that people are trying to fake
users into revealing information by sending from an outside domain but
with an internal reply to address and claiming to be administration, IT
or what not.

I can set up something that will reject if from is outside the domain by
reply to is internal. The problem is in some setups, there are fetchmail
setups. I do not want to reject these with a message. Which is what I am
currently doing for the others. Maybe I should discard them all without
rejecting.

However, my question is this:

Since such fetchmail messages will usually end up with two (at least
two?!?) Delivered-To headers, one for the fetchmail delivery and one for
the original target address's/system's delivery is it possible to do
something like this and have it work?


require ["fileinto", "regex","reject"];
if address :regex "Reply-To" ".*@<%= @name -%>" {
    if not address :regex "From" ".*@<%= @name -%>" {
        if not header :regex "Delivered-To" " .*@<%= @name -%> {
        reject "We do not allow emails from outside our
system to give Reply-To into our system!";
                    stop;
        }
    }
}

Please, not the <% =@name -%> is just that this is from a puppet module
I use to maintain these systems. It is the domain name for the mail
system. An example would be .*@middleearth.sapphiresunday.org here.

Thank you for any help in figuring this out.

The reason I want a reject in the case of non-fetchmail email is to let
users know if they try to do it (as many have multiple email accounts)
and may try it. But in fetchmail cases, no need to leak to the outside
world that users are doing fetchmail and what their account is in the
other system.

Thank you.

Trever




signature.asc
Description: OpenPGP digital signature

Replication Upgrade 2.2 -> 2.3 Question

[Cassidy B. Larson via dovecot]

Finally getting around to upgrading my setup to 2.3.x.

All of our front end proxies are fine on 2.3 connecting to a 2.2 backend
storage pair.

So the question is, if I stage the upgrade, will it break replication
between the two backend storage servers?  My thought was to upgrade one to
2.3, and keep the other on 2.2 for a couple of days?  If so, can 2.2
replicate to 2.3, and 2.3 replicate to 2.2 without any issue?  Or am I
going to have to go all-in on 2.3 at once on both? Heh.

Thanks,

-c

Listing expunged messages

[Mike via dovecot]

Hello,

	Following on my last question about broken mdboxes, removing 
dovecot.map.log and force-resync does work. My last problem now however 
is that after doing this, messages which were 'deleted' previously, come 
back in the inbox as new messages.


I have the original mailboxes and can subscribe to the expunged folders 
and everything in EXPUNGED/INBOX on the original, is appearing in INBOX 
on this migrated system. I would want to return everything to 
EXPUNDED/INBOX so my users don't see old dead resurrected messages. Is 
there a procedure to identify these?


Thanks.

Mike-

Re: [ext] 2.3.7 slower than 2.3.6?

[Ralf Hildebrandt via dovecot]

* Ralf Hildebrandt via dovecot :
> We're using dovecot (via LMTP) as a backup for all incoming mail.
> 
> I upgraded from 2.3.6 to 2.3.7 on the 12th:
> 2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 
> 2:2.3.7-8~bionic
> 
> and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup
> IMAP box is suddenly taking quite some time to get delivered and is piling up
> in the queue.

And alas, I had a look at the monitoring and found that disk IO has
increase A LOT after the update: 
https://www.arschkrebs.de/images/dovecot-disk-io-increase.png

I zoomed in and found that the sudden increace in IO coincides with
the update to 2.3.7 (14:35): https://www.arschkrebs.de/images/dovecot-io-2.png

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | https://www.charite.de

Weird response to LIST EXTENDED

[Emil Kalchev via dovecot]

Hello,

I was discussing weird response from Dovecot on Mailkit (this is client IMAP C# 
library) github page. We concluded that this is a bug in Dovecot server but I 
wanted to double check with you. Here are the details about the issue

https://github.com/jstedfast/MailKit/issues/879

Sent with OfficeSuite Mail for Windows PC
www.officesuitenow.com

Re: Authdb NSS module

[Aki Tuomi via dovecot]

 
 
  
   Dovecot can do cached ldap queries too. NSS module isn't going to do a comeback, sorry. 
  
  
   
  
  
   Aki
  
  
   
On 15/07/2019 16:51 Eugene Bright  wrote:
   
   

   
   

   I use LDAP right now, but local cached sssd queries are much faster and reliable.
   
   
   
On July 15, 2019 6:49:18 AM GMT+03:00, Aki Tuomi  wrote:

 
  
 
 
  
   On 15/07/2019 02:54 Eugene via dovecot < 
   dovecot@dovecot.org> wrote:
  
  
   
  
  
   
  
  
   Hello!
  
  
   
  
  
   Upgrading manual tells that authdb [NSS module was removed][1] some time ago.
  
  
   
  
  
   [1]: 
   https://wiki2.dovecot.org/Upgrading/2.3#line-100
  
  
   
  
  
   
userdb nss was removed. Use userdb passwd instead.
   
  
  
   Can this change be reverted?
  
  
   I'd like to use only libnss_sss.so.2 as dovecot userdb source. It's also essential for me to enable files backend in nsswitch.conf so the system could use local user db. At the same time dovecot must not see local users at all. Authdb NSS module could help me there.
  
  
   The other solution would be to use another instance of nsswitch.conf for dovecot authdb passwd module. Is it possible?
  
  
   
  
  
   Thanks!
  
  
   --
  
  
   Eugene Bright
  
  
   IT engineer
  
  
   Tel: + 79257289622
  
 
 
  
 
 
  passwd uses getpwent library call which goes thru nssswitch. Your explanation was strange though. If you don't want system users, why not use ldap or sql directly? 
 
 
  ---
Aki Tuomi
 

   
   Eugene Bright
   IT-engineer
   Tel.: +7 925 728 96 22
   
  
  
   
  
  
   ---
Aki Tuomi
   
 

Re: Error: o_stream_send_istream and Disconnected in APPEND

[Alessio Cecchi via dovecot]

Il 11/07/19 23:31, Timo Sirainen ha scritto:
On 11 Jul 2019, at 10.13, Alessio Cecchi via dovecot 
mailto:dovecot@dovecot.org>> wrote:


Hi,

I'm running some Dovecot servers configured with LVS + Director + 
Backend + NFS and version 2.2.36.3 (a7d78f5a2).


In the last days I see an increased number of these error:

Error: 
o_stream_send_istream(/nfs/mail/company.com/info/Maildir/.Sent/tmp/1562771349.M255624P9151.pop01 
) 
failed: Broken pipe


always with action "Disconnected in APPEND", when users try to upload 
a message in Sent or Drafts.




I think it simply means that Dovecot sees that the client disconnected 
while it was APPENDing the mail. Although I don't know why they would 
suddenly start now. And I especially don't understand why the error is 
"Broken pipe". Dovecot uses it internally when it closes input 
streams, so it's possibly that, but why would isn't that happening 
elsewhere then.. Did you upgrade your kernel recently? I guess it's 
also possible that there is some bug in Dovecot, but I don't remember 
any changes related to this for a long time.


I guess it could actually be writing as well, because "Broken pipe" is 
set also for closed output streams, so maybe some failed NFS write 
could cause it (although it really should have logged a different 
error in that case, so if that was the reason this is a bug).


Dovecot v2.3.x would log a different error depending on if the problem 
was reading or writing, which would make this clearer.


Thanks Timo,

the operating system is CentOS 6 from years, and we doing regular update 
every time available. And also the NFS storage is the same from years.


The error is not starting to show now, I see it occasionally since 2017 
but after the last network upgrade (Firewall and Switch) are coming more 
frequently. And only for Thunderbird and sometimes Apple iOS Mail.


The error never occurred on old servers when we didn't have a physical 
firewall, but we used iptables on individual servers, and MTU on network 
interfaces ha MTU 9000, but many others components was updated in the 
meantime.


We will upgrade to Dovecot 2.3 in the next months.

--
Alessio Cecchi
Postmaster @ http://www.qboxmail.it
https://www.linkedin.com/in/alessice

Re: Authdb NSS module

[Eugene Bright via dovecot]

I use LDAP right now, but local cached sssd queries are much faster and 
reliable.

On July 15, 2019 6:49:18 AM GMT+03:00, Aki Tuomi  
wrote:
>
>
>On 15/07/2019 02:54 Eugene via dovecot < dovecot@dovecot.org> wrote: 
>
>
>
>Hello! 
>
>
>Upgrading manual tells that authdb [NSS module was removed][1] some
>time ago. 
>
>
>[1]: https://wiki2.dovecot.org/Upgrading/2.3#line-100 
>
>
>userdb nss was removed. Use userdb passwd instead. 
>
>Can this change be reverted? 
>
>I'd like to use only libnss_sss.so.2 as dovecot userdb source. It's
>also essential for me to enable files backend in nsswitch.conf so the
>system could use local user db. At the same time dovecot must not see
>local users at all. Authdb NSS module could help me there. 
>
>The other solution would be to use another instance of nsswitch.conf
>for dovecot authdb passwd module. Is it possible? 
>
>
>Thanks! 
>
>-- 
>
>Eugene Bright 
>
>IT engineer 
>
>Tel: + 79257289622 
>
>
>passwd uses getpwent library call which goes thru nssswitch. Your
>explanation was strange though. If you don't want system users, why not
>use ldap or sql directly?  
>
>--- Aki Tuomi 

---
Eugene Bright
IT-engineer
Tel.: +7 925 728 96 22

2.3.7 slower than 2.3.6?

[Ralf Hildebrandt via dovecot]

We're using dovecot (via LMTP) as a backup for all incoming mail.

I upgraded from 2.3.6 to 2.3.7 on the 12th:
2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 
2:2.3.7-8~bionic

and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup
IMAP box is suddenly taking quite some time to get delivered and is piling up
in the queue.

doveconf -n:


default_vsz_limit = 2 G
lmtp_user_concurrency_limit = 1
mail_attachment_dir = /home/copymail/attachments
mail_fsync = never
mail_location = mdbox:~/mdbox
mail_plugins = zlib fts
mdbox_rotate_size = 128 M
... namespaces ...
plugin {
  fts_autoindex = yes
  fts_languages = de,en
  zlib_save = gz
  zlib_save_level = 5
}
protocols = " imap lmtp"
userdb {
  args = username_format=%u /etc/dovecot/passwd
  driver = passwd-file
}
protocol lmtp {
  mail_plugins = zlib fts
}
  

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | https://www.charite.de

Re: Sieve problem with duplicate and fileinto in the same set of rules

[Alessio Cecchi via dovecot]

Il 15/07/19 14:34, Gianluca Scaglia via dovecot ha scritto:


Hi there,

on my mail server (postfix, dovecot 2.2.27 in Debian 9) I have an 
automatic forwarding (with sender_bcc_maps in Postfix) for all the 
emails sent in smtp from the same server, that are then put in the 
Sent folder with a sieve rule.


In this way, however, when a user sends an e-mail to himself, both 
copies end up in the Sent folder and it's not good.


To resolve, I tried using the Sieve “duplicate” extension along with 
“fileinto” but I can't get it to work.



I solved the problem of duplicated email with this Sieve rule:

require ["duplicate", "fileinto", "mailbox"];

if duplicate :seconds 60 {
    fileinto "Trash";
}

Hope this can hel you.

Ciao

--
Alessio Cecchi
Postmaster @ http://www.qboxmail.it
https://www.linkedin.com/in/alessice

Sieve problem with duplicate and fileinto in the same set of rules

[Gianluca Scaglia via dovecot]

Hi there,

on my mail server (postfix, dovecot 2.2.27 in Debian 9) I have an automatic
forwarding (with sender_bcc_maps in Postfix) for all the emails sent in smtp
from the same server, that are then put in the Sent folder with a sieve
rule.

 

In this way, however, when a user sends an e-mail to himself, both copies
end up in the Sent folder and it's not good.

 

To resolve, I tried using the Sieve "duplicate" extension along with
"fileinto" but I can't get it to work.

 

I then activated also the "editheader" extension to better understand what
the problem was and I get these results.

 

Sieve rules without using "fileinto:"

 

require ["duplicate", "editheader"];

if duplicate { 

addheader "x-sieve-test1" "duplicate"; 

}

if allof (header :regex ["from"] [".*info@example\.com"], 

header :regex ["x-sieve-test1"] [".*duplicate"]) { 

addheader "x-sieve-test2" "confirm duplicate"; 

stop; 

} elsif header :regex ["from"] [".*info@example\.com"] { 

addheader "x-sieve-test3" "not duplicate"; 

stop;

}  

 

and I get these headers in the first mail (in INBOX):

X-Sieve-Test2: confirm duplicate

X-Sieve-Test1: duplicate

 

and in the second email (always in INBOX) this other:

X-Sieve-Test3: not duplicate

 

Everything works perfectly.

 

When I add the fileinto statement to the last if as below

 

require ["duplicate", "editheader"];

if duplicate { 

addheader "x-sieve-test1" "duplicate"; 

}

if allof (header :regex ["from"] [".*info@example\.com"], 

header :regex ["x-sieve-test1"] [".*duplicate"]) { 

addheader "x-sieve-test2" "confirm duplicate"; 

stop; 

} elsif header :regex ["from"] [".*info@example\.com"] { 

addheader "x-sieve-test3" "not duplicate"; 

fileinto "Sent";

stop;

}

 

I get a completely incorrect result, both mails in the Sent folder with this
header line:

X-Sieve-Test3: not duplicate

 

in practice it no longer detects the duplicate email.

 

If I use "discard" rather than "fileinto" it works fine and if I put
"fileinto" in the first "if" ("if duplicate") it does't work. 

 

I did hundreds of tests but the "duplicate" extension stops working in the
presence of the fileinto statement (there are exceptions: if I put the
"fileinto" in the second "if" - that is in the middle one instead of the
last one - it works fine, but it's not that that I want).

 

Any idea how to solve the problem or how to get the same result (one mail in
Sent folder and one in INBOX, when a user sends an e-mail to himself)
without using the "duplicate" extension?

 

Thanks a lot,

Luca

Re: Error since Dovecot v2.3.7

[Paul Hecker via dovecot]

Hi,

> On 15. Jul 2019, at 10:59, Timo Sirainen  wrote:
> 
> On 15 Jul 2019, at 10.58, Paul Hecker via dovecot  wrote:
>> 
>> Hi,
>> 
>> since upgrading to Dovecot 2.3.7 I get the following new errors:
> 
> What was the old version? Did you upgrade your kernel as well?

It was 2.3.6. But you are true, this seems to be an issue related to AppArmor 
(in complain mode) that I enabled a day earlier. Have to investigate why 
AppArmor is given me this?!

Sorry for the noise...

> 
>> 2019-07-15 09:10:52 mail dovecot:  
>> imap(p...@iwascoding.com)<32484>: Error: file_lock_free(): 
>> Unexpectedly failed to retry locking 
>> /var/spool/mail/iwascoding/paul/mdbox/mailboxes/INBOX/dbox-Mails/dovecot-vsize.lock:
>>  
>> fcntl(/var/spool/mail/iwascoding/paul/mdbox/mailboxes/INBOX/dbox-Mails/dovecot-vsize.lock,
>>  write-lock, F_SETLK) locking failed: No such file or directory
> 
> What filesystem are you using for mdboxes?
> 
> The lock fd is already open. Locking is not supposed to fail this way. Also I 
> can't think of any recent changes that could be related to this. v2.2.36 
> already had the same locking code.
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Error since Dovecot v2.3.7

[Timo Sirainen via dovecot]

On 15 Jul 2019, at 10.58, Paul Hecker via dovecot  wrote:
> 
> Hi,
> 
> since upgrading to Dovecot 2.3.7 I get the following new errors:

What was the old version? Did you upgrade your kernel as well?

> 2019-07-15 09:10:52 mail dovecot:  
> imap(p...@iwascoding.com)<32484>: Error: file_lock_free(): 
> Unexpectedly failed to retry locking 
> /var/spool/mail/iwascoding/paul/mdbox/mailboxes/INBOX/dbox-Mails/dovecot-vsize.lock:
>  
> fcntl(/var/spool/mail/iwascoding/paul/mdbox/mailboxes/INBOX/dbox-Mails/dovecot-vsize.lock,
>  write-lock, F_SETLK) locking failed: No such file or directory

What filesystem are you using for mdboxes?

The lock fd is already open. Locking is not supposed to fail this way. Also I 
can't think of any recent changes that could be related to this. v2.2.36 
already had the same locking code.

Error since Dovecot v2.3.7

[Paul Hecker via dovecot]

Hi,

since upgrading to Dovecot 2.3.7 I get the following new errors:

2019-07-15 09:10:52 mail dovecot:  
imap(p...@iwascoding.com)<32484>: Error: file_lock_free(): 
Unexpectedly failed to retry locking 
/var/spool/mail/iwascoding/paul/mdbox/mailboxes/INBOX/dbox-Mails/dovecot-vsize.lock:
 
fcntl(/var/spool/mail/iwascoding/paul/mdbox/mailboxes/INBOX/dbox-Mails/dovecot-vsize.lock,
 write-lock, F_SETLK) locking failed: No such file or directory

It seems that this is related to moving or deleting mails (IMAP). The file at 
least sometimes exists, the folder exists. You find my doveconf below.

Thanks,
Paul



# 2.3.7 (494d20bdc): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.7 (5a4e63b5)
# OS: Linux 4.9.0-9-amd64 x86_64 Debian 9.9 
# Hostname: mail.iwascoding.com
auth_cache_size = 10 M
auth_cache_ttl = 2 hours
auth_mechanisms = plain login
default_vsz_limit = 1 G
first_valid_gid = 8
first_valid_uid = 8
hostname = mail.iwascoding.com
imap_capability = +SPECIAL-USE
last_valid_gid = 8
last_valid_uid = 8
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
mail_attachment_detection_options = add-flags-on-save
mail_attribute_dict = file:/var/mail/iwascoding/%n/dovecot-attributes
mail_gid = 8
mail_home = /var/mail/iwascoding/%n
mail_location = mdbox:~/mdbox
mail_plugins = quota listescape fts fts_xapian
mail_prefetch_count = 200
mail_uid = 8
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext spamtest spamtestplus
mdbox_rotate_size = 16 M
namespace inbox {
  inbox = yes
  location = 
  mailbox "Sent Messages" {
auto = no
special_use = \Sent
  }
  mailbox archive {
auto = no
special_use = \Archive
  }
  mailbox drafts {
auto = no
special_use = \Drafts
  }
  mailbox junk {
auto = no
special_use = \Junk
  }
  mailbox sent {
auto = no
special_use = \Sent
  }
  mailbox trash {
auto = no
special_use = \Trash
  }
  prefix = 
  separator = /
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  fts = xapian
  fts_autoindex = yes
  fts_autoindex_exclude = \Junk
  fts_autoindex_exclude2 = \Trash
  fts_autoindex_exclude3 = \Drafts
  fts_enforced = yes
  fts_xapian = partial=2 full=20
  plugin = fts fts_xapian
  quota = count:User quota
  quota_grace = 10%%
  quota_rule = *:storage=200M
  quota_vsizes = yes
  recipient_delimiter = +
  sieve = /var/mail/iwascoding/%n/.dovecot.sieve
  sieve_default = /var/lib/dovecot/sieve/default.sieve
  sieve_dir = /var/mail/iwascoding/%n/sieve
  sieve_extensions = +spamtest +spamtestplus
  sieve_global_dir = /var/lib/dovecot/sieve/global
  sieve_spamtest_max_value = 200
  sieve_spamtest_status_header = X-Spam-Score-Int
  sieve_spamtest_status_type = score
}
postmaster_address = postmas...@iwascoding.com
protocols = imap pop3 lmtp sieve submission
quota_full_tempfail = yes
service auth {
  unix_listener auth-client {
mode = 0600
user = Debian-exim
  }
  unix_listener auth-userdb {
user = mail
  }
  vsz_limit = 4 G
}
service imap-login {
  inet_listener imap {
port = 0
  }
}
service indexer-worker {
  vsz_limit = 2 G
}
service lmtp {
  user = mail
}
service managesieve-login {
  inet_listener sieve {
port = 4190
  }
}
service pop3-login {
  inet_listener pop3 {
port = 0
  }
}
service stats {
  unix_listener stats-writer {
user = mail
  }
}
service submission-login {
  inet_listener submission_ssl {
port = 465
ssl = yes
  }
}
ssl = required
ssl_alt_cert = 

smime.p7s
Description: S/MIME cryptographic signature

Re: broken mdboxes

[Mike via dovecot]

On 7/14/19 11:30 PM, Aki Tuomi via dovecot wrote:
>> Try with one mailbox first.
>>
>> Delete dovecot.index.map files and run force-resync.
>>
>> ---
>> Aki Tuomi
>
> Sorry I ment dovecot.map.index there



Hey, that seemed to work!


I'll need to do more testing because I still have those false mdbox
files in the directories that need to be cleared out but I have a
process for that too.


Thanks again!


Mike-

Re: broken mdboxes

[Aki Tuomi via dovecot]

 
 
  
   
  
  
   
On 15/07/2019 09:22 Aki Tuomi via dovecot  wrote:
   
   

   
   

   
   

   
   

 On 15/07/2019 09:00 Mike via dovecot < 
 dovecot@dovecot.org> wrote:


 


 


 Hello,


 


 I was forced to do an emergency server migration with about 3000


 mailboxes and I botched it somewhat. Hoping to find some help here.


 


 My symptoms are, users showing apparently deleted mail messages, some


 replicated mail, and a time period of mail that is just 'missing'.


 


 The source server is centos 6.10 with dovecot 2.2.5, and the destination


 is ubuntu 16 / dovecot 2.2.22. User mailboxes are mdbox format and I did


 an rsync from the old to the new. My screw up however was that my rsync


 was not --delete files, and had to be run multiple times over the course


 of a few weeks due to to bandwidth disparities and the sheer size of the


 mail spool approaching 700GB (source server had 10mbps. Should have used


 sneakernet instead). The bottom line however is that I wound up having


 many extra m.* files in each user storage dir that were not actually


 present on the source when the final, final sync was made and I went


 live with the migration server.  I know this is my fsckup but now I have


 to deal with it.


 


 I first tried to fix the symptoms with a force-resync on the boxes not


 realizing the above, and that seemed to make things worse by only


 showing new mail as of the server migration date. Ugh.


 


 New mail is being delivered and this does appear in user mail boxes. But


 as I said, there seems to be some users with a problem with old mail


 presumably deleted on the prior server suddenly re-appearing (probbly


 due to the extraneous m.* mdbox files present). A more pressing problem


 is that there is simply a time period of mail that is in user mailboxes


 (on the old server) from before the migration, which is not displaying


 now for these users on the new server.


 


 I have investigated this and hacked on "mdbox-recover.pl" somewhat so


 that it iterates over all mdbox files for a user and I can extract ALL


 messages, and see that, yes, the mail really is present in the mdbox


 files, but pop/imap never shows it. This is puzzling to me. I have a


 test user that when I check imap, it shows only 21 message in inbox and


 1 in sent. But if I extract the messages from the mdbox files, then I


 actually have 1117 messages. 


 


 I know I can't just rsync the source server again with the right


 --delete flag, there has been lots of mail deliveries to the migrated


 boxes and so I don't want to lose all of that and create yet more


 problems. But I need to find some way to make these present mail


 messages appear for my users again. Im thinking it's got to be a flag of


 some kind since the data is there it's just ignored. Wondering what you


 would reccomend?


 


 


 Thanks.


 


 Mike-

   
   

   
   
Try with one mailbox first.
   
   

   
   
Delete dovecot.index.map files and run force-resync.
   
   

   
   
---
Aki Tuomi
   
  
  
   
  
  
   Sorry I ment dovecot.map.index there
  
  
   ---
Aki Tuomi
   
 

Re: broken mdboxes

[Aki Tuomi via dovecot]

 
 
  
   
  
  
   
On 15/07/2019 09:00 Mike via dovecot <
dovecot@dovecot.org> wrote:
   
   

   
   

   
   
Hello,
   
   

   
   
I was forced to do an emergency server migration with about 3000
   
   
mailboxes and I botched it somewhat. Hoping to find some help here.
   
   

   
   
My symptoms are, users showing apparently deleted mail messages, some
   
   
replicated mail, and a time period of mail that is just 'missing'.
   
   

   
   
The source server is centos 6.10 with dovecot 2.2.5, and the destination
   
   
is ubuntu 16 / dovecot 2.2.22. User mailboxes are mdbox format and I did
   
   
an rsync from the old to the new. My screw up however was that my rsync
   
   
was not --delete files, and had to be run multiple times over the course
   
   
of a few weeks due to to bandwidth disparities and the sheer size of the
   
   
mail spool approaching 700GB (source server had 10mbps. Should have used
   
   
sneakernet instead). The bottom line however is that I wound up having
   
   
many extra m.* files in each user storage dir that were not actually
   
   
present on the source when the final, final sync was made and I went
   
   
live with the migration server.  I know this is my fsckup but now I have
   
   
to deal with it.
   
   

   
   
I first tried to fix the symptoms with a force-resync on the boxes not
   
   
realizing the above, and that seemed to make things worse by only
   
   
showing new mail as of the server migration date. Ugh.
   
   

   
   
New mail is being delivered and this does appear in user mail boxes. But
   
   
as I said, there seems to be some users with a problem with old mail
   
   
presumably deleted on the prior server suddenly re-appearing (probbly
   
   
due to the extraneous m.* mdbox files present). A more pressing problem
   
   
is that there is simply a time period of mail that is in user mailboxes
   
   
(on the old server) from before the migration, which is not displaying
   
   
now for these users on the new server.
   
   

   
   
I have investigated this and hacked on "mdbox-recover.pl" somewhat so
   
   
that it iterates over all mdbox files for a user and I can extract ALL
   
   
messages, and see that, yes, the mail really is present in the mdbox
   
   
files, but pop/imap never shows it. This is puzzling to me. I have a
   
   
test user that when I check imap, it shows only 21 message in inbox and
   
   
1 in sent. But if I extract the messages from the mdbox files, then I
   
   
actually have 1117 messages. 
   
   

   
   
I know I can't just rsync the source server again with the right
   
   
--delete flag, there has been lots of mail deliveries to the migrated
   
   
boxes and so I don't want to lose all of that and create yet more
   
   
problems. But I need to find some way to make these present mail
   
   
messages appear for my users again. Im thinking it's got to be a flag of
   
   
some kind since the data is there it's just ignored. Wondering what you
   
   
would reccomend?
   
   

   
   

   
   
Thanks.
   
   

   
   
Mike-
   
  
  
   
  
  
   Try with one mailbox first.
  
  
   
  
  
   Delete dovecot.index.map files and run force-resync.
  
  
   
  
  
   ---
Aki Tuomi
   
 

broken mdboxes

[Mike via dovecot]

Hello,

I was forced to do an emergency server migration with about 3000
mailboxes and I botched it somewhat. Hoping to find some help here.

My symptoms are, users showing apparently deleted mail messages, some
replicated mail, and a time period of mail that is just 'missing'.

The source server is centos 6.10 with dovecot 2.2.5, and the destination
is ubuntu 16 / dovecot 2.2.22. User mailboxes are mdbox format and I did
an rsync from the old to the new. My screw up however was that my rsync
was not --delete files, and had to be run multiple times over the course
of a few weeks due to to bandwidth disparities and the sheer size of the
mail spool approaching 700GB (source server had 10mbps. Should have used
sneakernet instead). The bottom line however is that I wound up having
many extra m.* files in each user storage dir that were not actually
present on the source when the final, final sync was made and I went
live with the migration server.  I know this is my fsckup but now I have
to deal with it.

I first tried to fix the symptoms with a force-resync on the boxes not
realizing the above, and that seemed to make things worse by only
showing new mail as of the server migration date. Ugh.

New mail is being delivered and this does appear in user mail boxes. But
as I said, there seems to be some users with a problem with old mail
presumably deleted on the prior server suddenly re-appearing (probbly
due to the extraneous m.* mdbox files present). A more pressing problem
is that there is simply a time period of mail that is in user mailboxes
(on the old server) from before the migration, which is not displaying
now for these users on the new server.

I have investigated this and hacked on "mdbox-recover.pl" somewhat so
that it iterates over all mdbox files for a user and I can extract ALL
messages, and see that, yes, the mail really is present in the mdbox
files, but pop/imap never shows it. This is puzzling to me. I have a
test user that when I check imap, it shows only 21 message in inbox and
1 in sent. But if I extract the messages from the mdbox files, then I
actually have 1117 messages. 

I know I can't just rsync the source server again with the right
--delete flag, there has been lots of mail deliveries to the migrated
boxes and so I don't want to lose all of that and create yet more
problems. But I need to find some way to make these present mail
messages appear for my users again. Im thinking it's got to be a flag of
some kind since the data is there it's just ignored. Wondering what you
would reccomend?


Thanks.

Mike-