Re: server migration
in my case, 99% of mailboxes are imap Il gio 11 apr 2024, 00:08 Michael Peddemors via dovecot ha scritto: Of course, anyone who is stilling using POP (Leave on Server) presents a different challenge.. Depending on the client, and how the client treated the UID of the message.. The rest should present no issue.. On 2024-04-10 14:25, Kirill Miazine via dovecot wrote: > > > • Gandalf Corvotempesta via dovecot [2024-04-10 23:18]: >> Il giorno mer 10 apr 2024 alle ore 23:12 Kirill Miazine via dovecot >> ha scritto: >>> UIDVALIDITY change >> >> In which case uidvalidity would change ? > > if you do rsync, it doesn't. UIDVALIDITY is stored in dovecot- uidlist in > maildirs, as described in > https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#imap- uid-mapping > ___ > dovecot mailing list -- dovecot@dovecot.org > To unsubscribe send an email to dovecot-le...@dovecot.org -- "Catch the Magic of Linux..." - --- Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Reg. TradeMark of Wizard Tower TechnoServices Ltd. - --- 604-682-0300 Beautiful British Columbia, Canada ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: server migration
Of course, anyone who is stilling using POP (Leave on Server) presents a different challenge.. Depending on the client, and how the client treated the UID of the message.. The rest should present no issue.. On 2024-04-10 14:25, Kirill Miazine via dovecot wrote: • Gandalf Corvotempesta via dovecot [2024-04-10 23:18]: Il giorno mer 10 apr 2024 alle ore 23:12 Kirill Miazine via dovecot ha scritto: UIDVALIDITY change In which case uidvalidity would change ? if you do rsync, it doesn't. UIDVALIDITY is stored in dovecot-uidlist in maildirs, as described in https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#imap-uid-mapping ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org -- "Catch the Magic of Linux..." Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Reg. TradeMark of Wizard Tower TechnoServices Ltd. 604-682-0300 Beautiful British Columbia, Canada ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: server migration
• Gandalf Corvotempesta via dovecot [2024-04-10 23:18]: Il giorno mer 10 apr 2024 alle ore 23:12 Kirill Miazine via dovecot ha scritto: UIDVALIDITY change In which case uidvalidity would change ? if you do rsync, it doesn't. UIDVALIDITY is stored in dovecot-uidlist in maildirs, as described in https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#imap-uid-mapping ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: server migration
Il giorno mer 10 apr 2024 alle ore 23:12 Kirill Miazine via dovecot ha scritto: > UIDVALIDITY change In which case uidvalidity would change ? Manually changes in config file ? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: server migration
• Gandalf Corvotempesta via dovecot [2024-04-10 22:59]: What could trigger a new re-download of message ? UIDVALIDITY change ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: server migration
Il giorno mer 10 apr 2024 alle ore 22:32 Marc via dovecot ha scritto: > Why? The whole idea about having a LTS distribution is that you almost never > need to do this? It is not like the imap/pop/smtp standards are having yearly > innovations. Or is this a service you provide for clients? In my case the server migration has to be done because the old datacenter is closing and i have to move all datas to a new server on a different location. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: server migration
Il giorno mer 10 apr 2024 alle ore 21:40 Kirill Miazine via dovecot ha scritto: > What you describe is exactly what I have been doing since ... forever > > - reduce TTL > - setup new server > - rsync > - stop ALL mail services on old server (also anything which might be > doing deliveries, this is important), kill client connections, if any > - rsync again > - update DNS > - start mail service on new server > - verify > - increase TTL this is what i've planned, but I have to be 1% sure that clients wont re-download all messages again. What could trigger a new re-download of message ? The new server would be able to handle the mails stored with the old hostname and at the same time handle the mails stored with the new hostname (and thus different file name) ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: server migration
Can you expand and explain this: Why? The whole idea about having a LTS distribution is that you almost never need to do this? Can you provide a link for context? On 4/10/2024 3:25 PM, Marc via dovecot wrote: • Gandalf Corvotempesta via dovecot [2024-04-10 21:07]: Guys, any help? What you describe is exactly what I have been doing since ... forever Why? The whole idea about having a LTS distribution is that you almost never need to do this? It is not like the imap/pop/smtp standards are having yearly innovations. Or is this a service you provide for clients? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org -- Christopher Wensink IS Administrator Five Star Plastics, Inc 1339 Continental Drive Eau Claire, WI 54701 Office: 715-831-1682 Mobile: 715-563-3112 Fax: 715-831-6075 cwens...@five-star-plastics.com www.five-star-plastics.com ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: server migration
> > > > • Gandalf Corvotempesta via dovecot [2024-04-10 21:07]: > > Guys, any help? > > What you describe is exactly what I have been doing since ... forever > Why? The whole idea about having a LTS distribution is that you almost never need to do this? It is not like the imap/pop/smtp standards are having yearly innovations. Or is this a service you provide for clients? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: server migration
> > Guys, any help? this lacks context. > Also, what would happen if the new server has a different hostname ? So put temporary haproxy infront of it? > Il giorno dom 10 mar 2024 alle ore 14:28 Gandalf Corvotempesta > ha scritto: > > > > Hi guys > > I have to migrate around 10k mailboxes from dovecot 2.13 to (i think) > > the same version but on a different server. Why not just mount the data on the other server? > > I have to reduce as much as possible the inconveniences to the users, > > at least in this (temporary) phase. Just prepare two vm configs and stop the old and start the new. > > What do you suggest to move everything ? Same config, same maildir > > location and rsync everything ? You have to have a good reason to change storage formats, it is not really related to what instagrammers reeling about. What is your good reason to change? > > Better ideas ? i've thought to use the exact same config on both > > servers, then start multiple rsync to sync as much as possible and > > when ready, drop the old dovecot in the old server, rsync the latest > > changes, and then move the dns pointment from the old ip to the new > > one. I would go for just mount the data images on the other server. If that is not possible and you don't seem to know what is on the other server. Do the doveadm sync. I would suggest thinking a bit more about this and what you want in the future. Because you don't want to do this to often. There was just a discussion about these folder names what you might consider looking at. Als the availability of 2.4, maybe wait? > > But what about the MUA downloading emails ? I think this would be > > safe...or there is a chance that some MUA would re-download everything > > ? This would be unacceptable. > > You will see when you are testing. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: server migration
• Gandalf Corvotempesta via dovecot [2024-04-10 21:07]: Guys, any help? What you describe is exactly what I have been doing since ... forever - reduce TTL - setup new server - rsync - stop ALL mail services on old server (also anything which might be doing deliveries, this is important), kill client connections, if any - rsync again - update DNS - start mail service on new server - verify - increase TTL You mention multiple rsyncs, I wouldn't bother... Also, what would happen if the new server has a different hostname ? You'd get new filenames in Maildir, and this is it. Il giorno dom 10 mar 2024 alle ore 14:28 Gandalf Corvotempesta ha scritto: Hi guys I have to migrate around 10k mailboxes from dovecot 2.13 to (i think) the same version but on a different server. I have to reduce as much as possible the inconveniences to the users, at least in this (temporary) phase. What do you suggest to move everything ? Same config, same maildir location and rsync everything ? Better ideas ? i've thought to use the exact same config on both servers, then start multiple rsync to sync as much as possible and when ready, drop the old dovecot in the old server, rsync the latest changes, and then move the dns pointment from the old ip to the new one. But what about the MUA downloading emails ? I think this would be safe...or there is a chance that some MUA would re-download everything ? This would be unacceptable. thank you ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: server migration
Guys, any help? Also, what would happen if the new server has a different hostname ? Il giorno dom 10 mar 2024 alle ore 14:28 Gandalf Corvotempesta ha scritto: > > Hi guys > I have to migrate around 10k mailboxes from dovecot 2.13 to (i think) > the same version but on a different server. > > I have to reduce as much as possible the inconveniences to the users, > at least in this (temporary) phase. > > What do you suggest to move everything ? Same config, same maildir > location and rsync everything ? > > Better ideas ? i've thought to use the exact same config on both > servers, then start multiple rsync to sync as much as possible and > when ready, drop the old dovecot in the old server, rsync the latest > changes, and then move the dns pointment from the old ip to the new > one. > > But what about the MUA downloading emails ? I think this would be > safe...or there is a chance that some MUA would re-download everything > ? This would be unacceptable. > > thank you ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Server migration, password scheme/hashing, argon2i, argon2d, argon2id, sha512, sha512-crypt, tiger2, salt?
Should work then if you have roundcube 1.6.x and php8.2 which is also the bookworm package version. Depends on your server spec / number of users if you use argon2 over bcrypt. One approach might be to just migrate all users to BLF-CRYPT anyway, and then set the recommended dovecot member settings and selectively change a few users to ARGON2ID to see the impact. If you stored both hashes in the database, this would allow you to switch back. If someone gained write access to the database somehow, they could possibly replace user's password hash with a new one, thereby allowing them to gain access to user accounts. Two-factor authentication of course is the way to go with roundcube, but by default, there's nothing stopping access using the same credentials via IMAP/Submission without the 2FA, so roundcube 2FA isn't effective by itself if users also have access to IMAP/Submission. I improved things a bit by using roundcube plugins:- mmvi/twofactor_webauthn - FIDO2/webauthn 2FA. And: https://github.com/openSUSE/ap4rc I modified it a bit to allow using the same username and added some features to it: https://github.com/listerr/ap4rc/tree/last-access Ultimately the goal is to eliminate passwords using OAUTH2 etc but not quite there yet. R. On 2023-06-24 14:54, David Mehler wrote: Hello, Thanks. The other utility I would be using is the Roundcube webmail password plugin. Still trying to figure the best option. More opinions? Thanks. Dave. -- Robert Lister - email: r...@lentil.org - tel: 020 7043 7996 ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Server migration, password scheme/hashing, argon2i, argon2d, argon2id, sha512, sha512-crypt, tiger2, salt?
Hello, Thanks. The other utility I would be using is the Roundcube webmail password plugin. Still trying to figure the best option. More opinions? Thanks. Dave. On 6/24/23, Robert Lister wrote: > > I did a similar upgrade, and now in the process of migrating from > SHA512-CRYPT > to BLF-CRYPT with an appropriately set rounds, as I think the default > rounds > is a little low. > > A good write-up on migrating passwords and calculating the rounds: > https://kaworu.ch/blog/2016/04/20/strong-crypt-scheme-with-dovecot-postfixadmin-and-roundcube/ > > > I would take into consideration the following factors when deciding the > hashing algo. > > 1. Other tools/scripts that need to update or check passwords in the > database, > for example: > - roundcube webmail has a plugin to allow users to change their > password > using a variety of methods. > - postfixadmin > > For a long time, bcrypt wasn't natively supported by either the > version of php > or underlying OS libs, so these tools had to rely on calling "doveadm > pw " > to generate BLF-CRYPT hashes. And assumed that doveadm was available > on the same server as it. > > The latest versions support bcrypt and newer hashing algos natively. > > Some tools might rely on the database (mysql/mariadb) to hash > passwords, so > this may also be a consideration. > > 2. Server load / libs: > > - The Dovecot docs: > https://doc.dovecot.org/configuration_manual/authentication/password_schemes/ > has this to say on ARGON2I/ARGON2ID: > > "Argon2 is the winner of password hashing competition held at July > 2015. The password will >start with $argon2i$ or $argon2id$. You can use -r to tune > computational complexity, >minimum is 3. ARGON2ID is only available if your libsodium is > recent enough. >ARGON2 can require quite a hefty amount of virtual memory, so we > recommend that you set >service auth { vsz_limit = 2G } at least, or more." > > There's a good write up of considering the various algos: > > https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html > > I considered BLF-CRYPT (for the time being) to be strong enough and a > good balance between compatibility, strength and server load, given the > number of users etc. > > Rob > > > On 2023-06-23 02:14, David Mehler wrote: >> Hello, >> >> I'm migrating to a new server. It's running Debian 11 currently though >> that's going 12 this weekend. Currently it uses Openssl v3.0.9, and >> dovecot 2.3.13 and MySQL (in this case Mariadb) for storing user >> account information v10.6.14. My question is in regards password >> storage and scheme/encryption/salts. >> >> Currently they are stored in Mariadb password field with a type of >> varchar and a 255 character length, and are stored as SHA512-CRYPT. >> I'm wondering if I should keep this as is or when I migrate go to >> another scheme? I'm thinking argon2i, argon2d, argon2id, sha512, >> sha512-crypt, tiger2, saltt? > > > -- > Robert Lister - email: r...@lentil.org - tel: 020 7043 7996 > ___ > dovecot mailing list -- dovecot@dovecot.org > To unsubscribe send an email to dovecot-le...@dovecot.org > ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Server migration, password scheme/hashing, argon2i, argon2d, argon2id, sha512, sha512-crypt, tiger2, salt?
I did a similar upgrade, and now in the process of migrating from SHA512-CRYPT to BLF-CRYPT with an appropriately set rounds, as I think the default rounds is a little low. A good write-up on migrating passwords and calculating the rounds: https://kaworu.ch/blog/2016/04/20/strong-crypt-scheme-with-dovecot-postfixadmin-and-roundcube/ I would take into consideration the following factors when deciding the hashing algo. 1. Other tools/scripts that need to update or check passwords in the database, for example: - roundcube webmail has a plugin to allow users to change their password using a variety of methods. - postfixadmin For a long time, bcrypt wasn't natively supported by either the version of php or underlying OS libs, so these tools had to rely on calling "doveadm pw " to generate BLF-CRYPT hashes. And assumed that doveadm was available on the same server as it. The latest versions support bcrypt and newer hashing algos natively. Some tools might rely on the database (mysql/mariadb) to hash passwords, so this may also be a consideration. 2. Server load / libs: - The Dovecot docs: https://doc.dovecot.org/configuration_manual/authentication/password_schemes/ has this to say on ARGON2I/ARGON2ID: "Argon2 is the winner of password hashing competition held at July 2015. The password will start with $argon2i$ or $argon2id$. You can use -r to tune computational complexity, minimum is 3. ARGON2ID is only available if your libsodium is recent enough. ARGON2 can require quite a hefty amount of virtual memory, so we recommend that you set service auth { vsz_limit = 2G } at least, or more." There's a good write up of considering the various algos: https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html I considered BLF-CRYPT (for the time being) to be strong enough and a good balance between compatibility, strength and server load, given the number of users etc. Rob On 2023-06-23 02:14, David Mehler wrote: Hello, I'm migrating to a new server. It's running Debian 11 currently though that's going 12 this weekend. Currently it uses Openssl v3.0.9, and dovecot 2.3.13 and MySQL (in this case Mariadb) for storing user account information v10.6.14. My question is in regards password storage and scheme/encryption/salts. Currently they are stored in Mariadb password field with a type of varchar and a 255 character length, and are stored as SHA512-CRYPT. I'm wondering if I should keep this as is or when I migrate go to another scheme? I'm thinking argon2i, argon2d, argon2id, sha512, sha512-crypt, tiger2, saltt? -- Robert Lister - email: r...@lentil.org - tel: 020 7043 7996 ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Server migration
I've asked this before, but now it's time to move one server to another, I can't delay the operation anymore (the older server is failing) Both server are pretty old: 1.2.15 Probably, faster way would be to rsync all mailboxes from the older server to the newer one. I can start migrating everything while running then, stop the older server and sync only what is changed, keeping downtime at minimum. Any better solution ? It depends on your situation. If you only have a small userbase with modest amount of mail, you can probably take your whole system down, do a one time migration, then cut over to the new system without too much downtime. If you have medium sized operation, then what you propose above would work. If you have a large userbase (or large user mailboxes), or you want to minimize downtime, you can incrementally migrate per user so that any particular user experiences a small window of outage, but the system is online for everybody else. This, of course, requires a lot more setup and planning. Joseph Tam
Re: Server migration
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, 24 Nov 2017, Gandalf Corvotempesta wrote: I've asked this before, but now it's time to move one server to another, I can't delay the operation anymore (the older server is failing) Both server are pretty old: 1.2.15 Probably, faster way would be to rsync all mailboxes from the older server to the newer one. I can start migrating everything while running then, stop the older server and sync only what is changed, keeping downtime at minimum. Any better solution ? No, it would go this way. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEVAwUBWhuqEMQnQQNheMxiAQJxDQf/UHW0IdjQclo81XtGIzs2Wo6L/h6Zw1gd BBwpS8KaqKSprxOVJY375ybzvwU+POuujmaN2v8TXPRuJY6ptyy57cqfgPPMN1gG eDp4SoDtQQk0Y1rocM9GdNx5yWb3RLukvpAxLXHaFoQlNRkbIB7kCvNofxiCTcdA 1xcQ7rB1gh+HxCOxf+tLWR/S29EqJeIhxlBUGjTcY42t2hQLBnVwqUJN53GkSWet h+V10iihSkpd3mXPbc49DV0NWUZTVMuspFNWp74sEeJSaOTYbPQU+im60n93ZWBO wotPioiQfES561G2+/SOe0ySvG0h92b2ICZWXKRwSRhcCGI4sNdeiw== =pxDV -END PGP SIGNATURE-
Re: Server migration
Don't lose any of the dovecot-* files and your clients should be fine. I've done 1) a couple of times and nobody got hurt.What you should do is keep the two servers (the old and the new one), and once the new one is ready test with your client only (change your client's IMAP/POP server settings). Once you make sure it works right, you can change the config for the other clients. I have no idea about 2) -- Yassine. On Monday, March 20, 2017 8:49 AM, Gandalf Corvotempesta wrote: Hi to all. It's time to migrate an old server to a newer platform Some questions: 1) what happens by changing the pop3/IMAP server on the client? Is the client (Outlook, Thunderbird,...) smart enough to not download every message again? I'm asking this because the easier way to migrate would be move all mailboxes to the new server and then change the hostname on the client 2) what if I add a dovecot proxy on the new server, proxing back all requests to the older one, if the mailbox is still not migrated? Would the whole pop3/IMAP transaction happen through the proxy or there is something an http redirect (or anything similiar to the SIP protocol) ? 3) I think the response to this is no: is dovecot able to log the hostname used for the connection? I have multiple domains pointing to the same IP. Something like the Host header in Http.
Re: Server migration
> On 1 Nov 2016, at 14.10, Tanstaafl wrote: > > On 11/1/2016 3:58 AM, Sami Ketola wrote: >> On 31 Oct 2016, at 13.11, Tanstaafl wrote: >>> Ok, interesting. So... how does dsync do it? Or would it only work >>> between two dovecot servers? >>> >>> I'm interested in migrating from other servers (Office 365 in one case). > >> Dsync does not use IMAP protocol to store the mails to storage but instead >> uses the >> dovecot storage API to do that. Internally we can set what ever properties >> we want to >> including IMAP UIDs and POP3 UIDLs. >> >> Migrating from legacy system should then be done by pulling the mails from >> the >> legacy platform by using the imapc connector and storing them by using the >> internal apis. >> >> We can also store mails to imapc: location but in that case there is many >> properties that >> will be lost due to limitations of the IMAP protocol. > > Thanks Sami, but I don't see a definitive answer top my question in the > above... > > So, when migrating from legacy system (legacy = non-dovecot) using > imapc, is dovecot able to preserver the UIDs? If you fetch emails over imapc and store to dovecot dsync will preserve IMAP UIDs. Sami
Re: Server migration
On 11/1/2016 3:58 AM, Sami Ketola wrote: > On 31 Oct 2016, at 13.11, Tanstaafl wrote: >> Ok, interesting. So... how does dsync do it? Or would it only work >> between two dovecot servers? >> >> I'm interested in migrating from other servers (Office 365 in one case). > Dsync does not use IMAP protocol to store the mails to storage but instead > uses the > dovecot storage API to do that. Internally we can set what ever properties we > want to > including IMAP UIDs and POP3 UIDLs. > > Migrating from legacy system should then be done by pulling the mails from the > legacy platform by using the imapc connector and storing them by using the > internal apis. > > We can also store mails to imapc: location but in that case there is many > properties that > will be lost due to limitations of the IMAP protocol. Thanks Sami, but I don't see a definitive answer top my question in the above... So, when migrating from legacy system (legacy = non-dovecot) using imapc, is dovecot able to preserver the UIDs? Thanks, and my apologies for being a bit dense...
Re: Server migration
> On 31 Oct 2016, at 13.11, Tanstaafl wrote: > > On 10/30/2016 5:32 AM, Sami Ketola wrote: >> On 28 Oct 2016, at 16.54, Tanstaafl wrote: >>> Oh... I thought the --useuid option eliminated this problem? >>> >>> https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt > >> It does not. There is no option at IMAP level to set the UID. >> >> In this case —useuid seems to keep track on source:uid -> dest:uid >> pairs on multiple syncs and uses uid numbers to avoid syncing mails >> as duplicates instead of using headers to do that. > > Ok, interesting. So... how does dsync do it? Or would it only work > between two dovecot servers? > > I'm interested in migrating from other servers (Office 365 in one case). Dsync does not use IMAP protocol to store the mails to storage but instead uses the dovecot storage API to do that. Internally we can set what ever properties we want to including IMAP UIDs and POP3 UIDLs. Migrating from legacy system should then be done by pulling the mails from the legacy platform by using the imapc connector and storing them by using the internal apis. We can also store mails to imapc: location but in that case there is many properties that will be lost due to limitations of the IMAP protocol. Sami
Re: Server migration
On 10/30/2016 5:32 AM, Sami Ketola wrote: > On 28 Oct 2016, at 16.54, Tanstaafl wrote: >> Oh... I thought the --useuid option eliminated this problem? >> >> https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt > It does not. There is no option at IMAP level to set the UID. > > In this case —useuid seems to keep track on source:uid -> dest:uid > pairs on multiple syncs and uses uid numbers to avoid syncing mails > as duplicates instead of using headers to do that. Ok, interesting. So... how does dsync do it? Or would it only work between two dovecot servers? I'm interested in migrating from other servers (Office 365 in one case). Thanks, Charles
Re: Server migration
> On 28 Oct 2016, at 16.54, Tanstaafl wrote: > > On 10/27/2016 8:36 AM, Timo Sirainen wrote: >> On 27 Oct 2016, at 15:29, Tanstaafl wrote: >>> On 10/26/2016 2:38 AM, Gandalf Corvotempesta my only question is: how to manage the email received on the new server during the last rsync phase? >>> >>> Use IMAPSync - much better than rsync for this. > >> imapsync will change IMAP UIDs and cause clients to redownload all >> mails. http://wiki2.dovecot.org/Migration/Dsync should work though. > > Oh... I thought the --useuid option eliminated this problem? > > https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt It does not. There is no option at IMAP level to set the UID. In this case —useuid seems to keep track on source:uid -> dest:uid pairs on multiple syncs and uses uid numbers to avoid syncing mails as duplicates instead of using headers to do that. Sami
Re: Server migration
On 10/27/2016 8:36 AM, Timo Sirainen wrote: > On 27 Oct 2016, at 15:29, Tanstaafl wrote: >> On 10/26/2016 2:38 AM, Gandalf Corvotempesta >>> my only question is: how to manage the email received on the new server >>> during the last rsync phase? >> >> Use IMAPSync - much better than rsync for this. > imapsync will change IMAP UIDs and cause clients to redownload all > mails. http://wiki2.dovecot.org/Migration/Dsync should work though. Oh... I thought the --useuid option eliminated this problem? https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt
Re: Server migration
On 27 Oct 2016, at 15:58, Gandalf Corvotempesta wrote: > > 2016-10-27 14:36 GMT+02:00 Timo Sirainen : >> imapsync will change IMAP UIDs and cause clients to redownload all mails. >> http://wiki2.dovecot.org/Migration/Dsync should work though. > > Just to be sure: dsync from the *new* node would connect via IMAP to > the older node and transfer everything ? > By running this: > > doveadm -o mail_fsync=never sync -1 -R -u user@domain imapc: > > should be OK if newer mails are arrived on the new server ? Yes.
Re: Server migration
2016-10-27 14:36 GMT+02:00 Timo Sirainen : > imapsync will change IMAP UIDs and cause clients to redownload all mails. > http://wiki2.dovecot.org/Migration/Dsync should work though. Just to be sure: dsync from the *new* node would connect via IMAP to the older node and transfer everything ? By running this: doveadm -o mail_fsync=never sync -1 -R -u user@domain imapc: should be OK if newer mails are arrived on the new server ?
Re: Server migration
On 27 Oct 2016, at 15:29, Tanstaafl wrote: > > On 10/26/2016 2:38 AM, Gandalf Corvotempesta > wrote: >> This is much easier than dovecot replication as i can start immedialy with >> no need to upgrade the old server >> >> my only question is: how to manage the email received on the new server >> during the last rsync phase? > > Use IMAPSync - much better than rsync for this. imapsync will change IMAP UIDs and cause clients to redownload all mails. http://wiki2.dovecot.org/Migration/Dsync should work though.
Re: Server migration
On 10/26/2016 2:38 AM, Gandalf Corvotempesta wrote: > This is much easier than dovecot replication as i can start immedialy with > no need to upgrade the old server > > my only question is: how to manage the email received on the new server > during the last rsync phase? Use IMAPSync - much better than rsync for this.
Re: Server migration
2016-10-26 8:57 GMT+02:00 Aki Tuomi : > If you are moving from 1.x to 2.x, I think you should make some trials > first, and preferably move the user one at a time, blocking access to > old server/new server during move. It is very forklift upgrade, much danger. Yes, I'll do some test migration before moving the whole server. Maildir structure isn't changed between 1.x and 2.x, thus all emails should be safe. I have to test the new 2.2 configuration to see if existing users are able to log-in but how can I test if existing client would be able to preserve the mail ids without downloading everything again?
Re: Server migration
On 26.10.2016 09:38, Gandalf Corvotempesta wrote: > Il 26 ott 2016 8:30 AM, "Aki Tuomi" ha scritto: >> I would recommend using same major release with replication. >> >> If you are using maildir++ format, it should be enough to copy all the >> maildir files over and start dovecot on new server. >> > This is much easier than dovecot replication as i can start immedialy with > no need to upgrade the old server > > my only question is: how to manage the email received on the new server > during the last rsync phase? > As i wrote previously, i have some huge maildirs where rsync take hours to > scan all files > i can't keep the server down for hours or customers won't receive any new > emails, so, after the initial sync i have to move the mailbox on the new > server (only for deliveries) . In this way I'll not loose any emails but > the new servers as newer data than the old server. > When doing rsync with --delete, the news mails would be removed > > A solution could be to disable customer access to the new server and put > "new" directory in rsync exclude. Doing this won't delete the newly > received emails as the "new" directory isn't synced. > and no one osd able to move from new to cur as users are blocked for login. If you are moving from 1.x to 2.x, I think you should make some trials first, and preferably move the user one at a time, blocking access to old server/new server during move. It is very forklift upgrade, much danger. Aki
Re: Server migration
Il 26 ott 2016 8:30 AM, "Aki Tuomi" ha scritto: > I would recommend using same major release with replication. > > If you are using maildir++ format, it should be enough to copy all the > maildir files over and start dovecot on new server. > This is much easier than dovecot replication as i can start immedialy with no need to upgrade the old server my only question is: how to manage the email received on the new server during the last rsync phase? As i wrote previously, i have some huge maildirs where rsync take hours to scan all files i can't keep the server down for hours or customers won't receive any new emails, so, after the initial sync i have to move the mailbox on the new server (only for deliveries) . In this way I'll not loose any emails but the new servers as newer data than the old server. When doing rsync with --delete, the news mails would be removed A solution could be to disable customer access to the new server and put "new" directory in rsync exclude. Doing this won't delete the newly received emails as the "new" directory isn't synced. and no one osd able to move from new to cur as users are blocked for login.
Re: Server migration
On 26.10.2016 09:27, Gandalf Corvotempesta wrote: > Il 24 ott 2016 5:11 PM, "Michael Seevogel" ha scritto: >> I meant your old server. With "old" I was expecting something like Debian > Sarge or SuSE Linux 9.3. That would have been really old, but since you are > on Debian Squeeze, I would definitely choose the way with an upgraded > Dovecot version and its replication service. > > Is 2.1 from squeeze-backports enough to start the replication over a newer > server with dovecot 2.2? Is this supported or both server must run the same > version? > > I've looked around but the replication system is still not clear to me. > Any howto explaining this in details? Hi! I would recommend using same major release with replication. If you are using maildir++ format, it should be enough to copy all the maildir files over and start dovecot on new server. Aki Tuomi Dovecot oy
Re: Server migration
Il 24 ott 2016 5:11 PM, "Michael Seevogel" ha scritto: > I meant your old server. With "old" I was expecting something like Debian Sarge or SuSE Linux 9.3. That would have been really old, but since you are on Debian Squeeze, I would definitely choose the way with an upgraded Dovecot version and its replication service. Is 2.1 from squeeze-backports enough to start the replication over a newer server with dovecot 2.2? Is this supported or both server must run the same version? I've looked around but the replication system is still not clear to me. Any howto explaining this in details?
Re: Server migration
Am 24.10.2016 um 15:25 schrieb Gandalf Corvotempesta: 2016-10-24 14:47 GMT+02:00 Michael Seevogel : If your server OS supports newer Dovecot versions then I would highly suggest you to upgrade to Dovecot 2.2.xx (or at least to the latest 2.1) and set up Dovecot's replication[1] feature. Are you talking about the new server or the older one that I have to replace? The new server has to be installed from scratch, so, yes, I can use Dovecot 2.2 from Jessie I meant your old server. With "old" I was expecting something like Debian Sarge or SuSE Linux 9.3. That would have been really old, but since you are on Debian Squeeze, I would definitely choose the way with an upgraded Dovecot version and its replication service. The "old" server is based on Squeeze, I can upgrade that to Wheezy and install Dovecot 2.2 from wheezy-backports but I have huge trouble when I've tried to do the same on a similiar server. I was unable to upgrade the dovecot configuration by following the documentation as this didn't work: doveconf -n -c /etc/dovecot/dovecot.conf > dovecot-2.conf I had an empty dovecot-2.conf file, no warning or output at all. It did nothing. Well, I'am not too familiar with Debian since I'am a Red Hatter but perhaps you could use the binaries from there: http://wiki2.dovecot.org/PrebuiltBinaries Dunno if you have to rebuild the binaries, or if you can install them straight on Squeeze. You could also try to convert your old dovecot.conf on a different machine (maybe your new server?) and then just copy it back to your old server. As a last straw you could certainly adapt the dovecot.conf for Dovecot 2.2 manually, it shouldn't be too complicated, but this is totally up to you. With this method you can actually archieve a smooth migration while your current server replicates all emails in real time to your new server, including new incoming emails and also mailbox changes to your new server and when the migration is done you'll just have to change your DNS and disable the Replication service. Cool. Any guide about this ? Should I start the replication on one side and wait for finish before pointing the mailbox to the new server? How to setup and start replication is described here: http://wiki2.dovecot.org/Replication Also make sure that you migrate/copy your userdb from the old server to the new server and that you properly test the user-mailbox access on the new server before you start the replication process. Regarding replication: I would wait with adjusting the DNS records until the replication has finished and you know that the new server works as expected. However, you may want to keep the replication process running for one or two more days to catch emails still arriving due to DNS caching times on your old server. The same may apply to mailusers that still access your old server via POP3/IMAP. Best regards Michael Seevogel
Re: Server migration
2016-10-24 14:47 GMT+02:00 Michael Seevogel : > P.S. You should think about to use on the new server mdbox as mailbox > format. > That's kinda a hybrid of mbox and maildir and benefits of features of both > its predecessors. However, backup and restoring is in case of mdbox "a bit" > different. Just have a read... No, I don't like that format, for this: This also means that you must not lose the dbox index files, they can't be regenerated without data loss additionally, this means to change even our LDA, as neither Exim or Postfix are able to deliver messages.
Re: Server migration
2016-10-24 14:47 GMT+02:00 Michael Seevogel : > If your server OS supports newer Dovecot versions then I would highly > suggest you to upgrade to Dovecot 2.2.xx (or at least to the latest 2.1) and > set up Dovecot's replication[1] feature. Are you talking about the new server or the older one that I have to replace? The new server has to be installed from scratch, so, yes, I can use Dovecot 2.2 from Jessie The "old" server is based on Squeeze, I can upgrade that to Wheezy and install Dovecot 2.2 from wheezy-backports but I have huge trouble when I've tried to do the same on a similiar server. I was unable to upgrade the dovecot configuration by following the documentation as this didn't work: doveconf -n -c /etc/dovecot/dovecot.conf > dovecot-2.conf I had an empty dovecot-2.conf file, no warning or output at all. It did nothing. > With this method you can actually archieve a smooth migration while your > current server replicates all emails in real time to your new server, > including new incoming emails and also mailbox changes to your new server > and when the migration is done you'll just have to change your DNS and > disable the Replication service. Cool. Any guide about this ? Should I start the replication on one side and wait for finish before pointing the mailbox to the new server? > If you don't want or cannot set up replication you could still do a one-shot > migration via Dovecot's dsync[2] on the new server, pulling the mails from > the old. 50GB isn't that much as long as your two servers are at least > connected with 100 Mbit to the inet. You may want to block for the time of > the migration via iptables your users accessing Dovecot. However, under the > bottom-line, if this is really necessary depends on you and the needs of > your mailusers/customers. I can't block the whole server. I have to migrate 1 user at once. But I can disable the pop3/imap access for that user, so noone is changing the files during the migration (except for the postfix/exim delivery agent) > P.S. You should think about to use on the new server mdbox as mailbox > format. > That's kinda a hybrid of mbox and maildir and benefits of features of both > its predecessors. However, backup and restoring is in case of mdbox "a bit" > different. Just have a read... > > > [1] http://wiki.dovecot.org/Replication > [2] http://wiki2.dovecot.org/Migration/Dsync Thank you
Re: Server migration
2016-10-24 11:23 GMT+02:00 Karol Augustin : > When I am doing this I just turn off both servers for the third sync. > Its short enough to not cause much problem. And then after third sync I > start the new server and all clients can connect to it so I also > mitigate any problems resulting from clients that would be still > connected to the old server. The last issue depends on the way you force > everyone to use new server (DNS, routing, etc). The speed for third sync depends on the number of files to be scanned. I have mailboxes with tons of very small emails, thus even if the first two sync has transferred all datas, the scan made by rsync to check which files needs to be transferred requires many hours. My own mailbox has 80GB of mails. I can sync everything on a new server and then start a new rsync phase. this new phase requires exactly 1 hours and 49 minutes (as I can see from the last night backup). Transferred data: 78MB. 1 hours and 49 minutes to transfer only 78MB. > Remember that beside the new emails that could arrive during sync you > have also all sorts of user-generated operations as move, delete etc. So > if you just do 3rd rsync without --delete you can end up duplicating > users' emails if they move them during procedure. By shutting down both servers, the "--delete" argument could be used with no issues.
Re: Server migration
Am 24.10.2016 um 09:00 schrieb Gandalf Corvotempesta: Hi i have to migrate, online, a dovecot 1.2.15 to a new server. Which is the best way to accomplish this? I have 2 possibility: 1) migrate from the very old server to a newer server with the same dovecot version 2) migrate from the very old server to a new server with the latest dovecot version can i simply use rsync to sync everything and, when the sync is quick, move the mailbox from the old server to the new server? My biggest concern is how to manage the the emails that are coming during the server switch. Let's assume a 50gb maildir , the first sync would require hours to complete (tons of very small files) do i can't shutdown the mailbox. The second sync would require much less time and would also sync the email received during the first sync (but the mailbox is still receiving new emails) now, as third phase, i can move the mailbox to the new server (by changing the postfix configuration) so that all new emails are received on the new server and then start the last rsync (by removing the --delete flag or any new emails would be deleted as not existsnt on the older server) Any better solution? If your server OS supports newer Dovecot versions then I would highly suggest you to upgrade to Dovecot 2.2.xx (or at least to the latest 2.1) and set up Dovecot's replication[1] feature. With this method you can actually archieve a smooth migration while your current server replicates all emails in real time to your new server, including new incoming emails and also mailbox changes to your new server and when the migration is done you'll just have to change your DNS and disable the Replication service. If you don't want or cannot set up replication you could still do a one-shot migration via Dovecot's dsync[2] on the new server, pulling the mails from the old. 50GB isn't that much as long as your two servers are at least connected with 100 Mbit to the inet. You may want to block for the time of the migration via iptables your users accessing Dovecot. However, under the bottom-line, if this is really necessary depends on you and the needs of your mailusers/customers. Best regards Michael Seevogel P.S. You should think about to use on the new server mdbox as mailbox format. That's kinda a hybrid of mbox and maildir and benefits of features of both its predecessors. However, backup and restoring is in case of mdbox "a bit" different. Just have a read... [1] http://wiki.dovecot.org/Replication [2] http://wiki2.dovecot.org/Migration/Dsync
Re: Server migration
On 2016-10-24 8:00, Gandalf Corvotempesta wrote: > can i simply use rsync to sync everything and, when the sync is quick, move > the mailbox from the old server to the new server? My biggest concern is > how to manage the the emails that are coming during the server switch. > > Let's assume a 50gb maildir , the first sync would require hours to > complete (tons of very small files) do i can't shutdown the mailbox. The > second sync would require much less time and would also sync the email > received during the first sync (but the mailbox is still receiving new > emails) > now, as third phase, i can move the mailbox to the new server (by changing > the postfix configuration) so that all new emails are received on the new > server and then start the last rsync (by removing the --delete flag or any > new emails would be deleted as not existsnt on the older server) > > Any better solution? When I am doing this I just turn off both servers for the third sync. Its short enough to not cause much problem. And then after third sync I start the new server and all clients can connect to it so I also mitigate any problems resulting from clients that would be still connected to the old server. The last issue depends on the way you force everyone to use new server (DNS, routing, etc). Remember that beside the new emails that could arrive during sync you have also all sorts of user-generated operations as move, delete etc. So if you just do 3rd rsync without --delete you can end up duplicating users' emails if they move them during procedure. Best, Karol
[Dovecot] SOLVED (user error) - WAS Re: Server Migration Attempt - new messages DELETED after secondary rsyncs
On 2013-12-27 9:33 AM, Charles Marcus wrote: Damn! How did I miss that. The 'T' is for 'trashed', so of course, when the Inbox is expunged, they will be deleted... Thanks Rick, for the gentle clue stick... Apologies to all for the noise. -- Best regards, */Charles/*