Aw: Re: convert mdbox to maildir
Yes, you are right. The problems are not of technical nature. The reason seems to be some sort of fear (and "admins"). We have all we need. We have the old dovecot config, we have the mdbox files and the complete directory structure. We are simply not allowed to do all the stuff on the live system. Even if dsync backup exists that does not modify anything. Currently it is planned that the real conversion will take place on a weekend. We are setting up an environment that enables us to do that (how well does it work, how long will it take). They are only around 500 users and ca 4TB (incl public folders). I don't see a problem with that. And if they feel some sort of fear, then I can understand it (except the "admins" who should know better). We do what the customer wants us to do. And yes, they pay pretty well for working on weekends. What I would like to do is set up the real new machine with everything - the new live system. Put all the mdbox files/structures into a directory and run the conversion for each user. So the first question is if the mdbox files can really be "simple files in folders" or if they have to be "hosted" by an imap server, too. (As I understood, the destination can be our final imap server.) And I believe we can have the source in the filesystem (what would be mach faster). The other question is if we can run some of these conversion in parallel? I would say "why not". > Gesendet: Sonntag, 14. August 2022 um 18:24 Uhr > Von: "Steve Litt" > An: dovecot@dovecot.org > Betreff: Re: convert mdbox to maildir > > On Sat, 2022-08-13 at 18:36 +0200, lutz.niede...@gmx.net wrote: > > > > The real problem is that we must not use the running, old dovecot > > installation. > > So we are not able to connect to the old server, pull all folders and mails > > and > > create a new maildir structure. Currently, we can't do anything against > > it. What > > we get are the users' mdbox files. > > Why not? Is the old server broken beyond repair? If not, is there an actual > reason > behind, or is it just a arbitrary decision capable of being swayed by facts? > Is the > customer willing to pay for the large increase in time to rebuild the whole > thing? > > Will they at least let you rsync the old server's entire mdbox structure to a > machine where you can do your conversion? I don't know, to me their act of > giving > you some files and saying "it's your problem now" seems arbitrary, and you > should > charge them a lot of money. > > > > > > Is there any way to convert mdbox files and structures to maildir directly > > from > > filesystem? > > Or do we have to build a copy of the old machine (dovecot only, or -maybe > > better- > > a vm) and then use doveadm backup? > > Or is it ok to just set up the completely new installation, set > > mail_location to > > where the new Maildirs will be, like maildir:~/Maildir and then run > > something like > > doveadm backup mdbox:/tmp/$user/mdbox -u $user? Will this transfer all > > mails and > > folders or do we have to keep an eye on some specific things? > > All I know about mdbox comes from this document: > https://doc.dovecot.org/admin_manual/mailbox_formats/dbox/ > > Quoting a specific sentence: "One of the main reasons for dbox’s high > performance is > that it uses Dovecot’s index files as the only storage for message flags and > keywords, so the indexes don’t have to be “synchronized”. Dovecot trusts that > they’re always up-to-date (unless it sees that something is clearly broken). > This > also means that you must not lose the dbox index files, as they can’t be > regenerated > without data loss." > > The quote says *dbox*, but it's in a section devoted to both dbox and mdbox, > so I'm > thinking it might be true of both. Have they given you the index files? If > not, it > sounds to me like any regeneration would be an approximation at best. > > Do you have a way of accurately putting together the directory structure of > the > former mdbox system? > > My experience 10 years ago converting about a quarter million kmail emails to > Dovecot Maildir is it takes about an hour to transfer between 25,000 and > 50,000 > emails, but of course that was on a much more anemic machine than I have > today. I'd > guess that if you have both databases on the same machine, the way I did ten > years > ago, the process will go pretty fast. Here's a count of my Dovecot Maildir > today: > > [root@mydesk Maildir]# du -hs > 16G . > [root@mydesk Maildir]# find . | wc -l > 734906 > [root@mydesk Maildir]# > > I don't know much about your particular situation, but it seems to me like the > majority of your problem isn't technical. > > SteveT >
Aw: RE: convert mdbox to maildir
mdbox is a proprietary format, only dovecot can handle it (afaik). Postfix can write maildir natively, other MTAs, too. There are many tools to help in case something strange happens. I believe that using maildir seems to be not that bad. And I don't see the real benefits of mdbox. Maybe with "old" spindle driven devices, you may have a performance benefit, but with SDDs... If you use the right filesystem, use raid 10 Yes, it is said that mdbox is performing better. The largest installations we had with dovecot were about 2k users (maildir was used) and another one had about 3k users (with mdbox), they all have branch offices that are connected to their "home" via some sort of vpn or direct connect by the ISP. We did not see a real difference between maildir and mdbox, maybe because of the surrounding conditions. And I believe mdbox may be faster, but we did not have the critical masses to see that. > Gesendet: Sonntag, 14. August 2022 um 11:47 Uhr > Von: "Marc" > An: "Patrick Domack" , "dovecot@dovecot.org" > > Betreff: RE: convert mdbox to maildir > > So? that is why you have this lmtp not? Afaik was mdbox created to solve the > (performance) issues with mbox and maildir etc. So I just wonder what the > logics is behind chosing maildir current day. > > > > > Not sure what this had to do with the question asked. > > You're using mbox, but he is using mdbox. > > Postfix is unable to write to mdbox itself. > > > > > > Quoting justina colmena ~biz : > > > > > *My* inbox gets filled with thousands of emails, more or less > > > commercial content and trivial notifications from shopping online, > > > and postfix crashes and will not accept new messages if the file > > > "/var/mail/justina" becomes too large. > > > > > > Configuring postfix to deliver the mail to "~/Maildir" solved that > > problem. > > > > > > I still need to configure a sieve or a filter or some nicer > > > mechanism to clear out messages that are either outright spam or too > > > old or no longer of interest to me. > > > > > > On August 13, 2022 10:00:36 AM AKDT, Marc > > wrote: > > >>> > > >>> We need to move all users from one (pretty old) installation of > > dovecot > > >>> to a new one. > > >>> The old one uses mdbox for users' mailboxes and maildir for > > >>> shared/public mailboxes. > > >>> The new one must be maildir only. > > >> > > >> why did you decide to move to maildir? > >
Aw: Re: convert mdbox to maildir
Thank you. Do you means two servers? One as source and one as destination? As I understood, destination does not need an imap server... Description of dsync says: ... All of these can be used within the same server ... Adapted it says: mdbox -> maildir migration. Set mail_location=maildir:~/Maildir and run dsync -u username mirror mdbox:~/mdbox For me it looks like Maildir is on our new server (imap) and mdbox is in the user's homedir on the new server (without the need of imap). Gesendet: Samstag, 13. August 2022 um 18:58 Uhr Von: "Olivier DUCROT" An: lutz.niede...@gmx.net Betreff: Re: convert mdbox to maildir Hi You should create a new server with the same mbox structure, si that you can easyly connect to it and then create a brand new server with maildir. Le 13 août 2022 à 18:36, lutz.niede...@gmx.net a écrit : Hi! We need to move all users from one (pretty old) installation of dovecot to a new one. The old one uses mdbox for users' mailboxes and maildir for shared/public mailboxes. The new one must be maildir only. I believe that I can just copy the shared/public maildir structure to the new installation without problems. Am I right? Do I have to expect any trouble? (Set owner, permissions etc...) The real problem is that we must not use the running, old dovecot installation. So we are not able to connect to the old server, pull all folders and mails and create a new maildir structure. Currently, we can't do anything against it. What we get are the users' mdbox files. Is there any way to convert mdbox files and structures to maildir directly from filesystem? Or do we have to build a copy of the old machine (dovecot only, or -maybe better- a vm) and then use doveadm backup? Or is it ok to just set up the completely new installation, set mail_location to where the new Maildirs will be, like maildir:~/Maildir and then run something like doveadm backup mdbox:/tmp/$user/mdbox -u $user? Will this transfer all mails and folders or do we have to keep an eye on some specific things? I thank you very much and cheers! -lutzn
Aw: RE: convert mdbox to maildir
It was a decision we have not been involved in. That's it. We have to do what we are paid for. But I did not want to start a meta-discussion. I was looking for ideas and technical experience. (And I agree with Justina that maildir seems to be much more manageable. It is a well-known format.) Gesendet: Samstag, 13. August 2022 um 21:08 Uhr Von: "justina colmena ~biz" An: dovecot@dovecot.org Betreff: RE: convert mdbox to maildir *My* inbox gets filled with thousands of emails, more or less commercial content and trivial notifications from shopping online, and postfix crashes and will not accept new messages if the file "/var/mail/justina" becomes too large. Configuring postfix to deliver the mail to "~/Maildir" solved that problem. I still need to configure a sieve or a filter or some nicer mechanism to clear out messages that are either outright spam or too old or no longer of interest to me. On August 13, 2022 10:00:36 AM AKDT, Marc wrote: We need to move all users from one (pretty old) installation of dovecot to a new one. The old one uses mdbox for users' mailboxes and maildir for shared/public mailboxes. The new one must be maildir only. why did you decide to move to maildir? -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
