On Mar 22, 2013, at 2:47 AM, Timo Sirainen wrote: > On Wed, 2013-03-20 at 17:40 -0600, dormitionsk...@hotmail.com wrote: >> On Mar 20, 2013, at 8:59 AM, Timo Sirainen wrote: >> >>> On Wed, 2013-03-20 at 08:15 -0600, dormitionsk...@hotmail.com wrote: >>> >>>> My experience with IMAP over the internet with a couple of servers outside >>>> our monastery (while I was in it, and we have considerably better download >>>> speeds than upload) has always been that IMAP has always been incredibly >>>> slow. So, I've always just allowed users to connect to the IMAP server >>>> via webmail. It's slow, but usable. >>> >>> Another idea: Get some cheap server from outside, use dsync replication >>> to keep it synced with your internal one, and set up DNS so that users >>> get directed to the fastest server. http://wiki2.dovecot.org/Replication >>> >>> >> >> I LIKE this idea, but I have a few questions about it to see if it >> would be appropriate for our situation. There are a few other things >> to consider that I didn't mention before because they did not seem >> relevant earlier. >> >> First off, I'd just like to say that we have a web server set up at a >> location outside of our monastery that hosts all of our websites. I'm >> currently in the process of building new servers to replace both it >> and our current email server. So, assuming this is both plausible for >> our situation, and within my capabilities, I should be able to work on >> this at my leisure, and get the initial sync of our emails done while >> on the same LAN. >> >> So, the additional info and questions are the following: >> >> 1.) Our download speeds are decent enough, but in addition to having >> poor upload speeds, we also have very strict limits on how much we are >> able to download. And we use almost every bit of it every day. We >> cannot get more, either. We have unlimited downloads for four hours >> at night, however. > > If a delayed sync isn't a problem, you could do it only once at nights. > You wouldn't need to use the replicator service at all, just run > "doveadm sync -f -A -d" in a cronjob. > >> 2.) We have very large message archives. We basically have 95% of >> the emails we've received for the past 16 years. So, the sync *must* >> only update items that have been changed. Is this how it it would >> work? > > dsync can do full sync (= all messages' metadata is sent + new messages' > contents), "changed sync" (= same as full sync, but only for changed > folders) or incremental sync (= only new messages' metadata + contents > are sent). The incremental sync is what replicator service does while > it's running, but it's still currently doing a full sync at startup. > > A nightly cronjob could do incremental syncing also, but it would have > to run dsync separately for each user and store the sync state to some > file. > > The "changed sync" works well enough usually, but it has a problem if > both replicas have had exactly the same amount of changes it doesn't > realize that there may be differences between them and skip it. > >> 3.) We are currently using uw-imap with mbox. If we switch to >> Dovecot, using Maildir format, will the sync only update the new >> messages and the header files for any folders that have been changed? > > It works the same with all mailbox formats. Headers and bodies aren't > synced separately, but metadata (= ~100 bytes/msg maybe) is. > >> 4.) I thought I read somewhere in Dovecot's documentation last night >> that it has a 50 mb limit on folders. It can't write anything larger >> than that. Does this sound familiar? (Now I can't find it!) If so, >> is that for mbox? We currently have some mbox folders whose files are >> significantly larger than that. If we convert to Maildir format, >> where the individual messages are in their own files, could a folder >> contain messages totaling more than 50 MB using Dovecot? > > Dovecot has no such limit. Postfix by default has set a file size limit > for 50 MB, which effectively limits mbox sizes to 50 MB, but it can be > removed with Postfix mailbox_size_limit setting. > >> 4a. -- Oops. I just noticed this: "NOTE2: sdbox/mdbox mailbox >> formats are recommended for replication. Maildir still has some issues >> (although probably not noticeable in normal use)." Should I consider >> this a show-stopper for syncing like this? > > With v2.2 I don't think there's much of a difference anymore. > >> 5.) In the http://wiki2.dovecot.org/Replication page, would this be >> continuously synced each time a user sends, receives, deletes, or >> moves messages, etc.? Or would it be periodically synced? > > With replicator it syncs immediately when something changes. > >> 6.) Also, that page does not make it clear if one server is like the >> "master" and the other the "slave". Do I do the same changes to both >> servers? > > Both servers are equal. Setup both servers exactly the same. > >> If, given the above additional information, it would not be an >> appropriate solution for us, this suggestion about syncing the two >> servers gave me another idea. >> >> I was thinking, "Well, I wonder if I could just sync the Inboxes? We >> don't really need the folders synced. In the highly unlikely event a >> person would ever need something from one of his folders, he could >> always just log into the (slow) monastery server through web mail and >> get it that way." > > If you're syncing via ssh, you can give "-m inbox" parameter to > dsync_remote_cmd setting and it syncs only INBOX. But it's still > unnecessarily running dsync whenever anything changes. With some > hardcoding it would be easy to change that though. > >> (When we are on the road, we are generally working real hard, and we >> don't answer any more emails or do any other computer work than we >> absolutely have to.) >> >> So, that led me to the idea to simply set up some message rules in >> procmail in our (slow) monastery server to copy any incoming messages >> to the server offsite in addition to delivering them locally. For the >> most part, that would be sufficient for us -- and considerably >> easier. >> >> The only downsides to this are that when we reply to messages, they >> would not be marked as having been replied to, and we wouldn't have >> copies of our replies on our main server. >> >> The not being marked as replied to is not a big deal. I know we could >> manually copy any sent messages from one server to the other when we >> returned to the monastery, if we really wanted to, but does anyone >> know of a better way to do it? > > The users would then need to have two accounts I think, one for internal > and one for outside mail. Otherwise whenever they switch between servers > they need a full resync. > > >
Thank you very, very much, Timo. I appreciate this more than you can possibly imagine. And thank you, everyone else who contributed to this thread, as well. I appreciate all of your ideas and suggestions very much, too. My current plan is to to move the bulk of our archives to another server -- actually, a different zone on the same Solaris machine -- and dsync just our primary email server at night. The archives are rarely used, and don't need to be accessed from outside. In fact, I'll probably leave them in uw-imap so I don't have to convert them to dovecot. I think that should probably work the best for us, if I can pull it off. I'm a lot more of a programmer than a system admin. Thank you all again.
