My plan is to do that nifty trick with the copying over of the user
tables then point the port forward at the new host so we don't drop
any email.
Then I'm looking at the "best" way of moving the bulk of the email
over. I don't want to just mysqldump it over, I want it all in the
single instance storage system.
Any recommendations? The last few times I have tried imapcopy and
imapsync they seem to crash or cause dbmail to crash or something
equally silly (not to mention they are slow as a wet week). The dump
to mbox tool lost/merged or some such a few emails last time I tried
it as i recall although it was nice and fast.
Well it looks like the dbmail mbox exporter and the mbox-import tool
like to play nicley now, I copied over a gig worth of email to test the
new server and it seems pretty good. I did have imap crash though with
no error in dbmail.err or anything in dbmail.log however this was in
mail.log
Nov 17 03:00:09 mail dbmail/maintenance[12270]: [0x9f60958]
Warning:[db] db_set_headercache(+1604): error caching headers for
physmessage: [11406]
Nov 17 03:00:09 mail dbmail/maintenance[12270]: [0x9f60958] Warning:[db]
db_set_headercache(+1604): error caching headers for physmessage: [9262]
Nov 17 03:00:09 mail dbmail/maintenance[12270]: [0x9f60958] Warning:[db]
db_set_headercache(+1604): error caching headers for physmessage: [9246]
Nov 17 03:16:40 mail dbmail/deliver[7708]: [0x95ee9c0] Error:[message]
_header_cache(+1510): SQLException: Duplicate entry
'[EMAIL PROTECTED]' for key 3
Nov 17 03:18:13 mail dbmail/deliver[12193]: [0x9f429c0] Error:[message]
_header_cache(+1510): SQLException: Duplicate entry
'[EMAIL PROTECTED]' for key 3
That might be my fault for loading the data several times but it seems
unlikely as its still loaded through dbmail-deliver.
Whilst on the subject of loading data I made a frontend for mbox2dbmail
that should load all the accounts and mailboxes dumped by a wildcard
dump from the source system and as a bonus do it in parallel (ie 3
simultaneous insertion programs).
Its tailored pretty specifically to the setup i was using and isn't
general purpose in anyway yet however it was useful to me ;->
basically copy the results of your dbmail-export -u *
to a directory called "migrate" on the target system
(so ~/migrate/[EMAIL PROTECTED] would be a result)
put this program thus named migrate2dbmail.py version 0.01 and
mbox2dbmail in the parent directory (IE ~)
set how many threads you would like to use to insert (i used 3 because
thats how many CPU's i gave the VM)
then run it
python migrate2dbmail.py
It should spit out N text files for you
thread-1 thread-2 thread-3
with the commands to run mbox2dbmail in it.
so chmod +x thread*
screen (its probably going to take a while)
(./thread-1 &) ; (./thread-2 &) ; (./thread-3 &)
Its not tuned in any way shape or form it just splits the mbox files
directly into the 3 processes but its better than nothing.
It got me a pretty linear improvement in performance, I was hitting
~240% CPU use by dbmail-deliver 10% by python and 50% or so by mysql.
The disk was pretty relaxed, nowhere near hitting that. I estimate it
was inserting 5-10 emails a second or so, into a paravirtualised machine
on a quad core host with a generic 7200RPM 80GB drive as the single disk.
The emails themselves start appearing a minute or 2 after the thing says
they were inserted so don't get too impatient ;->
==========================================================
# Copyright (C) 2008 Jake Anderson <jake at vapourforge dot com>
# migrate2dbmail Version 0.01
#
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either
# version 2 of the License, or (at your option) any later
# version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
#
import os
import re
from os.path import join, getsize
os.chdir('migrate/')
numthreads = 3
outputfile = {}
for i in range(1,numthreads+1):
outputfile[i] = ''
currthread = 0
for root, dirs, files in os.walk('.'):
currthread += 1
if currthread > numthreads:
currthread = 1
print currthread
if root.count('/') == 1:
#this is the top level folder, set the account name
curract = root.split('/')[1]
for file in files:
mboxpath = os.path.abspath(os.path.join(root,file))
dest_path = '/'.join(root.split('/')[2:])+'/'+file[:-5]
if dest_path[0:1] == "/":
dest_path = dest_path[1:]
# print "python mbox2dbmail -p /usr/sbin/dbmail-deliver -t mbox
-u \"%s\" -m \"%s\" -b \"%s\"" % (curract,mboxpath,dest_path)
outputfile[currthread] += "python mbox2dbmail -p
/usr/sbin/dbmail-deliver -t mbox -u \"%s\" -m \"%s\" -b \"%s\"\n" %
(curract,mboxpath,dest_path)
# print dest_path
os.chdir('..')
for i in range(1,numthreads+1):
f = open("thread-%s" % (i),'w')
f.write(outputfile[i])
_______________________________________________
DBmail mailing list
[email protected]
https://mailman.fastxs.nl/mailman/listinfo/dbmail
