Re: Preserving seen state when converting from mbox
Ho boy ... that make me cry ... However, after verification I was *WAY* off-track concerning the size of the mbox we need to transfer. Apparently, there is only around 800 MB (phew!). The machine is a recent quad-XEON box 2 GB of RAM. By contrast, what does your hardware looked like ? On Thu, Sep 25, 2003 at 10:52:59PM +0200, Jure Pecar wrote: On Thu, 25 Sep 2003 15:04:48 -0400 (EDT) Igor Brezac [EMAIL PROTECTED] wrote: I do not think you have enough time to transfer several gigs (if not more) of email in just a few hours over IMAP protocol. Just for the reference, it took us about two weeks to transfer about 300gb of mailspool from one live system to another (with perdition in front), on a reasonably fast hw, with custom php scripts doing imap2imap copy (and related user imgration) in parallel. -- Jure Pecar -- Etienne GoyerLinux Québec Technologies Inc. http://www.LinuxQuebec.com [EMAIL PROTECTED]
Re: Preserving seen state when converting from mbox
On Thu, Sep 25, 2003 at 03:04:48PM -0400, Igor Brezac wrote:
On Thu, 25 Sep 2003, Etienne Goyer wrote:
On Thu, Sep 25, 2003 at 01:29:30PM -0400, Igor Brezac wrote:
mailutil append /path/to/mbox
{new.cyrus.server/user=mbox_cyrus_user/authuser=proxy_user/norsh}INBOX
You made my day !!! Exactly what I needed.
You will be prompted for the 'proxy_user' password. You can hardcode the
password with a small change in mailutil.c. Let me know if you need help
with this...
I'd like help, if possible. The code is pretty dense, so if you know
where I have to plug the password, that would save me ton of time.
Thanks a lot !
--
Etienne GoyerLinux Québec Technologies Inc.
http://www.LinuxQuebec.com [EMAIL PROTECTED]
Re: Preserving seen state when converting from mbox
On Fri, 26 Sep 2003, Etienne Goyer wrote:
On Thu, Sep 25, 2003 at 03:04:48PM -0400, Igor Brezac wrote:
On Thu, 25 Sep 2003, Etienne Goyer wrote:
On Thu, Sep 25, 2003 at 01:29:30PM -0400, Igor Brezac wrote:
mailutil append /path/to/mbox
{new.cyrus.server/user=mbox_cyrus_user/authuser=proxy_user/norsh}INBOX
You made my day !!! Exactly what I needed.
You will be prompted for the 'proxy_user' password. You can hardcode the
password with a small change in mailutil.c. Let me know if you need help
with this...
I'd like help, if possible. The code is pretty dense, so if you know
where I have to plug the password, that would save me ton of time.
Try:
--- mailutil.c.orig Fri Sep 26 11:39:12 2003
+++ mailutil.c Fri Sep 26 11:39:38 2003
@@ -598,7 +598,7 @@
username[NETMAXUSER-1] = '\0';
if (s = strchr (username,'\n')) *s = '\0';
}
- strcpy (password,getpass (password: ));
+ strcpy (password, your_pass);
}
--
Igor
Re: Preserving seen state when converting from mbox
Thank you very much, work perfectly.
On Fri, Sep 26, 2003 at 11:43:15AM -0400, Igor Brezac wrote:
On Fri, 26 Sep 2003, Etienne Goyer wrote:
On Thu, Sep 25, 2003 at 03:04:48PM -0400, Igor Brezac wrote:
On Thu, 25 Sep 2003, Etienne Goyer wrote:
On Thu, Sep 25, 2003 at 01:29:30PM -0400, Igor Brezac wrote:
mailutil append /path/to/mbox
{new.cyrus.server/user=mbox_cyrus_user/authuser=proxy_user/norsh}INBOX
You made my day !!! Exactly what I needed.
You will be prompted for the 'proxy_user' password. You can hardcode the
password with a small change in mailutil.c. Let me know if you need help
with this...
I'd like help, if possible. The code is pretty dense, so if you know
where I have to plug the password, that would save me ton of time.
Try:
--- mailutil.c.orig Fri Sep 26 11:39:12 2003
+++ mailutil.c Fri Sep 26 11:39:38 2003
@@ -598,7 +598,7 @@
username[NETMAXUSER-1] = '\0';
if (s = strchr (username,'\n')) *s = '\0';
}
- strcpy (password,getpass (password: ));
+ strcpy (password, your_pass);
}
--
Igor
--
Etienne GoyerLinux Québec Technologies Inc.
http://www.LinuxQuebec.com [EMAIL PROTECTED]
Re: Preserving seen state when converting from mbox
Take a look at the mailutil program in the UW-IMAP distro. Etienne Goyer wrote: Hi list, I am currently migrating a mail server running qpopper to Cyrus imapd. Part of the job involve migrating the user mailbox in mbox format to Cyrus. A lot of these users use pine and other POP client that set X-UIDL or Status header to set a kind of seen flag. Since the migration have to be transparent, it is required to preserve that flag. The user base will continue to use POP for a while with their current client (eventually migrating to IMAP). I make the assumption that setting the Seen flag on the revelant messages will make the Cyrus pop3d behave the same as qpopper wrt to these messages. Is this correct ? For the moment, I thought of three way I could migrate the mailboxes. My first plan was to use a script I dug from this mailing list called mboxtocyrus. It split the mbox file in messages with formail and inject them with deliver. However, the seen state is not preserved. Reading the deliver man page, I can't find a way to have it set the Seen flag. I wish I could have used that option since I feel it would have been the simplest and fastest (I have about 20 GB of mbox to migrate) way to do my migration. Is there a way I could preserve the Seen state with deliver ? My second plan is to follow the first one and work around the problem by writing a Sieve script that will look for the X-UIDL or Status header and set the Seen flag accordingly at delivery. If somebody already have such a Sieve script, I would be grateful if you could share it. I am a beginner with Sieve and not sure where to start. Also, pointer toward information about the X-UIDL and Status header will be welcome as I need to understand what these mean and their implication. Third and last resort plan is to run both Cyrus imapd and an imap daemon that read mbox file (most likely UW-imapd) and programatically copy the messages from the old mailstore to Cyrus. I wish to avoid that path as it is the one that require the most work from my part and I feel will probably be quite slow due to protocol overhead and so on. Any comments ? I know mbox to Cyrus migration is a FAQ on the list, but I never paid attention to these discussion and searching the archive prove quite tedious and donèt yeld clear answer so far. Thanks for your insight. -- Kenneth Murchison Oceana Matrix Ltd. Software Engineer 21 Princeton Place 716-662-8973 x26 Orchard Park, NY 14127 --PGP Public Key--http://www.oceana.com/~ken/ksm.pgp
Re: Preserving seen state when converting from mbox
On Thu, Sep 25, 2003 at 11:45:13AM -0400, Ken Murchison wrote: Take a look at the mailutil program in the UW-IMAP distro. I did. It does'nt do the job for me. The mbxcopy command want to create the mailbox it copy to, but since INBOX obviously already exist, it receive a Permission denied error. Also, it ask for crendential to open the connection, which is not something I wish to enter by hand considering I have ~7000 to migrate. The imapxfer command may do the job, but again it won't if I need to enter credential for each user (which I don't have access to anyway). I would also prefer to avoid the overhead of configuring/running a second imap daemon for the migration. imap-utils *almost* do the job, but using deliver instead would really rock if I could. -- Etienne GoyerLinux Québec Technologies Inc. http://www.LinuxQuebec.com [EMAIL PROTECTED]
Re: Preserving seen state when converting from mbox
On Thu, 25 Sep 2003, Etienne Goyer wrote: Third and last resort plan is to run both Cyrus imapd and an imap daemon that read mbox file (most likely UW-imapd) and programatically copy the messages from the old mailstore to Cyrus. I wish to avoid that path as it is the one that require the most work from my part and I feel will probably be quite slow due to protocol overhead and so on. This is typically the recommended path of transition actually. UW ships with a mailutil tool that allows you to move mailboxes between severs pretty easily. (Provided, of course, that UW gets the IMAP seen state right given your current files). Note that Cyrus probably uses different format UIDs than qpopper does, which is a problem that can likely only be solved by modifying cyrus. -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: Preserving seen state when converting from mbox
On Thu, 25 Sep 2003, Etienne Goyer wrote:
On Thu, Sep 25, 2003 at 11:45:13AM -0400, Ken Murchison wrote:
Take a look at the mailutil program in the UW-IMAP distro.
I did. It does'nt do the job for me. The mbxcopy command want to
create the mailbox it copy to, but since INBOX obviously already exist,
it receive a Permission denied error. Also, it ask for crendential to
open the connection, which is not something I wish to enter by hand
considering I have ~7000 to migrate.
You really need to use 'mailutil'. UW support proxy authentication and
with a very simple patch you can hard code proxy user password for the
conversion purpose. The command to convert an mbox may look like:
mailutil append /path/to/mbox
{new.cyrus.server/user=mbox_cyrus_user/authuser=proxy_user/norsh}INBOX
Make sure that cyrus allows plain mech, you can run this over ssl/tls if
security is important to you.
The imapxfer command may do the job, but again it won't if I need to
enter credential for each user (which I don't have access to anyway). I
would also prefer to avoid the overhead of configuring/running a second
imap daemon for the migration.
Why? You can run cyrus while the conversion is running.
imap-utils *almost* do the job, but using deliver instead would really
rock if I could.
--
Igor
Re: Preserving seen state when converting from mbox
On Thu, Sep 25, 2003 at 12:42:42PM -0400, Rob Siemborski wrote: Note that Cyrus probably uses different format UIDs than qpopper does, which is a problem that can likely only be solved by modifying cyrus. I don't quite understand what you mean here. Could you explain,or givepointer to ressource that explain ? Are you talking about the X-UID header ? -- Etienne GoyerLinux Québec Technologies Inc. http://www.LinuxQuebec.com [EMAIL PROTECTED]
Re: Preserving seen state when converting from mbox
On Thu, Sep 25, 2003 at 01:22:17PM -0400, Bennett Crowell wrote: When we had to do a similar migration, I changed the mailutil sourcecode to get the username and password from environment variables instead of prompting for them. At migration time, we made sure we had a backup of the password database, then changed all passwords to one that we used for the migration. After all the mailboxes were copied, the saved passwords were restored. Only the mm_login function needs to be changed in mailutil.c for this technique to work. If you are interested, let me know, and I will send you the changes needed. I'll take it. Maybe you could post it on the list, in case it interest somebody else ? This way, it will also get into the archive. Use mailutil append for mailboxes that already exist, and mailutil copy for others. Thank you so much for pointing this out ! -- Etienne GoyerLinux Québec Technologies Inc. http://www.LinuxQuebec.com [EMAIL PROTECTED]
Re: Preserving seen state when converting from mbox
On Thu, 25 Sep 2003, Etienne Goyer wrote: On Thu, Sep 25, 2003 at 12:42:42PM -0400, Rob Siemborski wrote: Note that Cyrus probably uses different format UIDs than qpopper does, which is a problem that can likely only be solved by modifying cyrus. I don't quite understand what you mean here. Could you explain,or givepointer to ressource that explain ? Are you talking about the X-UID header ? Read section 7 of: http://asg.web.cmu.edu/rfc/rfc1939.html The part that would concern me is: The unique-id of a message is an arbitrary server-determined string Cyrus uses IMAPUIDVALIDITY.IMAPUID for this string which is very likely to not be what the qpopper implementation uses (IMAPUIDVALIDITY and IMAPUID come from what is used in the IMAP protocol). -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: Preserving seen state when converting from mbox
On Thu, Sep 25, 2003 at 01:29:30PM -0400, Igor Brezac wrote:
mailutil append /path/to/mbox
{new.cyrus.server/user=mbox_cyrus_user/authuser=proxy_user/norsh}INBOX
You made my day !!! Exactly what I needed.
The imapxfer command may do the job, but again it won't if I need to
enter credential for each user (which I don't have access to anyway). I
would also prefer to avoid the overhead of configuring/running a second
imap daemon for the migration.
Why? You can run cyrus while the conversion is running.
I am going to make timing test with a few of the largest mbox to get an
estimate of how long it will be running. I will have a window of a few
hours one night on weekend to do it (can yuo spell espresso !). Mail
will be spooled at the MTA in the meantime, then Cyrus will be brought
up to replace qpopper. Timing is important as the service will be down
while the conversion is running.
Thanks again !
--
Etienne GoyerLinux Québec Technologies Inc.
http://www.LinuxQuebec.com [EMAIL PROTECTED]
Re: Preserving seen state when converting from mbox
On Thu, 25 Sep 2003, Etienne Goyer wrote:
On Thu, Sep 25, 2003 at 01:29:30PM -0400, Igor Brezac wrote:
mailutil append /path/to/mbox
{new.cyrus.server/user=mbox_cyrus_user/authuser=proxy_user/norsh}INBOX
You made my day !!! Exactly what I needed.
You will be prompted for the 'proxy_user' password. You can hardcode the
password with a small change in mailutil.c. Let me know if you need help
with this...
The imapxfer command may do the job, but again it won't if I need to
enter credential for each user (which I don't have access to anyway). I
would also prefer to avoid the overhead of configuring/running a second
imap daemon for the migration.
Why? You can run cyrus while the conversion is running.
I am going to make timing test with a few of the largest mbox to get an
estimate of how long it will be running. I will have a window of a few
hours one night on weekend to do it (can yuo spell espresso !). Mail
I do not think you have enough time to transfer several gigs (if not more)
of email in just a few hours over IMAP protocol. You can speed things up
if you run multiple mailutil's at the same time, you can also disable
duplicate suppresion while you run the conversion. Otherwise, I would
allow imap/pop/lmtp connections and run the conversion. Some folks will
not see their old email until the conversion completes.
will be spooled at the MTA in the meantime, then Cyrus will be brought
up to replace qpopper. Timing is important as the service will be down
while the conversion is running.
--
Igor
Re: Preserving seen state when converting from mbox
On Thu, Sep 25, 2003 at 01:29:30PM -0400, Igor Brezac wrote:
mailutil append /path/to/mbox
{new.cyrus.server/user=mbox_cyrus_user/authuser=proxy_user/norsh}INBOX
Ok, when I try :
[EMAIL PROTECTED] root]# mailutil copy /var/spool/mail/user1 \
{localhost/user=user1/authuser=cyrus/norsh/notls}user/user1
I get :
Can't do /authuser with this server
I have 'sasl_mech_list: PLAIN DIGEST-MD5'. When doing CAPABILITY, I
have AUTH=DIGEST-MD5, but not AUTH=PLAIN. I think that Cyrus imapd does
not advertise PLAIN, even if he accept it. If I sniff the connection
, I can see that mailutil hang up right after asking for CAPABILITY.
I guess it does not do DIGEST-MD5 and give up since AUTH=PLAIN is not
advertised. Does that make sense ? If yes, is there a way to make
Cyrus advertise AUTH=PLAIN ?
--
Etienne GoyerLinux Québec Technologies Inc.
http://www.LinuxQuebec.com [EMAIL PROTECTED]
Re: Preserving seen state when converting from mbox
--On Thursday, September 25, 2003 15:14:34 -0400 Etienne Goyer [EMAIL PROTECTED] wrote: I have 'sasl_mech_list: PLAIN DIGEST-MD5'. When doing CAPABILITY, I have AUTH=DIGEST-MD5, but not AUTH=PLAIN. I think that Cyrus imapd does not advertise PLAIN, even if he accept it. If I sniff the connection , I can see that mailutil hang up right after asking for CAPABILITY. I guess it does not do DIGEST-MD5 and give up since AUTH=PLAIN is not advertised. Does that make sense ? If yes, is there a way to make Cyrus advertise AUTH=PLAIN ? It may only be advertising PLAIN over an encrypted connection. What happens if you remove the /notls ? -Pat
Re: Preserving seen state when converting from mbox
On Thu, 25 Sep 2003, Etienne Goyer wrote:
On Thu, Sep 25, 2003 at 01:29:30PM -0400, Igor Brezac wrote:
mailutil append /path/to/mbox
{new.cyrus.server/user=mbox_cyrus_user/authuser=proxy_user/norsh}INBOX
Ok, when I try :
[EMAIL PROTECTED] root]# mailutil copy /var/spool/mail/user1 \
{localhost/user=user1/authuser=cyrus/norsh/notls}user/user1
I get :
Can't do /authuser with this server
I have 'sasl_mech_list: PLAIN DIGEST-MD5'. When doing CAPABILITY, I
have AUTH=DIGEST-MD5, but not AUTH=PLAIN. I think that Cyrus imapd does
not advertise PLAIN, even if he accept it. If I sniff the connection
, I can see that mailutil hang up right after asking for CAPABILITY.
I guess it does not do DIGEST-MD5 and give up since AUTH=PLAIN is not
advertised. Does that make sense ? If yes, is there a way to make
Cyrus advertise AUTH=PLAIN ?
PLAIN will not be advertised over non-secured transport. Try running over
tls session.
--
Igor
Re: Preserving seen state when converting from mbox
On Thu, Sep 25, 2003 at 12:35:21PM -0700, Pat Lashley wrote: --On Thursday, September 25, 2003 15:14:34 -0400 Etienne Goyer [EMAIL PROTECTED] wrote: I have 'sasl_mech_list: PLAIN DIGEST-MD5'. When doing CAPABILITY, I have AUTH=DIGEST-MD5, but not AUTH=PLAIN. I think that Cyrus imapd does not advertise PLAIN, even if he accept it. If I sniff the connection , I can see that mailutil hang up right after asking for CAPABILITY. I guess it does not do DIGEST-MD5 and give up since AUTH=PLAIN is not advertised. Does that make sense ? If yes, is there a way to make Cyrus advertise AUTH=PLAIN ? It may only be advertising PLAIN over an encrypted connection. What happens if you remove the /notls ? I receive Certificate failure I can fix that, but is there a way make mailutil accept self-signed certificate ? -- Etienne GoyerLinux Québec Technologies Inc. http://www.LinuxQuebec.com [EMAIL PROTECTED]
Re: Preserving seen state when converting from mbox
On Thu, 25 Sep 2003, Etienne Goyer wrote: On Thu, Sep 25, 2003 at 12:35:21PM -0700, Pat Lashley wrote: --On Thursday, September 25, 2003 15:14:34 -0400 Etienne Goyer [EMAIL PROTECTED] wrote: I have 'sasl_mech_list: PLAIN DIGEST-MD5'. When doing CAPABILITY, I have AUTH=DIGEST-MD5, but not AUTH=PLAIN. I think that Cyrus imapd does not advertise PLAIN, even if he accept it. If I sniff the connection , I can see that mailutil hang up right after asking for CAPABILITY. I guess it does not do DIGEST-MD5 and give up since AUTH=PLAIN is not advertised. Does that make sense ? If yes, is there a way to make Cyrus advertise AUTH=PLAIN ? It may only be advertising PLAIN over an encrypted connection. What happens if you remove the /notls ? I receive Certificate failure I can fix that, but is there a way make mailutil accept self-signed certificate ? Add /novalidate-cert. -- Igor
Re: Preserving seen state when converting from mbox
On Thu, 25 Sep 2003 15:04:48 -0400 (EDT) Igor Brezac [EMAIL PROTECTED] wrote: I do not think you have enough time to transfer several gigs (if not more) of email in just a few hours over IMAP protocol. Just for the reference, it took us about two weeks to transfer about 300gb of mailspool from one live system to another (with perdition in front), on a reasonably fast hw, with custom php scripts doing imap2imap copy (and related user imgration) in parallel. -- Jure Pecar
