Relation of filesystem to Cyrus mailbox structure
Heya folks, Is there any documentation on how Cyrus mailboxes are structured, as relative to the file system? Man. That's a poorly worded question, but I can't seem to find the best way to say it... I'm used to dealing with mbox mailboxes until very recently. The one thing that mbox format has going for it is that it's really conrete, for lack of a better word. The file in /var/mail/username is the user's inbox; No internal database or whatever. Other folders work similarly. Where I've been stuck recently is trying to figure out things like how to manipulate mailboxes by using the file system. Maybe this isn't really practical using mdir. Here's the latest: I have a user that we migrated from mbox. She has her inbox folder, which migrated fine. Then she has a couple of other folder, which migrated fine. Then she has folders within folders, and those refuse to show up, and it won't let her subscribe to them. Here's how it they're in the filesystem: /var/mail/j/user/janedoe -- Inbox: Migrated fine /var/mail/j/user/janedoe/folder1 -- Also migrated fine /var/mail/j/user/janedoe/folder1/folderA -- I can't get to show I don't think the user really NEEDS the folder called folder1 above, but wants the folders inside of it. So I tried to move folderA back one level so that it was sitting inside the /var/mail/j/user/janedoe folder, but that doesn't work. I figured I had to run a reconstruct command, but no avail there either. I guess the fundamental question here are these: - If I make changes to the file system, how do I get these reflected in Cyrus mailbox views? - Is this even a good idea, or something to be avoided, and done another way? - Is there any documentation on how the Cyrus internal database on mailboxes, quotas, and subscriptions work? Thanks, Rick Kunkel Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Relation of filesystem to Cyrus mailbox structure
--On Friday, September 14, 2007 8:27 -0700 Rick Kunkel [EMAIL PROTECTED] wrote: Where I've been stuck recently is trying to figure out things like how to manipulate mailboxes by using the file system. Maybe this isn't really practical using mdir. Here's the latest: I have a user that we migrated from mbox. She has her inbox folder, which migrated fine. Then she has a couple of other folder, which migrated fine. Then she has folders within folders, and those refuse to show up, and it won't let her subscribe to them. Here's how it they're in the filesystem: /var/mail/j/user/janedoe -- Inbox: Migrated fine /var/mail/j/user/janedoe/folder1 -- Also migrated fine /var/mail/j/user/janedoe/folder1/folderA -- I can't get to show I don't think the user really NEEDS the folder called folder1 above, but wants the folders inside of it. So I tried to move folderA back one level so that it was sitting inside the /var/mail/j/user/janedoe folder, but that doesn't work. I figured I had to run a reconstruct command, but no avail there either. I guess the fundamental question here are these: - If I make changes to the file system, how do I get these reflected in Cyrus mailbox views? It's less painful if you can use Cyrus to do this stuff. I know :-) The main reason you might have to do this would be restoring entire mailboxes off backups. Given /var/mail/j/user/janedoe/folder1/folderA , Cyrus might have three mailboxes named user.janedoe, user.janedoe.folder, and user.janedoe.folder.folderA, but not necessarily. cyradm will show you what folders exist, in Cyrus's view, and it lets you create, delete, and rename mailboxes to make the Cyrus view be what you want it to be. Cyrus will move the files around in the filesystem accordingly. To do batch work a perl script can use Cyrus::IMAP::Admin to do what cyradm would do. When we migrated from mbox, we created Cyrus mailboxes for the directories too. For ~janedoe/mail/foo/bar (mbox file) we created both user.janedoe.foo and user.janedoe.foo.bar, even though the user had never stored mail in foo (since it was a directory). This seemed simpler than having to explain later that foo could be part of a mailbox name but not exist as a mailbox! Joseph Brennan Lead Email Systems Engineer Columbia University Information Technology Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Relation of filesystem to Cyrus mailbox structure
Rick Kunkel wrote, at 09/14/2007 11:27 AM: Where I've been stuck recently is trying to figure out things like how to manipulate mailboxes by using the file system. Don't do that. That's the black box part of Cyrus IMAP. Forget about all the cool things you could do by directly manipulating mbox files. Here's the latest: I have a user that we migrated from mbox. She has her inbox folder, which migrated fine. Then she has a couple of other folder, which migrated fine. Then she has folders within folders, and those refuse to show up, and it won't let her subscribe to them. Here's how it they're in the filesystem: /var/mail/j/user/janedoe -- Inbox: Migrated fine /var/mail/j/user/janedoe/folder1 -- Also migrated fine /var/mail/j/user/janedoe/folder1/folderA -- I can't get to show I don't think the user really NEEDS the folder called folder1 above, but wants the folders inside of it. So I tried to move folderA back one level so that it was sitting inside the /var/mail/j/user/janedoe folder, but that doesn't work. I figured I had to run a reconstruct command, but no avail there either. This is one area migration tools failed me, as well. None of them were able to automatically create the mailbox that corresponds to the *directory* that held mbox files, even via IMAP. They were able to preserve the structure, however. If Cyrus IMAP complains: localhost lam user.janedoe.folder1 Mailbox does not exist Then you must create the mailbox: localhost cm user.janedoe.folder1 Don't worry, it won't delete or overwrite anything on the filesystem, it's just registering the existence of the mailbox in the blackbox system. Check again, then repeat the process for any other mailboxes that still don't exist. I never did find a way to completely automate this for an entire mailstore. I was quite surprised to discover that most users don't even bother to move mail out the INBOX, let alone organize them in multiple levels of subdirectories, so I managed to do most of it manually. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Relation of filesystem to Cyrus mailbox structure
Jorey Bump [EMAIL PROTECTED] wrote: This is one area migration tools failed me, as well. None of them were able to automatically create the mailbox that corresponds to the *directory* that held mbox files, even via IMAP. They were able to preserve the structure, however. But analyzing the old file structure is not difficult to script. find gives a nice list. if (-z $file || -d $file), just create a Cyrus mailbox but skip the message moving tool. Detail will vary depending how the old mail was structured. We had to grab /var/spool/mail/user, ~user/mbox, and ~user/mail/* which might be common in U Wash for example. It might have been cool to test the first five characters in ~user/* for From to catch stray files not in the mail directory. You're right, it's amazing how many users have only inbox and sent and trash folders (of varying names). Joseph Brennan Lead Email Systems Engineer Columbia University Information Technology Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Relation of filesystem to Cyrus mailbox structure
look at cyrus' reconstruct. From the man page: ... Reconstruct rebuilds one or more IMAP mailboxes. When invoked with the -m switch, it rebuilds the master mailboxes file. It can be used to recover from almost any sort of data corruption. So you should be able to run (as cyrus) reconstruct -m -r user.janedoe For FYI, if you do a a dump of the mailbox file you *could* manually add the mailboxes/folders and reload the mailbox file etc., but you'd still have to run reconstruct... HTH, B Rick Kunkel wrote: Heya folks, Is there any documentation on how Cyrus mailboxes are structured, as relative to the file system? Man. That's a poorly worded question, but I can't seem to find the best way to say it... I'm used to dealing with mbox mailboxes until very recently. The one thing that mbox format has going for it is that it's really conrete, for lack of a better word. The file in /var/mail/username is the user's inbox; No internal database or whatever. Other folders work similarly. Where I've been stuck recently is trying to figure out things like how to manipulate mailboxes by using the file system. Maybe this isn't really practical using mdir. Here's the latest: I have a user that we migrated from mbox. She has her inbox folder, which migrated fine. Then she has a couple of other folder, which migrated fine. Then she has folders within folders, and those refuse to show up, and it won't let her subscribe to them. Here's how it they're in the filesystem: /var/mail/j/user/janedoe -- Inbox: Migrated fine /var/mail/j/user/janedoe/folder1 -- Also migrated fine /var/mail/j/user/janedoe/folder1/folderA -- I can't get to show I don't think the user really NEEDS the folder called folder1 above, but wants the folders inside of it. So I tried to move folderA back one level so that it was sitting inside the /var/mail/j/user/janedoe folder, but that doesn't work. I figured I had to run a reconstruct command, but no avail there either. I guess the fundamental question here are these: - If I make changes to the file system, how do I get these reflected in Cyrus mailbox views? - Is this even a good idea, or something to be avoided, and done another way? - Is there any documentation on how the Cyrus internal database on mailboxes, quotas, and subscriptions work? Thanks, Rick Kunkel Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Relation of filesystem to Cyrus mailbox structure
I am currently pulling my hair out, trying to get this to work... Lemme show you a cyradm session, with some carriage returns thrown in for reading ease: Incidentally, we're running unixhierarchysep: yes. localhost ls -l /var/mail/c/user/calvizio -rw-r--r-- 1 cyrus mail 1911537 2007-09-07 02:43 1. -rw-r--r-- 1 cyrus mail 3717492 2007-09-07 02:43 2. -rw-r--r-- 1 cyrus mail6364 2007-09-07 02:43 3. -rw--- 1 cyrus mail 32252 2007-09-14 11:52 cyrus.cache -rw--- 1 cyrus mail 183 2007-09-14 12:32 cyrus.header -rw--- 1 cyrus mail1636 2007-09-14 11:52 cyrus.index localhost lam user/calvizio* user/calvizio: cyrus lrswipcda calvizio lrswipcda user/calvizio/INBOX: cyrus lrswipcda calvizio lrswipcda localhost cm user/calvizio/INBOX createmailbox: Mailbox already exists localhost sam user/calvizio cyrus all localhost sam user/calvizio/INBOX cyrus all setaclmailbox: cyrus: lrswipcda: System I/O error localhost reconstruct user/calvizio localhost reconstruct user/calvizio/INBOX reconstruct: System I/O error localhost cm user/calvizio/INBOX createmailbox: Mailbox already exists localhost dm user/calvizio/INBOX deletemailbox: System I/O error AGH I tried actually creating the dirs in the file system to match, and re-owning them to cyrus:mail, to no avail. Furthermore, ONE of the time I tried to create the mailbox, I typed cm INBOX instead of cm user/calvizio/INBOX by mistake, and now I have this wacko little mailbox called INBOX at the top of everything: localhost lm INBOX (\HasNoChildren) I can't seem to delete this little monster either Any ideas? Thanks, Rick Kunkel On Fri, 14 Sep 2007, Joseph Brennan wrote: --On Friday, September 14, 2007 8:27 -0700 Rick Kunkel [EMAIL PROTECTED] wrote: Where I've been stuck recently is trying to figure out things like how to manipulate mailboxes by using the file system. Maybe this isn't really practical using mdir. Here's the latest: I have a user that we migrated from mbox. She has her inbox folder, which migrated fine. Then she has a couple of other folder, which migrated fine. Then she has folders within folders, and those refuse to show up, and it won't let her subscribe to them. Here's how it they're in the filesystem: /var/mail/j/user/janedoe -- Inbox: Migrated fine /var/mail/j/user/janedoe/folder1 -- Also migrated fine /var/mail/j/user/janedoe/folder1/folderA -- I can't get to show I don't think the user really NEEDS the folder called folder1 above, but wants the folders inside of it. So I tried to move folderA back one level so that it was sitting inside the /var/mail/j/user/janedoe folder, but that doesn't work. I figured I had to run a reconstruct command, but no avail there either. I guess the fundamental question here are these: - If I make changes to the file system, how do I get these reflected in Cyrus mailbox views? It's less painful if you can use Cyrus to do this stuff. I know :-) The main reason you might have to do this would be restoring entire mailboxes off backups. Given /var/mail/j/user/janedoe/folder1/folderA , Cyrus might have three mailboxes named user.janedoe, user.janedoe.folder, and user.janedoe.folder.folderA, but not necessarily. cyradm will show you what folders exist, in Cyrus's view, and it lets you create, delete, and rename mailboxes to make the Cyrus view be what you want it to be. Cyrus will move the files around in the filesystem accordingly. To do batch work a perl script can use Cyrus::IMAP::Admin to do what cyradm would do. When we migrated from mbox, we created Cyrus mailboxes for the directories too. For ~janedoe/mail/foo/bar (mbox file) we created both user.janedoe.foo and user.janedoe.foo.bar, even though the user had never stored mail in foo (since it was a directory). This seemed simpler than having to explain later that foo could be part of a mailbox name but not exist as a mailbox! Joseph Brennan Lead Email Systems Engineer Columbia University Information Technology Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Relation of filesystem to Cyrus mailbox structure
On Fri, 14 Sep 2007, Rick Kunkel wrote: I am currently pulling my hair out, trying to get this to work... Lemme show you a cyradm session, with some carriage returns thrown in for reading ease: Incidentally, we're running unixhierarchysep: yes. localhost ls -l /var/mail/c/user/calvizio -rw-r--r-- 1 cyrus mail 1911537 2007-09-07 02:43 1. -rw-r--r-- 1 cyrus mail 3717492 2007-09-07 02:43 2. -rw-r--r-- 1 cyrus mail6364 2007-09-07 02:43 3. -rw--- 1 cyrus mail 32252 2007-09-14 11:52 cyrus.cache -rw--- 1 cyrus mail 183 2007-09-14 12:32 cyrus.header -rw--- 1 cyrus mail1636 2007-09-14 11:52 cyrus.index localhost lam user/calvizio* user/calvizio: cyrus lrswipcda calvizio lrswipcda user/calvizio/INBOX: cyrus lrswipcda calvizio lrswipcda localhost cm user/calvizio/INBOX createmailbox: Mailbox already exists localhost sam user/calvizio cyrus all localhost sam user/calvizio/INBOX cyrus all setaclmailbox: cyrus: lrswipcda: System I/O error localhost reconstruct user/calvizio localhost reconstruct user/calvizio/INBOX reconstruct: System I/O error localhost cm user/calvizio/INBOX createmailbox: Mailbox already exists localhost dm user/calvizio/INBOX deletemailbox: System I/O error AGH I tried actually creating the dirs in the file system to match, and re-owning them to cyrus:mail, to no avail. user/calvizio *IS* user calvizio's INBOX. INBOX is a reserved word in IMAP, so you cannot create a folder by that name in my experience anyways. Furthermore, ONE of the time I tried to create the mailbox, I typed cm INBOX instead of cm user/calvizio/INBOX by mistake, and now I have this wacko little mailbox called INBOX at the top of everything: localhost lm INBOX (\HasNoChildren) I can't seem to delete this little monster either That might be tricky. I assume you've tried sam INBOX cyrus c; dm INBOX? Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
