Re: cyrus: delete previously deleted messages(after backup recovery)
Thank you for your help, I managed to write a cyrus.index parser that checks all the effective mail UIDs (=filenames) and erases all the unnecessary files (previously deleted and thus not referenced in the index file). Just in case someone faces the same issue someday, here is the script : (instead of directly deleting the emails, it creates a bash file with all the rm commands, so it can be reviewed before, just in case something goes wrong...) In my case, I used the script (cleanMailbox.sh) this way (to clean only the INBOXes) : for userdir in `find /data/mail/imap/spool/*/user/ -maxdepth 1 -mindepth 1 -name *`; do echo $userdir; ~/adminScripts/cleanMailbox.sh $userdir; done; and the script itself: * #!/bin/bash # cleanMailbox.sh : remove all mailfiles not referenced in the cyrus.index file if [ ! -d $1 ]; then echo USAGE: argument must be a valid cyrus user mail directory exit 0 fi INDEXPATH=$1/cyrus.index if [ ! -f $INDEXPATH ]; then echo No cyrus.index in $1. ABORT exit 0 fi #1- read start offset in cyrus.index at position 0x0c ==12 (4 bytes) STARTOFFSET=`od --skip-bytes=12 --read-bytes=4 -An -t x1 $INDEXPATH | tr -d ' '` STARTOFFSET_10=`printf '%d\n' 0x$STARTOFFSET` echo messages start offset =0x$STARTOFFSET (=$STARTOFFSET_10) #2- read record size in cyrus.index at position 0x10 ==16 (4 bytes) RECORDSIZE=`od --skip-bytes=16 --read-bytes=4 -An -t x1 $INDEXPATH | tr -d ' '` RECORDSIZE_10=`printf '%d\n' 0x$RECORDSIZE` echo message record size =0x$RECORDSIZE (=$RECORDSIZE_10) #3- read number of emails in cyrus.index at position 0x14 ==20 (4 bytes) NBMESSAGES=`od --skip-bytes=20 --read-bytes=4 -An -t x1 $INDEXPATH | tr -d ' '` NBMESSAGES_10=`printf '%d\n' 0x$NBMESSAGES` echo number of emails in metadata=0x$NBMESSAGES (=$NBMESSAGES_10) # ALL message UID (=filename) will be in cyrus.index at STARTOFFSET + i * RECORDSIZE # with i integer in [1..NBMESSAGES] # Build a reference file to keep track of mail filenames (UID.) that will NOT be deleted for i in `seq 0 $((NBMESSAGES_10 - 1))`; do MESSAGEOFFSET=$((STARTOFFSET_10 + i * RECORDSIZE_10)) UIDMESSAGE=`od --skip-bytes=$MESSAGEOFFSET --read-bytes=4 -An -t x1 $INDEXPATH | tr -d ' '` UIDMESSAGE_10=`printf '%d\n' 0x$UIDMESSAGE` echo $UIDMESSAGE_10. $1/KEEP_THIS.tmp done # instead of deleting directly the undesirable files, build a command file with all the rm commands: echo cd $1 /tmp/eraseOldMails.sh cd $1 # for all email files for mailfile in `ls *\.`; do # if mailfile not found in KEEP_THIS list, delete it: grep ^$mailfile$ $1/KEEP_THIS.tmp /dev/null || echo rm $mailfile /tmp/eraseOldMails.sh done chmod +x /tmp/eraseOldMails.sh rm $1/KEEP_THIS.tmp * Yannick Barbeaux 2014-04-23 17:06 GMT+02:00 Michael Menge michael.me...@zdv.uni-tuebingen.de : Hi Quoting Y B ybp...@gmail.com: OK, I managed to reconstruct my test mailbox. The files that were in my directory structure but not visible in the mailbox are now all visible in my email client. Unfortunately, the cyrreconstruct command is not verbose at all and I cannot see the filenames corresponding to rediscovered emails (so I cannot delete them...). Any clue? ... My version of cyrus : 2.2.13 Cyrus 2.2.13 is very old. AFSIR the -n feature was added in 2.4.x. Updating in this situation might do harm, as the metadata format changed in 2.4.x I think there are no reconstruct options or other tools, to remove the deleted mails out of the box, but the information which mails where in the mailbox are stored in the cyrus.index file. See http://git.cyrusimap.org/cyrus-imapd/tree/doc/internal/ mailbox-format.html?h=cyrus-imapd-2.2 So you may be able to write a tool to parse the file. Regards Michael Menge M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni- tuebingen.de Wächterstraße 76 72074 Tübingen Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyrus: delete previously deleted messages(after backup recovery)
thanks for the answer. I certainly still have the metadata since I have not lost any file, I have just too many files (all the previously deleted emails, recovered from my rsync backup) I am not exactly sure where the metadata are situated, /var/imap does not exist. In my case, all email files are situated at /data/mail/imap/spool/*/user/username (the star being any letter of the alphabet). For a given username, I have for example the following files: -rw--- 1 cyrus mail21183 Apr 23 11:01 4846. -rw--- 1 cyrus mail26585 Apr 23 10:49 4845. -rw--- 1 cyrus mail15720 Apr 23 10:39 4844. -rw--- 1 cyrus mail 4496548 Apr 23 13:36 cyrus.cache -rw--- 1 cyrus mail 251 Mar 10 16:52 cyrus.header -rw--- 1 cyrus mail 151816 Apr 23 13:36 cyrus.index -rw--- 1 cyrus mail 10087920 Jan 22 02:28 cyrus.squat drwx-- 2 cyrus mail69632 Apr 23 13:11 Sent drwx-- 2 cyrus mail 4096 Apr 23 09:38 Drafts drwx-- 4 cyrus mail69632 Apr 22 20:45 Trash drwx-- 2 cyrus mail 4096 Apr 22 20:30 spam Are the metadata in the cyrus.* files? If yes, is there an easy way to delete the old emails using these metadata? Thank you. Yannick 2014-04-22 15:06 GMT+02:00 Patrick Boutilier bouti...@ednet.ns.ca: On 04/22/2014 09:19 AM, Y B wrote: Hello, after a disk crash, I had to recover all the emails from a daily backup. Unfortunately, I noticed that this backup was made with the rsync command but WITHOUT the --delete options. That means that the mail backup was much larger than the original one since no message was ever deleted on the backup. Now I would like to clean the mailboxes and remove all those emails that were previously deleted but I cannot find how. Is there any easy way to do this? (I use cyrus+postfix on debian 6.0.4) Thank you for your help. Did you lose your metadata as well? Usually in /var/imap ? Yannick Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyrus: delete previously deleted messages(after backup recovery)
On 04/23/2014 08:45 AM, Y B wrote: thanks for the answer. I certainly still have the metadata since I have not lost any file, I have just too many files (all the previously deleted emails, recovered from my rsync backup) I am not exactly sure where the metadata are situated, /var/imap does not exist. In my case, all email files are situated at /data/mail/imap/spool/*/user/username (the star being any letter of the alphabet). For a given username, I have for example the following files: -rw--- 1 cyrus mail21183 Apr 23 11:01 4846. -rw--- 1 cyrus mail26585 Apr 23 10:49 4845. -rw--- 1 cyrus mail15720 Apr 23 10:39 4844. -rw--- 1 cyrus mail 4496548 Apr 23 13:36 cyrus.cache -rw--- 1 cyrus mail 251 Mar 10 16:52 cyrus.header -rw--- 1 cyrus mail 151816 Apr 23 13:36 cyrus.index -rw--- 1 cyrus mail 10087920 Jan 22 02:28 cyrus.squat drwx-- 2 cyrus mail69632 Apr 23 13:11 Sent drwx-- 2 cyrus mail 4096 Apr 23 09:38 Drafts drwx-- 4 cyrus mail69632 Apr 22 20:45 Trash drwx-- 2 cyrus mail 4096 Apr 22 20:30 spam Are the metadata in the cyrus.* files? If yes, is there an easy way to delete the old emails using these metadata? Thank you. Yannick Yes, metadata files are the cyrus.* files. What version of Cyrus are you running? You should be able to run reconstruct on the mailboxes with the -n option which should list the files that reconstruct would have added back to the index if the -n option was not used. Use this output to determine which files can be deleted. For example I added some bogus files called 1. , 101. , and 255. to my mailbox. reconstruct -n user.boutilpj user.boutilpj uid 1 rediscovered - appending user.boutilpj uid 101 rediscovered - appending user.boutilpj uid 255 rediscovered - appending user.boutilpj 2014-04-22 15:06 GMT+02:00 Patrick Boutilier bouti...@ednet.ns.ca mailto:bouti...@ednet.ns.ca: On 04/22/2014 09:19 AM, Y B wrote: Hello, after a disk crash, I had to recover all the emails from a daily backup. Unfortunately, I noticed that this backup was made with the rsync command but WITHOUT the --delete options. That means that the mail backup was much larger than the original one since no message was ever deleted on the backup. Now I would like to clean the mailboxes and remove all those emails that were previously deleted but I cannot find how. Is there any easy way to do this? (I use cyrus+postfix on debian 6.0.4) Thank you for your help. Did you lose your metadata as well? Usually in /var/imap ? Yannick Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/__pipermail/info-cyrus/ http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/__mailman/listinfo/info-cyrus https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus attachment: boutilpj.vcf Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyrus: delete previously deleted messages(after backup recovery)
this seems to be a good way to reach my goal. Unfortunately, my version of cyrreconstruct does not accept the -n option : cyrreconstruct [ -C config-file ] [ -p partition ] [ -x ] [ -r ] [ -f ] mailbox... cyrreconstruct [ -C config-file ] -m Using cyradm and reconstruct does not allow the -n either, only the recursive flag is allowed (-r). I tried to reconstruct the mailbox anyway but nothing seems to happen (i ran the command on a mailbox with one email in INBOX and ten email files but after the execution of the command, nothing changes). Related lines in syslog: Apr 23 15:06:37 srv1 cyrus/imap[30671]: Reconstructing 'user.davicere' (not recursive) for user 'cyrus' Apr 23 15:06:37 srv1 cyrus/imap[30672]: Regenerating quota roots starting with 'user.davicere' for user 'cyrus' My version of cyrus : 2.2.13 cyrus@srv1:~$ cyradm localhost localhost ver name : Cyrus IMAPD version: v2.2.13-Debian-2.2.13-19+squeeze3 2006/12/19 19:32:59 vendor : Project Cyrus support-url: http://asg.web.cmu.edu/cyrus os : Linux os-version : 2.6.26-1-xen-686 With which version of cyrus can you use the -n option? With cyradm or cyrreconstruct? Any solution in my case? Thanks Yannick 2014-04-23 14:00 GMT+02:00 Patrick Boutilier bouti...@ednet.ns.ca: On 04/23/2014 08:45 AM, Y B wrote: thanks for the answer. I certainly still have the metadata since I have not lost any file, I have just too many files (all the previously deleted emails, recovered from my rsync backup) I am not exactly sure where the metadata are situated, /var/imap does not exist. In my case, all email files are situated at /data/mail/imap/spool/*/user/username (the star being any letter of the alphabet). For a given username, I have for example the following files: -rw--- 1 cyrus mail21183 Apr 23 11:01 4846. -rw--- 1 cyrus mail26585 Apr 23 10:49 4845. -rw--- 1 cyrus mail15720 Apr 23 10:39 4844. -rw--- 1 cyrus mail 4496548 Apr 23 13:36 cyrus.cache -rw--- 1 cyrus mail 251 Mar 10 16:52 cyrus.header -rw--- 1 cyrus mail 151816 Apr 23 13:36 cyrus.index -rw--- 1 cyrus mail 10087920 Jan 22 02:28 cyrus.squat drwx-- 2 cyrus mail69632 Apr 23 13:11 Sent drwx-- 2 cyrus mail 4096 Apr 23 09:38 Drafts drwx-- 4 cyrus mail69632 Apr 22 20:45 Trash drwx-- 2 cyrus mail 4096 Apr 22 20:30 spam Are the metadata in the cyrus.* files? If yes, is there an easy way to delete the old emails using these metadata? Thank you. Yannick Yes, metadata files are the cyrus.* files. What version of Cyrus are you running? You should be able to run reconstruct on the mailboxes with the -n option which should list the files that reconstruct would have added back to the index if the -n option was not used. Use this output to determine which files can be deleted. For example I added some bogus files called 1. , 101. , and 255. to my mailbox. reconstruct -n user.boutilpj user.boutilpj uid 1 rediscovered - appending user.boutilpj uid 101 rediscovered - appending user.boutilpj uid 255 rediscovered - appending user.boutilpj 2014-04-22 15:06 GMT+02:00 Patrick Boutilier bouti...@ednet.ns.ca mailto:bouti...@ednet.ns.ca: On 04/22/2014 09:19 AM, Y B wrote: Hello, after a disk crash, I had to recover all the emails from a daily backup. Unfortunately, I noticed that this backup was made with the rsync command but WITHOUT the --delete options. That means that the mail backup was much larger than the original one since no message was ever deleted on the backup. Now I would like to clean the mailboxes and remove all those emails that were previously deleted but I cannot find how. Is there any easy way to do this? (I use cyrus+postfix on debian 6.0.4) Thank you for your help. Did you lose your metadata as well? Usually in /var/imap ? Yannick Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/__pipermail/info-cyrus/ http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/__mailman/listinfo/info-cyrus https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe:
Re: cyrus: delete previously deleted messages(after backup recovery)
OK, I managed to reconstruct my test mailbox. The files that were in my directory structure but not visible in the mailbox are now all visible in my email client. Unfortunately, the cyrreconstruct command is not verbose at all and I cannot see the filenames corresponding to rediscovered emails (so I cannot delete them...). Any clue? 2014-04-23 15:22 GMT+02:00 Y B ybp...@gmail.com: this seems to be a good way to reach my goal. Unfortunately, my version of cyrreconstruct does not accept the -n option : cyrreconstruct [ -C config-file ] [ -p partition ] [ -x ] [ -r ] [ -f ] mailbox... cyrreconstruct [ -C config-file ] -m Using cyradm and reconstruct does not allow the -n either, only the recursive flag is allowed (-r). I tried to reconstruct the mailbox anyway but nothing seems to happen (i ran the command on a mailbox with one email in INBOX and ten email files but after the execution of the command, nothing changes). Related lines in syslog: Apr 23 15:06:37 srv1 cyrus/imap[30671]: Reconstructing 'user.davicere' (not recursive) for user 'cyrus' Apr 23 15:06:37 srv1 cyrus/imap[30672]: Regenerating quota roots starting with 'user.davicere' for user 'cyrus' My version of cyrus : 2.2.13 cyrus@srv1:~$ cyradm localhost localhost ver name : Cyrus IMAPD version: v2.2.13-Debian-2.2.13-19+squeeze3 2006/12/19 19:32:59 vendor : Project Cyrus support-url: http://asg.web.cmu.edu/cyrus os : Linux os-version : 2.6.26-1-xen-686 With which version of cyrus can you use the -n option? With cyradm or cyrreconstruct? Any solution in my case? Thanks Yannick 2014-04-23 14:00 GMT+02:00 Patrick Boutilier bouti...@ednet.ns.ca: On 04/23/2014 08:45 AM, Y B wrote: thanks for the answer. I certainly still have the metadata since I have not lost any file, I have just too many files (all the previously deleted emails, recovered from my rsync backup) I am not exactly sure where the metadata are situated, /var/imap does not exist. In my case, all email files are situated at /data/mail/imap/spool/*/user/username (the star being any letter of the alphabet). For a given username, I have for example the following files: -rw--- 1 cyrus mail21183 Apr 23 11:01 4846. -rw--- 1 cyrus mail26585 Apr 23 10:49 4845. -rw--- 1 cyrus mail15720 Apr 23 10:39 4844. -rw--- 1 cyrus mail 4496548 Apr 23 13:36 cyrus.cache -rw--- 1 cyrus mail 251 Mar 10 16:52 cyrus.header -rw--- 1 cyrus mail 151816 Apr 23 13:36 cyrus.index -rw--- 1 cyrus mail 10087920 Jan 22 02:28 cyrus.squat drwx-- 2 cyrus mail69632 Apr 23 13:11 Sent drwx-- 2 cyrus mail 4096 Apr 23 09:38 Drafts drwx-- 4 cyrus mail69632 Apr 22 20:45 Trash drwx-- 2 cyrus mail 4096 Apr 22 20:30 spam Are the metadata in the cyrus.* files? If yes, is there an easy way to delete the old emails using these metadata? Thank you. Yannick Yes, metadata files are the cyrus.* files. What version of Cyrus are you running? You should be able to run reconstruct on the mailboxes with the -n option which should list the files that reconstruct would have added back to the index if the -n option was not used. Use this output to determine which files can be deleted. For example I added some bogus files called 1. , 101. , and 255. to my mailbox. reconstruct -n user.boutilpj user.boutilpj uid 1 rediscovered - appending user.boutilpj uid 101 rediscovered - appending user.boutilpj uid 255 rediscovered - appending user.boutilpj 2014-04-22 15:06 GMT+02:00 Patrick Boutilier bouti...@ednet.ns.ca mailto:bouti...@ednet.ns.ca: On 04/22/2014 09:19 AM, Y B wrote: Hello, after a disk crash, I had to recover all the emails from a daily backup. Unfortunately, I noticed that this backup was made with the rsync command but WITHOUT the --delete options. That means that the mail backup was much larger than the original one since no message was ever deleted on the backup. Now I would like to clean the mailboxes and remove all those emails that were previously deleted but I cannot find how. Is there any easy way to do this? (I use cyrus+postfix on debian 6.0.4) Thank you for your help. Did you lose your metadata as well? Usually in /var/imap ? Yannick Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/__pipermail/info-cyrus/ http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/__mailman/listinfo/info-cyrus https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/ pipermail/info-cyrus/ To
Re: cyrus: delete previously deleted messages(after backup recovery)
Hi Quoting Y B ybp...@gmail.com: OK, I managed to reconstruct my test mailbox. The files that were in my directory structure but not visible in the mailbox are now all visible in my email client. Unfortunately, the cyrreconstruct command is not verbose at all and I cannot see the filenames corresponding to rediscovered emails (so I cannot delete them...). Any clue? ... My version of cyrus : 2.2.13 Cyrus 2.2.13 is very old. AFSIR the -n feature was added in 2.4.x. Updating in this situation might do harm, as the metadata format changed in 2.4.x I think there are no reconstruct options or other tools, to remove the deleted mails out of the box, but the information which mails where in the mailbox are stored in the cyrus.index file. See http://git.cyrusimap.org/cyrus-imapd/tree/doc/internal/mailbox-format.html?h=cyrus-imapd-2.2 So you may be able to write a tool to parse the file. Regards Michael Menge M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen smime.p7s Description: S/MIME Signatur Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
cyrus: delete previously deleted messages(after backup recovery)
Hello, after a disk crash, I had to recover all the emails from a daily backup. Unfortunately, I noticed that this backup was made with the rsync command but WITHOUT the --delete options. That means that the mail backup was much larger than the original one since no message was ever deleted on the backup. Now I would like to clean the mailboxes and remove all those emails that were previously deleted but I cannot find how. Is there any easy way to do this? (I use cyrus+postfix on debian 6.0.4) Thank you for your help. Yannick Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyrus: delete previously deleted messages(after backup recovery)
On 04/22/2014 09:19 AM, Y B wrote: Hello, after a disk crash, I had to recover all the emails from a daily backup. Unfortunately, I noticed that this backup was made with the rsync command but WITHOUT the --delete options. That means that the mail backup was much larger than the original one since no message was ever deleted on the backup. Now I would like to clean the mailboxes and remove all those emails that were previously deleted but I cannot find how. Is there any easy way to do this? (I use cyrus+postfix on debian 6.0.4) Thank you for your help. Did you lose your metadata as well? Usually in /var/imap ? Yannick Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus attachment: boutilpj.vcf Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
