Re: backup rsync
El 08/09/2014, a las 18:32, Patrick Goetz pgo...@mail.utexas.edu escribió: On 09/06/2014 06:52 AM, Egoitz Aurrekoetxea wrote: We are using Cyrus replication for some years now. It’s just fine. You can play too with expunge_delayed for avoid removing mail immediately. Hi - Do you mind explaining this in more detail? Thanks. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Hi all!, Using Cyrus replication is the way as Bron said of having a just fine replicated (and in consistent state) mailbox server. And yes as Bron said too :) replication rocks! :) We are too an ISP and as said it rocks!. The other part is about expunge… when Cyrus performs a Expunge on a message it removes from the Cyrus databases of the mailbox and the own mail file from the mailbox. You can alter this working mode, by just making the message disappearing from the mailbox by removing from the own mailbox database (not the mailboxes database which apart here has nothing to do with which we’re talking about…. I mean the own mailbox we’re dealing with from it’s own databases) and causing this way the message to disappear as if it would have been fully removed. Later you can in an accident event in that mailbox remove the cyrus.* databases do a reconstruct -rf and you will get back again all those deleted mails only if they have been removed before than expire days specified to cyr_expire who runs within the schedule defined in cyrus.conf. Note that doing this make all IMAP flags placed on the messages of the mailbox to disappear (if using IMAP). Hope it helps, Best regards, Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
On 09/06/2014 06:52 AM, Egoitz Aurrekoetxea wrote: We are using Cyrus replication for some years now. It’s just fine. You can play too with expunge_delayed for avoid removing mail immediately. Hi - Do you mind explaining this in more detail? Thanks. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
A long time ago for a much older version of Cyrus, we hacked lmtpd (I think, it’s been years) to log the messages as it wrote them down. Then we just processed the log every hour or so to backup only those files. That saved having to traverse the entire in ode tree most of the time. -- Stephen On Sep 6, 2014, at 7:32 PM, Bron Gondwana br...@fastmail.fm wrote: No, no - we do replication. Replication rocks. You could easily stop the replica and take a snapshot of that, but our real backup solution is much more evil. I've posted it to this list before, but it's basically a perl daemon which knows far too much about how Cyrus locks its data files. It actually reads and parses cyrus.index files to work out what it needs to do. Bron. On Sun, Sep 7, 2014, at 04:50 AM, Marcus Schopen wrote: Hi Bron, Am Samstag, den 06.09.2014, 22:17 +1000 schrieb Bron Gondwana: That's what we do :) Thanks for your feedbeek. What's your workaround for not stopping cyrus before taking a lvm snapshot and run rsnapshot? Ciao Marcus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
Did you back up Sent folders too? On Mon, Sep 8, 2014, at 11:48 AM, Stephen Ulmer wrote: A long time ago for a much older version of Cyrus, we hacked lmtpd (I think, it’s been years) to log the messages as it wrote them down. Then we just processed the log every hour or so to backup only those files. That saved having to traverse the entire in ode tree most of the time. -- Stephen On Sep 6, 2014, at 7:32 PM, Bron Gondwana [1]br...@fastmail.fm wrote: No, no - we do replication. Replication rocks. You could easily stop the replica and take a snapshot of that, but our real backup solution is much more evil. I've posted it to this list before, but it's basically a perl daemon which knows far too much about how Cyrus locks its data files. It actually reads and parses cyrus.index files to work out what it needs to do. Bron. On Sun, Sep 7, 2014, at 04:50 AM, Marcus Schopen wrote: Hi Bron, Am Samstag, den 06.09.2014, 22:17 +1000 schrieb Bron Gondwana: That's what we do :) Thanks for your feedbeek. What's your workaround for not stopping cyrus before taking a lvm snapshot and run rsnapshot? Ciao Marcus -- Bron Gondwana [2]br...@fastmail.fm Cyrus Home Page:[3]http://www.cyrusimap.org/ List Archives/Info:[4]http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: [5]https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm References 1. mailto:br...@fastmail.fm 2. mailto:br...@fastmail.fm 3. http://www.cyrusimap.org/ 4. http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 5. https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
I think at that point we may not have cared… We did still run TSM incrementals, basically back-to-back, but those didn’t finish in 24 hours. We needed a better-effort way to get most message for a disaster. I looked for the patch a bit ago (not that it would apply, but I would at least see what I had changed!) but couldn’t find it. I haven’t been employed by that organization for just over 8 years now, so it’s nowhere in my cache. The general point of my comment was: If you’re backing up files, looking for them is expensive. If you can get Cyrus to tell you what it changed, then you don’t have to ask the filesystem — which usually involves asking every file. The other option would be to use a filesystem that supports DMAPI, and write a DMAPI application that kept track of changed files and added them to a backup queue. It seems like having Cyrus log the delivery would have other benefits, though, and would work on any filesystem. I’m not currently a Cyrus admin, though, so it could already do that for all I know. :) Liberty, -- Stephen On Sep 7, 2014, at 10:03 PM, Bron Gondwana br...@fastmail.fm wrote: Did you back up Sent folders too? On Mon, Sep 8, 2014, at 11:48 AM, Stephen Ulmer wrote: A long time ago for a much older version of Cyrus, we hacked lmtpd (I think, it’s been years) to log the messages as it wrote them down. Then we just processed the log every hour or so to backup only those files. That saved having to traverse the entire in ode tree most of the time. -- Stephen On Sep 6, 2014, at 7:32 PM, Bron Gondwana br...@fastmail.fm wrote: No, no - we do replication. Replication rocks. You could easily stop the replica and take a snapshot of that, but our real backup solution is much more evil. I've posted it to this list before, but it's basically a perl daemon which knows far too much about how Cyrus locks its data files. It actually reads and parses cyrus.index files to work out what it needs to do. Bron. On Sun, Sep 7, 2014, at 04:50 AM, Marcus Schopen wrote: Hi Bron, Am Samstag, den 06.09.2014, 22:17 +1000 schrieb Bron Gondwana: That's what we do :) Thanks for your feedbeek. What's your workaround for not stopping cyrus before taking a lvm snapshot and run rsnapshot? Ciao Marcus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page:http://www.cyrusimap.org/ List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
We have an option called 'auditlog' now, which logs every append, regardless of where it came from, along with the sha1 of the spool file, and of course the mailbox name and UID. From that, you could calculate exactly which emails were new. I keep talking of writing a real incremental dump mode for Cyrus, but I haven't had a chance yet :( Bron. On Mon, Sep 8, 2014, at 02:28 PM, Stephen Ulmer wrote: I think at that point we may not have cared… We did still run TSM incrementals, basically back-to-back, but those didn’t finish in 24 hours. We needed a better-effort way to get most message for a disaster. I looked for the patch a bit ago (not that it would apply, but I would at least see what I had changed!) but couldn’t find it. I haven’t been employed by that organization for just over 8 years now, so it’s nowhere in my cache. The general point of my comment was: If you’re backing up files, looking for them is expensive. If you can get Cyrus to tell you what it changed, then you don’t have to ask the filesystem — which usually involves asking every file. The other option would be to use a filesystem that supports DMAPI, and write a DMAPI application that kept track of changed files and added them to a backup queue. It seems like having Cyrus log the delivery would have other benefits, though, and would work on any filesystem. I’m not currently a Cyrus admin, though, so it could already do that for all I know. :) Liberty, -- Stephen On Sep 7, 2014, at 10:03 PM, Bron Gondwana [1]br...@fastmail.fm wrote: Did you back up Sent folders too? On Mon, Sep 8, 2014, at 11:48 AM, Stephen Ulmer wrote: A long time ago for a much older version of Cyrus, we hacked lmtpd (I think, it’s been years) to log the messages as it wrote them down. Then we just processed the log every hour or so to backup only those files. That saved having to traverse the entire in ode tree most of the time. -- Stephen On Sep 6, 2014, at 7:32 PM, Bron Gondwana [2]br...@fastmail.fm wrote: No, no - we do replication. Replication rocks. You could easily stop the replica and take a snapshot of that, but our real backup solution is much more evil. I've posted it to this list before, but it's basically a perl daemon which knows far too much about how Cyrus locks its data files. It actually reads and parses cyrus.index files to work out what it needs to do. Bron. On Sun, Sep 7, 2014, at 04:50 AM, Marcus Schopen wrote: Hi Bron, Am Samstag, den 06.09.2014, 22:17 +1000 schrieb Bron Gondwana: That's what we do :) Thanks for your feedbeek. What's your workaround for not stopping cyrus before taking a lvm snapshot and run rsnapshot? Ciao Marcus -- Bron Gondwana [3]br...@fastmail.fm Cyrus Home Page:[4]http://www.cyrusimap.org/ List Archives/Info:[5]http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: [6]https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana [7]br...@fastmail.fm -- Bron Gondwana br...@fastmail.fm References 1. mailto:br...@fastmail.fm 2. mailto:br...@fastmail.fm 3. mailto:br...@fastmail.fm 4. http://www.cyrusimap.org/ 5. http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 6. https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 7. mailto:br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
Good morning, We are using Cyrus replication for some years now. It’s just fine. You can play too with expunge_delayed for avoid removing mail immediately. Regards, El 05/09/2014, a las 17:25, Marcus Schopen li...@localguru.de escribió: Am Samstag, den 30.08.2014, 17:10 +0200 schrieb Simon Matter: Am Freitag, den 29.08.2014, 07:56 -0700 schrieb David R Bosso: --On August 29, 2014 at 4:27:57 PM +0200 Marcus Schopen li...@localguru.de wrote: Hi, I'm planing to use lvm snaps and rsync for a daily disaster recovery backup on my master cyrus (2.4.12 Ubuntu 12.04 LTS): Have you tested lvm snaps for this purpose yet? When I tried to use lvm snapshots in this way a few years ago, it absolutely killed performance on the volume with the active snapshots. Hopefully things have improved, but I wouldn't bet on it. Yes, I've tested it and don't see any performance problems while the snaphot is active. My problem is how to backup about 140 GB mailspool in an acceptable time slot. What tools are recommended? If using rsync what options are best practice? I suggest -aH to preserve single instance storage in the backup. Just bought some good wine for my weekend project: what about rsnapshot combined with lvm snapshots (in the pre/postexec scripts of rsnapshot)? rsnapshot works with rsync and hardlinks. Rotation and some other nice features come out of the box. Any experiences with cyrus and this tool? Ciao! Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
In fact really…. I should say is “the way” of having a proper HA. Best regards, El 06/09/2014, a las 13:52, Egoitz Aurrekoetxea ego...@ramattack.net escribió: Good morning, We are using Cyrus replication for some years now. It’s just fine. You can play too with expunge_delayed for avoid removing mail immediately. Regards, El 05/09/2014, a las 17:25, Marcus Schopen li...@localguru.de escribió: Am Samstag, den 30.08.2014, 17:10 +0200 schrieb Simon Matter: Am Freitag, den 29.08.2014, 07:56 -0700 schrieb David R Bosso: --On August 29, 2014 at 4:27:57 PM +0200 Marcus Schopen li...@localguru.de wrote: Hi, I'm planing to use lvm snaps and rsync for a daily disaster recovery backup on my master cyrus (2.4.12 Ubuntu 12.04 LTS): Have you tested lvm snaps for this purpose yet? When I tried to use lvm snapshots in this way a few years ago, it absolutely killed performance on the volume with the active snapshots. Hopefully things have improved, but I wouldn't bet on it. Yes, I've tested it and don't see any performance problems while the snaphot is active. My problem is how to backup about 140 GB mailspool in an acceptable time slot. What tools are recommended? If using rsync what options are best practice? I suggest -aH to preserve single instance storage in the backup. Just bought some good wine for my weekend project: what about rsnapshot combined with lvm snapshots (in the pre/postexec scripts of rsnapshot)? rsnapshot works with rsync and hardlinks. Rotation and some other nice features come out of the box. Any experiences with cyrus and this tool? Ciao! Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
That's what we do :) Bron. On Sat, Sep 6, 2014, at 09:52 PM, Egoitz Aurrekoetxea wrote: In fact really…. I should say is “the way” of having a proper HA. Best regards, El 06/09/2014, a las 13:52, Egoitz Aurrekoetxea [1]ego...@ramattack.net escribió: Good morning, We are using Cyrus replication for some years now. It’s just fine. You can play too with expunge_delayed for avoid removing mail immediately. Regards, El 05/09/2014, a las 17:25, Marcus Schopen [2]li...@localguru.de escribió: Am Samstag, den 30.08.2014, 17:10 +0200 schrieb Simon Matter: Am Freitag, den 29.08.2014, 07:56 -0700 schrieb David R Bosso: --On August 29, 2014 at 4:27:57 PM +0200 Marcus Schopen [3]li...@localguru.de wrote: Hi, I'm planing to use lvm snaps and rsync for a daily disaster recovery backup on my master cyrus (2.4.12 Ubuntu 12.04 LTS): Have you tested lvm snaps for this purpose yet? When I tried to use lvm snapshots in this way a few years ago, it absolutely killed performance on the volume with the active snapshots. Hopefully things have improved, but I wouldn't bet on it. Yes, I've tested it and don't see any performance problems while the snaphot is active. My problem is how to backup about 140 GB mailspool in an acceptable time slot. What tools are recommended? If using rsync what options are best practice? I suggest -aH to preserve single instance storage in the backup. Just bought some good wine for my weekend project: what about rsnapshot combined with lvm snapshots (in the pre/postexec scripts of rsnapshot)? rsnapshot works with rsync and hardlinks. Rotation and some other nice features come out of the box. Any experiences with cyrus and this tool? Ciao! Cyrus Home Page:[4]http://www.cyrusimap.org/ List Archives/Info:[5]http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: [6]https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: [7]http://www.cyrusimap.org/ List Archives/Info: [8]http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: [9]https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm References 1. mailto:ego...@ramattack.net 2. mailto:li...@localguru.de 3. mailto:li...@localguru.de 4. http://www.cyrusimap.org/ 5. http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 6. https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 7. http://www.cyrusimap.org/ 8. http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 9. https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
Hi Bron, Am Samstag, den 06.09.2014, 22:17 +1000 schrieb Bron Gondwana: That's what we do :) Thanks for your feedbeek. What's your workaround for not stopping cyrus before taking a lvm snapshot and run rsnapshot? Ciao Marcus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
No, no - we do replication. Replication rocks. You could easily stop the replica and take a snapshot of that, but our real backup solution is much more evil. I've posted it to this list before, but it's basically a perl daemon which knows far too much about how Cyrus locks its data files. It actually reads and parses cyrus.index files to work out what it needs to do. Bron. On Sun, Sep 7, 2014, at 04:50 AM, Marcus Schopen wrote: Hi Bron, Am Samstag, den 06.09.2014, 22:17 +1000 schrieb Bron Gondwana: That's what we do :) Thanks for your feedbeek. What's your workaround for not stopping cyrus before taking a lvm snapshot and run rsnapshot? Ciao Marcus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
Am Samstag, den 30.08.2014, 17:10 +0200 schrieb Simon Matter: Am Freitag, den 29.08.2014, 07:56 -0700 schrieb David R Bosso: --On August 29, 2014 at 4:27:57 PM +0200 Marcus Schopen li...@localguru.de wrote: Hi, I'm planing to use lvm snaps and rsync for a daily disaster recovery backup on my master cyrus (2.4.12 Ubuntu 12.04 LTS): Have you tested lvm snaps for this purpose yet? When I tried to use lvm snapshots in this way a few years ago, it absolutely killed performance on the volume with the active snapshots. Hopefully things have improved, but I wouldn't bet on it. Yes, I've tested it and don't see any performance problems while the snaphot is active. My problem is how to backup about 140 GB mailspool in an acceptable time slot. What tools are recommended? If using rsync what options are best practice? I suggest -aH to preserve single instance storage in the backup. Just bought some good wine for my weekend project: what about rsnapshot combined with lvm snapshots (in the pre/postexec scripts of rsnapshot)? rsnapshot works with rsync and hardlinks. Rotation and some other nice features come out of the box. Any experiences with cyrus and this tool? Ciao! Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
Am Samstag, den 30.08.2014, 17:10 +0200 schrieb Simon Matter: Am Freitag, den 29.08.2014, 07:56 -0700 schrieb David R Bosso: --On August 29, 2014 at 4:27:57 PM +0200 Marcus Schopen li...@localguru.de wrote: Hi, I'm planing to use lvm snaps and rsync for a daily disaster recovery backup on my master cyrus (2.4.12 Ubuntu 12.04 LTS): Have you tested lvm snaps for this purpose yet? When I tried to use lvm snapshots in this way a few years ago, it absolutely killed performance on the volume with the active snapshots. Hopefully things have improved, but I wouldn't bet on it. Yes, I've tested it and don't see any performance problems while the snaphot is active. My problem is how to backup about 140 GB mailspool in an acceptable time slot. What tools are recommended? If using rsync what options are best practice? I suggest -aH to preserve single instance storage in the backup. So, rsync -aH to keep hardlinks in the spool. What about -A, --aclspreserve ACLs (implies -p) -X, --xattrs reserve extended attributes Ciao! Marcus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
Am Samstag, den 30.08.2014, 14:37 -0500 schrieb Patrick Goetz: On 8/30/2014 10:10 AM, Simon Matter wrote: I suggest -aH to preserve single instance storage in the backup. Does cyrus use a lot of hard links? I use rsync a lot to create snapshot backups, and use hard links across snapshots to preserve space; however, for a single instance backup and unless the filesystem includes hard links (not normal), then the -H won't do much for you. Of course one should always use -a. The biggest concern I have about backing up mail spools is keeping the index and message stores in sync while the backup is taking place. This is why I use lvm. A long time ago someone suggested using cyrdump, but when I looked into this, I couldn't find any documentation whatsoever. Is cyrdump a real thing, or did I imagine all this? Ciao Marcus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
Am Freitag, den 29.08.2014, 07:56 -0700 schrieb David R Bosso: --On August 29, 2014 at 4:27:57 PM +0200 Marcus Schopen li...@localguru.de wrote: Hi, I'm planing to use lvm snaps and rsync for a daily disaster recovery backup on my master cyrus (2.4.12 Ubuntu 12.04 LTS): Have you tested lvm snaps for this purpose yet? When I tried to use lvm snapshots in this way a few years ago, it absolutely killed performance on the volume with the active snapshots. Hopefully things have improved, but I wouldn't bet on it. Yes, I've tested it and don't see any performance problems while the snaphot is active. My problem is how to backup about 140 GB mailspool in an acceptable time slot. What tools are recommended? If using rsync what options are best practice? Ciao Marcus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
Am Freitag, den 29.08.2014, 07:56 -0700 schrieb David R Bosso: --On August 29, 2014 at 4:27:57 PM +0200 Marcus Schopen li...@localguru.de wrote: Hi, I'm planing to use lvm snaps and rsync for a daily disaster recovery backup on my master cyrus (2.4.12 Ubuntu 12.04 LTS): Have you tested lvm snaps for this purpose yet? When I tried to use lvm snapshots in this way a few years ago, it absolutely killed performance on the volume with the active snapshots. Hopefully things have improved, but I wouldn't bet on it. Yes, I've tested it and don't see any performance problems while the snaphot is active. My problem is how to backup about 140 GB mailspool in an acceptable time slot. What tools are recommended? If using rsync what options are best practice? I suggest -aH to preserve single instance storage in the backup. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
On 8/30/2014 10:10 AM, Simon Matter wrote: I suggest -aH to preserve single instance storage in the backup. Does cyrus use a lot of hard links? I use rsync a lot to create snapshot backups, and use hard links across snapshots to preserve space; however, for a single instance backup and unless the filesystem includes hard links (not normal), then the -H won't do much for you. Of course one should always use -a. The biggest concern I have about backing up mail spools is keeping the index and message stores in sync while the backup is taking place. A long time ago someone suggested using cyrdump, but when I looked into this, I couldn't find any documentation whatsoever. Is cyrdump a real thing, or did I imagine all this? Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
On 08/30/2014 02:37 PM, Patrick Goetz wrote: On 8/30/2014 10:10 AM, Simon Matter wrote: I suggest -aH to preserve single instance storage in the backup. Does cyrus use a lot of hard links? I use rsync a lot to create snapshot backups, and use hard links across snapshots to preserve space; however, for a single instance backup and unless the filesystem includes hard links (not normal), then the -H won't do much for you. Cyrus uses hard links among copies of the same message in different mailboxes, when singleinstancestore: 1 is set, which is the default. From the man page imapd.conf(5): If enabled, imapd, lmtpd and nntpd attempt to only write one copy of a message per partition and create hard links, resulting in a potentially large disk savings. Of course one should always use -a. The biggest concern I have about backing up mail spools is keeping the index and message stores in sync while the backup is taking place. A long time ago someone suggested using cyrdump, but when I looked into this, I couldn't find any documentation whatsoever. Is cyrdump a real thing, or did I imagine all this? Are you thinking of ctl_mboxlist? It allows one to dump the mailboxes database to a flat file. In an out-of-the-box Cyrus installation the indexes are stored with the messages in the mailbox hierarchy. If you decide to store meta-data separately, you should simply snapshot that at the same time you snapshot your mailstore. At the beginning of this thread, Marcus Schopen wrote this example: ctl_cyrusdb -c ctl_mboxlist -d mailboxes.db.dump stop cyrus lvm snaps start cyrus rsync/var/lib/cyrus/ and /var/spool/cyrus to backup host remove snaps One could simply include any meta-data volumes in the snapshotting process. Cheers, -nic Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
Thanks for that explanation! I don't have time to read every email that comes to the cyrus list, and missed that one. One point of clarification: On 8/30/2014 2:49 PM, Nic Bernstein wrote: ctl_cyrusdb -c ctl_mboxlist -d mailboxes.db.dump stop cyrus lvm snaps start cyrus rsync/var/lib/cyrus/ and /var/spool/cyrus to backup host remove snaps Wouldn't you want to stop cyrus *before* running ctl_mboxlist -d mailboxes.db.dump ? Also, are these ctl_* commands documented anywhere? Thanks! Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
backup rsync
Hi, I'm planing to use lvm snaps and rsync for a daily disaster recovery backup on my master cyrus (2.4.12 Ubuntu 12.04 LTS): ctl_cyrusdb -c ctl_mboxlist -d mailboxes.db.dump stop cyrus lvm snaps start cyrus rsync /var/lib/cyrus/ and /var/spool/cyrus to backup host remove snaps Is there something to be aware with rsync especially with /var/spool/cyrus directories? Beside that the master is in master-slave replication too. In a case of disaster recovery - if you don't want to make the slave to the master - would it work out to rsync /var/lib/cyrus/ and /var/spool/cyrus from the slave to the master or is that not a good idea? Ciao Marcus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
--On August 29, 2014 at 4:27:57 PM +0200 Marcus Schopen li...@localguru.de wrote: Hi, I'm planing to use lvm snaps and rsync for a daily disaster recovery backup on my master cyrus (2.4.12 Ubuntu 12.04 LTS): Have you tested lvm snaps for this purpose yet? When I tried to use lvm snapshots in this way a few years ago, it absolutely killed performance on the volume with the active snapshots. Hopefully things have improved, but I wouldn't bet on it. -David ctl_cyrusdb -c ctl_mboxlist -d mailboxes.db.dump stop cyrus lvm snaps start cyrus rsync /var/lib/cyrus/ and /var/spool/cyrus to backup host remove snaps Is there something to be aware with rsync especially with /var/spool/cyrus directories? Beside that the master is in master-slave replication too. In a case of disaster recovery - if you don't want to make the slave to the master - would it work out to rsync /var/lib/cyrus/ and /var/spool/cyrus from the slave to the master or is that not a good idea? Ciao Marcus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: backup rsync
There's something to be worried about with using lvm snapshots. My understanding is that this can dramatically slow down your system. I'm too busy right now to pull up a reference, but google is your friend. On 08/29/2014 09:27 AM, Marcus Schopen wrote: Hi, I'm planing to use lvm snaps and rsync for a daily disaster recovery backup on my master cyrus (2.4.12 Ubuntu 12.04 LTS): ctl_cyrusdb -c ctl_mboxlist -d mailboxes.db.dump stop cyrus lvm snaps start cyrus rsync /var/lib/cyrus/ and /var/spool/cyrus to backup host remove snaps Is there something to be aware with rsync especially with /var/spool/cyrus directories? Beside that the master is in master-slave replication too. In a case of disaster recovery - if you don't want to make the slave to the master - would it work out to rsync /var/lib/cyrus/ and /var/spool/cyrus from the slave to the master or is that not a good idea? Ciao Marcus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
