Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On Wed, Feb 24, 2010 at 7:05 PM, Gavin Carr ga...@openfusion.com.au wrote: On Wed, Feb 24, 2010 at 05:06:08PM +0530, Agnello George wrote: On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen eero.voloti...@iki.fi wrote: 2010/2/24 Agnello George agnello.dso...@gmail.com: On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr ga...@openfusion.com.au wrote: You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository). Cheers, Gavin is it possible with brackup to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same . rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!) Does http://code.google.com/p/brackup/ also work in on remote machines . Brackup will backup to local disk, or remotely to ftp, sftp, Amazon S3, or Rackspace CloudFiles targets/servers. So yes, on a lan you can backup over ftp or sftp just fine. Re docs, install brackup, 'man Brackup::Manual::Overview'. I've also written a few blog posts on it: http://www.openfusion.net/tags/brackup. Cheers, Gavin I am trying to install the brackup app on my system, the documentations seems very helpful ( http://www.openfusion.net/net/fun_with_brackup) But i have a few queries with the config file : [TARGET:backups] type = Filesystem path = /backup [SOURCE:imapsource] path = /var/spool/imap chunk_size = 5m # what does this mean gpg_recipient = 5E1B3EC5 # what does this mean [SOURCE:bradhome] chunk_size = 64MB path = /raid/bradfitz/ ignore = ^\.thumbnails/ ignore = ^\.kde/share/thumbnails/ ignore = ^\.ee/minis/ ignore = ^build/ ignore = ^(gqview|nautilus)/thumbnails/ and suppose i want to backup it up to another server with scp / ssh how is this attatined . secondly in whant format is the backup maintained . -- Regards Agnello D'souza ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
Have you read Brackup::Manual::Overview? Your questions are all answered in the man pages there or linked from there. On Thu, Feb 25, 2010 at 05:21:47PM +0530, Agnello George wrote: On Wed, Feb 24, 2010 at 7:05 PM, Gavin Carr ga...@openfusion.com.au wrote: Brackup will backup to local disk, or remotely to ftp, sftp, Amazon S3, or Rackspace CloudFiles targets/servers. So yes, on a lan you can backup over ftp or sftp just fine. Re docs, install brackup, 'man Brackup::Manual::Overview'. I've also written a few blog posts on it: http://www.openfusion.net/tags/brackup. I am trying to install the brackup app on my system, the documentations seems very helpful ( http://www.openfusion.net/net/fun_with_brackup) But i have a few queries with the config file : [TARGET:backups] type = Filesystem path = /backup [SOURCE:imapsource] path = /var/spool/imap chunk_size = 5m # what does this mean gpg_recipient = 5E1B3EC5 # what does this mean man Brackup::Manual::Overview; man Brackup::Root [SOURCE:bradhome] chunk_size = 64MB path = /raid/bradfitz/ ignore = ^\.thumbnails/ ignore = ^\.kde/share/thumbnails/ ignore = ^\.ee/minis/ ignore = ^build/ ignore = ^(gqview|nautilus)/thumbnails/ and suppose i want to backup it up to another server with scp / ssh how is this attatined . man Brackup::Target::Sftp secondly in whant format is the backup maintained . Backups are trees of file chunks, and a metadata file to put the chunks back together as files. So you get de-duplication for free between files and across backups. Cheers, Gavin ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On 02/24/2010 07:44 PM, Les Mikesell wrote: Err.. raid is NOT backup solution. Neither is a snapshot in another location on the same machine. Thats not true, raid is an online setup - different location could be point in time, and on blockdev;s that dont share user access load. Which in turn makes it easier to do intensive complete backups to offsite without impacting user level of service the machine can deliver, amongst other things[1]. Dont compare apples to banana's and call them oranges. - KB [1] Changeset and data/system model over time relation mapping for an adaptive system sizing feedback loop! ( how'se that for buzzword injection! ) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
Karanbir Singh wrote: On 02/24/2010 07:44 PM, Les Mikesell wrote: Err.. raid is NOT backup solution. Neither is a snapshot in another location on the same machine. Thats not true, raid is an online setup - different location could be point in time, and on blockdev;s that dont share user access load. Which in turn makes it easier to do intensive complete backups to offsite without impacting user level of service the machine can deliver, amongst other things[1]. Dont compare apples to banana's and call them oranges. Yes and no... There's an overlapping set of possibilities that they do and don't back up. They both cover single disk failures. They don't cover big operator errors (rm -rf /), building/site disasters, some types of controller/electrical issues, etc. The snapshots give you a short history that can help with small user/operator errors at the expense of being out of date when the live disk fails. So have several types of fruit to stay healthy. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On Thu, 2010-02-25 at 12:33 +, Karanbir Singh wrote: [1] Changeset and data/system model over time relation mapping for an adaptive system sizing feedback loop! ( how'se that for buzzword injection! ) --- Well if you run vacum on a Postgres DB then all that goes to the crapper... So we resort to real time backups or replication. Of which with replication on a postgres db with the 2GB Blobs being inserted there is going to be a problem... even if you have enough shared memory configured. The thing is regardless you are never in RT Replication. John ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
From: Agnello George agnello.dso...@gmail.com is it possible with brackup to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same . It apparently support: Brackup::Target::Amazon backup to Amazon's S3 service Brackup::Target::CloudFiles backup to Rackspace's CloudFiles Service Brackup::Target::Filebased Brackup::Target::Filesystem backup to a locally mounted filesystem Brackup::Target::Ftp backup to an FTP server Brackup::Target::Sftpbackup to an SSH/SFTP server So, you could use ftp or sftp... JD ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
Agnello George wrote:
The requirement fro backup is not primarily for HDD failure , but human
error failure . In case one of our user ( eg: the COO with huge mailbox
size has delete all his certain very important mails, and he want to recover
them , the contacts us as we are supposed to maintain his mail backup for a
week, and we should restore his backup immediately ) this the main
requirement for the backup and that too on the same server different
partition .
Have you considered using a snapshot approach? By that, I mean one
which uses hard links to create the backup, and as files get added/
modified, the data are copied, and links are created. Usually, one
has a snapshot directory with something like a daily snapshot, and
24 hourly ones, something like that.
Mike
--
p=p=%c%s%c;main(){printf(p,34,p,34);};main(){printf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I speak only for myself, and I am unanimous in that!
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
Karanbir Singh wrote:
On 02/24/2010 07:44 PM, Les Mikesell wrote:
Err.. raid is NOT backup solution.
Neither is a snapshot in another location on the same machine.
Thats not true, raid is an online setup - different location could be
point in time, and on blockdev;s that dont share user access load. Which
I think that, without causing any more dispute, I can point out
that backup covers a wide range of solutions to a less broad
but still not uniquely one set of needs. No one of the means to
backup is a full solution to all the needs which backup satisfies.
Even when one is using the term backup narrowly in the sense
of protection from disaster, there are still different kinds
of backup. For example, there is the full disaster recovery
or bare metal backup, which is intended to work with another
piece of identical hardware, starting with blank fixed storage,
and ending up with a working system which looks identical to the
original at the epoch at which the backup was made. This is
significantly different from one intended merely to restore the
user altered or created data on a machine which has been newly
installed with a compatible version of the OS, for example.
That's why one needs to know the intended use of the backup set
before making any recommendations on procedure and content of the
backup set.
Mike
--
p=p=%c%s%c;main(){printf(p,34,p,34);};main(){printf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I speak only for myself, and I am unanimous in that!
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
Years ago, I set up a backup tool that wrapped rsync. It has faithfully and reliably backed up a dozen hosts and too many TB of data to mention, offsite, automatically saving as many backup points as disk space allows. You're certainly welcome to try it! http://www.effortlessis.com/thisisnotbackupbuddy/ It works on an ascending powers basis, EG: 1 day ago, 2 days ago, 4 days ago, 8 days ago, 16 days ago... until out of disk space. =) -Ben On Thursday 25 February 2010 10:22:13 am Mike McCarty wrote: Agnello George wrote: The requirement fro backup is not primarily for HDD failure , but human error failure . In case one of our user ( eg: the COO with huge mailbox size has delete all his certain very important mails, and he want to recover them , the contacts us as we are supposed to maintain his mail backup for a week, and we should restore his backup immediately ) this the main requirement for the backup and that too on the same server different partition . Have you considered using a snapshot approach? By that, I mean one which uses hard links to create the backup, and as files get added/ modified, the data are copied, and links are created. Usually, one has a snapshot directory with something like a daily snapshot, and 24 hourly ones, something like that. Mike -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
Greetings, On Fri, Feb 26, 2010 at 5:50 AM, Benjamin Smith li...@benjamindsmith.comwrote: Years ago, I set up a backup tool that wrapped rsync. It has faithfully and reliably backed up a dozen hosts and too many TB of data to mention, offsite, automatically saving as many backup points as disk space allows. You're certainly welcome to try it! http://www.effortlessis.com/thisisnotbackupbuddy/ It works on an ascending powers basis, EG: 1 day ago, 2 days ago, 4 days ago, 8 days ago, 16 days ago... until out of disk space. =) grin|smile|whatever it is a criminal offence (in free software world) to hide this gem from the world for all this long. ;) Regards, Rajagopal ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
2010/2/24 Agnello George agnello.dso...@gmail.com: Hi We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . So, you need to add more disk i/o? (so, add some disk space with faster raid?) Take a look at: http://rdiff-backup.nongnu.org/ -- Eero ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
Dne 24.2.2010 10:00, Agnello George napsal(a): Hi We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . Is there any back up solution that you can think of , that can work in this situation - open source or proprietary Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. David ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On Wed, Feb 24, 2010 at 3:08 PM, David Hrbáč hrbac.c...@seznam.cz wrote: Dne 24.2.2010 10:00, Agnello George napsal(a): Hi We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . Is there any back up solution that you can think of , that can work in this situation - open source or proprietary Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. David backup directory structure is /var/spool/imap/a /adomain.com/a/agnello^dsouza/ -- Regards Agnello D'souza ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
2010/2/24 Agnello George agnello.dso...@gmail.com: On Wed, Feb 24, 2010 at 3:08 PM, David Hrbáč hrbac.c...@seznam.cz wrote: Dne 24.2.2010 10:00, Agnello George napsal(a): Hi We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . Is there any back up solution that you can think of , that can work in this situation - open source or proprietary Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. David backup directory structure is /var/spool/imap/a /adomain.com/a/agnello^dsouza/ Well, does that directory contains one file or lot of files ? Usually maildir structure is like dir++/tmp/current/new directories and each message is in own file on mailbox all files are inside one file. -- Eero ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote: Dne 24.2.2010 10:00, Agnello George napsal(a): We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . Is there any back up solution that you can think of , that can work in this situation - open source or proprietary Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. rsync and rdiff should handle mbox format okay though. Though I agree Maildir is generally nicer for differential backups. Agnello, how long is a lot of time? A backup is always going to have to walk the entire tree and checksum (or at least stat) every file, so there's a minimum cost you're always going to have. How long does a 'find /var/spool/imap -ls' take, for instance? You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository). Cheers, Gavin ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
2010/2/24 Eero Volotinen eero.voloti...@iki.fi 2010/2/24 Agnello George agnello.dso...@gmail.com: On Wed, Feb 24, 2010 at 3:08 PM, David Hrbáč hrbac.c...@seznam.cz wrote: Dne 24.2.2010 10:00, Agnello George napsal(a): Hi We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . Is there any back up solution that you can think of , that can work in this situation - open source or proprietary Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. David backup directory structure is /var/spool/imap/a /adomain.com/a/agnello^dsouza/ Well, does that directory contains one file or lot of files ? Usually maildir structure is like dir++/tmp/current/new directories and each message is in own file on mailbox all files are inside one file. its in a maildir format and the structure slightly different from what i mentioned earlier : r...@server1 ~]# ls -la /var/spool/imap/a/user/ajay/ total 5180 -rw--- 1 cyrus mail 34616 Feb 24 16:02 4790. -rw--- 1 cyrus mail 4490 Feb 24 16:03 4791. -rw--- 1 cyrus mail 199253 Feb 24 16:07 4792. -rw--- 1 cyrus mail 22930 Feb 24 16:09 4793. -rw--- 1 cyrus mail 8485 Feb 24 16:11 4794. -rw--- 1 cyrus mail 12664 Feb 24 16:13 4795. -rw--- 1 cyrus mail 4296 Feb 24 16:13 4796. -rw--- 1 cyrus mail 5337 Feb 24 16:15 4797. -rw--- 1 cyrus mail 111030 Feb 24 16:21 4798. -rw--- 1 cyrus mail 4805500 Feb 24 16:23 4799. -rw--- 1 cyrus mail 22920 Feb 24 16:23 cyrus.cache -rw--- 1 cyrus mail 204 Dec 10 16:27 cyrus.header -rw--- 1 cyrus mail 896 Feb 24 16:23 cyrus.index -rw--- 1 cyrus mail 8669 Feb 24 11:28 cyrus.squat this is Just a very small user and a example -- Regards Agnello D'souza ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr ga...@openfusion.com.au wrote: On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote: Dne 24.2.2010 10:00, Agnello George napsal(a): We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . Is there any back up solution that you can think of , that can work in this situation - open source or proprietary Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. rsync and rdiff should handle mbox format okay though. Though I agree Maildir is generally nicer for differential backups. Agnello, how long is a lot of time? A backup is always going to have to walk the entire tree and checksum (or at least stat) every file, so there's a minimum cost you're always going to have. How long does a 'find /var/spool/imap -ls' take, for instance? You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository). Cheers, Gavin http://lists.centos.org/mailman/listinfo/centos is it possible with brackup http://code.google.com/p/brackup/ to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same . -- Regards Agnello D'souza ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
2010/2/24 Agnello George agnello.dso...@gmail.com: On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr ga...@openfusion.com.au wrote: On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote: Dne 24.2.2010 10:00, Agnello George napsal(a): We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . Is there any back up solution that you can think of , that can work in this situation - open source or proprietary Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. rsync and rdiff should handle mbox format okay though. Though I agree Maildir is generally nicer for differential backups. Agnello, how long is a lot of time? A backup is always going to have to walk the entire tree and checksum (or at least stat) every file, so there's a minimum cost you're always going to have. How long does a 'find /var/spool/imap -ls' take, for instance? You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository). Cheers, Gavin is it possible with brackup to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same . rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!) -- Eero ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen eero.voloti...@iki.fiwrote: 2010/2/24 Agnello George agnello.dso...@gmail.com: On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr ga...@openfusion.com.au wrote: On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote: Dne 24.2.2010 10:00, Agnello George napsal(a): We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . Is there any back up solution that you can think of , that can work in this situation - open source or proprietary Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. rsync and rdiff should handle mbox format okay though. Though I agree Maildir is generally nicer for differential backups. Agnello, how long is a lot of time? A backup is always going to have to walk the entire tree and checksum (or at least stat) every file, so there's a minimum cost you're always going to have. How long does a 'find /var/spool/imap -ls' take, for instance? You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository). Cheers, Gavin is it possible with brackup to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same . rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!) Does http://code.google.com/p/brackup/ also work in on remote machines . -- Regards Agnello D'souza ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
2010/2/24 Agnello George agnello.dso...@gmail.com: On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen eero.voloti...@iki.fi wrote: 2010/2/24 Agnello George agnello.dso...@gmail.com: On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr ga...@openfusion.com.au wrote: On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote: Dne 24.2.2010 10:00, Agnello George napsal(a): We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . Is there any back up solution that you can think of , that can work in this situation - open source or proprietary Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. rsync and rdiff should handle mbox format okay though. Though I agree Maildir is generally nicer for differential backups. Agnello, how long is a lot of time? A backup is always going to have to walk the entire tree and checksum (or at least stat) every file, so there's a minimum cost you're always going to have. How long does a 'find /var/spool/imap -ls' take, for instance? You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository). Cheers, Gavin is it possible with brackup to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same . rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!) Summary from webpage: Flexible backup tool. Slices, dices, encrypts, and sprays across the net, notably to Amazon's S3. -- Eero ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On 02/24/2010 11:21 AM, Agnello George wrote: -rw--- 1 cyrus mail 4805500 Feb 24 16:23 4799. -rw--- 1 cyrus mail 22920 Feb 24 16:23 cyrus.cache -rw--- 1 cyrus mail 204 Dec 10 16:27 cyrus.header -rw--- 1 cyrus mail 896 Feb 24 16:23 cyrus.index -rw--- 1 cyrus mail 8669 Feb 24 11:28 cyrus.squat this is Just a very small user and a example About 90% of your problem is already solved here, you are using cyrus which has built in mail level replication. All you need to do is setup a lvm volume away from this main store and run your mail replica over to it. then just backup using whatever tools you want. Free win you get is online failover, backup in whatever manner you want! - KB ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On Wed, Feb 24, 2010 at 6:19 PM, Karanbir Singh mail-li...@karan.orgwrote: On 02/24/2010 11:21 AM, Agnello George wrote: -rw--- 1 cyrus mail 4805500 Feb 24 16:23 4799. -rw--- 1 cyrus mail 22920 Feb 24 16:23 cyrus.cache -rw--- 1 cyrus mail 204 Dec 10 16:27 cyrus.header -rw--- 1 cyrus mail 896 Feb 24 16:23 cyrus.index -rw--- 1 cyrus mail 8669 Feb 24 11:28 cyrus.squat this is Just a very small user and a example About 90% of your problem is already solved here, you are using cyrus which has built in mail level replication. All you need to do is setup a lvm volume away from this main store and run your mail replica over to it. then just backup using whatever tools you want. Free win you get is online failover, backup in whatever manner you want! yes just spoke to my senior and confrimed that this was alreday tried out a delayed replication is possible . but the current suitation is we need to take backup on the same server on a different partition /backup :( -- Regards Agnello D'souza ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On Wed, Feb 24, 2010 at 05:06:08PM +0530, Agnello George wrote: On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen eero.voloti...@iki.fiwrote: 2010/2/24 Agnello George agnello.dso...@gmail.com: On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr ga...@openfusion.com.au wrote: You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository). Cheers, Gavin is it possible with brackup to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same . rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!) Does http://code.google.com/p/brackup/ also work in on remote machines . Brackup will backup to local disk, or remotely to ftp, sftp, Amazon S3, or Rackspace CloudFiles targets/servers. So yes, on a lan you can backup over ftp or sftp just fine. Re docs, install brackup, 'man Brackup::Manual::Overview'. I've also written a few blog posts on it: http://www.openfusion.net/tags/brackup. Cheers, Gavin ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On Wed, Feb 24, 2010 at 7:05 PM, Gavin Carr ga...@openfusion.com.au wrote: On Wed, Feb 24, 2010 at 05:06:08PM +0530, Agnello George wrote: On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen eero.voloti...@iki.fi wrote: 2010/2/24 Agnello George agnello.dso...@gmail.com: On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr ga...@openfusion.com.au wrote: You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository). Cheers, Gavin is it possible with brackup to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same . rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!) Does http://code.google.com/p/brackup/ also work in on remote machines . Brackup will backup to local disk, or remotely to ftp, sftp, Amazon S3, or Rackspace CloudFiles targets/servers. So yes, on a lan you can backup over ftp or sftp just fine. Re docs, install brackup, 'man Brackup::Manual::Overview'. I've also written a few blog posts on it: http://www.openfusion.net/tags/brackup. Cheers, Gavin ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos it will take me some time to try this .. will get back on its output !! .. thanks -- Regards Agnello D'souza ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
Agnello George wrote: Hi We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O . Is there any back up solution that you can think of , that can work in this situation - open source or proprietary If you are just concerned about a single disk failure you could set up RAID1 on the disks (with some downtime to rebuild...) to keep the copy in realtime with little loss of speed. Rsync should work as well as anything for snapshots but you might need to update to a 3.x version to speed up handling large numbers of files. The 2.x version included in Centos will read the entire directory tree into memory before starting the comparisons and copies. The rpmforge repo has a packaged 3.0.7 version but I haven't tried it. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On 02/24/2010 01:07 PM, Agnello George wrote: yes just spoke to my senior and confrimed that this was alreday tried out a delayed replication is possible . but the current suitation is we need to take backup on the same server on a different partition /backup :( you can replicate to a local mail store as well. just make sure you put it on a block device that is suiteable and fits in with the rest of your backup strategy. And if you put it in an isolated enough place on the block dev, it wont contest with the users access. - KB ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
Agnello George wrote:
Hi
We have an issue with one of our clients , they have a mail server with
the /var/spool/imap partition as 150 GB . They need to take differential
backup on to /backup partition ( a different HDD of total 250 GB space ) .
You've stated things in terms of solutions. You may possibly get better
answers if you state your goal. There is some capability you are
trying to achieve. Tell us what that is, and you may make more progress.
IOW, what is the purpose of the backup? As one mentioned, RAID may
handle your needs.
Mike
--
p=p=%c%s%c;main(){printf(p,34,p,34);};main(){printf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I speak only for myself, and I am unanimous in that!
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
2010/2/24 Mike McCarty mike.mcca...@sbcglobal.net: Agnello George wrote: Hi We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . You've stated things in terms of solutions. You may possibly get better answers if you state your goal. There is some capability you are trying to achieve. Tell us what that is, and you may make more progress. IOW, what is the purpose of the backup? As one mentioned, RAID may handle your needs. Err.. raid is NOT backup solution. -- Eero ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On 2/24/2010 1:31 PM, Eero Volotinen wrote: We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . You've stated things in terms of solutions. You may possibly get better answers if you state your goal. There is some capability you are trying to achieve. Tell us what that is, and you may make more progress. IOW, what is the purpose of the backup? As one mentioned, RAID may handle your needs. Err.. raid is NOT backup solution. Neither is a snapshot in another location on the same machine. But both will cover the most likely thing to fail, with raid doing it transparently, the snapshot losing data from the time the last snapshot copy happened. Usually what you want is raid _and_ a history of backups kept elsewhere. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
Eero Volotinen wrote:
2010/2/24 Mike McCarty mike.mcca...@sbcglobal.net:
Agnello George wrote:
Hi
We have an issue with one of our clients , they have a mail server with
the /var/spool/imap partition as 150 GB . They need to take differential
backup on to /backup partition ( a different HDD of total 250 GB space ) .
You've stated things in terms of solutions. You may possibly get better
answers if you state your goal. There is some capability you are
trying to achieve. Tell us what that is, and you may make more progress.
IOW, what is the purpose of the backup? As one mentioned, RAID may
handle your needs.
Err.. raid is NOT backup solution.
Of course not. RAID is a means to achieve availability,
which may be his goal. Karanbir already stated a means to do
what he seemed to want, but it seemed not to satisfy his needs.
Unless the query is placed in terms of requirements and goals,
instead of solutions, it'll be difficult to achieve satisfactory
results.
The purpose of backup is some degree of disaster recovery, and
perhaps also migration. If that's truly his goal, then ISTM
that Karanbir suggested a viable solution to achieving avialability
while also performing backup, by doing on-the-fly duplication
of the data onto another file system which can then be backed up
at liesure.
Doing so in a manner which ensures a true snapshot may be more
difficult to achieve, while still ensuring availability. I normally
do my backups in single user mode with all file systems mounted read
only, except the one to receive the backup. That of course precludes
availability during the backup procedure.
That's why I would like to see what he actually wants to achieve,
instead of how he has chosen to go about it.
Mike
--
p=p=%c%s%c;main(){printf(p,34,p,34);};main(){printf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I speak only for myself, and I am unanimous in that!
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Backup solution to backup /var/spool/imap above 150GB data
On Thu, Feb 25, 2010 at 1:19 AM, Mike McCarty mike.mcca...@sbcglobal.netwrote: Eero Volotinen wrote: 2010/2/24 Mike McCarty mike.mcca...@sbcglobal.net: Agnello George wrote: Hi We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . You've stated things in terms of solutions. You may possibly get better answers if you state your goal. There is some capability you are trying to achieve. Tell us what that is, and you may make more progress. IOW, what is the purpose of the backup? As one mentioned, RAID may handle your needs. Err.. raid is NOT backup solution. Of course not. RAID is a means to achieve availability, which may be his goal. Karanbir already stated a means to do what he seemed to want, but it seemed not to satisfy his needs. Unless the query is placed in terms of requirements and goals, instead of solutions, it'll be difficult to achieve satisfactory results. The purpose of backup is some degree of disaster recovery, and perhaps also migration. If that's truly his goal, then ISTM that Karanbir suggested a viable solution to achieving avialability while also performing backup, by doing on-the-fly duplication of the data onto another file system which can then be backed up at liesure. Doing so in a manner which ensures a true snapshot may be more difficult to achieve, while still ensuring availability. I normally do my backups in single user mode with all file systems mounted read only, except the one to receive the backup. That of course precludes availability during the backup procedure. That's why I would like to see what he actually wants to achieve, instead of how he has chosen to go about it. Mike -- The requirement fro backup is not primarily for HDD failure , but human error failure . In case one of our user ( eg: the COO with huge mailbox size has delete all his certain very important mails, and he want to recover them , the contacts us as we are supposed to maintain his mail backup for a week, and we should restore his backup immediately ) this the main requirement for the backup and that too on the same server different partition . -- Regards Agnello D'souza ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
