Re: Synching a Backup Server

2011-01-25 Thread Kaspar Schleiser
On 01/22/2011 02:55 PM, Hubert Kario wrote: It looks like ZFS, Btrfs, and LVM should work in similar manners, but the overloaded terminology (pool, volume, sub-volume, filesystem are different in all three) and new terminology that's only in Btrfs is confusing. With btrfs you need to have *a*

Re: Synching a Backup Server

2011-01-25 Thread Hubert Kario
On Tuesday, January 25, 2011 18:29:35 Kaspar Schleiser wrote: On 01/22/2011 02:55 PM, Hubert Kario wrote: It looks like ZFS, Btrfs, and LVM should work in similar manners, but the overloaded terminology (pool, volume, sub-volume, filesystem are different in all three) and new terminology

Re: Synching a Backup Server

2011-01-25 Thread Hubert Kario
On Tuesday, January 25, 2011 18:59:39 Freddie Cash wrote: On Tue, Jan 25, 2011 at 9:43 AM, Hubert Kario h...@qbs.com.pl wrote: Besides, I don't see *why* this should be done... And as far as I know ZFS doesn't support different reduncancy levels for different files residing in the same

Re: Synching a Backup Server

2011-01-24 Thread Freddie Cash
On Sat, Jan 22, 2011 at 5:45 AM, Hugo Mills hugo-l...@carfax.org.uk wrote: On Fri, Jan 21, 2011 at 11:28:19AM -0800, Freddie Cash wrote: So, is Btrfs pooled storage or not?  Do you throw 24 disks into a single Btrfs filesystem, and then split that up into separate sub-volumes as needed?  

Re: Synching a Backup Server

2011-01-22 Thread Hugo Mills
On Fri, Jan 21, 2011 at 11:28:19AM -0800, Freddie Cash wrote: On Sun, Jan 9, 2011 at 10:30 AM, Hugo Mills hugo-l...@carfax.org.uk wrote: On Sun, Jan 09, 2011 at 09:59:46AM -0800, Freddie Cash wrote: Let see if I can match up the terminology and layers a bit: LVM Physical Volume == Btrfs

Re: Synching a Backup Server

2011-01-21 Thread Freddie Cash
On Sun, Jan 9, 2011 at 10:30 AM, Hugo Mills hugo-l...@carfax.org.uk wrote: On Sun, Jan 09, 2011 at 09:59:46AM -0800, Freddie Cash wrote: Let see if I can match up the terminology and layers a bit: LVM Physical Volume == Btrfs disk == ZFS disk / vdevs LVM Volume Group == Btrfs filesystem ==

Re: Synching a Backup Server

2011-01-10 Thread Hubert Kario
On Sunday 09 of January 2011 12:46:59 Alan Chandler wrote: On 07/01/11 16:20, Hubert Kario wrote: I usually create subvolumes in btrfs root volume: /mnt/btrfs/ |- server-a |- server-b \- server-c then create snapshots of these

Re: Synching a Backup Server

2011-01-09 Thread Alan Chandler
On 07/01/11 16:20, Hubert Kario wrote: I usually create subvolumes in btrfs root volume: /mnt/btrfs/ |- server-a |- server-b \- server-c then create snapshots of these directories: /mnt/btrfs/ |- server-a |- server-b |- server-c

Re: Synching a Backup Server

2011-01-09 Thread Fajar A. Nugraha
On Sun, Jan 9, 2011 at 6:46 PM, Alan Chandler a...@chandlerfamily.org.uk wrote: then create snapshots of these directories: /mnt/btrfs/         |- server-a         |- server-b         |- server-c         |- snapshots-server-a                  |- @GMT-2010.12.21-16.48.09                    

Re: Synching a Backup Server

2011-01-09 Thread Alan Chandler
On 09/01/11 13:54, Fajar A. Nugraha wrote: By default, when you do something like mount /dev/sdb1 /mnt/btrfs the default subvolume will be mounted under /mnt/btrfs. Snapshots and subvolumes will be visible as subdirectories under it, regardless whether it's in the root or several

Re: Synching a Backup Server

2011-01-09 Thread Freddie Cash
On Sun, Jan 9, 2011 at 7:32 AM, Alan Chandler a...@chandlerfamily.org.uk wrote: I think I start to get it now.  Its the fact that subvolumes can be snapshotted etc without mounting them that is the difference.  I guess I am too used to thinking like LVM and I was thinking subvolumes where like

Re: Synching a Backup Server

2011-01-09 Thread Alan Chandler
On 09/01/11 18:30, Hugo Mills wrote: No, subvolumes are a part of the whole filesystem. In btrfs, there is only one filesystem. There are 6 main B-trees that store metadata in btrfs (plus a couple of others). One of those is the filesystem tree (or FS tree), which contains all the

Re: Synching a Backup Server

2011-01-09 Thread Hugo Mills
On Sun, Jan 09, 2011 at 08:57:12PM +, Alan Chandler wrote: On 09/01/11 18:30, Hugo Mills wrote: No, subvolumes are a part of the whole filesystem. In btrfs, there is only one filesystem. There are 6 main B-trees that store metadata in btrfs (plus a couple of others). One of those is

Re: Synching a Backup Server

2011-01-09 Thread Alan Chandler
On 09/01/11 22:01, Hugo Mills wrote: I find the wiki also confusing because it talks about subvolumes having to be at the first level of the filesystem, but again further up this thread there is an example which is used for real of it not being at the first level, but at one level down inside

Re: Synching a Backup Server

2011-01-09 Thread Fajar A. Nugraha
On Mon, Jan 10, 2011 at 5:01 AM, Hugo Mills hugo-l...@carfax.org.uk wrote:   There is a root subvolume namespace (subvolid=0), which may contain files, directories, and other subvolumes. This root subvolume is what you see when you mount a newly-created btrfs filesystem. Is there a detailed

Re: Synching a Backup Server

2011-01-07 Thread Hubert Kario
On Friday, January 07, 2011 00:07:37 Carl Cook wrote: On Thu 06 January 2011 14:26:30 Carl Cook wrote: According To Doyle... Er, Hoyle... I am trying to create a multi-device BTRFS system using two identical drives. I want them to be raid 0 for no redunancy, and a total of 4TB. But in

Re: Synching a Backup Server

2011-01-07 Thread Hubert Kario
On Thursday, January 06, 2011 22:52:25 Freddie Cash wrote: On Thu, Jan 6, 2011 at 1:42 PM, Carl Cook cac...@quantum-sci.com wrote: On Thu 06 January 2011 11:16:49 Freddie Cash wrote: Also with this system, I'm concerned that if there is corruption on the HTPC, it could be propagated to

Synching a Backup Server

2011-01-06 Thread Carl Cook
I am setting up a backup server for the garage, to back up my HTPC in case of theft or fire. The HTPC has a 4TB RAID10 array (mdadm, JFS), and will be connected to the backup server using GB ethernet. The backup server will have a 4TB BTRFS RAID0 array. Debian Testing running on both. I

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 9:35 AM, Carl Cook cac...@quantum-sci.com wrote: I am setting up a backup server for the garage, to back up my HTPC in case of theft or fire.  The HTPC has a 4TB RAID10 array (mdadm, JFS), and will be connected to the backup server using GB ethernet.  The backup server

Re: Synching a Backup Server

2011-01-06 Thread Marcin Kuk
2011/1/6 Freddie Cash fjwc...@gmail.com: On Thu, Jan 6, 2011 at 9:35 AM, Carl Cook cac...@quantum-sci.com wrote: I am setting up a backup server for the garage, to back up my HTPC in case of theft or fire.  The HTPC has a 4TB RAID10 array (mdadm, JFS), and will be connected to the backup

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 11:33 AM, Marcin Kuk marcin@gmail.com wrote: Rsync is good, but not for all cases. Be aware of databases files - you should do snapshot filesystem before rsyncing. We script a dump of all databases before the rsync runs, so we get both text and binary backups. If

Re: Synching a Backup Server

2011-01-06 Thread C Anthony Risinger
On Thu, Jan 6, 2011 at 1:47 PM, Freddie Cash fjwc...@gmail.com wrote: On Thu, Jan 6, 2011 at 11:33 AM, Marcin Kuk marcin@gmail.com wrote: Rsync is good, but not for all cases. Be aware of databases files - you should do snapshot filesystem before rsyncing. We script a dump of all

Re: Synching a Backup Server

2011-01-06 Thread Fajar A. Nugraha
On Fri, Jan 7, 2011 at 12:35 AM, Carl Cook cac...@quantum-sci.com wrote: I want to keep a duplicate copy of the HTPC data, on the backup server Is there a BTRFS tool that would do this? AFAIK zfs is the only opensource filesystem today that can transfer block-level delta between two snapshots,

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 12:07 PM, C Anthony Risinger anth...@extof.me wrote: On Thu, Jan 6, 2011 at 1:47 PM, Freddie Cash fjwc...@gmail.com wrote: On Thu, Jan 6, 2011 at 11:33 AM, Marcin Kuk marcin@gmail.com wrote: Rsync is good, but not for all cases. Be aware of databases files - you

Re: Synching a Backup Server

2011-01-06 Thread C Anthony Risinger
On Thu, Jan 6, 2011 at 2:13 PM, Freddie Cash fjwc...@gmail.com wrote: On Thu, Jan 6, 2011 at 12:07 PM, C Anthony Risinger anth...@extof.me wrote: On Thu, Jan 6, 2011 at 1:47 PM, Freddie Cash fjwc...@gmail.com wrote: On Thu, Jan 6, 2011 at 11:33 AM, Marcin Kuk marcin@gmail.com wrote: Rsync

Re: Synching a Backup Server

2011-01-06 Thread Gordan Bobic
Unfortunately, we don't use btrfs or LVM on remote servers, so there's no snapshotting available during the backup run. In a perfect world, btrfs would be production-ready, ZFS would be available on Linux, and we'd no longer need the abomination called LVM. :) As a matter of fact, ZFS _IS_

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 1:06 PM, Gordan Bobic gor...@bobich.net wrote: Unfortunately, we don't use btrfs or LVM on remote servers, so there's no snapshotting available during the backup run.  In a perfect world, btrfs would be production-ready, ZFS would be available on Linux, and we'd no

Re: Synching a Backup Server

2011-01-06 Thread Carl Cook
On Thu 06 January 2011 11:16:49 Freddie Cash wrote: Just run rsync on the backup server, tell it to connect via ssh to the remote server, and rsync / (root filesystem) into /backups/htpc/ (or whatever directory you want). Use an exclude file to exclude the directories you don't want backed up

Re: Synching a Backup Server

2011-01-06 Thread Carl Cook
On Thu 06 January 2011 12:12:13 Fajar A. Nugraha wrote: With other filesystems, something like rsync + LVM snapshot is probably your best bet, and it doesn't really care what filesystem you use. I'm not running LVM though. Is this where the snapshotting ability comes from? -- To unsubscribe

Re: Synching a Backup Server

2011-01-06 Thread Carl Cook
On Thu 06 January 2011 12:07:17 C Anthony Risinger wrote: as for the DB stuff, you definitely need to snapshot _before_ rsync. roughly: ) read lock and flush tables ) snapshot ) unlock tables ) mount snapshot ) rsync from snapshot ie. the same as whats needed for LVM:

Re: Synching a Backup Server

2011-01-06 Thread Goffredo Baroncelli
On 01/06/2011 06:35 PM, Carl Cook wrote: I want to keep a duplicate copy of the HTPC data, on the backup server, and I think a regular full file copy is not optimal and may take days to do. So I'm looking for a way to sync the arrays at some interval. Ideally the sync would scan the HTPC

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 1:42 PM, Carl Cook cac...@quantum-sci.com wrote: On Thu 06 January 2011 11:16:49 Freddie Cash wrote: Also with this system, I'm concerned that if there is corruption on the HTPC, it could be propagated to the backup server.  Is there some way to address this?  

Re: Synching a Backup Server

2011-01-06 Thread Gordan Bobic
On 01/06/2011 09:44 PM, Carl Cook wrote: On Thu 06 January 2011 12:07:17 C Anthony Risinger wrote: as for the DB stuff, you definitely need to snapshot _before_ rsync. roughly: ) read lock and flush tables ) snapshot ) unlock tables ) mount snapshot ) rsync from snapshot ie. the same as

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 1:44 PM, Carl Cook cac...@quantum-sci.com wrote: On Thu 06 January 2011 12:07:17 C Anthony Risinger wrote: as for the DB stuff, you definitely need to snapshot _before_ rsync.   roughly: ) read lock and flush tables ) snapshot ) unlock tables ) mount snapshot )

Re: Synching a Backup Server

2011-01-06 Thread Carl Cook
On Thu 06 January 2011 13:58:41 Freddie Cash wrote: Simplest solution is to write a script to create a mysqldump of all databases into a directory, add that to cron so that it runs at the same time everyday, 10-15 minutes before the rsync run is done. That way, rsync to the backup server

Re: Synching a Backup Server

2011-01-06 Thread Gordan Bobic
On 01/06/2011 10:26 PM, Carl Cook wrote: On Thu 06 January 2011 13:58:41 Freddie Cash wrote: Simplest solution is to write a script to create a mysqldump of all databases into a directory, add that to cron so that it runs at the same time everyday, 10-15 minutes before the rsync run is done.

Re: Synching a Backup Server

2011-01-06 Thread Carl Cook
On Thu 06 January 2011 14:26:30 Carl Cook wrote: According To Doyle... Er, Hoyle... I am trying to create a multi-device BTRFS system using two identical drives. I want them to be raid 0 for no redunancy, and a total of 4TB. But in the wiki it says nothing about using fdisk to set up the drive

Re: Synching a Backup Server

2011-01-06 Thread Fajar A. Nugraha
On Fri, Jan 7, 2011 at 5:26 AM, Carl Cook cac...@quantum-sci.com wrote: On Thu 06 January 2011 13:58:41 Freddie Cash wrote: Simplest solution is to write a script to create a mysqldump of all databases into a directory, add that to cron so that it runs at the same time everyday, 10-15 minutes