Re: Synching a Backup Server

2011-01-25 Thread Hubert Kario
On Tuesday, January 25, 2011 18:59:39 Freddie Cash wrote: > On Tue, Jan 25, 2011 at 9:43 AM, Hubert Kario wrote: > > Besides, I don't see *why* this should be done... > > > > And as far as I know ZFS doesn't support different reduncancy levels for > > different files residing in the same director

Re: Synching a Backup Server

2011-01-25 Thread Freddie Cash
On Tue, Jan 25, 2011 at 9:43 AM, Hubert Kario wrote: > Besides, I don't see *why* this should be done... > > And as far as I know ZFS doesn't support different reduncancy levels for > different files residing in the same directory. You can have > ~/1billion$-project.tar.gz with triple redundancy a

Re: Synching a Backup Server

2011-01-25 Thread Hubert Kario
On Tuesday, January 25, 2011 18:29:35 Kaspar Schleiser wrote: > On 01/22/2011 02:55 PM, Hubert Kario wrote: > >> It looks like ZFS, Btrfs, and LVM should work in similar manners, but > >> the overloaded terminology (pool, volume, sub-volume, filesystem are > >> different in all three) and new termi

Re: Synching a Backup Server

2011-01-25 Thread Kaspar Schleiser
On 01/22/2011 02:55 PM, Hubert Kario wrote: It looks like ZFS, Btrfs, and LVM should work in similar manners, but the overloaded terminology (pool, volume, sub-volume, filesystem are different in all three) and new terminology that's only in Btrfs is confusing. With btrfs you need to have *a* f

Re: Synching a Backup Server

2011-01-24 Thread Freddie Cash
On Sat, Jan 22, 2011 at 5:45 AM, Hugo Mills wrote: > On Fri, Jan 21, 2011 at 11:28:19AM -0800, Freddie Cash wrote: >> So, is Btrfs pooled storage or not?  Do you throw 24 disks into a >> single Btrfs filesystem, and then split that up into separate >> sub-volumes as needed? > >   Yes, except that

Re: Synching a Backup Server

2011-01-22 Thread Hubert Kario
On Friday 21 of January 2011 20:28:19 Freddie Cash wrote: > On Sun, Jan 9, 2011 at 10:30 AM, Hugo Mills wrote: > > On Sun, Jan 09, 2011 at 09:59:46AM -0800, Freddie Cash wrote: > >> Let see if I can match up the terminology and layers a bit: > >> > >> LVM Physical Volume == Btrfs disk == ZFS disk

Re: Synching a Backup Server

2011-01-22 Thread Hugo Mills
On Fri, Jan 21, 2011 at 11:28:19AM -0800, Freddie Cash wrote: > On Sun, Jan 9, 2011 at 10:30 AM, Hugo Mills wrote: > > On Sun, Jan 09, 2011 at 09:59:46AM -0800, Freddie Cash wrote: > >> Let see if I can match up the terminology and layers a bit: > >> > >> LVM Physical Volume == Btrfs disk == ZFS d

Re: Synching a Backup Server

2011-01-21 Thread Freddie Cash
On Sun, Jan 9, 2011 at 10:30 AM, Hugo Mills wrote: > On Sun, Jan 09, 2011 at 09:59:46AM -0800, Freddie Cash wrote: >> Let see if I can match up the terminology and layers a bit: >> >> LVM Physical Volume == Btrfs disk == ZFS disk / vdevs >> LVM Volume Group == Btrfs "filesystem" == ZFS storage poo

Re: Synching a Backup Server

2011-01-11 Thread Hugo Mills
On Mon, Jan 10, 2011 at 09:22:49AM +0700, Fajar A. Nugraha wrote: > On Mon, Jan 10, 2011 at 5:01 AM, Hugo Mills wrote: > >   There is a root subvolume namespace (subvolid=0), which may contain > > files, directories, and other subvolumes. This root subvolume is what > > you see when you mount a ne

Re: Synching a Backup Server

2011-01-11 Thread Hugo Mills
On Sun, Jan 09, 2011 at 11:32:20PM +, Alan Chandler wrote: > On 09/01/11 22:01, Hugo Mills wrote: > >> I find the wiki > >>also confusing because it talks about subvolumes having to be at the > >>first level of the filesystem, but again further up this thread > >>there is an example which is u

Re: Synching a Backup Server

2011-01-10 Thread Hubert Kario
On Sunday 09 of January 2011 12:46:59 Alan Chandler wrote: > On 07/01/11 16:20, Hubert Kario wrote: > > I usually create subvolumes in btrfs root volume: > > > > /mnt/btrfs/ > > > > |- server-a > > |- server-b > > > > \- server-c > > > > then create snapshots

Re: Synching a Backup Server

2011-01-09 Thread Fajar A. Nugraha
On Mon, Jan 10, 2011 at 5:01 AM, Hugo Mills wrote: >   There is a root subvolume namespace (subvolid=0), which may contain > files, directories, and other subvolumes. This root subvolume is what > you see when you mount a newly-created btrfs filesystem. Is there a detailed explanation in the wiki

Re: Synching a Backup Server

2011-01-09 Thread Alan Chandler
On 09/01/11 22:01, Hugo Mills wrote: I find the wiki also confusing because it talks about subvolumes having to be at the first level of the filesystem, but again further up this thread there is an example which is used for real of it not being at the first level, but at one level down inside a

Re: Synching a Backup Server

2011-01-09 Thread Hugo Mills
On Sun, Jan 09, 2011 at 08:57:12PM +, Alan Chandler wrote: > On 09/01/11 18:30, Hugo Mills wrote: > > > >No, subvolumes are a part of the whole filesystem. In btrfs, there > >is only one filesystem. There are 6 main B-trees that store metadata > >in btrfs (plus a couple of others). One of t

Re: Synching a Backup Server

2011-01-09 Thread Alan Chandler
On 09/01/11 18:30, Hugo Mills wrote: No, subvolumes are a part of the whole filesystem. In btrfs, there is only one filesystem. There are 6 main B-trees that store metadata in btrfs (plus a couple of others). One of those is the "filesystem tree" (or FS tree), which contains all the metada

Re: Synching a Backup Server

2011-01-09 Thread Freddie Cash
On Sun, Jan 9, 2011 at 7:32 AM, Alan Chandler wrote: > I think I start to get it now.  Its the fact that subvolumes can be > snapshotted etc without mounting them that is the difference.  I guess I am > too used to thinking like LVM and I was thinking subvolumes where like an > LV.  They are, but

Re: Synching a Backup Server

2011-01-09 Thread Alan Chandler
On 09/01/11 13:54, Fajar A. Nugraha wrote: By default, when you do something like mount /dev/sdb1 /mnt/btrfs the default subvolume will be mounted under /mnt/btrfs. Snapshots and subvolumes will be visible as subdirectories under it, regardless whether it's in the root or several directories

Re: Synching a Backup Server

2011-01-09 Thread Fajar A. Nugraha
On Sun, Jan 9, 2011 at 6:46 PM, Alan Chandler wrote: >> then create snapshots of these directories: >> >> /mnt/btrfs/ >>         |- server-a >>         |- server-b >>         |- server-c >>         |- snapshots-server-a >>                  |- @GMT-2010.12.21-16.48.09 >>                    \- @GMT-

Re: Synching a Backup Server

2011-01-09 Thread Alan Chandler
On 07/01/11 16:20, Hubert Kario wrote: I usually create subvolumes in btrfs root volume: /mnt/btrfs/ |- server-a |- server-b \- server-c then create snapshots of these directories: /mnt/btrfs/ |- server-a |- server-b |- server-c |

Re: Synching a Backup Server

2011-01-07 Thread Hubert Kario
On Thursday, January 06, 2011 22:52:25 Freddie Cash wrote: > On Thu, Jan 6, 2011 at 1:42 PM, Carl Cook wrote: > > On Thu 06 January 2011 11:16:49 Freddie Cash wrote: > >> > Also with this system, I'm concerned that if there is corruption on > >> > the HTPC, it could be propagated to the backup ser

Re: Synching a Backup Server

2011-01-07 Thread Hubert Kario
On Friday, January 07, 2011 00:07:37 Carl Cook wrote: > On Thu 06 January 2011 14:26:30 Carl Cook wrote: > > According To Doyle... > > Er, Hoyle... > > I am trying to create a multi-device BTRFS system using two identical > drives. I want them to be raid 0 for no redunancy, and a total of 4TB. >

Re: Synching a Backup Server

2011-01-06 Thread Fajar A. Nugraha
On Fri, Jan 7, 2011 at 5:26 AM, Carl Cook wrote: > On Thu 06 January 2011 13:58:41 Freddie Cash wrote: >> Simplest solution is to write a script to create a mysqldump of all >> databases into a directory, add that to cron so that it runs at the >> same time everyday, 10-15 minutes before the rsync

Re: Synching a Backup Server

2011-01-06 Thread Carl Cook
On Thu 06 January 2011 14:26:30 Carl Cook wrote: > According To Doyle... Er, Hoyle... I am trying to create a multi-device BTRFS system using two identical drives. I want them to be raid 0 for no redunancy, and a total of 4TB. But in the wiki it says nothing about using fdisk to set up the drive

Re: Synching a Backup Server

2011-01-06 Thread Gordan Bobic
On 01/06/2011 10:26 PM, Carl Cook wrote: On Thu 06 January 2011 13:58:41 Freddie Cash wrote: Simplest solution is to write a script to create a mysqldump of all databases into a directory, add that to cron so that it runs at the same time everyday, 10-15 minutes before the rsync run is done. Th

Re: Synching a Backup Server

2011-01-06 Thread Carl Cook
On Thu 06 January 2011 13:58:41 Freddie Cash wrote: > Simplest solution is to write a script to create a mysqldump of all > databases into a directory, add that to cron so that it runs at the > same time everyday, 10-15 minutes before the rsync run is done. That > way, rsync to the backup server p

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 1:44 PM, Carl Cook wrote: > On Thu 06 January 2011 12:07:17 C Anthony Risinger wrote: >> as for the DB stuff, you definitely need to snapshot _before_ rsync.   >> roughly: >> >> ) read lock and flush tables >> ) snapshot >> ) unlock tables >> ) mount snapshot >> ) rsync from

Re: Synching a Backup Server

2011-01-06 Thread Gordan Bobic
On 01/06/2011 09:44 PM, Carl Cook wrote: On Thu 06 January 2011 12:07:17 C Anthony Risinger wrote: as for the DB stuff, you definitely need to snapshot _before_ rsync. roughly: ) read lock and flush tables ) snapshot ) unlock tables ) mount snapshot ) rsync from snapshot ie. the same as whats

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 1:42 PM, Carl Cook wrote: > On Thu 06 January 2011 11:16:49 Freddie Cash wrote: >> > Also with this system, I'm concerned that if there is corruption on the >> > HTPC, it could be propagated to the backup server.  Is there some way to >> > address this?  Longer intervals t

Re: Synching a Backup Server

2011-01-06 Thread Goffredo Baroncelli
On 01/06/2011 06:35 PM, Carl Cook wrote: > I want to keep a duplicate copy of the HTPC data, on the backup > server, and I think a regular full file copy is not optimal and may > take days to do. So I'm looking for a way to sync the arrays at some > interval. Ideally the sync would scan the HTPC

Re: Synching a Backup Server

2011-01-06 Thread Carl Cook
On Thu 06 January 2011 12:07:17 C Anthony Risinger wrote: > as for the DB stuff, you definitely need to snapshot _before_ rsync. roughly: > > ) read lock and flush tables > ) snapshot > ) unlock tables > ) mount snapshot > ) rsync from snapshot > > ie. the same as whats needed for LVM: > > http

Re: Synching a Backup Server

2011-01-06 Thread Carl Cook
On Thu 06 January 2011 12:12:13 Fajar A. Nugraha wrote: > With other filesystems, something like rsync + LVM snapshot is > probably your best bet, and it doesn't really care what filesystem you > use. I'm not running LVM though. Is this where the snapshotting ability comes from? -- To unsubscri

Re: Synching a Backup Server

2011-01-06 Thread Carl Cook
On Thu 06 January 2011 11:16:49 Freddie Cash wrote: > Just run rsync on the backup server, tell it to connect via ssh to the > remote server, and rsync / (root filesystem) into /backups/htpc/ (or > whatever directory you want). Use an exclude file to exclude the > directories you don't want backed

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 1:06 PM, Gordan Bobic wrote: >>> Unfortunately, we don't use btrfs or LVM on remote servers, so there's >>> no snapshotting available during the backup run.  In a perfect world, >>> btrfs would be production-ready, ZFS would be available on Linux, and >>> we'd no longer need

Re: Synching a Backup Server

2011-01-06 Thread Gordan Bobic
Unfortunately, we don't use btrfs or LVM on remote servers, so there's no snapshotting available during the backup run. In a perfect world, btrfs would be production-ready, ZFS would be available on Linux, and we'd no longer need the abomination called LVM. :) As a matter of fact, ZFS _IS_ ava

Re: Synching a Backup Server

2011-01-06 Thread C Anthony Risinger
On Thu, Jan 6, 2011 at 2:13 PM, Freddie Cash wrote: > On Thu, Jan 6, 2011 at 12:07 PM, C Anthony Risinger wrote: >> On Thu, Jan 6, 2011 at 1:47 PM, Freddie Cash wrote: >>> On Thu, Jan 6, 2011 at 11:33 AM, Marcin Kuk wrote: Rsync is good, but not for all cases. Be aware of databases files -

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 12:07 PM, C Anthony Risinger wrote: > On Thu, Jan 6, 2011 at 1:47 PM, Freddie Cash wrote: >> On Thu, Jan 6, 2011 at 11:33 AM, Marcin Kuk wrote: >>> Rsync is good, but not for all cases. Be aware of databases files - >>> you should do snapshot filesystem before rsyncing. >>

Re: Synching a Backup Server

2011-01-06 Thread Fajar A. Nugraha
On Fri, Jan 7, 2011 at 12:35 AM, Carl Cook wrote: > I want to keep a duplicate copy of the HTPC data, on the backup server > Is there a BTRFS tool that would do this? AFAIK zfs is the only opensource filesystem today that can transfer block-level delta between two snapshots, making it ideal for

Re: Synching a Backup Server

2011-01-06 Thread C Anthony Risinger
On Thu, Jan 6, 2011 at 1:47 PM, Freddie Cash wrote: > On Thu, Jan 6, 2011 at 11:33 AM, Marcin Kuk wrote: >> Rsync is good, but not for all cases. Be aware of databases files - >> you should do snapshot filesystem before rsyncing. > > We script a dump of all databases before the rsync runs, so we

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 11:33 AM, Marcin Kuk wrote: > Rsync is good, but not for all cases. Be aware of databases files - > you should do snapshot filesystem before rsyncing. We script a dump of all databases before the rsync runs, so we get both text and binary backups. If restoring the binary f

Re: Synching a Backup Server

2011-01-06 Thread Marcin Kuk
2011/1/6 Freddie Cash : > On Thu, Jan 6, 2011 at 9:35 AM, Carl Cook wrote: >> >> I am setting up a backup server for the garage, to back up my HTPC in case >> of theft or fire.  The HTPC has a 4TB RAID10 array (mdadm, JFS), and will be >> connected to the backup server using GB ethernet.  The ba

Re: Synching a Backup Server

2011-01-06 Thread Freddie Cash
On Thu, Jan 6, 2011 at 9:35 AM, Carl Cook wrote: > > I am setting up a backup server for the garage, to back up my HTPC in case of > theft or fire.  The HTPC has a 4TB RAID10 array (mdadm, JFS), and will be > connected to the backup server using GB ethernet.  The backup server will > have a 4TB