Re: [BackupPC-users] large archives and scalability issues

2008-02-27 Thread Les Mikesell
dan wrote: > sure would be nice to have native ZFS on linux. :( for this reason > i'm considering nexenta and backuppc, as well as a faster network stack in > the solaris kernel or freebsd 'cause freebsd rocks! Your timing is right - freebsd 7.0 was just released. -- Les Mikesell [EMAIL

Re: [BackupPC-users] Hardware upgrade advice

2008-02-27 Thread dan
RAM RAM RAM! 512MB of ram is definitely your issue here. swapping causes more periods of head movement which means less periods of head reads/writes on your disk. With your specs, moving to 1GB+ of ram will be huge. that Celeron processor is probably your next big issue specifically because it

Re: [BackupPC-users] large archives and scalability issues

2008-02-27 Thread dan
just a comment, It sure would be nice if LVM supported a raid5 or raidz style raid within the volume manager so we didn't have to put LVM on top of md*. sure would be nice to have native ZFS on linux. :( for this reason i'm considering nexenta and backuppc, as well as a faster network stack in th

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread dan
the $TopDIR must be on the same volume as the backup files. so if $TopDIR is /var/lib/backuppc and /var is a 500GB drive then your pc directory needs to be somewhere in /var. What I do and what is common to do is to mount up the target drive somewhere temporarily and `cp -Rp /var/lib/backuppc /ta

Re: [BackupPC-users] Hardware upgrade advice

2008-02-27 Thread Stephen Joyce
On Wed, 27 Feb 2008, John Pettitt wrote: > So my take - if your box is swapping that's the #1 upgrade because that > will kill any server performance and memory is cheap. Next I'd look at > disk, with the right controller more spindles will give you a > performance boost however raid 5 is not a

Re: [BackupPC-users] Hardware upgrade advice

2008-02-27 Thread John Pettitt
Nils Breunese (Lemonbit) wrote: > Hello all, > > We're running a BackupPC 3.1.0 installation on CentOS 4 32-bit on a > machine with the following specs: > > - Intel Celeron CPU 2.66 GHz > - 512 MB RAM > - BackupPC pool on a single 250 GB ATA 133 drive > > We currently running one backup at a time

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Nils Breunese (Lemonbit)
Kimball Larsen wrote: > Ok, I finally found and tracked down my problem. > > I'm using BackupPC on Ubuntu 7.10, which installs BackupPC 3.0.0. > Turns out, changing the TopDir location does NOT change the location > of the pool files - they are hardcoded to live in /var/lib/backuppc. > > Acco

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Kimball Larsen
Ok, I finally found and tracked down my problem. I'm using BackupPC on Ubuntu 7.10, which installs BackupPC 3.0.0. Turns out, changing the TopDir location does NOT change the location of the pool files - they are hardcoded to live in /var/lib/backuppc. According to here: http://backuppc.wi

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Les Mikesell
Kimball Larsen wrote: > Ok, I found the problem. Looks like my cpool is in the wrong place, > apparently: > > ln: creating hard link `/var/lib/backuppc/cpool/b/7/6/ > b763efe12301fe7645e82fb46d53b4e3' to `/mnt/plump/backuppc/pc/ > 192.168.0.8/10/attrib': Invalid cross-device link > > It look

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Kimball Larsen
Ok, I freely admit that I'm in over my head here... I am NOT a perl programmer. However, here is the method that is puking when trying to create the links: sub MakeFileLink { my($bpc, $name, $d, $newFile, $compress) = @_; my($i, $rawFile); return -1 if ( !-f $name ); for ( $

Re: [BackupPC-users] large archives and scalability issues

2008-02-27 Thread David Rees
On Wed, Feb 27, 2008 at 2:54 AM, Tomasz Chmielewski <[EMAIL PROTECTED]> wrote: > Stripe size is 64k. > Also, the system was made with just "mkfs.ext3 -j /dev/sdX", so without > the stride option (or other useful options, like online resizing, which > is enabled by default only in the recent rel

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Kimball Larsen
Ok, I found the problem. Looks like my cpool is in the wrong place, apparently: ln: creating hard link `/var/lib/backuppc/cpool/b/7/6/ b763efe12301fe7645e82fb46d53b4e3' to `/mnt/plump/backuppc/pc/ 192.168.0.8/10/attrib': Invalid cross-device link It looks like my cpool is in /var/lib/backupp

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Les Mikesell
Kimball Larsen wrote: > >>> Pool is used if you don't have compression enabled, cpool if you do. >>> Whichever is used must be on the same filesystem as the pc directory >>> tree so that hardlinks work between them. If you are getting errors >>> making the links, check that you have space (df)

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Kimball Larsen
On Feb 27, 2008, at 1:36 PM, Kimball Larsen wrote: On Feb 27, 2008, at 1:33 PM, Les Mikesell wrote: Kimball Larsen wrote: Actually, I may have answered my own question somewhat: It appears (based on my reading of a message here: http://www.mail-archive.com/backuppc-users@lists.sourceforge.ne

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Kimball Larsen
On Feb 27, 2008, at 1:33 PM, Les Mikesell wrote: Kimball Larsen wrote: Actually, I may have answered my own question somewhat: It appears (based on my reading of a message here: http://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg09017.html ) that my pool is hosed. Specifically,

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Les Mikesell
Kimball Larsen wrote: > Actually, I may have answered my own question somewhat: > > It appears (based on my reading of a message here: > http://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg09017.html > ) that my pool is hosed. > > Specifically, my cpool directory has several gigs

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Kimball Larsen
Actually, I may have answered my own question somewhat: It appears (based on my reading of a message here: http://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg09017.html ) that my pool is hosed. Specifically, my cpool directory has several gigs of data in it, but my "pool" direc

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Kimball Larsen
On Feb 27, 2008, at 1:11 PM, Les Mikesell wrote: > Kimball Larsen wrote: I notice that the cpool directory is owned by root. Should it be owned by the backuppc user instead? >>> Yes, it must be writable by the backuppc user. Your logs are >>> probably full of "unable to link" e

Re: [BackupPC-users] Backup keeps failing for one machine

2008-02-27 Thread Nils Breunese (Lemonbit)
Nils Breunese (Lemonbit) wrote: > I'm backing up a bunch of machine using BackupPC 3.1.0. Everything has > been working fine for a long time, but since a couple of days backups > for one machine have been failing. The XferLOG for the last partial > backup has this error message: > > Unable

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Tomasz Chmielewski
Kimball Larsen schrieb: > On Feb 27, 2008, at 11:49 AM, Les Mikesell wrote: >> Kimball Larsen wrote: >>> I've recently had to move my backuppc filesystem from one machine >>> to another. My machines are all backing up just fine, I'm able to >>> see all my old backups and restore files from t

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Les Mikesell
Kimball Larsen wrote: > >>> I notice that the cpool directory is owned by root. Should it be >>> owned by the backuppc user instead? >> Yes, it must be writable by the backuppc user. Your logs are >> probably full of "unable to link" errors. >> > > Yeah, I've been getting stuff like this:

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Kimball Larsen
On Feb 27, 2008, at 11:49 AM, Les Mikesell wrote: > Kimball Larsen wrote: >> I've recently had to move my backuppc filesystem from one machine >> to another. My machines are all backing up just fine, I'm able to >> see all my old backups and restore files from them with no >> problem.. h

Re: [BackupPC-users] Hardware upgrade advice

2008-02-27 Thread Les Mikesell
Ski Kacoroski wrote: > iowait means disk wait problems. Either move to a 3ware raid card or > to a scsi drive to improve matters. It generally means you are waiting for head motion. The controller isn't going to matter and faster drives only make a little bit of difference. -- Les Mikesel

Re: [BackupPC-users] Hardware upgrade advice

2008-02-27 Thread Les Mikesell
Nils Breunese (Lemonbit) wrote: > Hello all, > > We're running a BackupPC 3.1.0 installation on CentOS 4 32-bit on a > machine with the following specs: > > - Intel Celeron CPU 2.66 GHz > - 512 MB RAM > - BackupPC pool on a single 250 GB ATA 133 drive > > We currently running one backup at a t

Re: [BackupPC-users] Hardware upgrade advice

2008-02-27 Thread Carl Wilhelm Soderstrom
On 02/27 07:47 , Nils Breunese (Lemonbit) wrote: > We currently running one backup at a time ($Conf{MaxBackups} = 1;). > This already maxes out the iowait%. The machine is also swapping > sometimes. If we'd want to do more backups simultaneously, how would > you prioritize the following possi

Re: [BackupPC-users] Hardware upgrade advice

2008-02-27 Thread Ski Kacoroski
iowait means disk wait problems. Either move to a 3ware raid card or to a scsi drive to improve matters. cheers, ski On Wed, 27 Feb 2008 19:47:45 +0100 "Nils Breunese (Lemonbit)" <[EMAIL PROTECTED]> wrote: > Hello all, > > We're running a BackupPC 3.1.0 installation on CentOS 4 32-bit on a

[BackupPC-users] Hardware upgrade advice

2008-02-27 Thread Nils Breunese (Lemonbit)
Hello all, We're running a BackupPC 3.1.0 installation on CentOS 4 32-bit on a machine with the following specs: - Intel Celeron CPU 2.66 GHz - 512 MB RAM - BackupPC pool on a single 250 GB ATA 133 drive We currently running one backup at a time ($Conf{MaxBackups} = 1;). This already maxes o

Re: [BackupPC-users] Pool Size is Zero

2008-02-27 Thread Les Mikesell
Kimball Larsen wrote: > I've recently had to move my backuppc filesystem from one machine to > another. My machines are all backing up just fine, I'm able to see > all my old backups and restore files from them with no problem.. > however, my pool size is reporting as 0MB. I checked the fil

[BackupPC-users] Pool Size is Zero

2008-02-27 Thread Kimball Larsen
I've recently had to move my backuppc filesystem from one machine to another. My machines are all backing up just fine, I'm able to see all my old backups and restore files from them with no problem.. however, my pool size is reporting as 0MB. I checked the filesystem, and the size of the

Re: [BackupPC-users] Transforming a filled incr backup into a full backup

2008-02-27 Thread dan
On Wed, Feb 27, 2008 at 8:15 AM, John Rouillard <[EMAIL PROTECTED]> wrote: > On Tue, Feb 26, 2008 at 11:51:12AM -0600, Brad Triem wrote: > > I'd like to bring this topic back into discussion. I'm trying to keep a > > single daily full. To also save on bandwidth, I run a full, and then I > > run

Re: [BackupPC-users] Transforming a filled incr backup into a full backup

2008-02-27 Thread John Rouillard
On Tue, Feb 26, 2008 at 11:51:12AM -0600, Brad Triem wrote: > I'd like to bring this topic back into discussion. I'm trying to keep a > single daily full. To also save on bandwidth, I run a full, and then I > run a daily incremental. I'd like to just merge each days incremental > into the full.

Re: [BackupPC-users] large archives and scalability issues

2008-02-27 Thread Les Mikesell
Tomasz Chmielewski wrote: >>> I indeed use iSCSI. >>> The storage is on the SAN, and it's accessible via iSCSI. >>> BackupPC itself runs as a Xen guest. >> Is there some point to splitting the box where backuppc runs and its >> storage? I'd expect it to be faster/cheaper to shove some big SATA

[BackupPC-users] Backup keeps failing for one machine

2008-02-27 Thread Nils Breunese (Lemonbit)
Hello, I'm backing up a bunch of machine using BackupPC 3.1.0. Everything has been working fine for a long time, but since a couple of days backups for one machine have been failing. The XferLOG for the last partial backup has this error message: Unable to open /mnt/backup/backuppc

Re: [BackupPC-users] large archives and scalability issues

2008-02-27 Thread Tomasz Chmielewski
David Rees schrieb: > On Tue, Feb 26, 2008 at 4:39 PM, David Rees <[EMAIL PROTECTED]> wrote: >> So there you go. IMO, unless you are willing to overhaul your storage >> system or slightly increase the risk of data corruption (IMO, >> data=writeback instead of the default data=ordered should be a

Re: [BackupPC-users] large archives and scalability issues

2008-02-27 Thread Tomasz Chmielewski
David Rees schrieb: > On Tue, Feb 26, 2008 at 2:23 PM, Tomasz Chmielewski <[EMAIL PROTECTED]> wrote: >> > Can you give us more details on your disk array? Controller, disks, >> > RAID layout, ext3 fs creation options, etc... >> >> I said some of that already - but here are some missing parts. >>

Re: [BackupPC-users] large archives and scalability issues

2008-02-27 Thread Tomasz Chmielewski
Adam Goryachev schrieb: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Tomasz Chmielewski wrote: > | Les Mikesell schrieb: > |> Tomasz Chmielewski wrote: > |>> Although - with IO::Dirent "wa" is now 100% almost all the time, and > |>> the system feels much slower. Hm. Let's hope it's coinc

Re: [BackupPC-users] large archives and scalability issues

2008-02-27 Thread Tomasz Chmielewski
Les Mikesell schrieb: > Tomasz Chmielewski wrote: >> >> I indeed use iSCSI. >> The storage is on the SAN, and it's accessible via iSCSI. >> BackupPC itself runs as a Xen guest. > > Is there some point to splitting the box where backuppc runs and its > storage? I'd expect it to be faster/cheaper