On Tue, May 25, 2010 at 11:13:44AM -0700, Robin Lee Powell wrote:
> On Wed, May 19, 2010 at 11:14:48AM -0700, Robin Lee Powell wrote:
> > On Tue, May 18, 2010 at 09:30:43PM +0000, John Rouillard wrote:
> > > On Tue, May 18, 2010 at 02:04:46PM -0700, Robin Lee Powell
> > > wrote:
> > > > A customer we're backing up has a directory with ~500
> > > > subdirs and hundreds of GiB of data. We're using BackupPC
> > > > in rsync+ssh mode.
> > > >
> > > > As a first pass at breaking that up, I made a bunch of
> > > > seperate host entries like /A/*0, /A/*1, ... (all the dirs
> > > > have numeric names).
> > > >
> > > > That seems to select the right files, but it doesn't work
> > > > because BackupPC ends up running a bunch of them at once,
> > > > hammering that customer's machine.
> > >
> > > You can set it up so only a few of those hosts will run at the
> > > same time using
> > >
> > > $Conf{UserCmdCheckStatus} = 1;
> > >
> > > and a $Conf{DumpPreUserCmd}/$Conf{DumpPostUserCmd} that know
> > > the host names and implement a counting semaphore to make sure
> > > only some number of them are running at the same time. I
> > > posted a longer sketch of how I limit the number of parallel
> > > backups to remote sites in the archives some time ago.
> >
> > That's a fantastic idea! I don't even need to do anything
> > complicated; just use "lockfile /tmp/backuppc" OSLT, since I
> > only care about not overloading single hosts.
>
> There is a problem with it, though: it can lead to starvation.
> I've got one host broken up into 10 or so backuppc jobs, and one
> of them has yet to run, 10 days later. Each of the jobs takes
> about 6 hours, which means it takes more than a day for them all
> to complete, which means 5+ jobs are always available to run, and
> that one just hasn't gotten lucky yet.
>
> Haven't come up with a good solution yet.
I'm going to try increasing the IncrPeriod for that host to 4 days;
I'll see how it goes.
-Robin
--
http://singinst.org/ : Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
is "na nei". My personal page: http://www.digitalkingdom.org/rlp/
------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
