Re: [BackupPC-users] How do I use an external USB drive as backup target?
Note excuse my top posting since my comments reference the entire thread here... John, may I politely suggest that you will get more help and more sympathy if you focus your time asking precise, relevant, and well-formed questions rather than repeatedly criticizing the documentation, rehashing FAQ's, and asking broad hypotheticals that are not relevant to most people's conception of backups. This is an open source project. If you think the documentation or anything else is lacking, you are more than welcome to contribute enhancements. John Hudak wrote at about 14:32:51 -0500 on Sunday, February 14, 2010: > my comments are interspersed below... > > On Sat, Feb 13, 2010 at 6:40 PM, Jeffrey J. Kosowsky > wrote: > > > John Hudak wrote at about 12:06:29 -0500 on Saturday, February 13, 2010: > > > Hello: > > > I am considering using an external USB drive as the storage for my > > backups. > > > I am running backup pc under Debian 5.0. > > External USB drives are a *BAD* idea for multiple reasons: > > - Slow > > - Unreliable > > - Subject to being disconnected > > etc. > > > > Yes, I know 1, and 2 is 'it depends', and 3 is exactly the reason why I > want to use usb drives > > > > > Part 1 > > > What do I need to do to configure the USB disk as the target? (e.g. how > > do I > > > do it?) > > > The USB disk is currently formatted as a NTFS file system. Do I *need* > > to > > > reformat it to ext3? or other? > > - NTFS is not usually used - need to check whether it supports the > > types of hard links required for BackupPC > > > > > > > > Part 2 > > > Assume I am crazy paranoid about preserving backup data and I get a > > second > > > USB drive to serve as a backup to the first USB drive. > > > Also assume that I am not concerned about the bandwidth across the > > network > > > or the various buses. > > > > > > >From a data reliability standpoint, is it better to run a backup > > session to > > > USB drive 1, and then repeat the backup to USB drive 2? OR > > > run a backup session to USB drive 1, and then copy the backup > > directories to > > > USB drive 2??? > > Look at the archives and FAQ - this has been discussed *many* times so > > no point in wasting peoples time in rehashing. > > > > I did a quick search of the archives before asking - I did not find a > definitive answer... > > > > > The first approach could have errors in different backed up files on > > disk 1 > > > or 2 but given the odds, very unlikely that the same exact error would > > show > > > up > > > in the same exact way in the same file across both USB disks. > > > OTOH, the second approach would allow the exact error in the backup on > > USB > > > disk 1 to be copied to USB disk 2. > > > > > > I am leaning towards repeating the backup on two drives. > > > > > > My understanding is that files that are backed up (using either rsync or > > > smb) are 'encrypted' (for lack of a better word), and to view them I > > need to > > > use zcat.-True? > > > > There is a better word -- *compressed* > > > So that is the word that is not clearly used in the documentation. There > are many ways that backups can be manipulated: stored in a completely > nonstandard/proprietary file system and protocol such as z-san, they can be > encrypted, and they can be compressed. The backup PC doc talks about using > compression, but does not state if any compression is used in the default > configuration. Compression is often configuration parameter. It does not > make sense to compress many audio and video formats. If the data to be > backed up consists predominantly of these types of files, then it makes no > sense to waste CPU cycles applying compression to get < 5% compression. > > > > > Also, can the backup profile be specified to perform complete data > > copies > > > periodically, as opposed to a baseline and then periodic incrementals? > > > > Read the documentation and FAQ. > > > I have read the doc, (where IMHO) this should have been clearly stated. > Instead the doc frequently introduces a topic with 1-2 sentences, then goes > off on a tangent for 1-3 paragraphs about how things were done in a previous > verson (completely irrevelant), or talks about what will be comming (again, > irrelevant), or points one to another section of the doc, in the middle of > some other thread that is related to the topic but does not address the > topic at hand, or, introduces a topic, then talks about 3-4 other ways to > accomplish the same thing, without telling the reader exactly how to do the > initial topic to begin with. So my fault...I need to also read FAQs. > > > > > > Lastly, does anyone have a statistical number that represents the > > > probability of a backup file (e.g. on the target backup disk) containing > > an > > > error introduced > > > by the backup procedure? I know there are error probabilities for
Re: [BackupPC-users] How do I use an external USB drive as backup target?
John Hudak wrote:
> External USB drives are a *BAD* idea for multiple reasons:
> - Slow
> - Unreliable
> - Subject to being disconnected
> etc.
>
> Yes, I know 1, and 2 is 'it depends', and 3 is exactly the reason why I
> want to use usb drives
Note that there are other ways to disconnect drives that don't share the USB
disadvantages, like hot-swap SATA enclosures, or ESATA connections.
> > >From a data reliability standpoint, is it better to run a backup
> session to
> > USB drive 1, and then repeat the backup to USB drive 2? OR
> > run a backup session to USB drive 1, and then copy the backup
> directories to
> > USB drive 2???
> Look at the archives and FAQ - this has been discussed *many* times so
> no point in wasting peoples time in rehashing.
>
> I did a quick search of the archives before asking - I did not find a
> definitive answer...
I doubt if there is a definitive answer. It will depend on the scale and your
circumstances. I'm more comfortable with an always-mirrored internal pair plus
offsite copies.
> > My understanding is that files that are backed up (using either
> rsync or
> > smb) are 'encrypted' (for lack of a better word), and to view
> them I need to
> > use zcat.-True?
>
> There is a better word -- *compressed*
>
> So that is the word that is not clearly used in the documentation.
Filename mangling isn't optional, compression is, as
http://backuppc.sourceforge.net/faq/BackupPC.html#configuration_parameters
should make clear. See the entry for $Conf{CompressLevel}.
> There are many ways that backups can be manipulated: stored in a
> completely nonstandard/proprietary file system and protocol such as
> z-san, they can be encrypted, and they can be compressed. The backup PC
> doc talks about using compression, but does not state if any compression
> is used in the default configuration.
??? The defaults are shown, but with a packaged version you need to check to
see
if the packager made his own changes.
> Compression is often
> configuration parameter. It does not make sense to compress many audio
> and video formats. If the data to be backed up consists predominantly
> of these types of files, then it makes no sense to waste CPU cycles
> applying compression to get < 5% compression.
Note that backuppc doesn't compress/uncompress every time you touch a file so
the tradeoffs may be different than you expect. Once the initial version of a
file has been compressed in the pool, matching copies from the same or other
backups won't have the operation repeated.
>
> > Also, can the backup profile be specified to perform complete
> data copies
> > periodically, as opposed to a baseline and then periodic
> incrementals?
>
> Read the documentation and FAQ.
>
> I have read the doc, (where IMHO) this should have been clearly stated.
> Instead the doc frequently introduces a topic with 1-2 sentences,
I'm not sure what you are reading here, but set the first 3 variables under the
'What to backup and when to do it' section if you don't like the default
once-a-week full:
$Conf{FullPeriod} = 6.97;
$Conf{IncrPeriod} = 0.97;
$Conf{FullKeepCnt} = 1;
The rest are for special cases that you can figure out if you need them.
> then
> goes off on a tangent for 1-3 paragraphs about how things were done in a
> previous verson (completely irrevelant),
Irrelevant to you, perhaps. Not so for people who understood the importance of
backups a few years ago and now are upgrading.
> or talks about what will be
> comming (again, irrelevant), or points one to another section of the
> doc, in the middle of some other thread that is related to the topic but
> does not address the topic at hand, or, introduces a topic, then talks
> about 3-4 other ways to accomplish the same thing, without telling the
> reader exactly how to do the initial topic to begin with. So my
> fault...I need to also read FAQs.
Just look at what the config options let you specify.
> The probability is either 0 if no bugs in the software (or your
> configuration of it) or 100% if bugs in the software and your dataset
> triggers the bug. Your question is not very well-framed and pretty
> meaningless. I suggest you learn a bit more about backup in general
> and backuppc in particular. There is a lot of good documentation on
> BackupPC in the Wikki and in the archives, I suggest you reference it...
>
>
> Well, in the extremely simplistic and ideal case, it is 0 or 1. In the
> real world, where algorithms are badly designed, or implemented, or
> both, then problems arise due to things not originally considered in the
> development which cause errors to arise in the backup procedure that is
> not about the conversion of 0/1's in computer memory or the transferring
> of them to tape, disk, etc.
It is still 0 or 1, and if you knew what was wrong with a proces
Re: [BackupPC-users] Odd 'unexpected repeated share name error'
John writes:
> The reason for the current layout is to allow the existing critical
> backups to complete even if it takes days to backup the newer shares
> (filesystems). So it's a temporary thing, but I could see somebody
> wanting to put /home first to prioritize it's backup over say /.
> Is the bug going to be fixed for the beta3 or final release?
Yes.
> Also on another topic, is the elimination of zombie processes on the
> todo list for the 3.2 release?
Yes.
If you want to try the two fixes, here's a patch against 3.2.0beta1
(note: I haven't tested this yet - just did this sitting on a plane).
Craig
--- bin/BackupPC_dump 2010-01-24 17:30:43.0 -0800
+++ bin/BackupPC_dump 2010-02-14 12:02:14.859375000 -0800
@@ -623,6 +623,7 @@
#
# Now backup each of the shares
#
+my $shareDuplicate = {};
for my $shareName ( @$ShareNames ) {
local(*RH, *WH);
@@ -632,11 +633,17 @@
$shareName = encode("utf8", $shareName);
$stat{xferOK} = $stat{hostAbort} = undef;
$stat{hostError} = $stat{lastOutputLine} = undef;
-if ( -d "$Dir/new/$shareName" ) {
+if ( $shareName eq "" ) {
+print(LOG $bpc->timeStamp,
+ "unexpected empty share name skipped\n");
+next;
+}
+if ( $shareDuplicate->{$shareName} ) {
print(LOG $bpc->timeStamp,
"unexpected repeated share name $shareName skipped\n");
next;
}
+$shareDuplicate->{$shareName} = 1;
UserCommandRun("DumpPreShareCmd", $shareName);
if ( $? && $Conf{UserCmdCheckStatus} ) {
@@ -915,6 +922,10 @@
#
last;
}
+#
+# Wait for any child processes to exit
+#
+1 while ( wait() >= 0 );
}
#
--
SOLARIS 10 is the OS for Data Centers - provides features such as DTrace,
Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW
http://p.sf.net/sfu/solaris-dev2dev
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC not cleaning out cpool.
Joe, Yes, you have the IO::Dirent problem, and it sounds like it is fixed, since BackupPC_nightly reports non-zero information. However, it doesn't report that it removed any files, so every cpool file (at least for the output you included) has at least 2 links. However, BackupPC_trashClean (which is started once when BackupPC first starts) will also have the same problem. Did you re-start BackupPC after patching $IODirentOk? You should check to see if /var/data/backuppc/trash has a lot of files. If so, that's the problem. After those files are removed, then BackupPC_nightly will find files that only have one link, and will remove them. Craig -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] How do I use an external USB drive as backup target?
my comments are interspersed below... On Sat, Feb 13, 2010 at 6:40 PM, Jeffrey J. Kosowsky wrote: > John Hudak wrote at about 12:06:29 -0500 on Saturday, February 13, 2010: > > Hello: > > I am considering using an external USB drive as the storage for my > backups. > > I am running backup pc under Debian 5.0. > External USB drives are a *BAD* idea for multiple reasons: > - Slow > - Unreliable > - Subject to being disconnected > etc. > > Yes, I know 1, and 2 is 'it depends', and 3 is exactly the reason why I want to use usb drives > > Part 1 > > What do I need to do to configure the USB disk as the target? (e.g. how > do I > > do it?) > > The USB disk is currently formatted as a NTFS file system. Do I *need* > to > > reformat it to ext3? or other? > - NTFS is not usually used - need to check whether it supports the > types of hard links required for BackupPC > > > > > Part 2 > > Assume I am crazy paranoid about preserving backup data and I get a > second > > USB drive to serve as a backup to the first USB drive. > > Also assume that I am not concerned about the bandwidth across the > network > > or the various buses. > > > > >From a data reliability standpoint, is it better to run a backup > session to > > USB drive 1, and then repeat the backup to USB drive 2? OR > > run a backup session to USB drive 1, and then copy the backup > directories to > > USB drive 2??? > Look at the archives and FAQ - this has been discussed *many* times so > no point in wasting peoples time in rehashing. > > I did a quick search of the archives before asking - I did not find a definitive answer... > > The first approach could have errors in different backed up files on > disk 1 > > or 2 but given the odds, very unlikely that the same exact error would > show > > up > > in the same exact way in the same file across both USB disks. > > OTOH, the second approach would allow the exact error in the backup on > USB > > disk 1 to be copied to USB disk 2. > > > > I am leaning towards repeating the backup on two drives. > > > > My understanding is that files that are backed up (using either rsync or > > smb) are 'encrypted' (for lack of a better word), and to view them I > need to > > use zcat.-True? > > There is a better word -- *compressed* > So that is the word that is not clearly used in the documentation. There are many ways that backups can be manipulated: stored in a completely nonstandard/proprietary file system and protocol such as z-san, they can be encrypted, and they can be compressed. The backup PC doc talks about using compression, but does not state if any compression is used in the default configuration. Compression is often configuration parameter. It does not make sense to compress many audio and video formats. If the data to be backed up consists predominantly of these types of files, then it makes no sense to waste CPU cycles applying compression to get < 5% compression. > > Also, can the backup profile be specified to perform complete data > copies > > periodically, as opposed to a baseline and then periodic incrementals? > > Read the documentation and FAQ. > I have read the doc, (where IMHO) this should have been clearly stated. Instead the doc frequently introduces a topic with 1-2 sentences, then goes off on a tangent for 1-3 paragraphs about how things were done in a previous verson (completely irrevelant), or talks about what will be comming (again, irrelevant), or points one to another section of the doc, in the middle of some other thread that is related to the topic but does not address the topic at hand, or, introduces a topic, then talks about 3-4 other ways to accomplish the same thing, without telling the reader exactly how to do the initial topic to begin with. So my fault...I need to also read FAQs. > > > Lastly, does anyone have a statistical number that represents the > > probability of a backup file (e.g. on the target backup disk) containing > an > > error introduced > > by the backup procedure? I know there are error probabilities for both > disk > > and tape reads/writes failures, but am wondering if anything like that > > exists for the backup software. (A group I used to work with did this > sort > > of testing, and actually had some statistics on the reliability of > backup > > programs, wrt types of files, sizes, w/wo compression, and the types of > > compression. Not sure the open source community would go through this > type > > of assessment - but thought I'd ask. > > The probability is either 0 if no bugs in the software (or your > configuration of it) or 100% if bugs in the software and your dataset > triggers the bug. Your question is not very well-framed and pretty > meaningless. I suggest you learn a bit more about backup in general > and backuppc in particular. There is a lot of good documentation on > BackupPC in the Wikki and in the archives, I suggest you reference it... Well, in the extremely simplistic and ideal c
Re: [BackupPC-users] BackupPC not cleaning out cpool.
Joseph Holland wrote: > # du -xsmh /var/data/backuppc/ > 544G/var/data/backuppc/ > > > Joe. > > On 11/02/2010 22:45, Matthias Meyer wrote: >> Joseph Holland wrote: >> >> >>> We are having a problem with many of our BackupPC servers in our company >>> at the moment. We are running Debian Lenny and BackupPC 3.1.0. The >>> data volumes since we upgraded to this version have seemed to just be >>> filling up at a constant rate. >>> >>> The web interface is saying that the pool is 136.98GB but when you do a >>> "du -smh" on the topdir "/var/data/backuppc" it returns 543GB used. >>> >> Try du -xsmh >> >> br >> Matthias >> > strangely, indeed. du delivers a roughly estimation, not an exact value. But we are speaking about 140GB vs 540GB. 1) "du" and GUI use the same directory? I have /var/lib/backuppc and not /var/data/backuppc. 2) There are no extra files from you on your /var/data/backuppc? I don't know how the GUI calculates the usage of space. Sorry Matthias -- Don't Panic -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Trying to get backuppc working
Chris, What happened when you ran the commands I sent? Do you have output? It sounds like something caused dpkg to stop during a configuration step, which might be resolved with: apt-get -f install apt-get dist-upgrade I'll try to test a karmic install later. I can say certainly that backuppc works just fine on Ubuntu hardy (the last LTS release), either with the stock packages or the ones from my repository. Regards, Tyler On Sunday 14 February 2010 01:49:47 bigonroad wrote: > I don't really know what to do now. After a day on it, I still haven't got > backuppc running on Ubuntu 9.10 server. > > Should I try a different variant, or can you advise me what to do? I am > doing a fresh install, and will start from the beginning again. > > I will install everything apache, samba, samba-common-bin, and everything > else before backuppc. However, if that doesn't work, I am out of ideas! > > Bless, > Chris > > +-- > > |This was sent by bigonr...@gmail.com via Backup Central. > |Forward SPAM to ab...@backupcentral.com. > > +-- > > > > --- > --- SOLARIS 10 is the OS for Data Centers - provides features such as > DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW > http://p.sf.net/sfu/solaris-dev2dev > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > -- "Every normal man must be tempted at times to spit on his hands, hoist the black flag, and begin to slit throats." -- H. L. Mencken -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Trying to get backuppc working
I don't really know what to do now. After a day on it, I still haven't got backuppc running on Ubuntu 9.10 server. Should I try a different variant, or can you advise me what to do? I am doing a fresh install, and will start from the beginning again. I will install everything apache, samba, samba-common-bin, and everything else before backuppc. However, if that doesn't work, I am out of ideas! Bless, Chris +-- |This was sent by bigonr...@gmail.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
