Re: [BackupPC-users] since upgrading to 4.0 I can no longer do full backups

[An Ancient BackupPC User]

So I know its a VERRRY long time ago but I finally got around to creating a 
rsync config file and putting the charset=UTF-8 in it.  So it might help 
others, I can confirm then I got a protocol streaming error so removed the 
protected args parameter as you suggested and it did a backup.

Thank you so much for such a great product

Paul


On May 15, 2018 at 9:30:24 PM, Craig Barratt (cbarr...@users.sourceforge.net) 
wrote:

I'm glad that it's working now.

For OSX, the default rsync is really old.  On High Sierra 10.13.4, it's rsync 
2.6.9, circa 2006.  I strongly recommend installing a recent version in, eg, 
/usr/local/bin.

Craig

On Tue, May 15, 2018 at 6:27 PM, An Ancient BackupPC User 
 wrote:

Ok so I can report my windows10 machines are completing full backups now 
without any other changes - THANK YOU :)

My mac needs more work based on the way I have the daemon set up.  I will 
report back some time tomorrow, once I have had chance to work on it.

Thanks again, I really appreciate your help and backup pc is the best, have 
used it for ever.

Paul


On May 15, 2018 at 4:45:46 PM, Craig Barratt via BackupPC-users 
(backuppc-users@lists.sourceforge.net) wrote:

On your client the rsyncd.conf file should have a setting such as "charset = 
utf-8".  You'll need to make sure rsyncd reloads the rsyncd.conf file before 
retesting.

If it still fails, the next thing to try is to remove --protect-args from 
$Conf{RsyncArgs}.

Craig

On Tue, May 15, 2018 at 1:17 PM, Paul Farrow x...@xxx.com wrote:
Ok I updated as you said and now I get this when trying to do a full backup on 
my mac.

This is the rsync child about to exec /usr/bin/rsync_bpc
bpc_attrib_backwardCompat: WriteOldStyleAttribFile = 0, KeepOldAttribFiles = 0
rsync: on remote machine: -slHogDtprc: unknown option


On May 15, 2018 at 3:28:04 PM, Craig Barratt via BackupPC-users 
(backuppc-users@lists.sourceforge.net) wrote:

Ok, you are using rsyncd, so $Conf{RsyncSshArgs} doesn't matter.

It seems that when you upgraded to 4.0 the config file was not updated (or 
perhaps these are per-host settings, which don't get updated during an 
upgrade).  The correct settings should be:

$Conf{RsyncFullArgsExtra} = [
            '--checksum',
];

#
# Arguments to rsync for backup.  Do not edit the first set unless you
# have a good understanding of rsync options.
#
$Conf{RsyncArgs} = [
            '--super',
            '--recursive',
            '--protect-args',
            '--numeric-ids',
            '--perms',
            '--owner',
            '--group',
            '-D',
            '--times',
            '--links',
            '--hard-links',
            '--delete',
            '--delete-excluded',
            '--partial',
            '--log-format=log: %o %i %B %8U,%8G %9l %f%L',
            '--stats',
];

Craig


On Tue, May 15, 2018 at 10:26 AM, An Ancient BackupPC User 
 wrote:
Thanks Craig for such a quick response.

I upgraded some time ago but have been pretty lazy in posting to the mailing 
list about it.

XferLOG file /somefolder/XferLOG.840.z created 2018-05-15 11:51:27 
Backup prep: type = full, case = 4, inPlace = 0, doDuplicate = 0, newBkupNum = 
840, newBkupIdx = 45, lastBkupNum = 839, lastBkupIdx = 44 (FillCycle = , 
noFillCnt = 20)
Running: /usr/bin/rsync_bpc --bpc-top-dir /somefolder --bpc-host-name somehost 
--bpc-share-name auser --bpc-bkup-num 840 --bpc-bkup-comp 3 --bpc-bkup-prevnum 
839 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 693052 --bpc-attrib-new 
--bpc-log-level 2 --numeric-ids --perms --owner --group -D --links --hard-links 
--times --block-size=2048 --recursive  --timeout=72000 
--password-file=/passwordfile --exclude=Virtual\ Machines/ --exclude=.Trash/ 
--exclude=Downloads/ --exclude=updater/ --exclude=Library/ 
--exclude=Movies/BBC\ iPlayer\ Downloads/ --exclude=Microsoft\ User\ Data/ 
--exclude=AdobeStockPhotos/ --exclude=Data.noindex/ --exclude=.nuget/ 
--exclude=.dnx/ --exclude=.Android/ auser@somehost@::auser /
full backup started for directory some directory
Xfer PIDs are now 17304

RsyncSshArgs - non existent in config.pl and clients config.
RsyncClientPath - $Conf{RsyncClientPath} = 'rsync';
RsyncBackupPCPath - $Conf{RsyncBackupPCPath} = '/usr/bin/rsync_bpc';
RsyncFullArgsExtra - non existent
RsyncArgsExtra = $Conf{RsyncArgsExtra} = [];
RsyncArgs - $Conf{RsyncArgs} = [
  '--numeric-ids',
  '--perms',
  '--owner',
  '--group',
  '-D',
  '--links',
  '--hard-links',
  '--times',
  '--block-size=2048',
  '--recursive'
];
 

Appreciate your help - thanks

Paul

On May 15, 2018 at 1:04:19 PM, Craig Barratt via BackupPC-users 
(backuppc-users@lists.sourceforge.net) wrote:

Please send the first part of the log file with the full rsync command (feel 
free to redact confidential arguments, but maintain any special characters etc).

Also send this client's settings for $Conf{RsyncSshArgs}, 
$Conf{RsyncClientPath}, $Conf{RsyncBackupPCPath}, $Conf{RsyncFullArgsExtra} and 
$Conf{RsyncArgs}.

Craig

On Tue, May 15, 2018 at

Re: [BackupPC-users] BackupPC 4.2.1 apparently in an infinite loop.

[Michael Stowe]

On 2018-09-16 11:59, Mike Hughes wrote:

Michael,

Condescending and belittling treatment of others in this list is not
the norm. Your personal attacks are unwarranted and unhelpful.


I certainly didn't mean anything that way, and let me apologize should 
anybody have taken anything that way.  My general comments and generic 
example were meant to be taken as that, and not a commentary on the OP.



Since you reached out and asked for help in relating to others I will
share with you this piece from Dr. Phil Agre which helped me immensely
when I was learning the ropes at the helpdesk and I still refer to it
regularly to help keep me on track:


I appreciate the reference (and am familiar with it) and anybody who has 
spent time doing general tech support knows that a lot of finding the 
right level of help involves properly gauging the level of expertise, 
which is not always a simple task, especially on this list:  we've had 
people ask for help who have needed basic background on what a web 
server is, and people who have been mortally insulted by any suggestion 
that they are not foremost experts in Linux, perl, and the field of 
backup software.  Sometimes, these are the same people.


I've gotten it wrong quite a bit.  I've written long passages on how to 
diagnose a [presumably] complex problem only to find the questioner 
simply didn't know how to invoke an executable that wasn't in the path, 
and I've inquired about simple, in-the-manual, obvious steps and it 
became clear later that the questioner was well into debugging a tough 
corner case.  With this group in particular, some questioners only need 
a tiny nudge, and some need handholding through the most basic 
operations, and some uncover weird corner cases that need addressing.  
Hopefully I get it right more often than not, but it's something I can 
always improve upon.



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackupPC 4.2.1 apparently in an infinite loop.

[Mike Hughes]

Michael,

Condescending and belittling treatment of others in this list is not the norm. 
Your personal attacks are unwarranted and unhelpful.


Since you reached out and asked for help in relating to others I will share 
with you this piece from Dr. Phil Agre which helped me immensely when I was 
learning the ropes at the helpdesk and I still refer to it regularly to help 
keep me on track:


"How to help someone use a computer.


Computer people are generally fine human beings, but nonetheless they do a lot 
of inadvertent harm in the ways they "help" other people with their computer 
problems. Now that we're trying to get everyone on the net, I thought it might 
be helpful to write down everything I've been taught about helping people use 
computers.

First you have to tell yourself some things:

  1.  Nobody is born knowing this stuff.
  2.  You've forgotten what it's like to be a 
beginner.
  3.  If it's not obvious to them, it's not obvious.
  4.  A computer is a means to an end. The person you're helping probably cares 
mostly about the end. This is reasonable.

Their knowledge of the computer is grounded in what they can do and see -- 
"when I do this, it does that". They need to develop a deeper understanding, of 
course, but this can only happen slowly, and not through abstract theory but 
through the real, concrete situations they encounter in their work.

By the time they ask you for help, they've probably tried several different 
things. As a result, their computer might be in a strange state. This is 
natural.

The best way to learn is through apprenticeship -- that is, by doing some real 
task together with someone who has skills that you don't have.

Your primary goal is not to solve their problem. Your primary goal is to help 
them become one notch more capable of solving their problem on their own. So 
it's okay if they take notes.

Most user interfaces are terrible. When people make mistakes it's usually the 
fault of the interface. You've forgotten how many ways you've learned to adapt 
to bad interfaces. You've forgotten how many things you once assumed that the 
interface would be able to do for you.

Knowledge lives in communities, not individuals. A computer user who's not part 
of a community of computer users is going to have a harder time of it than one 
who is.

Having convinced yourself of these things, you are more likely to follow some 
important rules:

  *   Don't take the keyboard. Let them do all the typing, even if it's slower 
that way, and even if you have to point them to each and every key they need to 
type. That's the only way they're going to learn from the interaction.
  *   Find out what they're really trying to do. Is there another way to go 
about it?
  *   Attend to the symbolism of the interaction. Try to squat down so your 
eyes are just below the level of theirs. When they're looking at the computer, 
look at the computer. When they're looking at you, look back at them.
  *   Explain your thinking. Don't make it mysterious. If something is true, 
show them how they can see it's true. When you don't know, say "I don't know".  
When you're guessing, say "let's try ... because ...". Resist the temptation to 
appear all-knowing. Help them learn to think like you.
  *   Be aware of how abstract your language is. For example, "Get into the 
editor" is abstract and "press this key" is concrete. Don't say anything unless 
you intend for them to understand it. Keep adjusting your language downward 
towards concrete units until they start to get it, then slowly adjust back up 
towards greater abstraction so long as they're following you. When formulating 
a take-home lesson ("when it does this and that, you should check 
such-and-such"), check once again that you're using language of the right 
degree of abstraction for this user right now.
  *   Whenever they start to blame themselves, blame the computer, no matter 
how many times it takes, in a calm, authoritative tone of voice. If you need to 
show off, show off your ability to criticize the bad interface. When they get 
nailed by a false assumption about the computer's behavior, tell them their 
assumption was reasonable. Tell *yourself* that it was reasonable.
  *   Formulate a take-home lesson.
  *   Take a long-term view. Who do users in this community get help from? If 
you focus on building that person's skills, the skills will diffuse outward to 
everyone else.
  *   Never do something for someone that they are capable of doing for 
themselves.

* Don't say "it's in the manual". (You probably knew that.)"


Source: http://polaris.gseis.ucla.edu/pagre/how-to-help.html

This is just a basic framework to keep in mind when helping others learn 
something that you already know. Adapt it as necessary to fit the situation.

I hope this helps.

Sincerely,
Mike



From: Michael Stowe 
Sent: Sunday, September 16, 2018 11:39:51 AM
To: G

Re: [BackupPC-users] BackupPC 4.2.1 apparently in an infinite loop.

[Michael Stowe]

On 2018-09-16 05:27, G.W. Haywood via BackupPC-users wrote:


Once again, thank you for your input.  Do consider some restraint.


That's fair.  Consider the context:  you've already admitted to breaking 
BackupPC after an upgrade and instead of methodically going through the 
steps of getting it working again, you upgrade to a completely new 
version, misunderstand how it works, and start mucking about with 
internal tools from the command line, while demonstrating that you don't 
know the side effects.


Is there a quick fix for this?  Not if you're trying to solve one 
problem with a completely unrelated tool:  if you want to solve taking 
too much space, there are better ways to do it.  If you want to solve 
rsync problems, there are better ways to do it.


You do have my apologies for any confusion I have caused -- so yes, 
BackupPC_backupDelete is capable of removing directories from a share 
that were backed up.  Will this solve your problem, or is it a good 
idea?  No.  Are you interpreting what it's doing and its side effects 
properly?  No.  Did you get the syntax right?  Hard to say, as you've 
edited the command line and its responses -- and as I've already pointed 
out, most people wouldn't run it from the command line, so if there are 
quirks in how it handles directory input, a test case would be 
warranted.


In many years of helping users who have wide variety in degrees of 
expertise, I do struggle with people who demonstrate an incomplete 
understanding of the tool they're trying to use, while insisting on 
doing things that are, at best, orthogonal to their problem.  It is 
usually coupled with a healthy dose of failing to follow reasonable 
steps up until that point, and unwinding that is difficult, if not 
impossible.  It is often unclear whether it is a problem of attention to 
detail, ability to comprehend, missing fundamentals, poor 
troubleshooting techniques, or an outsized ego.


Tech Support: "Okay, let's see what's in the directory.  Type 'ls -l' "
Customer: [typing sounds continue for several minutes]
Tech Support: "Uh.  Do you have a directory listing yet?"
Customer: "I read that hard drives need fsck'ing every now and then"
Tech Support: "It would help to know what's in that directory"
Customer: "It's rebooting now"

I'm open to suggestions as to how to be helpful in such situations.  If 
obliquely pointing toward the documentation doesn't work, would it be 
better to be more direct?  Would you prefer no response?  What is the 
best way to gently suggest that leaping to a solution before 
understanding the problem is unlikely to yield satisfying results?


Thanks


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] BackupPC 4.2.1 apparently in an infinite loop.

[G.W. Haywood via BackupPC-users]

Hi there,

On Sat, 15 Sep 2018, Michael Stowe wrote:


... 4.2.1 seemed to be eating a LOT more disc space than V3.x ...
tried used BackupPC_backupDelete to delete some cruft ...


And I used "rm -r *" to list the contents of a directory.


Thanks very much for your reply, and if you ever find yourself
confronted by a mob of respectable physicists, run like hell.


... BackupPC_backupDelete is not a tool which removes directories
you did not mean to back up.  It deletes entire backups or shares.


Not wishing to contradict you, I'll let the documentation do that:

http://backuppc.sourceforge.net/BackupPC-4.2.1.html#Other-Command-Line-Utilities

states (the emphasis is mine):

"BackupPC_backupDelete deletes an entire backup, OR A DIRECTORY PATH WITHIN A 
BACKUP."


Given what I've done, is any of this to be expected?


Yes, it is absolutely to be expected, as the tool does exactly what
it says it does.  It does not do what you want it to do ...


Maybe the name of the tool does say something, but the documentation
says it does exactly what I want it to do, and the debug output (given
below) shows it attempting to do it.  The tool, however, does not seem
to do what the documentation says it does (the debug output shows it
failing, perhaps because it's looking for non-existent directories).

I suspect that this is a fault in BackupPC_backupDelete, but if it's
just a case of incorrect documentation then it's unfortunate that the
tool apparently breaks, and then also breaks BackupPC -- and that I
had to find out this way.


... which is better accomplished in a different way.


Is this different way documented?  If so a pointer would be welcome.


... actual disc usage as reported by the OS (df) isn't changing,
but BackupPC appears to think that it has now used 5TB of a 3TB
disc and the claimed usage is growing. ...
...
Pool is 663.71+4946.35GiB ...
Pool file system was recently at 80% ...


Today's pool figures: Pool is 663.71+6922.70GiB

So the pool is using ~7.5TB?!

The pool is in a partition on a 3TB drive.  Apparently it is expected
that if I use BackupPC_backupDelete in the way that's documented (and
in which you say that it is not intended to be used), then not only
will the tool not do what it's documented to do, but also BackupPC may
not recover from the shock -- symptoms of which include what appears
to be an infinite loop which I've yet to break (the incremental backup
which was started on 8th September completed in just under 8 minutes,
but the one started on 9th September is still running a week later);
fanciful calculations of the storage in use by the pools displayed on
the Web interface; and what looks like a propensity to incrementally
delete all the V4 backups that were made in previous runs.

tornado:/var/lib/backuppc/pc# >>> date
Sun 16 Sep 13:04:38 BST 2018
tornado:/var/lib/backuppc/pc# >>> tail localhost/LOG.092018
2018-09-07 20:17:25 Finished BackupPC_backupDelete, status = 0 (running time: 
104 sec)
2018-09-08 20:00:02 incr backup started for directory Config
2018-09-08 20:00:27 incr backup started for directory Homes
2018-09-08 20:08:23 incr backup 735 complete, 916597 files, 190430491468 bytes, 
0 xferErrs (0 bad files, 0 bad shares, 0 other)
2018-09-08 20:08:23 Removing unfilled backup 720
2018-09-08 20:08:23 BackupPC_backupDelete: removing pre-v4 backup #720
2018-09-08 20:10:07 BackupPC_backupDelete: got 0 errors
2018-09-08 20:10:07 Finished BackupPC_backupDelete, status = 0 (running time: 
104 sec)
2018-09-09 20:00:04 incr backup started for directory Config
2018-09-09 20:00:41 incr backup started for directory Homes


You can, of course, delete the entire backup, then set excludes, and
back up what you want.  This is probably simplest if you're going to
fool around at the storage level.


Yes, of course I can, but I'll lose a lot that way.  Just to be clear,
my intention was not to "fool around", but to prevent BackupPC V4 from
using MORE THAN THREE TIMES AS MUCH STORAGE as BackupPC V3 used TO DO
EXACTLY THE SAME JOB and in the process run out of space on the backup
partition.  The partition was originally sized three times bigger than
needed when it was running BackupPC V3, but is now proving to be much
less than adequate.  So much so that I might as well go back to using
rsync from a cron job (and which, incidentally, I still do for all the
really important stuff).


... don't try to screw around with the contents of individual
backups.  It's possible, but you'll need to have a deep
understanding of what tools to use and the ramifications ...


Even supposing that my usage of BackupPC_backupDelete was incorrect,
one would hardly expect that to throw the backup service into a fit.
I've done things like this for years with V3.  Apparently V4 is less
forgiving than V3 in this respect.

Below is the debug output mentioned earlier in this message.

8<--
BackupPC_backupDelete debug output: userid is redacted, no othe