[BackupPC-users] Full backup not backing up all files

2014-10-14 Thread Mikko Kortelainen
I have a problem with two BackupPC hosts, yet there's a further third
host that is working ok.

The problem is that a full backup does not seem to back up all files
on Windows hosts. An incremental backs up many more files than a full
one, or so it seems, looking at the version history.

On a typical C$ share, full backup omits directories such as Program
Files and WINDOWS. The incremental backs them up.

The only error messages I get are a couple of sharing violations in
the error log, but nothing that should fail those whole directories.

So far I have seen this on two BackupPC hosts, both Ubuntu 14.04 LTS.
A third host which I have kept at 12.04 does not seem to suffer from
this. The BackupPC version on all three of them is 3.3.0. Smbclient
version is 3.5.11 on the 12.04 host and 4.1.6 on the 14.04 hosts.

For testing purposes, I set up one 12.04 host and one 14.04 host to
back up the same target with similar configuration files. Looking at
the XferLOGs, the total number of files on share C$ for the 12.04 host
was:

66691 filesTotal taking 9 GB on disk, while the 14.04 host had:
23910 filesTotal taking 6 GB on disk.

So around 4 files of total 3 GB are omitted.

Full backup command is:

/usr/bin/smbclient -A /etc/backuppc/smbauth HOSTNAME\\C\$ -E -d 1
-c tarmode\ full -Tc -

Incremental:

/usr/bin/smbclient -A /etc/backuppc/smbauth HOSTNAME\\C\$ -E -d 1
-c tarmode\ full -TcN /var/lib/backuppc/pc/HOSTNAME/timeStamp.level0 -

Here's is a screenshot of the history page. Backups number 0, 2 and 5
are full backups, while the rest are incrementals:

https://drive.google.com/file/d/0B8tcfJc8vLAkZW1UQ25qUTRtbzQ/view?usp=sharing

What could be the problem here? Something to do with the Samba version
(3.5 vs. 4.1)? What can I do to diagnose this?

-Mikko

--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Full backup not backing up all files

2014-10-14 Thread Mikko Kortelainen
Holger, thanks for your response.

The smb.conf should be pretty much stock on both machines. Besides
comments and whitespace, the only difference is this:

   client ntlmv2 auth = no

That's set on the Samba 3.5 host but not on the 4.1. Actually it seems
that the default for that has changed from Samba 3 (no) to Samba 4
(yes). I set it to no and ran a full backup again but the result was
no different.

A colleague of mine just told me he did a Samba downgrade from 4.1 to
3.5 and tested a quick full backup which was fine. I haven't tried it
myself yet.

What I did do is I ran a full backup command from the command line
with smbclient 4.1 and it stops here:

   0 (0,0 kb/s) \MSDOS.SYS
   94720 (  818,6 kb/s) \msizap.exe
   47772 ( 2744,3 kb/s) \NTDETECT.COM
  297072 ( 4917,1 kb/s) \ntldr
NT_STATUS_SHARING_VIOLATION opening remote file \pagefile.sys (\)
NT_STATUS_SHARING_VIOLATION listing \\*
tar: dumped 26297 files and directories
Total bytes written: 17752618496

So it stops after the first NT_STATUS_SHARING_VIOLATION on a file in
the root directory of C$. There's a bug related to it and tagged to
Samba 3.6:

smbclient TAR omits remainder of directory on access denial.
https://bugzilla.samba.org/show_bug.cgi?id=10605

Perhaps Samba 4.1 suffers from the same bug.

This is pretty bad as the backup is still listed as a good backup,
even though it is far from complete. And the error log can have more
NT_STATUS_SHARING_VIOLATION messages so it is not immediately obvious
that the backup stopped before copying everything it can. It only
omits the rest of the files and subdirectories in the same directory
with that error and continues with the next same or upper level
directory. Being the root of C$ there are no same level or upper level
directories left to back up, so it stops completely.

As time goes by, the incrementals get deleted from disk, and any files
not included in the full backups will be gone, if I have understood
correctly (?). So there will be at most IncrAgeMax days of complete
backups available. The full backups are just partial.

I guess the only thing I can do now is downgrade to Samba 3.5.

It is quite alarming to realize this thing may bite every user out
there with the default packages and settings on any relatively new
distro. And there is no good warning about it. Looking at the error
log, the problem is not immediately obvious. It is obvious only after
you browse your full backups and discover some stuff is not there.

-Mikko


2014-10-14 15:26 GMT+03:00 Holger Parplies wb...@parplies.de:
 Hi,

 Mikko Kortelainen wrote on 2014-10-14 12:18:00 +0300 [[BackupPC-users] Full 
 backup not backing up all files]:
 I have a problem with two BackupPC hosts, yet there's a further third
 host that is working ok.

 The problem is that a full backup does not seem to back up all files
 on Windows hosts. An incremental backs up many more files than a full
 one, or so it seems, looking at the version history.
 [...]
 What could be the problem here? Something to do with the Samba version
 (3.5 vs. 4.1)?

 so it would seem. Or maybe only a changed default configuration. I don't know
 enough about smbclient to guess what it might be, but I know that smb.conf is
 relevant for the smbclient command as well.

 What can I do to diagnose this?

 A 'diff' of the two smb.conf files might shed some light, presuming it is
 configuration related. Otherwise, I would expect the old smbclient package
 (along with the same version samba-common) to work on the newer system. You
 have several options for trying the downgrade (***Note: I am assuming you have
 nothing installed on the system that depends on smbclient/samba-common, i.e.
 no samba server, samba configuration tool etc. If in doubt, check with
 'apt-cache rdepends samba-common smbclient'***):

 - download the packages with a browser and install with 'dpkg -i',
 - add a sources.list line for the old distribution on the host(s) with the
   new one and do something like

 apt-get update
 apt-get install --reinstall smbclient=xyz samba-common=xyz

   (where 'xyz' is the version of smbclient/samba-common on the old system;
   'apt-cache policy smbclient' might help figuring things out), or
 - download the packages on the old system with

 apt-get -d install --reinstall smbclient samba-common

   (the '-d' switch means 'download only'). Copy them from
   /var/cache/apt/archives over to the new host and install with 'dpkg -i'.

 You'll probably need to confirm the downgrade for 'apt-get' and possibly
 use a '--force-downgrade' switch on dpkg, but I don't think so.

 Presuming downgrading helps and you want to keep things that way, you
 should put the packages on hold (so they won't be inadvertantly
 upgraded at some point), and you should probably browse the changelogs to
 see if there were any important security fixes that the old version misses.

 Otherwise, 'apt-get install smbclient samba-common' or simply
 'apt-get upgrade