Unfortunately, I did not save the pool state before "fixing" the
problem, and I haven't been able to reproduce it. Hopefully there is
something in the logs that will help narrow down the cause. The
snippets I sent to the mailing list were partial greps of the log, but
if you do see something that you need more context for, I still do
have the full XferLog at logging level 6, with rsync -vvv. I saved it
before deleting the bad backups.
Regards,
Raman
On Thu, May 9, 2019 at 2:31 AM Craig Barratt via BackupPC-users
<backuppc-users@lists.sourceforge.net> wrote:
>
> Thanks for the update.
>
> Can you re-create the problem? It would be great to track down what went
> wrong. (I still need to look at your log file snippets in more detail,
> hopefully this weekend.)
>
> Craig
>
> On Wed, May 8, 2019 at 10:24 PM Raman Gupta <rocketra...@gmail.com> wrote:
>>
>> So the issue was definitely with the prior backups, probably related
>> to borkage due to the cancellation / partial backup I mentioned in my
>> previous message. The last six or seven backups were all incrementals,
>> with the last one being filled. I had tried deleting the last filled
>> incremental several times, but this had never fixed the problem --
>> probably because deleting that backup caused bpc to fill the prior
>> incremental, and whatever was borked in the deleted backup was copied
>> over to the prior one. The solution was to delete all the backups all
>> the way back to the previous filled full backup. Once this was done, a
>> manually triggered full backup then finally completed with no issues.
>>
>> Regards,
>> Raman
>>
>> On Wed, May 8, 2019 at 11:32 AM Raman Gupta <rocketra...@gmail.com> wrote:
>> >
>> > The backup is large, but not huge. I created a new test host pointing
>> > only to /home/raman/x and it worked just fine. I forgot to mention I
>> > also have another BackupPC server, backing up the same client with the
>> > same configuration, and it backs up all of these files from this
>> > client without any issue, so the issue is definitely on the server
>> > side, not on the client side.
>> >
>> > I ran a test with XferLogLevel=6 and "-vvv". Here are some relevant
>> > grep outputs -- `home/raman/x/y/2018` is a directory created in 2018,
>> > while all of the previously created directories in `y` are backing up
>> > without any issue --- this happens to be tax data so I have one
>> > directory for every year, and all 2015 directories and before are
>> > fine, but 2016 and later are not:
>> >
>> > $ BackupPC_zcat /var/lib/BackupPC/pc/edison/XferLOG.3972.z | grep
>> > "x/y/2018"
>> > [sender] showing directory home/raman/x/y/2018 because of pattern /home/**
>> > [sender] make_file(home/raman/x/y/2018,*,2)
>> > recv_file_name(home/raman/x/y/2018)
>> > recv_file_name(home/raman/x/y/2018)
>> > G bpc_lstat(home/raman/x/y/2018)
>> > file has vanished: "/home/raman/x/y/2018"
>> > recv_generator(home/raman/x/y/2018,216871)
>> > G bpc_lstat(home/raman/x/y/2018)
>> > G bpc_mkdir(home/raman/x/y/2018, 0700)
>> > [sender] pushing local filters for /home/raman/x/y/2018/
>> > [sender] showing file home/raman/x/y/2018/a.txt.pdf because of pattern
>> > /home/**
>> > [... same thing for all other subdirectories and files ...]
>> > recv_file_name(home/raman/x/y/2018/a.txt.pdf)
>> > [... same thing for all other subdirectories and files ...]
>> > recv_generator(home/raman/x/y/2018,217359)
>> > G bpc_lstat(home/raman/x/y/2018)
>> > G bpc_mkdir(home/raman/x/y/2018, 040775)
>> > G bpc_lstat(home/raman/x/y/2018)
>> > rsync_bpc: stat "/home/raman/x/y/2018" failed: No such file or directory
>> > (2)
>> > delete_in_dir(home/raman/x/y/2018)
>> > [generator] pushing local filters for /home/raman/x/y/2018/
>> > G bpc_opendir(home/raman/x/y/2018)
>> > G bpc_lstat(home/raman/x/y/2018/a.txt.pdf)
>> > file has vanished: "/home/raman/x/y/2018/a.txt.pdf"
>> > [... same thing for all other subdirectories and files ...]
>> > recv_generator(home/raman/x/y/2018/a.txt.pdf,217366)
>> > G bpc_lstat(home/raman/x/y/2018/a.txt.pdf)
>> > G bpc_sysCall_poolFileCheck(home/raman/x/y/2018/a.txt.pdf): potential
>> > match /var/lib/BackupPC//cpool/28/56/29571824e1fe8922304bd1924f02ce22
>> > (len = 171990)
>> > G bpc_lstat(home/raman/x/y/2018/a.txt.pdf)
>> > G bpc_open(home/raman/x/y/2018/a.txt.pdf, 0x0, 00) -> 3
>> > G bpc_lgetxattr(home/raman/x/y/2018/a.txt.pdf, user.rsync.%aacl)
>> > G bpc_llistxattr(home/raman/x/y/2018/a.txt.pdf)
>> > G bpc_read(3 (home/raman/x/y/2018/a.txt.pdf), buf, 171990) tmpFd = -1
>> > G bpc_close(3 (home/raman/x/y/2018/a.txt.pdf))
>> > [... same thing for all other subdirectories and files ...]
>> >
>> > and some context around the "file has vanished" message:
>> >
>> > $ BackupPC_zcat /var/lib/BackupPC/pc/edison/XferLOG.3972.z | grep -C
>> > 5 'file has vanished: "/home/raman/x/y/2018"'
>> > G bpc_readdir -> 1996
>> > G bpc_lstat(home/raman/x/y/1996)
>> > [generator] make_file(home/raman/x/y/1996,*,2)
>> > G bpc_readdir -> 2018
>> > G bpc_lstat(home/raman/x/y/2018)
>> > file has vanished: "/home/raman/x/y/2018"
>> > G bpc_readdir -> 2014
>> > G bpc_lstat(home/raman/x/y/2014)
>> > [generator] make_file(home/raman/x/y/2014,*,2)
>> > G bpc_readdir -> 2011
>> > G bpc_lstat(home/raman/x/y/2011)
>> >
>> > Lastly, I've tried a test of placing a new file into the 2015
>> > directory, and this file does back up successfully. So it seems like
>> > new files are captured, as long as they fall into an existing
>> > directory, but new directories are not.
>> >
>> > One thing I did do on this server is cancel the first full backup
>> > after removing `--one-file-system` and deleted the backup from the web
>> > UI, whereas the other server completed its backup without
>> > intervention. I wonder if the server pool was corrupted somehow by
>> > doing that? Also, the `--one-file-system` was probably added at the
>> > same time as v4 and the resulting pool migrations on subsequent
>> > backups, which would have *removed* a lot of stuff that was being
>> > backed up by v3. I have run a BackupPC_fsck and there were no errors.
>> >
>> > Regards,
>> > Raman
>> >
>> > On Wed, May 8, 2019 at 2:11 AM Craig Barratt via BackupPC-users
>> > <backuppc-users@lists.sourceforge.net> wrote:
>> > >
>> > > Thanks for confirming; it's not a charset issue then.
>> > >
>> > > The empty /home due to --one-file-system from 2017 shouldn't make a
>> > > difference by now, but it's good to keep in mind.
>> > >
>> > > When you look through the XferLOG file, please look out for any other
>> > > unexpected errors. You could also increase the XferLogLevel (eg, to 5
>> > > or 6).
>> > >
>> > > Also, if the backup size is large, you could create a new test host (use
>> > > ClientNameAlias to point it back to the right host) that just backs up
>> > > /home/raman/x or even /home/raman/x/y. If the same problem happens,
>> > > then it will be much easier to browse shorter log files. If the problem
>> > > doesn't happen in that case, then that's a useful clue too.
>> > >
>> > > Craig
>> > >
>> > > On Tue, May 7, 2019 at 11:02 PM Raman Gupta <rocketra...@gmail.com>
>> > > wrote:
>> > >>
>> > >> It is an ext4 filesystem. The directories are plain ASCII -- no
>> > >> strange characters in any way. The `rsync` process on the Linux client
>> > >> runs as root, and I have verified root has access to these files
>> > >> without any issue. There are plenty of inodes free (`df -i` shows that
>> > >> filesystem as only 5% used). There is no file corruption -- all the
>> > >> data is good.
>> > >>
>> > >> There is a seemingly weird coincidence here though... my backups have
>> > >> been running nightly for years. However, in late 2017 I updated
>> > >> BackupPC on my Fedora box, and `--one-file-system` was added to the
>> > >> rsync args without me realizing. This caused all backups from that
>> > >> point forward to a few days ago to be missing all the new files and
>> > >> modifications in `/home`, which is a mountpoint for a local LVM ext4
>> > >> partition. It seems that every file and directory that falls into this
>> > >> category are now failing with this "vanished" error. Is it possible
>> > >> that BackupPC is confused because it is expecting to find these files
>> > >> in prior backups?
>> > >>
>> > >> I will try the debugging you suggest.
>> > >>
>> > >> Regards,
>> > >> Raman
>> > >>
>> > >> On Wed, May 8, 2019 at 1:44 AM Craig Barratt via BackupPC-users
>> > >> <backuppc-users@lists.sourceforge.net> wrote:
>> > >> >
>> > >> > What sort of filesystem is this? Do those directory names contain
>> > >> > non-ascii characters?
>> > >> >
>> > >> > However, the "file has vanished" error shouldn't occur on a
>> > >> > directory, so something strange is going on.
>> > >> >
>> > >> > I'd recommend turning on additional debug in rsync (eg, add -vvv to
>> > >> > $Conf{RsyncArgs}, and also look at the --debug option) and looking in
>> > >> > the XferLOG file. When the initial file list is sent, are those
>> > >> > directories and their contents present in the file list?
>> > >> >
>> > >> > Craig
>> > >> >
>> > >> > On Tue, May 7, 2019 at 4:26 PM Michael Stowe
>> > >> > <michael.st...@member.mensa.org> wrote:
>> > >> >>
>> > >> >> On 2019-05-07 13:39, Raman Gupta wrote:
>> > >> >>
>> > >> >> Certain directories (and their contents) on one of my hosts are not
>> > >> >> getting backed up at all, even with a “Full” backup.
>> > >> >>
>> > >> >> I use rsync as my Xfer method, with BackupPC 4.3.0 on Fedora (rpms
>> > >> >> BackupPC-4.3.0-1.fc29.x86_64, BackupPC-XS-0.58-1.fc29.x86_64).
>> > >> >>
>> > >> >> Looking at the backup logs, I see messages like the following
>> > >> >> related to the directories that are not being backed up:
>> > >> >>
>> > >> >> file has vanished: “/home/raman/x/y/a” file has vanished:
>> > >> >> “/home/raman/x/y/b” file has vanished: “/home/raman/x/y/c”
>> > >> >>
>> > >> >> I have other directories and files successfully backed up in
>> > >> >> “/home/raman/x/y”, but the directories “a”, “b”, and “c” (and their
>> > >> >> content) are not being backed up.
>> > >> >>
>> > >> >> Note that these files have not vanished — they are not ephemeral and
>> > >> >> they haven't been touched in days. For example:
>> > >> >>
>> > >> >> File: /home/raman/x/y/a
>> > >> >> Size: 4096 Blocks: 8 IO Block: 4096 directory
>> > >> >>
>> > >> >> Device: fd08h/64776d Inode: 33037482 Links: 5 Access:
>> > >> >> (0775/drwxrwxr-x) Uid: ( 1000/ raman) Gid: ( 1000/ raman) Context:
>> > >> >> unconfined_u:object_r:user_home_t:s0 Access: 2019-05-07
>> > >> >> 05:05:17.288857497 -0400 Modify: 2019-04-30 00:56:22.914849594 -0400
>> > >> >> Change: 2019-04-30 00:56:22.914849594 -0400 Birth: –
>> > >> >>
>> > >> >> Any idea what might be happening here?
>> > >> >>
>> > >> >> Regards, Raman
>> > >> >>
>> > >> >> “File has vanished” issues can be tricky to diagnose if the file
>> > >> >> appears to be there. What rsync is really telling you is that it
>> > >> >> built a file list, and some of the files or directories from that
>> > >> >> list are not accessible when it actually went to read them. Actually
>> > >> >> being deleted or ephemeral files are two reasons, but there are
>> > >> >> others, from filename encoding issues to inode changes to
>> > >> >> complications with remotely mounted filesystems to corruption issues
>> > >> >> to complex file permissions.
>> > >> >>
>> > >> >> While I might check the file's details both before and after the
>> > >> >> rsync run to look for changes, I recommend ensuring that these files
>> > >> >> are reliably accessible by the rsync user, check the logs for any
>> > >> >> problems, and working through filesystem issues. (XFS is notorious
>> > >> >> for this sort of thing.) Also, if the volume is anything other than
>> > >> >> a local mount, that's where I'd look first for issues; be aware that
>> > >> >> rsync's high read volume often exposes issues not evident under less
>> > >> >> stressful usage.
>> > >> >>
>> > >> >> _______________________________________________
>> > >> >> BackupPC-users mailing list
>> > >> >> BackupPC-users@lists.sourceforge.net
>> > >> >> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> > >> >> Wiki: http://backuppc.wiki.sourceforge.net
>> > >> >> Project: http://backuppc.sourceforge.net/
>> > >> >
>> > >> > _______________________________________________
>> > >> > BackupPC-users mailing list
>> > >> > BackupPC-users@lists.sourceforge.net
>> > >> > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> > >> > Wiki: http://backuppc.wiki.sourceforge.net
>> > >> > Project: http://backuppc.sourceforge.net/
>> > >>
>> > >>
>> > >> _______________________________________________
>> > >> BackupPC-users mailing list
>> > >> BackupPC-users@lists.sourceforge.net
>> > >> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> > >> Wiki: http://backuppc.wiki.sourceforge.net
>> > >> Project: http://backuppc.sourceforge.net/
>> > >
>> > > _______________________________________________
>> > > BackupPC-users mailing list
>> > > BackupPC-users@lists.sourceforge.net
>> > > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> > > Wiki: http://backuppc.wiki.sourceforge.net
>> > > Project: http://backuppc.sourceforge.net/
>>
>>
>> _______________________________________________
>> BackupPC-users mailing list
>> BackupPC-users@lists.sourceforge.net
>> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki: http://backuppc.wiki.sourceforge.net
>> Project: http://backuppc.sourceforge.net/
>
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
