Thanks for the update.
Can you re-create the problem? It would be great to track down what went
wrong. (I still need to look at your log file snippets in more detail,
hopefully this weekend.)
Craig
On Wed, May 8, 2019 at 10:24 PM Raman Gupta <rocketra...@gmail.com> wrote:
> So the issue was definitely with the prior backups, probably related
> to borkage due to the cancellation / partial backup I mentioned in my
> previous message. The last six or seven backups were all incrementals,
> with the last one being filled. I had tried deleting the last filled
> incremental several times, but this had never fixed the problem --
> probably because deleting that backup caused bpc to fill the prior
> incremental, and whatever was borked in the deleted backup was copied
> over to the prior one. The solution was to delete all the backups all
> the way back to the previous filled full backup. Once this was done, a
> manually triggered full backup then finally completed with no issues.
>
> Regards,
> Raman
>
> On Wed, May 8, 2019 at 11:32 AM Raman Gupta <rocketra...@gmail.com> wrote:
> >
> > The backup is large, but not huge. I created a new test host pointing
> > only to /home/raman/x and it worked just fine. I forgot to mention I
> > also have another BackupPC server, backing up the same client with the
> > same configuration, and it backs up all of these files from this
> > client without any issue, so the issue is definitely on the server
> > side, not on the client side.
> >
> > I ran a test with XferLogLevel=6 and "-vvv". Here are some relevant
> > grep outputs -- `home/raman/x/y/2018` is a directory created in 2018,
> > while all of the previously created directories in `y` are backing up
> > without any issue --- this happens to be tax data so I have one
> > directory for every year, and all 2015 directories and before are
> > fine, but 2016 and later are not:
> >
> > $ BackupPC_zcat /var/lib/BackupPC/pc/edison/XferLOG.3972.z | grep
> "x/y/2018"
> > [sender] showing directory home/raman/x/y/2018 because of pattern
> /home/**
> > [sender] make_file(home/raman/x/y/2018,*,2)
> > recv_file_name(home/raman/x/y/2018)
> > recv_file_name(home/raman/x/y/2018)
> > G bpc_lstat(home/raman/x/y/2018)
> > file has vanished: "/home/raman/x/y/2018"
> > recv_generator(home/raman/x/y/2018,216871)
> > G bpc_lstat(home/raman/x/y/2018)
> > G bpc_mkdir(home/raman/x/y/2018, 0700)
> > [sender] pushing local filters for /home/raman/x/y/2018/
> > [sender] showing file home/raman/x/y/2018/a.txt.pdf because of pattern
> /home/**
> > [... same thing for all other subdirectories and files ...]
> > recv_file_name(home/raman/x/y/2018/a.txt.pdf)
> > [... same thing for all other subdirectories and files ...]
> > recv_generator(home/raman/x/y/2018,217359)
> > G bpc_lstat(home/raman/x/y/2018)
> > G bpc_mkdir(home/raman/x/y/2018, 040775)
> > G bpc_lstat(home/raman/x/y/2018)
> > rsync_bpc: stat "/home/raman/x/y/2018" failed: No such file or directory
> (2)
> > delete_in_dir(home/raman/x/y/2018)
> > [generator] pushing local filters for /home/raman/x/y/2018/
> > G bpc_opendir(home/raman/x/y/2018)
> > G bpc_lstat(home/raman/x/y/2018/a.txt.pdf)
> > file has vanished: "/home/raman/x/y/2018/a.txt.pdf"
> > [... same thing for all other subdirectories and files ...]
> > recv_generator(home/raman/x/y/2018/a.txt.pdf,217366)
> > G bpc_lstat(home/raman/x/y/2018/a.txt.pdf)
> > G bpc_sysCall_poolFileCheck(home/raman/x/y/2018/a.txt.pdf): potential
> > match /var/lib/BackupPC//cpool/28/56/29571824e1fe8922304bd1924f02ce22
> > (len = 171990)
> > G bpc_lstat(home/raman/x/y/2018/a.txt.pdf)
> > G bpc_open(home/raman/x/y/2018/a.txt.pdf, 0x0, 00) -> 3
> > G bpc_lgetxattr(home/raman/x/y/2018/a.txt.pdf, user.rsync.%aacl)
> > G bpc_llistxattr(home/raman/x/y/2018/a.txt.pdf)
> > G bpc_read(3 (home/raman/x/y/2018/a.txt.pdf), buf, 171990) tmpFd = -1
> > G bpc_close(3 (home/raman/x/y/2018/a.txt.pdf))
> > [... same thing for all other subdirectories and files ...]
> >
> > and some context around the "file has vanished" message:
> >
> > $ BackupPC_zcat /var/lib/BackupPC/pc/edison/XferLOG.3972.z | grep -C
> > 5 'file has vanished: "/home/raman/x/y/2018"'
> > G bpc_readdir -> 1996
> > G bpc_lstat(home/raman/x/y/1996)
> > [generator] make_file(home/raman/x/y/1996,*,2)
> > G bpc_readdir -> 2018
> > G bpc_lstat(home/raman/x/y/2018)
> > file has vanished: "/home/raman/x/y/2018"
> > G bpc_readdir -> 2014
> > G bpc_lstat(home/raman/x/y/2014)
> > [generator] make_file(home/raman/x/y/2014,*,2)
> > G bpc_readdir -> 2011
> > G bpc_lstat(home/raman/x/y/2011)
> >
> > Lastly, I've tried a test of placing a new file into the 2015
> > directory, and this file does back up successfully. So it seems like
> > new files are captured, as long as they fall into an existing
> > directory, but new directories are not.
> >
> > One thing I did do on this server is cancel the first full backup
> > after removing `--one-file-system` and deleted the backup from the web
> > UI, whereas the other server completed its backup without
> > intervention. I wonder if the server pool was corrupted somehow by
> > doing that? Also, the `--one-file-system` was probably added at the
> > same time as v4 and the resulting pool migrations on subsequent
> > backups, which would have *removed* a lot of stuff that was being
> > backed up by v3. I have run a BackupPC_fsck and there were no errors.
> >
> > Regards,
> > Raman
> >
> > On Wed, May 8, 2019 at 2:11 AM Craig Barratt via BackupPC-users
> > <backuppc-users@lists.sourceforge.net> wrote:
> > >
> > > Thanks for confirming; it's not a charset issue then.
> > >
> > > The empty /home due to --one-file-system from 2017 shouldn't make a
> difference by now, but it's good to keep in mind.
> > >
> > > When you look through the XferLOG file, please look out for any other
> unexpected errors. You could also increase the XferLogLevel (eg, to 5 or
> 6).
> > >
> > > Also, if the backup size is large, you could create a new test host
> (use ClientNameAlias to point it back to the right host) that just backs up
> /home/raman/x or even /home/raman/x/y. If the same problem happens, then
> it will be much easier to browse shorter log files. If the problem doesn't
> happen in that case, then that's a useful clue too.
> > >
> > > Craig
> > >
> > > On Tue, May 7, 2019 at 11:02 PM Raman Gupta <rocketra...@gmail.com>
> wrote:
> > >>
> > >> It is an ext4 filesystem. The directories are plain ASCII -- no
> > >> strange characters in any way. The `rsync` process on the Linux client
> > >> runs as root, and I have verified root has access to these files
> > >> without any issue. There are plenty of inodes free (`df -i` shows that
> > >> filesystem as only 5% used). There is no file corruption -- all the
> > >> data is good.
> > >>
> > >> There is a seemingly weird coincidence here though... my backups have
> > >> been running nightly for years. However, in late 2017 I updated
> > >> BackupPC on my Fedora box, and `--one-file-system` was added to the
> > >> rsync args without me realizing. This caused all backups from that
> > >> point forward to a few days ago to be missing all the new files and
> > >> modifications in `/home`, which is a mountpoint for a local LVM ext4
> > >> partition. It seems that every file and directory that falls into this
> > >> category are now failing with this "vanished" error. Is it possible
> > >> that BackupPC is confused because it is expecting to find these files
> > >> in prior backups?
> > >>
> > >> I will try the debugging you suggest.
> > >>
> > >> Regards,
> > >> Raman
> > >>
> > >> On Wed, May 8, 2019 at 1:44 AM Craig Barratt via BackupPC-users
> > >> <backuppc-users@lists.sourceforge.net> wrote:
> > >> >
> > >> > What sort of filesystem is this? Do those directory names contain
> non-ascii characters?
> > >> >
> > >> > However, the "file has vanished" error shouldn't occur on a
> directory, so something strange is going on.
> > >> >
> > >> > I'd recommend turning on additional debug in rsync (eg, add -vvv to
> $Conf{RsyncArgs}, and also look at the --debug option) and looking in the
> XferLOG file. When the initial file list is sent, are those directories
> and their contents present in the file list?
> > >> >
> > >> > Craig
> > >> >
> > >> > On Tue, May 7, 2019 at 4:26 PM Michael Stowe <
> michael.st...@member.mensa.org> wrote:
> > >> >>
> > >> >> On 2019-05-07 13:39, Raman Gupta wrote:
> > >> >>
> > >> >> Certain directories (and their contents) on one of my hosts are
> not getting backed up at all, even with a “Full” backup.
> > >> >>
> > >> >> I use rsync as my Xfer method, with BackupPC 4.3.0 on Fedora (rpms
> BackupPC-4.3.0-1.fc29.x86_64, BackupPC-XS-0.58-1.fc29.x86_64).
> > >> >>
> > >> >> Looking at the backup logs, I see messages like the following
> related to the directories that are not being backed up:
> > >> >>
> > >> >> file has vanished: “/home/raman/x/y/a” file has vanished:
> “/home/raman/x/y/b” file has vanished: “/home/raman/x/y/c”
> > >> >>
> > >> >> I have other directories and files successfully backed up in
> “/home/raman/x/y”, but the directories “a”, “b”, and “c” (and their
> content) are not being backed up.
> > >> >>
> > >> >> Note that these files have not vanished — they are not ephemeral
> and they haven't been touched in days. For example:
> > >> >>
> > >> >> File: /home/raman/x/y/a
> > >> >> Size: 4096 Blocks: 8 IO Block: 4096 directory
> > >> >>
> > >> >> Device: fd08h/64776d Inode: 33037482 Links: 5 Access:
> (0775/drwxrwxr-x) Uid: ( 1000/ raman) Gid: ( 1000/ raman) Context:
> unconfined_u:object_r:user_home_t:s0 Access: 2019-05-07 05:05:17.288857497
> -0400 Modify: 2019-04-30 00:56:22.914849594 -0400 Change: 2019-04-30
> 00:56:22.914849594 -0400 Birth: –
> > >> >>
> > >> >> Any idea what might be happening here?
> > >> >>
> > >> >> Regards, Raman
> > >> >>
> > >> >> “File has vanished” issues can be tricky to diagnose if the file
> appears to be there. What rsync is really telling you is that it built a
> file list, and some of the files or directories from that list are not
> accessible when it actually went to read them. Actually being deleted or
> ephemeral files are two reasons, but there are others, from filename
> encoding issues to inode changes to complications with remotely mounted
> filesystems to corruption issues to complex file permissions.
> > >> >>
> > >> >> While I might check the file's details both before and after the
> rsync run to look for changes, I recommend ensuring that these files are
> reliably accessible by the rsync user, check the logs for any problems, and
> working through filesystem issues. (XFS is notorious for this sort of
> thing.) Also, if the volume is anything other than a local mount, that's
> where I'd look first for issues; be aware that rsync's high read volume
> often exposes issues not evident under less stressful usage.
> > >> >>
> > >> >> _______________________________________________
> > >> >> BackupPC-users mailing list
> > >> >> BackupPC-users@lists.sourceforge.net
> > >> >> List:
> https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > >> >> Wiki: http://backuppc.wiki.sourceforge.net
> > >> >> Project: http://backuppc.sourceforge.net/
> > >> >
> > >> > _______________________________________________
> > >> > BackupPC-users mailing list
> > >> > BackupPC-users@lists.sourceforge.net
> > >> > List:
> https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > >> > Wiki: http://backuppc.wiki.sourceforge.net
> > >> > Project: http://backuppc.sourceforge.net/
> > >>
> > >>
> > >> _______________________________________________
> > >> BackupPC-users mailing list
> > >> BackupPC-users@lists.sourceforge.net
> > >> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > >> Wiki: http://backuppc.wiki.sourceforge.net
> > >> Project: http://backuppc.sourceforge.net/
> > >
> > > _______________________________________________
> > > BackupPC-users mailing list
> > > BackupPC-users@lists.sourceforge.net
> > > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > > Wiki: http://backuppc.wiki.sourceforge.net
> > > Project: http://backuppc.sourceforge.net/
>
>
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
