Re: [BackupPC-users] Tar causing problems with removable drive on localhost
On Thu, Jan 19, 2023 at 4:41 AM Adam Hardy wrote: > > 50% of the time, the backup works. When I run the command above as the > backuppc user, it never fails. The removable drive is always connected. > > Following advice from this mailing list in October, I searched all my logs > and found nothing related. > > I also added this to ArchivePreUserCmd: ls -la /media/adam/Videos-2TB/Videos > > However the error has not been solved. > As a wild guess, I'd say the USB drive is going to sleep and takes more time to spin up. Your PreUserCmd is probably the right idea but also failing and continuing too quickly. 'fdisk -l' might be better but needs to run as root, or maybe just adding ";sleep 2' after your ls command would work. -- Les Mikesell lesmikes...@gmail.com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
[BackupPC-users] Tar causing problems with removable drive on localhost
This is picking up something that I didn't have time to sort out in October. I keep getting a problem from tar when backuppc goes to back up the localhost, which includes a removable drive in the backup targets: tarExtract: Done: 0 errors, 1 filesExist, 0 sizeExist, 0 sizeExistComp, 1 filesTotal, 0 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 9243 inodeLast Running: /usr/bin/sudo /bin/tar -c -v -f - -C /home/adam/.ssh --totals --newer=2023-01-19\ 03:00:00 . incr backup started back to 2023-01-19 03:00:00 (backup #1392) for directory /home/adam/.ssh Xfer PIDs are now 40706,40705 [ skipped 1 lines ] readOutput: sysread returns 0 and got EOF tarExtract: /usr/share/backuppc/bin/BackupPC_tarExtract: got Full = tarExtract: /usr/share/backuppc/bin/BackupPC_tarExtract starting... (XferLogLevel = 1) [ skipped 1 lines ] tarExtract: Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 9244 inodeLast Running: /usr/bin/sudo /bin/tar -c -v -f - -C /var/lib/backuppc/.ssh --totals --newer=2023-01-19\ 03:00:00 . incr backup started back to 2023-01-19 03:00:00 (backup #1392) for directory /var/lib/backuppc/.ssh Xfer PIDs are now 40709,40708 [ skipped 1 lines ] readOutput: sysread returns 0 and got EOF tarExtract: /usr/share/backuppc/bin/BackupPC_tarExtract: got Full = tarExtract: /usr/share/backuppc/bin/BackupPC_tarExtract starting... (XferLogLevel = 1) [ skipped 1 lines ] tarExtract: Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 9245 inodeLast Running: /usr/bin/sudo /bin/tar -c -v -f - -C /media/adam/Videos-2TB/Videos --totals --newer=2023-01-19\ 03:00:00 . incr backup started back to 2023-01-19 03:00:00 (backup #1392) for directory /media/adam/Videos-2TB/Videos Xfer PIDs are now 40712,40711 /bin/tar: /media/adam/Videos-2TB/Videos: Cannot open: No such file or directory /bin/tar: Error is not recoverable: exiting now readOutput: sysread returns 0 and got EOF Tar exited with error 512 () status tarExtract: /usr/share/backuppc/bin/BackupPC_tarExtract: got Full = tarExtract: /usr/share/backuppc/bin/BackupPC_tarExtract starting... (XferLogLevel = 1) tarExtract: Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 9246 inodeLast Xfer PIDs are now Got fatal error during xfer (Tar exited with error 512 () status) Backup aborted (Tar exited with error 512 () status) 50% of the time, the backup works. When I run the command above as the backuppc user, it never fails. The removable drive is always connected. Following advice from this mailing list in October, I searched all my logs and found nothing related. I also added this to ArchivePreUserCmd: ls -la /media/adam/Videos- 2TB/Videos However the error has not been solved. Here's the original email in the archive: https://sourceforge.net/p/backuppc/mailman/backuppc-users/thread/ed9ae59-1421-90b0-a546-d5cd95f0facf%40jubileegroup.co.uk/ Have you got any more advice on this for me? Thanks & regards Adam ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Tar causing problems with removable drive on localhost
You shouldn't need to do that if you're running systemd. Everything is written to the journal. Just do journalctl --since=today and you'll get everything since midnight. You really don't need the text logs at all. On 10/20/22 18:28, Kenneth Porter wrote: On 10/20/2022 6:52 AM, G.W. Haywood via BackupPC-users wrote: I checked in the syslog and I can't see any other log files that it might be using. You can spend hours trawling through logs, but mostly I'd search in /var/log/(daemon.log|debug|kern.log|messages|syslog) - not necessarily in that order. A trick I've learned is to issue "ls -lt /var/log | head -20". This will list the 20 log files most recently written to. Alas, it doesn't find journald stuff, and it doesn't look in subdirectories, so you need to do that separately. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project:https://backuppc.github.io/backuppc/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Tar causing problems with removable drive on localhost
I haven't found a good explanation of how the /media mount system works. It seems that mechanism changes frequently. For my CentOS 7 system with a USB external drive for backups, I use systemd mount units to mount the drive when anything tries to touch /var/lib/BackupPC. I'd suggest doing something similar to hold your video library. Find a suitable "permanent" mount point (like /opt/Videos or /mnt/Videos) and create a pair of systemd unit files in /etc/systemd/system. Here's the files I use for my backup drive. The file names are important and should reflect where you plan to mount. Only enable the .automount one. It starts the .mount file when the target directory is touched. The mount options here are from when I needed to upgrade the drive to a larger size and the copy process was extremely slow. # var-lib-BackupPC.automount [Unit] Description = External BackupPC media automount [Automount] Where = /var/lib/BackupPC [Install] WantedBy = multi-user.target ─ # var-lib-BackupPC.mount [Unit] Description = External BackupPC media [Mount] #What = LABEL=BackupPC20211109 What = LABEL=backuppc4a Where = /var/lib/BackupPC Type = ext4 # options for better read performance Options = defaults,noiversion,noatime,nodiratime,relatime,errors=remount-ro,commit=60,delalloc [Install] WantedBy = multi-user.target ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Tar causing problems with removable drive on localhost
On 10/20/2022 6:52 AM, G.W. Haywood via BackupPC-users wrote: I checked in the syslog and I can't see any other log files that it might be using. You can spend hours trawling through logs, but mostly I'd search in /var/log/(daemon.log|debug|kern.log|messages|syslog) - not necessarily in that order. A trick I've learned is to issue "ls -lt /var/log | head -20". This will list the 20 log files most recently written to. Alas, it doesn't find journald stuff, and it doesn't look in subdirectories, so you need to do that separately. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Tar causing problems with removable drive on localhost
Adam Hardy wrote at about 22:25:31 +0100 on Thursday, October 20, 2022: > Looking through those logs for anything at the time of the modification > timestamp on the backuppc xferlog, I can see this one line which > appears about 25 mins after the backup fails: > >Oct 17 16:13:48 localhost smartd[1302]: Device: /dev/sdb [SAT], 1 >Currently unreadable (pending) sectors > > The target USB drive is /dev/sdb1. > > But as I said, I scanned the drive and it didn't log any errors. Is > that useful info? > As others have advised, USB drives are not very reliable for critical storage needs like backup. It's not particularly surprising that issues could occur with large file reads/writes like you have with your 2TB of video. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Tar causing problems with removable drive on localhost
-Original Message-
> Be careful with smartctl if you use it for anything other than
> reading
> information from the drive. Heed the warnings in the 'man' page, and
> before you do anything like setting or changing drive characteristics
> search online for reports from people who've done similar things with
> your particular drive model. One wrong move could be serious. Maybe
> it won't be necessary. I've used it to stop drives from spinning
> down
> in problem cases but only when the timeouts interacted very badly
> with
> the Linux kernel timeouts for flushing sectors from RAM to disc.
> I've
> been prepared to take the risk of damaging drives, but only because
> my
> alternative would have been to put them in the recycling bin. :/
>
> Don't forget I'm still guessing. :)
>
> > I checked in the syslog and I can't see any other log files that it
> > might be using.
>
> You can spend hours trawling through logs, but mostly I'd search in
> /var/log/(daemon.log|debug|kern.log|messages|syslog) - not
> necessarily
> in that order.
>
> > How would I run something automatically before backup? Or do you
> > mean manually?
>
> I meant automatically. I was thinking of one of these:
>
> $Conf{DumpPreUserCmd}
> $Conf{DumpPostUserCmd}
> $Conf{DumpPreShareCmd}
> $Conf{DumpPostShareCmd}
> $Conf{RestorePreUserCmd}
> $Conf{RestorePostUserCmd}
> $Conf{ArchivePreUserCmd}
> $Conf{ArchivePostUserCmd}
>
> Check the documentation for "Optional commands to run before and
> after
> dumps and restores, and also before and after each share of a dump."
Warning registered re smartctl. Thanks for the references to the
config, which immediately draw attention to the fact that I have no
idea what the error is.
Looking through those logs for anything at the time of the modification
timestamp on the backuppc xferlog, I can see this one line which
appears about 25 mins after the backup fails:
Oct 17 16:13:48 localhost smartd[1302]: Device: /dev/sdb [SAT], 1
Currently unreadable (pending) sectors
The target USB drive is /dev/sdb1.
But as I said, I scanned the drive and it didn't log any errors. Is
that useful info?
Thanks
Adam
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Tar causing problems with removable drive on localhost
Hello again,
On Thu, 20 Oct 2022, Adam Hardy wrote:
I scanned the problem USB drive with smartctl and with
gnome-utilities and it logged nothing.?
Be careful with smartctl if you use it for anything other than reading
information from the drive. Heed the warnings in the 'man' page, and
before you do anything like setting or changing drive characteristics
search online for reports from people who've done similar things with
your particular drive model. One wrong move could be serious. Maybe
it won't be necessary. I've used it to stop drives from spinning down
in problem cases but only when the timeouts interacted very badly with
the Linux kernel timeouts for flushing sectors from RAM to disc. I've
been prepared to take the risk of damaging drives, but only because my
alternative would have been to put them in the recycling bin. :/
Don't forget I'm still guessing. :)
I checked in the syslog and I can't see any other log files that it
might be using.
You can spend hours trawling through logs, but mostly I'd search in
/var/log/(daemon.log|debug|kern.log|messages|syslog) - not necessarily
in that order.
How would I run something automatically before backup? Or do you
mean manually?
I meant automatically. I was thinking of one of these:
$Conf{DumpPreUserCmd}
$Conf{DumpPostUserCmd}
$Conf{DumpPreShareCmd}
$Conf{DumpPostShareCmd}
$Conf{RestorePreUserCmd}
$Conf{RestorePostUserCmd}
$Conf{ArchivePreUserCmd}
$Conf{ArchivePostUserCmd}
Check the documentation for "Optional commands to run before and after
dumps and restores, and also before and after each share of a dump."
--
73,
Ged.
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Tar causing problems with removable drive on localhost
-Original Message- > Don't assume what the problem is and try to solve it. First find it. > There may be something interesting in the system logs (in /var/log/). > > Your subject line says "tar causing problems" but I feel sure it > won't > be tar which is causing the problems. I also doubt that it's > anything > as complicated as Ubuntu doing things for GUIs. > > Would I be right in guessing that it's a USB-connected drive? > They've > almost always caused problems for me. USB disconnects happen all the > time, and for most USB devices a reconnect is fine. For mass storage > devices unfortunately that's often not the case. > > Is the drive spinning down when not busy? That's the default for > many > USB devices I've seen and that often causes problems too - but you > can > prevent it from doing that easily enough. Maybe you'd have more luck > if you run something just before the backup takes place which > accesses > the drive, to spin it up, if that's a problem. > > Are you backing up 2TB of videos? I'd probably exclude them from the > regular backups. > > Well I've done enough guessing for one post. :) Thanks Ged. Great guessing. It is a USB-connected drive. In fact I have my whole collection of backuppc backups on another USB drive which has never caused a problem in its life (5 years or so). OK so that's writing to the USB drive, not reading it like the issue here. I scanned the problem USB drive with smartctl and with gnome-utilities and it logged nothing. I checked in the syslog and I can't see any other log files that it might be using. How would I run something automatically before backup? Or do you mean manually? BTW you are right that I'm backing up 2TB of videos. Obviously though, video files never change so theoretically, backuppc should only ever have one copy of each video file. Space hasn't become an issue yet. Thanks Adam ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Tar causing problems with removable drive on localhost
Hi there, On Wed, 19 Oct 2022, Adam Hardy wrote: I keep getting a problem from tar when backuppc goes to back up the localhost, which includes a removable drive in the backup targets: Running: /usr/bin/sudo /bin/tar -c -v -f - -C /media/adam/Videos-2TB/Videos --totals --newer=2022-10-11\ 13:11:32 . incr backup started back to 2022-10-11 13:11:32 (backup #1343) for directory /media/adam/Videos-2TB/Videos Xfer PIDs are now 22832,22831 /bin/tar: /media/adam/Videos-2TB/Videos: Cannot open: No such file or directory /bin/tar: Error is not recoverable: exiting now readOutput: sysread returns 0 and got EOF Tar exited with error 512 () status 50% of the time, the backup works. When I run the command above as the backuppc user, it never fails. I'm not sure what to do now. The removable drive is always connected. I can only assume it has something to do with Ubuntu managing the drive somehow so that it is connected when any GUI process tries to access it, but I'm not sure why it shouldn't fail for me when I run it on the command line. Don't assume what the problem is and try to solve it. First find it. There may be something interesting in the system logs (in /var/log/). Your subject line says "tar causing problems" but I feel sure it won't be tar which is causing the problems. I also doubt that it's anything as complicated as Ubuntu doing things for GUIs. Would I be right in guessing that it's a USB-connected drive? They've almost always caused problems for me. USB disconnects happen all the time, and for most USB devices a reconnect is fine. For mass storage devices unfortunately that's often not the case. Is the drive spinning down when not busy? That's the default for many USB devices I've seen and that often causes problems too - but you can prevent it from doing that easily enough. Maybe you'd have more luck if you run something just before the backup takes place which accesses the drive, to spin it up, if that's a problem. Are you backing up 2TB of videos? I'd probably exclude them from the regular backups. Well I've done enough guessing for one post. :) -- 73, Ged. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
[BackupPC-users] Tar causing problems with removable drive on localhost
I keep getting a problem from tar when backuppc goes to back up the localhost, which includes a removable drive in the backup targets: Running: /usr/bin/sudo /bin/tar -c -v -f - -C /media/adam/Videos-2TB/Videos --totals --newer=2022-10-11\ 13:11:32 . incr backup started back to 2022-10-11 13:11:32 (backup #1343) for directory /media/adam/Videos-2TB/Videos Xfer PIDs are now 22832,22831 /bin/tar: /media/adam/Videos-2TB/Videos: Cannot open: No such file or directory /bin/tar: Error is not recoverable: exiting now readOutput: sysread returns 0 and got EOF Tar exited with error 512 () status 50% of the time, the backup works. When I run the command above as the backuppc user, it never fails. I'm not sure what to do now. The removable drive is always connected. I can only assume it has something to do with Ubuntu managing the drive somehow so that it is connected when any GUI process tries to access it, but I'm not sure why it shouldn't fail for me when I run it on the command line. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
