Re: [Bacula-users] Tricky restore issue (bacula 11)
Hi folks, thanks for your comments. We managed to clone the damaged VM, booted it using grml and were able to undo the damage done by the mistaken "find" command. We're now looking for a downtime to fix the production system in the next few days hopefully. All the best, Uwe ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tricky restore issue (bacula 11)
Hi Uwe, Josip, all, Am 31.10.2022 um 14:37 schrieb Josip Deanovic: On 2022-10-31 12:46, Uwe Schuerkamp wrote: Hi folks, due to user error part of a file system on one of our backup clients was compressed by a find command gone haywire. Among them are /usr/bin/bash & others (kernel, initrd, grub, root, authorized_keys etc), leading to a situation where we cannot log into the system remotely nor via the vmware console function anymore. I would assume that also quite a few libraries are now compressed. I've tried to restore /usr/bin/bash by its uncompressed version as the bacula-fd on the client is still running (today's incr. backup worked fine, too), but once the restore starts I'm getting the following error: 31-Oct 12:23 deniol2378-sd JobId 23316: Forward spacing Volume "deX-0183" to addr=269 31-Oct 12:23 deniol2378-sd JobId 23316: Elapsed time=00:00:01, Transfer rate=484.9 K Bytes/second 31-Oct 12:22 bacula-fd JobId 23316: Error: create_file.c:227 Could not create /usr/bin/bash: ERR=Permission denied That looks like a real challenge... however, for practical reasons, I would tend to agree with Josip: Boot from recovery media, and start fixing from there (possibly even reverting the find-triggered compression instead of restoring). The actual problem might be related to any sort of libraries or configuration no longer usable -- user authentication, permission checks, ACL processing may all need to do all sorts of getpw* calls. ... Replace: Always Start time: 31-Oct-2022 12:23:02 End time: 31-Oct-2022 12:23:02 Elapsed time: 1 sec Files Expected: 1 Files Restored: 1 Bytes Restored: 0 (0 B) Rate: 0.0 KB/s FD Errors: 0 FD termination status: OK SD termination status: OK Termination: Restore OK That does not look correct, although a system that messed up is no good foundation for a bug report :-) I'm clutching at straws here naturally, however before we rebuild the VM in question from a full backup I'd like to try as many options as possible. What makes this case even worse is that LVM is being used for / & other fileystems, so we cannot simply mount those drives on another VM (3 physical volumes) to repair the damage, right? Might actually be possible, although a bit risky. But there are snapshots for that purpose :-) Thanks for reading this far & for your help, Hello, It might be that your file systems are now mounted read-only and bacula-fd cannot write to file systems. Are you able to connect to the system and run any command? Unlikely, I think. If not, you will have to boot the system using boot iso image, prepare network, prepare LVM, mount file systems, copy the necessary bacula files, chroot into the file system, start bacula-fd and restore the data. Or give systemrescuecd a try. You should be able to boot that from an image / virtual CD, and volume management, file system maintenance, and (de)compression tools should be available. This sort of problem is possibly more quickly recovered from using a new system installation, FD installation, and then restoration of the actual data. The necessary forensic work is just too time consuming (for me at least...) Good luck! Arno -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tricky restore issue (bacula 11)
On 2022-10-31 12:46, Uwe Schuerkamp wrote: Hi folks, due to user error part of a file system on one of our backup clients was compressed by a find command gone haywire. Among them are /usr/bin/bash & others (kernel, initrd, grub, root, authorized_keys etc), leading to a situation where we cannot log into the system remotely nor via the vmware console function anymore. I've tried to restore /usr/bin/bash by its uncompressed version as the bacula-fd on the client is still running (today's incr. backup worked fine, too), but once the restore starts I'm getting the following error: 31-Oct 12:23 deniol2378-sd JobId 23316: Forward spacing Volume "deX-0183" to addr=269 31-Oct 12:23 deniol2378-sd JobId 23316: Elapsed time=00:00:01, Transfer rate=484.9 K Bytes/second 31-Oct 12:22 bacula-fd JobId 23316: Error: create_file.c:227 Could not create /usr/bin/bash: ERR=Permission denied 31-Oct 12:23 deniol2378-dir JobId 23316: Bacula deYY-dir 13.0.0 (04Jul22): the bacula-fd runs as root on the client so it should have no issue creating the file in the local filesystem... any idea what could be causing this? It's also weird that I'm seeing a "restore OK" message at the end of the job output: Replace:Always Start time: 31-Oct-2022 12:23:02 End time: 31-Oct-2022 12:23:02 Elapsed time: 1 sec Files Expected: 1 Files Restored: 1 Bytes Restored: 0 (0 B) Rate: 0.0 KB/s FD Errors: 0 FD termination status: OK SD termination status: OK Termination:Restore OK I'm clutching at straws here naturally, however before we rebuild the VM in question from a full backup I'd like to try as many options as possible. What makes this case even worse is that LVM is being used for / & other fileystems, so we cannot simply mount those drives on another VM (3 physical volumes) to repair the damage, right? Thanks for reading this far & for your help, Hello, It might be that your file systems are now mounted read-only and bacula-fd cannot write to file systems. Are you able to connect to the system and run any command? If not, you will have to boot the system using boot iso image, prepare network, prepare LVM, mount file systems, copy the necessary bacula files, chroot into the file system, start bacula-fd and restore the data. If yes, you could try to remount file systems rw. E.g. mount -o remount,rw / If os is heavily broken, you could help yourself using the statically linked busybox tool. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Tricky restore issue (bacula 11)
Hi folks, due to user error part of a file system on one of our backup clients was compressed by a find command gone haywire. Among them are /usr/bin/bash & others (kernel, initrd, grub, root, authorized_keys etc), leading to a situation where we cannot log into the system remotely nor via the vmware console function anymore. I've tried to restore /usr/bin/bash by its uncompressed version as the bacula-fd on the client is still running (today's incr. backup worked fine, too), but once the restore starts I'm getting the following error: 31-Oct 12:23 deniol2378-sd JobId 23316: Forward spacing Volume "deX-0183" to addr=269 31-Oct 12:23 deniol2378-sd JobId 23316: Elapsed time=00:00:01, Transfer rate=484.9 K Bytes/second 31-Oct 12:22 bacula-fd JobId 23316: Error: create_file.c:227 Could not create /usr/bin/bash: ERR=Permission denied 31-Oct 12:23 deniol2378-dir JobId 23316: Bacula deYY-dir 13.0.0 (04Jul22): the bacula-fd runs as root on the client so it should have no issue creating the file in the local filesystem... any idea what could be causing this? It's also weird that I'm seeing a "restore OK" message at the end of the job output: Replace:Always Start time: 31-Oct-2022 12:23:02 End time: 31-Oct-2022 12:23:02 Elapsed time: 1 sec Files Expected: 1 Files Restored: 1 Bytes Restored: 0 (0 B) Rate: 0.0 KB/s FD Errors: 0 FD termination status: OK SD termination status: OK Termination:Restore OK I'm clutching at straws here naturally, however before we rebuild the VM in question from a full backup I'd like to try as many options as possible. What makes this case even worse is that LVM is being used for / & other fileystems, so we cannot simply mount those drives on another VM (3 physical volumes) to repair the damage, right? Thanks for reading this far & for your help, Uwe ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
