Re: [Qubes OS Community Forum] [Mailing Lists/qubes-users] [qubes-users] R4.0.4 RC1 Unable to delete or backup certain qubes

[donoban]

Hi

On 11/15/20 8:29 PM, Steve Coleman wrote:
> Since checking each drive in this way is relatively efficient and easy
> it seems to me that there must be an automated way to check these error
> logs and notify the user when a drive is starting to fail. My Qubes
> system was completely silent and it was only because of the odd
> behaviour of the backup system that I was forced to investigate. If the
> backup process didn't just hang then all my future backups could have
> been trash, and I would have not even noticed the issue until it was too
> late. Why wait until the system is completely unusable?
> 
> So, my question to the Qubes community is, has anyone out there set up
> this kind of "smart" disk check up on Qubes? What are the best tools for
> a quick check, say upon each boot, or one that could easilly be put in
> cron for a periodic/daily go-no-go health check?

I personally would recommend btrfs specially if you have a ssd hard
disk. Although it supposes some performance lost you will get a more
reliable data consistency and you can check all your data just doing
"btrfs scrub start /" (or "btrfs scrub start / -c Idle" if you don't
want it lags too much your system while working).

I was using also btrfs-send/receieve but ultimately it seems that there
is some problem that causes a CPU bottleneck with my big non-ssd hard
disk. The main ssd disk stills working fine but I am thinking on another
way for incremental backups.

I would like to experiment with borg so I could do backups at file level
adding some cool features like ignore some paths (e.g. '~/.cache') or
restore a single file without uncompressing/unencrypting the whole image
and also have deduplication.

Regards.

-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/a0820f19-2665-089c-9d9e-a2ae9d410949%40riseup.net.


OpenPGP_0x141310D8E3ED08A5.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature

Re: [Qubes OS Community Forum] [Mailing Lists/qubes-users] [qubes-users] R4.0.4 RC1 Unable to delete or backup certain qubes

[Steve Coleman]

On Sat, Nov 14, 2020 at 12:51 PM Mike Keehan via Qubes OS Community Forum <
qubes...@discoursemail.com> wrote:

> Mike_Keehan 
> November 14
>
> Well, the thin pool is LVM, but if the VM is offline, there should not be
> a problem. Guess you'll have to investigate all the logs you can
> find.
>
I finally have the answer!

Thankfully this problem has nothing to do with R4.0.4 but rather a brand
new disk drive failing (MTBF<=5.2 days, likely earlier)  in a rather odd
way. What had me stumped is why the VMs would would seem to run fine but
completely hung the backup process while reading the exact same volumes. It
appears that all the VMs that were acting odd were all allocated on the
same physical drive, but nothing ever gave any kind of an error when they
were reading the drive. It was likely the per-VM metadata needed for the
backup system that failed first.

Fortunatly the drives built in "smart" log holds the records for the last 4
errors, which can be easilly checked, and this allowed me to identify which
physical drive needed to be yanked and replaced. Being a brand new system I
did not yet know which logical drive mapped to which physical drive. To
analyse the problem I used a "smartctl" tool variant on another system to
check the logs that are stored physically within the drive.

Since checking each drive in this way is relatively efficient and easy it
seems to me that there must be an automated way to check these error logs
and notify the user when a drive is starting to fail. My Qubes system was
completely silent and it was only because of the odd behaviour of the
backup system that I was forced to investigate. If the backup process
didn't just hang then all my future backups could have been trash, and I
would have not even noticed the issue until it was too late. Why wait until
the system is completely unusable?

So, my question to the Qubes community is, has anyone out there set up this
kind of "smart" disk check up on Qubes? What are the best tools for a quick
check, say upon each boot, or one that could easilly be put in cron for a
periodic/daily go-no-go health check?

Thanks,

Steve

> --
>
> Visit Topic
> 
> or reply to this email to respond.
>
> To unsubscribe from these emails, click here
> 
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/CAJ5FDniPDdysU6isxXotQvdkPWgkZ9LFCRVLVbkRwp12%2BMacVA%40mail.gmail.com.

Re: [qubes-users] R4.0.4 RC1 Unable to delete or backup certain qubes

[Mike Keehan]

On 11/14/20 4:39 PM, Steve Coleman wrote:



On Sat, Nov 14, 2020, 4:48 AM Mike Keehan > wrote:


On 11/14/20 3:29 AM, Steve Coleman wrote:
 > I installed R4.0.4 RC1 and have been having some very odd issues
with
 > just a few of the VM's I restored from backups, thus I have been
 > restoring, cloning, testing, and deleting clones while trying to
figure
 > out a few things.
 >
 > The first problem I was originally chasing was why one VM in
particular
 > never completes a backup and just hangs at 0% while the file
grows only
 > to about 40kb (the header info?). That specific VM starts and
runs just
 > fine but just won't complete a backup. A Clone of it runs fine as
well.
 >

>  The backup not completing can occur if the VM is online, and you
are not
using LVM.

The VM is definitely not online/running, and it is not using LVM but
rather the default thin pool mechanism set up by the R4.0.4 RC1
default installer options. This is a brand new install, and backups
were working just fine for about three days without any issues.


Well, the thin pool is LVM, but if the VM is offline, there should not 
be a problem.  Guess you'll have to investigate all the logs you can

find.

Mike.

--
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/9c17f76e-38c1-d595-f627-aedb07f42808%40keehan.net.

Re: [qubes-users] R4.0.4 RC1 Unable to delete or backup certain qubes

[Steve Coleman]

On Sat, Nov 14, 2020, 4:48 AM Mike Keehan  wrote:

> On 11/14/20 3:29 AM, Steve Coleman wrote:
> > I installed R4.0.4 RC1 and have been having some very odd issues with
> > just a few of the VM's I restored from backups, thus I have been
> > restoring, cloning, testing, and deleting clones while trying to figure
> > out a few things.
> >
> > The first problem I was originally chasing was why one VM in particular
> > never completes a backup and just hangs at 0% while the file grows only
> > to about 40kb (the header info?). That specific VM starts and runs just
> > fine but just won't complete a backup. A Clone of it runs fine as well.
> >
>
> > The backup not completing can occur if the VM is online, and you are not
> using LVM.
>
> The VM is definitely not online/running, and it is not using LVM but
> rather the default thin pool mechanism set up by the R4.0.4 RC1 default
> installer options. This is a brand new install, and backups were working
> just fine for about three days without any issues.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/CAJ5FDni-AjV4Fq76K9itYo48pkE%2B4QKms_HEaPDxb_qXciJX7w%40mail.gmail.com.

Re: [qubes-users] R4.0.4 RC1 Unable to delete or backup certain qubes

[Mike Keehan]

On 11/14/20 3:29 AM, Steve Coleman wrote:
I installed R4.0.4 RC1 and have been having some very odd issues with 
just a few of the VM's I restored from backups, thus I have been 
restoring, cloning, testing, and deleting clones while trying to figure 
out a few things.


The first problem I was originally chasing was why one VM in particular 
never completes a backup and just hangs at 0% while the file grows only 
to about 40kb (the header info?). That specific VM starts and runs just 
fine but just won't complete a backup. A Clone of it runs fine as well.




The backup not completing can occur if the VM is online, and you are not
using LVM.

Mike.

--
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/350ff3b9-d889-8e32-2ac4-05d00a69df70%40keehan.net.

[qubes-users] R4.0.4 RC1 Unable to delete or backup certain qubes

[Steve Coleman]

I installed R4.0.4 RC1 and have been having some very odd issues with just
a few of the VM's I restored from backups, thus I have been restoring,
cloning, testing, and deleting clones while trying to figure out a few
things.

The first problem I was originally chasing was why one VM in particular
never completes a backup and just hangs at 0% while the file grows only to
about 40kb (the header info?). That specific VM starts and runs just fine
but just won't complete a backup. A Clone of it runs fine as well.

At present, I now have a few VM's that I am unable to delete for some
reason. Both the command line tool and Qube manager fail to remove these
VM's. From the command line tool I get a "qvm-remove: error: Got empty
response from qubesd." message. In the journalctl I see a "domain not
found" error, when it gets a second exception and then terminates.

I'm guessing that these two problems are likely related, but I'm not sure
how. I'm guessing there is something wrong with qubesd. I have attached the
relevant logs for the delete problem.

Any ideas?

thanks,

Steve Coleman

-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/CAJ5FDngd4MhiA_VzL0CGWmWUvu9gsCtOQ-FdhmNJs%3D8QrK76fw%40mail.gmail.com.

Nov 13 21:55:29 dom0 qubesd[3019]: unhandled exception while calling 
src=b'dom0' meth=b'admin.vm.Remove' dest=b'Win10-clone-1' arg=b'' 
len(untrusted_payload)=0
Nov 13 21:55:29 dom0 qubesd[3019]: Traceback (most recent call last):
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/qubes/vm/qubesvm.py", line 691, in 
libvirt_domain
Nov 13 21:55:29 dom0 qubesd[3019]: self.uuid.bytes)
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/qubes/app.py", line 142, in wrapper
Nov 13 21:55:29 dom0 qubesd[3019]: return self._wrap_domain(attr(*args, 
**kwargs))
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib64/python3.5/site-packages/libvirt.py", line 4047, in lookupByUUID
Nov 13 21:55:29 dom0 qubesd[3019]: if ret is None:raise 
libvirtError('virDomainLookupByUUID() failed', conn=self)
Nov 13 21:55:29 dom0 qubesd[3019]: libvirt.libvirtError: Domain not found
Nov 13 21:55:29 dom0 qubesd[3019]: During handling of the above exception, 
another exception occurred:
Nov 13 21:55:29 dom0 qubesd[3019]: Traceback (most recent call last):
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/qubes/api/__init__.py", line 275, in respond
Nov 13 21:55:29 dom0 qubesd[3019]: untrusted_payload=untrusted_payload)
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib64/python3.5/asyncio/futures.py", line 381, in __iter__
Nov 13 21:55:29 dom0 qubesd[3019]: yield self  # This tells Task to wait 
for completion.
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib64/python3.5/asyncio/tasks.py", line 310, in _wakeup
Nov 13 21:55:29 dom0 qubesd[3019]: future.result()
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib64/python3.5/asyncio/futures.py", line 294, in result
Nov 13 21:55:29 dom0 qubesd[3019]: raise self._exception
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib64/python3.5/asyncio/tasks.py", line 240, in _step
Nov 13 21:55:29 dom0 qubesd[3019]: result = coro.send(None)
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/qubes/api/admin.py", line 1144, in vm_remove
Nov 13 21:55:29 dom0 qubesd[3019]: del self.app.domains[self.dest]
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/qubes/app.py", line 537, in __delitem__
Nov 13 21:55:29 dom0 qubesd[3019]: if vm.libvirt_domain:
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/qubes/vm/qubesvm.py", line 694, in 
libvirt_domain
Nov 13 21:55:29 dom0 qubesd[3019]: self._update_libvirt_domain()
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/qubes/vm/qubesvm.py", line 2126, in 
_update_libvirt_domain
Nov 13 21:55:29 dom0 qubesd[3019]: domain_config = self.create_config_file()
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/qubes/vm/__init__.py", line 373, in 
create_config_file
Nov 13 21:55:29 dom0 qubesd[3019]: ]).render(vm=self)
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/jinja2/environment.py", line 989, in render
Nov 13 21:55:29 dom0 qubesd[3019]: return 
self.environment.handle_exception(exc_info, True)
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/jinja2/environment.py", line 754, in 
handle_exception
Nov 13 21:55:29 dom0 qubesd[3019]: reraise(exc_type, exc_value, tb)
Nov 13 21:55:29 dom0 qubesd[3019]:   File 
"/usr/lib/python3.5/site-packages/jinja2/_compat.py", line 37, in reraise
Nov 13