Re: [qubes-users] Re: Qubes R4.0 broken by 'TypeError: not enough arguments...' for most qvm-* commands
According to the LVM documentation, the thin pool will become read-only for operations that require a change to metadata in the event of the metadata volume becoming exhausted. It seems I was lucky enough to experience the system failure prior to data loss occurring (although that would have cost only another reinstall, as it was my second day using the new OS). On Fri, 13 Apr 2018, 13:34 'awokd' via qubes-users, < qubes-users@googlegroups.com> wrote: > On Fri, April 13, 2018 12:33 am, Pablo Di Noto wrote: > > > Ah! The joy! Now back to R4.0 without massive reinstall. > > > > > > Thanks Marek, awokd and techg...! > > I think you taught me more than I helped on this one! Thanks for sharing > your resolution. > > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "qubes-users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/qubes-users/SV_BlWTvB2g/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > qubes-users+unsubscr...@googlegroups.com. > To post to this group, send email to qubes-users@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/qubes-users/04dac88f740f67b95a41f3d69060abe7.squirrel%40tt3j2x4k5ycaa5zt.onion > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/CAJrT%2BY%3DKtGAHQuu%2Boba24FZ99u5xmE7mqzrnuqvJaXGnn8toQg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [qubes-users] Re: Qubes R4.0 broken by 'TypeError: not enough arguments...' for most qvm-* commands
On Fri, April 13, 2018 12:33 am, Pablo Di Noto wrote: > Ah! The joy! Now back to R4.0 without massive reinstall. > > > Thanks Marek, awokd and techg...! I think you taught me more than I helped on this one! Thanks for sharing your resolution. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/04dac88f740f67b95a41f3d69060abe7.squirrel%40tt3j2x4k5ycaa5zt.onion. For more options, visit https://groups.google.com/d/optout.
Re: [qubes-users] Re: Qubes R4.0 broken by "TypeError: not enough arguments..." for most qvm-* commands
> > > > You could also try to revert to earlier revision using "qvm-volume > > revert sys-net:private" for example. > > Will try that tonight. > Unfortunately, that was not possible for the service VMs. For the only important AppVM that had -back and -back missing, the command `qvm-block revert p-2018d:private` was enough. > > > I remember there was some Saltstack magic to recreate the services vms, > > > but could not find anything for R4.0... So I had to revert to R3.2 for > > > the time being. > > > > https://www.qubes-os.org/doc/salt/ > > Especially links at the bottom: > > https://github.com/QubesOS/qubes-mgmt-salt-dom0-virtual-machines/blob/master/README.rst > > Thanks. Will also try that tonight. The way I recreated all sys-* service VMs was: # You need to unset sys-firewall as netvm for AppVMs # and unset sys-net as netvm for sys-firewall: dom0$ qvm-prefs --set netvm '' dom0$ qvm-prefs --set sys-firewall netvm '' # The delete all service VMs qubes: dom0$ qvm-remove sys-usb dom0$ qvm-remove sys-firewall dom0$ qvm-remove sys-net # Make sure the right salt top files are enabled: dom0$ sudo qubesctl top.enable qvm.sys-net dom0$ sudo qubesctl top.enable qvm.sys-firewall dom0$ sudo qubesctl top.enable qvm.sys-usb dom0$ sudo qubesctl state.highstate and that should be all. You have to restore 'sys-firewall' as netvm for all the qubes you want networking and thing should be pretty much as they were after install. Note that all new service VMs will have different IP address than their deceased counterparts, but firewall would be able to cope with that. Ah! The joy! Now back to R4.0 without massive reinstall. Thanks Marek, awokd and techg...! -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/d343c920-7401-475d-a340-39df7c4b3afa%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [qubes-users] Re: Qubes R4.0 broken by 'TypeError: not enough arguments...' for most qvm-* commands
Check for Meta% above 80% On Thu, 12 Apr 2018 at 21:21 awokd wrote: > On Thu, April 12, 2018 3:20 am, techgee...@gmail.com wrote: > > On Wednesday, 11 April 2018 08:29:48 UTC+12, Pablo Di Noto wrote: > > > >> Hello, > >> > >> > >> Any debugging tips? > >> > > > > You have almost exactly described my scenario. > > > > > > You can work around the TypeError exception by editing > > /usr/lib/python3.5/site-packages/qubesadmin/exc.py, as I did. Please > > note, however, that the lines _must_ be indented with spaces (not tabs); > > as Python3 is very particular about indentation style. > > > > If, as I suspect, the root cause of your problem is a lack of metadata > > space on pool00; you can confirm this by typing "sudo lvs" into a > > console. You will then need to figure out a way to enlarge that metadata > > volume. > > That's certainly a non-intuitive failure mode. How did you find that? I'm > not experiencing it myself, but what would one look for in "sudo lvs"- > Meta% at 100% on one of the pools? > > > -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/CAJrT%2BYmYVZtk%3DWQDNEOUx%3DgSU5M9svf%3DH_Tev3Q0GXwCHveCTQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [qubes-users] Re: Qubes R4.0 broken by "TypeError: not enough arguments..." for most qvm-* commands
> > > If, as I suspect, the root cause of your problem is a lack of metadata > > > space on pool00; you can confirm this by typing "sudo lvs" into a > > > console. You will then need to figure out a way to enlarge that metadata > > > volume. > > > > Yes, you are right, the `pool00` volume metadata was >96% when this > > happened. The thing is that the volume metadata was set to a quite small > > size after install (96mb on a 46gb pool) and after install was on ~20% > > usage. I started to use the system, testing stuff with DispVMs, restoring > > my debian templates and some work VMs. After a couple of days of usage the > > metadata climbed very little, to 27-28%. > > > > I tried to have a second pool to hold my machines, precisely to avoid > > issues with thin provisioning on the pool holding `root` and `swap` and > > services vms. But the lack of support for cloning/moving between pools made > > that effort moot. > > > > So I `lvextend`ed `pool00` and forgot to properly enlarge it's > > `pool00_tmeta` counterpart. > > What sizes you have there? > For me tmeta is 118MB for a ~450GB pool00. And after few months of usage > it's still at 33%... An install on a 60G disk partition had a pool00 of 43G created with a pool00_tmeta of 44M (11 extents). Later, the pool00 was extended to 147.7G and the pool00_tmeta left as is by mistake. The metadata became full, and after that the pool00_tmeta was extended to 300M by adding 256M. > > When doing some more customization, including restoring more larger sized > > qubes and cloning/renaming qubes it seems the metadata usage climbed really > > fast and hit this bug. > > > > Unfortunately, could not recover from that. > > > > It looks like qubes lvm actions while metadata was full may have corrupted > > the metadata somehow, since I could enlarge and repair the thin metadata > > from a live cd, but many of the volumes that where in use where never > > available again. The -private and -snap for the qubes that were running > > (not sure how to discard them) and also all the volumes of the qubes being > > restored and services vms are lost ("NOT available" as lvm status) > > You could also try to revert to earlier revision using "qvm-volume > revert sys-net:private" for example. Will try that tonight. > > I remember there was some Saltstack magic to recreate the services vms, but > > could not find anything for R4.0... So I had to revert to R3.2 for the time > > being. > > https://www.qubes-os.org/doc/salt/ > Especially links at the bottom: > https://github.com/QubesOS/qubes-mgmt-salt-dom0-virtual-machines/blob/master/README.rst Thanks. Will also try that tonight. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/cd525562-ead8-4559-9326-c31787b438dc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [qubes-users] Re: Qubes R4.0 broken by "TypeError: not enough arguments..." for most qvm-* commands
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Thu, Apr 12, 2018 at 08:19:10AM -0700, Pablo Di Noto wrote: > > techg...@gmail.com > > If, as I suspect, the root cause of your problem is a lack of metadata > > space on pool00; you can confirm this by typing "sudo lvs" into a console. > > You will then need to figure out a way to enlarge that metadata volume. > > Yes, you are right, the `pool00` volume metadata was >96% when this happened. > The thing is that the volume metadata was set to a quite small size after > install (96mb on a 46gb pool) and after install was on ~20% usage. I started > to use the system, testing stuff with DispVMs, restoring my debian templates > and some work VMs. After a couple of days of usage the metadata climbed very > little, to 27-28%. > > I tried to have a second pool to hold my machines, precisely to avoid issues > with thin provisioning on the pool holding `root` and `swap` and services > vms. But the lack of support for cloning/moving between pools made that > effort moot. > > So I `lvextend`ed `pool00` and forgot to properly enlarge it's > `pool00_tmeta` counterpart. What sizes you have there? For me tmeta is 118MB for a ~450GB pool00. And after few months of usage it's still at 33%... > When doing some more customization, including restoring more larger sized > qubes and cloning/renaming qubes it seems the metadata usage climbed really > fast and hit this bug. > > Unfortunately, could not recover from that. > > It looks like qubes lvm actions while metadata was full may have corrupted > the metadata somehow, since I could enlarge and repair the thin metadata from > a live cd, but many of the volumes that where in use where never available > again. The -private and -snap for the qubes that were running (not sure how > to discard them) and also all the volumes of the qubes being restored and > services vms are lost ("NOT available" as lvm status) You could also try to revert to earlier revision using "qvm-volume revert sys-net:private" for example. > I remember there was some Saltstack magic to recreate the services vms, but > could not find anything for R4.0... So I had to revert to R3.2 for the time > being. https://www.qubes-os.org/doc/salt/ Especially links at the bottom: https://github.com/QubesOS/qubes-mgmt-salt-dom0-virtual-machines/blob/master/README.rst > I will keep the failing install for debugging, or may be able to recover if > someone can provide any tips about: > > - How to recreate sys-net, sys-firewall and sys-usb on a R4.0 system > - how to recover a qube whose -snap volumes are no longer available (I have > no problem losing these short-term data) > > Thanks for pointing to the right direction! > - -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlrMuusACgkQ24/THMrX 1yy3lQf/cO0oe9uOUviiKwgdf6+fEzhCbn6XUkmAU7MLLAkYC1uCAwE3DoT8MBGt bbGkpmWq9gijUCJeWzUD0Z2k1QkZWDdiMgEE8nSgiqyS1O6uNxqqO0ucozWe69Ud FWwmxkCATwX+FK239+HJSO9Jq6/Izb59qbvB1kwewQheqGkZVF9ISNE3AopkMjG8 4RBy1J0dVjHH3wxHtl9N3Z6/4mVwFquLwlE7cM+kTRpFfPtwvrBrNavfYTrEX5lz ALBvsh/eunXBOmc4FNSGHj2yaKnNZibfBVDOoBGaexXt1G0ykpu9aou8tQrKv0zl FqhhNHp9DeOdHm3kP0h1d6PZW1EGiw== =5Ksa -END PGP SIGNATURE- -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/20180412184252.GB2275%40mail-itl. For more options, visit https://groups.google.com/d/optout.
[qubes-users] Re: Qubes R4.0 broken by "TypeError: not enough arguments..." for most qvm-* commands
> techg...@gmail.com > If, as I suspect, the root cause of your problem is a lack of metadata space > on pool00; you can confirm this by typing "sudo lvs" into a console. You > will then need to figure out a way to enlarge that metadata volume. Yes, you are right, the `pool00` volume metadata was >96% when this happened. The thing is that the volume metadata was set to a quite small size after install (96mb on a 46gb pool) and after install was on ~20% usage. I started to use the system, testing stuff with DispVMs, restoring my debian templates and some work VMs. After a couple of days of usage the metadata climbed very little, to 27-28%. I tried to have a second pool to hold my machines, precisely to avoid issues with thin provisioning on the pool holding `root` and `swap` and services vms. But the lack of support for cloning/moving between pools made that effort moot. So I `lvextend`ed `pool00` and forgot to properly enlarge it's `pool00_tmeta` counterpart. When doing some more customization, including restoring more larger sized qubes and cloning/renaming qubes it seems the metadata usage climbed really fast and hit this bug. Unfortunately, could not recover from that. It looks like qubes lvm actions while metadata was full may have corrupted the metadata somehow, since I could enlarge and repair the thin metadata from a live cd, but many of the volumes that where in use where never available again. The -private and -snap for the qubes that were running (not sure how to discard them) and also all the volumes of the qubes being restored and services vms are lost ("NOT available" as lvm status) I remember there was some Saltstack magic to recreate the services vms, but could not find anything for R4.0... So I had to revert to R3.2 for the time being. I will keep the failing install for debugging, or may be able to recover if someone can provide any tips about: - How to recreate sys-net, sys-firewall and sys-usb on a R4.0 system - how to recover a qube whose -snap volumes are no longer available (I have no problem losing these short-term data) Thanks for pointing to the right direction! -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/6dbc17a5-6f49-46ac-8e5c-902eae7ee44d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [qubes-users] Re: Qubes R4.0 broken by 'TypeError: not enough arguments...' for most qvm-* commands
On Thu, April 12, 2018 3:20 am, techgee...@gmail.com wrote: > On Wednesday, 11 April 2018 08:29:48 UTC+12, Pablo Di Noto wrote: > >> Hello, >> >> >> Any debugging tips? >> > > You have almost exactly described my scenario. > > > You can work around the TypeError exception by editing > /usr/lib/python3.5/site-packages/qubesadmin/exc.py, as I did. Please > note, however, that the lines _must_ be indented with spaces (not tabs); > as Python3 is very particular about indentation style. > > If, as I suspect, the root cause of your problem is a lack of metadata > space on pool00; you can confirm this by typing "sudo lvs" into a > console. You will then need to figure out a way to enlarge that metadata > volume. That's certainly a non-intuitive failure mode. How did you find that? I'm not experiencing it myself, but what would one look for in "sudo lvs"- Meta% at 100% on one of the pools? -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/7d73628942f687b771626a3e16fd2b4f.squirrel%40tt3j2x4k5ycaa5zt.onion. For more options, visit https://groups.google.com/d/optout.
[qubes-users] Re: Qubes R4.0 broken by "TypeError: not enough arguments..." for most qvm-* commands
On Wednesday, 11 April 2018 08:29:48 UTC+12, Pablo Di Noto wrote: > Hello, > > Any debugging tips? You have almost exactly described my scenario. You can work around the TypeError exception by editing /usr/lib/python3.5/site-packages/qubesadmin/exc.py, as I did. Please note, however, that the lines _must_ be indented with spaces (not tabs); as Python3 is very particular about indentation style. If, as I suspect, the root cause of your problem is a lack of metadata space on pool00; you can confirm this by typing "sudo lvs" into a console. You will then need to figure out a way to enlarge that metadata volume. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/c2e5bb2f-bc9d-436f-9b9c-23ca0bc2357c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[qubes-users] Re: Qubes R4.0 broken by "TypeError: not enough arguments..." for most qvm-* commands
And also, there is possible reason for this to happen as stated [here](https://github.com/QubesOS/qubes-issues/issues/3809). -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/f2ef3c50-ae7c-49c9-b61c-8294462a7ef6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[qubes-users] Re: Qubes R4.0 broken by "TypeError: not enough arguments..." for most qvm-* commands
It seems that there are other users with similar issues: https://github.com/QubesOS/qubes-issues/issues/3810 (not 100% sure is the same issue, but I have seen that message from UI tools while having this problem) -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/499efc2e-92ac-4741-a6e4-5e9264cd8977%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.