Re: [qubes-users] Re: Qubes R4.0 broken by 'TypeError: not enough arguments...' for most qvm-* commands

2018-04-16 Thread Brendon Green 
According to the LVM documentation, the thin pool will become read-only for
operations that require a change to metadata in the event of the metadata
volume becoming exhausted. It seems I was lucky enough to experience the
system failure prior to data loss occurring (although that would have cost
only another reinstall, as it was my second day using the new OS).

On Fri, 13 Apr 2018, 13:34 'awokd' via qubes-users, <
qubes-users@googlegroups.com> wrote:

> On Fri, April 13, 2018 12:33 am, Pablo Di Noto wrote:
>
> > Ah! The joy! Now back to R4.0 without massive reinstall.
> >
> >
> > Thanks Marek, awokd and techg...!
>
> I think you taught me more than I helped on this one! Thanks for sharing
> your resolution.
>
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "qubes-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/qubes-users/SV_BlWTvB2g/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> qubes-users+unsubscr...@googlegroups.com.
> To post to this group, send email to qubes-users@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/qubes-users/04dac88f740f67b95a41f3d69060abe7.squirrel%40tt3j2x4k5ycaa5zt.onion
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To post to this group, send email to qubes-users@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/CAJrT%2BY%3DKtGAHQuu%2Boba24FZ99u5xmE7mqzrnuqvJaXGnn8toQg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [qubes-users] Re: Qubes R4.0 broken by 'TypeError: not enough arguments...' for most qvm-* commands

2018-04-12 Thread 'awokd' via qubes-users 
On Fri, April 13, 2018 12:33 am, Pablo Di Noto wrote:

> Ah! The joy! Now back to R4.0 without massive reinstall.
>
>
> Thanks Marek, awokd and techg...!

I think you taught me more than I helped on this one! Thanks for sharing
your resolution.



-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To post to this group, send email to qubes-users@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/04dac88f740f67b95a41f3d69060abe7.squirrel%40tt3j2x4k5ycaa5zt.onion.
For more options, visit https://groups.google.com/d/optout.

Re: [qubes-users] Re: Qubes R4.0 broken by "TypeError: not enough arguments..." for most qvm-* commands

2018-04-12 Thread Pablo Di Noto 

> > > > You could also try to revert to earlier revision using "qvm-volume
> > revert sys-net:private" for example.
> 
> Will try that tonight.
> 

Unfortunately, that was not possible for the service VMs.
For the only important AppVM that had -back and -back missing, the command 
`qvm-block revert p-2018d:private` was enough.

> > > I remember there was some Saltstack magic to recreate the services vms, 
> > > but could not find anything for R4.0... So I had to revert to R3.2 for 
> > > the time being.
> > 
> > https://www.qubes-os.org/doc/salt/
> > Especially links at the bottom:
> > https://github.com/QubesOS/qubes-mgmt-salt-dom0-virtual-machines/blob/master/README.rst
> 
> Thanks. Will also try that tonight.

The way I recreated all sys-* service VMs was:

# You need to unset sys-firewall as netvm for AppVMs
# and unset sys-net as netvm for sys-firewall:
dom0$ qvm-prefs --set  netvm ''
dom0$ qvm-prefs --set sys-firewall netvm ''

# The delete all service VMs qubes:
dom0$ qvm-remove sys-usb
dom0$ qvm-remove sys-firewall
dom0$ qvm-remove sys-net

# Make sure the right salt top files are enabled:
dom0$ sudo qubesctl top.enable qvm.sys-net
dom0$ sudo qubesctl top.enable qvm.sys-firewall
dom0$ sudo qubesctl top.enable qvm.sys-usb
dom0$ sudo qubesctl state.highstate

and that should be all. You have to restore 'sys-firewall' as netvm for all the 
qubes you want networking and thing should be pretty much as they were after 
install. Note that all new service VMs will have different IP address than 
their deceased counterparts, but firewall would be able to cope with that.

Ah! The joy! Now back to R4.0 without massive reinstall.

Thanks Marek, awokd and techg...!

-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To post to this group, send email to qubes-users@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/d343c920-7401-475d-a340-39df7c4b3afa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [qubes-users] Re: Qubes R4.0 broken by 'TypeError: not enough arguments...' for most qvm-* commands

2018-04-12 Thread Brendon Green 
Check for Meta% above 80%

On Thu, 12 Apr 2018 at 21:21 awokd  wrote:

> On Thu, April 12, 2018 3:20 am, techgee...@gmail.com wrote:
> > On Wednesday, 11 April 2018 08:29:48 UTC+12, Pablo Di Noto  wrote:
> >
> >> Hello,
> >>
> >>
> >> Any debugging tips?
> >>
> >
> > You have almost exactly described my scenario.
> >
> >
> > You can work around the TypeError exception by editing
> > /usr/lib/python3.5/site-packages/qubesadmin/exc.py, as I did.  Please
> > note, however, that the lines _must_ be indented with spaces (not tabs);
> > as Python3 is very particular about indentation style.
> >
> > If, as I suspect, the root cause of your problem is a lack of metadata
> > space on pool00; you can confirm this by typing "sudo lvs" into a
> > console.  You will then need to figure out a way to enlarge that metadata
> > volume.
>
> That's certainly a non-intuitive failure mode. How did you find that? I'm
> not experiencing it myself, but what would one look for in "sudo lvs"-
> Meta% at 100% on one of the pools?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To post to this group, send email to qubes-users@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/CAJrT%2BYmYVZtk%3DWQDNEOUx%3DgSU5M9svf%3DH_Tev3Q0GXwCHveCTQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [qubes-users] Re: Qubes R4.0 broken by "TypeError: not enough arguments..." for most qvm-* commands

2018-04-12 Thread Pablo Di Noto 
> > > If, as I suspect, the root cause of your problem is a lack of metadata 
> > > space on pool00; you can confirm this by typing "sudo lvs" into a 
> > > console.  You will then need to figure out a way to enlarge that metadata 
> > > volume.
> > 
> > Yes, you are right, the `pool00` volume metadata was >96% when this 
> > happened. The thing is that the volume metadata was set to a quite small 
> > size after install (96mb on a 46gb pool) and after install was on ~20% 
> > usage. I started to use the system, testing stuff with DispVMs, restoring 
> > my debian templates and some work VMs. After a couple of days of usage the 
> > metadata climbed very little, to 27-28%.
> > 
> > I tried to have a second pool to hold my machines, precisely to avoid 
> > issues with thin provisioning on the pool holding `root` and `swap` and 
> > services vms. But the lack of support for cloning/moving between pools made 
> > that effort moot.
> > 
> > So I `lvextend`ed `pool00` and forgot to properly enlarge it's  
> > `pool00_tmeta` counterpart.
> 
> What sizes you have there?
> For me tmeta is 118MB for a ~450GB pool00. And after few months of usage
> it's still at 33%...

An install on a 60G disk partition had a pool00 of 43G created with a 
pool00_tmeta of 44M (11 extents). Later, the pool00 was extended to 147.7G and 
the pool00_tmeta left as is by mistake. 

The metadata became full, and after that the pool00_tmeta was extended to 300M 
by adding 256M.

> > When doing some more customization, including restoring more larger sized 
> > qubes and cloning/renaming qubes it seems the metadata usage climbed really 
> > fast and hit this bug.
> > 
> > Unfortunately, could not recover from that.
> > 
> > It looks like qubes lvm actions while metadata was full may have corrupted 
> > the metadata somehow, since I could enlarge and repair the thin metadata 
> > from a live cd, but many of the volumes that where in use where never 
> > available again. The -private and -snap for the qubes that were running 
> > (not sure how to discard them) and also all the volumes of the qubes being 
> > restored and services vms are lost ("NOT available" as lvm status)
> 
> You could also try to revert to earlier revision using "qvm-volume
> revert sys-net:private" for example.

Will try that tonight.

> > I remember there was some Saltstack magic to recreate the services vms, but 
> > could not find anything for R4.0... So I had to revert to R3.2 for the time 
> > being.
> 
> https://www.qubes-os.org/doc/salt/
> Especially links at the bottom:
> https://github.com/QubesOS/qubes-mgmt-salt-dom0-virtual-machines/blob/master/README.rst

Thanks. Will also try that tonight. 

-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To post to this group, send email to qubes-users@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/cd525562-ead8-4559-9326-c31787b438dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [qubes-users] Re: Qubes R4.0 broken by "TypeError: not enough arguments..." for most qvm-* commands

2018-04-12 Thread Marek Marczykowski-Górecki 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Thu, Apr 12, 2018 at 08:19:10AM -0700, Pablo Di Noto wrote:
> > techg...@gmail.com
> > If, as I suspect, the root cause of your problem is a lack of metadata 
> > space on pool00; you can confirm this by typing "sudo lvs" into a console.  
> > You will then need to figure out a way to enlarge that metadata volume.
> 
> Yes, you are right, the `pool00` volume metadata was >96% when this happened. 
> The thing is that the volume metadata was set to a quite small size after 
> install (96mb on a 46gb pool) and after install was on ~20% usage. I started 
> to use the system, testing stuff with DispVMs, restoring my debian templates 
> and some work VMs. After a couple of days of usage the metadata climbed very 
> little, to 27-28%.
> 
> I tried to have a second pool to hold my machines, precisely to avoid issues 
> with thin provisioning on the pool holding `root` and `swap` and services 
> vms. But the lack of support for cloning/moving between pools made that 
> effort moot.
> 
> So I `lvextend`ed `pool00` and forgot to properly enlarge it's  
> `pool00_tmeta` counterpart.

What sizes you have there?
For me tmeta is 118MB for a ~450GB pool00. And after few months of usage
it's still at 33%...

> When doing some more customization, including restoring more larger sized 
> qubes and cloning/renaming qubes it seems the metadata usage climbed really 
> fast and hit this bug.
> 
> Unfortunately, could not recover from that.
> 
> It looks like qubes lvm actions while metadata was full may have corrupted 
> the metadata somehow, since I could enlarge and repair the thin metadata from 
> a live cd, but many of the volumes that where in use where never available 
> again. The -private and -snap for the qubes that were running (not sure how 
> to discard them) and also all the volumes of the qubes being restored and 
> services vms are lost ("NOT available" as lvm status)

You could also try to revert to earlier revision using "qvm-volume
revert sys-net:private" for example.

> I remember there was some Saltstack magic to recreate the services vms, but 
> could not find anything for R4.0... So I had to revert to R3.2 for the time 
> being.

https://www.qubes-os.org/doc/salt/
Especially links at the bottom:
https://github.com/QubesOS/qubes-mgmt-salt-dom0-virtual-machines/blob/master/README.rst

> I will keep the failing install for debugging, or may be able to recover if 
> someone can provide any tips about:
> 
> - How to recreate sys-net, sys-firewall and sys-usb on a R4.0 system
> - how to recover a qube whose -snap volumes are no longer available (I have 
> no problem losing these short-term data)
> 
> Thanks for pointing to the right direction!
> 


- -- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlrMuusACgkQ24/THMrX
1yy3lQf/cO0oe9uOUviiKwgdf6+fEzhCbn6XUkmAU7MLLAkYC1uCAwE3DoT8MBGt
bbGkpmWq9gijUCJeWzUD0Z2k1QkZWDdiMgEE8nSgiqyS1O6uNxqqO0ucozWe69Ud
FWwmxkCATwX+FK239+HJSO9Jq6/Izb59qbvB1kwewQheqGkZVF9ISNE3AopkMjG8
4RBy1J0dVjHH3wxHtl9N3Z6/4mVwFquLwlE7cM+kTRpFfPtwvrBrNavfYTrEX5lz
ALBvsh/eunXBOmc4FNSGHj2yaKnNZibfBVDOoBGaexXt1G0ykpu9aou8tQrKv0zl
FqhhNHp9DeOdHm3kP0h1d6PZW1EGiw==
=5Ksa
-END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To post to this group, send email to qubes-users@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/20180412184252.GB2275%40mail-itl.
For more options, visit https://groups.google.com/d/optout.

Re: [qubes-users] Re: Qubes R4.0 broken by 'TypeError: not enough arguments...' for most qvm-* commands

2018-04-12 Thread 'awokd' via qubes-users 
On Thu, April 12, 2018 3:20 am, techgee...@gmail.com wrote:
> On Wednesday, 11 April 2018 08:29:48 UTC+12, Pablo Di Noto  wrote:
>
>> Hello,
>>
>>
>> Any debugging tips?
>>
>
> You have almost exactly described my scenario.
>
>
> You can work around the TypeError exception by editing
> /usr/lib/python3.5/site-packages/qubesadmin/exc.py, as I did.  Please
> note, however, that the lines _must_ be indented with spaces (not tabs);
> as Python3 is very particular about indentation style.
>
> If, as I suspect, the root cause of your problem is a lack of metadata
> space on pool00; you can confirm this by typing "sudo lvs" into a
> console.  You will then need to figure out a way to enlarge that metadata
> volume.

That's certainly a non-intuitive failure mode. How did you find that? I'm
not experiencing it myself, but what would one look for in "sudo lvs"-
Meta% at 100% on one of the pools?


-- 
You received this message because you are subscribed to the Google Groups 
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to qubes-users+unsubscr...@googlegroups.com.
To post to this group, send email to qubes-users@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/qubes-users/7d73628942f687b771626a3e16fd2b4f.squirrel%40tt3j2x4k5ycaa5zt.onion.
For more options, visit https://groups.google.com/d/optout.