Re: amazon/xen... any way at all to pass a message/signal/semaphoere/morse-code to the boot loader?

[Julian Elischer]

On 13/4/17 2:07 am, Jeremiah Lott via freebsd-cloud wrote:

On Wed, Apr 12, 2017 at 1:30 AM, Leif Pedersen  wrote:


I keep an extra EBS volume handy that has a simple recovery image. If I get
stuck into a trouble, I change the normal boot disk to sdb, and attach my
recovery volume as sda1. Essentially, the extra volume is my "recovery
partition". To make it cheaper, keep only a snapshot of it.


I tried for a while to get some sort of bootloader-based recovery plan in
place for our cloud-based systems, like what was originally asked for. We
already have a primary and a backup partition in our boot disk, but there
was no way I found in EC2 to easily switch the partition to boot from. In
the end, I gave up on passing information to the bootloader and used
something like the above with multiple images. I actually wrote a script at
one point using the aws CLI that you could run from any FreeBSD VM in the
same availability zone. It detached the original boot volume from the
"broken" instance; attached it as a secondary disk to the recovery image,
changed the boot partition, detached it from the recovery image, then
re-attached it to the original image. It took a while to run, but required
little user input. We kind of kept that as "good enough" for the rare case
that a instance became un-bootable and we cared to recover it rather than
replace it. I'm not sure we actually ever used it on a customer system. It
was used more during development when you are more likely to break stuff
(and want to recover coredumps, etc. so you can fix the broken code).


Thanks for your comments. It appears that you have the same issues 
that we do.
Andriy Gapon has been doing some stuff where nextboot information is 
saved onto the drive,

and it knows how much it has beooted which may be good enough for us.
basically a 'drops to recovery mode after N failures' would be enough 
for me.


Is there any chance you can make your "recovery" system available?
(especially if you can give source for the aws CLI stuff..  I think 
having that as an example
and starting point might be a good start to making something truely 
useful.
It may even be worth adding it to the regular FreeBSD AMI so that any 
FreeBSD

EC2 system could be used to recover other systems.

In our system there is a single zpool with two ZFS datasets and we use 
the "bootfs" parameter to select the new image, but it can be 
overridden from the boot menu, except of course on AWS due to the lack 
of console.







If you go down the route of implementing EC2 network driver(s) in the
bootloader, then you could read the instance metadata via http and use a
tag to control the boot behavior. However, a bootloader driver, even a very
simplistic one, for xn0 (and potentially for both ixv and ena, if you
support EC2 Enhanced Networking) was more work that we wanted to undertake
for this.

   Jeremiah Lott
   Avere Systems
___
freebsd-cl...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-cloud
To unsubscribe, send any mail to "freebsd-cloud-unsubscr...@freebsd.org"



___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: amazon/xen... any way at all to pass a message/signal/semaphoere/morse-code to the boot loader?

[Jeremiah Lott via freebsd-xen]

On Wed, Apr 12, 2017 at 1:30 AM, Leif Pedersen  wrote:

> I keep an extra EBS volume handy that has a simple recovery image. If I get
> stuck into a trouble, I change the normal boot disk to sdb, and attach my
> recovery volume as sda1. Essentially, the extra volume is my "recovery
> partition". To make it cheaper, keep only a snapshot of it.
>

I tried for a while to get some sort of bootloader-based recovery plan in
place for our cloud-based systems, like what was originally asked for. We
already have a primary and a backup partition in our boot disk, but there
was no way I found in EC2 to easily switch the partition to boot from. In
the end, I gave up on passing information to the bootloader and used
something like the above with multiple images. I actually wrote a script at
one point using the aws CLI that you could run from any FreeBSD VM in the
same availability zone. It detached the original boot volume from the
"broken" instance; attached it as a secondary disk to the recovery image,
changed the boot partition, detached it from the recovery image, then
re-attached it to the original image. It took a while to run, but required
little user input. We kind of kept that as "good enough" for the rare case
that a instance became un-bootable and we cared to recover it rather than
replace it. I'm not sure we actually ever used it on a customer system. It
was used more during development when you are more likely to break stuff
(and want to recover coredumps, etc. so you can fix the broken code).

If you go down the route of implementing EC2 network driver(s) in the
bootloader, then you could read the instance metadata via http and use a
tag to control the boot behavior. However, a bootloader driver, even a very
simplistic one, for xn0 (and potentially for both ixv and ena, if you
support EC2 Enhanced Networking) was more work that we wanted to undertake
for this.

  Jeremiah Lott
  Avere Systems
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: amazon/xen... any way at all to pass a message/signal/semaphoere/morse-code to the boot loader?

[Julian Elischer]

On 12/4/17 2:09 pm, Toomas Soome wrote:

On 12. apr 2017, at 9:04, Julian Elischer  wrote:

On 12/4/17 12:34 pm, Colin Percival wrote:

[CCing freebsd-cloud, which is the right place for discussions of FreeBSD/EC2]

On 04/11/17 21:03, Julian Elischer wrote:

In Amazon ec2 they have no console access (though I heard rumors that it was
available I have not seen any sign of it) so I'd like to put a "recovery
partition" into an AMI.
The trick is how to convince it to boot to that instead of the regular action.

Can you get what you want via gptboot's support for selecting the partition
to boot via "bootonce" and "bootme" flags?

not if you can't get onto to the machine.
When I talk about a recovery partition I mean it in the same way that apple 
means it..
"system won't boot? press the power button and hold down the 'option' key.
it will give you the option to boot to a recovery mode"
(* actually I can't remember the keys but you get the idea..)

in our case we would like to be able to recover a customer's AMI by giving a 
simple set of instructions over the phone.
We can assume they know how to get into the amazon menus, but we would like to 
not have to assume much more.


The ideal thing would be if there was  way to 'influence' one of the smbios
values in some way, and have the boot code see it, but I'm open to any
suggestions.
I really need only 1 bit of information to get through.

Possibilties include "changing the VM to have only 2G of ram" (we'd never do
that in a real machine).
or maybe temporarily removing all the disks other than the root drive? Almost
anything I could do to signal the boot code to behave differently.

I don't think adding/removing disks will be useful, since the extra disks will
be Xen blkfront devices; AFAIK the boot loader doesn't know anything about
these.  (The boot device is also a blkfront device but gets ATA emulation for
the benefit of boot loaders.)

Maybe you can repurpose some of the logic used for booting over NFS?  I've
never heard of people booting over NFS when the initial bootstrap comes from
disk rather than PXE, but I assume it's possible...?

Oh I've done it, in the past but you still have the same issue..
how do you signal the boot code to do this?

(does an AMI have a bios capable of doing network operations?) I was thinking
about whether we could add a really simple xn driver into the bootcode to allow
us to have an console of sorts (accessible from an adjacent machine only??)





basically you want what zfsbootcfg does but in reverse — with fallback to 
recovery…

yes,
that is also a possibility  N failed boots in a row..



rgds,
toomas





___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: amazon/xen... any way at all to pass a message/signal/semaphoere/morse-code to the boot loader?

[Julian Elischer]

On 12/4/17 1:30 pm, Leif Pedersen wrote:
I keep an extra EBS volume handy that has a simple recovery image. 
If I get stuck into a trouble, I change the normal boot disk to sdb, 
and attach my recovery volume as sda1. Essentially, the extra volume 
is my "recovery partition". To make it cheaper, keep only a snapshot 
of it.


yes that's a technique I've used in the past.

I'd prefer to find something simpler to do, which is why it'd be nice 
if one could just control some single bit that the bootloader could read.




Same idea on Google Compute Engine.




On Apr 11, 2017 11:34 PM, "Colin Percival" > wrote:


[CCing freebsd-cloud, which is the right place for discussions
of FreeBSD/EC2]

On 04/11/17 21:03, Julian Elischer wrote:
> In Amazon ec2 they have no console access (though I heard
rumors that it was
> available I have not seen any sign of it) so I'd like to put a
"recovery
> partition" into an AMI.
> The trick is how to convince it to boot to that instead of the
regular action.

Can you get what you want via gptboot's support for selecting
the partition
to boot via "bootonce" and "bootme" flags?

> The ideal thing would be if there was  way to 'influence' one
of the smbios
> values in some way, and have the boot code see it, but I'm
open to any
> suggestions.
> I really need only 1 bit of information to get through.
>
> Possibilties include "changing the VM to have only 2G of ram"
(we'd never do
> that in a real machine).
> or maybe temporarily removing all the disks other than the
root drive? Almost
> anything I could do to signal the boot code to behave differently.

I don't think adding/removing disks will be useful, since the
extra disks will
be Xen blkfront devices; AFAIK the boot loader doesn't know
anything about
these.  (The boot device is also a blkfront device but gets ATA
emulation for
the benefit of boot loaders.)

Maybe you can repurpose some of the logic used for booting over
NFS?  I've
never heard of people booting over NFS when the initial
bootstrap comes from
disk rather than PXE, but I assume it's possible...?

--
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com  |
Online backups for the truly paranoid
___
freebsd-cl...@freebsd.org 
mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-cloud

To unsubscribe, send any mail to
"freebsd-cloud-unsubscr...@freebsd.org
"



___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: amazon/xen... any way at all to pass a message/signal/semaphoere/morse-code to the boot loader?

[Toomas Soome]

> On 12. apr 2017, at 9:04, Julian Elischer  wrote:
> 
> On 12/4/17 12:34 pm, Colin Percival wrote:
>> [CCing freebsd-cloud, which is the right place for discussions of 
>> FreeBSD/EC2]
>> 
>> On 04/11/17 21:03, Julian Elischer wrote:
>>> In Amazon ec2 they have no console access (though I heard rumors that it was
>>> available I have not seen any sign of it) so I'd like to put a "recovery
>>> partition" into an AMI.
>>> The trick is how to convince it to boot to that instead of the regular 
>>> action.
>> Can you get what you want via gptboot's support for selecting the partition
>> to boot via "bootonce" and "bootme" flags?
> not if you can't get onto to the machine.
> When I talk about a recovery partition I mean it in the same way that apple 
> means it..
> "system won't boot? press the power button and hold down the 'option' key.
> it will give you the option to boot to a recovery mode"
> (* actually I can't remember the keys but you get the idea..)
> 
> in our case we would like to be able to recover a customer's AMI by giving a 
> simple set of instructions over the phone.
> We can assume they know how to get into the amazon menus, but we would like 
> to not have to assume much more.
> 
>> 
>>> The ideal thing would be if there was  way to 'influence' one of the smbios
>>> values in some way, and have the boot code see it, but I'm open to any
>>> suggestions.
>>> I really need only 1 bit of information to get through.
>>> 
>>> Possibilties include "changing the VM to have only 2G of ram" (we'd never do
>>> that in a real machine).
>>> or maybe temporarily removing all the disks other than the root drive? 
>>> Almost
>>> anything I could do to signal the boot code to behave differently.
>> I don't think adding/removing disks will be useful, since the extra disks 
>> will
>> be Xen blkfront devices; AFAIK the boot loader doesn't know anything about
>> these.  (The boot device is also a blkfront device but gets ATA emulation for
>> the benefit of boot loaders.)
>> 
>> Maybe you can repurpose some of the logic used for booting over NFS?  I've
>> never heard of people booting over NFS when the initial bootstrap comes from
>> disk rather than PXE, but I assume it's possible...?
> 
> Oh I've done it, in the past but you still have the same issue..
> how do you signal the boot code to do this?
> 
> (does an AMI have a bios capable of doing network operations?) I was thinking
> about whether we could add a really simple xn driver into the bootcode to 
> allow
> us to have an console of sorts (accessible from an adjacent machine only??)
> 
> 
> 


basically you want what zfsbootcfg does but in reverse — with fallback to 
recovery…

rgds,
toomas

___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: amazon/xen... any way at all to pass a message/signal/semaphoere/morse-code to the boot loader?

[Julian Elischer]

On 12/4/17 12:34 pm, Colin Percival wrote:

[CCing freebsd-cloud, which is the right place for discussions of FreeBSD/EC2]

On 04/11/17 21:03, Julian Elischer wrote:

In Amazon ec2 they have no console access (though I heard rumors that it was
available I have not seen any sign of it) so I'd like to put a "recovery
partition" into an AMI.
The trick is how to convince it to boot to that instead of the regular action.

Can you get what you want via gptboot's support for selecting the partition
to boot via "bootonce" and "bootme" flags?

not if you can't get onto to the machine.
When I talk about a recovery partition I mean it in the same way that 
apple means it..

"system won't boot? press the power button and hold down the 'option' key.
it will give you the option to boot to a recovery mode"
(* actually I can't remember the keys but you get the idea..)

in our case we would like to be able to recover a customer's AMI by 
giving a simple set of instructions over the phone.
We can assume they know how to get into the amazon menus, but we would 
like to not have to assume much more.





The ideal thing would be if there was  way to 'influence' one of the smbios
values in some way, and have the boot code see it, but I'm open to any
suggestions.
I really need only 1 bit of information to get through.

Possibilties include "changing the VM to have only 2G of ram" (we'd never do
that in a real machine).
or maybe temporarily removing all the disks other than the root drive? Almost
anything I could do to signal the boot code to behave differently.

I don't think adding/removing disks will be useful, since the extra disks will
be Xen blkfront devices; AFAIK the boot loader doesn't know anything about
these.  (The boot device is also a blkfront device but gets ATA emulation for
the benefit of boot loaders.)

Maybe you can repurpose some of the logic used for booting over NFS?  I've
never heard of people booting over NFS when the initial bootstrap comes from
disk rather than PXE, but I assume it's possible...?


Oh I've done it, in the past but you still have the same issue..
how do you signal the boot code to do this?

(does an AMI have a bios capable of doing network operations?) I was 
thinking
about whether we could add a really simple xn driver into the bootcode 
to allow
us to have an console of sorts (accessible from an adjacent machine 
only??)




___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: amazon/xen... any way at all to pass a message/signal/semaphoere/morse-code to the boot loader?

[Leif Pedersen]

I keep an extra EBS volume handy that has a simple recovery image. If I get
stuck into a trouble, I change the normal boot disk to sdb, and attach my
recovery volume as sda1. Essentially, the extra volume is my "recovery
partition". To make it cheaper, keep only a snapshot of it.

Same idea on Google Compute Engine.




On Apr 11, 2017 11:34 PM, "Colin Percival"  wrote:

> [CCing freebsd-cloud, which is the right place for discussions of
> FreeBSD/EC2]
>
> On 04/11/17 21:03, Julian Elischer wrote:
> > In Amazon ec2 they have no console access (though I heard rumors that it
> was
> > available I have not seen any sign of it) so I'd like to put a "recovery
> > partition" into an AMI.
> > The trick is how to convince it to boot to that instead of the regular
> action.
>
> Can you get what you want via gptboot's support for selecting the partition
> to boot via "bootonce" and "bootme" flags?
>
> > The ideal thing would be if there was  way to 'influence' one of the
> smbios
> > values in some way, and have the boot code see it, but I'm open to any
> > suggestions.
> > I really need only 1 bit of information to get through.
> >
> > Possibilties include "changing the VM to have only 2G of ram" (we'd
> never do
> > that in a real machine).
> > or maybe temporarily removing all the disks other than the root drive?
> Almost
> > anything I could do to signal the boot code to behave differently.
>
> I don't think adding/removing disks will be useful, since the extra disks
> will
> be Xen blkfront devices; AFAIK the boot loader doesn't know anything about
> these.  (The boot device is also a blkfront device but gets ATA emulation
> for
> the benefit of boot loaders.)
>
> Maybe you can repurpose some of the logic used for booting over NFS?  I've
> never heard of people booting over NFS when the initial bootstrap comes
> from
> disk rather than PXE, but I assume it's possible...?
>
> --
> Colin Percival
> Security Officer Emeritus, FreeBSD | The power to serve
> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
> ___
> freebsd-cl...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-cloud
> To unsubscribe, send any mail to "freebsd-cloud-unsubscr...@freebsd.org"
>
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: amazon/xen... any way at all to pass a message/signal/semaphoere/morse-code to the boot loader?

[Colin Percival]

[CCing freebsd-cloud, which is the right place for discussions of FreeBSD/EC2]

On 04/11/17 21:03, Julian Elischer wrote:
> In Amazon ec2 they have no console access (though I heard rumors that it was
> available I have not seen any sign of it) so I'd like to put a "recovery
> partition" into an AMI.
> The trick is how to convince it to boot to that instead of the regular action.

Can you get what you want via gptboot's support for selecting the partition
to boot via "bootonce" and "bootme" flags?

> The ideal thing would be if there was  way to 'influence' one of the smbios
> values in some way, and have the boot code see it, but I'm open to any
> suggestions.
> I really need only 1 bit of information to get through.
> 
> Possibilties include "changing the VM to have only 2G of ram" (we'd never do
> that in a real machine).
> or maybe temporarily removing all the disks other than the root drive? Almost
> anything I could do to signal the boot code to behave differently.

I don't think adding/removing disks will be useful, since the extra disks will
be Xen blkfront devices; AFAIK the boot loader doesn't know anything about
these.  (The boot device is also a blkfront device but gets ATA emulation for
the benefit of boot loaders.)

Maybe you can repurpose some of the logic used for booting over NFS?  I've
never heard of people booting over NFS when the initial bootstrap comes from
disk rather than PXE, but I assume it's possible...?

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

amazon/xen... any way at all to pass a message/signal/semaphoere/morse-code to the boot loader?

[Julian Elischer]

In Amazon ec2 they have no console access (though I heard rumors that 
it was available I have not seen any sign of it) so I'd like to put a 
"recovery partition" into an AMI.
The trick is how to convince it to boot to that instead of the regular 
action.



The ideal thing would be if there was  way to 'influence' one of the 
smbios values in some way, and have the boot code see it, but I'm open 
to any suggestions.

I really need only 1 bit of information to get through.

Possibilties include "changing the VM to have only 2G of ram" (we'd 
never do that in a real machine).
or maybe temporarily removing all the disks other than the root drive? 
Almost anything I could do to signal the boot code to behave differently.




___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"