Re: installboot with amd64 root on softraid crypto, NOT 'a' partition

[Theo de Raadt]

> On Sun, 8 Nov 2015 21:22:04 -0800
> Nathan Wheeler  wrote:
> > I ran into this exact same issue when I was trying to create a
> > rollback install with CRYPTO for a sort of appliance I develop/manage
> > for my company. We only have remote access with console and remote
> > hands aren't easy to get so when upgrading it'd be nice to have a
> > rollback in case something happens.
> > 
> > You can definitely boot a kernel off a different partition, but the
> > kernel still assumes the root disk is 'a'. You have to tell the kernel
> > to ask you for the root partition with "boot -a". Or you can compile
> > your own kernel with the root disk hardcoded as it mentions in this
> > post: http://www.undeadly.org/cgi?action=article=20110530221728
> 
> Isn't a story of the kernel configuration file?
> It doesn't seem 'a' is hard coded (cannot change) since actually -a
> can override that value.

People should be careful what they ask for:

- The entropy file
- /etc/boot.conf

I think this is a clear case of (1) don't waste timet explaining why
it is the way it is, (2) no warranty, (3) people who wish to hurt
themselves can go into the kitchen, (4) people who run into problems
are asked to not submit bug reports from their frankenstein machines.

Re: installboot with amd64 root on softraid crypto, NOT 'a' partition

[YASUOKA Masahiko]

On Sun, 8 Nov 2015 21:22:04 -0800
Nathan Wheeler  wrote:
> I ran into this exact same issue when I was trying to create a
> rollback install with CRYPTO for a sort of appliance I develop/manage
> for my company. We only have remote access with console and remote
> hands aren't easy to get so when upgrading it'd be nice to have a
> rollback in case something happens.
> 
> You can definitely boot a kernel off a different partition, but the
> kernel still assumes the root disk is 'a'. You have to tell the kernel
> to ask you for the root partition with "boot -a". Or you can compile
> your own kernel with the root disk hardcoded as it mentions in this
> post: http://www.undeadly.org/cgi?action=article=20110530221728

Isn't a story of the kernel configuration file?
It doesn't seem 'a' is hard coded (cannot change) since actually -a
can override that value.

> But I've come to the same conclusion that most people would say on
> this list that its just not a good idea. I definitely don't plan to
> put that practice in production. But if its just a personal use
> laptop, maybe it'll be OK, up to you.

I understand that story.

I'm not complaining but just wondering why "boot hd0j:/bsd" works but
"boot sr0j:/bsd" doesn't work.

--yasuoka

Re: installboot with amd64 root on softraid crypto, NOT 'a' partition

[Stefan Sperling]

On Sat, Nov 07, 2015 at 07:57:05PM -0500, Jonathan Thornburg wrote:
> At this point the machine can boot and run sd1[aeg] fine.  *But* if I
> enter "boot sr1d:/bsd" at the "boot>" prompt, the machine boots sd1[aeg],
> not the desired sd1[dfh].  In other words, at this point my "backup"
> sd1[dfh] partitions aren't usefully bootable :(.

You cannot boot a kernel from a non-'a' partition.
The bootloader won't look elsewhere.

In theory, you could copy your 5.6 kernel to the 'a' partition
of your 5.8 install (cp /mnt/bsd /bsd.56) and then boot it from
there with the '-a' option: boot> boot sr1a:/bsd.56 -a
The '-a' option causes the kernel to ask for the root and swap
partitions after booting, so you could try pointing this kernel
at your 5.6 root filesystem on sd1d.

But this is rather fragile.
Perhaps it would have been easier to temporarily install 5.8 and
your perl modules on a USB stick or into a qemu virtual machine
for testing purposes.

Overall, I'd suggest moving everything to 5.8 now and forgetting
about your sinking 5.6 life boat.

> I read with interest krw@'s recent Undeadly description
> 
> of fixing a softraid-crypto bug in installboot(8),
> | I spent some time helping stsp@ fix installboot(8) to handle softraid
> | crypto installs, while stoutly maintaining my ignorance of softraid
> | crypto. I failed to see a bug stsp@ introduced, which lead to some
> | hilarity a few days later when I backed out the change at the same
> | time that stsp@ was committing a real fix. Fortunately stsp@ was
> | able to set things right, and installboot(8) now works for softraid
> | crypto setups and non-softraid setups.
> 
> Is this bug relevant to my situation?

No. This bug didn't affect 5.8, it affected -current for about 2 days.

Reading this quote it seems Ken got a bit confused in his report.
The problem was not about softraid vs. non-softraid, but softraid
on top of a GPT disk vs. softraid on top of an MBR disk.
The bug broke boot entirely for softraid on an MBR disk (you would
just see `ERR M' trying to boot).

Re: installboot with amd64 root on softraid crypto, NOT 'a' partition

[YASUOKA Masahiko]

On Sun, 8 Nov 2015 11:52:48 +0100
Stefan Sperling  wrote:
> On Sat, Nov 07, 2015 at 07:57:05PM -0500, Jonathan Thornburg wrote:
>> At this point the machine can boot and run sd1[aeg] fine.  *But* if I
>> enter "boot sr1d:/bsd" at the "boot>" prompt, the machine boots sd1[aeg],
>> not the desired sd1[dfh].  In other words, at this point my "backup"
>> sd1[dfh] partitions aren't usefully bootable :(.
> 
> You cannot boot a kernel from a non-'a' partition.
> The bootloader won't look elsewhere.

As far as I know The boot loader itself should be located on 'a'
partition at least on i386 or amd64 BIOS boot loader, but kernel seems
to be able to be booted from non-'a' parition partition.  This
actually works at least on hd?? on amd64.

When starting the kernel, the boot loaders pass the 'bootdev'
parameter to the kernel.   Below is the definition of 'bootdev' in
sys/reboot.h.

/*
 * Constants for converting boot-style device number to type,
 * adaptor (uba, mba, etc), unit number and partition number.
 * Type (== major device number) is in the low byte
 * for backward compatibility.  Except for that of the "magic
 * number", each mask applies to the shifted value.
 * Format:
 *   (4) (4) (4) (4)  (8) (8)
 *  
 *  |MA | AD| CT| UN| PART  | TYPE |
 *  
 */

it includes the parition number.  So booting from non-'a' partition
seems to be intended.

--yasuoka

Re: installboot with amd64 root on softraid crypto, NOT 'a' partition

[Nathan Wheeler]

I ran into this exact same issue when I was trying to create a
rollback install with CRYPTO for a sort of appliance I develop/manage
for my company. We only have remote access with console and remote
hands aren't easy to get so when upgrading it'd be nice to have a
rollback in case something happens.

You can definitely boot a kernel off a different partition, but the
kernel still assumes the root disk is 'a'. You have to tell the kernel
to ask you for the root partition with "boot -a". Or you can compile
your own kernel with the root disk hardcoded as it mentions in this
post: http://www.undeadly.org/cgi?action=article=20110530221728

But I've come to the same conclusion that most people would say on
this list that its just not a good idea. I definitely don't plan to
put that practice in production. But if its just a personal use
laptop, maybe it'll be OK, up to you.

installboot with amd64 root on softraid crypto, NOT 'a' partition

[Jonathan Thornburg]

I've run into an odd problem trying to use with installboot(8) on a
softraid-crypto disk, and I wonder if anyone can suggest a
workaround/solution.


Background
--

As described in ,
I have an amd64 laptop (Thinkpad T60p) which has multiple sets of
softraid-crypto root/var/usr partitions, sharing common swap and home
partitions:

  sd0
  ---
   |  a-+- (sd1) softraid crypt, size = 44.5G
   || a = root   256M
   || d = root2  256M
   || e = var2G
   || f = var2   2G
   || g = usr20G
   || h = usr2   20G
   |   -+-
   |  b   swap   6G
   |  j-+- (sd2) softraid crypt, size = all remaining space
   || j = home
  ---  -+-

sd0 is the physical disk.  It has 3 openbsd-partitions: a, b, and j.

sd1 is a softraid-crypto disk living inside sd0a.  sd1 stores all the
OS partitions.  More precisely, sd1 stores *two* sets of root, var, and
usr partitions.  My starting point is that sd1[aeg] contain a working
OpenBSD 5.6-stable/amd64, and sd1[dfh] are empty.

sd2 is a softraid-crypt disk living inside sd0j.  sd2 stores /home,
and (I think) isn't relevant here.

The boot procedure is to turn the laptop on, and (after memory tests etc)
enter the sd1 passphrase.  The machine will then boot from sd1[aeg] (or
sd1[dfh] if I enter 'boot sr1d:/bsd' at the "boot>" prompt, assuming there's
an OpenBSD installation on those partitions).  I activate sd2 and mount
/home via a perl script run after the machine is running multiuser, i.e.,
this doesn't form part of the OS-level boot sequence.


Problem
---

I want to move to 5.8... *but* I want to keep the ability to
boot and run 5.6-stable as a backup even after 5.8 is running.
[I'm pretty confident that 5.8 will install/run fine,
but I'm less confident that all the perl modules I
currently use under 5.6 will grok 5.8 properly.]
That is, I'd like to have a new 5.8 install in sd1[aeg], and (a copy of)
my previous 5.6-stable in sd1[dfh], with the latter bootable via entering
"boot sr1d:/bsd" at the "boot>" prompt.

My plan was the following:
1. copy the 5.6-stable OpenBSD installation from sd1[aeg] to sd1[dfh]
2. install 5.8-release on sd1[aeg] from the CD set
3. update 5.8-release on sd1[aeg] to 5.8-stable as usual

However, I've run into trouble with step 1.  In more detail, I did this:

(boot 5.6-stable from partitions sd1[aeg])
# newfs /dev/rsd1d
# mount -o softdep,noatime /dev/sd1d /mnt
# tar|tar copy from / to /mnt
#
# newfs /dev/rsd1f
# mount -o softdep,noatime,nodev,nosuid /dev/sd1d /mnt/var
# tar|tar copy from /var to /mnt/var
#
# newfs /dev/rsd1h
# mount -o softdep,noatime,nodev /dev/sd1d /mnt/usr
# tar|tar copy from /usr to /mnt/usr
#
# edit /mnt/etc/fstab to change from mounting sd1[aeg] to sd1[dfh]
# fiddle around a bit, including unmounting /mnt/var
#
# mount
/dev/sd1a on / type ffs (local, noatime, softdep)
mfs:1891 on /tmp type mfs (asynchronous, local, nodev, nosuid, size=500 
1024-blocks)
/dev/sd1e on /var type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd1g on /usr type ffs (local, noatime, nodev, softdep)
mfs:17343 on /usr/tmp type mfs (asynchronous, local, nodev, nosuid, 
size=200 1024-blocks)
/dev/sd1d on /mnt type ffs (local, noatime, softdep)
/dev/sd1h on /mnt/usr type ffs (local, noatime, nodev, nosuid, softdep)
# installboot -v -r /mnt sd1 <- note  -r /mnt  here
Using /mnt as root
installing bootstrap on /dev/rsd1c
using first-stage /mnt/usr/mdec/biosboot, second-stage /mnt/usr/mdec/boot
sd1: softraid volume with 1 disk(s)
sd1: installing boot loader on softraid volume
/mnt/usr/mdec/boot is 5 blocks x 16384 bytes
sd0a: installing boot blocks on /dev/rsd0c, part offset 208
master boot record (MBR) at sector 0
partition 3: type 0xA6 offset 64 size 1465144001
/mnt/usr/mdec/biosboot will be written at sector 64
# reboot

(N.b. all the tar|tar copies use GNU tar from the gtar-1.27.1-static
package.  This handles sparse files, and can copy (create) /dev entries
fine.)

At this point the machine can boot and run sd1[aeg] fine.  *But* if I
enter "boot sr1d:/bsd" at the "boot>" prompt, the machine boots sd1[aeg],
not the desired sd1[dfh].  In other words, at this point my "backup"
sd1[dfh] partitions aren't usefully bootable :(.


Questions
-
I guess my basic question is, what's wrong here and what's the cleanest
way to achieve my desired ends (assuming they're achievable)?

Of course, I could go ahead install 5.8 on sd1[aeg]... but then I wouldn't
have the ability to (easily) boot 5.6-stable as a backup.

I read with interest krw@'s recent Undeadly description

of fixing a softraid-crypto bug in installboot(8),
| I spent some time helping stsp@ fix installboot(8) to handle softraid
| crypto installs, while stoutly maintaining my ignorance of softraid
| crypto. I failed to 

Re: installboot with amd64 root on softraid crypto, NOT 'a' partition

[Nick Holland]

On 11/07/15 19:56, Jonathan Thornburg wrote:
...
> (or
> sd1[dfh] if I enter 'boot sr1d:/bsd' at the "boot>" prompt, assuming there's
   ^
> an OpenBSD installation on those partitions).  

...
> Questions
> -
> I guess my basic question is, what's wrong here and what's the cleanest
> way to achieve my desired ends (assuming they're achievable)?

What's wrong is you are violating the basic rule of "boot from 'a', root
is 'a'".

Anything you do to work around it today may break in the future, I don't
think there has been any real interest with developers in supporting
multiple OpenBSD systems on one physical (or logical) disk.

Going backwards to ignore a problem isn't the OpenBSD Way.

If your goal is to wind-back a bad upgrade, you have your backups
(RIGHT??) of all your config files and data files...just reinstall the
old version that worked and all associated packages (list is at
/var/db/pkg on your backup).

Nick.