Re: KARL relinking deletes second kernel install set
In case this problem is of any concern to anyone else and not just myself, the following is the minimal patch that I could come up with that preserves the idempotency of /usr/libexec/reorder_kernel and thus does not create any unnecessary corner cases that need to be taken into account separately: --- reorder_kernel.orig Thu May 7 10:52:19 2020 +++ reorder_kernel Fri May 29 03:56:58 2020 @@ -46,7 +46,6 @@ if [[ -f $KERNEL_DIR.tgz ]]; then # stdout again to a new logfile. exec 1>$LOGFILE tar -C $KERNEL_DIR -xzf $KERNEL_DIR.tgz $KERNEL - rm -f $KERNEL_DIR.tgz fi if ! sha256 -C $SHA256 /bsd; then On Thu, May 28, 2020 at 4:25 AM Connor Schech wrote: > It's important to note that the SHA256 hash checked against /bsd stored in > /var/db each time reorder_kernel is called has no bearing on the integrity > or consistency of the files present at compile-time for the next reordering > in /usr/share/relink. If a signed SHA256.sig file (or two, GENERIC.sig and > GENERIC.MP.sig) for all the individual link-kit objects from the 53mb > kernel.tgz file embedded in base67.tgz were also present inside base67.tgz > in the relink dir next to kernel.tgz, all the signatures could be verified > before relinking for either GENERIC or GENERIC.MP and the objects > reordered each time from either relink location would be guaranteed to be > the same ones from the initial release. Then repopulating them if they are > deleted in either or both locations wouldn't require reordering two kernels > or maintaining two sets of uncompressed objects, if need be kernel.tgz > could be extracted again for either configuration and the signatures > verified before running make once on one set of objects. > > On Thu, May 28, 2020, 03:57 Stuart Henderson wrote: > >> On 2020/05/27 22:50, Connor Schech wrote: >> > I compressed the GENERIC link kit with tar czf and it becomes 114MB, all >> > other things being equal to what is done now. That becomes significant >> for >> > users with many instances or embedded devices. There are trade-offs >> > involved, so to speak. >> >> Relinking once is already quite heavy and makes some systems unusable, >> this at least applies to slower machines using the architectures with >> wide hardware support in the kernel, i386 probably being the worst case >> - some of the smaller arches like landisk cope better, partly because >> there's less in the kernel and partly because they use ld.bfd which >> uses less RAM. Extracting from tar.gz and linking twice is going to >> be way too much for these. >> >>
Re: KARL relinking deletes second kernel install set
It's important to note that the SHA256 hash checked against /bsd stored in /var/db each time reorder_kernel is called has no bearing on the integrity or consistency of the files present at compile-time for the next reordering in /usr/share/relink. If a signed SHA256.sig file (or two, GENERIC.sig and GENERIC.MP.sig) for all the individual link-kit objects from the 53mb kernel.tgz file embedded in base67.tgz were also present inside base67.tgz in the relink dir next to kernel.tgz, all the signatures could be verified before relinking for either GENERIC or GENERIC.MP and the objects reordered each time from either relink location would be guaranteed to be the same ones from the initial release. Then repopulating them if they are deleted in either or both locations wouldn't require reordering two kernels or maintaining two sets of uncompressed objects, if need be kernel.tgz could be extracted again for either configuration and the signatures verified before running make once on one set of objects. On Thu, May 28, 2020, 03:57 Stuart Henderson wrote: > On 2020/05/27 22:50, Connor Schech wrote: > > I compressed the GENERIC link kit with tar czf and it becomes 114MB, all > > other things being equal to what is done now. That becomes significant > for > > users with many instances or embedded devices. There are trade-offs > > involved, so to speak. > > Relinking once is already quite heavy and makes some systems unusable, > this at least applies to slower machines using the architectures with > wide hardware support in the kernel, i386 probably being the worst case > - some of the smaller arches like landisk cope better, partly because > there's less in the kernel and partly because they use ld.bfd which > uses less RAM. Extracting from tar.gz and linking twice is going to > be way too much for these. > >
Re: KARL relinking deletes second kernel install set
I compressed the GENERIC link kit with tar czf and it becomes 114MB, all other things being equal to what is done now. That becomes significant for users with many instances or embedded devices. There are trade-offs involved, so to speak. On Wed, May 27, 2020, 21:30 Theo de Raadt wrote: > Connor Schech wrote: > > > I tried sending this via sendbug but my host was invalid. this is a > > 'system' problem. > > > > When installing OpenBSD 6.7 amd64 and selecting both bsd.mp and bsd in > the > > install sets, indicating that both are desired, on a single core machine > or > > VM only bsd is reordered and bsd.mp is discarded; if switching to SMP on > > the VM or migrating to another SMP host, there is no way for KARL to > > recover and relink the GENERIC.MP objects because it only reordered > objects > > for GENERIC the first time, even after the SHA256 hash for /bsd is > updated. > > The installed is quite clear: it picks one, or the other. > > > This single path of execution of KARL has bothered me for several > releases, > > since it ignores the fact there are two types of kernels and also lots of > > usage scenarios that need both or handle reordering both types properly. > > The machine you install on remains the same. > > If you change the machine, why don't you reinstall. > > Seriously, why is this our problem? > > > Since reordering the kernel is not optional in the install nor > controllable > > after the install with rcctl like library_aslr, reordering both the sp > and > > mp kernels if both install sets are selected instead of removing one > makes > > sense, because all users then have access to both types of > > properly-reordered official kernels from the default install, and can > also > > choose one or the other install set to begin with if both are not needed. > > Reordering a kernel should work at any point in time when it has been > > checkpointed using its current SHA256 hash, regardless of whether the > > system was initially booted in MP mode or SP mode. > > On amd64, the GENERIC.MP link-kit is 347M. Upon install, we delete > the other kernel link kit to save space. > > I think the GENERIC link kit is around 300MB. > > If we keep an additional 300MB of storage on the disk, shall we direct > all the complaints to you? What do you think the response will be? >
Re: KARL relinking deletes second kernel install set
Connor Schech wrote: > I tried sending this via sendbug but my host was invalid. this is a > 'system' problem. > > When installing OpenBSD 6.7 amd64 and selecting both bsd.mp and bsd in the > install sets, indicating that both are desired, on a single core machine or > VM only bsd is reordered and bsd.mp is discarded; if switching to SMP on > the VM or migrating to another SMP host, there is no way for KARL to > recover and relink the GENERIC.MP objects because it only reordered objects > for GENERIC the first time, even after the SHA256 hash for /bsd is updated. The installed is quite clear: it picks one, or the other. > This single path of execution of KARL has bothered me for several releases, > since it ignores the fact there are two types of kernels and also lots of > usage scenarios that need both or handle reordering both types properly. The machine you install on remains the same. If you change the machine, why don't you reinstall. Seriously, why is this our problem? > Since reordering the kernel is not optional in the install nor controllable > after the install with rcctl like library_aslr, reordering both the sp and > mp kernels if both install sets are selected instead of removing one makes > sense, because all users then have access to both types of > properly-reordered official kernels from the default install, and can also > choose one or the other install set to begin with if both are not needed. > Reordering a kernel should work at any point in time when it has been > checkpointed using its current SHA256 hash, regardless of whether the > system was initially booted in MP mode or SP mode. On amd64, the GENERIC.MP link-kit is 347M. Upon install, we delete the other kernel link kit to save space. I think the GENERIC link kit is around 300MB. If we keep an additional 300MB of storage on the disk, shall we direct all the complaints to you? What do you think the response will be?
KARL relinking deletes second kernel install set
Hi, I tried sending this via sendbug but my host was invalid. this is a 'system' problem. When installing OpenBSD 6.7 amd64 and selecting both bsd.mp and bsd in the install sets, indicating that both are desired, on a single core machine or VM only bsd is reordered and bsd.mp is discarded; if switching to SMP on the VM or migrating to another SMP host, there is no way for KARL to recover and relink the GENERIC.MP objects because it only reordered objects for GENERIC the first time, even after the SHA256 hash for /bsd is updated. This single path of execution of KARL has bothered me for several releases, since it ignores the fact there are two types of kernels and also lots of usage scenarios that need both or handle reordering both types properly. Since reordering the kernel is not optional in the install nor controllable after the install with rcctl like library_aslr, reordering both the sp and mp kernels if both install sets are selected instead of removing one makes sense, because all users then have access to both types of properly-reordered official kernels from the default install, and can also choose one or the other install set to begin with if both are not needed. Reordering a kernel should work at any point in time when it has been checkpointed using its current SHA256 hash, regardless of whether the system was initially booted in MP mode or SP mode. Hopefully this makes sense, Connor
