Re: [gentoo-user] Re: USE flag 'split-usr' is now global
On Tue, Aug 6, 2019 at 7:41 PM Grant Taylor wrote: > > On 8/6/19 10:28 AM, Rich Freeman wrote: > > An initramfs is just a userspace bootloader that runs on top of linux. > > I disagree. > > To me: > > · A boot loader is something that boots / executes a kernel with > various parameters. > · An initramfs / initrd (concept) is the micro installation that runs > under the kernel that was booted (by the boot loader). Bootloaders are just programs that run other programs, ultimately. We can argue all day about what kinds of programs count as each. But, there is no point arguing definitions. > The initramfs / initrd micro installation does the following: > > 1) fulfills prerequisites (e.g. loads requisite drivers, initializes > networking for and discovers storage, decrypts block devices) > 2) mounts root (and some other) file systems > 3) launches additional init scripts. So, an initramfs doesn't actually have to do ANY of those things, though typically it does. > None of those steps include booting / executing an alternate kernel. Nothing prevents an initramfs from booting an alternate kernel actually, though if it does it will need its own root filesystem initialized one way or another (which could be an initramfs). Linux bootstrapping linux is basically the whole premise behind coreboot. > > > and you don't need to have cruft all over your filesystem to do it. > > Actually yes you do need that cruft. Sure, but you don't need it ALL OVER YOUR FILESYSTEM. Some of the proposed solutions in this thread involved sticking stuff in /bin and so on. An initramfs is nicely bundled up in a single file. > If anything, having an initramfs / initrd means that you're going to > have an additional copy of said core system files in a slightly > different format (initramfs / initrd) that's not usable by the main system. Absolutely true, which means it won't interfere with the main system, as opposed to sticking it in /bin or somewhere where it might. > > and they make your bootstrapping process infinitely flexible. > > Nope. I don't agree. > > There have been many things that I've wanted to do in the past 20 years > that initramfs / initrd aren't flexible enough to do. Such as? It is a linux userspace. It can literally do anything. > But adding an init script that calls tools on the root file system is > infinitely more flexible. I'm not restricted to doing things that > dracut (et al.) know how to do. You don't need to use dracut (et al) to have an initramfs. In fact, you could create your super root filesystem that does all the fancy stuff you claim you can't do with an initramfs, then create a cpio archive of that root filesystem, and now you have an initramfs which does the job. > If I can boot the kernel, point it at a root disk, and launch an init > system, I can do anything I want with the system. Sure, but only if the kernel can find that disk without any userspace code. What if that disk is on the other side of an ssh tunnel? > Let me say it this way. If I can put together commands to do something, > I can get thee system to do it on boot. I don't have to wait for dracut > to be updated to support the next wiz-bang feature. If you know the commands to do something, why would you have to wait for somebody else to implement them? > > > If you want root to be a zip file hosted on a webserver somewhere > > that isn't a problem. If you want root to be a rar file on a > > gpg-encrypted attachment to a message stored on some IMAP server, > > you could do that too. To make all that work you can just script it > > in bash using regular userspace tools like curl or fetchmail, without > > any need to go mucking around with lower-level code. Then once your > > root filesystem is mounted all that bootstrapping code just deletes > > itself and the system operates as if it was never there (strictly > > speaking this isn't required but this is usually how it is done). > > You need /something/ to be your starting root file system. For most > servers / workstations / VMs / containers, that's a local directory > structure. Actually, for most of these the initramfs is the starting root filesystem (just about all linux server installs use one). Usually it just mounts a simple filesystem on a local disk as root and then pivots. However, an initramfs frees you up from the limitation that it be something that can be passed directly on the command line. > Sadly, I think people have thrust additional (IMHO) largely unnecessary > complexity into the init process just to be able to support more exotic > installations. Then, they have said "We want to be consistent in how we > boot, so we need to use this more complex thing everywhere." To which, > many greybeards have responded "I don't need nor want that on my simple > server." Initramfs is popular because it is way more flexible: 1. You can build a fully modular kernel so that you can use the same kernel on every system but not be loading
Re: [gentoo-user] Re: USE flag 'split-usr' is now global
On Tue, Aug 6, 2019 at 7:39 PM Ian Zimmerman wrote: > > On 2019-08-06 12:28, Rich Freeman wrote: > > > > Arguing against this trivial (and IMHO, elegant) solution is tilting > > > at windmills. Specially if it is for ideological reasons instead of > > > technical ones. > > > Some of the solutions I've seen tossed out in this thread are more > > complex than just building your own initramfs from scratch. > > > > An initramfs is just a userspace bootloader that runs on top of linux. > > Nobody has any problem with conventional bootloaders, and if you want > > to do anything with one of those you have to muck around in low-level > > C or assembly. > > There is a difference, and that difference is the reason I dislike > initramfs, not one of the other possible reasons you hypothesize. The > difference is that real Unix processes (not just kernel threads and not > just PID 1) survive from the initramfs stage into the "real Unix" > stage. It's like being able to trace matter back before the Big Bang. They only persist if you allow them to. Most implementations I've seen don't leave anything behind. Typically an initramfs will unlink everything inside, kill any stray processes (if it even forks anything in the first place), and then will exec the new init from the previous init. That starts up init as PID 1 with nothing else running, and no trace of the initramfs remaining. But, sure, you can stick a fork bomb in an initramfs and have it create 10GB of junk in the ramfs as well so that you boot up a crippled system if you really want to. Unless something has changed the kernel is actually built with an empty initramfs built-in by default which loads no matter what. So, the framework of an initramfs is always there, and the only thing that changes is whether it does anything. -- Rich
Re: [gentoo-user] Re: USE flag 'split-usr' is now global
On 8/6/19 10:28 AM, Rich Freeman wrote: An initramfs is just a userspace bootloader that runs on top of linux. I disagree. To me: · A boot loader is something that boots / executes a kernel with various parameters. · An initramfs / initrd (concept) is the micro installation that runs under the kernel that was booted (by the boot loader). The initramfs / initrd micro installation does the following: 1) fulfills prerequisites (e.g. loads requisite drivers, initializes networking for and discovers storage, decrypts block devices) 2) mounts root (and some other) file systems 3) launches additional init scripts. None of those steps include booting / executing an alternate kernel. Remember that the contents of an initramfs / initrd is a micro instillation that simply includes the minimum number of things to do steps 1–3 above. The initrd is literally an image of a block device with a root file system on it that includes the minimum number of things to do steps 1–3 above. The initramfs is a CPIO archive of the minimum number of things to do steps 1–3 above which get extracted to a temporary location (usually a ram disk). Both an initrd and an initramfs are a micro installation for the purpose of initializing the main installation. They are not a boot loader. Nobody has any problem with conventional bootloaders, and if you want to do anything with one of those you have to muck around in low-level C or assembly. That's because a boot loader, e.g. GRUB, lilo, loadlin, do something different and operate at a lower level than system init scripts. There has been talk of gathering up all the stuff you need from /usr to bootstap everything, and then adding some scripting to mount everything. The proposals have been to shove all that in / somewhere (perhaps even in /bin or /sbin). If instead you just stick it all in a cpio archive you basically have an initramfs, Not basically. That /is/ what an initramfs / initrd contains. and you don't need to have cruft all over your filesystem to do it. Actually yes you do need that cruft. Let's back up and talk about what that cruft is. It's the minimum libraries and binaries required to: 1) Requisite libraries - C library - other similar / minimal / unavoidable libraries 2) Requisite binaries - fsck - mount - networking utilites (for iSCSI / NFS / etc.) - other similar / minimal / unavoidable binaries 3) Scripts to bring the rest of the system up - minimal init scripts to go from a post-kernel-execution (what was traditionally /sbin/init) to launching second stage init scripts To me, all of these things are going to exist on the main installation /anyway/. There is going to be minimal, if any, difference between the version put in the initramfs / initrd and what's in the main / (root). So, to me, what you're calling "cruft" is core system files that are going to exist anyway. If anything, having an initramfs / initrd means that you're going to have an additional copy of said core system files in a slightly different format (initramfs / initrd) that's not usable by the main system. There are already configurable and modular solutions like dracut which do a really nice job of building one, Yes. We've had many different tools in the last 28 years of Linux and longer for Unix for making the management of the boot process simpler. It comes down to loading the kernel from something and starting the kernel with proper parameters (that's the boot loader's job) and starting something (traditionally /sbin/init) from some storage somewhere. and they make your bootstrapping process infinitely flexible. Nope. I don't agree. There have been many things that I've wanted to do in the past 20 years that initramfs / initrd aren't flexible enough to do. But adding an init script that calls tools on the root file system is infinitely more flexible. I'm not restricted to doing things that dracut (et al.) know how to do. If I can boot the kernel, point it at a root disk, and launch an init system, I can do anything I want with the system. Let me say it this way. If I can put together commands to do something, I can get thee system to do it on boot. I don't have to wait for dracut to be updated to support the next wiz-bang feature. If you want root to be a zip file hosted on a webserver somewhere that isn't a problem. If you want root to be a rar file on a gpg-encrypted attachment to a message stored on some IMAP server, you could do that too. To make all that work you can just script it in bash using regular userspace tools like curl or fetchmail, without any need to go mucking around with lower-level code. Then once your root filesystem is mounted all that bootstrapping code just deletes itself and the system operates as if it was never there (strictly speaking this isn't required but this is usually how it is done). You need /something/ to be
[gentoo-user] Re: USE flag 'split-usr' is now global
On 2019-08-06 12:28, Rich Freeman wrote: > > Arguing against this trivial (and IMHO, elegant) solution is tilting > > at windmills. Specially if it is for ideological reasons instead of > > technical ones. > Some of the solutions I've seen tossed out in this thread are more > complex than just building your own initramfs from scratch. > > An initramfs is just a userspace bootloader that runs on top of linux. > Nobody has any problem with conventional bootloaders, and if you want > to do anything with one of those you have to muck around in low-level > C or assembly. There is a difference, and that difference is the reason I dislike initramfs, not one of the other possible reasons you hypothesize. The difference is that real Unix processes (not just kernel threads and not just PID 1) survive from the initramfs stage into the "real Unix" stage. It's like being able to trace matter back before the Big Bang. -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com.
Re: [gentoo-user] Re: USE flag 'split-usr' is now global
On Tue, Aug 6, 2019 at 5:54 PM Grant Taylor < gtay...@gentoo.tnetconsulting.net> wrote: [...] > > Arguing against this trivial (and IMHO, elegant) solution is tilting at > > windmills. Specially if it is for ideological reasons instead of > > technical ones. > > Please clarify what "this trivial solution" is. Are you referring to > initramfs / initrd or the 'split-user' USE flag? The trivial solution (IMO) is to use an initramfs. Rich gave a much more elaborated answer. Regards. -- Dr. Canek Peláez Valdés Profesor de Carrera Asociado C Departamento de Matemáticas Facultad de Ciencias Universidad Nacional Autónoma de México
Re: [gentoo-user] Re: USE flag 'split-usr' is now global
On 8/6/19 9:54 AM, Canek Peláez Valdés wrote: If it's computable it can be done, of course. Therefore it can be done, currently. I don't think nobody has said it absolutely cannot be done. >.< So it sounds like it's a question of /how/ compatible / possible it is. It seems as if there is enough incompatibility / problems that multiple people are comfortable saying that it can't be done on some level. The thing is: 1. How much work implies to get it done. 2. Who is gonna do said work. The answer to 1 is "a lot", since (as someone mentioned in the thread) it involves changing not only the init (nevermind systemd; *ALL* init systems), but all applications that may require to use binaries in /usr before it's mounted. The answer to 2 is, effectively, "nobody", since it requires a big coordinated effort, stepping into the toes of several projects, significantly augmenting their code complexity for a corner case[1] that can be trivially be solved with an initramfs, which it just works. I don't currently feel like I can give a proper response to this. 1) I don't have the time to lab this and try things. 2) I don't want to further hijack this thread and start discussing what precisely is and is not broken. 3) I question — from a place of ignorance — just how much effort there is for #1. Arguing against this trivial (and IMHO, elegant) solution is tilting at windmills. Specially if it is for ideological reasons instead of technical ones. Please clarify what "this trivial solution" is. Are you referring to initramfs / initrd or the 'split-user' USE flag? -- Grant. . . . unix || die
Re: [gentoo-user] No profile 17.1 for 32-bit (x86) installs?
On Tue, Aug 06, 2019 at 07:56:49AM +0200, Raffaele Belardi wrote > On x86 system there is no lib32 nor lib64, everything is already under lib. On 32-bit systems, everything is in /lib. On 64-bit systems, everything is now in /lib64. I wonder if they're laying the groundwork for the appearance of /lib128 one of these days . -- Walter Dnes I don't run "desktop environments"; I run useful applications
Re: [gentoo-user] Re: USE flag 'split-usr' is now global
On Tue, Aug 6, 2019 at 11:54 AM Canek Peláez Valdés wrote: > > Arguing against this trivial (and IMHO, elegant) solution is tilting at > windmills. Specially if it is for ideological reasons instead of technical > ones. > ++ Some of the solutions I've seen tossed out in this thread are more complex than just building your own initramfs from scratch. An initramfs is just a userspace bootloader that runs on top of linux. Nobody has any problem with conventional bootloaders, and if you want to do anything with one of those you have to muck around in low-level C or assembly. There has been talk of gathering up all the stuff you need from /usr to bootstap everything, and then adding some scripting to mount everything. The proposals have been to shove all that in / somewhere (perhaps even in /bin or /sbin). If instead you just stick it all in a cpio archive you basically have an initramfs, and you don't need to have cruft all over your filesystem to do it. There are already configurable and modular solutions like dracut which do a really nice job of building one, and they make your bootstrapping process infinitely flexible. If you want root to be a zip file hosted on a webserver somewhere that isn't a problem. If you want root to be a rar file on a gpg-encrypted attachment to a message stored on some IMAP server, you could do that too. To make all that work you can just script it in bash using regular userspace tools like curl or fetchmail, without any need to go mucking around with lower-level code. Then once your root filesystem is mounted all that bootstrapping code just deletes itself and the system operates as if it was never there (strictly speaking this isn't required but this is usually how it is done). I doubt we'll see a /usr merge anytime soon, or any huge rush to break a separate /usr completely without an initramfs. However, there are already use cases known where a separate /usr can cause problems without either an initramfs or some kind of early-boot shell script (there have already been solutions tossed out for that which are much simpler than the ones in this thread). The biggest example I've heard is bluetooth keyboards - that was what made most of the zealots give up on supporting anything that could possibly be needed for bootstrapping being in /, as bluetooth has a bunch of deps and at that point you might as well shove gnome in /bin. So, for the forseeable future your system probably won't break if you are using a separate /usr, but if it does the policy has been established for a long time that nobody is obligated to fix it (nor are they prohibited from doing so). Really, though, you should take the time to appreciate an initramfs whether you decide to use one or not. They're a really elegant solution to the problem. Sometimes I think that half the objection is due to the fact that they use cpio which is a bit obscure - if they were tarballs people would be tearing them apart and playing with them more. -- Rich
Re: [gentoo-user] Re: USE flag 'split-usr' is now global
On Mon, Aug 5, 2019 at 9:38 PM Grant Taylor < gtay...@gentoo.tnetconsulting.net> wrote: [...] > I don't have any current first hand experience with /usr being a > separate file system without using an initramfs / initrd. So I'm going > to have to take what you, and others, say on faith that it can't > /currently/ be done. But I've got to say, that I find that idea > disturbing and highly suspicious. If it's computable it can be done, of course. Therefore it can be done, currently. I don't think nobody has said it absolutely cannot be done. The thing is: 1. How much work implies to get it done. 2. Who is gonna do said work. The answer to 1 is "a lot", since (as someone mentioned in the thread) it involves changing not only the init (nevermind systemd; *ALL* init systems), but all applications that may require to use binaries in /usr before it's mounted. The answer to 2 is, effectively, "nobody", since it requires a big coordinated effort, stepping into the toes of several projects, significantly augmenting their code complexity for a corner case[1] that can be trivially be solved with an initramfs, which it just works. Arguing against this trivial (and IMHO, elegant) solution is tilting at windmills. Specially if it is for ideological reasons instead of technical ones. Regards. [1] I firmly believe that's the situation nowadays. -- Dr. Canek Peláez Valdés Profesor de Carrera Asociado C Departamento de Matemáticas Facultad de Ciencias Universidad Nacional Autónoma de México
Re: [gentoo-user] No profile 17.1 for 32-bit (x86) installs?
On Tuesday, 6 August 2019 06:56:49 BST Raffaele Belardi wrote: > Walter Dnes wrote: > >I just updated Gentoo on my old backup machine, an 11-year-old Dell > > > > Inspiron 530 desktop, and there's no mention of profile 17.1 in either > > "eselect profile list" or "eselect news list". I'm not looking for > > extra hassle, but I wanted to make sure I didn't miss anything obvious. > > Same here, nothing to worry about: > > "In the new profiles, the lib->lib64 compatibility symlink is removed. > 64-bit libraries need to be installed directly to lib64. /lib > and /usr/lib become real directories, that are used for cross-arch > and native non-library packages (gcc, clang) and 32-bit libraries > on the multilib profile (which improves compatibility with prebuilt x86 > packages)." [1] > > On x86 system there is no lib32 nor lib64, everything is already under lib. > > raffaele > > [1] > https://www.gentoo.org/support/news-items/2019-06-05-amd64-17-1-profiles-are > -now-stable.html As raffaele said there was an earlier news time for testing 17.1, which appears to have been removed: $ eselect news read 10 2017-12-26-experimental-amd64-17-1-profiles !!! Warning: News item "2017-12-26-experimental-amd64-17-1-profiles" no longer exists But there is another more up to date news item titled: "2019-06-05-amd64-17-1-profiles-are-now-stable" which is as per URL [1] shown above and contains all the instructions needed to undertake the fs migration. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Re: HACK: Boot without an initramfs / initrd while maintaining a separate /usr file system.
On Mon, 5 Aug 2019 18:04:00 -0600, Grant Taylor wrote: > > Don't you have to go through some extra hoops (a flag to the mount > > command or something) to mount over a non-empty directory? > > Nope. > > I don't recall ever needing to do anything like that in Linux. Fuse complain about this, that's the only time I've seen it. -- Neil Bothwick Top Oxymorons Number 39: Almost exactly pgplSNCeSYf1p.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Re: acct-group packages ??
On Mon, 5 Aug 2019 09:17:17 -0400, Michael Orlitzky wrote: > > I've not checked lately, but policy was that if an ebuild change did > > not result in differences in the installed files, there was no need > > for a version bump. This avoids needless recompiling of packages. > > > > Realistically, almost all ebuild changes should incur a new revision. I > would much rather recompile 100 packages *and have it work* than compile > 10 packages and have it crash three times requiring manual intervention > because the tree is so screwed up. > > We have better guidelines these days: > > https://devmanual.gentoo.org/general-concepts/ebuild-revisions > > but they still give developers too much freedom to be lazy and commit > important changes without a revision. The "straight to stable" advice > contradicts our existing stabilization policy, and the USE flag advice > says that you can rely on a non-default, portage-only feature to prevent > breakage. That's pretty much how I remember it. If the existing version crash, then the binaries have changed so it should be bumped, but if a dev missed out a new DEPEND for chromium of libreoffice that I happen to have already installed, I don't want to have to waste hours of CPU time recompiling to exactly the same end point. The most important statement in the policy is also the hardest to enforce "Developers are encouraged to use common sense" :-O -- Neil Bothwick What is a "free" gift ? Aren't all gifts free? pgpod207Cbmq1.pgp Description: OpenPGP digital signature
Re: [gentoo-user] HACK: Boot without an initramfs / initrd while maintaining a separate /usr file system.
On Mon, 5 Aug 2019 10:30:58 -0600, Grant Taylor wrote: > > How will you make sure installations of the same binaries are > > installed/copied in both underlying and mounted /usr/* fs and kept > > in sync? By changing all affected ebuilds? > > I don't have an answer to this qustion. I've not needed an answer to > this question. > > I think I would likely create a script that would copy specific files > from the /usr path to the underlying /(usr) path as needed. > > I doubt there would be many files. You could use rsync with the --existing flag to update all files in your bind mounted directories, and run it as a post-emerge script. It's a horrible hack, I like it! -- Neil Bothwick C Error #011: First C Program, huh? pgpeh_Lp_t0fz.pgp Description: OpenPGP digital signature
