[gentoo-dev] Re: usr merge
Richard Yao posted on Wed, 06 Apr 2016 00:15:58 -0400 as excerpted: >> On Apr 4, 2016, at 9:19 PM, William Hubbs wrote: >> >> All, >> >> I thought that since the usr merge is coming up again, and since I lost >> track of the message where it was brought up, I would open a new thread >> to discuss it. >> >> When it came up before, some were saying that the /usr merge violates >> the fhs. I don't remember the specifics of what the claim was at the >> time, (I'm sure someone will point it out if it is still a concern). > > Here are the violations: > > http://refspecs.linuxfoundation.org/FHS_3.0/ fhs-3.0.html#binEssentialUserCommandBinaries > > http://refspecs.linuxfoundation.org/FHS_3.0/ fhs-3.0.html#sbinSystemBinaries > > http://refspecs.linuxfoundation.org/FHS_3.0/ fhs-3.0.html#libEssentialSharedLibrariesAndKern (Those links are wrapped and I'm not bothering to jump thru the hoops to unwrap them, since readers can either unwrap them manually or refer to the parent post I'm quoting for the unwrapped versions.) If those are the "violations", then putting everything in /usr and making the /bin and /sbin locations symlinks isn't going to be a problem, since /bin and /sbin are specifically allowed to contain symlinks to the executables, instead of the executables themselves, and if the dirs themselves are symlinks to the locations in /usr with the files, that fulfills that requirement. And the requirement for /lib is rather vague, saying only that it contains the libs linked by the executables in /bin and /sbin. So once / bin and /sbin are symlinks to the dirs with the executables, /lib (or the arch-specific alternative libdirs) can be a symlink as well. Tho I must say doing the reverse, making either /usr itself or /usr/bin and /usr/sbin symlinks to the root dirs, as I did here, actually makes more sense and bends the rules less. Basically, what the FHS says, at least in the 3.0 version you linked, is that the executables must be reachable via whatever specific path, but using symlinks to do it is fine. Which means the merge is allowed, as long as symlinks allow the executables to be reached by their specifically defined paths. And I'm not aware of anyone seriously proposing that said symlinks be omitted, so... -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Re: [gentoo-dev] usr merge
> On Apr 4, 2016, at 9:19 PM, William Hubbs wrote: > > All, > > I thought that since the usr merge is coming up again, and since I lost > track of the message where it was brought up, I would open a > new thread to discuss it. > > When it came up before, some were saying that the /usr merge violates > the fhs. I don't remember the specifics of what the claim was at the > time, (I'm sure someone will point it out if it is still a concern). Here are the violations: http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#binEssentialUserCommandBinaries http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#sbinSystemBinaries http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#libEssentialSharedLibrariesAndKern > > I don't think creating usr merged stages would be that difficult. I > think it would just be a matter of creating a new version of baselayout > that puts these symlinks in place: > > /bin->usr/bin > /lib->usr/lib > /lib32->usr/lib32 > /lib64->usr/lib64 > /sbin->usr/bin > /usr/sbin->bin > > Once that is in place in a new baselayout, I think portage's colission > detection would be able to catch files that had the same names and were > originally in different paths when building the new stages. We will have users whose system configurations rely on the FHS complain about us breaking boot if we force this. > I put some thought also in how to nigrate live systems, and I'm not sure > what the best way to do that is. I wrote a script, which would do it in > theory, but I haven't tested because I only have one system and if > it breaks it the only way back would be to reinstall. > > The script is attached. > > > Thoughts on any of this? This was invented in Solaris and copied by RHEL. The upgrade path for the /usr merge on those systems is a complete reinstall. Upgrading from RHEL6 to RHEL7 this Solaris 10 to Solaris 11 is not supported. The reason being that there are ways of configuring the system boot process with the original layout that break if you try using scripts to migrate to the new one. A USE flag for the /usr merge that is off by default would allow us to have both worlds without putting any systems at risk. This has been an almost annual debate. I do not have much incentive to keep up with it. The reason I ever bothered to explain why this is a bad idea for Gentoo was that I was concerned for our user base. My systems would not be negatively affected and arguing against changes that originated in Solaris is awkward for me. If others are not willing to be advocates for those users that would only make themselves known after an a fundamental change has been made and people are determined to go ahead with this, I suggest having and testing a plan for backing out the change should the backlash from users after systems break be more than people can stomach. This is not the sort of change we should make without an "exit strategy". > William > >
[gentoo-dev] Re: usr merge
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 04/05/2016 12:53 PM, Alexis Ballier wrote: > On Tuesday, April 5, 2016 2:26:53 PM CEST, Duncan wrote: >> >> As I said in the other thread, I'm running merged /usr and >> bin/sbin here, except that I merged them the other way, with /usr >> -> . so everything in /usr is now in /. >> >> Portage has long "just worked" in that regard, tho I've no idea >> whether the other PMs do. Portage has enough intelligence to >> avoid replacing a file with a symlink pointing to it (and thus to >> itself once the replacement is done), regardless of which way the >> directory symlinks point. >> >> As such, coreutils "just works". If the two would end up in the >> same canonical location, the file wins and the symlink isn't >> installed. > > What about the unlikely case with two files ? > Having actually run this way myself, I did find one case that I haven't filed a bug for yet: the plymouth ebuild tries to install symlinks in /sbin pointing at /usr/sbin, and portage chose to install the symlinks instead of the real files, for whatever reason (apparently because the $ED/sbin directory is created after the $ED/usr directory). Because of this, it might be best to ensure that packages that do install in both places are modified not to do so under such a configuration. - -- Jonathan Callen -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBCgAGBQJXBFMbAAoJEEIQbvYRB3mg2e4P/2lPBxpyjY311LP7gN2Nndn4 Dd4EtFbh8tQWoedPJQgr2CIeVgpPFA7l/stuvcoZAqLVDuFnn4ZmMWSIQOgHmgPp +mIiCDPuLMjhqw/yINlTGGVVhffHFG4PrHcd2MwP6Gm9ME0NH8+Z0cgAznHsHQ5c lgNdfXDsgBdrSrKu5/JTw7jDOv1A1TwIACJoLpEYZTlVCBClp6J01kqH1oyEzPf8 FO6fqAvFJXCq1um6/+ve8LOpS0OLBpg0dh5kcdkFgV1430FqNwUczMINhav5J0mp qTAIZTO4OSLxyswOUiKoxROl4xrQ1ByYi1ZF7g24oh7M1fmkreNClrhJ1kA3M6ff OJ3LJ6m350LEIVzAED66pnKOTNDOLJSaz6MsPk8CHzuJ2RCMatKjBA3Lb0tkkepp 5LOCBXbnVfSPRI+TQM91cHXVnh87T1zZSeGT8qOCfNoF7rFWNSlpIRnxMeeFlv2n 0kXfJo9YeiUAA9BYXBryMIsWr4StM4I9oq0ITc7h9WmB/WKW6zJhl7WHd7SgiePW Lb2fHJtz0R8dUIc53Yxuls1Cbt8AUAFYmN9Ve615cVLs3+jO8HWmwiuFfiYH71k1 JaS51cgBjPBnQuiET0iNxu/gjIekwIjoNptn/cCr9IZ4jnZ9L13ai6Wug49vUwwK bed4Tt3nl8GSbRtlV+rk =PHpB -END PGP SIGNATURE-
[gentoo-dev] Last-rites: dev-libs/ptypes
# David Seifert (05 Apr 2016) # No upstream release since 2007, no revdeps in tree, # version in tree outdated, bug #454702. # Masking for removal in 30 days dev-libs/ptypes
Re: [gentoo-dev] usr merge
On Tuesday, April 5, 2016 2:26:53 PM CEST, Duncan wrote: Alexis Ballier posted on Tue, 05 Apr 2016 12:10:51 +0200 as excerpted: On Tuesday, April 5, 2016 3:19:59 AM CEST, William Hubbs wrote: [...] ... As I said in the other thread, I'm running merged /usr and bin/sbin here, except that I merged them the other way, with /usr -> . so everything in /usr is now in /. Portage has long "just worked" in that regard, tho I've no idea whether the other PMs do. Portage has enough intelligence to avoid replacing a file with a symlink pointing to it (and thus to itself once the replacement is done), regardless of which way the directory symlinks point. As such, coreutils "just works". If the two would end up in the same canonical location, the file wins and the symlink isn't installed. What about the unlikely case with two files ? There are a few individual package bugs, including one open right now where the gcc-config ebuild does an unconditional rm -f of any old versions found in its old sbin location, even when it removes the executable it just installed into the bin location, because they're the same canonical location. (Bug number for that and other bugs in the reply on the other thread.) And cmake can get mixed up in some instances so a few packages (baloo) have problems with some cmake versions. But the bugs aren't with portage, they're with the ebuild or the upstream sources, and the number of them I've run into in the two years plus I've been running merged can fit on one hand. Certainly, they're few enough to deal with on a case-by-case basis. Yeah, these cases need to be handled on a case by case basis, there's no other choice anyway :) If we want to move on on this, we should definitely track these properly. [...] 1) Unless one is sure of the actual install path used and uses it, equery belongs and I assume q and similar tools with the same query, need the bare file name, not the full path, because you might use the wrong one. For instance, /bin/grep, /sbin/grep, /usr/sbin/grep, and /usr/bin/grep, are all the same file due to directory level symlinks. However, if you try equery belongs with all four paths, only one of them will return the hit you're looking for. Easily solved by simply doing equery belongs grep (no path), and letting equery supply the installed path in its results. That's actually how I find out which path a file was actually installed to, these days, as well. =:^) no real issue here, and anyway, since it parses portage tree & profiles, support for guessing usr-merge can be added to restore old behavior 2) revdep-rebuild will, in its default configuration, end up processing files using all paths. So grep, to use the same example as above, will be processed four times, one each for /bin and /sbin, /usr/bin and /usr/ sbin. While it's possible to reconfigure revdep-rebuild to ignore three of those paths and only check one of them (and similarly, ignore one of /lib64 and /usr/lib64, etc), doing so will result in revdep-rebuild complaining about unowned broken binaries if they're installed using a different path than the one it processed. That's not a big problem, because equery belongs (without the path) will tell you what owns it as well as the installed path it used, and then that package can be remerged manually, if needed. kind of defeats the point of revdep-rebuild though :) [...] # Create the /usr merge compatibility symlinks for dir in /bin /sbin; do run_command rm -rf $dir ---> where are the 'ln' and 'rm' taken from after this step ? If this fails here, you're also leaving the system in a broken state. run_command ln -s usr/bin $dir done In my case I was using the mc binary, which continued to run after the transfer, so it wasn't an issue. But using the individual ln and rm binaries, while they'll still be on the path, you may need to run hash -r in the shell so it forgets the old location and checks the path again. Similar thing for the libs, since the lib cache will be screwed after the move, until the symlink has been created so the old paths work again, at least. In my case I was using the mc binary which continued to run and thus could be used to create the symlink, but for one-shot executables like ln, that could be an issue. One way around the problem would be to create a few static-linked executables for the purpose, and ship them in a tarball that's unpacked to an unchanging tmp location for the run, so they can be called from there to finish up regardless of whether the dynamically linked normal executables can still be invoked. Smart use of the shell's builtin read, echo and redirection could probably do some of it too, but can ln be emulated using shell builtins? I'd rather use a binary that ensures everything that needs to be loaded is loaded at the beginning and even better if it can enusre system consistency and can do rollbacks in case of failure. A pyth
[gentoo-dev] Re: usr merge
Alexis Ballier posted on Tue, 05 Apr 2016 12:10:51 +0200 as excerpted: > On Tuesday, April 5, 2016 3:19:59 AM CEST, William Hubbs wrote: > [...] >> I don't think creating usr merged stages would be that difficult. I >> think it would just be a matter of creating a new version of baselayout >> that puts these symlinks in place: >> >> /bin->usr/bin /lib->usr/lib /lib32->usr/lib32 /lib64->usr/lib64 > > (OT: maybe it'd be a good oportunity to kill SYMLINK_LIB too :p) > >> /sbin->usr/bin /usr/sbin->bin >> >> Once that is in place in a new baselayout, I think portage's colission >> detection would be able to catch files that had the same names and were >> originally in different paths when building the new stages. > > > I think that prior to that we have to ensure that packages with intra > collisions are not merged: What happens with current coreutils ebuilds > that install, e.g., /bin/seq and a /usr/bin/seq symlink to it ? > I haven't looked at the actual code, thus I can only assume there are no > guarantees, which is definitely bad. As I said in the other thread, I'm running merged /usr and bin/sbin here, except that I merged them the other way, with /usr -> . so everything in /usr is now in /. Portage has long "just worked" in that regard, tho I've no idea whether the other PMs do. Portage has enough intelligence to avoid replacing a file with a symlink pointing to it (and thus to itself once the replacement is done), regardless of which way the directory symlinks point. As such, coreutils "just works". If the two would end up in the same canonical location, the file wins and the symlink isn't installed. There are a few individual package bugs, including one open right now where the gcc-config ebuild does an unconditional rm -f of any old versions found in its old sbin location, even when it removes the executable it just installed into the bin location, because they're the same canonical location. (Bug number for that and other bugs in the reply on the other thread.) And cmake can get mixed up in some instances so a few packages (baloo) have problems with some cmake versions. But the bugs aren't with portage, they're with the ebuild or the upstream sources, and the number of them I've run into in the two years plus I've been running merged can fit on one hand. Certainly, they're few enough to deal with on a case-by-case basis. >> I put some thought also in how to nigrate live systems, and I'm not >> sure what the best way to do that is. I wrote a script, which would do >> it in theory, but I haven't tested because I only have one system and >> if it breaks it the only way back would be to reinstall. > > > Does it behave properly wrt portage's way of tracking of package files? > I remember that modifying files owned by portage used to cause issues. What I did for my migration was simply move everything from /usr to / and create the /usr -> . symlink. I did that with mc, and kept it running just in case I ended up not being able to start something, until I had the symlink in place and had tested starting various things, including X/ KDE, so I knew it was working. Similarly for the sbin -> bin moves and symlinks. The moves worked fine, and with the directory symlinks replacing the old dirs, everything else, including portage on updates after that, worked just fine. There are a couple things that behave slightly differently regarding packages, that one needs to be aware of, but in general it just works. Those couple things are: 1) Unless one is sure of the actual install path used and uses it, equery belongs and I assume q and similar tools with the same query, need the bare file name, not the full path, because you might use the wrong one. For instance, /bin/grep, /sbin/grep, /usr/sbin/grep, and /usr/bin/grep, are all the same file due to directory level symlinks. However, if you try equery belongs with all four paths, only one of them will return the hit you're looking for. Easily solved by simply doing equery belongs grep (no path), and letting equery supply the installed path in its results. That's actually how I find out which path a file was actually installed to, these days, as well. =:^) 2) revdep-rebuild will, in its default configuration, end up processing files using all paths. So grep, to use the same example as above, will be processed four times, one each for /bin and /sbin, /usr/bin and /usr/ sbin. While it's possible to reconfigure revdep-rebuild to ignore three of those paths and only check one of them (and similarly, ignore one of /lib64 and /usr/lib64, etc), doing so will result in revdep-rebuild complaining about unowned broken binaries if they're installed using a different path than the one it processed. That's not a big problem, because equery belongs (without the path) will tell you what owns it as well as the installed path it used, and then that package can be remerged manually, if needed. So with revdep-rebuil
Re: [gentoo-dev] usr merge
On Tuesday, April 5, 2016 3:19:59 AM CEST, William Hubbs wrote: [...] I don't think creating usr merged stages would be that difficult. I think it would just be a matter of creating a new version of baselayout that puts these symlinks in place: /bin->usr/bin /lib->usr/lib /lib32->usr/lib32 /lib64->usr/lib64 (OT: maybe it'd be a good oportunity to kill SYMLINK_LIB too :p) /sbin->usr/bin /usr/sbin->bin Once that is in place in a new baselayout, I think portage's colission detection would be able to catch files that had the same names and were originally in different paths when building the new stages. I think that prior to that we have to ensure that packages with intra collisions are not merged: What happens with current coreutils ebuilds that install, e.g., /bin/seq and a /usr/bin/seq symlink to it ? I haven't looked at the actual code, thus I can only assume there are no guarantees, which is definitely bad. I put some thought also in how to nigrate live systems, and I'm not sure what the best way to do that is. I wrote a script, which would do it in theory, but I haven't tested because I only have one system and if it breaks it the only way back would be to reinstall. Does it behave properly wrt portage's way of tracking of package files? I remember that modifying files owned by portage used to cause issues. What should baselayout ebuild do on systems that have not run that script ? Also, I think your script may not work: # copy binaries for dir in /bin /sbin /usr/sbin; do run_command cp -a $dir/* /usr/bin done ---> Here it is important to ensure nothing /usr/bin/ is a symlink to /bin, otherwise this would just copy, e.g., /bin/seq onto /bin/seq # Create the /usr merge compatibility symlinks for dir in /bin /sbin; do run_command rm -rf $dir ---> where are the 'ln' and 'rm' taken from after this step ? If this fails here, you're also leaving the system in a broken state. run_command ln -s usr/bin $dir done