[gentoo-dev] Re: usr merge

[Duncan]

Richard Yao posted on Wed, 06 Apr 2016 00:15:58 -0400 as excerpted:


>> On Apr 4, 2016, at 9:19 PM, William Hubbs  wrote:
>> 
>> All,
>> 
>> I thought that since the usr merge is coming up again, and since I lost
>> track of the message where it was brought up, I would open a new thread
>> to discuss it.
>> 
>> When it came up before, some were saying that the /usr merge violates
>> the fhs. I don't remember the specifics of what the claim was at the
>> time, (I'm sure someone will point it out if it is still a concern).
> 
> Here are the violations:
> 
> http://refspecs.linuxfoundation.org/FHS_3.0/
fhs-3.0.html#binEssentialUserCommandBinaries
> 
> http://refspecs.linuxfoundation.org/FHS_3.0/
fhs-3.0.html#sbinSystemBinaries
> 
> http://refspecs.linuxfoundation.org/FHS_3.0/
fhs-3.0.html#libEssentialSharedLibrariesAndKern

(Those links are wrapped and I'm not bothering to jump thru the hoops to 
unwrap them, since readers can either unwrap them manually or refer to 
the parent post I'm quoting for the unwrapped versions.)

If those are the "violations", then putting everything in /usr and making 
the /bin and /sbin locations symlinks isn't going to be a problem, since 
/bin and /sbin are specifically allowed to contain symlinks to the 
executables, instead of the executables themselves, and if the dirs 
themselves are symlinks to the locations in /usr with the files, that 
fulfills that requirement.

And the requirement for /lib is rather vague, saying only that it 
contains the libs linked by the executables in /bin and /sbin.  So once /
bin and /sbin are symlinks to the dirs with the executables, /lib (or the 
arch-specific alternative libdirs) can be a symlink as well.

Tho I must say doing the reverse, making either /usr itself or /usr/bin 
and /usr/sbin symlinks to the root dirs, as I did here, actually makes 
more sense and bends the rules less.

Basically, what the FHS says, at least in the 3.0 version you linked, is 
that the executables must be reachable via whatever specific path, but 
using symlinks to do it is fine.

Which means the merge is allowed, as long as symlinks allow the 
executables to be reached by their specifically defined paths.  And I'm 
not aware of anyone seriously proposing that said symlinks be omitted, 
so...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] usr merge

[Richard Yao]

> On Apr 4, 2016, at 9:19 PM, William Hubbs  wrote:
> 
> All,
> 
> I thought that since the usr merge is coming up again, and since I lost
> track of the message where it was brought up, I would open a
> new thread to discuss it.
> 
> When it came up before, some were saying that the /usr merge violates
> the fhs. I don't remember the specifics of what the claim was at the
> time, (I'm sure someone will point it out if it is still a concern).

Here are the violations:

http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#binEssentialUserCommandBinaries

http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#sbinSystemBinaries

http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#libEssentialSharedLibrariesAndKern

> 
> I don't think creating usr merged stages would be that difficult. I
> think it would just be a matter of creating a new version of baselayout
> that puts these symlinks in place:
> 
> /bin->usr/bin
> /lib->usr/lib
> /lib32->usr/lib32
> /lib64->usr/lib64
> /sbin->usr/bin
> /usr/sbin->bin
> 
> Once that is in place in a new baselayout, I think portage's colission
> detection would be able to catch files that had the same names and were
> originally in different paths when building the new stages.

We will have users whose system configurations rely on the FHS complain about 
us breaking boot if we force this.

> I put some thought also in how to nigrate live systems, and I'm not sure
> what the best way to do that is. I wrote a script, which would do it in
> theory, but I haven't tested because I only have one system and if
> it breaks it the only way back would be to reinstall.
> 
> The script is attached.
> 
> 
> Thoughts on any of this?


This was invented in Solaris and copied by RHEL. The upgrade path for the /usr 
merge on those systems is a complete reinstall. Upgrading from RHEL6 to RHEL7 
this Solaris 10 to Solaris 11 is not supported. The reason being that there are 
ways of configuring the system boot process with the original layout that break 
if you try using scripts to migrate to the new one. A USE flag for the /usr 
merge that is off by default would allow us to have both worlds without putting 
any systems at risk.

This has been an almost annual debate. I do not have much incentive to keep up 
with it. The reason I ever bothered to explain why this is a bad idea for 
Gentoo was that I was concerned for our user base. My systems would not be 
negatively affected and arguing against changes that originated in Solaris is 
awkward for me.

If others are not willing to be advocates for those users that would only make 
themselves known after an a fundamental change has been made and people are 
determined to go ahead with this, I suggest having and testing a plan for 
backing out the change should the backlash from users after systems break be 
more than people can stomach. This is not the sort of change we should make 
without an "exit strategy".

> William
> 
> 

[gentoo-dev] Re: usr merge

[Jonathan Callen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 04/05/2016 12:53 PM, Alexis Ballier wrote:
> On Tuesday, April 5, 2016 2:26:53 PM CEST, Duncan wrote:
>> 
>> As I said in the other thread, I'm running merged /usr and
>> bin/sbin here, except that I merged them the other way, with /usr
>> -> . so everything in /usr is now in /.
>> 
>> Portage has long "just worked" in that regard, tho I've no idea 
>> whether the other PMs do.  Portage has enough intelligence to
>> avoid replacing a file with a symlink pointing to it (and thus to
>> itself once the replacement is done), regardless of which way the
>> directory symlinks point.
>> 
>> As such, coreutils "just works".  If the two would end up in the
>> same canonical location, the file wins and the symlink isn't
>> installed.
> 
> What about the unlikely case with two files ?
> 

Having actually run this way myself, I did find one case that I
haven't filed a bug for yet: the plymouth ebuild tries to install
symlinks in /sbin pointing at /usr/sbin, and portage chose to install
the symlinks instead of the real files, for whatever reason
(apparently because the $ED/sbin directory is created after the
$ED/usr directory).  Because of this, it might be best to ensure that
packages that do install in both places are modified not to do so
under such a configuration.

- -- 
Jonathan Callen
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCgAGBQJXBFMbAAoJEEIQbvYRB3mg2e4P/2lPBxpyjY311LP7gN2Nndn4
Dd4EtFbh8tQWoedPJQgr2CIeVgpPFA7l/stuvcoZAqLVDuFnn4ZmMWSIQOgHmgPp
+mIiCDPuLMjhqw/yINlTGGVVhffHFG4PrHcd2MwP6Gm9ME0NH8+Z0cgAznHsHQ5c
lgNdfXDsgBdrSrKu5/JTw7jDOv1A1TwIACJoLpEYZTlVCBClp6J01kqH1oyEzPf8
FO6fqAvFJXCq1um6/+ve8LOpS0OLBpg0dh5kcdkFgV1430FqNwUczMINhav5J0mp
qTAIZTO4OSLxyswOUiKoxROl4xrQ1ByYi1ZF7g24oh7M1fmkreNClrhJ1kA3M6ff
OJ3LJ6m350LEIVzAED66pnKOTNDOLJSaz6MsPk8CHzuJ2RCMatKjBA3Lb0tkkepp
5LOCBXbnVfSPRI+TQM91cHXVnh87T1zZSeGT8qOCfNoF7rFWNSlpIRnxMeeFlv2n
0kXfJo9YeiUAA9BYXBryMIsWr4StM4I9oq0ITc7h9WmB/WKW6zJhl7WHd7SgiePW
Lb2fHJtz0R8dUIc53Yxuls1Cbt8AUAFYmN9Ve615cVLs3+jO8HWmwiuFfiYH71k1
JaS51cgBjPBnQuiET0iNxu/gjIekwIjoNptn/cCr9IZ4jnZ9L13ai6Wug49vUwwK
bed4Tt3nl8GSbRtlV+rk
=PHpB
-END PGP SIGNATURE-

[gentoo-dev] Last-rites: dev-libs/ptypes

[David Seifert]

# David Seifert  (05 Apr 2016)
# No upstream release since 2007, no revdeps in tree,
# version in tree outdated, bug #454702.
# Masking for removal in 30 days
dev-libs/ptypes

Re: [gentoo-dev] usr merge

[Alexis Ballier]

On Tuesday, April 5, 2016 2:26:53 PM CEST, Duncan wrote:

Alexis Ballier posted on Tue, 05 Apr 2016 12:10:51 +0200 as excerpted:


On Tuesday, April 5, 2016 3:19:59 AM CEST, William Hubbs wrote:
[...] ...


As I said in the other thread, I'm running merged /usr and bin/sbin here, 
except that I merged them the other way, with /usr -> . so everything in 
/usr is now in /.


Portage has long "just worked" in that regard, tho I've no idea whether 
the other PMs do.  Portage has enough intelligence to avoid replacing a 
file with a symlink pointing to it (and thus to itself once the 
replacement is done), regardless of which way the directory symlinks 
point.


As such, coreutils "just works".  If the two would end up in the same 
canonical location, the file wins and the symlink isn't installed.


What about the unlikely case with two files ?


There are a few individual package bugs, including one open right now 
where the gcc-config ebuild does an unconditional rm -f of any old 
versions found in its old sbin location, even when it removes the 
executable it just installed into the bin location, because they're the 
same canonical location.  (Bug number for that and other bugs in the 
reply on the other thread.)  And cmake can get mixed up in some instances 
so a few packages (baloo) have problems with some cmake versions.


But the bugs aren't with portage, they're with the ebuild or the upstream 
sources, and the number of them I've run into in the two years plus I've 
been running merged can fit on one hand.  Certainly, they're few enough 
to deal with on a case-by-case basis.


Yeah, these cases need to be handled on a case by case basis, there's no 
other choice anyway :)


If we want to move on on this, we should definitely track these properly.

[...]
1) Unless one is sure of the actual install path used and uses it, equery 
belongs and I assume q and similar tools with the same query, need the 
bare file name, not the full path, because you might use the wrong one.


For instance, /bin/grep, /sbin/grep, /usr/sbin/grep, and /usr/bin/grep, 
are all the same file due to directory level symlinks.  However, if you 
try equery belongs with all four paths, only one of them will return the 
hit you're looking for.


Easily solved by simply doing equery belongs grep (no path), and letting 
equery supply the installed path in its results.  That's actually how I 
find out which path a file was actually installed to, these days, as 
well. =:^)



no real issue here, and anyway, since it parses portage tree & profiles, 
support for guessing usr-merge can be added to restore old behavior


2) revdep-rebuild will, in its default configuration, end up processing 
files using all paths.  So grep, to use the same example as above, will 
be processed four times, one each for /bin and /sbin, /usr/bin and /usr/

sbin.

While it's possible to reconfigure revdep-rebuild to ignore three of 
those paths and only check one of them (and similarly, ignore one of 
/lib64 and /usr/lib64, etc), doing so will result in revdep-rebuild 
complaining about unowned broken binaries if they're installed using a 
different path than the one it processed.


That's not a big problem, because equery belongs  (without the 
path) will tell you what owns it as well as the installed path it used, 
and then that package can be remerged manually, if needed.



kind of defeats the point of revdep-rebuild though :)

[...]

# Create the /usr merge compatibility symlinks for dir in /bin /sbin; do
run_command rm -rf $dir

---> where are the 'ln' and 'rm' taken from after this step ?
If this fails here, you're also leaving the system in a broken state.

run_command ln -s usr/bin $dir done


In my case I was using the mc binary, which continued to run after the 
transfer, so it wasn't an issue.


But using the individual ln and rm binaries, while they'll still be on 
the path, you may need to run hash -r in the shell so it forgets the old 
location and checks the path again.


Similar thing for the libs, since the lib cache will be screwed after the 
move, until the symlink has been created so the old paths work again, at 
least.  In my case I was using the mc binary which continued to run and 
thus could be used to create the symlink, but for one-shot executables 
like ln, that could be an issue.


One way around the problem would be to create a few static-linked 
executables for the purpose, and ship them in a tarball that's unpacked 
to an unchanging tmp location for the run, so they can be called from 
there to finish up regardless of whether the dynamically linked normal 
executables can still be invoked.


Smart use of the shell's builtin read, echo and redirection could 
probably do some of it too, but can ln be emulated using shell builtins?





I'd rather use a binary that ensures everything that needs to be loaded is 
loaded at the beginning and even better if it can enusre system consistency 
and can do rollbacks in case of failure.


A 

[gentoo-dev] Re: usr merge

[Duncan]

Alexis Ballier posted on Tue, 05 Apr 2016 12:10:51 +0200 as excerpted:

> On Tuesday, April 5, 2016 3:19:59 AM CEST, William Hubbs wrote:
> [...]
>> I don't think creating usr merged stages would be that difficult. I
>> think it would just be a matter of creating a new version of baselayout
>> that puts these symlinks in place:
>>
>> /bin->usr/bin /lib->usr/lib /lib32->usr/lib32 /lib64->usr/lib64
> 
> (OT: maybe it'd be a good oportunity to kill SYMLINK_LIB too :p)
> 
>> /sbin->usr/bin /usr/sbin->bin
>>
>> Once that is in place in a new baselayout, I think portage's colission
>> detection would be able to catch files that had the same names and were
>> originally in different paths when building the new stages.
> 
> 
> I think that prior to that we have to ensure that packages with intra
> collisions are not merged: What happens with current coreutils ebuilds
> that install, e.g., /bin/seq and a /usr/bin/seq symlink to it ?
> I haven't looked at the actual code, thus I can only assume there are no
> guarantees, which is definitely bad.

As I said in the other thread, I'm running merged /usr and bin/sbin here, 
except that I merged them the other way, with /usr -> . so everything in 
/usr is now in /.

Portage has long "just worked" in that regard, tho I've no idea whether 
the other PMs do.  Portage has enough intelligence to avoid replacing a 
file with a symlink pointing to it (and thus to itself once the 
replacement is done), regardless of which way the directory symlinks 
point.

As such, coreutils "just works".  If the two would end up in the same 
canonical location, the file wins and the symlink isn't installed.

There are a few individual package bugs, including one open right now 
where the gcc-config ebuild does an unconditional rm -f of any old 
versions found in its old sbin location, even when it removes the 
executable it just installed into the bin location, because they're the 
same canonical location.  (Bug number for that and other bugs in the 
reply on the other thread.)  And cmake can get mixed up in some instances 
so a few packages (baloo) have problems with some cmake versions.

But the bugs aren't with portage, they're with the ebuild or the upstream 
sources, and the number of them I've run into in the two years plus I've 
been running merged can fit on one hand.  Certainly, they're few enough 
to deal with on a case-by-case basis.

>> I put some thought also in how to nigrate live systems, and I'm not
>> sure what the best way to do that is. I wrote a script, which would do
>> it in theory, but I haven't tested because I only have one system and
>> if it breaks it the only way back would be to reinstall.
> 
> 
> Does it behave properly wrt portage's way of tracking of package files?
> I remember that modifying files owned by portage used to cause issues.

What I did for my migration was simply move everything from /usr to / and 
create the /usr -> . symlink.  I did that with mc, and kept it running 
just in case I ended up not being able to start something, until I had 
the symlink in place and had tested starting various things, including X/
KDE, so I knew it was working.

Similarly for the sbin -> bin moves and symlinks.

The moves worked fine, and with the directory symlinks replacing the old 
dirs, everything else, including portage on updates after that, worked 
just fine.


There are a couple things that behave slightly differently regarding 
packages, that one needs to be aware of, but in general it just works.  
Those couple things are:

1) Unless one is sure of the actual install path used and uses it, equery 
belongs and I assume q and similar tools with the same query, need the 
bare file name, not the full path, because you might use the wrong one.

For instance, /bin/grep, /sbin/grep, /usr/sbin/grep, and /usr/bin/grep, 
are all the same file due to directory level symlinks.  However, if you 
try equery belongs with all four paths, only one of them will return the 
hit you're looking for.

Easily solved by simply doing equery belongs grep (no path), and letting 
equery supply the installed path in its results.  That's actually how I 
find out which path a file was actually installed to, these days, as 
well. =:^)

2) revdep-rebuild will, in its default configuration, end up processing 
files using all paths.  So grep, to use the same example as above, will 
be processed four times, one each for /bin and /sbin, /usr/bin and /usr/
sbin.

While it's possible to reconfigure revdep-rebuild to ignore three of 
those paths and only check one of them (and similarly, ignore one of 
/lib64 and /usr/lib64, etc), doing so will result in revdep-rebuild 
complaining about unowned broken binaries if they're installed using a 
different path than the one it processed.

That's not a big problem, because equery belongs  (without the 
path) will tell you what owns it as well as the installed path it used, 
and then that package can be remerged manually, if needed.

So with 

Re: [gentoo-dev] usr merge

[Alexis Ballier]

On Tuesday, April 5, 2016 3:19:59 AM CEST, William Hubbs wrote:
[...]

I don't think creating usr merged stages would be that difficult. I
think it would just be a matter of creating a new version of baselayout
that puts these symlinks in place:

/bin->usr/bin
/lib->usr/lib
/lib32->usr/lib32
/lib64->usr/lib64


(OT: maybe it'd be a good oportunity to kill SYMLINK_LIB too :p)


/sbin->usr/bin
/usr/sbin->bin

Once that is in place in a new baselayout, I think portage's colission
detection would be able to catch files that had the same names and were
originally in different paths when building the new stages.



I think that prior to that we have to ensure that packages with intra 
collisions are not merged: What happens with current coreutils ebuilds that 
install, e.g., /bin/seq and a /usr/bin/seq symlink to it ?
I haven't looked at the actual code, thus I can only assume there are no 
guarantees, which is definitely bad.




I put some thought also in how to nigrate live systems, and I'm not sure
what the best way to do that is. I wrote a script, which would do it in
theory, but I haven't tested because I only have one system and if
it breaks it the only way back would be to reinstall.



Does it behave properly wrt portage's way of tracking of package files? I 
remember that modifying files owned by portage used to cause issues.


What should baselayout ebuild do on systems that have not run that script ?

Also, I think your script may not work:

# copy binaries
for dir in /bin /sbin /usr/sbin; do
run_command cp -a $dir/* /usr/bin
done

---> Here it is important to ensure nothing /usr/bin/ is a symlink to /bin, 
otherwise this would just copy, e.g., /bin/seq onto /bin/seq



# Create the /usr merge compatibility symlinks
for dir in /bin /sbin; do
run_command rm -rf $dir

---> where are the 'ln' and 'rm' taken from after this step ?
If this fails here, you're also leaving the system in a broken state.

run_command ln -s usr/bin $dir
done