Re: merged-/usr-via-symlinks vs a-different-layout

[Sam Hartman]

First, I hear your frustration.
I suspect that our current path is going to lead to a fair bit of pain,
sufficient even that I expect us to make some changes after the release
of buster.  Swallowing that will be hard, but I tend to think we're
being somewhat narrow in which requirements are being considered by the
current plan, and we'll realize that we're not serving the needs of
significant user communities with our current approach.

> "Guillem" == Guillem Jover  writes:

Guillem> On Sun, 2019-02-24 at 09:23:14 -0500, Sam Hartman wrote:
>> > "Guillem" == Guillem Jover  writes:
Guillem> On Sun, 2019-02-24 at 08:27:20 +0100, Ansgar wrote:
>> >> Guillem Jover writes: > You are still conflating the concept
>> with >> the deployment. The > underlaying properties of merging
>> /usr is >> that the contents for > directories that are present
>> in both / >> and /usr get merged into > /usr.  >>· >> No, I'm
>> saying that you are proposing yet another different file >>
>> system layout.  Which is quite simple to see: the file system >>
>> would differ.
>> >>
>> >> You just claim it follows similar "ideas" in some way.
>> 
Guillem> Again, no, the important part is that the contents get
Guillem> *moved* properly and *automatically* within the .deb
Guillem> packags,
>> 
>> This is the important part to *you*.

Guillem> This (and the other part you omitted) was an attempt at
Guillem> describing how this alternative deployment method follows
Guillem> the same underlying concept.  And not about how it is
Guillem> important to me.
Guillem> Sure (even though that was unrelated to my point), and I've
Guillem> acknowledged that in previous threads. I do understand some
Guillem> people might value using a deployment method that gives
Guillem> immediate results so they can use the underlying properties
Guillem> of the merged-/usr right away, and providing a readily
Guillem> packaged solution for that seemed acceptable (even though
Guillem> it complicates packaging and alternative deployments). I
Guillem> can also see the apparent appeal of using directory
Guillem> symlinks, as that avoids the forest of compat symlinks.

I don't have a formal definition of deployment vs concept.
However, I think they are similar to concepts of design+requirements vs
implementation.  (Implementation is similar to what you're calling
deployment, design/requirements are I think what you're similar to
calling concept).

That boundary between design and implementation depends on your
requirements.

In your model, both /bin/true and /usr/bin/true might exist and be
different inodes.  One would be a symlink, one would be presumably a
binary.

In the usrmerge model, that situation never exists.  Instead, if on a
non-usrmerge system, /usr/bin/true doesn't exist, but on a usrmerge
system, /bin/true and /usr/bin/true are the same filesystem entity.

What people are telling you is that they consider that a requirement,
not a detail of the implementation--in your words, they consider that
part of the concept.

I do agree that you are repeating yourself.  I might suggest focusing on
asking *why* people feel this is part of the concept rather than an
implementation detail.  I understand if you're too frustrated or
burned-out to do that, but I'd suggest that would be a good next step
for someone holding your position.


For myself, I consider that important because I believe that managing
the dependencies of the sort of transition you talk about would be more
difficult than the pain of the current approach.  Say that the next
version of bash goes and moves bash to /usr/bin/bash with a
compatibility symlink.  We now have a more complex problem with build
systems.  Today, if you build on a usrmerge system, you run into the
possibility that /usr/bin/bash will be discovered by a build system.
You can avoid this class of problems by not building on a usrmerge
system.  In your deployment, once that package enters the archive, I
need to either guarantee that my build system won't choose
/usr/bin/bash, or I need to depend on a new enough version of bash that
I'm guaranteed to get /usr/bin/bash.  Moreover, because the state space
of packages that are converted and not is larger, the reproducible
builds approach of testing to see if a build veries in this condition is
a lot harder to implement.


In addition, I find the eventual /bin -> /usr/bin symlink critical.
Interfaces like #!/bin/sh are long-standing conventions in POSIX.  We
should not deprecate those interfaces.  Maintaining a certain small set
of compatibility symlinks for common interfaces like /bin/sh is
unappealing.  Users will not know which /bin entries they can depend on.
Also, users familiar with AIX or other Linux distributions that do have
a /bin->/usr/bin symlink will assume that they can depend on /bin/foo
for anything in /usr/bin.  I think the value of making 

Re: merged-/usr-via-symlinks vs a-different-layout

[Guillem Jover]

On Sun, 2019-02-24 at 09:23:14 -0500, Sam Hartman wrote:
> > "Guillem" == Guillem Jover  writes:
> Guillem> On Sun, 2019-02-24 at 08:27:20 +0100, Ansgar wrote:
> >> Guillem Jover writes: > You are still conflating the concept with
> >> the deployment. The > underlaying properties of merging /usr is
> >> that the contents for > directories that are present in both /
> >> and /usr get merged into > /usr.
> >>·
> >> No, I'm saying that you are proposing yet another different file
> >> system layout.  Which is quite simple to see: the file system
> >> would differ.
> >>
> >> You just claim it follows similar "ideas" in some way.
> 
> Guillem> Again, no, the important part is that the contents get
> Guillem> *moved* properly and *automatically* within the .deb
> Guillem> packags,
> 
> This is the important part to *you*.

This (and the other part you omitted) was an attempt at describing how
this alternative deployment method follows the same underlying concept.
And not about how it is important to me.

> Other things are important to other people.

Sure (even though that was unrelated to my point), and I've acknowledged
that in previous threads. I do understand some people might value using
a deployment method that gives immediate results so they can use the
underlying properties of the merged-/usr right away, and providing a
readily packaged solution for that seemed acceptable (even though it
complicates packaging and alternative deployments). I can also see the
apparent appeal of using directory symlinks, as that avoids the forest
of compat symlinks.

> Instead of working to understand their requirements, you are saying
> things  like "you are still conflating the concept with the deployment."

Well, if we are trying to communicate, and there's a clear distinction
between concepts/ideas/designs and how to implement/deploy them, taking
issue with trying to avoid conflating them, seems to me like a path to
muddling the discussion. :/

> I ask you to please stop and to instead take the time to understand the
> people who disagree with you.

I do think I have this understanding. I've kept finding, though, extremely
frustrating, draining and exhausting to deal and discuss this issue,
because people pushing forward with the merged-usr-via-symlinks deployment
method have been doing that, even though:

  - we've had sustained technical opposition to it, with complete
disregard to it,
  - it's an irreversible process,
  - it's being forced into all new installations by default,

when not forcing it on new installations, still makes it possible for
people that value the properties of the merged-usr-via-symlinks
deployment method to still get it via the usrmerge package! :(


At this point, I feel I'm repeating myself, which also exhausts me, so
I don't think I have the energy right now to keep discussing this. :(

Thanks,
Guillem

Re: merged-/usr-via-symlinks vs a-different-layout

[Sam Hartman]

> "Guillem" == Guillem Jover  writes:

Guillem> On Sun, 2019-02-24 at 08:27:20 +0100, Ansgar wrote:
>> Guillem Jover writes: > You are still conflating the concept with
>> the deployment. The > underlaying properties of merging /usr is
>> that the contents for > directories that are present in both /
>> and /usr get merged into > /usr.
>> 
>> No, I'm saying that you are proposing yet another different file
>> system layout.  Which is quite simple to see: the file system
>> would differ.
>> 
>> You just claim it follows similar "ideas" in some way.

Guillem> Again, no, the important part is that the contents get
Guillem> *moved* properly and *automatically* within the .deb
Guillem> packags,

This is the important part to *you*.

Other things are important to other people.
Instead of working to understand their requirements, you are saying
things  like "you are still conflating the concept with the deployment."

I ask you to please stop and to instead take the time to understand the
people who disagree with you.

I've found it's a valuable exercise to write up the position of those I
disagree with and get to a point where they say that yes, I've
accurately represented their position.
Then I can talk about why I disagree with that approach.
I urge you to do something similar here.

From where I sit, other people in the discussion actually value ending
up with the symlink from /bin to /usr/bin and from /sbin to /usr/sbin.

I can demonstrate that those symlinks have different technical
properties than a system without those symlinks.
Instead of debating those tradeoffs, you're using language like
"botched the final part," which don't actually lead to building
understanding and actually having a technical debate.

I urge you to work to understand those who disagree with you.

Thanks for your consideration,

--Sam


signature.asc
Description: PGP signature

Re: merged-/usr-via-symlinks vs a-different-layout

[Guillem Jover]

On Sun, 2019-02-24 at 08:27:20 +0100, Ansgar wrote:
> Guillem Jover writes:
> > You are still conflating the concept with the deployment. The
> > underlaying properties of merging /usr is that the contents for
> > directories that are present in both / and /usr get merged into
> > /usr.
> 
> No, I'm saying that you are proposing yet another different file system
> layout.  Which is quite simple to see: the file system would differ.
> 
> You just claim it follows similar "ideas" in some way.

Again, no, the important part is that the contents get *moved*
properly and *automatically* within the .deb packags, the only
difference is that for a transition period we'd have most of those
files that currently live in / as symlinks to /usr. Most of those
(not all, obviously) would eventually disappear.

The main difference between the two deployments is how the move is
done, and how and which compat symlinks are setup.

> I can come up with other variations that move files to /usr too: have
> /bin a symlink to /usr/root/bin (some for other directories).  That
> would also be a different file system layout which shouldn't be called
> "merged-/usr" either.

Err, well sure, because that'd not be merged-/usr, obviously. The
contents would not end up being moved/merged into their counterparts
on /usr. This just looks like a straw man…

> (That would have no filesystem aliasing between /bin and /usr/bin;
> policy has special rules to allow replacing top-level directories with
> symlinks; but besides that it is different for no good reason so I don't
> think it would be a good idea.)

The rationale for the policy and dpkg support for moving directories
into other places and then symlinking them, was for space constrained
purposes (say move /lib to /srv/lib, or even /usr to /srv/usr). These
never create filesystem aliasing problems, because no one had before
tried to point those dpkg owned directories contents into another dpkg
owned directory and then symlinking the directories. This should be
obvious from the fact that before the merged-/usr-via-symlinks hack
got forced in, many packages needed to be changed (not fixed really)
to cope with the newly introduced aliasing problems.

> > See for example OpenSUSE which did this transition correctly:
> >
> >   
> 
> "The special comments #UsrMerge, #EndUsrMerge will help us to clean up
> the spec files once everything has been moved and we can create a
> general link from /bin to /usr/bin for example. "
> 
> So they want a /bin -> /usr/bin symlink (and the same for the other
> directories) just as in the proposal you call broken...
> 
> Maybe you have misunderstood what SuSE plans to do?

Well, they did a proper transition, and AFAIUI they have not yet
botched the final part.

> Having dpkg track a symlink /bin/${something} and a file
> /usr/bin/${something} will break should /bin be turned into a symlink
> happen.

Only if that symlink is aliasing an existing dpkg owned directory. Which
well, clearly should not be done. If the admin moves the directory
somwehere else (not owned by dpkg), that does *not* create aliasing
problems.

> So there is no nice way to finish the transition.  I'm not sure
> how that would work on RPM; maybe SuSE got stuck with that?  That would
> be another good reason to not follow that way...

If by "finishing the transition" you mean ending up deploying
merged-usr-via-symlinks, well then I guess you missed the entire point
of my mail. It would be pointless to do it by properly moving the
files in the .debs, creating compat symlinks to then botch it at the
end again with the directory symlinks, we might as well botch it from
the beginning. :/

The transition would be finished simply when no (non-required) compat
symlinks are present under the / directories. So we'd remain with just
a few things like /bin/sh and similar required for historical reasons.

> I wouldn't be surprised if that took about as long as the /usr/doc ->
> /usr/share/doc transition.  (If the change process for most simple
> changes like file locations takes 10+ years then I say the process is
> broken...)

As mentioned in my earlier mail, the main difference is that in this
case the move can be fully automated, w/o any initial maintainer
intervention whatsoever. The only thing that requires maintainer
internvation is the analysis and cleanup of the compat symlinks in
the .deb, but that's really not urgent at all in any case.

Regards,
Guillem

Re: merged-/usr-via-symlinks vs a-different-layout

[Ansgar]

Guillem Jover writes:
> On Tue, 2019-02-19 at 08:54:12 +0100, Ansgar wrote:
>> Guillem Jover writes:
>> > 3) Switching packages to the merged-/usr layout could have been
>> >accomplished automatically via debhelper for a coverage of around
>> >99% (?) of the archive. With something along the lines of:
>> >
>> >  ,---
>> >  D=debian/tmp
>> >  for d in /bin /sbin /lib; do
>> >for p in $(find $D/$d ! -type d); do
>> >  b="${p##$D/}"
>> >  m="$D/usr$b"
>> >  if [ ! -e "$m" ]; then
>> >mkdir -p "$(dirname $m)"
>> >mv "$p" "$m"
>> >ln -s "${m##$D}" "$p"
>> >  fi
>> >done
>> >  done
>> >  `---
>> >
>> >With the property that it would handle gracefully all cases were the
>> >maintainer has checked that no compat symlinks are required and has
>> >then progressively moved the pathname installation to their final
>> >destination under /usr.
>> 
>> That is not merged-/usr, but a different filesystem layout.
>> 
>> So, no, it is not possible to switch packages to merged-/usr this way.
>
> You are still conflating the concept with the deployment. The
> underlaying properties of merging /usr is that the contents for
> directories that are present in both / and /usr get merged into
> /usr.

No, I'm saying that you are proposing yet another different file system
layout.  Which is quite simple to see: the file system would differ.

You just claim it follows similar "ideas" in some way.

I can come up with other variations that move files to /usr too: have
/bin a symlink to /usr/root/bin (some for other directories).  That
would also be a different file system layout which shouldn't be called
"merged-/usr" either.

(That would have no filesystem aliasing between /bin and /usr/bin;
policy has special rules to allow replacing top-level directories with
symlinks; but besides that it is different for no good reason so I don't
think it would be a good idea.)

> See for example OpenSUSE which did this transition correctly:
>
>   

"The special comments #UsrMerge, #EndUsrMerge will help us to clean up
the spec files once everything has been moved and we can create a
general link from /bin to /usr/bin for example. "

So they want a /bin -> /usr/bin symlink (and the same for the other
directories) just as in the proposal you call broken...

Maybe you have misunderstood what SuSE plans to do?

Having dpkg track a symlink /bin/${something} and a file
/usr/bin/${something} will break should /bin be turned into a symlink
happen.  So there is no nice way to finish the transition.  I'm not sure
how that would work on RPM; maybe SuSE got stuck with that?  That would
be another good reason to not follow that way...

I wouldn't be surprised if that took about as long as the /usr/doc ->
/usr/share/doc transition.  (If the change process for most simple
changes like file locations takes 10+ years then I say the process is
broken...)

Ansgar

Re: merged-/usr-via-symlinks vs a-different-layout

[Guillem Jover]

On Tue, 2019-02-19 at 08:54:12 +0100, Ansgar wrote:
> Guillem Jover writes:
> > 3) Switching packages to the merged-/usr layout could have been
> >accomplished automatically via debhelper for a coverage of around
> >99% (?) of the archive. With something along the lines of:
> >
> >  ,---
> >  D=debian/tmp
> >  for d in /bin /sbin /lib; do
> >for p in $(find $D/$d ! -type d); do
> >  b="${p##$D/}"
> >  m="$D/usr$b"
> >  if [ ! -e "$m" ]; then
> >mkdir -p "$(dirname $m)"
> >mv "$p" "$m"
> >ln -s "${m##$D}" "$p"
> >  fi
> >done
> >  done
> >  `---
> >
> >With the property that it would handle gracefully all cases were the
> >maintainer has checked that no compat symlinks are required and has
> >then progressively moved the pathname installation to their final
> >destination under /usr.
> 
> That is not merged-/usr, but a different filesystem layout.
> 
> So, no, it is not possible to switch packages to merged-/usr this way.

You are still conflating the concept with the deployment. The
underlaying properties of merging /usr is that the contents for
directories that are present in both / and /usr get merged into
/usr.

See for example OpenSUSE which did this transition correctly:

  

> > 4) Due to having to support the broken merged-/usr-via-symlinks
> >deployment, when we want to move the contents of the binary
> >packages to the merged-/usr layout, we require now to include tons
> >of logic in (probably new) maintainer scripts, when we have been
> >trying to remove them altogether. :( With even more files untracked
> >by dpkg itself, bypassing the packaging system even more, when the
> >compat symlinks could have been shipped in the binary packages.
> 
> As far as I know maintainer scripts are only required for moving files
> from / to /usr when (a) a compat symlink is required, and (b) only when
> both merged-/usr and non-merged-/usr is supported.

I guess you are implying that we should start forcing and requiring
the broken deployment into all Debian systems… if that ever happens
I'm personally going to use any local technical means to avoid such
breakage to damage any of my systems…

Regards,
Guillem

Re: merged-/usr-via-symlinks vs a-different-layout

[Ansgar]

Guillem Jover writes:
> 3) Switching packages to the merged-/usr layout could have been
>accomplished automatically via debhelper for a coverage of around
>99% (?) of the archive. With something along the lines of:
>
>  ,---
>  D=debian/tmp
>  for d in /bin /sbin /lib; do
>for p in $(find $D/$d ! -type d); do
>  b="${p##$D/}"
>  m="$D/usr$b"
>  if [ ! -e "$m" ]; then
>mkdir -p "$(dirname $m)"
>mv "$p" "$m"
>ln -s "${m##$D}" "$p"
>  fi
>done
>  done
>  `---
>
>With the property that it would handle gracefully all cases were the
>maintainer has checked that no compat symlinks are required and has
>then progressively moved the pathname installation to their final
>destination under /usr.

That is not merged-/usr, but a different filesystem layout.

So, no, it is not possible to switch packages to merged-/usr this way.

> 4) Due to having to support the broken merged-/usr-via-symlinks
>deployment, when we want to move the contents of the binary
>packages to the merged-/usr layout, we require now to include tons
>of logic in (probably new) maintainer scripts, when we have been
>trying to remove them altogether. :( With even more files untracked
>by dpkg itself, bypassing the packaging system even more, when the
>compat symlinks could have been shipped in the binary packages.

As far as I know maintainer scripts are only required for moving files
from / to /usr when (a) a compat symlink is required, and (b) only when
both merged-/usr and non-merged-/usr is supported.

Ansgar