Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable
On Sat, 3 Aug 2019 00:07:13 +0100 James Le Cuirot wrote: > Any better? I think I would be personally aided by a description of what sort of environment is expecting the various values, eg: I know if I build for a target that I'll eventually have to "chroot" to get into, paths that get "made concrete" in the final image need to be from the perspective of inside that chroot, or they won't work, while paths that pertain to the build process need to be relative to the container of said chroot, or they won't know the absolute location to look at for its dependencies. All this talk of "prefix" with differing meanings just confuses me. :/ pgpDbvvHJ7VQX.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable
On Thu, 1 Aug 2019 16:27:22 +0200
Alexis Ballier wrote:
>
> > What I am trying to say (somewhat unsuccessfully!) is that the value
> > of (E)SYSROOT only changes how the package is built, not what the
> > resulting package looks like. It's where all the headers and libraries
> > are sourced from at build time, which is irrelevant at runtime.
>
> err, SYSROOT does change what the resulting package looks like: it
> defines ABI (headers and needed entries for example).
Well yes, in that sense it changes the resulting package but not in a
way that is significant here. Say you build a binpkg against SYSROOT A
that has libfoo-X and the same package against SYSROOT B that has
libfoo-Y. The resulting binpkgs may differ but it would be just the
same if you upgraded libfoo-X to libfoo-Y in SYSROOT A and built the
binpkg again. Your build system package versions aren't frozen in time
forever, right?
This doesn't mean that building your binpkgs against different SYSROOTs
is a great idea but it's not like this is going to happen by itself.
Despite ESYSROOT being influenced by ROOT, changing ROOT at install
time obviously isn't going to send your binpkgs into a time machine to
be rebuilt with a different ESYSROOT value.
Even at build time, I don't expect ESYSROOT to change under your feet.
This is simply about automatically giving it a meaningful value based
on other variables that are also not going to change at build time.
> > So why does ROOT affect it? Normally you install the packages for
> > BDEPEND, DEPEND, and RDEPEND to the same location. If BDEPEND and
> > RDEPEND are installed to different locations (ROOT!=/) then DEPEND
> > will almost always be installed to one of the other two. If either of
> > those two locations is prefixed then we need the prefix for DEPEND's
> > location to match, otherwise it wouldn't actually be the same
> > location. Using ROOT allows us to figure this out automatically in a
> > way that covers all sensible use cases and avoids accidentally
> > falling into an unsupported case.
>
>
> So, now let's simplify this a bit and forget about prefix and crossdev
> for a moment. Say I am building an atom chroot (for nfs root) on an
> haswell desktop. I set ROOT=SYSROOT=/atomchroot. My desktop has CFLAGS
> for haswell, and I have atom CFLAGS in
> /atomchroot/etc/portage/make.conf. When I build something and portage
> installs a BDEPEND to /, I want it to use my / configuration, and when
> it is in /atomchroot I want it to use the configuration from there.
> emerge has command line options to do that but I am not sure if this is
> properly specified in PMS.
>
>
> Now, say I am doing the same on a prefix haswell desktop. With your
> algorithm, we have SYSROOT==ROOT so ESYSROOT is EPREFIX/SYSROOT.
> However, what I want is a normal chroot with EPREFIX=/ here, so when
> should one use ESYSROOT in an ebuild in that case ? where does this
> EPREFIX comes from by the way ?
>
>
> To me, this looks more of a general problem of defining where the
> configuration is taken from, so that we can set EPREFIX independently
> if it is SYSROOT or BROOT.
First off, you said that ESYSROOT would be EPREFIX/SYSROOT when it
would actually be SYSROOT/EPREFIX. I don't know whether that was a typo
or genuine confusion. I'll assume the former.
I think I see why you're having trouble following this though. EPREFIX
is the prefix of the system you're building for and eventually
installing to. It has nothing to do with the system you're building on
or even the system you're building against.
The prefix for the system you're building on is BROOT. I initially
suggested the name BPREFIX but it was argued that it should be a *ROOT
variable because being rooted at / makes it an absolute path. Before
EAPI 7, we didn't have a proper variable for this so, for lack of
anything better, some people used EPREFIX. This was fine until you hit
the cross-prefix case, which is what you have described above. The true
value has also been available as PORTAGE_OVERRIDE_EPREFIX but, as the
name suggests, this is an internal Portage variable.
So in your example, you want EPREFIX=/ (or more accurately, blank) and
that's fine so you would define it as such. You would also define
SYSROOT=/atomchroot. As per PMS, ESYSROOT=${SYSROOT}/${EPREFIX} so the
actual calculated value of ESYSROOT would also be /atomchroot.
You're building on your prefixed Haswell system so BROOT would
automatically be set to /haswellprefix or something. You've already
defined EPREFIX=/ so where does the BROOT value actually come from?
It's hard coded into your prefixed Portage installation, in const.py
to be precise. The variable in that file is confusingly named EPREFIX
but that's simply because it predates BROOT. You can override this
value by defining the environment variable PORTAGE_OVERRIDE_EPREFIX but
I can't think of any legitimate reason to do that.
I should add that your example is not a case we readily support.
Although we've broken down many of
Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Wed, 31 Jul 2019 21:40:19 +0100 James Le Cuirot wrote: > On Wed, 31 Jul 2019 15:51:58 +0200 > Alexis Ballier wrote: > > > On Tue, 30 Jul 2019 23:26:27 +0100 > > James Le Cuirot wrote: > > > > > > Admittedly without a full understanding of the problem, but this > > > > looks wrong to me: SYSROOT, EPREFIX and BROOT are only relevant > > > > in build phases (src_*); (EPREFIX is a little special here but > > > > mostly for convenience). ROOT is only relevant in pkg_* phases. > > > > I don't see how this can work. Say I build a binpkg with ROOT=/ > > > > then use that binpkg with ROOT=/somewhere, you can't go back > > > > and change SYSROOT. > > > > > > ROOT is used to determine ESYSROOT, not the other way around. As > > > you say, (E)SYSROOT is only relevant in src phases so it doesn't > > > matter if ROOT has changed when installing a binpkg. > > > > I am missing something here: You are making ESYSROOT depend on the > > value of ROOT, so how can it not matter ? > > What I am trying to say (somewhat unsuccessfully!) is that the value > of (E)SYSROOT only changes how the package is built, not what the > resulting package looks like. It's where all the headers and libraries > are sourced from at build time, which is irrelevant at runtime. err, SYSROOT does change what the resulting package looks like: it defines ABI (headers and needed entries for example). > So why does ROOT affect it? Normally you install the packages for > BDEPEND, DEPEND, and RDEPEND to the same location. If BDEPEND and > RDEPEND are installed to different locations (ROOT!=/) then DEPEND > will almost always be installed to one of the other two. If either of > those two locations is prefixed then we need the prefix for DEPEND's > location to match, otherwise it wouldn't actually be the same > location. Using ROOT allows us to figure this out automatically in a > way that covers all sensible use cases and avoids accidentally > falling into an unsupported case. So, now let's simplify this a bit and forget about prefix and crossdev for a moment. Say I am building an atom chroot (for nfs root) on an haswell desktop. I set ROOT=SYSROOT=/atomchroot. My desktop has CFLAGS for haswell, and I have atom CFLAGS in /atomchroot/etc/portage/make.conf. When I build something and portage installs a BDEPEND to /, I want it to use my / configuration, and when it is in /atomchroot I want it to use the configuration from there. emerge has command line options to do that but I am not sure if this is properly specified in PMS. Now, say I am doing the same on a prefix haswell desktop. With your algorithm, we have SYSROOT==ROOT so ESYSROOT is EPREFIX/SYSROOT. However, what I want is a normal chroot with EPREFIX=/ here, so when should one use ESYSROOT in an ebuild in that case ? where does this EPREFIX comes from by the way ? To me, this looks more of a general problem of defining where the configuration is taken from, so that we can set EPREFIX independently if it is SYSROOT or BROOT. > I hope that's clearer. Sorry :-( -BEGIN PGP SIGNATURE- iHUEAREIAB0WIQSpOxaxaZikKNVNlsYOJUi7xgflrgUCXUL2ygAKCRAOJUi7xgfl rkwRAP9LDImZBCYxsWKMjT/ckCeK0hOLtctdpZuBwzjv5RIuiAD/cTUj7P7h9rBd gYaEEI+pqdN25ZNbjao/8w/j7SsXUMo= =WYef -END PGP SIGNATURE-
Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable
On Wed, 31 Jul 2019 21:40:19 +0100 James Le Cuirot wrote: > So why does ROOT affect it? Normally you install the packages for > BDEPEND, DEPEND, and RDEPEND to the same location. If BDEPEND and > RDEPEND are installed to different locations (ROOT!=/) then DEPEND will > almost always be installed to one of the other two. If either of those > two locations is prefixed then we need the prefix for DEPEND's location > to match, otherwise it wouldn't actually be the same location. Using > ROOT allows us to figure this out automatically in a way that covers > all sensible use cases and avoids accidentally falling into an > unsupported case. Just to clarify this in the binpkg case, here you would normally install BDEPEND, DEPEND, and usually RDEPEND to / on your build machine. On your other machines, you usually only care about RDEPEND. If you care about BDEPEND and DEPEND too then you'll need to use the same prefix everywhere but that's nothing new. You'd be pretty crazy to use different prefixes in this kind of environment anyway. -- James Le Cuirot (chewi) Gentoo Linux Developer pgpwVGu6zJELp.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable
On Wed, 31 Jul 2019 15:51:58 +0200 Alexis Ballier wrote: > On Tue, 30 Jul 2019 23:26:27 +0100 > James Le Cuirot wrote: > > > > Admittedly without a full understanding of the problem, but this > > > looks wrong to me: SYSROOT, EPREFIX and BROOT are only relevant in > > > build phases (src_*); (EPREFIX is a little special here but mostly > > > for convenience). ROOT is only relevant in pkg_* phases. I don't > > > see how this can work. Say I build a binpkg with ROOT=/ then use > > > that binpkg with ROOT=/somewhere, you can't go back and change > > > SYSROOT. > > > > ROOT is used to determine ESYSROOT, not the other way around. As you > > say, (E)SYSROOT is only relevant in src phases so it doesn't matter if > > ROOT has changed when installing a binpkg. > > I am missing something here: You are making ESYSROOT depend on the > value of ROOT, so how can it not matter ? What I am trying to say (somewhat unsuccessfully!) is that the value of (E)SYSROOT only changes how the package is built, not what the resulting package looks like. It's where all the headers and libraries are sourced from at build time, which is irrelevant at runtime. So why does ROOT affect it? Normally you install the packages for BDEPEND, DEPEND, and RDEPEND to the same location. If BDEPEND and RDEPEND are installed to different locations (ROOT!=/) then DEPEND will almost always be installed to one of the other two. If either of those two locations is prefixed then we need the prefix for DEPEND's location to match, otherwise it wouldn't actually be the same location. Using ROOT allows us to figure this out automatically in a way that covers all sensible use cases and avoids accidentally falling into an unsupported case. I hope that's clearer. -- James Le Cuirot (chewi) Gentoo Linux Developer pgp7vt5Id1Dzh.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Tue, 30 Jul 2019 23:26:27 +0100 James Le Cuirot wrote: > > Admittedly without a full understanding of the problem, but this > > looks wrong to me: SYSROOT, EPREFIX and BROOT are only relevant in > > build phases (src_*); (EPREFIX is a little special here but mostly > > for convenience). ROOT is only relevant in pkg_* phases. I don't > > see how this can work. Say I build a binpkg with ROOT=/ then use > > that binpkg with ROOT=/somewhere, you can't go back and change > > SYSROOT. > > ROOT is used to determine ESYSROOT, not the other way around. As you > say, (E)SYSROOT is only relevant in src phases so it doesn't matter if > ROOT has changed when installing a binpkg. I am missing something here: You are making ESYSROOT depend on the value of ROOT, so how can it not matter ? > I take your point that setting a src phase variable based on a pkg > phase variable seems odd but we're only using ROOT to determine the > applicable prefix. We're not taking the actual value of ROOT. When > Portage works all this out, it has access to all the necessary > variables. It only filters the variables based on the phase function > later on. The value does not really matter, the dependency of a variable used in src_* from a variable that can change when installing a binpkg is what worries me here. Alexis. -BEGIN PGP SIGNATURE- iHUEAREIAB0WIQSpOxaxaZikKNVNlsYOJUi7xgflrgUCXUGc/gAKCRAOJUi7xgfl rrlNAP9AEX0enc5Tzh++N6I+P4ZKp0hBdljlYBteYTuEipZLEAD/a3fT/pgOR3HJ LyGvu98wHn6sCldtBMjoV/RgrxfaKO8= =L9Hp -END PGP SIGNATURE-
Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable
On Mon, 29 Jul 2019 14:05:10 +0200 Alexis Ballier wrote: > There seems to be unneeded whitespace only changes here that make the > diff less readable Sorry about that. I've only changed one cell in that table. > Admittedly without a full understanding of the problem, but this looks > wrong to me: SYSROOT, EPREFIX and BROOT are only relevant in build > phases (src_*); (EPREFIX is a little special here but mostly for > convenience). ROOT is only relevant in pkg_* phases. I don't see how > this can work. Say I build a binpkg with ROOT=/ then use that binpkg > with ROOT=/somewhere, you can't go back and change SYSROOT. ROOT is used to determine ESYSROOT, not the other way around. As you say, (E)SYSROOT is only relevant in src phases so it doesn't matter if ROOT has changed when installing a binpkg. I take your point that setting a src phase variable based on a pkg phase variable seems odd but we're only using ROOT to determine the applicable prefix. We're not taking the actual value of ROOT. When Portage works all this out, it has access to all the necessary variables. It only filters the variables based on the phase function later on. > Also, I think the wording could be more "programmatic" (if ... then > ESYSROOT is ... else ... ) to avoid confusion. Given that there are three clauses, I thought that using "respectively" would be shorter and clearer but I'll try some other wording to see how it looks. -- James Le Cuirot (chewi) Gentoo Linux Developer pgpHV_CFv44RS.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable
On Sun, 28 Jul 2019 22:37:35 +0100
James Le Cuirot wrote:
[...]
> -& \t{BDEPEND} &
> \t{DEPEND} & \t{RDEPEND}, \t{PDEPEND} \\
> +& \t{BDEPEND} &
> \t{DEPEND}& \t{RDEPEND}, \t{PDEPEND} \\
> \midrule
There seems to be unneeded whitespace only changes here that make the
diff less readable
[...]
> \t{ESYSROOT} &
> Ditto &
> No &
> -Contains the concatenation of the paths in the \t{SYSROOT} and
> \t{EPREFIX} variables,
> -for convenience. See also the \t{EPREFIX} variable. Only for
> EAPIs listed
> -in table~\ref{tab:offset-env-vars-table} as supporting
> \t{ESYSROOT}. \\
> +Contains the concatenation of the \t{SYSROOT} path and
> applicable prefix value. The prefix value
> +is \t{EPREFIX}, \t{BROOT}, or blank if \t{SYSROOT} is equal to
> \t{ROOT}, \t{/}, or something
> +else respectively. Only for EAPIs listed in
> table~\ref{tab:offset-env-vars-table} as supporting
> +\t{ESYSROOT}. \\
> \t{BROOT} &
> Ditto &
> No &
Admittedly without a full understanding of the problem, but this looks
wrong to me: SYSROOT, EPREFIX and BROOT are only relevant in build
phases (src_*); (EPREFIX is a little special here but mostly for
convenience). ROOT is only relevant in pkg_* phases. I don't see how
this can work. Say I build a binpkg with ROOT=/ then use that binpkg
with ROOT=/somewhere, you can't go back and change SYSROOT.
Also, I think the wording could be more "programmatic" (if ... then
ESYSROOT is ... else ... ) to avoid confusion.
Alexis.
[gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable
It was originally envisaged (but not stated in PMS) that SYSROOT would
only ever need to equal / or ROOT as a distinct SYSROOT would have no
benefit. A check was added to Portage to ensure this held. Myself, the
ChromiumOS team, and others have since been caught out by this check
when trying to bootstrap brand new systems from scratch. You cannot
bootstrap with no headers at all! The check will therefore be adjusted
to merely ensure that SYSROOT is / when ROOT is /.
There were differing assumptions about how prefixes applied to the
above. EPREFIX is traditionally something the user sets so some
thought that it would be applied to SYSROOT, regardless of the
latter's value. In order to honor the rule about there being no
distinct SYSROOT, this would mean that if SYSROOT is / then EPREFIX
would have to match BROOT. Despite that limitation, ESYSROOT was
written into PMS with a fixed value of ${SYSROOT}${EPREFIX}. Being
somewhat unfamiliar with prefix at the time, I didn't realise that
this view didn't align with what I'd had in mind and it was only when
I came to need a distinct SYSROOT that I realised there was a problem.
crossdev toolchains are installed to ${EPREFIX}/usr/${CHOST} but have
no further prefix appended and packages subsequently installed with
cross-emerge are placed in this location by setting ROOT. Bug #642604
recently revealed that the build system's prefix was being erroneously
duplicated on the end but I have now fixed this.
What if we want to bootstrap a brand new prefixed system using the
crossdev system as SYSROOT? This is the distinct SYSROOT case. The
problem is that there is no distinct variable for SYSROOT's prefix
and, as already stated, ESYSROOT is always ${SYSROOT}${EPREFIX}. We
therefore cannot do it! If the crossdev prefix is blank then ROOT's
must be blank too.
I also never intended to have the aforementioned limitation where
EPREFIX must match BROOT when SYSROOT is /. These are both entirely
artificial restrictions.
So how should it work instead? We originally intended for SYSROOT to
equal either / or ROOT so I imagined the prefix would automatically be
adjusted to match the prefix applicable at the matching location,
namely BROOT or EPREFIX. This is obviously more flexible than forcing
it to match EPREFIX.
What about the distinct SYSROOT case? With no distinct variable, we
have no way to explicitly set a prefix but this is likely only needed
when bootstrapping against crossdev systems, which are unprefixed by
nature. We therefore simply assume that the prefix is blank in this
case.
What about the cross-prefix case? Here, SYSROOT matches both / and
ROOT so which prefix do we choose? The bootstrap-prefix.sh script sets
flags to build against the target prefix so EPREFIX is used in this
case. This happens to fit the current definition of ESYSROOT anyway.
Legitimate concerns have been raised about building for a system with
a different prefix to the one you're building against. The only
binaries that leak from SYSROOT to ROOT are static libraries. Headers
from SYSROOT will obviously also influence how ROOT's binaries are
built. It is entirely possible that SYSROOT's prefix may leak through
a header but grepping /usr/include on my own main system reveals only
a few paths from a small handful of packages. pkg-config files
invariably include paths but these are almost always used at build
time, not runtime. A differing prefix would likely only occur in cases
involving core packages like the libc and kernel headers anyway. Also
consider that we have never prevented this from happening in the
past. It has always been possible to do "EPREFIX=/foo emerge bar" from
some system with a different prefix or no prefix at all. All we're
doing here is including the prefix (if any) in the ESYSROOT variable.
Should this warrant a new EAPI? I don't think so. All existing usage
of ESYSROOT that I have seen still fits with this new definition and
most of that usage has come from me. We're not even changing what the
variable is used for, just loosening the constraints around what it
can be set to.
If you have doubts about whether this makes sense or actually works in
practise, I have experimented with a prefixed system using all the
different combinations I could think of, including cross-compiling,
and it all worked as expected. Keep in mind that ESYSROOT is not magic
and currently isn't used very much. As such, neither the toolchain nor
pkg-config will use this sysroot if you don't explicitly tell them
to. For the former, I find CC="${CHOST}-gcc --sysroot=${ESYSROOT}"
works well. For the latter, crossdev installs a cross-pkg-config
wrapper but it is completely lacking prefix support at the moment. I
have fixes waiting on this change.
Signed-off-by: James Le Cuirot
---
dependencies.tex| 24 ++--
ebuild-env-vars.tex | 7 ---
2 files changed, 22 insertions(+), 9 deletions(-)
diff --git a/dependencies.tex b/dependencies.tex
index 44382d1..fbe99ae 100644
--- a/
