[gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds
Recently, Prefix changes have been committed to the gentoo-x86 tree, it
was rather ambitious on my part, where I moved stuff that we are not
maintainer of ourself. It should have been communicated better for
these ebuilds. This is a formal apology for springing that onto you.
This will attempt to solve the confusion and answer questions that have
been raised.
Gentoo Prefix has mainly the following characteristics:
- allow for unprivileged installations,
- in an offset-prefix called the prefix, often referred to as EPREFIX,
in autoconf words: ./configure --prefix=${EPREFIX}/usr
In the Prefix Portage branch, that we have in use [1], we extended and
adapted regular Portage to support above two characteristics. I will
ignore the details of unprivileged installations in this email.
The offset-prefix changes in Portage introduce three new variables that
are available in the ebuild environment:
- EPREFIX: the configured offset-prefix
- ED: shorthand for ${D%/}${EPREFIX}/
- EROOT: ${ROOT%/}${EPREFIX}/
The offset-prefix changes require extensive changes to most eclasses,
and minor changes to many ebuilds. This is mainly because awareness
of the offset-prefix has to be added to places where ebuilds manually
deal with file-system locations. In particular:
- configure calls that specify some argument to find a component need to
do so in the offset prefix, e.g.
--with-libxml2=${EPREFIX}/usr/$(get_libdir)
- almost every place where ${D} is used, the offset-prefix ${EPREFIX}
has to be added. Because this is lengthy, Prefix Portage provides a
variable ${ED} that is the shortcut for ${D} plus the offset-prefix.
- the same holds for occurrences of ${ROOT}, where ${EROOT} is available
as shortcut
Why do we need both ${ED} and ${D}? Technically we don't, because we
can use ${EPREFIX} all the time. However, one cannot say that ${D}
includes ${EPREFIX} for Prefix Portage, because that means the following
src_install() function no longer works correctly:
src_install() {
emake DESTDIR=${D} install || die
mv ${D}/usr/bin/{,exuberant-}ctags || die
}
While the mv does a great job if ${D} would include ${EPREFIX} here,
the make install will cause double offset to be written, since configure
recorded the ${EPREFIX} before in src_compile using the argument
--prefix=${EPREFIX}/usr. In a previous version of Prefix Portage, the
variable EDEST was available as workaround for this, so the example
would read like this:
src_install() {
emake DESTDIR=${EDEST} install || die
mv ${D}/usr/bin/{,exuberant-}ctags || die
}
Apart from that this approach got very tricky and confusing in eclasses
and ebuilds that do special mungling in their src_install, it makes it
harder to reconstruct the variable in normal Portage and hence to make
existing ebuilds forward compatible.
The lengthy forward compatible version of the example src_install
function would look like this:
src_install() {
emake DESTDIR=${D} install || die
mv ${D%/}${EPREFIX}/usr/bin/{,exuberant-}ctags || die
}
As mentioned before, this is pretty lengthy, and quickly becomes too
much work and unreadable when ${D} is used more.
To avoid the confusion that implicit ${EPREFIX} in ${D} in the ${EDEST}
approach brought, ED and EROOT were chosen because they are easy to
understand and easy to reconstruct outside Prefix Portage. To help
with this, the Prefix profiles use.force the prefix USE-flag.
Non-Prefix profiles have this flag masked and unset in the base profile.
This USE-flag can be used to do conditional code for Prefix consumers.
In case of our ${ED} and ${EROOT} convenience variables, we can use it
to define ${ED} and ${EROOT} in case a normal Portage is used. For our
example function:
src_install() {
use prefix || local ED=${D}
emake DESTDIR=${D} install || die
mv ${ED}/usr/bin/{,exuberant-}ctags || die
}
Here, Prefix Portage (using a Prefix profile) will not set ${ED}, but
still do as intended because ${ED} is set correctly by Prefix Portage.
Normal Portage will set the convenience variable such that it does not
cause a sandbox violation due to the missing image path.
We will consult the maintainers of packages we would like to make
compatible with Gentoo Portage before adding the changes. In the
future, you will likely see more bug reports and more requests on IRC
from us.
At this point, we have reached a critical mass where we cannot maintain
the Prefix overlay much longer with its size and usage. We either
continue to grow on, requiring less maintenance on tree synchronisation,
or we stop the project -- an option we really don't like.
Bringing back most, and ideally all, ebuilds to gentoo-x86 significantly
improves the workload caused by synchronisation, leaving more time for
the real issues. Examples are fixing and porting packages and getting
the Prefix Portage branch merged with regular Portage some day. At that
point, the variables EPREFIX, ED and EROOT can become available in a
next
Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds
Hi, You know i am totaly supporting prefix but i have one point. Why on earth portage simply does not detect the prefix enviroment is being run and then INTERNALY switch D-ED and other variables. It would be much easier that way to migrate all stuff in portage instead of doing this || shebang. Mostly when it is done by eclasses its quite cool, but when you get into changing lots of ebuilds its quite hard for maintaining. Even the multilib overlay guys rather modify the portage than changing a load of ebuilds. Just my 2 cents. Tomas
Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds
On 18-10-2009 13:57:10 +0200, Tomáš Chvátal wrote: Hi, You know i am totaly supporting prefix but i have one point. Why on earth portage simply does not detect the prefix enviroment is being run and then INTERNALY switch D-ED and other variables. It would be much easier that way to migrate all stuff in portage instead of doing this || shebang. Mostly when it is done by eclasses its quite cool, but when you get into changing lots of ebuilds its quite hard for maintaining. Even the multilib overlay guys rather modify the portage than changing a load of ebuilds. Of course we would like to do that, but that was rejected for EAPI=3, so it will at least take until EAPI=4 is implemented, which is not the forseeable future, given that EAPI=3 isn't a fact yet either. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay
Mike Frysinger schrieb: another quick look at _setup_abi_env() looks like it needs work: - LD should not default to `ld` Whats your suggestion? - the -L paths to system dirs in LDFLAGS should not be there -- the toolchain can handle these just fine Last time i tried without, some packages failed to compile, will test it again to check, if its still needed - missing CPPFLAGS handling Added - see previous comments re-pkg-config Will have a look the next days - you dont declare pyver local fixed - the abi if check is uncommon syntax. ! [[ a == b ]] - [[ a != b ]] fixed how do you control whether the multilib headers are needed ? it'll probably make sense in general, but there are def some packages where this will only get in the way (the toolchain ones). What do you mean here? If the diff between ABIs makes sense to install seperate versions? how do you differentiate between packages where multi ABIs make no sense ? for example, most packages that dont install any libraries but just binaries. let's use the simple package app-text/tree. I dont differentiate. Currently i build for every ABI, if lib32 useflag is set and multilib useflag is not set. The idea behind it is to allow the installation of additional 32bit binaries, if requested. a lot of this multilib code should probably continue to live in the tree as it's a pretty big base of code to formalize that could do with fixes over time. i.e. we figure out that certain paths / define protections dont work so well, so changing them to something else would require PMS changing !? there has been talk before about pushing a lot of basic stuff to the tree so things dont have to be encoded in the PMS. How do you want to do this? Require package managers to inherit some file/eclass? how are packages handled that can only be used via 1 ABI ? any of the packages listed in the amd64 no-multilib package.mask for example. while these are mostly binary-only, this isnt a binary-specific issue. there are a number of packages which only work on x86/ppc but could easily work in a multilib amd64/ppc64 as a 32bit binary (source code sucks, is heavily asm, something else). The binary-only ones are easy. Since they dont interact with the env, they can be installed as usual. If they depend on a new enough package manager with multilib support, they could also depend on the useflag for the additional 32bit libs they need. 2. Adding the internal lib32 useflag and usedeps with some workarounds what exactly does this lib32 do ? naming USE flags according to specific ABI implementations is a bad idea. you have to forget special casing anything to lib32 vs lib64. amd64, while the most common, is hardly extensible. we must handle multiple ABIs which easily might have the same bitsize. lib32 is added to IUSE, you can enable as as every other useflag. Internally, portage does add [lib32?] usedeps to all dependencies. So if you enable the lib32 useflag, portage will require this useflag for all dependencies too. I dont mind renaming it, if there is some other sane naming for it. -- Thomas Sachau Gentoo Linux Developer signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay
Robin H. Johnson schrieb: On Mon, Oct 12, 2009 at 04:50:23PM -0400, Mike Frysinger wrote: what exactly does this lib32 do ? naming USE flags according to specific ABI implementations is a bad idea. you have to forget special casing anything to lib32 vs lib64. amd64, while the most common, is hardly extensible. we must handle multiple ABIs which easily might have the same bitsize. The canonical example for this does still remain MIPS I believe. The most common ABIs in MIPS are: o32, n32 - Both in Gentoo releases n64 - Was experimentally done in Gentoo (default-linux/mips/2007.1-dev/generic-be/n64) o64, eabi, meabi, nubi - Not sure if they were were ever released in any way. Crossdev DOES support the full swath of these last I checked it. There is a difference between creating a toolchain and supporting all packages for that arch and every possible ABI you can crosscompile. Currently i only support amd64 since thats the only ARCH i know and have access to. If i get enough details to implement other ARCHes and some way to test it there, i might try it for those other ARCHes too. -- Thomas Sachau Gentoo Linux Developer signature.asc Description: OpenPGP digital signature
[gentoo-dev] Re: openrc-0.5.1 arrived in the tree
Hi, Tomáš Chvátal scarab...@gentoo.org: Actualy i would like to hear what we in KDE did too, we publish into the tree as 0 days bump mostly since 4.2 and 4.1 was in the tree right away when we had working configuration. Your bumping is excellent, no discussion here. Gnome does a really good job in figuring out what packages need to go stable along with their newest release, maybe that is easier for them. With the ongoing KDE 4 stabilisation, I have seen a lot of stabilisation requests filed by Samuli, which should have been already there, filed by the KDE team. Figuring out what will be broken with such a major release and trying to get the stable tree on par is what eats a lot of time for us architecture developers...you can ease that by researching beforehand (with a stable chroot for example). V-Li -- Christian Faulhammer, Gentoo Lisp project URL:http://www.gentoo.org/proj/en/lisp/, #gentoo-lisp on FreeNode URL:http://gentoo.faulhammer.org/ signature.asc Description: PGP signature
[gentoo-dev] Automated Package Removal and Addition Tracker, for the week ending 2009-10-18 23h59 UTC
The attached list notes all of the packages that were added or removed from the tree, for the week ending 2009-10-18 23h59 UTC. Removals: sys-apps/minit 2009-10-12 15:39:42 vostorga kde-base/artsplugin-mpeglib 2009-10-12 23:27:53 abcd kde-base/artsplugin-mpg123 2009-10-12 23:28:37 abcd kde-base/mpeglib2009-10-12 23:31:20 abcd media-sound/beast 2009-10-13 11:05:46 ssuominen media-sound/beast-data 2009-10-13 11:05:47 ssuominen media-plugins/flexplaylist 2009-10-13 11:07:18 ssuominen net-wireless/bluez-bluefw 2009-10-13 11:08:37 ssuominen app-mobilephone/x70talk 2009-10-13 11:11:03 ssuominen sys-libs/uclibc++ 2009-10-13 11:14:14 ssuominen app-pda/multisync 2009-10-13 11:25:26 ssuominen app-admin/realpath 2009-10-13 17:13:42 ulm Additions: dev-perl/Test-Perl-Critic 2009-10-12 08:11:45 tove sys-apps/minit 2009-10-12 15:01:52 vostorga dev-ml/react2009-10-13 06:37:20 aballier app-misc/realpath 2009-10-13 16:58:23 ulm app-backup/backintime 2009-10-14 13:28:10 bangert net-wireless/kbluetooth 2009-10-15 16:06:41 ssuominen dev-python/wtforms 2009-10-16 10:02:48 djc x11-libs/qt-multimedia 2009-10-16 16:47:56 wired gnome-extra/gget2009-10-16 16:53:27 mrpouet net-misc/rwhoisd2009-10-16 20:16:36 kingtaco sys-boot/shlilo-lantank 2009-10-18 22:16:14 vapier mail-client/claws-mail-python 2009-10-18 23:47:58 fauli -- Robin Hugh Johnson Gentoo Linux Developer E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 Removed Packages: sys-apps/minit,removed,vostorga,2009-10-12 15:39:42 kde-base/artsplugin-mpeglib,removed,abcd,2009-10-12 23:27:53 kde-base/artsplugin-mpg123,removed,abcd,2009-10-12 23:28:37 kde-base/mpeglib,removed,abcd,2009-10-12 23:31:20 media-sound/beast,removed,ssuominen,2009-10-13 11:05:46 media-sound/beast-data,removed,ssuominen,2009-10-13 11:05:47 media-plugins/flexplaylist,removed,ssuominen,2009-10-13 11:07:18 net-wireless/bluez-bluefw,removed,ssuominen,2009-10-13 11:08:37 app-mobilephone/x70talk,removed,ssuominen,2009-10-13 11:11:03 sys-libs/uclibc++,removed,ssuominen,2009-10-13 11:14:14 app-pda/multisync,removed,ssuominen,2009-10-13 11:25:26 app-admin/realpath,removed,ulm,2009-10-13 17:13:42 Added Packages: dev-perl/Test-Perl-Critic,added,tove,2009-10-12 08:11:45 sys-apps/minit,added,vostorga,2009-10-12 15:01:52 dev-ml/react,added,aballier,2009-10-13 06:37:20 app-misc/realpath,added,ulm,2009-10-13 16:58:23 app-backup/backintime,added,bangert,2009-10-14 13:28:10 net-wireless/kbluetooth,added,ssuominen,2009-10-15 16:06:41 dev-python/wtforms,added,djc,2009-10-16 10:02:48 x11-libs/qt-multimedia,added,wired,2009-10-16 16:47:56 gnome-extra/gget,added,mrpouet,2009-10-16 16:53:27 net-misc/rwhoisd,added,kingtaco,2009-10-16 20:16:36 sys-boot/shlilo-lantank,added,vapier,2009-10-18 22:16:14 mail-client/claws-mail-python,added,fauli,2009-10-18 23:47:58 Done.
Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay
On Sunday 18 October 2009 14:49:09 Thomas Sachau wrote: Robin H. Johnson schrieb: On Mon, Oct 12, 2009 at 04:50:23PM -0400, Mike Frysinger wrote: what exactly does this lib32 do ? naming USE flags according to specific ABI implementations is a bad idea. you have to forget special casing anything to lib32 vs lib64. amd64, while the most common, is hardly extensible. we must handle multiple ABIs which easily might have the same bitsize. The canonical example for this does still remain MIPS I believe. The most common ABIs in MIPS are: o32, n32 - Both in Gentoo releases n64 - Was experimentally done in Gentoo (default-linux/mips/2007.1-dev/generic-be/n64) o64, eabi, meabi, nubi - Not sure if they were were ever released in any way. Crossdev DOES support the full swath of these last I checked it. There is a difference between creating a toolchain and supporting all packages for that arch and every possible ABI you can crosscompile. Currently i only support amd64 since thats the only ARCH i know and have access to. If i get enough details to implement other ARCHes and some way to test it there, i might try it for those other ARCHes too. we're not asking you to implement support for these ABIs, just to keep in mind that any implementation that does not scale and/or hardcodes to a single set of ABIs isnt a proper solution -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay
On Sun, Oct 18, 2009 at 10:26:37PM -0400, Mike Frysinger wrote:
On Sunday 18 October 2009 14:49:09 Thomas Sachau wrote:
Robin H. Johnson schrieb:
On Mon, Oct 12, 2009 at 04:50:23PM -0400, Mike Frysinger wrote:
what exactly does this lib32 do ? naming USE flags according to
specific ABI implementations is a bad idea. you have to forget special
casing anything to lib32 vs lib64. amd64, while the most common, is
hardly extensible. we must handle multiple ABIs which easily might have
the same bitsize.
The canonical example for this does still remain MIPS I believe.
The most common ABIs in MIPS are:
o32, n32 - Both in Gentoo releases
n64 - Was experimentally done in Gentoo
(default-linux/mips/2007.1-dev/generic-be/n64) o64, eabi, meabi, nubi -
Not sure if they were were ever released in any way.
Crossdev DOES support the full swath of these last I checked it.
There is a difference between creating a toolchain and supporting all
packages for that arch and every possible ABI you can crosscompile.
Currently i only support amd64 since thats the only ARCH i know and have
access to. If i get enough details to implement other ARCHes and some way
to test it there, i might try it for those other ARCHes too.
we're not asking you to implement support for these ABIs, just to keep in
mind
that any implementation that does not scale and/or hardcodes to a single set
of ABIs isnt a proper solution
My main objection thusfar is hardcoding the names as lib64/lib32.
Is there any specific reason you're using the content of the
LIBDIR_${ABI} variable? Maybe using the $ABI string as the USE flag
would be better, in some form of USE_EXPAND manner (cannot use them raw
as I believe we still have some code in the form of 'use $ARCH', which
might not behave sanely if say both amd64 and x86 were set.
Egs:
multilib AMD64:
USE=abi_x86 abi_amd64
Multilib PPC64:
USE=abi_ppc abi_ppc64
Multilib MIPS (SGI hardware)
USE=abi_n32 abi_n64
Possible valid multilib sets on MIPS are:
[n32,n64] - at least one Gentoo system like this has been built.
[o32,o64]
[eabi,meabi] - docs fuzzy
[nubi]
--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee Infrastructure Lead
E-Mail : robb...@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
pgpuMdHJrA6yA.pgp
Description: PGP signature
Re: [gentoo-portage-dev] Can an ebuild override CONFIG_PROTECT?
Amit Dor-Shifer wrote: Thanks. IMHO this usage of env.d should be documented in the ebuild howto, as it presents a useful-yet-not-trivial mechanism. I'd document it, if: a) I get no objection from others. Anyone? b) I'd be referred to the proper place to document it in. There's a page dealing w/ |CONFIG_PROTECT_MASK at http://devmanual.gentoo.org/general-concepts/config-protect/index.html, but I doubt if that's the proper place. It's touched upon in the User Handbook here: http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?full=1#book_part2_chap5 It might be nice to mention it somewhere in the Developer Handbook: http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?full=1 That's maintained by the devrel team, and you can file a bug for them at bugs.gentoo.org. -- Thanks, Zac
Re: [gentoo-portage-dev] Can an ebuild override CONFIG_PROTECT?
Does emerge actually call 'env-update'? If so, where (I can't find in in the code)? if not, who performs the environment update when an ebuild is emerged (I'm supposing env-update gets called. I didn't call it explicitly in my ebuild, yet still /etc/profile.env gets updated with CONFIG_PROTECT_MASK)? Amit Zac Medico wrote: Amit Dor-Shifer wrote: Thanks. IMHO this usage of env.d should be documented in the ebuild howto, as it presents a useful-yet-not-trivial mechanism. I'd document it, if: a) I get no objection from others. Anyone? b) I'd be referred to the proper place to document it in. There's a page dealing w/ |CONFIG_PROTECT_MASK at http://devmanual.gentoo.org/general-concepts/config-protect/index.html, but I doubt if that's the proper place. It's touched upon in the User Handbook here: http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?full=1#book_part2_chap5 It might be nice to mention it somewhere in the Developer Handbook: http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?full=1 That's maintained by the devrel team, and you can file a bug for them at bugs.gentoo.org.
Re: [gentoo-portage-dev] Can an ebuild override CONFIG_PROTECT?
Amit Dor-Shifer wrote: Does emerge actually call 'env-update'? If so, where (I can't find in in the code)? if not, who performs the environment update when an ebuild is emerged (I'm supposing env-update gets called. I didn't call it explicitly in my ebuild, yet still /etc/profile.env gets updated with CONFIG_PROTECT_MASK)? Amit Yes, emerge calls env-update automatically. But, you should call source /etc/profile afterwards in you running shell. Also, there's this bug which prevents new CONFIG_PROTECT* settings from taking immediate effect: http://bugs.gentoo.org/show_bug.cgi?id=276345 -- Thanks, Zac
