[gentoo-dev] [PATCH 0/3] Remove parallel run support from multilib multibuild
Hello, everyone. Following a similar change in distutils-r1, I would like to remove the parallel run support in multilib eclasses, and effectively from multibuild completely. The goal is to make things simpler, and follow PMS properly. Right now the parallel runs imply running commands in a subshell. This has three important implications: 1. variable exports and environment changes inside the implicit subshell don't affect the outer scope -- developers are sometimes confused by that, 2. parallel runs can collide if temporary files aren't named uniquely -- think of ffmpeg, waf, 3. 'die' called in a subshell is unsupported by the PMS, and didn't make it into EAPI 6. The side advantages are that we can get rid of multiprocessing inherit and locking in multibuild_merge_root(). Possible issues: if people were relying on commands inside parallel not to affect the outer environment (i.e. doing 'cd' and expecting the next call to start in the initial location, or exporting variables and expecting them to disappear), the ebuilds will fail randomly. However, that's quite unlikely. Your thoughts? -- Best regards, Michał Górny
[gentoo-dev] [PATCH 1/3] multilib-minimal: do not run src_configure() in parallel
---
eclass/multilib-minimal.eclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/eclass/multilib-minimal.eclass b/eclass/multilib-minimal.eclass
index 4e1843e..3f6cfbd 100644
--- a/eclass/multilib-minimal.eclass
+++ b/eclass/multilib-minimal.eclass
@@ -51,7 +51,7 @@ multilib-minimal_src_configure() {
popd /dev/null || die
}
- multilib_parallel_foreach_abi multilib-minimal_abi_src_configure
+ multilib_foreach_abi multilib-minimal_abi_src_configure
}
multilib-minimal_src_compile() {
--
2.2.0
[gentoo-dev] [PATCH 2/3] multilib-build: unparallelize multilib_parallel_foreach_abi
---
eclass/multilib-build.eclass | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/eclass/multilib-build.eclass b/eclass/multilib-build.eclass
index 2e35d47..b0a4ea1 100644
--- a/eclass/multilib-build.eclass
+++ b/eclass/multilib-build.eclass
@@ -223,18 +223,18 @@ multilib_foreach_abi() {
# @DESCRIPTION:
# If multilib support is enabled, sets the toolchain up for each
# supported ABI along with the ABI variable and correct BUILD_DIR,
-# and runs the given commands with them. The commands are run
-# in parallel with number of jobs being determined from MAKEOPTS.
+# and runs the given commands with them.
#
# If multilib support is disabled, it just runs the commands. No setup
# is done.
#
-# Useful for running configure scripts.
+# This function used to run multiple commands in parallel. Now it's just
+# a deprecated alias to multilib_foreach_abi.
multilib_parallel_foreach_abi() {
debug-print-function ${FUNCNAME} ${@}
local MULTIBUILD_VARIANTS=( $(multilib_get_enabled_abi_pairs) )
- multibuild_parallel_foreach_variant _multilib_multibuild_wrapper ${@}
+ multibuild_foreach_variant _multilib_multibuild_wrapper ${@}
}
# @FUNCTION: multilib_for_best_abi
--
2.2.0
[gentoo-dev] [PATCH 3/3] multibuild: remove parallel run support
---
eclass/multibuild.eclass | 48 +++-
1 file changed, 3 insertions(+), 45 deletions(-)
diff --git a/eclass/multibuild.eclass b/eclass/multibuild.eclass
index 03e6280..bb17421 100644
--- a/eclass/multibuild.eclass
+++ b/eclass/multibuild.eclass
@@ -26,8 +26,6 @@ esac
if [[ ! ${_MULTIBUILD} ]]; then
-inherit multiprocessing
-
# @ECLASS-VARIABLE: MULTIBUILD_VARIANTS
# @DESCRIPTION:
# An array specifying all enabled variants which multibuild_foreach*
@@ -138,8 +136,8 @@ multibuild_foreach_variant() {
# @USAGE: [argv...]
# @DESCRIPTION:
# Run the passed command repeatedly for each of the enabled package
-# variants alike multibuild_foreach_variant. Multiple invocations of the
command
-# will be performed in parallel, up to MULTIBUILD_JOBS tasks.
+# variants. This used to run the commands in parallel but now it's
+# just a deprecated alias to multibuild_foreach_variant.
#
# The function returns 0 if all commands return 0, or the first non-zero
# exit status otherwise. However, it performs all the invocations
@@ -148,31 +146,7 @@ multibuild_foreach_variant() {
multibuild_parallel_foreach_variant() {
debug-print-function ${FUNCNAME} ${@}
- local ret lret
-
- _multibuild_parallel() {
- (
- multijob_child_init
- ${@}
- )
- multijob_post_fork
- }
-
- local opts
- if [[ ${MULTIBUILD_JOBS} ]]; then
- opts=-j${MULTIBUILD_JOBS}
- else
- opts=${MAKEOPTS}
- fi
-
- multijob_init ${opts}
- multibuild_foreach_variant _multibuild_parallel ${@}
- ret=${?}
- multijob_finish
- lret=${?}
-
- [[ ${ret} -eq 0 ]] ret=${lret}
- return ${ret}
+ multibuild_foreach_variant ${@}
}
# @FUNCTION: multibuild_for_best_variant
@@ -252,25 +226,12 @@ run_in_build_dir() {
# Merge the directory tree (fake root) from src-root to dest-root
# (the real root). Both directories have to be real, absolute paths
# (i.e. including ${D}). Source root will be removed.
-#
-# This functions uses locking to support merging during parallel
-# installs.
multibuild_merge_root() {
local src=${1}
local dest=${2}
- local lockfile=${T}/.multibuild_merge_lock
- local lockfile_l=${lockfile}.${BASHPID}
local ret
- # Lock the install tree for merge. The touch+ln method ensures race
- # condition-free locking with maximum portability.
- touch ${lockfile_l} || die
- until ln ${lockfile_l} ${lockfile} /dev/null; do
- sleep 1
- done
- rm ${lockfile_l} || die
-
if use userland_BSD; then
# Most of BSD variants fail to copy broken symlinks, #447370
# also, they do not support --version
@@ -297,9 +258,6 @@ multibuild_merge_root() {
ret=${?}
fi
- # Remove the lock.
- rm ${lockfile} || die
-
if [[ ${ret} -ne 0 ]]; then
die ${MULTIBUILD_VARIANT:-(unknown)}: merging image failed.
fi
--
2.2.0
Re: [gentoo-dev] [PATCH 0/3] Remove parallel run support from multilib multibuild
On Thu, 11 Dec 2014 11:36:29 +0100 Michał Górny mgo...@gentoo.org wrote: Hello, everyone. Following a similar change in distutils-r1, I would like to remove the parallel run support in multilib eclasses, and effectively from multibuild completely. [...] Your thoughts? +1 I hate parallel builds as I can't follow build execution properly; and, on a different level, there's emerge --jobs already. Alexis.
Re: [gentoo-dev] Re: metadata.xml unherd/-ization, v2
On Thu, Dec 11, 2014 at 1:23 AM, Michael Palimaka kensing...@gentoo.org wrote: This would be by far the easiest solution. Some herds already have an alias like this eg. freedesktop - freedesktop-bugs. Much easier than mass-editing every single metadata.xml with what amounts to a cosmetic change. I think we're not all on the same page about what we're trying to accomplish. I think we need to agree on that before we can really agree on how to do it. There are many goals here that I can see: 1. Simplification of the xml schema so that we don't have so many different kinds of tags with different rules for each. 2. Simplification of how we track group (ie project/herd/alias/etc) member lists so that they're not in 5 different places with 5 different ways of determining who is in a group. 3. Avoiding having large groups of packages maintained by large groups of devs where the reality is that many packages aren't maintained by anybody but this is opaque. 4. If not all of the emails associated with metadata are considered true maintainers, making it easy to tell which ones are and aren't. I don't think all of these have equal support, which is why we end up debating different solutions (obviously a solution which addresses all of these is going to be more intrusive than a solution that only addresses some of these). I've been a proponent of solving all of these, but perhaps it would make more sense to start smaller than that. The only catch is that if we remove the distinction in metadata between maintainers/proxies/projects/herds/etc then if that distinction becomes more important in the future it becomes harder to tell which ones are which. Part of me is wondering if worrying about #3-4 is actually all that productive. Does it really matter if a package is maintained or not, when it all comes down to it? Does it make more sense to focus on whether packages have serious problems? Maybe if a package is completely unmaintained it makes it easier for developers to drop in and make changes without asking anybody about it first, versus logging a bug, waiting for the maintainer to drop the ball, and then making the change anyway. -- Rich
Re: [gentoo-dev] Lastrites: net-im/linpopup, app-office/teapot, net-irc/bitchx, sys-power/cpufrequtils, x11-plugins/gkrellm-cpufreq, media-sound/gnome-alsamixer, sys-devel/ac-archive, net-misc/emirror
On Tue, 09 Dec 2014 11:05:31 +0100
Pacho Ramos pa...@gentoo.org wrote:
Regarding the gkrellm-plugins, looks like Fedora is supplying this
one:
http://pkgs.fedoraproject.org/cgit/gkrellm-freq.git/tree/gkrellm-freq.spec
I rewrote ebuild for gkrellm-gkfreq plugin a bit
https://github.com/cerebrum/dr/blob/master/x11-plugins/gkrellm-gkfreq/gkrellm-gkfreq-2.3.ebuild
And sent this patch upstream
https://github.com/cerebrum/dr/blob/master/x11-plugins/gkrellm-gkfreq/files/gkrellm-gkfreq-2.3-make.patch
https://sourceforge.net/p/gkrellm-gkfreq/feature-requests/2/
Why do we have a function tc-getCC for CC environment variable and why
we don't use CC environment variable automatically? With this patch
CFLAGS and LDFLAGS are substituted automatically without garbage in
ebuild. Why don't we use the same strategy for CC variable?
Sometimes a package will try to use a bizarre compiler, or will need
to be told which compiler to use. In these situations, the tc-getCC()
function from toolchain-funcs.eclass should be used...
Note: It is not correct to use the ${CC} variable for this purpose.
https://devmanual.gentoo.org/ebuild-writing/functions/src_compile/building/index.html
And why CC variable gives me cc but not x86_64-pc-linux-gnu-gcc as
it should be? Am I right that it's a bit strange?
Re: [gentoo-dev] sys-devel/gcc::mgorny up for testing
On 12/09/2014 11:21, Michał Górny wrote: Dnia 2014-12-08, o godz. 09:56:11 Alexis Ballier aball...@gentoo.org napisał(a): On Sun, 7 Dec 2014 11:37:57 +0100 Michał Górny mgo...@gentoo.org wrote: 1. No cross-compilation support. If the project proves being a success it will be readded at some point. However, I will likely fork glibc first and work on a sane crossdev alternative. Could you please elaborate on this ? How and why this broke, what is wrong in glibc, what would be a sane crossdev alternative and how crossdev is not. 1. eblits are broken in glibc. Another case of incorrect interpretation of code sharing with love for breaking stable ebuilds. Plus I'd love to add multilib flags to it but we'll likely doing that to main glibc as well soon, if eblits don't get into our way. Originally, eblits were supposed to be ways of sharing common blocks of code between ebuilds in a specific package, but not anything that would warrant a new eclass (local package eclasses, kinda?). They never really took off for reasons lost to time. I use them in mips-sources, but version them so that if I change an eblit, it gets a new version and the old version is removed only when there are no more consumers. The idea of eblits itself isn't bad, but probably could use some re-thinking and/or re-implementation. Especially if a decent amount of kilobytes can be shaved off a given package's directory. 2. crossdev is broken because it creates semi-random, unmaintained ebuilds in randomly chosen overlay, those ebuilds relying on toolchain.eclass behavior. I would argue this isn't necessarily broken. crossdev produces working cross-toolchains and is a valuable tool for anyone running multiple arches, especially slower arches. How it produces said working cross-toolchains is an implementation detail. Don't be so hasty as to throw the baby out with the bathwater. -- Joshua Kinard Gentoo/MIPS ku...@gentoo.org 4096R/D25D95E3 2011-03-28 The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between. --Emperor Turhan, Centauri Republic
Re: [gentoo-portage-dev] [PATCH] search._xmatch: handle aux_get KeyError (525718)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I don't have a suggestion for where to put it, but I really dislike having the exact same thing twice like this. - -- Alexander berna...@gentoo.org https://secure.plaimi.net/~alexander -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlSJUPUACgkQRtClrXBQc7UnSgD+MwGTCNZNY41PMgakwD/3ASs2 gpknBhUet2oI5WteoKwA/jjj191j9VmxsaK7xD0Zv8uhs4MxdbYgJ7ydRKFKANL6 =6vpE -END PGP SIGNATURE-
Re: [gentoo-portage-dev] [PATCH] Support @profile package set for bug #532224
Dnia 2014-12-10, o godz. 18:08:00 Zac Medico zmed...@gentoo.org napisał(a): Add support for a new @profile set which allows the profile to pull in additional packages that do not belong to the @system set. The motivation to have @profile separate from @system is that @system packages may have incomplete dependency specifications (due to long-standing Gentoo policy), and incomplete dependency specifications have deleterious effects on the ability of emerge --jobs to parallelize builds. So, unlike @system, packages added to @profile do not hurt emerge --jobs parallelization. Packages are added to the @profile set in the same way that they are added to the @system set, except that atoms in the @profile set are not preceded with a '*' character. Also, the @profile package set is only supported when 'profile-set' is listed in the layout.conf profile-formats field of the containing repository. PMS says PMs ought to ignore atoms without '*'. This means we can't use it without profile EAPI change, or other PMs will start losing packages. -- Best regards, Michał Górny pgpPG6RfXCJXS.pgp Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH v2] search._xmatch: handle aux_get KeyError (525718)
Since commit 55c8c8bc7a781e3f71ce92922eea64ad4cafce3c, emerge --search
can raise an unhandled KeyError if a pkg_desc_index contains a stale
package. Handle it with the same error message used for other aux_get
failures.
Fixes: 55c8c8bc7a78 (Add emerge --search-index option.)
X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
---
PATCH v2 splits out a _aux_get_error method to unify the error output
formatting.
pym/_emerge/search.py | 29 +
1 file changed, 21 insertions(+), 8 deletions(-)
diff --git a/pym/_emerge/search.py b/pym/_emerge/search.py
index 90dbcec..e7f6f44 100644
--- a/pym/_emerge/search.py
+++ b/pym/_emerge/search.py
@@ -1,6 +1,8 @@
# Copyright 1999-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
+from __future__ import unicode_literals
+
import re
import portage
from portage import os
@@ -86,6 +88,11 @@ class search(object):
pass
raise KeyError(args[0])
+ def _aux_get_error(self, cpv):
+ portage.writemsg(emerge: search:
+ aux_get('%s') failed, skipping\n % cpv,
+ noiselevel=-1)
+
def _findname(self, *args, **kwargs):
for db in self._dbs:
if db is not self._portdb:
@@ -166,8 +173,12 @@ class search(object):
else:
db_keys = list(db._aux_cache_keys)
for cpv in db.match(atom):
- metadata = zip(db_keys,
- db.aux_get(cpv,
db_keys))
+ try:
+ metadata = zip(db_keys,
+ db.aux_get(cpv,
db_keys))
+ except KeyError:
+ self._aux_get_error(cpv)
+ continue
if not self._visible(db, cpv,
metadata):
continue
matches.add(cpv)
@@ -197,8 +208,12 @@ class search(object):
for cpv in reversed(matches):
if portage.cpv_getkey(cpv) !=
cp:
continue
- metadata = zip(db_keys,
- db.aux_get(cpv,
db_keys))
+ try:
+ metadata = zip(db_keys,
+ db.aux_get(cpv,
db_keys))
+ except KeyError:
+ self._aux_get_error(cpv)
+ continue
if not self._visible(db, cpv,
metadata):
continue
if not result or cpv ==
portage.best([cpv, result]):
@@ -257,9 +272,7 @@ class search(object):
full_desc = self._aux_get(
full_package,
[DESCRIPTION])[0]
except KeyError:
- portage.writemsg(
- emerge: search: aux_get()
failed, skipping\n,
- noiselevel=-1)
+ self._aux_get_error(full_package)
continue
if not self.searchre.search(full_desc):
continue
@@ -337,7 +350,7 @@ class search(object):
metadata =
dict(zip(metadata_keys,
self._aux_get(full_package, metadata_keys)))
except KeyError:
- msg.append(emerge: search:
aux_get() failed, skipping\n)
+
self._aux_get_error(full_package)
continue
desc = metadata[DESCRIPTION]
--
2.0.4
Re: [gentoo-portage-dev] [PATCH v2] search._xmatch: handle aux_get KeyError (525718)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Yeah this is nicer, go ahead merge. - -- Alexander berna...@gentoo.org https://secure.plaimi.net/~alexander -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlSJV4IACgkQRtClrXBQc7W+VgD7BUuZb940KY5hiQZUsEaLLLAD PGDZKd2yIVtJB0uLJqAA/3/AQmjfht8+TAPmwWImiaykO3+oSCZXYARJfsnEv47X =LMHO -END PGP SIGNATURE-
Re: [gentoo-portage-dev] [PATCH] Support @profile package set for bug #532224
On 12/11/2014 12:25 AM, Michał Górny wrote: Dnia 2014-12-10, o godz. 18:08:00 Zac Medico zmed...@gentoo.org napisał(a): Add support for a new @profile set which allows the profile to pull in additional packages that do not belong to the @system set. The motivation to have @profile separate from @system is that @system packages may have incomplete dependency specifications (due to long-standing Gentoo policy), and incomplete dependency specifications have deleterious effects on the ability of emerge --jobs to parallelize builds. So, unlike @system, packages added to @profile do not hurt emerge --jobs parallelization. Packages are added to the @profile set in the same way that they are added to the @system set, except that atoms in the @profile set are not preceded with a '*' character. Also, the @profile package set is only supported when 'profile-set' is listed in the layout.conf profile-formats field of the containing repository. PMS says PMs ought to ignore atoms without '*'. This means we can't use it without profile EAPI change, or other PMs will start losing packages. It's hidden behind a layout.conf profile-formats flag, so it's beyond the scope of PMS. Package managers should reject the profile if they don't recognize the profile-formats flags that it declares. -- Thanks, Zac
