Re: [gentoo-portage-dev] [PATCH 16/18] Disallow helpers in global scope in EAPI 6
On 01/16/2015 02:14 AM, Sergei Trofimovich wrote:
On Mon, 1 Dec 2014 22:28:34 +0100
Michał Górny mgo...@gentoo.org wrote:
Disallow calling most of the ebuild helpers in global scope since they
are meaningless in that context. Most of them are also prohibited by PMS
for all EAPIs, so EAPI 6 sounds like a good moment to finally enforce
that restriction.
---
bin/eapi.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/eapi.sh b/bin/eapi.sh
index e0ade02..5ab92f4 100644
--- a/bin/eapi.sh
+++ b/bin/eapi.sh
@@ -159,7 +159,7 @@ ___eapi_helpers_can_die() {
}
___eapi_disallows_helpers_in_global_scope() {
-[[ ${1-${EAPI}} =~ ^(4-python|5-progress)$ ]]
+[[ ! ${1-${EAPI}} =~ ^(0|1|2|3|4|4-slot-abi|5|5-hdepend)$ ]]
}
___eapi_unpack_is_case_sensitive() {
Looks like it cries on ebuilds with unset EAPI in main tree (attached).
Can be easily fixed in tree by adding EAPI=0, but better handle it
here: ${EAPI-0} (untested!)
Using ${EAPI-0} would not be the correct fix, because portage is
supposed to export the EAPI that was previously determined via the
_parse_eapi_ebuild_head function (preprocessor).
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH 16/18] Disallow helpers in global scope in EAPI 6
On 01/16/2015 10:18 AM, Zac Medico wrote:
On 01/16/2015 02:14 AM, Sergei Trofimovich wrote:
On Mon, 1 Dec 2014 22:28:34 +0100
Michał Górny mgo...@gentoo.org wrote:
Disallow calling most of the ebuild helpers in global scope since they
are meaningless in that context. Most of them are also prohibited by PMS
for all EAPIs, so EAPI 6 sounds like a good moment to finally enforce
that restriction.
---
bin/eapi.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/eapi.sh b/bin/eapi.sh
index e0ade02..5ab92f4 100644
--- a/bin/eapi.sh
+++ b/bin/eapi.sh
@@ -159,7 +159,7 @@ ___eapi_helpers_can_die() {
}
___eapi_disallows_helpers_in_global_scope() {
- [[ ${1-${EAPI}} =~ ^(4-python|5-progress)$ ]]
+ [[ ! ${1-${EAPI}} =~ ^(0|1|2|3|4|4-slot-abi|5|5-hdepend)$ ]]
}
___eapi_unpack_is_case_sensitive() {
Looks like it cries on ebuilds with unset EAPI in main tree (attached).
Can be easily fixed in tree by adding EAPI=0, but better handle it
here: ${EAPI-0} (untested!)
Using ${EAPI-0} would not be the correct fix, because portage is
supposed to export the EAPI that was previously determined via the
_parse_eapi_ebuild_head function (preprocessor).
Actually, ebuild.sh unsets the EAPI just before it sources the ebuild,
and then it does this after it sources the ebuild:
[ ${EAPI+set} = set ] || EAPI=0
So, for any code that is called while the ebuild is being sourced, using
${EAPI-0} would be correct.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH] Support USE_EXPAND prefixes in package.use and relevant files
On 01/20/2015 03:18 AM, Michał Górny wrote: Support prefixing groups of USE_EXPAND flags with 'USE_EXPAND:' in user configuration package.use and similar files. This provides a convenient way of declaring specifying multiple USE_EXPAND flags and matches the syntax supported by Paludis. Example: dev-util/netbeans NETBEANS_MODULES: php webcommon extide media-libs/mesa osmesa VIDEO_CARDS: intel nouveau --- pym/portage/package/ebuild/_config/UseManager.py | 20 +++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/pym/portage/package/ebuild/_config/UseManager.py b/pym/portage/package/ebuild/_config/UseManager.py index 3a4ec22..59f474e 100644 --- a/pym/portage/package/ebuild/_config/UseManager.py +++ b/pym/portage/package/ebuild/_config/UseManager.py @@ -202,10 +202,17 @@ class UseManager(object): useflag_re = _get_useflag_re(eapi) for k, v in file_dict.items(): useflags = [] + use_expand_prefix = '' for prefixed_useflag in v: + if extended_syntax and prefixed_useflag[-1] == :: + use_expand_prefix = prefixed_useflag[:-1].lower() + _ + continue + if prefixed_useflag[:1] == -: useflag = prefixed_useflag[1:] + prefixed_useflag = - + use_expand_prefix + useflag else: + prefixed_useflag = use_expand_prefix + prefixed_useflag useflag = prefixed_useflag if useflag_re.match(useflag) is None: writemsg(_(--- Invalid USE flag for '%s' in '%s': '%s'\n) % We could match useflag_re against (use_expand_prefix + useflag) here, so that the use_expand_prefix is also validated. Otherwise, the patch looks good. -- Thanks, Zac
[gentoo-portage-dev] [PATCH 0/7] Add FEATURES=binpkg-multi-instance (bug 150031)
FEATURES=binpkg-multi-instance causes an integer build-id to be
associated with each binary package instance. Inclusion of the build-id
in the file name of the binary package file makes it possible to store
an arbitrary number of binary packages built from the same ebuild.
Having multiple instances is useful for a number of purposes, such as
retaining builds that were built with different USE flags or linked
against different versions of libraries. The location of any particular
package within PKGDIR can be expressed as follows:
${PKGDIR}/${CATEGORY}/${PN}/${PF}-${BUILD_ID}.xpak
The build-id starts at 1 for the first build of a particular ebuild,
and is incremented by 1 for each new build. It is possible to share a
writable PKGDIR over NFS, and locking ensures that each package added
to PKGDIR will have a unique build-id. It is not necessary to migrate
an existing PKGDIR to the new layout, since portage is capable of
working with a mixed PKGDIR layout, where packages using the old layout
are allowed to remain in place.
The new PKGDIR layout is backward-compatible with binhost clients
running older portage, since the file format is identical, the
per-package PATH attribute in the 'Packages' index directs them to
download the file from the correct URI, and they automatically use
BUILD_TIME metadata to select the latest builds.
There is currently no automated way to prune old builds from PKGDIR,
although it is possible to remove packages manually, and then run
'emaint --fix binhost' to update the ${PKGDIR}/Packages index. Support
for FEATURES=binpkg-multi-instance is planned for eclean-pkg.
X-Gentoo-Bug: 150031
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=150031
Zac Medico (7):
binpkg-multi-instance 1 of 7
binpkg-multi-instance 2 of 7
binpkg-multi-instance 3 of 7
binpkg-multi-instance 4 of 7
binpkg-multi-instance 5 of 7
binpkg-multi-instance 6 of 7
binpkg-multi-instance 7 of 7
bin/quickpkg | 1 -
man/make.conf.5| 27 +
man/portage.5 | 8 +-
pym/_emerge/Binpkg.py | 33 +-
pym/_emerge/BinpkgFetcher.py | 13 +-
pym/_emerge/BinpkgVerifier.py | 6 +-
pym/_emerge/EbuildBinpkg.py| 9 +-
pym/_emerge/EbuildBuild.py | 36 +-
pym/_emerge/Package.py | 67 +-
pym/_emerge/Scheduler.py | 6 +-
pym/_emerge/clear_caches.py| 1 -
pym/_emerge/is_valid_package_atom.py | 5 +-
pym/_emerge/resolver/output.py | 21 +-
pym/portage/_sets/ProfilePackageSet.py | 3 +-
pym/portage/_sets/profiles.py | 3 +-
pym/portage/const.py | 2 +
pym/portage/dbapi/__init__.py | 10 +-
pym/portage/dbapi/bintree.py | 843 +++--
pym/portage/dbapi/vartree.py | 8 +-
pym/portage/dbapi/virtual.py | 113 ++-
pym/portage/dep/__init__.py| 35 +-
pym/portage/emaint/modules/binhost/binhost.py | 47 +-
.../package/ebuild/_config/KeywordsManager.py | 3 +-
.../package/ebuild/_config/LocationsManager.py | 8 +-
pym/portage/package/ebuild/_config/MaskManager.py | 21 +-
pym/portage/package/ebuild/_config/UseManager.py | 14 +-
pym/portage/package/ebuild/config.py | 15 +-
pym/portage/repository/config.py | 2 +-
pym/portage/tests/dep/test_isvalidatom.py | 8 +-
pym/portage/tests/resolver/ResolverPlayground.py | 25 +-
.../resolver/binpkg_multi_instance/__init__.py | 2 +
.../resolver/binpkg_multi_instance/__test__.py | 2 +
.../test_build_id_profile_format.py| 134
.../binpkg_multi_instance/test_rebuilt_binaries.py | 101 +++
pym/portage/util/__init__.py | 13 +-
pym/portage/versions.py| 28 +-
36 files changed, 1144 insertions(+), 529 deletions(-)
create mode 100644 pym/portage/tests/resolver/binpkg_multi_instance/__init__.py
create mode 100644 pym/portage/tests/resolver/binpkg_multi_instance/__test__.py
create mode 100644
pym/portage/tests/resolver/binpkg_multi_instance/test_build_id_profile_format.py
create mode 100644
pym/portage/tests/resolver/binpkg_multi_instance/test_rebuilt_binaries.py
--
2.0.5
[gentoo-portage-dev] [PATCH 1/7] binpkg-multi-instance 1 of 7
Extend the _pkg_str class with build_id, build_time, file_size, and
mtime attributes. These will be used to distinguish binary package
instances that have the same cpv. Package sorting accounts for
build_time, which will be used to prefer newer builds over older builds
when their versions are identical.
---
pym/_emerge/Package.py | 51 +-
pym/_emerge/resolver/output.py | 21 +
pym/portage/dbapi/__init__.py | 10 +++--
pym/portage/dbapi/vartree.py | 8 +--
pym/portage/versions.py| 28 +--
5 files changed, 92 insertions(+), 26 deletions(-)
diff --git a/pym/_emerge/Package.py b/pym/_emerge/Package.py
index e8a13cb..975335d 100644
--- a/pym/_emerge/Package.py
+++ b/pym/_emerge/Package.py
@@ -41,12 +41,12 @@ class Package(Task):
_validated_atoms, _visible)
metadata_keys = [
- BUILD_TIME, CHOST, COUNTER, DEPEND, EAPI,
- HDEPEND, INHERITED, IUSE, KEYWORDS,
- LICENSE, PDEPEND, PROVIDE, RDEPEND,
- repository, PROPERTIES, RESTRICT, SLOT, USE,
- _mtime_, DEFINED_PHASES, REQUIRED_USE, PROVIDES,
- REQUIRES]
+ BUILD_ID, BUILD_TIME, CHOST, COUNTER, DEFINED_PHASES,
+ DEPEND, EAPI, HDEPEND, INHERITED, IUSE, KEYWORDS,
+ LICENSE, MD5, PDEPEND, PROVIDE, PROVIDES,
+ RDEPEND, repository, REQUIRED_USE,
+ PROPERTIES, REQUIRES, RESTRICT, SIZE,
+ SLOT, USE, _mtime_]
_dep_keys = ('DEPEND', 'HDEPEND', 'PDEPEND', 'RDEPEND')
_buildtime_keys = ('DEPEND', 'HDEPEND')
@@ -114,13 +114,14 @@ class Package(Task):
return self._metadata[EAPI]
@property
+ def build_id(self):
+ return self.cpv.build_id
+
+ @property
def build_time(self):
if not self.built:
raise AttributeError('build_time')
- try:
- return long(self._metadata['BUILD_TIME'])
- except (KeyError, ValueError):
- return 0
+ return self.cpv.build_time
@property
def defined_phases(self):
@@ -509,9 +510,15 @@ class Package(Task):
else:
cpv_color = PKG_NOMERGE
+ build_id_str =
+ if isinstance(self.cpv.build_id, long) and self.cpv.build_id
0:
+ build_id_str = -%s % self.cpv.build_id
+
s = (%s, %s \
- % (portage.output.colorize(cpv_color, self.cpv +
_slot_separator + \
- self.slot + / + self.sub_slot + _repo_separator +
self.repo) , self.type_name)
+ % (portage.output.colorize(cpv_color, self.cpv +
+ build_id_str + _slot_separator + self.slot + / +
+ self.sub_slot + _repo_separator + self.repo),
+ self.type_name)
if self.type_name == installed:
if self.root_config.settings['ROOT'] != /:
@@ -755,29 +762,41 @@ class Package(Task):
def __lt__(self, other):
if other.cp != self.cp:
return self.cp other.cp
- if portage.vercmp(self.version, other.version) 0:
+ result = portage.vercmp(self.version, other.version)
+ if result 0:
return True
+ if result == 0 and self.built and other.built:
+ return self.build_time other.build_time
return False
def __le__(self, other):
if other.cp != self.cp:
return self.cp = other.cp
- if portage.vercmp(self.version, other.version) = 0:
+ result = portage.vercmp(self.version, other.version)
+ if result = 0:
return True
+ if result == 0 and self.built and other.built:
+ return self.build_time = other.build_time
return False
def __gt__(self, other):
if other.cp != self.cp:
return self.cp other.cp
- if portage.vercmp(self.version, other.version) 0:
+ result = portage.vercmp(self.version, other.version)
+ if result 0:
return True
+ if result == 0 and self.built and other.built:
+ return self.build_time other.build_time
return False
def __ge__(self, other):
if other.cp != self.cp:
return self.cp = other.cp
- if portage.vercmp(self.version, other.version) = 0:
+ result = portage.vercmp(self.version, other.version)
+ if result = 0:
return True
+ if
[gentoo-portage-dev] [PATCH 3/7] binpkg-multi-instance 3 of 7
FEATURES=binpkg-multi-instance causes an integer build-id to be
associated with each binary package instance. Inclusion of the build-id
in the file name of the binary package file makes it possible to store
an arbitrary number of binary packages built from the same ebuild.
Having multiple instances is useful for a number of purposes, such as
retaining builds that were built with different USE flags or linked
against different versions of libraries. The location of any particular
package within PKGDIR can be expressed as follows:
${PKGDIR}/${CATEGORY}/${PN}/${PF}-${BUILD_ID}.xpak
The build-id starts at 1 for the first build of a particular ebuild,
and is incremented by 1 for each new build. It is possible to share a
writable PKGDIR over NFS, and locking ensures that each package added
to PKGDIR will have a unique build-id. It is not necessary to migrate
an existing PKGDIR to the new layout, since portage is capable of
working with a mixed PKGDIR layout, where packages using the old layout
are allowed to remain in place.
The new PKGDIR layout is backward-compatible with binhost clients
running older portage, since the file format is identical, the
per-package PATH attribute in the 'Packages' index directs them to
download the file from the correct URI, and they automatically use
BUILD_TIME metadata to select the latest builds.
There is currently no automated way to prune old builds from PKGDIR,
although it is possible to remove packages manually, and then run
'emaint --fix binhost' to update the ${PKGDIR}/Packages index. Support
for FEATURES=binpkg-multi-instance is planned for eclean-pkg.
X-Gentoo-Bug: 150031
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=150031
---
bin/quickpkg | 1 -
man/make.conf.5 | 27 +
pym/_emerge/Binpkg.py | 33 +-
pym/_emerge/BinpkgFetcher.py | 13 +-
pym/_emerge/BinpkgVerifier.py | 6 +-
pym/_emerge/EbuildBinpkg.py | 9 +-
pym/_emerge/EbuildBuild.py| 36 +-
pym/_emerge/Package.py| 16 +-
pym/_emerge/Scheduler.py | 6 +-
pym/_emerge/clear_caches.py | 1 -
pym/portage/const.py | 2 +
pym/portage/dbapi/bintree.py | 684 +-
pym/portage/emaint/modules/binhost/binhost.py | 47 +-
13 files changed, 601 insertions(+), 280 deletions(-)
diff --git a/bin/quickpkg b/bin/quickpkg
index 2c69a69..8b71c3e 100755
--- a/bin/quickpkg
+++ b/bin/quickpkg
@@ -63,7 +63,6 @@ def quickpkg_atom(options, infos, arg, eout):
pkgs_for_arg = 0
for cpv in matches:
excluded_config_files = []
- bintree.prevent_collision(cpv)
dblnk = vardb._dblink(cpv)
have_lock = False
diff --git a/man/make.conf.5 b/man/make.conf.5
index 84b7191..6ead61b 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -256,6 +256,33 @@ has a \fB\-\-force\fR option that can be used to force
regeneration of digests.
Keep logs from successful binary package merges. This is relevant only when
\fBPORT_LOGDIR\fR is set.
.TP
+.B binpkg\-multi\-instance
+Enable support for multiple binary package instances per ebuild.
+Having multiple instances is useful for a number of purposes, such as
+retaining builds that were built with different USE flags or linked
+against different versions of libraries. The location of any particular
+package within PKGDIR can be expressed as follows:
+
+ ${PKGDIR}/${CATEGORY}/${PN}/${PF}\-${BUILD_ID}.xpak
+
+The build\-id starts at 1 for the first build of a particular ebuild,
+and is incremented by 1 for each new build. It is possible to share a
+writable PKGDIR over NFS, and locking ensures that each package added
+to PKGDIR will have a unique build\-id. It is not necessary to migrate
+an existing PKGDIR to the new layout, since portage is capable of
+working with a mixed PKGDIR layout, where packages using the old layout
+are allowed to remain in place.
+
+The new PKGDIR layout is backward\-compatible with binhost clients
+running older portage, since the file format is identical, the
+per\-package PATH attribute in the 'Packages' index directs them to
+download the file from the correct URI, and they automatically use
+BUILD_TIME metadata to select the latest builds.
+
+There is currently no automated way to prune old builds from PKGDIR,
+although it is possible to remove packages manually, and then run
+\(aqemaint \-\-fix binhost' to update the ${PKGDIR}/Packages index.
+.TP
.B buildpkg
Binary packages will be created for all packages that are merged. Also see
\fBquickpkg\fR(1) and \fBemerge\fR(1) \fB\-\-buildpkg\fR and
diff --git a/pym/_emerge/Binpkg.py b/pym/_emerge/Binpkg.py
index ded6dfd..7b7ae17 100644
--- a/pym/_emerge/Binpkg.py
+++ b/pym/_emerge/Binpkg.py
@@ -121,16 +121,11 @@ class
[gentoo-portage-dev] [PATCH 7/7] binpkg-multi-instance 7 of 7
Support profile-formats = build-id setting for layout.conf. When this is enabled in layout.conf of the containing repository, a dependency atom in the profile can refer to a specific build, using the build-id that is assigned when FEATURES=binpkg-multi-instance is enabled. A build-id atom is identical to a version-specific atom, except that the version is followed by a hyphen and an integer build-id. With the build-id profile format, it is possible to assemble a system using specific builds of binary packages, as users of binary distros might be accustomed to. For example, an atom in the packages file can pull a specific build of a package into the @system set, and an atom in the package.keywords file can be used to modify the effective KEYWORDS of a specific build of a package. Refering to specific builds can be useful for a number of reasons. For example, if a particular build needs to undergo a large amount of testing in a complex environment in order to verify reliability, then it can be useful to lock a profile to a specific build that has been thoroughly tested. --- This patch is identical to [PATCH 2/2] Add profile-formats=build-id (bug 150031) which was sent to the list earlier, except that it adds a comment about smart defaults in the Atom class. man/portage.5 | 8 +- pym/_emerge/is_valid_package_atom.py | 5 +- pym/portage/_sets/ProfilePackageSet.py | 3 +- pym/portage/_sets/profiles.py | 3 +- pym/portage/dep/__init__.py| 38 +- .../package/ebuild/_config/KeywordsManager.py | 3 +- .../package/ebuild/_config/LocationsManager.py | 8 +- pym/portage/package/ebuild/_config/MaskManager.py | 21 +++- pym/portage/package/ebuild/_config/UseManager.py | 14 ++- pym/portage/package/ebuild/config.py | 15 ++- pym/portage/repository/config.py | 2 +- pym/portage/tests/dep/test_isvalidatom.py | 8 +- .../test_build_id_profile_format.py| 134 + pym/portage/util/__init__.py | 13 +- 14 files changed, 237 insertions(+), 38 deletions(-) create mode 100644 pym/portage/tests/resolver/binpkg_multi_instance/test_build_id_profile_format.py diff --git a/man/portage.5 b/man/portage.5 index ed5140d..e062f9f 100644 --- a/man/portage.5 +++ b/man/portage.5 @@ -1180,7 +1180,7 @@ and the newer/faster md5-dict format. Default is to detect dirs. The EAPI to use for profiles when unspecified. This attribute is supported only if profile-default-eapi is included in profile-formats. .TP -.BR profile\-formats = [pms] [portage-1] [portage-2] [profile-bashrcs] [profile-set] [profile-default-eapi] +.BR profile\-formats = [pms] [portage-1] [portage-2] [profile-bashrcs] [profile-set] [profile-default-eapi] [build-id] Control functionality available to profiles in this repo such as which files may be dirs, or the syntax available in parent files. Use portage-2 if you're unsure. The default is portage-1-compat mode which is meant to be compatible @@ -1190,7 +1190,11 @@ Setting profile-bashrcs will enable the per-profile bashrc mechanism profile \fBpackages\fR file to add atoms to the @profile package set. See the profile \fBpackages\fR section for more information. Setting profile-default-eapi enables support for the -profile_eapi_when_unspecified attribute. +profile_eapi_when_unspecified attribute. Setting build\-id allows +dependency atoms in the profile to refer to specific builds (see the +binpkg\-multi\-instance FEATURES setting in \fBmake.conf\fR(5)). A +build\-id atom is identical to a version-specific atom, except that the +version is followed by a hyphen and an integer build\-id. .RE .RE diff --git a/pym/_emerge/is_valid_package_atom.py b/pym/_emerge/is_valid_package_atom.py index 112afc1..17f7642 100644 --- a/pym/_emerge/is_valid_package_atom.py +++ b/pym/_emerge/is_valid_package_atom.py @@ -14,9 +14,10 @@ def insert_category_into_atom(atom, category): ret = None return ret -def is_valid_package_atom(x, allow_repo=False): +def is_valid_package_atom(x, allow_repo=False, allow_build_id=True): if / not in x.split(:)[0]: x2 = insert_category_into_atom(x, 'cat') if x2 != None: x = x2 - return isvalidatom(x, allow_blockers=False, allow_repo=allow_repo) + return isvalidatom(x, allow_blockers=False, allow_repo=allow_repo, + allow_build_id=allow_build_id) diff --git a/pym/portage/_sets/ProfilePackageSet.py b/pym/portage/_sets/ProfilePackageSet.py index 2fcafb6..fec9373 100644 --- a/pym/portage/_sets/ProfilePackageSet.py +++ b/pym/portage/_sets/ProfilePackageSet.py @@ -23,7 +23,8 @@ class ProfilePackageSet(PackageSet): def load(self): self._setAtoms(x for x in stack_lists(
[gentoo-portage-dev] [PATCH 4/7] binpkg-multi-instance 4 of 7
Add a test case to verify that emerge --rebuilt-binaries works with
binpkg-multi-instance. This relies on the fact that binary packages of
the same version are ordered by BUILD_TIME, so that the latest builds
are preferred when appropriate.
---
pym/portage/tests/resolver/ResolverPlayground.py | 25 -
.../resolver/binpkg_multi_instance/__init__.py | 2 +
.../resolver/binpkg_multi_instance/__test__.py | 2 +
.../binpkg_multi_instance/test_rebuilt_binaries.py | 101 +
4 files changed, 126 insertions(+), 4 deletions(-)
create mode 100644 pym/portage/tests/resolver/binpkg_multi_instance/__init__.py
create mode 100644 pym/portage/tests/resolver/binpkg_multi_instance/__test__.py
create mode 100644
pym/portage/tests/resolver/binpkg_multi_instance/test_rebuilt_binaries.py
diff --git a/pym/portage/tests/resolver/ResolverPlayground.py
b/pym/portage/tests/resolver/ResolverPlayground.py
index 84ad17c..6bdf2c7 100644
--- a/pym/portage/tests/resolver/ResolverPlayground.py
+++ b/pym/portage/tests/resolver/ResolverPlayground.py
@@ -39,6 +39,7 @@ class ResolverPlayground(object):
config_files = frozenset((eapi, layout.conf, make.conf,
package.accept_keywords,
package.keywords, package.license, package.mask,
package.properties,
+ package.provided, packages,
package.unmask, package.use, package.use.aliases,
package.use.stable.mask,
soname.provided,
unpack_dependencies, use.aliases, use.force, use.mask,
layout.conf))
@@ -208,12 +209,18 @@ class ResolverPlayground(object):
raise AssertionError('digest creation failed
for %s' % ebuild_path)
def _create_binpkgs(self, binpkgs):
- for cpv, metadata in binpkgs.items():
+ # When using BUILD_ID, there can be mutiple instances for the
+ # same cpv. Therefore, binpkgs may be an iterable instead of
+ # a dict.
+ items = getattr(binpkgs, 'items', None)
+ items = items() if items is not None else binpkgs
+ for cpv, metadata in items:
a = Atom(= + cpv, allow_repo=True)
repo = a.repo
if repo is None:
repo = test_repo
+ pn = catsplit(a.cp)[1]
cat, pf = catsplit(a.cpv)
metadata = metadata.copy()
metadata.setdefault(SLOT, 0)
@@ -225,8 +232,13 @@ class ResolverPlayground(object):
repo_dir = self.pkgdir
category_dir = os.path.join(repo_dir, cat)
- binpkg_path = os.path.join(category_dir, pf + .tbz2)
- ensure_dirs(category_dir)
+ if BUILD_ID in metadata:
+ binpkg_path = os.path.join(category_dir, pn,
+ %s-%s.xpak% (pf,
metadata[BUILD_ID]))
+ else:
+ binpkg_path = os.path.join(category_dir, pf +
.tbz2)
+
+ ensure_dirs(os.path.dirname(binpkg_path))
t = portage.xpak.tbz2(binpkg_path)
t.recompose_mem(portage.xpak.xpak_mem(metadata))
@@ -252,6 +264,7 @@ class ResolverPlayground(object):
unknown_keys = set(metadata).difference(
portage.dbapi.dbapi._known_keys)
unknown_keys.discard(BUILD_TIME)
+ unknown_keys.discard(BUILD_ID)
unknown_keys.discard(COUNTER)
unknown_keys.discard(repository)
unknown_keys.discard(USE)
@@ -749,7 +762,11 @@ class ResolverPlaygroundResult(object):
repo_str =
if x.repo != test_repo:
repo_str = _repo_separator +
x.repo
- mergelist_str = x.cpv + repo_str
+ build_id_str =
+ if (x.type_name == binary and
+ x.cpv.build_id is not None):
+ build_id_str = -%s %
x.cpv.build_id
+ mergelist_str = x.cpv + build_id_str +
repo_str
if x.built:
if x.operation == merge:
desc = x.type_name
diff --git a/pym/portage/tests/resolver/binpkg_multi_instance/__init__.py
b/pym/portage/tests/resolver/binpkg_multi_instance/__init__.py
new file mode 100644
index 000..4725d33
--- /dev/null
+++
[gentoo-portage-dev] [PATCH 6/7] binpkg-multi-instance 6 of 7
Remove unused binarytree _remove_symlink, _create_symlink,
prevent_collision, _move_to_all, and _move_from_all methods. These are
all related to the oldest PKGDIR layout, which put all of the tbz2
files in $PKGDIR/All, and created symlinks to them in the category
directories. The $PKGDIR/All layout should be practically extinct by
now. Now portage recognizes all existing layouts, or mixtures of them,
and uses the old packages in place. It never puts new packages in
$PKGDIR/All, so there's no need to move packages around to prevent file
name collisions between packages from different categories. It also
only uses regular files (any symlinks are ignored).
---
pym/portage/dbapi/bintree.py | 117 ++-
1 file changed, 4 insertions(+), 113 deletions(-)
diff --git a/pym/portage/dbapi/bintree.py b/pym/portage/dbapi/bintree.py
index 5636a5f..a475fb5 100644
--- a/pym/portage/dbapi/bintree.py
+++ b/pym/portage/dbapi/bintree.py
@@ -471,89 +471,11 @@ class binarytree(object):
return moves
- def _remove_symlink(self, cpv):
- Remove a ${PKGDIR}/${CATEGORY}/${PF}.tbz2 symlink and also
remove
- the ${PKGDIR}/${CATEGORY} directory if empty. The file will
not be
- removed if os.path.islink() returns False.
- mycat, mypkg = catsplit(cpv)
- mylink = os.path.join(self.pkgdir, mycat, mypkg + .tbz2)
- if os.path.islink(mylink):
- Only remove it if it's really a link so that this
method never
- removes a real package that was placed here to avoid a
collision.
- os.unlink(mylink)
- try:
- os.rmdir(os.path.join(self.pkgdir, mycat))
- except OSError as e:
- if e.errno not in (errno.ENOENT,
- errno.ENOTEMPTY, errno.EEXIST):
- raise
- del e
-
- def _create_symlink(self, cpv):
- Create a ${PKGDIR}/${CATEGORY}/${PF}.tbz2 symlink (and
- ${PKGDIR}/${CATEGORY} directory, if necessary). Any file that
may
- exist in the location of the symlink will first be removed.
- mycat, mypkg = catsplit(cpv)
- full_path = os.path.join(self.pkgdir, mycat, mypkg + .tbz2)
- self._ensure_dir(os.path.dirname(full_path))
- try:
- os.unlink(full_path)
- except OSError as e:
- if e.errno != errno.ENOENT:
- raise
- del e
- os.symlink(os.path.join(.., All, mypkg + .tbz2),
full_path)
-
def prevent_collision(self, cpv):
- Make sure that the file location ${PKGDIR}/All/${PF}.tbz2 is
safe to
- use for a given cpv. If a collision will occur with an existing
- package from another category, the existing package will be
bumped to
- ${PKGDIR}/${CATEGORY}/${PF}.tbz2 so that both can coexist.
- if not self._all_directory:
- return
-
- # Copy group permissions for new directories that
- # may have been created.
- for path in (All, catsplit(cpv)[0]):
- path = os.path.join(self.pkgdir, path)
- self._ensure_dir(path)
- if not os.access(path, os.W_OK):
- raise PermissionDenied(access('%s', W_OK) %
path)
-
- full_path = self.getname(cpv)
- if All == full_path.split(os.path.sep)[-2]:
- return
- Move a colliding package if it exists. Code below this
point only
- executes in rare cases.
- mycat, mypkg = catsplit(cpv)
- myfile = mypkg + .tbz2
- mypath = os.path.join(All, myfile)
- dest_path = os.path.join(self.pkgdir, mypath)
-
- try:
- st = os.lstat(dest_path)
- except OSError:
- st = None
- else:
- if stat.S_ISLNK(st.st_mode):
- st = None
- try:
- os.unlink(dest_path)
- except OSError:
- if os.path.exists(dest_path):
- raise
-
- if st is not None:
- # For invalid packages, other_cat could be None.
- other_cat =
portage.xpak.tbz2(dest_path).getfile(bCATEGORY)
- if other_cat:
- other_cat = _unicode_decode(other_cat,
- encoding=_encodings['repo.content'],
Re: [gentoo-portage-dev] [PATCH] bin/eapi.sh: Invert condition in ___eapi_unpack_supports_absolute_paths().
On 02/16/2015 10:21 AM, Ulrich Müller wrote:
This should return true starting with EAPI 6, and false for EAPI 5 and
earlier.
---
bin/eapi.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/eapi.sh b/bin/eapi.sh
index f1c677e..528e6f2 100644
--- a/bin/eapi.sh
+++ b/bin/eapi.sh
@@ -175,7 +175,7 @@ ___eapi_unpack_is_case_sensitive() {
}
___eapi_unpack_supports_absolute_paths() {
- [[ ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend)$ ]]
}
___eapi_die_can_respect_nonfatal() {
LGTM.
Please go ahead and push this. We want to make sure it's fixed for the
next release, since it's a sort of regression.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH 1/2] Add FEATURES=binpkg-multi-instance (bug 150031)
On 02/17/2015 10:42 AM, Brian Dolbec wrote: overall, there is no way I know the code well enough to know if you screwed up. But the code looks decent, so... My only questions are: pym/portage/dbapi/bintree.py: You removed several functions from the binarytree class and essentially reduced prevent_collision to a warning message. Can you briefly say why they are not needed please. Okay, I'll include this info in an updated patch: _pkgindex_cpv_map_latest_build: This is what binhost clients running older versions of portage will use to select to select the latest builds when their binhost server switches to FEATURES=binpkg-multi-instance. New portage won't need this anymore because it is capable of examining multiple builds and it uses sorting to ensure that the latest builds are preferred when appropriate. _remove_symlink, _create_symlink, prevent_collision, _move_to_all, and _move_from_all: These are all related to the oldest PKGDIR layout, which put all of the tbz2 files in $PKGDIR/All, and created symlinks to them in the category directories. The $PKGDIR/All layout should be practically extinct by now. New portage recognizes all existing layouts, or mixtures of them, and uses the old packages in place. It never puts new packages in $PKGDIR/All, so there's no need to move packages around to prevent file name collisions between packages from different categories. It also only uses regular files (any symlinks are ignored). _populate() == it is some 500+ LOC nasty. From what I can see I think you added slightly more loc than you deleted/changed. While I don't expect a breakup of this function to be directly a part of this commit. IT IS BADLY NEEDED. I'd like to see this function properly split into pieces and act as a driver for the smaller tasks. Currently it is a mess of nested if/else, for,... that is extremely difficult to keep straight and make logic changes to. It's almost as bad as repomans 1000+ LOC main loop. Why if TRUE: == just fix the indent... At the same time, I noticed _ensure_dirs, _file_permissions functions defined in that class. Shouldn't these just be util functions available everywhere. Yeah, I will work on cleaning that up, and submit it as a separate patch. pym/portage/versions.py class _pkg_str(_unicode) in __init__() if build_time is not None: try: build_time = long(build_time) except ValueError: if build_time: build_time = -1 else: build_time = 0 if build_id is not None: try: build_id = long(build_id) except ValueError: if build_id: build_id = -1 else: build_id = None if file_size is not None: try: file_size = long(file_size) except ValueError: if file_size: file_size = -1 else: file_size = None if mtime is not None: try: mtime = long(mtime) except ValueError: if mtime: mtime = -1 else: mtime = None I hate repeating code. This can be done using one universal small function and will clean up the __init__() @static_method def _long(var, default): if var is not None: try: var = long(var) except ValueError: if var: var = -1 else: var = default return var Okay, I'll do that and post the updated patch. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 2/2] Add profile-formats=build-id (bug 150031)
On 02/17/2015 10:58 AM, Brian Dolbec wrote: class Atom() if allow_repo is None: allow_repo = True + if allow_build_id is None: + allow_build_id = True these can be written as allow_repo = allow_repo or True allow_build_id = allow_build_id or True Actually, your version behaves differently than mine for the case where False has been passed in for these parameters. The parameters are designed are provide a smart default, as long as the caller has not passed in an explicit True or False value. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] Add FEATURES=binpkg-multi-instance (bug 150031)
On 02/17/2015 11:56 AM, Brian Dolbec wrote: On Tue, 17 Feb 2015 11:26:27 -0800 Zac Medico zmed...@gentoo.org wrote: On 02/17/2015 10:42 AM, Brian Dolbec wrote: overall, there is no way I know the code well enough to know if you screwed up. But the code looks decent, so... My only questions are: pym/portage/dbapi/bintree.py: You removed several functions from the binarytree class and essentially reduced prevent_collision to a warning message. Can you briefly say why they are not needed please. Okay, I'll include this info in an updated patch: Actually, I think this one patch could be split into a few logical ones. Tag them binpkg-multi-instance 1 of... in the commit message so it is clear they belong together. That way the commit messages can more clearly be relevant to the file(s) changed. The commit message is already a short story in length ;) before adding these new explanations. Okay, will do. -- Thanks, Zac
[gentoo-portage-dev] [PATCH 1/7 v2] binpkg-multi-instance 1 of 7
Extend the _pkg_str class with build_id, build_time, file_size, and
mtime attributes. These will be used to distinguish binary package
instances that have the same cpv. Package sorting accounts for
build_time, which will be used to prefer newer builds over older builds
when their versions are identical.
---
[PATCH 1/7 v2] updates pkg_desc_index._pkg_node to have a build_time
attribute, which fixes an AttributeError raised from dbapi._cmp_cpv
for some emerge search actions.
pym/_emerge/Package.py| 51 +--
pym/_emerge/resolver/output.py| 21 ++---
pym/portage/cache/index/pkg_desc_index.py | 1 +
pym/portage/dbapi/__init__.py | 10 --
pym/portage/dbapi/vartree.py | 8 +++--
pym/portage/versions.py | 28 +++--
6 files changed, 93 insertions(+), 26 deletions(-)
diff --git a/pym/_emerge/Package.py b/pym/_emerge/Package.py
index e8a13cb..975335d 100644
--- a/pym/_emerge/Package.py
+++ b/pym/_emerge/Package.py
@@ -41,12 +41,12 @@ class Package(Task):
_validated_atoms, _visible)
metadata_keys = [
- BUILD_TIME, CHOST, COUNTER, DEPEND, EAPI,
- HDEPEND, INHERITED, IUSE, KEYWORDS,
- LICENSE, PDEPEND, PROVIDE, RDEPEND,
- repository, PROPERTIES, RESTRICT, SLOT, USE,
- _mtime_, DEFINED_PHASES, REQUIRED_USE, PROVIDES,
- REQUIRES]
+ BUILD_ID, BUILD_TIME, CHOST, COUNTER, DEFINED_PHASES,
+ DEPEND, EAPI, HDEPEND, INHERITED, IUSE, KEYWORDS,
+ LICENSE, MD5, PDEPEND, PROVIDE, PROVIDES,
+ RDEPEND, repository, REQUIRED_USE,
+ PROPERTIES, REQUIRES, RESTRICT, SIZE,
+ SLOT, USE, _mtime_]
_dep_keys = ('DEPEND', 'HDEPEND', 'PDEPEND', 'RDEPEND')
_buildtime_keys = ('DEPEND', 'HDEPEND')
@@ -114,13 +114,14 @@ class Package(Task):
return self._metadata[EAPI]
@property
+ def build_id(self):
+ return self.cpv.build_id
+
+ @property
def build_time(self):
if not self.built:
raise AttributeError('build_time')
- try:
- return long(self._metadata['BUILD_TIME'])
- except (KeyError, ValueError):
- return 0
+ return self.cpv.build_time
@property
def defined_phases(self):
@@ -509,9 +510,15 @@ class Package(Task):
else:
cpv_color = PKG_NOMERGE
+ build_id_str =
+ if isinstance(self.cpv.build_id, long) and self.cpv.build_id
0:
+ build_id_str = -%s % self.cpv.build_id
+
s = (%s, %s \
- % (portage.output.colorize(cpv_color, self.cpv +
_slot_separator + \
- self.slot + / + self.sub_slot + _repo_separator +
self.repo) , self.type_name)
+ % (portage.output.colorize(cpv_color, self.cpv +
+ build_id_str + _slot_separator + self.slot + / +
+ self.sub_slot + _repo_separator + self.repo),
+ self.type_name)
if self.type_name == installed:
if self.root_config.settings['ROOT'] != /:
@@ -755,29 +762,41 @@ class Package(Task):
def __lt__(self, other):
if other.cp != self.cp:
return self.cp other.cp
- if portage.vercmp(self.version, other.version) 0:
+ result = portage.vercmp(self.version, other.version)
+ if result 0:
return True
+ if result == 0 and self.built and other.built:
+ return self.build_time other.build_time
return False
def __le__(self, other):
if other.cp != self.cp:
return self.cp = other.cp
- if portage.vercmp(self.version, other.version) = 0:
+ result = portage.vercmp(self.version, other.version)
+ if result = 0:
return True
+ if result == 0 and self.built and other.built:
+ return self.build_time = other.build_time
return False
def __gt__(self, other):
if other.cp != self.cp:
return self.cp other.cp
- if portage.vercmp(self.version, other.version) 0:
+ result = portage.vercmp(self.version, other.version)
+ if result 0:
return True
+ if result == 0 and self.built and other.built:
+ return self.build_time other.build_time
return False
def __ge__(self, other):
if other.cp != self.cp:
[gentoo-portage-dev] [PATCH] binarytree: avoid unecessary index regeneration
Since commit f1c1b8a77eebf7713b32e5f9945690f60f4f46de, binarytree regenerates the 'Packages' index unnecessarily, due to missing REQUIRES and PROVIDES entries. These entries are not required, so use default empty values in order to avoid the unnecessary regeneration. Fixes: f1c1b8a77eeb (Generate soname dependency metadata (bug 282639)) --- pym/portage/dbapi/bintree.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pym/portage/dbapi/bintree.py b/pym/portage/dbapi/bintree.py index b98b26e..726a7a1 100644 --- a/pym/portage/dbapi/bintree.py +++ b/pym/portage/dbapi/bintree.py @@ -344,7 +344,9 @@ class binarytree(object): PDEPEND : , PROPERTIES : , PROVIDE : , + PROVIDES: , RDEPEND : , + REQUIRES: , RESTRICT: , SLOT: 0, USE : , -- 2.0.5
[gentoo-portage-dev] [PATCH 1/2] Add FEATURES=binpkg-multi-instance (bug 150031)
FEATURES=binpkg-multi-instance causes an integer build-id to be
associated with each binary package instance. Inclusion of the build-id
in the file name of the binary package file makes it possible to store
an arbitrary number of binary packages built from the same ebuild.
Having multiple instances is useful for a number of purposes, such as
retaining builds that were built with different USE flags or linked
against different versions of libraries. The location of any particular
package within PKGDIR can be expressed as follows:
${PKGDIR}/${CATEGORY}/${PN}/${PF}-${BUILD_ID}.xpak
The build-id starts at 1 for the first build of a particular ebuild,
and is incremented by 1 for each new build. It is possible to share a
writable PKGDIR over NFS, and locking ensures that each package added
to PKGDIR will have a unique build-id. It is not necessary to migrate
an existing PKGDIR to the new layout, since portage is capable of
working with a mixed PKGDIR layout, where packages using the old layout
are allowed to remain in place.
The new PKGDIR layout is backward-compatible with binhost clients
running older portage, since the file format is identical, the
per-package PATH attribute in the 'Packages' index directs them to
download the file from the correct URI, and they automatically use
BUILD_TIME metadata to select the latest builds.
There is currently no automated way to prune old builds from PKGDIR,
although it is possible to remove packages manually, and then run
'emaint --fix binhost' to update the ${PKGDIR}/Packages index.
It is not necessary to migrate an existing PKGDIR to the new layout,
since portage is capable of working with a mixed PKGDIR layout, where
packages using the old layout are allowed to remain in-place.
There is currently no automated way to prune old builds from PKGDIR,
although it is possible to remove packages manually, and then run
'emaint --fix binhost' update the ${PKGDIR}/Packages index. Support for
FEATURES=binpkg-multi-instance is planned for eclean-pkg.
X-Gentoo-Bug: 150031
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=150031
---
bin/quickpkg | 1 -
man/make.conf.5| 27 +
pym/_emerge/Binpkg.py | 33 +-
pym/_emerge/BinpkgFetcher.py | 13 +-
pym/_emerge/BinpkgVerifier.py | 6 +-
pym/_emerge/EbuildBinpkg.py| 9 +-
pym/_emerge/EbuildBuild.py | 36 +-
pym/_emerge/Package.py | 67 +-
pym/_emerge/Scheduler.py | 6 +-
pym/_emerge/clear_caches.py| 1 -
pym/_emerge/resolver/output.py | 21 +-
pym/portage/const.py | 2 +
pym/portage/dbapi/__init__.py | 10 +-
pym/portage/dbapi/bintree.py | 842 +++--
pym/portage/dbapi/vartree.py | 8 +-
pym/portage/dbapi/virtual.py | 113 ++-
pym/portage/emaint/modules/binhost/binhost.py | 47 +-
pym/portage/package/ebuild/config.py | 3 +-
pym/portage/tests/resolver/ResolverPlayground.py | 26 +-
.../resolver/binpkg_multi_instance/__init__.py | 2 +
.../resolver/binpkg_multi_instance/__test__.py | 2 +
.../binpkg_multi_instance/test_rebuilt_binaries.py | 101 +++
pym/portage/versions.py| 48 +-
23 files changed, 932 insertions(+), 492 deletions(-)
create mode 100644 pym/portage/tests/resolver/binpkg_multi_instance/__init__.py
create mode 100644 pym/portage/tests/resolver/binpkg_multi_instance/__test__.py
create mode 100644
pym/portage/tests/resolver/binpkg_multi_instance/test_rebuilt_binaries.py
diff --git a/bin/quickpkg b/bin/quickpkg
index 2c69a69..8b71c3e 100755
--- a/bin/quickpkg
+++ b/bin/quickpkg
@@ -63,7 +63,6 @@ def quickpkg_atom(options, infos, arg, eout):
pkgs_for_arg = 0
for cpv in matches:
excluded_config_files = []
- bintree.prevent_collision(cpv)
dblnk = vardb._dblink(cpv)
have_lock = False
diff --git a/man/make.conf.5 b/man/make.conf.5
index 84b7191..6ead61b 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -256,6 +256,33 @@ has a \fB\-\-force\fR option that can be used to force
regeneration of digests.
Keep logs from successful binary package merges. This is relevant only when
\fBPORT_LOGDIR\fR is set.
.TP
+.B binpkg\-multi\-instance
+Enable support for multiple binary package instances per ebuild.
+Having multiple instances is useful for a number of purposes, such as
+retaining builds that were built with different USE flags or linked
+against different versions of libraries. The location of any particular
+package within PKGDIR can be expressed as follows:
+
+ ${PKGDIR}/${CATEGORY}/${PN}/${PF}\-${BUILD_ID}.xpak
+
[gentoo-portage-dev] [PATCH 2/2] Add profile-formats=build-id (bug 150031)
When profile-formats = build-id is enabled in layout.conf of the containing repository, a dependency atom in the profile can refer to a specific build, using the build-id that is assigned when FEATURES=binpkg-multi-instance is enabled. A build-id atom is identical to a version-specific atom, except that the version is followed by a hyphen and an integer build-id. With the build-id profile format, it is possible to assemble a system using specific builds of binary packages, as users of binary distros might be accustomed to. For example, an atom in the packages file can pull a specific build of a package into the @system set, and an atom in the package.keywords file can be used to modify the effective KEYWORDS of a specific build of a package. Refering to specific builds can be useful for a number of reasons. For example, if a particular build needs to undergo a large amount of testing in a complex environment in order to verify reliability, then it can be useful to lock a profile to a specific build that has been thoroughly tested. X-Gentoo-Bug: 150031 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=150031 --- man/portage.5 | 8 +- pym/_emerge/is_valid_package_atom.py | 5 +- pym/portage/_sets/ProfilePackageSet.py | 3 +- pym/portage/_sets/profiles.py | 3 +- pym/portage/dep/__init__.py| 35 +- .../package/ebuild/_config/KeywordsManager.py | 3 +- .../package/ebuild/_config/LocationsManager.py | 8 +- pym/portage/package/ebuild/_config/MaskManager.py | 21 +++- pym/portage/package/ebuild/_config/UseManager.py | 14 ++- pym/portage/package/ebuild/config.py | 15 ++- pym/portage/repository/config.py | 2 +- pym/portage/tests/dep/test_isvalidatom.py | 8 +- .../test_build_id_profile_format.py| 134 + pym/portage/util/__init__.py | 13 +- 14 files changed, 234 insertions(+), 38 deletions(-) create mode 100644 pym/portage/tests/resolver/binpkg_multi_instance/test_build_id_profile_format.py diff --git a/man/portage.5 b/man/portage.5 index ed5140d..e062f9f 100644 --- a/man/portage.5 +++ b/man/portage.5 @@ -1180,7 +1180,7 @@ and the newer/faster md5-dict format. Default is to detect dirs. The EAPI to use for profiles when unspecified. This attribute is supported only if profile-default-eapi is included in profile-formats. .TP -.BR profile\-formats = [pms] [portage-1] [portage-2] [profile-bashrcs] [profile-set] [profile-default-eapi] +.BR profile\-formats = [pms] [portage-1] [portage-2] [profile-bashrcs] [profile-set] [profile-default-eapi] [build-id] Control functionality available to profiles in this repo such as which files may be dirs, or the syntax available in parent files. Use portage-2 if you're unsure. The default is portage-1-compat mode which is meant to be compatible @@ -1190,7 +1190,11 @@ Setting profile-bashrcs will enable the per-profile bashrc mechanism profile \fBpackages\fR file to add atoms to the @profile package set. See the profile \fBpackages\fR section for more information. Setting profile-default-eapi enables support for the -profile_eapi_when_unspecified attribute. +profile_eapi_when_unspecified attribute. Setting build\-id allows +dependency atoms in the profile to refer to specific builds (see the +binpkg\-multi\-instance FEATURES setting in \fBmake.conf\fR(5)). A +build\-id atom is identical to a version-specific atom, except that the +version is followed by a hyphen and an integer build\-id. .RE .RE diff --git a/pym/_emerge/is_valid_package_atom.py b/pym/_emerge/is_valid_package_atom.py index 112afc1..17f7642 100644 --- a/pym/_emerge/is_valid_package_atom.py +++ b/pym/_emerge/is_valid_package_atom.py @@ -14,9 +14,10 @@ def insert_category_into_atom(atom, category): ret = None return ret -def is_valid_package_atom(x, allow_repo=False): +def is_valid_package_atom(x, allow_repo=False, allow_build_id=True): if / not in x.split(:)[0]: x2 = insert_category_into_atom(x, 'cat') if x2 != None: x = x2 - return isvalidatom(x, allow_blockers=False, allow_repo=allow_repo) + return isvalidatom(x, allow_blockers=False, allow_repo=allow_repo, + allow_build_id=allow_build_id) diff --git a/pym/portage/_sets/ProfilePackageSet.py b/pym/portage/_sets/ProfilePackageSet.py index 2fcafb6..fec9373 100644 --- a/pym/portage/_sets/ProfilePackageSet.py +++ b/pym/portage/_sets/ProfilePackageSet.py @@ -23,7 +23,8 @@ class ProfilePackageSet(PackageSet): def load(self): self._setAtoms(x for x in stack_lists( [grabfile_package(os.path.join(y.location, packages), - verify_eapi=True, eapi=y.eapi, eapi_default=None) + verify_eapi=True,
[gentoo-portage-dev] [PATCH] Generate soname dependency metadata (282639)
Generate soname dependency metadata for binary and installed packages,
in the form of PROVIDES and REQUIRES metadata. It is useful to generate
PROVIDES and REQUIRES metadata now, so that it will be available
when dependency resolver support is added in the future. Note that
slot-operator dependencies will not be able to serve as a substitute
for soname dependencies for the forseeable future, because system
dependencies are frequently unspecified (according to Gentoo policy).
The PROVIDES/REQUIRES system is very similar to the automatic Requires
and Provides system which is supported by RPM. The PROVIDES/REQUIRES
metadata is generated automatically from the ELF files that are
installed by a package. The PROVIDES/REQUIRES syntax is described in
the /var/db/pkg section of the portage(5) man page. REQUIRES_EXCLUDE
and PROVIDES_EXCLUDE ebuild variables allow for filtering of the
sonames that are saved in REQUIRES and PROVIDES (see the ebuild(5) man
page for details).
The /var/db/pkg NEEDED.ELF.2 format now includes an additional field
which indicates the multilib category, as discussed in bug #534206. The
multilib category is used to categorize the sonames that are listed in
PROVIDES/REQUIRES metadata, since sonames need to be resolved
separately for each multilib category. The complete list of supported
multilib categories is documented in the comments of the
portage.dep.soname.multilib_category module.
X-Gentoo-Bug: 282639
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=282639
---
bin/ebuild.sh | 2 +-
bin/phase-functions.sh | 2 +-
man/ebuild.5| 12 +++
man/portage.5 | 25 +
pym/_emerge/Package.py | 3 +-
pym/portage/dbapi/bintree.py| 5 +-
pym/portage/dbapi/vartree.py| 1 +
pym/portage/dep/soname/__init__.py | 2 +
pym/portage/dep/soname/multilib_category.py | 112 +++
pym/portage/package/ebuild/doebuild.py | 86 --
pym/portage/util/_dyn_libs/LinkageMapELF.py | 61 ++---
pym/portage/util/_dyn_libs/NeededEntry.py | 83 +
pym/portage/util/_dyn_libs/soname_deps.py | 136
pym/portage/util/elf/__init__.py| 2 +
pym/portage/util/elf/constants.py | 36
pym/portage/util/elf/header.py | 62 +
pym/portage/util/endian/__init__.py | 2 +
pym/portage/util/endian/decode.py | 56
18 files changed, 661 insertions(+), 27 deletions(-)
create mode 100644 pym/portage/dep/soname/__init__.py
create mode 100644 pym/portage/dep/soname/multilib_category.py
create mode 100644 pym/portage/util/_dyn_libs/NeededEntry.py
create mode 100644 pym/portage/util/_dyn_libs/soname_deps.py
create mode 100644 pym/portage/util/elf/__init__.py
create mode 100644 pym/portage/util/elf/constants.py
create mode 100644 pym/portage/util/elf/header.py
create mode 100644 pym/portage/util/endian/__init__.py
create mode 100644 pym/portage/util/endian/decode.py
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index e6f9cb9..b6b3723 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -578,7 +578,7 @@ if ! has $EBUILD_PHASE clean cleanrm ; then
# interaction begins.
unset EAPI DEPEND RDEPEND PDEPEND HDEPEND INHERITED IUSE
REQUIRED_USE \
ECLASS E_IUSE E_REQUIRED_USE E_DEPEND E_RDEPEND
E_PDEPEND \
- E_HDEPEND
+ E_HDEPEND PROVIDES_EXCLUDE REQUIRES_EXCLUDE
if [[ $PORTAGE_DEBUG != 1 || ${-/x/} != $- ]] ; then
source $EBUILD || die error sourcing ebuild
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index aec86fd..def2080 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -580,7 +580,7 @@ __dyn_install() {
for f in ASFLAGS CBUILD CC CFLAGS CHOST CTARGET CXX \
CXXFLAGS EXTRA_ECONF EXTRA_EINSTALL EXTRA_MAKE \
LDFLAGS LIBCFLAGS LIBCXXFLAGS QA_CONFIGURE_OPTIONS \
- QA_DESKTOP_FILE ; do
+ QA_DESKTOP_FILE PROVIDES_EXCLUDE REQUIRES_EXCLUDE ; do
x=$(echo -n ${!f})
[[ -n $x ]] echo $x $f
done
diff --git a/man/ebuild.5 b/man/ebuild.5
index b587264..c2cbe4b 100644
--- a/man/ebuild.5
+++ b/man/ebuild.5
@@ -480,6 +480,12 @@ source source\-build which is scheduled for merge
.TE
.RE
.TP
+.B PROVIDES_EXCLUDE\fR = \fI[space delimited list of fnmatch patterns]\fR
+Sonames and file paths matched by these fnmatch patterns will be
+excluded during genertion of \fBPROVIDES\fR metadata (see
+\fBportage\fR(5)). Patterns are delimited by whitespace, and it is
+possible to create patterns containing quoted whitespace.
+.TP
.B PORTAGE_LOG_FILE
[gentoo-portage-dev] [PATCH v2] Generate soname dependency metadata (282639)
Generate soname dependency metadata for binary and installed packages,
in the form of PROVIDES and REQUIRES metadata. It is useful to generate
PROVIDES and REQUIRES metadata now, so that it will be available
when dependency resolver support is added in the future. Note that
slot-operator dependencies will not be able to serve as a substitute
for soname dependencies for the forseeable future, because system
dependencies are frequently unspecified (according to Gentoo policy).
The PROVIDES/REQUIRES system is very similar to the automatic Requires
and Provides system which is supported by RPM. The PROVIDES/REQUIRES
metadata is generated automatically from the ELF files that are
installed by a package. The PROVIDES/REQUIRES syntax is described in
the /var/db/pkg section of the portage(5) man page. REQUIRES_EXCLUDE
and PROVIDES_EXCLUDE ebuild variables allow for filtering of the
sonames that are saved in REQUIRES and PROVIDES (see the ebuild(5) man
page for details).
The /var/db/pkg NEEDED.ELF.2 format now includes an additional field
which indicates the multilib category, as discussed in bug #534206. The
multilib category is used to categorize the sonames that are listed in
PROVIDES/REQUIRES metadata, since sonames need to be resolved
separately for each multilib category. The complete list of supported
multilib categories is documented in the comments of the
portage.dep.soname.multilib_category module.
X-Gentoo-Bug: 282639
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=282639
---
PATCH v2 includes the following changes:
* no relative imports
* NeededEntry.__str__ now uses ;.join
* portage.util.elf.constants refers to elf.h from elfutils
* EM_ALPHA uses correct 0x9026 value (tested with gentoo stage3)
* portage.util.endian.decode uses struct.unpack_from
bin/ebuild.sh | 2 +-
bin/phase-functions.sh | 2 +-
man/ebuild.5| 12 +++
man/portage.5 | 25 +
pym/_emerge/Package.py | 3 +-
pym/portage/dbapi/bintree.py| 5 +-
pym/portage/dbapi/vartree.py| 1 +
pym/portage/dep/soname/__init__.py | 2 +
pym/portage/dep/soname/multilib_category.py | 112 +++
pym/portage/package/ebuild/doebuild.py | 86 --
pym/portage/util/_dyn_libs/LinkageMapELF.py | 61 ++---
pym/portage/util/_dyn_libs/NeededEntry.py | 82 +
pym/portage/util/_dyn_libs/soname_deps.py | 136
pym/portage/util/elf/__init__.py| 2 +
pym/portage/util/elf/constants.py | 39
pym/portage/util/elf/header.py | 62 +
pym/portage/util/endian/__init__.py | 2 +
pym/portage/util/endian/decode.py | 48 ++
18 files changed, 655 insertions(+), 27 deletions(-)
create mode 100644 pym/portage/dep/soname/__init__.py
create mode 100644 pym/portage/dep/soname/multilib_category.py
create mode 100644 pym/portage/util/_dyn_libs/NeededEntry.py
create mode 100644 pym/portage/util/_dyn_libs/soname_deps.py
create mode 100644 pym/portage/util/elf/__init__.py
create mode 100644 pym/portage/util/elf/constants.py
create mode 100644 pym/portage/util/elf/header.py
create mode 100644 pym/portage/util/endian/__init__.py
create mode 100644 pym/portage/util/endian/decode.py
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index e6f9cb9..b6b3723 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -578,7 +578,7 @@ if ! has $EBUILD_PHASE clean cleanrm ; then
# interaction begins.
unset EAPI DEPEND RDEPEND PDEPEND HDEPEND INHERITED IUSE
REQUIRED_USE \
ECLASS E_IUSE E_REQUIRED_USE E_DEPEND E_RDEPEND
E_PDEPEND \
- E_HDEPEND
+ E_HDEPEND PROVIDES_EXCLUDE REQUIRES_EXCLUDE
if [[ $PORTAGE_DEBUG != 1 || ${-/x/} != $- ]] ; then
source $EBUILD || die error sourcing ebuild
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index aec86fd..def2080 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -580,7 +580,7 @@ __dyn_install() {
for f in ASFLAGS CBUILD CC CFLAGS CHOST CTARGET CXX \
CXXFLAGS EXTRA_ECONF EXTRA_EINSTALL EXTRA_MAKE \
LDFLAGS LIBCFLAGS LIBCXXFLAGS QA_CONFIGURE_OPTIONS \
- QA_DESKTOP_FILE ; do
+ QA_DESKTOP_FILE PROVIDES_EXCLUDE REQUIRES_EXCLUDE ; do
x=$(echo -n ${!f})
[[ -n $x ]] echo $x $f
done
diff --git a/man/ebuild.5 b/man/ebuild.5
index b587264..c2cbe4b 100644
--- a/man/ebuild.5
+++ b/man/ebuild.5
@@ -480,6 +480,12 @@ source source\-build which is scheduled for merge
.TE
.RE
.TP
+.B PROVIDES_EXCLUDE\fR = \fI[space delimited list of fnmatch patterns]\fR
[gentoo-portage-dev] [PATCH] lockfile: handle EINTR for bug #538314
Use portage._eintr_func_wrapper to handle EINTR from fcntl.lockf. Since fcntl.lockf raises IOError, make _eintr_func_wrapper handle EnvironmentError (which both OSError and IOError inherit from). X-Gentoo-Bug: 538314 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=538314 --- pym/portage/__init__.py | 2 +- pym/portage/locks.py| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pym/portage/__init__.py b/pym/portage/__init__.py index d96d733..1c85042 100644 --- a/pym/portage/__init__.py +++ b/pym/portage/__init__.py @@ -337,7 +337,7 @@ class _eintr_func_wrapper(object): try: rval = self._func(*args, **kwargs) break - except OSError as e: + except EnvironmentError as e: if e.errno != errno.EINTR: raise diff --git a/pym/portage/locks.py b/pym/portage/locks.py index fdfe4a5..42ff1e3 100644 --- a/pym/portage/locks.py +++ b/pym/portage/locks.py @@ -146,7 +146,7 @@ def lockfile(mypath, wantnewlockfile=0, unlinkfile=0, # try for a non-blocking lock, if it's held, throw a message # we're waiting on lockfile and use a blocking attempt. - locking_method = _default_lock_fn + locking_method = portage._eintr_func_wrapper(_default_lock_fn) try: if __PORTAGE_TEST_HARDLINK_LOCKS in os.environ: raise IOError(errno.ENOSYS, Function not implemented) -- 2.0.5
Re: [gentoo-portage-dev] [PATCH] dispatch-conf: avoid symlink File exists error (535850)
On 01/11/2015 05:21 PM, Brian Dolbec wrote: On Sun, 11 Jan 2015 00:49:50 -0800 Zac Medico zmed...@gentoo.org wrote: Since commit f17448317166bfac42dc279b8795cd581c189582, an existing symlink in /etc/config-archive could trigger a fatal File exists error. Handle this by removing the destination file if it exists. This was not necessary when dispatch-conf only supported regular files, since shutil.copy2 would simply overwrite the regular destination file. Fixes: f17448317166 (dispatch-conf: symlink support for bug #485598) X-Gentoo-Bug: 535850 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=535850 --- pym/portage/dispatch_conf.py | 24 1 file changed, 24 insertions(+) diff --git a/pym/portage/dispatch_conf.py b/pym/portage/dispatch_conf.py index b27e68b..7d55182 100644 --- a/pym/portage/dispatch_conf.py +++ b/pym/portage/dispatch_conf.py @@ -179,6 +179,12 @@ def rcs_archive(archive, curconf, newconf, mrgconf): if curconf_st is not None and \ (stat.S_ISREG(curconf_st.st_mode) or stat.S_ISLNK(curconf_st.st_mode)): +# Remove destination file in order to ensure that the following +# symlink or copy2 call won't fail (see bug #535850). +try: +os.unlink(archive) +except OSError: +pass try: if stat.S_ISLNK(curconf_st.st_mode): os.symlink(os.readlink(curconf), archive) @@ -208,6 +214,12 @@ def rcs_archive(archive, curconf, newconf, mrgconf): if has_branch: os.rename(archive, archive + '.dist') +# Remove destination file in order to ensure that the following +# symlink or copy2 call won't fail (see bug #535850). +try: +os.unlink(archive) +except OSError: +pass try: if stat.S_ISLNK(mystat.st_mode): os.symlink(os.readlink(newconf), archive) @@ -264,6 +276,12 @@ def file_archive(archive, curconf, newconf, mrgconf): if curconf_st is not None and \ (stat.S_ISREG(curconf_st.st_mode) or stat.S_ISLNK(curconf_st.st_mode)): +# Remove destination file in order to ensure that the following +# symlink or copy2 call won't fail (see bug #535850). +try: +os.unlink(archive) +except OSError: +pass try: if stat.S_ISLNK(curconf_st.st_mode): os.symlink(os.readlink(curconf), archive) @@ -285,6 +303,12 @@ def file_archive(archive, curconf, newconf, mrgconf): stat.S_ISLNK(mystat.st_mode)): # Save off new config file in the archive dir with .dist.new suffix newconf_archive = archive + '.dist.new' +# Remove destination file in order to ensure that the following +# symlink or copy2 call won't fail (see bug #535850). +try: +os.unlink(newconf_archive) +except OSError: +pass try: if stat.S_ISLNK(mystat.st_mode): os.symlink(os.readlink(newconf), newconf_archive) That makes duplicated code in 4 places :/ I know it's only 4 lines of actual code plus 2 more for the same repeated comments. BUT... This diff doens't show if it's all in the same class. The changes affect 2 different functions (rcs_archive and file_archive), and add 2 unlink calls per function. Both function are very similar, but only one of them is called, depending on whether or not the user has rcs support enabled. If possible I'd prefer a small breakout function to do the unlink. Decorate it with @staticmethod even. If it's named right, you won't even need the comments repeated throughout the code. Just the docstring with that comment info. Maybe should just omit the comments and leave the unlink calls inline? It feels like overkill to split out a function for this. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] dispatch_conf: factor out _archive_copy
This eliminates 4 instances of duplicate code from the rcs_archive and
file_archive functions.
Suggested-by: Brian Dolbec dol...@gentoo.org
---
pym/portage/dispatch_conf.py | 92 +---
1 file changed, 36 insertions(+), 56 deletions(-)
diff --git a/pym/portage/dispatch_conf.py b/pym/portage/dispatch_conf.py
index 7d55182..790eacb 100644
--- a/pym/portage/dispatch_conf.py
+++ b/pym/portage/dispatch_conf.py
@@ -158,6 +158,38 @@ def read_config(mandatory_opts):
return opts
+def _archive_copy(src_st, src_path, dest_path):
+
+ Copy file from src_path to dest_path. Regular files and symlinks
+ are supported. If an EnvironmentError occurs, then it is logged
+ to stderr.
+
+ @param src_st: source file lstat result
+ @type src_st: posix.stat_result
+ @param src_path: source file path
+ @type src_path: str
+ @param dest_path: destination file path
+ @type dest_path: str
+
+ # Remove destination file in order to ensure that the following
+ # symlink or copy2 call won't fail (see bug #535850).
+ try:
+ os.unlink(dest_path)
+ except OSError:
+ pass
+ try:
+ if stat.S_ISLNK(src_st.st_mode):
+ os.symlink(os.readlink(src_path), dest_path)
+ else:
+ shutil.copy2(src_path, dest_path)
+ except EnvironmentError as e:
+ portage.util.writemsg(
+ _('dispatch-conf: Error copying %(src_path)s to '
+ '%(dest_path)s: %(reason)s\n') % {
+ src_path: src_path,
+ dest_path: dest_path,
+ reason: e
+ }, noiselevel=-1)
def rcs_archive(archive, curconf, newconf, mrgconf):
Archive existing config in rcs (on trunk). Then, if mrgconf is
@@ -179,20 +211,7 @@ def rcs_archive(archive, curconf, newconf, mrgconf):
if curconf_st is not None and \
(stat.S_ISREG(curconf_st.st_mode) or
stat.S_ISLNK(curconf_st.st_mode)):
- # Remove destination file in order to ensure that the following
- # symlink or copy2 call won't fail (see bug #535850).
- try:
- os.unlink(archive)
- except OSError:
- pass
- try:
- if stat.S_ISLNK(curconf_st.st_mode):
- os.symlink(os.readlink(curconf), archive)
- else:
- shutil.copy2(curconf, archive)
- except(IOError, os.error) as why:
- print(_('dispatch-conf: Error copying %(curconf)s to
%(archive)s: %(reason)s; fatal') % \
- {curconf: curconf, archive: archive,
reason: str(why)}, file=sys.stderr)
+ _archive_copy(curconf_st, curconf, archive)
if os.path.lexists(archive + ',v'):
os.system(RCS_LOCK + ' ' + archive)
@@ -214,20 +233,7 @@ def rcs_archive(archive, curconf, newconf, mrgconf):
if has_branch:
os.rename(archive, archive + '.dist')
- # Remove destination file in order to ensure that the following
- # symlink or copy2 call won't fail (see bug #535850).
- try:
- os.unlink(archive)
- except OSError:
- pass
- try:
- if stat.S_ISLNK(mystat.st_mode):
- os.symlink(os.readlink(newconf), archive)
- else:
- shutil.copy2(newconf, archive)
- except(IOError, os.error) as why:
- print(_('dispatch-conf: Error copying %(newconf)s to
%(archive)s: %(reason)s; fatal') % \
- {newconf: newconf, archive: archive,
reason: str(why)}, file=sys.stderr)
+ _archive_copy(mystat, newconf, archive)
if has_branch:
if mrgconf and os.path.isfile(archive) and \
@@ -276,20 +282,7 @@ def file_archive(archive, curconf, newconf, mrgconf):
if curconf_st is not None and \
(stat.S_ISREG(curconf_st.st_mode) or
stat.S_ISLNK(curconf_st.st_mode)):
- # Remove destination file in order to ensure that the following
- # symlink or copy2 call won't fail (see bug #535850).
- try:
- os.unlink(archive)
- except OSError:
- pass
- try:
- if stat.S_ISLNK(curconf_st.st_mode):
- os.symlink(os.readlink(curconf), archive)
- else:
- shutil.copy2(curconf, archive)
-
Re: [gentoo-portage-dev] [PATCH] Allow virtuals in package.provided
On 01/12/2015 01:34 AM, Michał Górny wrote: Dnia 2014-12-06, o godz. 22:31:42 Zac Medico zmed...@gentoo.org napisał(a): @Robin: This patch reverts the changes from bug 161003. Maybe this constraint is not needed anymore? Considering the feedback on the bug, ok to push this one finally? Yes, please do. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] Use _SUBMODULE_PATH_MAP keys for option choices
Fix commit c8a850685232b938cf0896fc2d80e72d57edad17 to re-use the keys
from _SUBMODULE_PATH_MAP for option choices.
Fixes: c8a850685232 (Add --sync-submodule glsa|news|profiles (534070))
Suggested-by: Brian Dolbec dol...@gentoo.org
---
pym/_emerge/main.py | 3 ++-
pym/portage/emaint/modules/sync/__init__.py | 4 +++-
pym/portage/sync/__init__.py| 7 +++
pym/portage/sync/syncbase.py| 7 +--
4 files changed, 13 insertions(+), 8 deletions(-)
diff --git a/pym/_emerge/main.py b/pym/_emerge/main.py
index 93c1a15..3fcfcbf 100644
--- a/pym/_emerge/main.py
+++ b/pym/_emerge/main.py
@@ -20,6 +20,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
)
from portage import os
from portage.util._argparse import ArgumentParser
+from portage.sync import _SUBMODULE_PATH_MAP
if sys.hexversion = 0x300:
long = int
@@ -636,7 +637,7 @@ def parse_opts(tmpcmdline, silent=False):
--sync-submodule: {
help: (Restrict sync to the specified
submodule(s).
(--sync action only)),
- choices : (glsa, news, profiles),
+ choices : tuple(_SUBMODULE_PATH_MAP),
action : append,
},
diff --git a/pym/portage/emaint/modules/sync/__init__.py
b/pym/portage/emaint/modules/sync/__init__.py
index 620861f..2fc8d34 100644
--- a/pym/portage/emaint/modules/sync/__init__.py
+++ b/pym/portage/emaint/modules/sync/__init__.py
@@ -1,6 +1,8 @@
# Copyright 2014-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
+from sync import _SUBMODULE_PATH_MAP
+
doc = Check repos.conf settings and sync repositories.
__doc__ = doc[:]
@@ -43,7 +45,7 @@ module_spec = {
long: --sync-submodule,
help: ((sync module only): Restrict
sync
to the specified
submodule(s)),
- choices: (glsa, news, profiles),
+ choices: tuple(_SUBMODULE_PATH_MAP),
action: append,
dest: sync_submodule,
},
diff --git a/pym/portage/sync/__init__.py b/pym/portage/sync/__init__.py
index 0f6c566..e01f993 100644
--- a/pym/portage/sync/__init__.py
+++ b/pym/portage/sync/__init__.py
@@ -3,10 +3,17 @@
import os
+from portage import OrderedDict
from portage.module import Modules
from portage.sync.controller import SyncManager
from portage.sync.config_checks import check_type
+_SUBMODULE_PATH_MAP = OrderedDict([
+ ('glsa', 'metadata/glsa'),
+ ('news', 'metadata/news'),
+ ('profiles', 'profiles'),
+])
+
path = os.path.join(os.path.dirname(__file__), modules)
# initial development debug info
#print(module path:, path)
diff --git a/pym/portage/sync/syncbase.py b/pym/portage/sync/syncbase.py
index 9506616..04db2d0 100644
--- a/pym/portage/sync/syncbase.py
+++ b/pym/portage/sync/syncbase.py
@@ -11,12 +11,7 @@ import os
import portage
from portage.util import writemsg_level
-
-_SUBMODULE_PATH_MAP = {
- 'glsa': 'metadata/glsa',
- 'news': 'metadata/news',
- 'profiles': 'profiles',
-}
+from . import _SUBMODULE_PATH_MAP
class SyncBase(object):
'''Base Sync class for subclassing'''
--
2.0.5
Re: [gentoo-portage-dev] [PATCH] Add --sync-submodule glsa|news|profiles (534070)
On 01/06/2015 08:53 PM, Brian Dolbec wrote:
On Wed, 31 Dec 2014 00:07:19 -0800
Zac Medico zmed...@gentoo.org wrote:
This adds support for a new --sync-submodule option to both emerge and
emaint. When this option is used with the sync action, only the
selected submodules are synced. Each submodule is referenced using an
abstract identifier, which serves to hide the implementation details
involving the precise locations of specific submodules within each
repository.
Currently, --sync-submodule has no effect for sync protocols other
than rsync, but the new SyncBase._get_submodule_paths() method will
be useful for implementing support in other SyncBase subclasses.
X-Gentoo-Bug: 534070
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=534070
---
portage.process.spawn(command, diff --git
a/pym/portage/sync/syncbase.py b/pym/portage/sync/syncbase.py index
94d4aab..fcde51f 100644 --- a/pym/portage/sync/syncbase.py
+++ b/pym/portage/sync/syncbase.py
@@ -12,6 +12,11 @@ import os
import portage
from portage.util import writemsg_level
+_SUBMODULE_PATH_MAP = {
+'glsa': 'metadata/glsa',
+'news': 'metadata/news',
+'profiles': 'profiles',
+}
It mostly looks good, but...
I don't like the idea of hard-coding this in the base class.
I think this should be configurable per-repo or at least per repo type.
The paths should be the same as long as the repo conforms to existing
conventions. If the repo doesn't conform to existing conventions, then
existing tools won't know where to look for the glsa, news, and profiles
directories.
This patch clears the way for other repo types to have sub-modules, but
will they be the same mapping for a git tree with those as
sub-modules? Maybe I'm just not following how the system works
properly...
Yes, they should use the same mappings, as explained above.
Alternatively...if the above is wrong/not feasible. In the
sync/__init__.py you have the choices hard-coded there independently of
these definitions above. I would prefer that one get it's options from
the other. Since syncbase.py is not gauranteed to be loaded, I would
think that the above be declared in the sync module's __init__.py and
imported or passed in here. Then the:
choices: (glsa, news, profiles),
could become:
choices: list(_SUBMODULE_PATH_MAP)
Then there is only one place to edit as options or code change.
That's a good idea. I'll make a patch for that.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v2] Add early check for broken /dev/s (bug 538980)
On 02/13/2015 10:11 AM, Alexander Berntsen wrote: Signed-off-by: Alexander Berntsen berna...@gentoo.org Suggested-by: Zac Medico zmed...@gentoo.org --- Add some output. Use the more backwards compatible OSError instead of FileNotFoundError. pym/_emerge/main.py | 14 ++ 1 file changed, 14 insertions(+) diff --git a/pym/_emerge/main.py b/pym/_emerge/main.py index 5d5e936..f1a9e18 100644 --- a/pym/_emerge/main.py +++ b/pym/_emerge/main.py @@ -1100,6 +1100,20 @@ def emerge_main(args=None): # loading to allow new repos with non-existent directories portage._sync_mode = True + # Verify that /dev/null exists and is a device file as a cheap early + # filter for obviously broken /dev/s. + try: + if os.stat(os.devnull).st_rdev == 0: + writemsg_level(Failed to validate a sane '/dev'.\n + '/dev/null' is not a device file.\n, + level=logging.ERROR, noiselevel=-1) + return 1 + except OSError: + writemsg_level(Failed to validate a sane '/dev'.\n + '/dev/null' does not exist.\n, + level=logging.ERROR, noiselevel=-1) + return 1 + # Portage needs to ensure a sane umask for the files it creates. os.umask(0o22) emerge_config = load_emerge_config( LGTM. We should also test that bash process substitution works, as noted in bug 538980. -- Thanks, Zac
[gentoo-portage-dev] [PATCH 2/2 v2] depgraph: soname dependency resolution (282639)
Soname dependency resolution is disabled by default, since it will not work correctly unless all available installed and binary packages have been built by a version of portage which generates REQUIRES and PROVIDES metadata. Soname dependency resolution is enabled when --ignore-soname-deps=n is specified, and one of the following is true: * --usepkgonly option is enabled * removal actions (--depclean and --prune) Soname dependencies are automatically ignored for dependency calculations that can pull unbuilt ebuilds into the dependency graph, since unbuilt ebuilds do not have any soname dependency metadata, making it impossible to determine whether an unresolved soname dependency can be satisfied. Therefore, --usepkgonly must be used in order to enable soname depedency resolution when installing packages. A new soname.provided file is supported for profiles, making it possible to selectively ignore soname dependencies (see the portage(5) man page). When soname dependency resolution is enabled, the soname dependencies are represented as SonameAtom instances which expose an interface that is minimally compatible with Atom instances. This allows both types of atoms to be satisfied using mostly the same mechanisms, with minimal use of conditional logic to handle the differences. Both atom classes have soname and package attributes that make it convenient for conditional code to distinguish package atoms and soname atoms. Both classes also implement a match method, so that it is possible to match a Package instance using identical syntax for both types of atoms. Since soname dependencies and slot-operator := dependencies share many properties, the slot-operator rebuild code has been generalized to handle both types of dependencies. Many of the existing unit tests involving slot-operator dependencies have been copied and adapted to test soname dependencies (the new tests are located in the pym/portage/tests/resolver/soname/ directory). X-Gentoo-Bug: 282639 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=282639 --- PATCH 2/2 v2 fixes depgraph._iter_match_pkgs_soname to yield packages in descending order. man/emerge.1 | 13 + man/portage.5 | 23 ++ pym/_emerge/FakeVartree.py | 16 +- pym/_emerge/Package.py | 55 +++- pym/_emerge/actions.py | 9 +- pym/_emerge/create_depgraph_params.py | 6 + pym/_emerge/create_world_atom.py | 6 +- pym/_emerge/depgraph.py| 347 +++-- pym/_emerge/main.py| 10 + pym/_emerge/resolver/DbapiProvidesIndex.py | 101 ++ pym/_emerge/resolver/output.py | 10 +- pym/_emerge/resolver/package_tracker.py| 42 ++- pym/_emerge/resolver/slot_collision.py | 35 ++- pym/portage/dbapi/DummyTree.py | 16 + pym/portage/dep/__init__.py| 25 ++ pym/portage/dep/soname/SonameAtom.py | 72 + pym/portage/dep/soname/parse.py| 47 +++ pym/portage/package/ebuild/config.py | 13 + pym/portage/tests/resolver/ResolverPlayground.py | 3 + pym/portage/tests/resolver/soname/__init__.py | 2 + pym/portage/tests/resolver/soname/__test__.py | 2 + .../tests/resolver/soname/test_autounmask.py | 103 ++ pym/portage/tests/resolver/soname/test_depclean.py | 61 .../tests/resolver/soname/test_downgrade.py| 240 ++ .../tests/resolver/soname/test_or_choices.py | 92 ++ .../tests/resolver/soname/test_reinstall.py| 87 ++ .../tests/resolver/soname/test_skip_update.py | 86 + .../soname/test_slot_conflict_reinstall.py | 342 .../resolver/soname/test_slot_conflict_update.py | 117 +++ .../tests/resolver/soname/test_soname_provided.py | 78 + .../tests/resolver/soname/test_unsatisfiable.py| 71 + .../tests/resolver/soname/test_unsatisfied.py | 87 ++ pym/portage/tests/resolver/test_package_tracker.py | 4 +- 33 files changed, 2099 insertions(+), 122 deletions(-) create mode 100644 pym/_emerge/resolver/DbapiProvidesIndex.py create mode 100644 pym/portage/dbapi/DummyTree.py create mode 100644 pym/portage/dep/soname/SonameAtom.py create mode 100644 pym/portage/dep/soname/parse.py create mode 100644 pym/portage/tests/resolver/soname/__init__.py create mode 100644 pym/portage/tests/resolver/soname/__test__.py create mode 100644 pym/portage/tests/resolver/soname/test_autounmask.py create mode 100644 pym/portage/tests/resolver/soname/test_depclean.py create mode 100644 pym/portage/tests/resolver/soname/test_downgrade.py create mode 100644 pym/portage/tests/resolver/soname/test_or_choices.py create mode 100644
Re: [gentoo-portage-dev] [PATCH] Add another check for broken /dev/s (bug 538980)
On 02/13/2015 10:39 AM, Alexander Berntsen wrote: Signed-off-by: Alexander Berntsen berna...@gentoo.org --- pym/_emerge/main.py | 9 + 1 file changed, 9 insertions(+) diff --git a/pym/_emerge/main.py b/pym/_emerge/main.py index f1a9e18..84094ae 100644 --- a/pym/_emerge/main.py +++ b/pym/_emerge/main.py @@ -1114,6 +1114,15 @@ def emerge_main(args=None): level=logging.ERROR, noiselevel=-1) return 1 + # Verify that BASH process substitution works as another cheap early + # filter. Process substitution uses '/dev/fd'. + if portage.process.spawn_bash([[ $( (echo foo) ) == foo ]]) != 0: + writemsg_level(Failed to validate a sane '/dev'.\n + bash process substitution doesn't work; this may be an + indication of a broken '/dev/fd'.\n, + level=logging.ERROR, noiselevel=-1) + return 1 + # Portage needs to ensure a sane umask for the files it creates. os.umask(0o22) emerge_config = load_emerge_config( LGTM. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] Fix random SonameOrChoicesTestCase failure
In commit 11467fc640995e3dc8897c82bbc7130d5bf27d05, the atom_not_selected variable was initialized outside of a loop, when it should have been re-initialized for each iteration of the loop. This caused atom_not_selected to have an incorrect value in some cases, triggering random failures of SonameOrChoicesTestCase, as observed here: https://travis-ci.org/gentoo/portage/jobs/50673911#L1114 Fixes: 11467fc64099 (depgraph: soname dependency resolution (bug 282639)) --- pym/_emerge/depgraph.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py index 63c89a4..e8a3110 100644 --- a/pym/_emerge/depgraph.py +++ b/pym/_emerge/depgraph.py @@ -1860,9 +1860,9 @@ class depgraph(object): replacement_candidates = [] # Set of all packages all atoms can agree on. all_candidate_pkgs = None - atom_not_selected = False for atom in atoms: + atom_not_selected = False if not atom.package: unevaluated_atom = None -- 2.0.5
Re: [gentoo-portage-dev] [PATCH 0/2] Implement soname dependencies
On 02/09/2015 08:40 AM, Alexander Berntsen wrote: On 09/02/15 06:41, Zac Medico wrote: 49 files changed, 2766 insertions(+), 149 deletions(-) So... is anyone anywhere near qualified to (N)ACK this? Sebastian Luther it the only other person who is really familiar with portage's dependency resolver. @Sebastian: Would you like to review my patch for soname dependency resolution? https://github.com/zmedico/portage/tree/binpkg-soname-deps -- Thanks, Zac
Re: [gentoo-portage-dev] Re: [PATCH 2/2 v2] depgraph: soname dependency resolution (282639)
On 02/13/2015 07:03 PM, Duncan wrote: Zac Medico posted on Fri, 13 Feb 2015 10:35:09 -0800 as excerpted: Soname dependency resolution is enabled when --ignore-soname-deps=n is specified, and [...] Can that option possibly be renamed? Ignore is functionally a negative, as in don't consider. As a result, --ignore-sonames-deps=n is a double-negative that actually enables something, and it becomes rather difficult to reason about what you're actually telling portage to do. Unfortunately, I'm confused enough myself that I don't have any suggestions for any better option name. =:^( Well, the ignore prefix is there to express that something is being ignored. In a couple of years, we'll be able to enable soname dependency resolution by default, and the --ignore-soname-deps=y option will be there for people to use in rare cases where it might be helpful (like on outdated systems where soname metadata isn't available for installed packages). In case there's any doubt about how much trouble English speakers often have with multiple negation, I follow a linguists blog called LanguageLog, that regularly features examples of misnegation, in which all sorts of people have ended up saying the opposite of what they obviously intended because of an incorrect number of negations. People really do have trouble sorting it out, and some of the examples are actually quite humorous. Here's a google on misnegation. At least from here, the top three hits are Language log, with #3 being a big list of posts on the topic. https://www.google.com/search?q=misnegationie=UTF-8 And here's one of the more amusing ones, a billboard at a(n apparently UK) petrol/gas station: http://languagelog.ldc.upenn.edu/nll/?p=14927 (FWIW, I recently emailed the author of a popular FLOSS community standards blog about a misnegation as well. He fixed it, and sent me a nice reply/thanks.) So, umm... Let's come up with some other option name here, one that doesn't invoke multiple negation to turn ON a function, and thus hurt to think about, if at all possible. =:^/ Tho arguably with the double negation turning something ON, at least we're not misnegating here. It's confusing and hard to think about, but AFAICT, logically correct. =:^) Of course that doesn't mean people can actually /use/ it correctly, thus the problem. =:^\ I see what you mean, but I still think having ignore in then name is a good way to express the meaning. It's similar to the --nodeps option, which also begins with a negative. -- Thanks, Zac
Re: [gentoo-portage-dev] custom profiles?
On 03/14/2015 11:41 AM, Joakim Tjernlund wrote:
On Sat, 2015-03-14 at 11:08 -0700, Zac Medico wrote:
On 03/14/2015 09:12 AM, Joakim Tjernlund wrote:
On Wed, 2015-03-11 at 13:43 -0700, Zac Medico wrote:
On 03/11/2015 01:16 PM, Joakim Tjernlund wrote:
On Wed, 2015-03-11 at 10:56 -0700, Zac Medico wrote:
On 03/11/2015 08:48 AM, Joakim Tjernlund wrote:
On Sun, 2015-03-08 at 11:58 -0700, Zac Medico wrote:
On 03/08/2015 10:01 AM, Joakim Tjernlund wrote:
On Sun, 2015-03-08 at 15:47 +, Joakim Tjernlund wrote:
package.use/package.use.force is a bit different though:
cat /etc/portage/package.use/qemu
app-emulation/qemu vde -alsa -pulseaudio -bluetooth -opengl
qemu_user_targets_x86_64 xattr
virtfs
static-
user
#Needed by static-user
sys-libs/zlib static-libs
dev-libs/glib static-libs
sys-apps/attr static-libs
Moving this to package.use/package.use.force does not respect -alsa,
-pulseaudio, -opengl
all
flags which has a - on them, emerge wants to turn them on again.
Am I missing something?
Using portage 2.2.18
Appears one have to use package.use.mask for that.
cat package.use.mask
app-emulation/qemu alsa pulseaudio bluetooth opengl
It would be handy if one could use the same syntax as in
/etc/portage/package.use/qemu(-alsa -
opengl
etc.)
Jocke
Yes, the inverted use.mask logic can be confusing if you are not
familiar with it. The negative
flags
have a
special meaning within the context of of portage's incremental
stacking behavior, so they can
still
be
useful, though not in the same way that you you attempted to use them.
Just noticed that USE flags in profiles/package.use.mask override
everything so this
USE=thin emerge -av sys-fs/lvm2
will not turn on thin if thin is in profiles/package.use.mask
How can just change the default so a user can easily turn it on ?
Generally, setting the USE environment variable like that is poor practice,
because the setting will not
persist the next time that you rebuild the package. So, you should set the
flag in
I know, this was just an example to illustrate that it did not work.
/etc/portage/package.use. You can unmask the flag for lvm2 like this:
echo sys-fs/lvm2 -thin /etc/portage/profile/package.use.mask
You misunderstand, I have sys-fs/lvm2 thin in
/etc/portage/profile/package.use.mask
and I want a user to able to override this setting, using USE=.. or adding
it
to their local /etc/portage/package.use file/dir
There's no other way to negate a use mask than to use
/etc/portage/profile/{use.mask,package.use.mask} as I have described. I
don't think that it makes much sense to negate a use mask for a single
emerge invocation. If you want to do that, then why is the use flag
masked anyway?
--
Thanks,
Zac
Re: [gentoo-portage-dev] custom profiles?
On 03/14/2015 12:30 PM, Joakim Tjernlund wrote:
On Sat, 2015-03-14 at 11:57 -0700, Zac Medico wrote:
On 03/14/2015 11:41 AM, Joakim Tjernlund wrote:
On Sat, 2015-03-14 at 11:08 -0700, Zac Medico wrote:
On 03/14/2015 09:12 AM, Joakim Tjernlund wrote:
On Wed, 2015-03-11 at 13:43 -0700, Zac Medico wrote:
On 03/11/2015 01:16 PM, Joakim Tjernlund wrote:
On Wed, 2015-03-11 at 10:56 -0700, Zac Medico wrote:
On 03/11/2015 08:48 AM, Joakim Tjernlund wrote:
On Sun, 2015-03-08 at 11:58 -0700, Zac Medico wrote:
On 03/08/2015 10:01 AM, Joakim Tjernlund wrote:
On Sun, 2015-03-08 at 15:47 +, Joakim Tjernlund wrote:
package.use/package.use.force is a bit different though:
cat /etc/portage/package.use/qemu
app-emulation/qemu vde -alsa -pulseaudio -bluetooth -opengl
qemu_user_targets_x86_64
xattr
virtfs
static-
user
#Needed by static-user
sys-libs/zlib static-libs
dev-libs/glib static-libs
sys-apps/attr static-libs
Moving this to package.use/package.use.force does not respect
-alsa, -pulseaudio, -
opengl
all
flags which has a - on them, emerge wants to turn them on again.
Am I missing something?
Using portage 2.2.18
Appears one have to use package.use.mask for that.
cat package.use.mask
app-emulation/qemu alsa pulseaudio bluetooth opengl
It would be handy if one could use the same syntax as in
/etc/portage/package.use/qemu(-
alsa -
opengl
etc.)
Jocke
Yes, the inverted use.mask logic can be confusing if you are not
familiar with it. The
negative
flags
have a
special meaning within the context of of portage's incremental
stacking behavior, so they
can
still
be
useful, though not in the same way that you you attempted to use
them.
Just noticed that USE flags in profiles/package.use.mask override
everything so this
USE=thin emerge -av sys-fs/lvm2
will not turn on thin if thin is in profiles/package.use.mask
How can just change the default so a user can easily turn it on ?
Generally, setting the USE environment variable like that is poor
practice, because the setting will
not
persist the next time that you rebuild the package. So, you should set the
flag in
I know, this was just an example to illustrate that it did not work.
/etc/portage/package.use. You can unmask the flag for lvm2 like this:
echo sys-fs/lvm2 -thin /etc/portage/profile/package.use.mask
You misunderstand, I have sys-fs/lvm2 thin in
/etc/portage/profile/package.use.mask
and I want a user to able to override this setting, using USE=.. or
adding it
to their local /etc/portage/package.use file/dir
There's no other way to negate a use mask than to use
/etc/portage/profile/{use.mask,package.use.mask} as I
have described. I don't think that it makes much sense to negate a use mask
for a single emerge invocation.
If you want to do that, then why is the use flag masked anyway?
I am putting tougher a profile for my company where I want to specify default
USE flags for different apps/libs. Then a user who knows what he/she is doing
should be able to override these defaults. It is not possible as far
as I can tell to override negative USE flags or is there?
You can set default USE flags in the profile, and then the users can
override those settings locally (both positively and negatively). You
should not be using use.mask at all here. The profile can set
USE=-flag in make.defaults, or in packages.use, and the user can
override that without having to mess with use.mask.
--
Thanks,
Zac
[gentoo-portage-dev] [PATCH v2] _post_src_install_soname_symlinks: fix bug 543818
The SonameDepsProcessor.add() method raises AssertionError if the
multilib category of an ELF file is not recognized. It's not possible
to account for soname dependencies in this case (the file is probably
intended for a foreign architecture), so avoid the AssertionError and
generate an eqawarn message for this case. The eqawarn message is
suppressed for files matched by the QA_PREBUILT variable.
X-Gentoo-Bug: 543818
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=543818
---
PATCH v2 fixes inverted logic for the QA Notice, so that warnings will
be correctly suppressed for files matched by the QA_PREBUILT variable.
bin/phase-functions.sh | 2 +-
pym/portage/package/ebuild/doebuild.py | 59 +++---
2 files changed, 42 insertions(+), 19 deletions(-)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index def2080..2743e27 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -580,7 +580,7 @@ __dyn_install() {
for f in ASFLAGS CBUILD CC CFLAGS CHOST CTARGET CXX \
CXXFLAGS EXTRA_ECONF EXTRA_EINSTALL EXTRA_MAKE \
LDFLAGS LIBCFLAGS LIBCXXFLAGS QA_CONFIGURE_OPTIONS \
- QA_DESKTOP_FILE PROVIDES_EXCLUDE REQUIRES_EXCLUDE ; do
+ QA_DESKTOP_FILE QA_PREBUILT PROVIDES_EXCLUDE
REQUIRES_EXCLUDE ; do
x=$(echo -n ${!f})
[[ -n $x ]] echo $x $f
done
diff --git a/pym/portage/package/ebuild/doebuild.py
b/pym/portage/package/ebuild/doebuild.py
index 94785b5..3c31eba 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -8,6 +8,7 @@ __all__ = ['doebuild', 'doebuild_environment', 'spawn',
'spawnebuild']
import grp
import gzip
import errno
+import fnmatch
import io
from itertools import chain
import logging
@@ -2209,24 +2210,29 @@ def _post_src_install_soname_symlinks(mysettings, out):
if f is not None:
f.close()
- qa_no_symlink =
- f = None
- try:
- f = io.open(_unicode_encode(os.path.join(
- mysettings[PORTAGE_BUILDDIR],
- build-info, QA_SONAME_NO_SYMLINK),
- encoding=_encodings['fs'], errors='strict'),
- mode='r', encoding=_encodings['repo.content'],
- errors='replace')
- qa_no_symlink = f.read()
- except IOError as e:
- if e.errno not in (errno.ENOENT, errno.ESTALE):
- raise
- finally:
- if f is not None:
- f.close()
+ metadata = {}
+ for k in (QA_PREBUILT, QA_NO_SYMLINK):
+ try:
+ with io.open(_unicode_encode(os.path.join(
+ mysettings[PORTAGE_BUILDDIR],
+ build-info, k),
+ encoding=_encodings['fs'], errors='strict'),
+ mode='r', encoding=_encodings['repo.content'],
+ errors='replace') as f:
+ v = f.read()
+ except IOError as e:
+ if e.errno not in (errno.ENOENT, errno.ESTALE):
+ raise
+ else:
+ metadata[k] = v
+
+ qa_prebuilt = metadata.get(QA_PREBUILT, ).strip()
+ if qa_prebuilt:
+ qa_prebuilt = re.compile(|.join(
+ fnmatch.translate(x.lstrip(os.sep))
+ for x in portage.util.shlex_split(qa_prebuilt)))
- qa_no_symlink = qa_no_symlink.split()
+ qa_no_symlink = metadata.get(QA_NO_SYMLINK, ).split()
if qa_no_symlink:
if len(qa_no_symlink) 1:
qa_no_symlink = |.join((%s) % x for x in
qa_no_symlink)
@@ -2297,6 +2303,7 @@ def _post_src_install_soname_symlinks(mysettings, out):
requires_exclude =
missing_symlinks = []
+ unrecognized_elf_files = []
soname_deps = SonameDepsProcessor(
provides_exclude, requires_exclude)
@@ -2326,7 +2333,14 @@ def _post_src_install_soname_symlinks(mysettings, out):
entry.multilib_category = compute_multilib_category(elf_header)
needed_file.write(_unicode(entry))
- soname_deps.add(entry)
+ if entry.multilib_category is None:
+ if qa_prebuilt.match(
+
entry.filename[len(mysettings[EPREFIX]):].lstrip(
+ os.sep)) is None:
+ unrecognized_elf_files.append(entry)
+ else:
+ soname_deps.add(entry)
+
obj = entry.filename
soname = entry.soname
@@ -2365,6 +2379,15 @@ def
[gentoo-portage-dev] [PATCH v3] _post_src_install_soname_symlinks: fix bug 543818
The SonameDepsProcessor.add() method raises AssertionError if the
multilib category of an ELF file is not recognized. It's not possible
to account for soname dependencies in this case (the file is probably
intended for a foreign architecture), so avoid the AssertionError and
generate an eqawarn message for this case. The eqawarn message is
suppressed for files matched by the QA_PREBUILT variable.
X-Gentoo-Bug: 543818
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=543818
---
PATCH v3 fixes an AttributeError which could occur if the QA_PREBUILT
variable was undefined.
bin/phase-functions.sh | 2 +-
pym/portage/package/ebuild/doebuild.py | 59 +++---
2 files changed, 42 insertions(+), 19 deletions(-)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index def2080..2743e27 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -580,7 +580,7 @@ __dyn_install() {
for f in ASFLAGS CBUILD CC CFLAGS CHOST CTARGET CXX \
CXXFLAGS EXTRA_ECONF EXTRA_EINSTALL EXTRA_MAKE \
LDFLAGS LIBCFLAGS LIBCXXFLAGS QA_CONFIGURE_OPTIONS \
- QA_DESKTOP_FILE PROVIDES_EXCLUDE REQUIRES_EXCLUDE ; do
+ QA_DESKTOP_FILE QA_PREBUILT PROVIDES_EXCLUDE
REQUIRES_EXCLUDE ; do
x=$(echo -n ${!f})
[[ -n $x ]] echo $x $f
done
diff --git a/pym/portage/package/ebuild/doebuild.py
b/pym/portage/package/ebuild/doebuild.py
index 94785b5..1be83ad 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -8,6 +8,7 @@ __all__ = ['doebuild', 'doebuild_environment', 'spawn',
'spawnebuild']
import grp
import gzip
import errno
+import fnmatch
import io
from itertools import chain
import logging
@@ -2209,24 +2210,29 @@ def _post_src_install_soname_symlinks(mysettings, out):
if f is not None:
f.close()
- qa_no_symlink =
- f = None
- try:
- f = io.open(_unicode_encode(os.path.join(
- mysettings[PORTAGE_BUILDDIR],
- build-info, QA_SONAME_NO_SYMLINK),
- encoding=_encodings['fs'], errors='strict'),
- mode='r', encoding=_encodings['repo.content'],
- errors='replace')
- qa_no_symlink = f.read()
- except IOError as e:
- if e.errno not in (errno.ENOENT, errno.ESTALE):
- raise
- finally:
- if f is not None:
- f.close()
+ metadata = {}
+ for k in (QA_PREBUILT, QA_NO_SYMLINK):
+ try:
+ with io.open(_unicode_encode(os.path.join(
+ mysettings[PORTAGE_BUILDDIR],
+ build-info, k),
+ encoding=_encodings['fs'], errors='strict'),
+ mode='r', encoding=_encodings['repo.content'],
+ errors='replace') as f:
+ v = f.read()
+ except IOError as e:
+ if e.errno not in (errno.ENOENT, errno.ESTALE):
+ raise
+ else:
+ metadata[k] = v
+
+ qa_prebuilt = metadata.get(QA_PREBUILT, ).strip()
+ if qa_prebuilt:
+ qa_prebuilt = re.compile(|.join(
+ fnmatch.translate(x.lstrip(os.sep))
+ for x in portage.util.shlex_split(qa_prebuilt)))
- qa_no_symlink = qa_no_symlink.split()
+ qa_no_symlink = metadata.get(QA_NO_SYMLINK, ).split()
if qa_no_symlink:
if len(qa_no_symlink) 1:
qa_no_symlink = |.join((%s) % x for x in
qa_no_symlink)
@@ -2297,6 +2303,7 @@ def _post_src_install_soname_symlinks(mysettings, out):
requires_exclude =
missing_symlinks = []
+ unrecognized_elf_files = []
soname_deps = SonameDepsProcessor(
provides_exclude, requires_exclude)
@@ -2326,7 +2333,14 @@ def _post_src_install_soname_symlinks(mysettings, out):
entry.multilib_category = compute_multilib_category(elf_header)
needed_file.write(_unicode(entry))
- soname_deps.add(entry)
+ if entry.multilib_category is None:
+ if not qa_prebuilt or qa_prebuilt.match(
+
entry.filename[len(mysettings[EPREFIX]):].lstrip(
+ os.sep)) is None:
+ unrecognized_elf_files.append(entry)
+ else:
+ soname_deps.add(entry)
+
obj = entry.filename
soname = entry.soname
@@ -2365,6 +2379,15 @@ def
Re: [gentoo-portage-dev] custom profiles?
On 03/16/2015 09:31 AM, Joakim Tjernlund wrote: On Sun, 2015-03-15 at 16:37 +, Joakim Tjernlund wrote: On Sat, 2015-03-14 at 14:02 -0700, Zac Medico wrote: You can set default USE flags in the profile, and then the users can override those settings locally (both positively and negatively). You should not be using use.mask at all here. The profile can set USE=-flag in make.defaults, or in packages.use, and the user can override that without having to mess with use.mask. That is how I started, I added -thin etc. to profile's package.use and it didn't work so I moved over to package.mask. Now when I try with -thin again in profile's package.use it actually works! I have no idea what got wrong the first time. Anyhow, thank you for your patience. I will continue moving over all Transmode stuff to our new profile the next few days :) hmm, spoke too fast. Negative USE flags in profiles package.use works on other computers but mine :-( I have in profiles package.use: app-emulation/qemu usb usbredir vde qemu_user_targets_x86_64 xattr virtfs static-user -alsa -pulseaudio -bluetooth -opengl but on my computer the -USE flags(-alsa -pulseaudio -bluetooth -opengl) have no effect. I have been looking high and low what could case this but no luck, any pointers ? Maybe you have overridden the default USE_ORDER setting? You can use this command to check it: portageq envvar USE_ORDER -- Thanks, Zac
[gentoo-portage-dev] [PATCH] _post_src_install_soname_symlinks: fix bug 543818
The SonameDepsProcessor.add() method raises AssertionError if the
multilib category of an ELF file is not recognized. It's not possible
to account for soname dependencies in this case (the file is probably
intended for a foreign architecture), so avoid the AssertionError and
generate an eqawarn message for this case. The eqawarn message is
suppressed for files matched by the QA_PREBUILT variable.
X-Gentoo-Bug: 543818
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=543818
---
bin/phase-functions.sh | 2 +-
pym/portage/package/ebuild/doebuild.py | 59 +++---
2 files changed, 42 insertions(+), 19 deletions(-)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index def2080..2743e27 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -580,7 +580,7 @@ __dyn_install() {
for f in ASFLAGS CBUILD CC CFLAGS CHOST CTARGET CXX \
CXXFLAGS EXTRA_ECONF EXTRA_EINSTALL EXTRA_MAKE \
LDFLAGS LIBCFLAGS LIBCXXFLAGS QA_CONFIGURE_OPTIONS \
- QA_DESKTOP_FILE PROVIDES_EXCLUDE REQUIRES_EXCLUDE ; do
+ QA_DESKTOP_FILE QA_PREBUILT PROVIDES_EXCLUDE
REQUIRES_EXCLUDE ; do
x=$(echo -n ${!f})
[[ -n $x ]] echo $x $f
done
diff --git a/pym/portage/package/ebuild/doebuild.py
b/pym/portage/package/ebuild/doebuild.py
index 94785b5..6f49f70 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -8,6 +8,7 @@ __all__ = ['doebuild', 'doebuild_environment', 'spawn',
'spawnebuild']
import grp
import gzip
import errno
+import fnmatch
import io
from itertools import chain
import logging
@@ -2209,24 +2210,29 @@ def _post_src_install_soname_symlinks(mysettings, out):
if f is not None:
f.close()
- qa_no_symlink =
- f = None
- try:
- f = io.open(_unicode_encode(os.path.join(
- mysettings[PORTAGE_BUILDDIR],
- build-info, QA_SONAME_NO_SYMLINK),
- encoding=_encodings['fs'], errors='strict'),
- mode='r', encoding=_encodings['repo.content'],
- errors='replace')
- qa_no_symlink = f.read()
- except IOError as e:
- if e.errno not in (errno.ENOENT, errno.ESTALE):
- raise
- finally:
- if f is not None:
- f.close()
+ metadata = {}
+ for k in (QA_PREBUILT, QA_NO_SYMLINK):
+ try:
+ with io.open(_unicode_encode(os.path.join(
+ mysettings[PORTAGE_BUILDDIR],
+ build-info, k),
+ encoding=_encodings['fs'], errors='strict'),
+ mode='r', encoding=_encodings['repo.content'],
+ errors='replace') as f:
+ v = f.read()
+ except IOError as e:
+ if e.errno not in (errno.ENOENT, errno.ESTALE):
+ raise
+ else:
+ metadata[k] = v
+
+ qa_prebuilt = metadata.get(QA_PREBUILT, ).strip()
+ if qa_prebuilt:
+ qa_prebuilt = re.compile(|.join(
+ fnmatch.translate(x.lstrip(os.sep))
+ for x in portage.util.shlex_split(qa_prebuilt)))
- qa_no_symlink = qa_no_symlink.split()
+ qa_no_symlink = metadata.get(QA_NO_SYMLINK, ).split()
if qa_no_symlink:
if len(qa_no_symlink) 1:
qa_no_symlink = |.join((%s) % x for x in
qa_no_symlink)
@@ -2297,6 +2303,7 @@ def _post_src_install_soname_symlinks(mysettings, out):
requires_exclude =
missing_symlinks = []
+ unrecognized_elf_files = []
soname_deps = SonameDepsProcessor(
provides_exclude, requires_exclude)
@@ -2326,7 +2333,14 @@ def _post_src_install_soname_symlinks(mysettings, out):
entry.multilib_category = compute_multilib_category(elf_header)
needed_file.write(_unicode(entry))
- soname_deps.add(entry)
+ if entry.multilib_category is None:
+ if qa_prebuilt.match(
+
entry.filename[len(mysettings[EPREFIX]):].lstrip(
+ os.sep)) is not None:
+ unrecognized_elf_files.append(entry)
+ else:
+ soname_deps.add(entry)
+
obj = entry.filename
soname = entry.soname
@@ -2365,6 +2379,15 @@ def _post_src_install_soname_symlinks(mysettings, out):
errors='strict') as f:
Re: [gentoo-portage-dev] custom profiles?
On 03/11/2015 01:16 PM, Joakim Tjernlund wrote: On Wed, 2015-03-11 at 10:56 -0700, Zac Medico wrote: On 03/11/2015 08:48 AM, Joakim Tjernlund wrote: On Sun, 2015-03-08 at 11:58 -0700, Zac Medico wrote: On 03/08/2015 10:01 AM, Joakim Tjernlund wrote: On Sun, 2015-03-08 at 15:47 +, Joakim Tjernlund wrote: package.use/package.use.force is a bit different though: cat /etc/portage/package.use/qemu app-emulation/qemu vde -alsa -pulseaudio -bluetooth -opengl qemu_user_targets_x86_64 xattr virtfs static- user #Needed by static-user sys-libs/zlib static-libs dev-libs/glib static-libs sys-apps/attr static-libs Moving this to package.use/package.use.force does not respect -alsa, -pulseaudio, -opengl all flags which has a - on them, emerge wants to turn them on again. Am I missing something? Using portage 2.2.18 Appears one have to use package.use.mask for that. cat package.use.mask app-emulation/qemu alsa pulseaudio bluetooth opengl It would be handy if one could use the same syntax as in /etc/portage/package.use/qemu(-alsa -opengl etc.) Jocke Yes, the inverted use.mask logic can be confusing if you are not familiar with it. The negative flags have a special meaning within the context of of portage's incremental stacking behavior, so they can still be useful, though not in the same way that you you attempted to use them. So now I got to binary pkgs and profiles, the profile is typically part of ebuild src tree/overlay and a system using only binary pkgs does not need ebuild sources. How does one manage profiles is this case? Just sync an empty /usr/portage tree(sans profile) or is the a better way? Recent portage has emerge --sync --sync-submodule=profile, which might be useful. I would like to work toward handling this case better, so your feedback is welcome. That would the same as pointing portage to your own empty tree(sans profile). I was hoping for something connected to your BINHOST so one can get all in one go and stored in the same directory tree. Yeah, I'll have to think about that. Maybe we could include a profile URI in the header of the Packages file. Jocke PS. emerge --depclean refuses to clean a system which is lagging behind, would it be possible for --depclean to go ahead anyway somehow? --dynamic-deps=n comes to mind. You should probably put --dynamic-deps=n in EMERGE_DEFAULT_OPTS, since this option typically causes these kinds of problems with --depclean. You don't need --dynamic-deps if you use -- changed-deps when updating. If I do --dynamic-deps=n then man emerge suggest to also do fixpackages(I guess after every SYNC) which feels a bit heavy, is this still needed? Not really, because package moves are applied automatically after each emerge --sync. This happens unconditionally since FEATURES=fixpackages was removed [1], unless you use the emerge --package-moves=n option, in which case we assume that you will manage the consequences yourself. Why is --dynamic-deps=y default? This feels like lying about your true deps, I am probably missing something here, an example would be great:) It's a legacy behavior, since portage has always behaved this way, and ebuild developers have relied upon it (resulting in broken dependency calculations without it). Will --depclean with --dynamic-deps=n always succeed? I realize that it could be dangerous but sometimes would like to have the option. For the most part, yes. However, you may need to use --with-bdeps=y for updates, in order to avoid dependency breakage for build-time-only dependencies. BTW, this text is hard to parse: --binpkg-changed-deps [ y | n ] Tells emerge to ignore binary packages for which the corresponding ebuild depen‐ dencies have changed since the packages were built. In order to help avoid issues with resolving inconsistent dependencies, this option is automatically enabled unless the --usepkgonly option is enabled. Behavior with respect to changed build-time dependencies is controlled by the --with-bdeps option. --binpkg-changed-deps=y - Ignore bin pkgs with changed deps? Yeah, well it's not always easy to translate what the code does into human language. [1] https://github.com/gentoo/portage/commit/7ab4d5722a828167dd1bf946b26cfa80808f59fc -- Thanks, Zac
Re: [gentoo-portage-dev] custom profiles?
On 03/11/2015 01:27 PM, Brian Dolbec wrote: On Wed, 11 Mar 2015 20:16:06 + Joakim Tjernlund joakim.tjernl...@transmode.se wrote: Why is --dynamic-deps=y default? This feels like lying about your true deps, I am probably missing something here, an example would be great:) Because the last time we even discussed the possibility of changing this and steps that might be to fix problems,... There were a few individuals that raised such a stink about it, they even brought it to council to have us STOPPED. Well, stuff like this happens when you have lots of users. On the bright side, at least we have users! Something similar happened when I tried to make emerge --quiet-build the default. So, as a result much of the portage team don't feel like working on portage. Heaven forbid we actually make a change! It's not so bad, considering that you can simply use --dynamic-deps=n if you want, and now --changed-deps makes it feasible regardless of what ebuild developers do. -- Thanks, Zac
Re: [gentoo-portage-dev] running ebuild in src tree
On 03/11/2015 09:03 AM, Joakim Tjernlund wrote: When developing code it would be really nice if one could run your ebuild on that src tree as is(no fetch, unpack etc.) The existing convention is to create an ebuild with version and use one of the live vcs eclasses such as git-r3 to pull the live sources in the src_unpack function. In a future EAPI, we plan to add some features related to this [1]. Then one can compile, install, build a binary pkg etc. just for test. That way you shorten your development/test cycle, get to partially test your brand new ebuild, don't have write custom build scripts etc. Any thoughts on this? Jocke [1] https://bugs.gentoo.org/show_bug.cgi?id=182028 -- Thanks, Zac
Re: [gentoo-portage-dev] custom profiles?
On 03/11/2015 08:48 AM, Joakim Tjernlund wrote: On Sun, 2015-03-08 at 11:58 -0700, Zac Medico wrote: On 03/08/2015 10:01 AM, Joakim Tjernlund wrote: On Sun, 2015-03-08 at 15:47 +, Joakim Tjernlund wrote: package.use/package.use.force is a bit different though: cat /etc/portage/package.use/qemu app-emulation/qemu vde -alsa -pulseaudio -bluetooth -opengl qemu_user_targets_x86_64 xattr virtfs static- user #Needed by static-user sys-libs/zlib static-libs dev-libs/glib static-libs sys-apps/attr static-libs Moving this to package.use/package.use.force does not respect -alsa, -pulseaudio, -opengl all flags which has a - on them, emerge wants to turn them on again. Am I missing something? Using portage 2.2.18 Appears one have to use package.use.mask for that. cat package.use.mask app-emulation/qemu alsa pulseaudio bluetooth opengl It would be handy if one could use the same syntax as in /etc/portage/package.use/qemu(-alsa -opengl etc.) Jocke Yes, the inverted use.mask logic can be confusing if you are not familiar with it. The negative flags have a special meaning within the context of of portage's incremental stacking behavior, so they can still be useful, though not in the same way that you you attempted to use them. So now I got to binary pkgs and profiles, the profile is typically part of ebuild src tree/overlay and a system using only binary pkgs does not need ebuild sources. How does one manage profiles is this case? Just sync an empty /usr/portage tree(sans profile) or is the a better way? Recent portage has emerge --sync --sync-submodule=profile, which might be useful. I would like to work toward handling this case better, so your feedback is welcome. Jocke PS. emerge --depclean refuses to clean a system which is lagging behind, would it be possible for --depclean to go ahead anyway somehow? --dynamic-deps=n comes to mind. You should probably put --dynamic-deps=n in EMERGE_DEFAULT_OPTS, since this option typically causes these kinds of problems with --depclean. You don't need --dynamic-deps if you use --changed-deps when updating. -- Thanks, Zac
Re: [gentoo-portage-dev] running ebuild in src tree
On 03/11/2015 12:08 PM, Joakim Tjernlund wrote: On Wed, 2015-03-11 at 12:03 -0700, Zac Medico wrote: On 03/11/2015 11:56 AM, Joakim Tjernlund wrote: On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote: On 03/11/2015 09:03 AM, Joakim Tjernlund wrote: When developing code it would be really nice if one could run your ebuild on that src tree as is(no fetch, unpack etc.) The existing convention is to create an ebuild with version and use one of the live vcs eclasses such as git-r3 to pull the live sources in the src_unpack function. In a future EAPI, we plan to add some features related to this [1]. I think you misunderstand, [1] is not what I want to do(I think): Got my src working copy and made a few modds, not commitet yet. Now I just want build/test etc. before committing and to do that I just run mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my code is built which I already have in mytree. Well, you can create a - ebuild that copies your sources from $directory to $WORKDIR. Maybe use an environment to configure whether it pulls from a local directory or a vcs repository. No copy, just use the current sources. I realize this is not supported ATM but I think it would be a very useful extension. It could just be an extra option or two to ebuild You can already write the code to do this in the ebuild. Maybe have it create a symlink to your source directory. You can use environment variables, so there's no need to special options for this. It's really none of portage's business where your source code comes from. -- Thanks, Zac
Re: [gentoo-portage-dev] Security and Comparison of Portage with other Package Managers
On 03/08/2015 08:02 AM, Mark Kubacki wrote:
On 03/06/2015 09:50 AM, Mark Kubacki wrote:
And by default you cannot compare the result with any authoritative source.
2015-03-08 0:26 GMT+01:00 Zac Medico zmed...@gentoo.org:
Ideally, we can rely on security mechanisms built into git [1], possibly
involving signed commits.
Some brownfield thinking here, without GIT and not replacing GIT:
1. Find and compile all directories two levels deep in a file
category.idx and sign it.
2. Sign every Manifest.
3. Distribute that as usual.
Will need N+1 checks (N × Manifest + 1 × category present/missing) and
doesn't break anything already deployed.
I think it's an unnecessary expenditure of effort to implement our own
Merkle tree, considering that git's Merkle tree is good enough for the
time being, and will likely implement stronger security soon enough.
Contributors (individuals, teams) need to provide a public key before
submitting, and the mirror source (authority) just checks against
the author's signature
Ideally, this signature check would be implemented as a server-side git
hook, so that a push would be automatically rejected if any of the
pushed commits lacked a good signature.
and signs (1) and (2) with its own key
(official portage tree root key X). That way, in the end, it's
enough to announce only one signing key for every tree.
Or just rely on signed commits in git. We can automatically generate an
empty signed commit with the root key every 30 minutes or something like
that.
(It's easier with binhosts, because all you need to sign is Packages{,gz}.)
Yes, much easier. We might also want to embed signatures directly in
each binary package, so that they can be independently verified without
needing a copy of the original Packages file.
--
Thanks,
Zac
Re: [gentoo-portage-dev] Security and Comparison of Portage with other Package Managers
On 03/08/2015 07:59 AM, Patrick Schleizer wrote: Zac Medico: On 03/06/2015 09:50 AM, Mark Kubacki wrote: We're on the same side here. Do we have numbers showing the ratio portage used with defaults vs. where [webrsync-gpg] is described in many hardening guides for gentoo and widely used among the security conscious applies? DNS not being encrypted is just painting the whole picture. Point is, the default is that emerge --sync results in a transfer using RSYNC (or http). And by default you cannot compare the result with any authoritative source. Ideally, we can rely on security mechanisms built into git [1], possibly involving signed commits. [1] https://github.com/gentoo/gentoo-portage-rsync-mirror Then the question is, how secure are signatures when used wit hgit? And once we answer that question, the question is, is git secure enough for our needs? A while ago I wrote a blog post asking that question, referencing a lot related information, started a discussion and also posted this on the git mailing list. How safe are signed git tags? Only as safe as SHA-1 or somehow safer? [1] [2] Cheers, Patrick [1] https://www.whonix.org/blog/how-safe-are-signed-git-tags-only-as-safe-as-sha-1-or-somehow-safer [2] http://www.mail-archive.com/git@vger.kernel.org/msg61087.html For the time being, I think that git is secure enough for our needs, and I trust that git will implement stronger security soon enough. -- Thanks, Zac
Re: [gentoo-portage-dev] custom profiles?
On 03/06/2015 02:15 AM, Joakim Tjernlund wrote: I am trying to define a custom profile in one of our overlay(managed via layman) This profile should inherit one of /usr/portage/profiles/ and then I want to add custom USE flags, mask, keywords packages etc, then eselect this new profile. I cannot work out how to do this, any pointers? portage: 2.2.18 Jocke Set profile-formats = portage-2 in metadata/layout.conf of your overlay. The is documented in the portage man page. It allows you to specify a profile from another repository in the parent file of you profile. For example, gentoo:default/linux/amd64/13.0/desktop in your parent file will cause your overlay's profile to inherit from that gentoo profile. If you list your profiles in profiles/profiles.desc in you overlay, then eselect profile will see them. -- Thanks, Zac
Re: [gentoo-portage-dev] Security and Comparison of Portage with other Package Managers
On 03/06/2015 09:50 AM, Mark Kubacki wrote: We're on the same side here. Do we have numbers showing the ratio portage used with defaults vs. where [webrsync-gpg] is described in many hardening guides for gentoo and widely used among the security conscious applies? DNS not being encrypted is just painting the whole picture. Point is, the default is that emerge --sync results in a transfer using RSYNC (or http). And by default you cannot compare the result with any authoritative source. Ideally, we can rely on security mechanisms built into git [1], possibly involving signed commits. [1] https://github.com/gentoo/gentoo-portage-rsync-mirror -- Thanks, Zac
Re: [gentoo-portage-dev] Security and Comparison of Portage with other Package Managers
On 03/07/2015 05:24 PM, Brian Dolbec wrote: On Sat, 07 Mar 2015 15:26:26 -0800 Zac Medico zmed...@gentoo.org wrote: On 03/06/2015 09:50 AM, Mark Kubacki wrote: We're on the same side here. Do we have numbers showing the ratio portage used with defaults vs. where [webrsync-gpg] is described in many hardening guides for gentoo and widely used among the security conscious applies? DNS not being encrypted is just painting the whole picture. Point is, the default is that emerge --sync results in a transfer using RSYNC (or http). And by default you cannot compare the result with any authoritative source. Ideally, we can rely on security mechanisms built into git [1], possibly involving signed commits. [1] https://github.com/gentoo/gentoo-portage-rsync-mirror Actually, git makes it nearly impossible to get the correct info required to re-process the git commit signature. It is all embedded into the commit itself, but not in a way for gpg to be able to re-verify it. Git relies on the issuer's gpg verification status which it records in it's data before the push. So, without some major hacking on git data, it is not viable for routine verification purposes. Maybe we can use the relatively recently added git verify-commit command [1]. If we run that command with --verbose, then we can use that to verify that that a commit was signed with a trusted key. Shouldn't verification of the HEAD commit be enough to vouch for the integrity of the entire tree? [1] https://github.com/git/git/commit/d07b00b7f31d0cb6675b4bdba269ce7f087ddd24 -- Thanks, Zac
Re: [gentoo-portage-dev] custom profiles?
On 03/13/2015 05:08 AM, Joakim Tjernlund wrote: On Thu, 2015-03-12 at 17:51 -0700, Zac Medico wrote: On 03/12/2015 02:43 PM, Joakim Tjernlund wrote: Why is --dynamic-deps=y default? This feels like lying about your true deps, I am probably missing something here, an example would be great:) It's a legacy behavior, since portage has always behaved this way, and ebuild developers have relied upon it (resulting in broken dependency calculations without it). Here is odd difference: emerge --dynamic-deps=n changed-deps=y -a1 vanilla-sources ... Nothing to merge That's normal, because --changed-deps implies --selective (a number of options do this). If you add -- selective=n to the above command, you'll get the same result regardless of the --changed-deps option. I just did a sync and emerge -aNDu --dynamic-deps=n --changed-deps=y --selective=n world and again portage wanted to rebuild 150 pkgs. --selective=n seems to be the culprit, should I expect this from --selective=n ? Yes --selective=n is the opposite of --noreplace, so for the above command, it will rebuild everything in /var/lib/portage/world. -- Thanks, Zac
Re: [gentoo-portage-dev] custom profiles?
On 03/14/2015 05:55 AM, Joakim Tjernlund wrote: On Fri, 2015-03-13 at 10:51 -0700, Zac Medico wrote: On 03/13/2015 05:08 AM, Joakim Tjernlund wrote: On Thu, 2015-03-12 at 17:51 -0700, Zac Medico wrote: On 03/12/2015 02:43 PM, Joakim Tjernlund wrote: Why is --dynamic-deps=y default? This feels like lying about your true deps, I am probably missing something here, an example would be great:) It's a legacy behavior, since portage has always behaved this way, and ebuild developers have relied upon it (resulting in broken dependency calculations without it). Here is odd difference: emerge --dynamic-deps=n changed-deps=y -a1 vanilla-sources ... Nothing to merge That's normal, because --changed-deps implies --selective (a number of options do this). If you add -- selective=n to the above command, you'll get the same result regardless of the --changed-deps option. I just did a sync and emerge -aNDu --dynamic-deps=n --changed-deps=y --selective=n world and again portage wanted to rebuild 150 pkgs. --selective=n seems to be the culprit, should I expect this from --selective=n ? Yes --selective=n is the opposite of --noreplace, so for the above command, it will rebuild everything in /var/lib/portage/world. hmm, this kind of a bummer I don't understand your motivation for using --selective=n with that command. Isn't the command useful without it? --dynamic-deps=n implies --changed-deps=y which implies --selective=n and this makes the whole world to rebuild. No, don't use --selective=n. I only mentioned it in order to explain the behavior that you observed. Using just --dynamic-deps=n was not really safe if I understood corretly? It's safe, but you may need --changed-deps in order for your dependency calculations to work (depends on how the dependencies of your installed packages have changed). -- Thanks, Zac
Re: [gentoo-portage-dev] custom profiles?
On 03/14/2015 09:12 AM, Joakim Tjernlund wrote: On Wed, 2015-03-11 at 13:43 -0700, Zac Medico wrote: On 03/11/2015 01:16 PM, Joakim Tjernlund wrote: On Wed, 2015-03-11 at 10:56 -0700, Zac Medico wrote: On 03/11/2015 08:48 AM, Joakim Tjernlund wrote: On Sun, 2015-03-08 at 11:58 -0700, Zac Medico wrote: On 03/08/2015 10:01 AM, Joakim Tjernlund wrote: On Sun, 2015-03-08 at 15:47 +, Joakim Tjernlund wrote: package.use/package.use.force is a bit different though: cat /etc/portage/package.use/qemu app-emulation/qemu vde -alsa -pulseaudio -bluetooth -opengl qemu_user_targets_x86_64 xattr virtfs static- user #Needed by static-user sys-libs/zlib static-libs dev-libs/glib static-libs sys-apps/attr static-libs Moving this to package.use/package.use.force does not respect -alsa, -pulseaudio, -opengl all flags which has a - on them, emerge wants to turn them on again. Am I missing something? Using portage 2.2.18 Appears one have to use package.use.mask for that. cat package.use.mask app-emulation/qemu alsa pulseaudio bluetooth opengl It would be handy if one could use the same syntax as in /etc/portage/package.use/qemu(-alsa - opengl etc.) Jocke Yes, the inverted use.mask logic can be confusing if you are not familiar with it. The negative flags have a special meaning within the context of of portage's incremental stacking behavior, so they can still be useful, though not in the same way that you you attempted to use them. Just noticed that USE flags in profiles/package.use.mask override everything so this USE=thin emerge -av sys-fs/lvm2 will not turn on thin if thin is in profiles/package.use.mask How can just change the default so a user can easily turn it on ? Generally, setting the USE environment variable like that is poor practice, because the setting will not persist the next time that you rebuild the package. So, you should set the flag in /etc/portage/package.use. You can unmask the flag for lvm2 like this: echo sys-fs/lvm2 -thin /etc/portage/profile/package.use.mask Or umask it globally like this: echo -thin /etc/portage/profile/use.mask -- Thanks, Zac
Re: [gentoo-portage-dev] custom profiles?
On 03/14/2015 11:35 AM, Joakim Tjernlund wrote: On Sat, 2015-03-14 at 10:58 -0700, Zac Medico wrote: On 03/14/2015 05:55 AM, Joakim Tjernlund wrote: On Fri, 2015-03-13 at 10:51 -0700, Zac Medico wrote: On 03/13/2015 05:08 AM, Joakim Tjernlund wrote: On Thu, 2015-03-12 at 17:51 -0700, Zac Medico wrote: On 03/12/2015 02:43 PM, Joakim Tjernlund wrote: Why is --dynamic-deps=y default? This feels like lying about your true deps, I am probably missing something here, an example would be great:) It's a legacy behavior, since portage has always behaved this way, and ebuild developers have relied upon it (resulting in broken dependency calculations without it). Here is odd difference: emerge --dynamic-deps=n changed-deps=y -a1 vanilla-sources ... Nothing to merge That's normal, because --changed-deps implies --selective (a number of options do this). If you add -- selective=n to the above command, you'll get the same result regardless of the --changed-deps option. I just did a sync and emerge -aNDu --dynamic-deps=n --changed-deps=y --selective=n world and again portage wanted to rebuild 150 pkgs. --selective=n seems to be the culprit, should I expect this from --selective=n ? Yes --selective=n is the opposite of --noreplace, so for the above command, it will rebuild everything in /var/lib/portage/world. hmm, this kind of a bummer I don't understand your motivation for using --selective=n with that command. Isn't the command useful without it? I have it my default emerge options --dynamic-deps=n implies --changed-deps=y which implies --selective=n and this makes the whole world to rebuild. No, don't use --selective=n. I only mentioned it in order to explain the behavior that you observed. Using just --dynamic-deps=n was not really safe if I understood corretly? It's safe, but you may need --changed-deps in order for your dependency calculations to work (depends on how the dependencies of your installed packages have changed). I am trying to find out what to put in emerge default options and this may need does relay compute in terms of default options. Right, options that imply --selective are not well suited for EMERGE_DEFAULT_OPTS unless you also put --selective=n in EMERGE_DEFAULT_OPTS, and then use --selective just for the commands that require it. As an alternative, we can add support for command-specific default options, as discussed here: https://bugs.gentoo.org/show_bug.cgi?id=540250#c1 Does --dynamic-deps=n only work reliable with emerge -NDu world ? No, it's well suited for EMERGE_DEFAULT_OPTS. It's just that you may find that you encounter dependency conflicts for some calculations unless you use --changed-deps. I use --changed-deps for all of my world updates. -- Thanks, Zac
Re: [gentoo-portage-dev] ROOT=/usr/src/initramfs emerge -ave busybox cause host rebuilds
On 03/12/2015 10:39 AM, Joakim Tjernlund wrote: ROOT=/usr/src/initramfs emerge -ave busybox givs [ebuild R ] app-arch/xz-utils-5.0.8 USE=nls threads -static-libs [ebuild R ] sys-libs/zlib-1.2.8-r1 USE=-minizip -static-libs [ebuild R ] virtual/libintl-0-r1 [ebuild R ] sys-libs/ncurses-5.9-r3 USE=cxx unicode -ada -debug -doc -gpm -minimal -profile -static-libs -tinfo -trace [ebuild R ] sys-devel/gnuconfig-20140728 [ebuild R ] app-arch/bzip2-1.0.6-r6 USE=-static -static-libs [ebuild R ] virtual/libiconv-0-r2 [ebuild R ] sys-libs/timezone-data-2014j USE=nls -right_timezone [ebuild R ] sys-devel/binutils-config-3-r3 [ebuild R ] sys-libs/zlib-1.2.8-r1 to /usr/src/initramfs/ USE=-minizip -static-libs [ebuild U ] sys-libs/timezone-data-2014j [2014g] to /usr/src/initramfs/ USE=nls -right_timezone [ebuild R ] sys-devel/gcc-config-1.7.3 Notice how emerge wants to rebuild stuff in host(ROOT=/) This does not sit right with me, ROOT=/ is completely up to date and the is no USE flags that ha changed. Jocke Yes, this has already been reported: https://bugs.gentoo.org/show_bug.cgi?id=239114 -- Thanks, Zac
Re: [gentoo-portage-dev] custom profiles?
On 03/12/2015 02:43 PM, Joakim Tjernlund wrote: Why is --dynamic-deps=y default? This feels like lying about your true deps, I am probably missing something here, an example would be great:) It's a legacy behavior, since portage has always behaved this way, and ebuild developers have relied upon it (resulting in broken dependency calculations without it). Here is odd difference: emerge --dynamic-deps=n changed-deps=y -a1 vanilla-sources ... Nothing to merge That's normal, because --changed-deps implies --selective (a number of options do this). If you add --selective=n to the above command, you'll get the same result regardless of the --changed-deps option. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] dispatch-conf.conf: less-opts --quit-if-one-screen for bug 501886
From: Marc Schiffbauer msch...@gentoo.org Use --quit-if-one-screen instead of --QUIT-AT-EOF, so that less doesn't quit automatically when the user has scrolled to the bottom. X-Gentoo-Bug: 501886 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=501886 Signed-off-by: Zac Medico zmed...@gentoo.org --- cnf/dispatch-conf.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cnf/dispatch-conf.conf b/cnf/dispatch-conf.conf index 125b7cc..7efce8a 100644 --- a/cnf/dispatch-conf.conf +++ b/cnf/dispatch-conf.conf @@ -31,7 +31,7 @@ diff=diff -Nu '%s' '%s' pager= # Default options used if less is the pager -less-opts=--no-init --QUIT-AT-EOF +less-opts=--no-init --quit-if-one-screen # Diff for interactive merges. # %s output file -- 2.3.0
Re: [gentoo-portage-dev] [PATCH] repoman: add --straight-to-stable (-S) option
On 03/24/2015 02:10 AM, Michał Górny wrote:
Add an option to safely allow committing ebuilds straight to stable.
Before, this required either round trips with multiple commits or
--force option that ignored valid QA concerns and (surprisingly to many
developers) skipped some expensive QA checks.
---
bin/repoman | 28
1 file changed, 16 insertions(+), 12 deletions(-)
diff --git a/bin/repoman b/bin/repoman
index 13c220d..03664bd 100755
--- a/bin/repoman
+++ b/bin/repoman
@@ -191,6 +191,9 @@ def ParseArgs(argv, qahelp):
parser.add_argument('-f', '--force', dest='force', default=False,
action='store_true',
help='Commit with QA violations')
+ parser.add_argument('-S', '--straight-to-stable',
dest='straight_to_stable', default=False,
+ action='store_true', help='Allow committing straight to stable')
+
parser.add_argument('--vcs', dest='vcs',
help='Force using specific VCS instead of autodetection')
@@ -1908,18 +1911,19 @@ for x in effective_scanlist:
(relative_path, len(myaux['DESCRIPTION']),
max_desc_len))
keywords = myaux[KEYWORDS].split()
- stable_keywords = []
- for keyword in keywords:
- if not keyword.startswith(~) and \
- not keyword.startswith(-):
- stable_keywords.append(keyword)
- if stable_keywords:
- if ebuild_path in new_ebuilds and catdir != virtual:
- stable_keywords.sort()
- stats[KEYWORDS.stable] += 1
- fails[KEYWORDS.stable].append(
- x + / + y + .ebuild added with
stable keywords: %s % \
- .join(stable_keywords))
+ if not options.straight_to_stable:
+ stable_keywords = []
+ for keyword in keywords:
+ if not keyword.startswith(~) and \
+ not keyword.startswith(-):
+ stable_keywords.append(keyword
The indents above mix spaces with tabs.
+ if stable_keywords:
+ if ebuild_path in new_ebuilds and catdir !=
virtual:
+ stable_keywords.sort()
+ stats[KEYWORDS.stable] += 1
+ fails[KEYWORDS.stable].append(
+ x + / + y + .ebuild added with
stable keywords: %s % \
+ .join(stable_keywords))
In this scope, the relative_path variable is equivalent to x + / + y +
.ebuild, so you could use that instead.
ebuild_archs = set(kw.lstrip(~) for kw in keywords \
if not kw.startswith(-))
--
Thanks,
Zac
[gentoo-portage-dev] [PATCH 3/7 v2] binpkg-multi-instance 3 of 7
FEATURES=binpkg-multi-instance causes an integer build-id to be
associated with each binary package instance. Inclusion of the build-id
in the file name of the binary package file makes it possible to store
an arbitrary number of binary packages built from the same ebuild.
Having multiple instances is useful for a number of purposes, such as
retaining builds that were built with different USE flags or linked
against different versions of libraries. The location of any particular
package within PKGDIR can be expressed as follows:
${PKGDIR}/${CATEGORY}/${PN}/${PF}-${BUILD_ID}.xpak
The build-id starts at 1 for the first build of a particular ebuild,
and is incremented by 1 for each new build. It is possible to share a
writable PKGDIR over NFS, and locking ensures that each package added
to PKGDIR will have a unique build-id. It is not necessary to migrate
an existing PKGDIR to the new layout, since portage is capable of
working with a mixed PKGDIR layout, where packages using the old layout
are allowed to remain in place.
The new PKGDIR layout is backward-compatible with binhost clients
running older portage, since the file format is identical, the
per-package PATH attribute in the 'Packages' index directs them to
download the file from the correct URI, and they automatically use
BUILD_TIME metadata to select the latest builds.
There is currently no automated way to prune old builds from PKGDIR,
although it is possible to remove packages manually, and then run
'emaint --fix binhost' to update the ${PKGDIR}/Packages index. Support
for FEATURES=binpkg-multi-instance is planned for eclean-pkg.
X-Gentoo-Bug: 150031
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=150031
---
PATCH 3/7 v2 fixes BinpkgPrefetcher to pass the _pkg_path to
BinpkgVerifier. Thanks to David James davidja...@google.com.
bin/quickpkg | 1 -
man/make.conf.5 | 27 +
pym/_emerge/Binpkg.py | 33 +-
pym/_emerge/BinpkgFetcher.py | 13 +-
pym/_emerge/BinpkgPrefetcher.py | 2 +-
pym/_emerge/BinpkgVerifier.py | 6 +-
pym/_emerge/EbuildBinpkg.py | 9 +-
pym/_emerge/EbuildBuild.py| 36 +-
pym/_emerge/Package.py| 16 +-
pym/_emerge/Scheduler.py | 6 +-
pym/_emerge/clear_caches.py | 1 -
pym/portage/const.py | 2 +
pym/portage/dbapi/bintree.py | 684 +-
pym/portage/emaint/modules/binhost/binhost.py | 47 +-
14 files changed, 602 insertions(+), 281 deletions(-)
diff --git a/bin/quickpkg b/bin/quickpkg
index 2c69a69..8b71c3e 100755
--- a/bin/quickpkg
+++ b/bin/quickpkg
@@ -63,7 +63,6 @@ def quickpkg_atom(options, infos, arg, eout):
pkgs_for_arg = 0
for cpv in matches:
excluded_config_files = []
- bintree.prevent_collision(cpv)
dblnk = vardb._dblink(cpv)
have_lock = False
diff --git a/man/make.conf.5 b/man/make.conf.5
index cd1ae21..1b71b97 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -256,6 +256,33 @@ has a \fB\-\-force\fR option that can be used to force
regeneration of digests.
Keep logs from successful binary package merges. This is relevant only when
\fBPORT_LOGDIR\fR is set.
.TP
+.B binpkg\-multi\-instance
+Enable support for multiple binary package instances per ebuild.
+Having multiple instances is useful for a number of purposes, such as
+retaining builds that were built with different USE flags or linked
+against different versions of libraries. The location of any particular
+package within PKGDIR can be expressed as follows:
+
+ ${PKGDIR}/${CATEGORY}/${PN}/${PF}\-${BUILD_ID}.xpak
+
+The build\-id starts at 1 for the first build of a particular ebuild,
+and is incremented by 1 for each new build. It is possible to share a
+writable PKGDIR over NFS, and locking ensures that each package added
+to PKGDIR will have a unique build\-id. It is not necessary to migrate
+an existing PKGDIR to the new layout, since portage is capable of
+working with a mixed PKGDIR layout, where packages using the old layout
+are allowed to remain in place.
+
+The new PKGDIR layout is backward\-compatible with binhost clients
+running older portage, since the file format is identical, the
+per\-package PATH attribute in the 'Packages' index directs them to
+download the file from the correct URI, and they automatically use
+BUILD_TIME metadata to select the latest builds.
+
+There is currently no automated way to prune old builds from PKGDIR,
+although it is possible to remove packages manually, and then run
+\(aqemaint \-\-fix binhost' to update the ${PKGDIR}/Packages index.
+.TP
.B buildpkg
Binary packages will be created for all packages that are merged. Also see
\fBquickpkg\fR(1) and \fBemerge\fR(1) \fB\-\-buildpkg\fR and
[gentoo-portage-dev] [PATCH] make.conf: expand PORTAGE_CONFIGROOT (bug 511806)
This can be useful for making settings, such as PKGDIR, relative
to PORTAGE_CONFIGROOT.
X-Gentoo-Bug: 511806
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=511806
---
pym/portage/package/ebuild/config.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/pym/portage/package/ebuild/config.py
b/pym/portage/package/ebuild/config.py
index 71fe4df..3c0018f 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -394,6 +394,7 @@ class config(object):
# Allow make.globals to set default paths relative to
${EPREFIX}.
expand_map[EPREFIX] = eprefix
+ expand_map[PORTAGE_CONFIGROOT] = config_root
if portage._not_installed:
make_globals_path =
os.path.join(PORTAGE_BASE_PATH, cnf, make.globals)
--
2.0.5
Re: [gentoo-portage-dev] Impl. egetent in user.eclass using script from sys-apps/getent?
On 03/03/2015 10:39 AM, Joakim Tjernlund wrote: I wonder if it would be possible to use the script from sys-apps/getent(included below) to impl. getent instead of using glibc's getent? I cannot see any downside, is there one? Please re-send to the gentoo-dev mailing list. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] Don't spawn socks5-server.py for pkg_nofetch (bug 542052)
Don't spawn socks5-server.py for pkg_nofetch, since the spawn_nofetch function creates a private PORTAGE_TMPDIR. X-Gentoo-Bug: 542052 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=542052 --- pym/portage/package/ebuild/doebuild.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py index a5970d5..94785b5 100644 --- a/pym/portage/package/ebuild/doebuild.py +++ b/pym/portage/package/ebuild/doebuild.py @@ -1497,8 +1497,10 @@ def spawn(mystring, mysettings, debug=False, free=False, droppriv=False, keywords['unshare_net'] = not networked keywords['unshare_ipc'] = not ipc - if not networked: + if not networked and mysettings.get(EBUILD_PHASE) != nofetch: # Provide a SOCKS5-over-UNIX-socket proxy to escape sandbox + # Don't do this for pkg_nofetch, since the spawn_nofetch + # function creates a private PORTAGE_TMPDIR. try: proxy = get_socks5_proxy(mysettings) except NotImplementedError: -- 2.0.5
[gentoo-portage-dev] [PATCH] depgraph: fix 'operation' AttributeError (bug 541754)
Since commit 4f5e4f697e2593df164fc9864893768f7d2375fc, this faulty code has gone unreported because it's only triggered by invalid dependencies in /var/db/pkg. Fixes: 4f5e4f697e25 (Use Package instance attributes to clean up and simplify depgraph.validate_blockers().) X-Gentoo-Bug: 541754 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=541754 --- pym/_emerge/depgraph.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py index e8a3110..37292a6 100644 --- a/pym/_emerge/depgraph.py +++ b/pym/_emerge/depgraph.py @@ -6595,7 +6595,7 @@ class depgraph(object): if not success: replacement_pkgs = self._dynamic_config._package_tracker.match( myroot, pkg.slot_atom) - if any(replacement_pkg[0].operation == merge for \ + if any(replacement_pkg.operation == merge for replacement_pkg in replacement_pkgs): # This package is being replaced anyway, so # ignore invalid dependencies so as not to -- 2.0.5
Re: [gentoo-portage-dev] Pre RFC on RFC: Add compiler information to exported a Package Manger's Cached Information.
On 02/25/2015 04:07 PM, Zac Medico wrote:
On 02/25/2015 03:41 PM, Anthony G. Basile wrote:
On 02/25/15 15:38, Zac Medico wrote:
On 02/25/2015 12:01 PM, Anthony G. Basile wrote:
On 02/25/15 14:51, Anthony G. Basile wrote:
On 02/22/15 01:30, Zac Medico wrote:
On 02/21/2015 10:22 PM, Zac Medico wrote:
If we put the real/canonical libstdc++.so path in the DT_NEEDED
section,
then it will automatically work with existing soname dependency
support.
Actually, we'd also have to add a way for you to put the full path of
the libstdc++.so in PROVIDES. For example:
PROVIDES_ABSOLUTE=/usr/lib/gcc/*/*/libstdc++.so.6
I guess I don't understand how this would work exactly. What if
someone
has gcc-4.8.3. Builds library libfoo.so which uses c++. Then upgrades
to gcc-4.9, removes 4.8 and then tries to build bar which is also
written in c++ and links against libfoo.so. We would have mismatching
abis. How would this catch it and trigger the correct rebuilds?
Unless I'm misunderstanding your *'s in that line. Are you using
PROVIDES_ABSOLUTE as a way of recording what version of the compiler
libfoo.so was build with? So that you'd have a line that says
libfoo.so
links against /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so, so
that parsing that line gives 4.8.3?
Also if you had the absolute path in VDB somewhere, like in PROVIDES,
then you don't need it in the elf's rpath which would make me feel
better.
Actually no, you'd still need rpath for the elf itslef otherwise you can
still link against the wrong version of libstdc++.so. Note in my
following example that even though I build test.cpp with 4.7.3 I still
wind up linking aginast 4.8.3.
If DT_NEEDED contains the absolute libstdc++.so path, it's guaranteed to
link against the correct version, regardless of rpath.
How do you get DT_NEEDED to the absolute libstdc++.so path when building?
I'm not sure how we're going to accomplish that yet, but it should be
feasible. Ideally, the build system would support it. The worst case
would be that we would have to patch the DT_NEEDED sections after
src_install.
Using patchelf built from git [1], I tried to patch the DT_NEEDED of a
hello world program:
$ cat hello.cpp
#include iostream
using namespace std;
int main() { cout Hello World! endl ; }
$ g++ -o hello hello.cpp
$ scanelf -qF '%a;%p;%S;%r;%n' hello
EM_X86_64;hello;;;libstdc++.so.6,libm.so.6,libgcc_s.so.1,libc.so.6
$ ldd hello
linux-vdso.so.1 (0x7fff5992a000)
libstdc++.so.6 =
/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6 (0x7f38f6a82000)
libm.so.6 = /lib64/libm.so.6 (0x7f38f6783000)
libgcc_s.so.1 =
/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1 (0x7f38f656d000)
libc.so.6 = /lib64/libc.so.6 (0x7f38f61c7000)
/lib64/ld-linux-x86-64.so.2 (0x7f38f6d86000)
$ ./hello
Hello World!
$ patchelf --replace-needed libstdc++.so.6
/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6 hello
$ scanelf -qF '%a;%p;%S;%r;%n' hello
EM_X86_64;hello;;;/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6,libm.so.6,libgcc_s.so.1,libc.so.6
$ ldd hello
Inconsistency detected by ld.so: dl-version.c: 224:
_dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
$ ./hello
Inconsistency detected by ld.so: dl-version.c: 224:
_dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
Apparently the ld.so is not capable of resolving absolute paths in
DT_NEEDED. Maybe it's more practical to go with -rpath instead.
[1] https://github.com/NixOS/patchelf
--
Thanks,
Zac
Re: [gentoo-portage-dev] Pre RFC on RFC: Add compiler information to exported a Package Manger's Cached Information.
On 02/25/2015 12:01 PM, Anthony G. Basile wrote:
On 02/25/15 14:51, Anthony G. Basile wrote:
On 02/22/15 01:30, Zac Medico wrote:
On 02/21/2015 10:22 PM, Zac Medico wrote:
If we put the real/canonical libstdc++.so path in the DT_NEEDED
section,
then it will automatically work with existing soname dependency
support.
Actually, we'd also have to add a way for you to put the full path of
the libstdc++.so in PROVIDES. For example:
PROVIDES_ABSOLUTE=/usr/lib/gcc/*/*/libstdc++.so.6
I guess I don't understand how this would work exactly. What if someone
has gcc-4.8.3. Builds library libfoo.so which uses c++. Then upgrades
to gcc-4.9, removes 4.8 and then tries to build bar which is also
written in c++ and links against libfoo.so. We would have mismatching
abis. How would this catch it and trigger the correct rebuilds?
Unless I'm misunderstanding your *'s in that line. Are you using
PROVIDES_ABSOLUTE as a way of recording what version of the compiler
libfoo.so was build with? So that you'd have a line that says libfoo.so
links against /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so, so
that parsing that line gives 4.8.3?
Also if you had the absolute path in VDB somewhere, like in PROVIDES,
then you don't need it in the elf's rpath which would make me feel
better.
Actually no, you'd still need rpath for the elf itslef otherwise you can
still link against the wrong version of libstdc++.so. Note in my
following example that even though I build test.cpp with 4.7.3 I still
wind up linking aginast 4.8.3.
yellow tmp # cat test.cpp
#include iostream
using namespace std;
int main() { cout hello owrld endl ; }
yellow tmp # gcc-config -l
[1] x86_64-pc-linux-gnu-4.7.3 *
[2] x86_64-pc-linux-gnu-4.7.3-hardenednopie
[3] x86_64-pc-linux-gnu-4.7.3-hardenednopiessp
[4] x86_64-pc-linux-gnu-4.7.3-hardenednossp
[5] x86_64-pc-linux-gnu-4.7.3-vanilla
[6] x86_64-pc-linux-gnu-4.8.3
[7] x86_64-pc-linux-gnu-4.8.3-hardenednopie
[8] x86_64-pc-linux-gnu-4.8.3-hardenednopiessp
[9] x86_64-pc-linux-gnu-4.8.3-hardenednossp
[10] x86_64-pc-linux-gnu-4.8.3-vanilla
yellow tmp # g++ -o go test.cpp
yellow tmp # ldd go
linux-vdso.so.1 (0x033f63717000)
libstdc++.so.6 =
/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6 (0x033f631cf000)
libm.so.6 = /lib64/libm.so.6 (0x033f62ecb000)
libgcc_s.so.1 =
/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1 (0x033f62cb4000)
libc.so.6 = /lib64/libc.so.6 (0x033f628f8000)
/lib64/ld-linux-x86-64.so.2 (0x033f634f6000)
yellow tmp # g++ -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/ -o
go test.cpp
yellow tmp # ldd go
linux-vdso.so.1 (0x036035212000)
libstdc++.so.6 =
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libstdc++.so.6 (0x036034ccf000)
libm.so.6 = /lib64/libm.so.6 (0x0360349cb000)
libgcc_s.so.1 =
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libgcc_s.so.1 (0x0360347b4000)
libc.so.6 = /lib64/libc.so.6 (0x0360343f8000)
/lib64/ld-linux-x86-64.so.2 (0x036034ff1000)
If we use -rpath, then we need to decide how to encode the rpath
information for use in soname dependency resolution. For example, we can
introduce a PROVIDES_RPATH variable that the gcc ebuild will use to
export paths for use in soname dependency resolution. For libstdc++
consumers, we can introduce a corresponding REQUIRES_RPATH variable.
Instead of requiring ebuilds to explicitly set a REQUIRES_RPATH
variable, we might instead extract the information from DT_RPATH and
DT_RUNPATH sections. However, the result would probably need to be
subject to filtering by something like REQUIRES_EXCLUDE. Maybe it's
better to have explicit REQUIRES_RPATH settings.
--
Thanks,
Zac
[gentoo-portage-dev] [PATCH 3/7 v3] binpkg-multi-instance 3 of 7
FEATURES=binpkg-multi-instance causes an integer build-id to be
associated with each binary package instance. Inclusion of the build-id
in the file name of the binary package file makes it possible to store
an arbitrary number of binary packages built from the same ebuild.
Having multiple instances is useful for a number of purposes, such as
retaining builds that were built with different USE flags or linked
against different versions of libraries. The location of any particular
package within PKGDIR can be expressed as follows:
${PKGDIR}/${CATEGORY}/${PN}/${PF}-${BUILD_ID}.xpak
The build-id starts at 1 for the first build of a particular ebuild,
and is incremented by 1 for each new build. It is possible to share a
writable PKGDIR over NFS, and locking ensures that each package added
to PKGDIR will have a unique build-id. It is not necessary to migrate
an existing PKGDIR to the new layout, since portage is capable of
working with a mixed PKGDIR layout, where packages using the old layout
are allowed to remain in place.
The new PKGDIR layout is backward-compatible with binhost clients
running older portage, since the file format is identical, the
per-package PATH attribute in the 'Packages' index directs them to
download the file from the correct URI, and they automatically use
BUILD_TIME metadata to select the latest builds.
There is currently no automated way to prune old builds from PKGDIR,
although it is possible to remove packages manually, and then run
'emaint --fix binhost' to update the ${PKGDIR}/Packages index. Support
for FEATURES=binpkg-multi-instance is planned for eclean-pkg.
X-Gentoo-Bug: 150031
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=150031
---
PATCH 3/7 v3 fixes a couple of issues reported by David James:
* Fix depgraph to exhaustively search for a binary package with the
desired USE settings
* Fix binarytree.inject to preserve multiple package instances with
the same BUILD_ID (or even missing BUILD_ID). This is useful if the
client has FEATURES=binpkg-multi-instance enabled, in order to
preserve multiple instances from multiple binhosts that do not have
FEATURES=binpkg-multi-instance enabled. In this case, the number
in the local binpkg file name does not have to correspond to the
BUILD_ID metadata in the package.
bin/quickpkg | 1 -
man/make.conf.5 | 27 +
pym/_emerge/Binpkg.py | 33 +-
pym/_emerge/BinpkgFetcher.py | 13 +-
pym/_emerge/BinpkgPrefetcher.py | 2 +-
pym/_emerge/BinpkgVerifier.py | 6 +-
pym/_emerge/EbuildBinpkg.py | 9 +-
pym/_emerge/EbuildBuild.py| 36 +-
pym/_emerge/Package.py| 16 +-
pym/_emerge/Scheduler.py | 6 +-
pym/_emerge/clear_caches.py | 1 -
pym/_emerge/depgraph.py | 16 +-
pym/portage/const.py | 2 +
pym/portage/dbapi/bintree.py | 683 +-
pym/portage/emaint/modules/binhost/binhost.py | 47 +-
15 files changed, 613 insertions(+), 285 deletions(-)
diff --git a/bin/quickpkg b/bin/quickpkg
index 2c69a69..8b71c3e 100755
--- a/bin/quickpkg
+++ b/bin/quickpkg
@@ -63,7 +63,6 @@ def quickpkg_atom(options, infos, arg, eout):
pkgs_for_arg = 0
for cpv in matches:
excluded_config_files = []
- bintree.prevent_collision(cpv)
dblnk = vardb._dblink(cpv)
have_lock = False
diff --git a/man/make.conf.5 b/man/make.conf.5
index cd1ae21..1b71b97 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -256,6 +256,33 @@ has a \fB\-\-force\fR option that can be used to force
regeneration of digests.
Keep logs from successful binary package merges. This is relevant only when
\fBPORT_LOGDIR\fR is set.
.TP
+.B binpkg\-multi\-instance
+Enable support for multiple binary package instances per ebuild.
+Having multiple instances is useful for a number of purposes, such as
+retaining builds that were built with different USE flags or linked
+against different versions of libraries. The location of any particular
+package within PKGDIR can be expressed as follows:
+
+ ${PKGDIR}/${CATEGORY}/${PN}/${PF}\-${BUILD_ID}.xpak
+
+The build\-id starts at 1 for the first build of a particular ebuild,
+and is incremented by 1 for each new build. It is possible to share a
+writable PKGDIR over NFS, and locking ensures that each package added
+to PKGDIR will have a unique build\-id. It is not necessary to migrate
+an existing PKGDIR to the new layout, since portage is capable of
+working with a mixed PKGDIR layout, where packages using the old layout
+are allowed to remain in place.
+
+The new PKGDIR layout is backward\-compatible with binhost clients
+running older portage, since the file format is identical, the
+per\-package PATH
Re: [gentoo-portage-dev] Pre RFC on RFC: Add compiler information to exported a Package Manger's Cached Information.
On 02/25/2015 11:51 AM, Anthony G. Basile wrote: On 02/22/15 01:30, Zac Medico wrote: On 02/21/2015 10:22 PM, Zac Medico wrote: If we put the real/canonical libstdc++.so path in the DT_NEEDED section, then it will automatically work with existing soname dependency support. Actually, we'd also have to add a way for you to put the full path of the libstdc++.so in PROVIDES. For example: PROVIDES_ABSOLUTE=/usr/lib/gcc/*/*/libstdc++.so.6 I guess I don't understand how this would work exactly. What if someone has gcc-4.8.3. Builds library libfoo.so which uses c++. Then upgrades to gcc-4.9, removes 4.8 and then tries to build bar which is also written in c++ and links against libfoo.so. We would have mismatching abis. How would this catch it and trigger the correct rebuilds? If the absolute libstdc++ path is recorded in DT_NEEDED, then libfoo.so built against gcc-4.8 will break as soon as gcc-4.8 is unmerged. It's easy to see that a rebuild is needed, because the DT_NEEDED data in NEEDED.ELF.2 shows that libfoo.so is linked against gcc-4.8. The relevant part of the DT_NEEDED data is also recorded in REQUIRES (which is available during soname dependency resolution for binary packages). Unless I'm misunderstanding your *'s in that line. Are you using PROVIDES_ABSOLUTE as a way of recording what version of the compiler libfoo.so was build with? No, it's for the gcc ebuild, in order to indicate that the absolute libstdc++ path should be included in PROVIDES, for the purposes of soname dependency resolution. So that you'd have a line that says libfoo.so links against /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so, so that parsing that line gives 4.8.3? No, this dependency information would propagate via DT_NEEDED. Any builds that use the -std= flag would be responsible for ensuring that the absolute libstdc++ path is recorded in DT_NEEDED, rather than the plain libstdc++.so.6 soname. Also if you had the absolute path in VDB somewhere, like in PROVIDES, then you don't need it in the elf's rpath which would make me feel better. Yeah, the absolute path of libstdc++ will be recorded in gcc's PROVIDES, thanks to the ebuild setting the PROVIDES_ABSOLUTE variable. The absolute path of libstdc++ that libfoo.so links against will be recorded in both NEEDED.ELF.2 and REQUIRES. There's no need for rpath. -- Thanks, Zac
Re: [gentoo-portage-dev] Pre RFC on RFC: Add compiler information to exported a Package Manger's Cached Information.
On 02/25/2015 12:01 PM, Anthony G. Basile wrote: On 02/25/15 14:51, Anthony G. Basile wrote: On 02/22/15 01:30, Zac Medico wrote: On 02/21/2015 10:22 PM, Zac Medico wrote: If we put the real/canonical libstdc++.so path in the DT_NEEDED section, then it will automatically work with existing soname dependency support. Actually, we'd also have to add a way for you to put the full path of the libstdc++.so in PROVIDES. For example: PROVIDES_ABSOLUTE=/usr/lib/gcc/*/*/libstdc++.so.6 I guess I don't understand how this would work exactly. What if someone has gcc-4.8.3. Builds library libfoo.so which uses c++. Then upgrades to gcc-4.9, removes 4.8 and then tries to build bar which is also written in c++ and links against libfoo.so. We would have mismatching abis. How would this catch it and trigger the correct rebuilds? Unless I'm misunderstanding your *'s in that line. Are you using PROVIDES_ABSOLUTE as a way of recording what version of the compiler libfoo.so was build with? So that you'd have a line that says libfoo.so links against /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so, so that parsing that line gives 4.8.3? Also if you had the absolute path in VDB somewhere, like in PROVIDES, then you don't need it in the elf's rpath which would make me feel better. Actually no, you'd still need rpath for the elf itslef otherwise you can still link against the wrong version of libstdc++.so. Note in my following example that even though I build test.cpp with 4.7.3 I still wind up linking aginast 4.8.3. If DT_NEEDED contains the absolute libstdc++.so path, it's guaranteed to link against the correct version, regardless of rpath. -- Thanks, Zac
Re: [gentoo-portage-dev] Pre RFC on RFC: Add compiler information to exported a Package Manger's Cached Information.
On 02/21/2015 10:22 PM, Zac Medico wrote: If we put the real/canonical libstdc++.so path in the DT_NEEDED section, then it will automatically work with existing soname dependency support. Actually, we'd also have to add a way for you to put the full path of the libstdc++.so in PROVIDES. For example: PROVIDES_ABSOLUTE=/usr/lib/gcc/*/*/libstdc++.so.6 -- Thanks, Zac
Re: [gentoo-portage-dev] Pre RFC on RFC: Add compiler information to exported a Package Manger's Cached Information.
On 02/21/2015 12:18 PM, Zac Medico wrote: On 02/21/2015 08:14 AM, Anthony G. Basile wrote: Finally, a limitation of the above should be noted. Since the CFLAGS/CXXFLAGS cached are only those supplied by the user, it does not cover situations where the package build system or ebuild supply their own -std= flag. Since this information cannot and should not be cached by the package manager, utilities used to find any mismatches in C++ ABI must provide for this intelligence. I wonder if it's feasible to force the -rpath option when using the -std= flag, and force the real/canonical libstc++.so path into the rpath. There's some mention of -rpath here: http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html#AEN95 Have the -rpath information in the binaries should make it easy to correctly resolve the library dependencies. If we put the real/canonical libstdc++.so path in the DT_NEEDED section, then it will automatically work with existing soname dependency support. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] use_reduce: support non-string token_class (bug 541198)
Do not assume that token_class returns a basestring. X-Gentoo-Bug: 541198 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=541198 --- pym/portage/dep/__init__.py | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/pym/portage/dep/__init__.py b/pym/portage/dep/__init__.py index e2e416c..a8c748d 100644 --- a/pym/portage/dep/__init__.py +++ b/pym/portage/dep/__init__.py @@ -558,7 +558,8 @@ def use_reduce(depstr, uselist=[], masklist=[], matchall=False, excludeall=[], i stack[level].extend(l) continue - if stack[level]: + if stack[level] and isinstance(stack[level][-1], + basestring): if stack[level][-1] == || and not l: #Optimize: || ( ) - . stack[level].pop() @@ -583,7 +584,8 @@ def use_reduce(depstr, uselist=[], masklist=[], matchall=False, excludeall=[], i #ends in a non-operator. This is almost equivalent to stack[level][-1]==||, #expect that it skips empty levels. while k=0: - if stack[k]: + if stack[k] and isinstance(stack[k][-1], + basestring): if stack[k][-1] == ||: return k elif stack[k][-1][-1] != ?: -- 2.0.5
Re: [gentoo-portage-dev] [PATCH] Do not interrupt on SIGCONT
On 02/25/2015 08:30 PM, Mike Frysinger wrote: From: Bertrand SIMONNET bsimon...@chromium.org SIGCONT signals should not interrupt any system calls (locking or wait pid for example). URL: http://crbug.com/417800 X-Gentoo-Bug-URL: https://bugs.gentoo.org/500436 --- pym/_emerge/Scheduler.py | 1 + 1 file changed, 1 insertion(+) diff --git a/pym/_emerge/Scheduler.py b/pym/_emerge/Scheduler.py index d6db311..6e3bf1a 100644 --- a/pym/_emerge/Scheduler.py +++ b/pym/_emerge/Scheduler.py @@ -1017,6 +1017,7 @@ class Scheduler(PollScheduler): earlier_sigterm_handler = signal.signal(signal.SIGTERM, sighandler) earlier_sigcont_handler = \ signal.signal(signal.SIGCONT, self._sigcont_handler) + signal.siginterrupt(signal.SIGCONT, False) try: rval = self._merge() LGTM. We already have wrappers to handle EINTR for waitpid and lockf: https://github.com/gentoo/portage/commit/7c70eea2f607baffcbb9d465c03578d69b09decf https://github.com/gentoo/portage/commit/664684f0f17c0f3570ffd9382037994db1ec66bd I suppose it won't hurt to suppress EINTR though... -- Thanks, Zac
[gentoo-portage-dev] [PATCH] repoman: skip vcs calls for manifest modes (bug 540882)
For manifest and manifest-check modes, there's no need to call the vcs unless --if-modified=y is enabled. X-Gentoo-Bug: 540882 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=540882 --- bin/repoman | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/bin/repoman b/bin/repoman index 16ea055..13c220d 100755 --- a/bin/repoman +++ b/bin/repoman @@ -1187,7 +1187,10 @@ mychanged = [] mynew = [] myremoved = [] -if vcs == cvs: +if (options.if_modified != y and + options.mode in (manifest, manifest-check)): + pass +elif vcs == cvs: mycvstree = cvstree.getentries(./, recursive=1) mychanged = cvstree.findchanged(mycvstree, recursive=1, basedir=./) mynew = cvstree.findnew(mycvstree, recursive=1, basedir=./) -- 2.0.5
Re: [gentoo-portage-dev] Pre RFC on RFC: Add compiler information to exported a Package Manger's Cached Information.
On 02/25/2015 03:41 PM, Anthony G. Basile wrote: On 02/25/15 15:38, Zac Medico wrote: On 02/25/2015 12:01 PM, Anthony G. Basile wrote: On 02/25/15 14:51, Anthony G. Basile wrote: On 02/22/15 01:30, Zac Medico wrote: On 02/21/2015 10:22 PM, Zac Medico wrote: If we put the real/canonical libstdc++.so path in the DT_NEEDED section, then it will automatically work with existing soname dependency support. Actually, we'd also have to add a way for you to put the full path of the libstdc++.so in PROVIDES. For example: PROVIDES_ABSOLUTE=/usr/lib/gcc/*/*/libstdc++.so.6 I guess I don't understand how this would work exactly. What if someone has gcc-4.8.3. Builds library libfoo.so which uses c++. Then upgrades to gcc-4.9, removes 4.8 and then tries to build bar which is also written in c++ and links against libfoo.so. We would have mismatching abis. How would this catch it and trigger the correct rebuilds? Unless I'm misunderstanding your *'s in that line. Are you using PROVIDES_ABSOLUTE as a way of recording what version of the compiler libfoo.so was build with? So that you'd have a line that says libfoo.so links against /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so, so that parsing that line gives 4.8.3? Also if you had the absolute path in VDB somewhere, like in PROVIDES, then you don't need it in the elf's rpath which would make me feel better. Actually no, you'd still need rpath for the elf itslef otherwise you can still link against the wrong version of libstdc++.so. Note in my following example that even though I build test.cpp with 4.7.3 I still wind up linking aginast 4.8.3. If DT_NEEDED contains the absolute libstdc++.so path, it's guaranteed to link against the correct version, regardless of rpath. How do you get DT_NEEDED to the absolute libstdc++.so path when building? I'm not sure how we're going to accomplish that yet, but it should be feasible. Ideally, the build system would support it. The worst case would be that we would have to patch the DT_NEEDED sections after src_install. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] actions.py: fix missing localization import (bug 541302)
Since commit 206efe5f6341bce99a5e9994a0458c304513b2c3, FEATURES=buildpkg triggers a NameError due to a missing localization import. Fix the NameError, and also allow read-only PKGDIR if the --usepkgonly option is enabled. Fixes: 206efe5f6341 (emerge: check for writable PKGDIR (490732)) X-Gentoo-Bug: 541302 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=541302 --- pym/_emerge/actions.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py index fa4fe19..b667537 100644 --- a/pym/_emerge/actions.py +++ b/pym/_emerge/actions.py @@ -63,6 +63,7 @@ from portage.util._async.SchedulerInterface import SchedulerInterface from portage.util._eventloop.global_event_loop import global_event_loop from portage._global_updates import _global_updates from portage.sync.old_tree_timestamp import old_tree_timestamp_warn +from portage.localization import _ from portage.metadata import action_metadata from _emerge.clear_caches import clear_caches @@ -433,7 +434,8 @@ def action_build(settings, trees, mtimedb, _opts_no_self_update.intersection(myopts) need_write_bindb = not any(x in myopts for x in - (--fetchonly, --fetch-all-uri, --pretend)) and \ + (--fetchonly, --fetch-all-uri, + --pretend, --usepkgonly)) and \ (any(buildpkg in trees[eroot][root_config]. settings.features for eroot in trees) or any(buildsyspkg in trees[eroot][root_config]. -- 2.0.5
Re: [gentoo-portage-dev] [PATCH v2] Support escaping network-sandbox through SOCKSv5 proxy
On 01/25/2015 06:00 AM, Michał Górny wrote: diff --git a/bin/socks5-server.py b/bin/socks5-server.py new file mode 100644 index 000..c079018 --- /dev/null +++ b/bin/socks5-server.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python +# SOCKSv5 proxy server for network-sandbox +# Copyright 2015 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 + +import asyncore +import errno +import socket +import struct +import sys + + +class ProxyConnection(asyncore.dispatcher_with_send): + _addr = None + _connected = False + _family = socket.AF_INET + _proxy_conn = None + + def __init__(self, proxy_conn): + self._proxy_conn = proxy_conn + asyncore.dispatcher_with_send.__init__(self) + self.create_socket(self._family, socket.SOCK_STREAM) + + def start_connection(self, host, port): + try: + self.connect((host, port)) + except: + self.handle_error() This except handler should at least allow SystemExit and KeyboardInterrupt to raise. diff --git a/pym/portage/package/ebuild/_config/special_env_vars.py b/pym/portage/package/ebuild/_config/special_env_vars.py index 6bb3c95..905d5e7 100644 --- a/pym/portage/package/ebuild/_config/special_env_vars.py +++ b/pym/portage/package/ebuild/_config/special_env_vars.py @@ -71,7 +71,7 @@ environ_whitelist += [ PORTAGE_PYM_PATH, PORTAGE_PYTHON, PORTAGE_PYTHONPATH, PORTAGE_QUIET, PORTAGE_REPO_NAME, PORTAGE_REPOSITORIES, PORTAGE_RESTRICT, - PORTAGE_SIGPIPE_STATUS, + PORTAGE_SIGPIPE_STATUS, PORTAGE_SOCKS5_PROXY, PORTAGE_TMPDIR, PORTAGE_UPDATE_ENV, PORTAGE_USERNAME, PORTAGE_VERBOSE, PORTAGE_WORKDIR_MODE, PORTAGE_XATTR_EXCLUDE, PORTDIR, PORTDIR_OVERLAY, PREROOTPATH, The DISTCC_SOCKS_PROXY variable should also be added to the whitelist. Other than these 2 minor issues, the patch looks to me. I guess there's no point in using portage's event loop instead of asyncore, since we want the proxy to drop privileges, and therefore it can't run in the main portage process. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH v2] Support escaping network-sandbox through SOCKSv5 proxy
On 01/25/2015 06:00 AM, Michał Górny wrote: diff --git a/bin/socks5-server.py b/bin/socks5-server.py new file mode 100644 index 000..c079018 --- /dev/null +++ b/bin/socks5-server.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python +# SOCKSv5 proxy server for network-sandbox +# Copyright 2015 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 + +import asyncore +import errno +import socket +import struct +import sys + + +class ProxyConnection(asyncore.dispatcher_with_send): + _addr = None + _connected = False + _family = socket.AF_INET + _proxy_conn = None + + def __init__(self, proxy_conn): + self._proxy_conn = proxy_conn + asyncore.dispatcher_with_send.__init__(self) + self.create_socket(self._family, socket.SOCK_STREAM) + + def start_connection(self, host, port): + try: + self.connect((host, port)) + except: + self.handle_error() + + def handle_read(self): + buf = self.recv(4096) + self._proxy_conn.send(buf) The self.recv calls should probably handle BlockingIOError, since the docs say it can be raised even though select.select() or select.poll() has reported the socket ready for reading. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH v2] Support escaping network-sandbox through SOCKSv5 proxy
On 01/25/2015 06:00 AM, Michał Górny wrote: + def __bool__(self):a + return self.socket_path is not None You should also implement __nonzero__ for python-2.x. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH v3] Support escaping network-sandbox through SOCKSv5 proxy
On 01/25/2015 02:34 PM, Michał Górny wrote: diff --git a/bin/socks5-server.py b/bin/socks5-server.py new file mode 100644 index 000..45cf76b --- /dev/null +++ b/bin/socks5-server.py @@ -0,0 +1,233 @@ +#!/usr/bin/env python +# SOCKSv5 proxy server for network-sandbox +# Copyright 2015 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 + +import asyncore +import errno +import os +import socket +import struct +import sys + + +if sys.hexversion 0x0300: + from io import BlockingIOError + + +class ProxyConnection(asyncore.dispatcher_with_send): + _addr = None + _connected = False + _family = socket.AF_INET + _proxy_conn = None You've defined these as class variables, but they should be instance variables (initialized in the constructor). Since the class is a singleton, it works either way, but it's poor style to use class variables like this. +class ProxyHandler(asyncore.dispatcher_with_send): + _my_buf = b'' + _my_conn = None + _my_state = 0 + _my_addr = None These class variables should also be changed to instance variables. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] Allow IPC networking during pkg_config pkg_info
On 01/25/2015 03:42 AM, Michał Górny wrote: --- pym/portage/package/ebuild/doebuild.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py index 0d71f01..050f6c4 100644 --- a/pym/portage/package/ebuild/doebuild.py +++ b/pym/portage/package/ebuild/doebuild.py @@ -86,7 +86,7 @@ _unsandboxed_phases = frozenset([ # phases in which IPC with host is allowed _ipc_phases = frozenset([ - setup, pretend, + setup, pretend, config, info, preinst, postinst, prerm, postrm, ]) LGTM. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] Generate soname dependency metadata (282639)
On 01/29/2015 04:54 PM, Anthony G. Basile wrote:
On 01/26/15 22:16, Zac Medico wrote:
ELF gets us like 99% of the way but will eventually have to think about
Mach-O for gentoo-on-mac etc. Maybe COFF, a.out. Not sure where to
draw the line.
Yeah, I guess we can let the Prefix team decide on that stuff.
I'm a bit confused here. So PROVIDES and REQUIRES are new files in
/var/db/pkg/cat/pkg?
Right.
And the format of NEEDED.ELF.2 is not changing? Correct?
It has one new field, for the multilib category (same as the abstract
abi identifier discussed in bug 534206). I didn't bother to document it,
since is was already completely undocumented. I suppose it would be nice
to have it documented in portage.5, though.
We should note at some point that arm_32 means EABI and not OABI which
are two different 32-bit arm abis. We don't support oabi in gentoo
anymore, but still this can be confusing. If it did come up in some
context, we could distinguish it as arm_o32 like mips below.
Yeah, I'm inclined to just wait for someone to request support for
something like this, and deal with it then.
We probably don't have to note that arm_64 is aarch6 which is really a
different ISA, not backwards compatible to 32-bit arm. As you say below
this is just a naming convention, but in every other case each line does
refer to one ISA, except maybe ia_32 can be confused with x86.
Well, wikipedia says AArch64 provides user-space compatibility with
ARMv7-A ISA [1], so I think that's a good enough reason to stick with
arm_{32,64} for now.
+#hppa_{32,64}
+#ia_{32,64}
+#m68k_{32,64}
+#mips_{eabi32,eabi64,n32,n64,o32,o64}
+#ppc_{32,64}
+#s390_{32,64}
+#sh_{32,64}
+#sparc_{32,64}
+#x86_{32,64,x32}
+#
+# NOTES:
+#
+# * The ABIs referenced by some of the above *_32 and *_64 categories
+# may be imaginary, but they are listed anyway, since the goal is to
+# establish a naming convention that is as consistent and uniform as
+# possible.
+#
+# * The Elf header's e_ident[EI_OSABI] byte is completely ignored,
+# since OS-independence is one of the goals. The assumption is that,
+# for given installation, we are only interested in tracking multilib
+# ABIs for a single OS.
If you run readelf -h on (say) bash in any of our stage3's tarballs you
always get OS/ABI: UNIX - System V irrespective of arch and abi. I
don't know what you would get on BSD, but the field is totally
irrelevant for our purposes despite the name. As far as I can tell, it
is totally invariant across arches and abis.
You can even unpack the the stage3's on an amd64 host and run readelf
form the host on the chroot target and you'll get the elf header, so you
don't need access to native hardware.
The comment suggests that there might be some interesting information
there, but there isn't. Maybe I'm just reading too much into it.
Well, a quick google search seems to indicate that FreeBSD uses
EI_OSABI. I was specifically thinking about FreeBSD when I wrote that
comment, because I was aware that Gentoo/FBSD was using ELF, and I just
assumed that they would have a different EI_OSABI than Linux.
diff --git a/pym/portage/util/_dyn_libs/NeededEntry.py
b/pym/portage/util/_dyn_libs/NeededEntry.py
new file mode 100644
index 000..5de59a0
--- /dev/null
+++ b/pym/portage/util/_dyn_libs/NeededEntry.py
@@ -0,0 +1,83 @@
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+from __future__ import unicode_literals
+
+import sys
+
+from portage import _encodings, _unicode_encode
+from portage.exception import InvalidData
+from portage.localization import _
+
+class NeededEntry(object):
+
+Represents one entry (line) from a NEEDED.ELF.2 file. The entry
+must have 5 or more semicolon-delimited fields in order to be
+considered valid. The sixth field is optional, corresponding
+to the multilib category. The multilib_category attribute is
+None if the corresponding field is either empty or missing.
+
+
+__slots__ = (arch, filename, multilib_category, needed,
+runpaths, soname)
Looks like this answers my question above about the format of NEEDED.ELF.2
Yeah, maybe this is enough documentation for those who would be interested.
Nice logic here :) The only thing that I don't get is why we might need
{provides,requires}_exclude patterns. I guess its good design
principles but I can't think of a use case.
I copied the idea from Fedora [2]. Since the soname dependencies are
automatically generated, we want to be able to filter them if necessary,
though hopefully it won't be needed very often.
b/pym/portage/util/elf/constants.py
new file mode 100644
index 000..3857b71
--- /dev/null
+++ b/pym/portage/util/elf/constants.py
Document where these are coming from else we'll loose the connection to
the standard. They should all be in elf.h provided
Re: [gentoo-portage-dev] [PATCH 2/3] portage/sync/syncbase.py: Change has_bin to an @property function
On 01/29/2015 11:45 AM, Brian Dolbec wrote: This avoids that self.logger is None error in the __init__(). --- pym/portage/sync/syncbase.py | 24 +++- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/pym/portage/sync/syncbase.py b/pym/portage/sync/syncbase.py index c820bcf..973d545 100644 --- a/pym/portage/sync/syncbase.py +++ b/pym/portage/sync/syncbase.py @@ -35,17 +35,23 @@ class SyncBase(object): self.xterm_titles = None self.spawn_kwargs = None self.bin_command = None - self.has_bin = False + self._bin_command = bin_command + self.bin_pkg = bin_pkg if bin_command: self.bin_command = portage.process.find_binary(bin_command) - if self.bin_command is None: - msg = [Command not found: %s % bin_command, - Type \emerge %s\ to enable %s support. % (bin_pkg, bin_command)] - for l in msg: - writemsg_level(!!! %s\n % l, - level=self.logger.ERROR, noiselevel=-1) - else: - self.has_bin = True + + + @property + def has_bin(self): + if self.bin_command is None: + msg = [Command not found: %s % self._bin_command, + Type \emerge %s\ to enable %s support. + % (self.bin_pkg, self._bin_command)] + for l in msg: + writemsg_level(!!! %s\n % l, + level=self.logger.ERROR, noiselevel=-1) + return False + return True def _kwargs(self, kwargs): Maybe it should be named _has_bin instead, since it should only be called by a subclass after it has called _kwargs to initialize self.logger. Otherwise, somebody might think they can access has_bin before _kwargs has been called. -- Thanks, Zac
[gentoo-portage-dev] Re: [gentoo-dev] [PATCH] pym/portage/news.py: let slackers copy+paste the news read command
On 01/30/2015 12:14 AM, Jason Zaman wrote: On Thu, Jan 29, 2015 at 11:00:21PM -0800, Zac Medico wrote: Also, when the work read appears twice on a short line like that, it gives a wordy/redundant feeling. Perhaps better would be 'eselect news read' to view new items? ie. view instead of the second read. Yeah, that's much better. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 3/3] portage/sync: Some final pyflakes code cleanup
On 01/29/2015 11:45 AM, Brian Dolbec wrote: --- pym/portage/sync/modules/cvs/cvs.py | 1 - pym/portage/sync/modules/git/git.py | 6 -- pym/portage/sync/modules/svn/svn.py | 5 ++--- pym/portage/sync/modules/websync/websync.py | 4 4 files changed, 2 insertions(+), 14 deletions(-) LGTM. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/3] portage/sync: Break out a NewBase class from SyncBase
On 01/29/2015 11:45 AM, Brian Dolbec wrote: From mgorny's suggestion, rename _sync() to update(). Raise NotImplementedError in base classes. Directly override sync() in websync module. Fix up line spacing to be consistent. --- pym/portage/sync/modules/cvs/cvs.py | 10 +++--- pym/portage/sync/modules/git/git.py | 8 ++--- pym/portage/sync/modules/rsync/rsync.py | 12 +++ pym/portage/sync/modules/svn/svn.py | 10 +++--- pym/portage/sync/modules/websync/websync.py | 21 +-- pym/portage/sync/syncbase.py| 56 ++--- 6 files changed, 64 insertions(+), 53 deletions(-) Looks good, but it feels a little bit crazy to split out a NewBase class when WebRsync is the only one that doesn't inherit from it. WebRsync is a very special case, and it could just as well inherit from NewBase, with the new() method calling update(). -- Thanks, Zac
[gentoo-portage-dev] [PATCH v3] dispatch-conf: fix unicode handling (bug 545270)
This avoids UnicodeDecodeError problems by using UTF-8 encoding
regardless of the locale.
X-Gentoo-Bug: 545270
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=545270
---
[PATCH v3] adds related fixes in bin/dispatch-conf
bin/dispatch-conf| 9 +
pym/portage/dispatch_conf.py | 4 +---
2 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/bin/dispatch-conf b/bin/dispatch-conf
index b679910..678a66d 100755
--- a/bin/dispatch-conf
+++ b/bin/dispatch-conf
@@ -11,12 +11,11 @@
# dialog menus
#
-from __future__ import print_function
+from __future__ import print_function, unicode_literals
import atexit
import io
import re
-import shutil
import sys
from stat import ST_GID, ST_MODE, ST_UID
@@ -27,7 +26,7 @@ if
osp.isfile(osp.join(osp.dirname(osp.dirname(osp.realpath(__file__))), .porta
sys.path.insert(0,
osp.join(osp.dirname(osp.dirname(osp.realpath(__file__))), pym))
import portage
portage._internal_caller = True
-from portage import os
+from portage import os, shutil
from portage import _encodings, _unicode_decode
from portage.dispatch_conf import diffstatusoutput, diff_mixed_wrapper
from portage.process import find_binary, spawn
@@ -403,7 +402,9 @@ class dispatch:
newconfigs.sort ()
for nconf in newconfigs:
-nconf = nconf.rstrip ()
+# Use strict mode here, because we want to know if it fails,
+# and portage only merges files with valid UTF-8 encoding.
+nconf = _unicode_decode(nconf, errors='strict').rstrip()
conf = re.sub (r'\._cfg\d+_', '', nconf)
dirname = os.path.dirname(nconf)
conf_map = {
diff --git a/pym/portage/dispatch_conf.py b/pym/portage/dispatch_conf.py
index 790eacb..98939fd 100644
--- a/pym/portage/dispatch_conf.py
+++ b/pym/portage/dispatch_conf.py
@@ -10,15 +10,13 @@ from __future__ import print_function, unicode_literals
import io
import functools
-import os
-import shutil
import stat
import subprocess
import sys
import tempfile
import portage
-from portage import _encodings
+from portage import _encodings, os, shutil
from portage.env.loaders import KeyValuePairFileLoader
from portage.localization import _
from portage.util import shlex_split, varexpand
--
2.3.1
[gentoo-portage-dev] [PATCH v2] repoman: fix dependency.unknown to ignore USE deps (bug 525376)
The surrounding code is ignorant of USE flags, because it calls use_reduce(matchall=True), therefore it makes sense for the dependency.unknown code to ignore USE deps. X-Gentoo-Bug: 525376 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525376 --- [PATCH v2] only changes the bug references from bug 545294 to bug 525376 bin/repoman | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/bin/repoman b/bin/repoman index 7101a00..4a21e37 100755 --- a/bin/repoman +++ b/bin/repoman @@ -2049,9 +2049,10 @@ for x in effective_scanlist: # Skip dependency.unknown for blockers, so that we # don't encourage people to remove necessary blockers, - # as discussed in bug #382407. + # as discussed in bug 382407. We use atom.without_use + # due to bug 525376. if not is_blocker and \ - not portdb.xmatch(match-all, atom) and \ + not portdb.xmatch(match-all, atom.without_use) and \ not atom.cp.startswith(virtual/): unknown_pkgs.add((mytype, atom.unevaluated_atom)) -- 2.3.1
Re: [gentoo-portage-dev] [PATCHv3 1/2] MEDIUM: misc-functions: Be more quiet when removing non existing INSTALL_MASK
These are in the master branch now:
https://gitweb.gentoo.org/proj/portage.git/commit/?id=5a1b870fe54ac06f864a648c3ea5cc118f6ce911
https://gitweb.gentoo.org/proj/portage.git/commit/?id=e74e2670e3f043608fced9847e54bdbb19f35169
On 04/20/2015 12:45 PM, Bertrand Jaquin wrote:
# normal stuff
- rm -Rf ${root}/${no_inst} /dev/null
+ if [[ -e ${root}/${no_inst} ]] ; then
+ __quiet_mode || einfo Removing ${no_inst}
+ rm -Rf ${root}/${no_inst} /dev/null
+ fi
I modified the test as follows in order to ensure that it still works
with shell globs:
if [[ -e ${root}/${no_inst} || ${root}/${no_inst} != $(echo
${root}/${no_inst}) ]] ; then
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCHv3 1/2] MEDIUM: misc-functions: Be more quiet when removing non existing INSTALL_MASK
On 04/20/2015 05:31 PM, Bertrand Jacquin wrote:
On 21/04/2015 00:39, Zac Medico wrote:
These are in the master branch now:
https://gitweb.gentoo.org/proj/portage.git/commit/?id=5a1b870fe54ac06f864a648c3ea5cc118f6ce911
https://gitweb.gentoo.org/proj/portage.git/commit/?id=e74e2670e3f043608fced9847e54bdbb19f35169
Thanks for that :)
I modified the test as follows in order to ensure that it still works
with shell globs:
if [[ -e ${root}/${no_inst} || ${root}/${no_inst} != $(echo
${root}/${no_inst}) ]] ; then
I do not really understand the use case here, do you have an example ?
Cheers,
Well, I don't use INSTALL_MASK myself, so I don't have a real-world
use-case for you. However, it's clear that the code will expand shell
globs, so I preserved that behavior for compatibility.
--
Thanks,
Zac
Re: [gentoo-portage-dev] Re: [PATCHv3 1/2] MEDIUM: misc-functions: Be more quiet when removing non existing INSTALL_MASK
On 04/21/2015 10:27 AM, Zac Medico wrote:
On 04/21/2015 02:48 AM, Duncan wrote:
Zac Medico posted on Mon, 20 Apr 2015 17:37:15 -0700 as excerpted:
On 04/20/2015 05:31 PM, Bertrand Jacquin wrote:
On 21/04/2015 00:39, Zac Medico wrote:
These are in the master branch now:
https://gitweb.gentoo.org/proj/portage.git/commit/?
id=5a1b870fe54ac06f864a648c3ea5cc118f6ce911
https://gitweb.gentoo.org/proj/portage.git/commit/?
id=e74e2670e3f043608fced9847e54bdbb19f35169
Thanks for that :)
I modified the test as follows in order to ensure that it still works
with shell globs:
if [[ -e ${root}/${no_inst} || ${root}/${no_inst} != $(echo
${root}/${no_inst}) ]] ; then
I do not really understand the use case here, do you have an example ?
Cheers,
Well, I don't use INSTALL_MASK myself, so I don't have a real-world
use-case for you. However, it's clear that the code will expand shell
globs, so I preserved that behavior for compatibility.
I do, with shell globs, tho I didn't bother checking the above to see if
they'd have been affected.
The two install-masks with globs I use here are:
*.la
Since it doesn't have an absolute path, this glob is handled by the
find/rm code which comes just after the shell glob code that I was
talking about.
(Unmasked on libtool itself, since it has a *.la file that other
package's .configure scripts test for.)
/etc/cron.*/
(I use systemd's timers in place of cron and thus crontablets.)
This is absolute, so it is handled by the shell glob code in question.
Oh, I see what you mean now. It's reconciled by a temporary 'set -o
noglob', followed by 'set +o noglob'.
--
Thanks,
Zac
Re: [gentoo-portage-dev] Re: [PATCHv3 1/2] MEDIUM: misc-functions: Be more quiet when removing non existing INSTALL_MASK
On 04/21/2015 02:48 AM, Duncan wrote:
Zac Medico posted on Mon, 20 Apr 2015 17:37:15 -0700 as excerpted:
On 04/20/2015 05:31 PM, Bertrand Jacquin wrote:
On 21/04/2015 00:39, Zac Medico wrote:
These are in the master branch now:
https://gitweb.gentoo.org/proj/portage.git/commit/?
id=5a1b870fe54ac06f864a648c3ea5cc118f6ce911
https://gitweb.gentoo.org/proj/portage.git/commit/?
id=e74e2670e3f043608fced9847e54bdbb19f35169
Thanks for that :)
I modified the test as follows in order to ensure that it still works
with shell globs:
if [[ -e ${root}/${no_inst} || ${root}/${no_inst} != $(echo
${root}/${no_inst}) ]] ; then
I do not really understand the use case here, do you have an example ?
Cheers,
Well, I don't use INSTALL_MASK myself, so I don't have a real-world
use-case for you. However, it's clear that the code will expand shell
globs, so I preserved that behavior for compatibility.
I do, with shell globs, tho I didn't bother checking the above to see if
they'd have been affected.
The two install-masks with globs I use here are:
*.la
Since it doesn't have an absolute path, this glob is handled by the
find/rm code which comes just after the shell glob code that I was
talking about.
(Unmasked on libtool itself, since it has a *.la file that other
package's .configure scripts test for.)
/etc/cron.*/
(I use systemd's timers in place of cron and thus crontablets.)
This is absolute, so it is handled by the shell glob code in question.
--
Thanks,
Zac
Re: [gentoo-portage-dev] Re: [PATCHv3 1/2] MEDIUM: misc-functions: Be more quiet when removing non existing INSTALL_MASK
On 04/21/2015 04:21 AM, Michael Orlitzky wrote: On 04/21/2015 05:48 AM, Duncan wrote: Well, I don't use INSTALL_MASK myself, so I don't have a real-world use-case for you. However, it's clear that the code will expand shell globs, so I preserved that behavior for compatibility. I do, with shell globs, tho I didn't bother checking the above to see if they'd have been affected. The docs for INSTALL_MASK (man 5 make.conf) don't mention that globs will work. It's expecting a space delimited list of file names. Does it really take a space-delimited list of globs instead? If so, how does that reconcile with the fact that * could match spaces? How does it conflict? -- Thanks, Zac
[gentoo-portage-dev] [PATCH] LinkageMapElf.rebuild: pass error_leader to varexpand (bug 542796)
Since commit f1c1b8a77eebf7713b32e5f9945690f60f4f46de,
LinkageMapElf.rebuild could produce mysterious bad substitution
messages.
Fixes: f1c1b8a77eeb (Generate soname dependency metadata (bug 282639))
X-Gentoo-Bug: 542796
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=542796
X-Gentoo-forum-thread: https://forums.gentoo.org/viewtopic-t-1014842.html
---
pym/portage/util/_dyn_libs/LinkageMapELF.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/pym/portage/util/_dyn_libs/LinkageMapELF.py
b/pym/portage/util/_dyn_libs/LinkageMapELF.py
index c44666a..f4d8b5d 100644
--- a/pym/portage/util/_dyn_libs/LinkageMapELF.py
+++ b/pym/portage/util/_dyn_libs/LinkageMapELF.py
@@ -339,7 +339,8 @@ class LinkageMapELF(object):
obj = entry.filename
soname = entry.soname
expand = {ORIGIN: os.path.dirname(entry.filename)}
- path = frozenset(normalize_path(varexpand(x, expand))
+ path = frozenset(normalize_path(
+ varexpand(x, expand, error_leader=lambda: %s:
% location))
for x in entry.runpaths)
path = frozensets.setdefault(path, path)
needed = frozenset(entry.needed)
--
2.3.1
Re: [gentoo-portage-dev] [PATCH] portage/util.py: Identify the source of a bad sustitution error message (bug 542796)
On 04/23/2015 09:01 AM, Brian Dolbec wrote: This bug caused great difficulty in determining the cause of the error message. This adds identifying code which states where the error message is coming from. It also adds the actual source which caused the error message in the first place. X-Gentoo-Bug: 542796 X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=542796 X-Gentoo-forum-thread: https://forums.gentoo.org/viewtopic-t-1014842.html Since varexpand already has an error_leader argument, we could use that to report the source. The relevant varexpand calls are in pym/portage/util/_dyn_libs/LinkageMapELF.py, added in commit f1c1b8a77eebf7713b32e5f9945690f60f4f46de. -- Thanks, Zac
Re: [gentoo-portage-dev] emerge -a @preserved-rebuild wants to rebuild dev-java/oracle-jdk-bin-1.8.0.45
On 04/23/2015 11:20 PM, Joakim Tjernlund wrote: On Thu, 2015-04-23 at 18:57 -0700, Zac Medico wrote: On 04/23/2015 04:35 AM, Joakim Tjernlund wrote: On Wed, 2015-04-22 at 18:49 -0700, Zac Medico wrote: On 04/22/2015 04:07 PM, Joakim Tjernlund wrote: emerge -a @preserved-rebuild [ebuild Rf ] dev-java/oracle-jdk-bin-1.8.0.45 but this does not help, how can I see what @preserved-rebuild is looking for? Jocke Probably ffmpeg/libav libraries. Here's a bug report: https://bugs.gentoo.org/show_bug.cgi?id=460468 Yes, that is it :) I would really like to have some verbose mode to @preserved-rebuild which shows what lib(s) is/are causing a rebuild. Usually the !!! existing preserved libs message suffices. For example, I just hit the exact same issue and the output looked like this: !!! existing preserved libs: package: media-video/libav-9.17 * - /usr/lib64/libavformat.so.54 * - /usr/lib64/libavformat.so.54.20.4 * used by /opt/oracle-jdk-bin-1.8.0.45/jre/lib/amd64/libavplugin-54.so (dev-java/oracle-jdk-bin-1.8.0.45) * - /usr/lib64/libavcodec.so.54 * - /usr/lib64/libavcodec.so.54.35.0 * used by /opt/oracle-jdk-bin-1.8.0.45/jre/lib/amd64/libavplugin-54.so (dev-java/oracle-jdk-bin-1.8.0.45) That is exactly what I needed too but nothing appeared other than !!! than existing preserved libs I guess I have silenced this warning somehow: # portageq envvar FEATURES assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge -orphans userfetch userpriv usersandbox usersync xattr # portageq envvar EMERGE_DEFAULT_OPTS --quiet --jobs=6 --keep-going --load-average=12 --buildpkg --quiet perhaps? I did try emerge -v.. too override Yes, --quiet does it, and -v doesn't override it. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] search: fix addCP so only the specified results are displayed (bug 547736)
Since commit 4938b8a8a72e719b394a5c5b0c5030c160091d57, the search.addCP method has not worked as intended. Fixes: 4938b8a8a72e (Display emerge search results incrementally (412471)) X-Gentoo-Bug: 547736 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=547736 --- pym/_emerge/search.py | 14 +++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/pym/_emerge/search.py b/pym/_emerge/search.py index 5a8143c..41c182d 100644 --- a/pym/_emerge/search.py +++ b/pym/_emerge/search.py @@ -36,6 +36,7 @@ class search(object): self.verbose = verbose self.searchdesc = searchdesc self.searchkey = None + self._results_specified = False # Disable the spinner since search results are displayed # incrementally. self.spinner = None @@ -295,6 +296,12 @@ class search(object): yield (set, setname) def addCP(self, cp): + + Add a specific cp to the search results. This modifies the + behavior of the output method, so that it only displays specific + packages added via this method. + + self._results_specified = True if not self._xmatch(match-all, cp): return self.matches[pkg].append(cp) @@ -315,11 +322,12 @@ class search(object): metadata_keys.update([DESCRIPTION, HOMEPAGE, LICENSE, SRC_URI]) metadata_keys = tuple(metadata_keys) - if self.searchkey is None: + if self._results_specified: # Handle results added via addCP addCP_matches = [] - for mytype, match in self.matches.items(): - addCP_matches.append(mytype, match) + for mytype, matches in self.matches.items(): + for match in matches: + addCP_matches.append((mytype, match)) iterator = iter(addCP_matches) else: -- 2.3.5
[gentoo-portage-dev] [PATCH] Make the USE variable readonly (bug 325009)
This requires the EBUILD_FORCE_TEST code from dyn_test to execute
before USE is declared readonly.
X-Gentoo-Bug: 325009
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=325009
---
bin/ebuild.sh | 10 +-
bin/phase-functions.sh | 9 +
2 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index b6b3723..4e26f87 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -1,5 +1,5 @@
#!/bin/bash
-# Copyright 1999-2013 Gentoo Foundation
+# Copyright 1999-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
PORTAGE_BIN_PATH=${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}
@@ -746,6 +746,14 @@ else
declare -r ED EPREFIX EROOT
fi
+ # If ${EBUILD_FORCE_TEST} == 1 and USE came from ${T}/environment
+ # then it might not have USE=test like it's supposed to here.
+ if [[ ${EBUILD_PHASE} == test ${EBUILD_FORCE_TEST} == 1
+ test =~ ${PORTAGE_IUSE} ]] ! has test ${USE} ; then
+ export USE=${USE} test
+ fi
+ declare -r USE
+
if [[ -n $EBUILD_SH_ARGS ]] ; then
(
# Don't allow subprocesses to inherit the pipe which
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index 2743e27..7bf4d63 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -1,5 +1,5 @@
#!/bin/bash
-# Copyright 1999-2013 Gentoo Foundation
+# Copyright 1999-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# Hardcoded bash lists are needed for backward compatibility with
@@ -489,13 +489,6 @@ __dyn_test() {
elif [[ ${EBUILD_FORCE_TEST} != 1 ]] ! has test ${FEATURES} ; then
__vecho Test phase [not enabled]: ${CATEGORY}/${PF}
else
- # If ${EBUILD_FORCE_TEST} == 1 and USE came from
${T}/environment
- # then it might not have USE=test like it's supposed to here.
- if [[ ${EBUILD_FORCE_TEST} == 1 test =~ ${PORTAGE_IUSE} ]]
\
- ! has test ${USE} ; then
- export USE=${USE} test
- fi
-
local save_sp=${SANDBOX_PREDICT}
addpredict /
__ebuild_phase pre_src_test
--
2.3.5
[gentoo-portage-dev] [PATCH] VdbMetadataDelta.applyDelta: remove replaced versions (bug 547532)
Since commit d800d224ab38c0f524d3fe858ebe201cbfa903c1, emerge --search
could randomly report incorrect results for the installed version
due to the replaced version of a given slot remaining in the cache.
Fixes: d800d224ab38 (Log changes between vdb_metadata.pickle updates)
X-Gentoo-Bug: 547532
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=547532
---
pym/_emerge/search.py | 10 --
pym/portage/dbapi/_VdbMetadataDelta.py | 22 +-
2 files changed, 29 insertions(+), 3 deletions(-)
diff --git a/pym/_emerge/search.py b/pym/_emerge/search.py
index e7f6f44..5a8143c 100644
--- a/pym/_emerge/search.py
+++ b/pym/_emerge/search.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2014 Gentoo Foundation
+# Copyright 1999-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
from __future__ import unicode_literals
@@ -442,7 +442,13 @@ class search(object):
def getInstallationStatus(self,package):
installed_package = self._vardb.match(package)
if installed_package:
- installed_package = installed_package[-1]
+ try:
+ self._vardb.match_unordered
+ except AttributeError:
+ installed_package = installed_package[-1]
+ else:
+ installed_package =
portage.best(installed_package)
+
else:
installed_package =
result =
diff --git a/pym/portage/dbapi/_VdbMetadataDelta.py
b/pym/portage/dbapi/_VdbMetadataDelta.py
index 3e3ff18..2dbb07a 100644
--- a/pym/portage/dbapi/_VdbMetadataDelta.py
+++ b/pym/portage/dbapi/_VdbMetadataDelta.py
@@ -1,4 +1,4 @@
-# Copyright 2014 Gentoo Foundation
+# Copyright 2014-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
import errno
@@ -8,6 +8,7 @@ import os
from portage import _encodings
from portage.util import atomic_ofstream
+from portage.versions import cpv_getkey
class VdbMetadataDelta(object):
@@ -138,10 +139,12 @@ class VdbMetadataDelta(object):
def applyDelta(self, data):
packages = self._vardb._aux_cache[packages]
+ added_slots = {}
for delta in data[deltas]:
cpv = delta[package] + - + delta[version]
event = delta[event]
if event == add:
+ added_slots[cpv] = delta
# Use aux_get to populate the cache
# for this cpv.
if cpv not in packages:
@@ -151,3 +154,20 @@ class VdbMetadataDelta(object):
pass
elif event == remove:
packages.pop(cpv, None)
+
+ # Remove replaced versions from updated slots
+ for cached_cpv, (mtime, metadata) in list(packages.items()):
+ if cached_cpv in added_slots:
+ continue
+ replaced = False
+ for cpv, delta in added_slots.items():
+ if (cached_cpv.startswith(delta[package]) and
+ metadata.get(SLOT) == delta[slot]
and
+ cpv_getkey(cached_cpv) ==
delta[package]):
+ replaced = True
+ break
+ if replaced:
+ del packages[cached_cpv]
+ del added_slots[cpv]
+ if not added_slots:
+ break
--
2.3.5
Re: [gentoo-portage-dev] emerge -a @preserved-rebuild wants to rebuild dev-java/oracle-jdk-bin-1.8.0.45
On 04/22/2015 04:07 PM, Joakim Tjernlund wrote: emerge -a @preserved-rebuild [ebuild Rf ] dev-java/oracle-jdk-bin-1.8.0.45 but this does not help, how can I see what @preserved-rebuild is looking for? Jocke Probably ffmpeg/libav libraries. Here's a bug report: https://bugs.gentoo.org/show_bug.cgi?id=460468 -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 2/2] MEDIUM: misc-functions: Be more verbose when removing INSTALL_MASK glob
On 04/19/2015 03:35 PM, Bertrand Jaquin wrote:
+ find ${root} \( -path ${no_inst} -or -name ${no_inst} \)
2 /dev/null \
+ | while read; do
+ __quiet_mode || einfo Removing /${REPLY#${root}}
+ rm -Rf ${REPLY} /dev/null
+ done
+
Please use find -print0 | while read -r -d '' to ensure that it works
will all possible paths.
--
Thanks,
Zac
[gentoo-portage-dev] [PATCH] _doebuild_path: add fallback for temp PORTAGE_BIN_PATH (bug 547086)
X-Gentoo-Bug: 547086 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=547086 --- pym/portage/package/ebuild/doebuild.py | 19 +-- 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py index 1be83ad..4d926c7 100644 --- a/pym/portage/package/ebuild/doebuild.py +++ b/pym/portage/package/ebuild/doebuild.py @@ -195,7 +195,10 @@ def _doebuild_path(settings, eapi=None): # Note: PORTAGE_BIN_PATH may differ from the global constant # when portage is reinstalling itself. - portage_bin_path = settings[PORTAGE_BIN_PATH] + portage_bin_path = [settings[PORTAGE_BIN_PATH]] + if portage_bin_path[0] != portage.const.PORTAGE_BIN_PATH: + # Add a fallback path for restarting failed builds (bug 547086) + portage_bin_path.append(portage.const.PORTAGE_BIN_PATH) eprefix = portage.const.EPREFIX prerootpath = [x for x in settings.get(PREROOTPATH, ).split(:) if x] rootpath = [x for x in settings.get(ROOTPATH, ).split(:) if x] @@ -210,18 +213,22 @@ def _doebuild_path(settings, eapi=None): path = overrides if xattr in settings.features: - path.append(os.path.join(portage_bin_path, ebuild-helpers, xattr)) + for x in portage_bin_path: + path.append(os.path.join(x, ebuild-helpers, xattr)) if uid != 0 and \ unprivileged in settings.features and \ fakeroot not in settings.features: - path.append(os.path.join(portage_bin_path, - ebuild-helpers, unprivileged)) + for x in portage_bin_path: + path.append(os.path.join(x, + ebuild-helpers, unprivileged)) if settings.get(USERLAND, GNU) != GNU: - path.append(os.path.join(portage_bin_path, ebuild-helpers, bsd)) + for x in portage_bin_path: + path.append(os.path.join(x, ebuild-helpers, bsd)) - path.append(os.path.join(portage_bin_path, ebuild-helpers)) + for x in portage_bin_path: + path.append(os.path.join(x, ebuild-helpers)) path.extend(prerootpath) for prefix in prefixes: -- 2.3.1
[gentoo-portage-dev] [PATCH] UseManager: handle newlines for USE_EXPAND prefixes (bug 546512)
From: Albert Safin xzf...@gmail.com
Since commit b65e1fc5fe723bd07bf034f6e08878fa3eb033cb, USE_EXPAND
prefixes affected subsequent lines of the same atom in package.use.
Fixes: b65e1fc5fe72 (Support USE_EXPAND prefixes in package.use and relevant
files)
X-Gentoo-Bug: 546512
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=546512
---
pym/portage/package/ebuild/_config/UseManager.py | 8 +++-
pym/portage/util/__init__.py | 10 +++---
2 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/pym/portage/package/ebuild/_config/UseManager.py
b/pym/portage/package/ebuild/_config/UseManager.py
index a93ea5c..6041802 100644
--- a/pym/portage/package/ebuild/_config/UseManager.py
+++ b/pym/portage/package/ebuild/_config/UseManager.py
@@ -208,6 +208,9 @@ class UseManager(object):
useflags = []
use_expand_prefix = ''
for prefixed_useflag in v:
+ if extended_syntax and prefixed_useflag == \n:
+ use_expand_prefix =
+ continue
if extended_syntax and prefixed_useflag[-1] ==
::
use_expand_prefix =
prefixed_useflag[:-1].lower() + _
continue
@@ -236,11 +239,14 @@ class UseManager(object):
ret = ExtendedAtomDict(dict)
if user_config:
pusedict = grabdict_package(
- os.path.join(location, file_name), recursive=1,
allow_wildcard=True, allow_repo=True, verify_eapi=False)
+ os.path.join(location, file_name), recursive=1,
newlines=1, allow_wildcard=True, allow_repo=True, verify_eapi=False)
for k, v in pusedict.items():
l = []
use_expand_prefix = ''
for flag in v:
+ if flag == \n:
+ use_expand_prefix =
+ continue
if flag[-1] == ::
use_expand_prefix =
flag[:-1].lower() + _
continue
diff --git a/pym/portage/util/__init__.py b/pym/portage/util/__init__.py
index 4b82e71..48cd1b7 100644
--- a/pym/portage/util/__init__.py
+++ b/pym/portage/util/__init__.py
@@ -340,7 +340,7 @@ def stack_lists(lists, incremental=1,
remember_source_file=False,
else:
return list(new_list)
-def grabdict(myfilename, juststrings=0, empty=0, recursive=0, incremental=1):
+def grabdict(myfilename, juststrings=0, empty=0, recursive=0, incremental=1,
newlines=0):
This function grabs the lines in a file, normalizes whitespace and
returns lines in a dictionary
@@ -354,6 +354,8 @@ def grabdict(myfilename, juststrings=0, empty=0,
recursive=0, incremental=1):
@type recursive: Boolean (integer)
@param incremental: Append to the return list, don't overwrite
@type incremental: Boolean (integer)
+ @param newlines: Append newlines
+ @type newlines: Boolean (integer)
@rtype: Dictionary
@return:
1. Returns the lines in a file in a dictionary, for example:
@@ -379,6 +381,8 @@ def grabdict(myfilename, juststrings=0, empty=0,
recursive=0, incremental=1):
continue
if len(myline) 1 and empty == 1:
continue
+ if newlines:
+ myline.append(\n)
if incremental:
newdict.setdefault(myline[0], []).extend(myline[1:])
else:
@@ -424,7 +428,7 @@ def read_corresponding_eapi_file(filename, default=0):
return default
return eapi
-def grabdict_package(myfilename, juststrings=0, recursive=0,
+def grabdict_package(myfilename, juststrings=0, recursive=0, newlines=0,
allow_wildcard=False, allow_repo=False, allow_build_id=False,
verify_eapi=False, eapi=None, eapi_default=0):
Does the same thing as grabdict except it validates keys
@@ -438,7 +442,7 @@ def grabdict_package(myfilename, juststrings=0, recursive=0,
atoms = {}
for filename in file_list:
d = grabdict(filename, juststrings=False,
- empty=True, recursive=False, incremental=True)
+ empty=True, recursive=False, incremental=True,
newlines=newlines)
if not d:
continue
if verify_eapi and eapi is None:
--
2.3.1
Re: [gentoo-portage-dev] [PATCHv2 2/2] MEDIUM: misc-functions: Be more verbose when removing INSTALL_MASK glob
On 04/19/2015 05:01 PM, Bertrand Jaquin wrote:
diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index 36a3bb8..4c37f10 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -276,7 +276,13 @@ install_mask() {
# we also need to handle globs (*.a, *.h, etc)
find ${root} \( -path ${no_inst} -or -name ${no_inst} \) \
- -exec rm -fR {} \; /dev/null 21
+ -print0 2 /dev/null \
+ | sort \
sort -z
Maybe also set LANG=C for locale independence?
--
Thanks,
Zac
