Re: [gentoo-portage-dev] [PATCH] ebuild: allow RESTRICT=network-sandbox in ebuilds
On 16 Jan 2018 23:32, Mike Gilbert wrote: > On Tue, Jan 16, 2018 at 4:46 PM, Mike Frysinger wrote: > > Some ebuilds are a bit hard to fix their use of the network in src > > phases, so allow them to disable things. This allows us to turn off > > access by default and for the vast majority while we work out how to > > fix the few broken packages. > > If we are going to allow network sandboxing to be disabled in > individual ebuilds, we should also allow the other sandboxes to be > disabled for the same reasons. sys-apps/sandbox has been notoriously > buggy, for example. that sandbox can already be disabled dynamically in ebuilds as needed. i don't really see a reason for it to be a RESTRICT value. -mike signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH v2] ebuild: allow RESTRICT=network-sandbox in ebuilds
From: Mike Frysinger <vap...@chromium.org> Some ebuilds are a bit hard to fix their use of the network in src phases, so allow them to disable things. This allows us to turn off access by default and for the vast majority while we work out how to fix the few broken packages. URL: https://crbug.com/731905 --- man/ebuild.5 | 4 pym/portage/package/ebuild/doebuild.py | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/man/ebuild.5 b/man/ebuild.5 index 42a0599fe3a9..1f2d4219e93e 100644 --- a/man/ebuild.5 +++ b/man/ebuild.5 @@ -689,6 +689,10 @@ binaries that are not compatible with debugedit. .I mirror files in \fBSRC_URI\fR will not be downloaded from the \fBGENTOO_MIRRORS\fR. .TP +.I network\-sandbox +Disables the network namespace for specific packages. +Should not be used in the main Gentoo tree. +.TP .I preserve\-libs Disables preserve\-libs for specific packages. Note than when a package is merged, RESTRICT=preserve\-libs applies if either the new instance or the diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py index 0be148fd408e..f75f11a1a424 100644 --- a/pym/portage/package/ebuild/doebuild.py +++ b/pym/portage/package/ebuild/doebuild.py @@ -148,7 +148,8 @@ def _doebuild_spawn(phase, settings, actionmap=None, **kwargs): kwargs['ipc'] = 'ipc-sandbox' not in settings.features or \ phase in _ipc_phases kwargs['networked'] = 'network-sandbox' not in settings.features or \ - phase in _networked_phases + phase in _networked_phases or \ + 'network-sandbox' in settings['PORTAGE_RESTRICT'].split() if phase == 'depend': kwargs['droppriv'] = 'userpriv' in settings.features -- 2.15.1
[gentoo-portage-dev] [PATCH] ebuild: allow RESTRICT=network-sandbox in ebuilds
From: Mike Frysinger <vap...@chromium.org> Some ebuilds are a bit hard to fix their use of the network in src phases, so allow them to disable things. This allows us to turn off access by default and for the vast majority while we work out how to fix the few broken packages. URL: https://crbug.com/731905 --- man/ebuild.5 | 4 pym/portage/package/ebuild/doebuild.py | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/man/ebuild.5 b/man/ebuild.5 index 42a0599fe3a9..a8fc3f86b68a 100644 --- a/man/ebuild.5 +++ b/man/ebuild.5 @@ -689,6 +689,10 @@ binaries that are not compatible with debugedit. .I mirror files in \fBSRC_URI\fR will not be downloaded from the \fBGENTOO_MIRRORS\fR. .TP +.I network-sandbox +Disables the network namespace for specific packages. +Should not be used in the main Gentoo tree. +.TP .I preserve\-libs Disables preserve\-libs for specific packages. Note than when a package is merged, RESTRICT=preserve\-libs applies if either the new instance or the diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py index 0be148fd408e..f75f11a1a424 100644 --- a/pym/portage/package/ebuild/doebuild.py +++ b/pym/portage/package/ebuild/doebuild.py @@ -148,7 +148,8 @@ def _doebuild_spawn(phase, settings, actionmap=None, **kwargs): kwargs['ipc'] = 'ipc-sandbox' not in settings.features or \ phase in _ipc_phases kwargs['networked'] = 'network-sandbox' not in settings.features or \ - phase in _networked_phases + phase in _networked_phases or \ + 'network-sandbox' in settings['PORTAGE_RESTRICT'].split() if phase == 'depend': kwargs['droppriv'] = 'userpriv' in settings.features -- 2.15.1
[gentoo-portage-dev] [PATCH v2] repoman: flag URIs using http:// when https:// is available
---
v2
- add more sites
- check the trailing URL to filter false positives
repoman/pym/repoman/modules/scan/ebuild/checks.py | 22 ++
repoman/pym/repoman/modules/scan/ebuild/errors.py | 2 ++
repoman/pym/repoman/qa_data.py| 4 +++-
3 files changed, 27 insertions(+), 1 deletion(-)
diff --git a/repoman/pym/repoman/modules/scan/ebuild/checks.py
b/repoman/pym/repoman/modules/scan/ebuild/checks.py
index 15e225156db4..83f9362b7506 100644
--- a/repoman/pym/repoman/modules/scan/ebuild/checks.py
+++ b/repoman/pym/repoman/modules/scan/ebuild/checks.py
@@ -682,6 +682,28 @@ class EMakeParallelDisabledViaMAKEOPTS(LineCheck):
error = errors.EMAKE_PARALLEL_DISABLED_VIA_MAKEOPTS
+class UriUseHttps(LineCheck):
+ """Check that we use https:// for known good sites."""
+ repoman_check_name = 'uri.https'
+ _SITES = (
+ '([-._a-zA-Z0-9]*\.)?apache\.org',
+ # Most FDO sites support https, but not all (like tango).
+ # List the most common ones here for now.
+
'((anongit|bugs|cgit|patchwork|people|specifications|www|xorg)\.)?freedesktop\.org',
+ '((bugs|dev|www)\.)?gentoo\.org',
+ 'github\.(io|com)',
+ 'savannah\.(non)?gnu\.org',
+ '((gcc|www)\.)?gnu\.org',
+ 'curl\.haxx\.se',
+ '(sf|sourceforge)\.net',
+ '(www\.)?sourceware\.org',
+ )
+ # Try to anchor the end of the URL so we don't get false positives
+ # with http://github.com.foo.bar.com/. Unlikely, but possible.
+ re = re.compile(r'.*\bhttp://(%s)(\s|["\'/]|$)' % r'|'.join(_SITES))
+ error = errors.URI_HTTPS
+
+
class NoAsNeeded(LineCheck):
"""Check for calls to the no-as-needed function."""
repoman_check_name = 'upstream.workaround'
diff --git a/repoman/pym/repoman/modules/scan/ebuild/errors.py
b/repoman/pym/repoman/modules/scan/ebuild/errors.py
index 3090de0d1a2c..14e47e35877e 100644
--- a/repoman/pym/repoman/modules/scan/ebuild/errors.py
+++ b/repoman/pym/repoman/modules/scan/ebuild/errors.py
@@ -47,3 +47,5 @@ USEQ_ERROR = (
'Ebuild calls deprecated useq function on line: %d')
HASQ_ERROR = (
'Ebuild calls deprecated hasq function on line: %d')
+URI_HTTPS = (
+ 'Ebuild uses http:// but should use https:// on line: %d')
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index 48ab389d086e..03711b6ed5d0 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -224,7 +224,8 @@ qahelp = {
"The ebuild makes use of an obsolete construct"),
"upstream.workaround": (
"The ebuild works around an upstream bug,"
- " an upstream bug should be filed and tracked in
bugs.gentoo.org")
+ " an upstream bug should be filed and tracked in
bugs.gentoo.org"),
+ "uri.https": "URI uses http:// but should use https://;,
}
qacats = list(qahelp)
@@ -273,6 +274,7 @@ qawarnings = set((
"LIVEVCS.stable",
"LIVEVCS.unmasked",
"IUSE.rubydeprecated",
+ "uri.https",
))
--
2.8.2
[gentoo-portage-dev] [PATCH] repoman: simplify wrapper for running locally
Running the repoman program locally is a bit unreliable in its
detection. Add a dedicated repoman.git program for people to
run directly if they want.
---
repoman/.repoman_not_installed | 0
repoman/bin/repoman | 10 --
repoman/bin/repoman.git | 37 +
repoman/pym/repoman/__init__.py | 4 ++--
repoman/setup.py| 10 --
5 files changed, 39 insertions(+), 22 deletions(-)
delete mode 100644 repoman/.repoman_not_installed
create mode 100755 repoman/bin/repoman.git
diff --git a/repoman/.repoman_not_installed b/repoman/.repoman_not_installed
deleted file mode 100644
index e69de29bb2d1..
diff --git a/repoman/bin/repoman b/repoman/bin/repoman
index 7082a968f502..66211b26bc5e 100755
--- a/repoman/bin/repoman
+++ b/repoman/bin/repoman
@@ -25,16 +25,6 @@ try:
except KeyboardInterrupt:
sys.exit(1)
-from os import path as osp
-here = osp.realpath(__file__)
-if osp.isfile(osp.join(osp.dirname(osp.dirname(here)),
".repoman_not_installed")):
- # Add the repoman subpkg
- pym_path = osp.join(osp.dirname(osp.dirname(here)), "pym")
- sys.path.insert(0, pym_path)
- if osp.isfile(osp.join(osp.dirname(osp.dirname(osp.dirname(here))),
".portage_not_installed")):
- # Add the base portage pkg
- pym_path =
osp.join(osp.dirname(osp.dirname(osp.dirname(here))), "pym")
- sys.path.insert(0, pym_path)
import portage
portage._internal_caller = True
diff --git a/repoman/bin/repoman.git b/repoman/bin/repoman.git
new file mode 100755
index ..67d4b17ba295
--- /dev/null
+++ b/repoman/bin/repoman.git
@@ -0,0 +1,37 @@
+#!/usr/bin/python -bO
+# Copyright 1999-2016 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+"""Run repoman from git using local modules/scripts."""
+
+from __future__ import print_function
+
+
+import os
+import sys
+
+
+def main(argv):
+ """The main entry point"""
+ source_root =
os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+
+ # Add the repoman source root.
+ pympath = os.path.join(source_root, 'pym')
+ pythonpath = os.environ.get('PYTHONPATH')
+ if pythonpath is None:
+ pythonpath = pympath
+ else:
+ pythonpath = pympath + ':' + pythonpath
+ # Add the portage source root.
+ pythonpath += ':' + os.path.join(os.path.dirname(source_root), 'pym')
+ os.environ['PYTHONPATH'] = pythonpath
+
+ # Signal to some repoman code that we're not installed.
+ os.environ['REPOMAN_NOT_INSTALLED'] = 'true'
+
+ cmd = [os.path.join(source_root, 'bin', 'repoman')]
+ os.execvp(cmd[0], cmd + argv)
+
+
+if __name__ == '__main__':
+ main(sys.argv[1:])
diff --git a/repoman/pym/repoman/__init__.py b/repoman/pym/repoman/__init__.py
index 5f0f9f873b1b..40259e61f92a 100644
--- a/repoman/pym/repoman/__init__.py
+++ b/repoman/pym/repoman/__init__.py
@@ -1,6 +1,6 @@
-import os.path
+import os
REPOMAN_BASE_PATH = os.path.join(os.sep,
os.sep.join(os.path.realpath(__file__.rstrip("co")).split(os.sep)[:-3]))
-_not_installed = os.path.isfile(os.path.join(REPOMAN_BASE_PATH,
".repoman_not_installed"))
+_not_installed = os.environ.get('REPOMAN_NOT_INSTALLED') == 'true'
diff --git a/repoman/setup.py b/repoman/setup.py
index 47ed15574de0..930d61060baf 100755
--- a/repoman/setup.py
+++ b/repoman/setup.py
@@ -145,11 +145,6 @@ class x_clean(clean):
print('removing %s symlink' % repr(conf_dir))
os.unlink(conf_dir)
- pni_file = os.path.join(top_dir, '.repoman_not_installed')
- if os.path.exists(pni_file):
- print('removing %s' % repr(pni_file))
- os.unlink(pni_file)
-
def clean_man(self):
man_dir = os.path.join(self.build_base, 'man')
if os.path.exists(man_dir):
@@ -350,11 +345,6 @@ class build_tests(x_build_scripts_custom):
print('Symlinking %s -> %s' % (conf_dir, conf_src))
os.symlink(conf_src, conf_dir)
- # create $build_lib/../.repoman_not_installed
- # to enable proper paths in tests
- with open(os.path.join(self.top_dir, '.repoman_not_installed'),
'w'):
- pass
-
class test(Command):
""" run tests """
--
2.8.2
Re: [gentoo-portage-dev] [PATCH] repoman: filter out duplicate dependencies in error messages
On 04 Jan 2016 19:17, Brian Dolbec wrote: > I immediately want to say REJECT!, REJECT!, REJECT!,... > ... > I would much prefer you re-base your patch on the rewrite code. reject != delay ... i don't mind waiting and rebasing -mike signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH] repoman: filter out duplicate dependencies in error messages
Some packages list the same atom multiple times (e.g. behind diff USE flags). If one of them throws an error, we end up listing it more than once, and the output can get verbose/useless. --- pym/repoman/scanner.py | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/pym/repoman/scanner.py b/pym/repoman/scanner.py index d1c10d7..94ada90 100644 --- a/pym/repoman/scanner.py +++ b/pym/repoman/scanner.py @@ -704,13 +704,22 @@ class Scanner(object): # we have some unsolvable deps # remove ! deps, which always show up as unsatisfiable - atoms = [ + all_atoms = [ str(atom.unevaluated_atom) for atom in atoms if not atom.blocker] # if we emptied out our list, continue: - if not atoms: + if not all_atoms: continue + + # Filter out duplicates. We do this by hand (rather + # than use a set) so the order is stable and better + # matches the order that's in the ebuild itself. + atoms = [] + for atom in all_atoms: + if atom not in atoms: + atoms.append(atom) + if self.options.output_style in ['column']: self.qatracker.add_error(mykey, "%s: %s: %s(%s) %s" -- 2.6.2
Re: [gentoo-portage-dev] cat/pkg::x-portage ?
On 24 Nov 2015 15:41, Joakim Tjernlund wrote: > Emerging on my embedded gentoo ppc target I see this: > > dev-lang/python-exec-2.0.1-r1::x-portage > sys-apps/install-xattr-0.5::x-portage > sys-libs/timezone-data-2015f::x-portage > > Where does the name x-portage come from? I do not have a repo > named x-portage, it is still named gentoo. it has no name so one is auto-created using the path. pym/portage/repository/config.py: @staticmethod def _read_valid_repo_name(repo_path): name, missing = RepoConfig._read_repo_name(repo_path) # We must ensure that the name conforms to PMS 3.1.5 # in order to avoid InvalidAtom exceptions when we # use it to generate atoms. name = _gen_valid_repo(name) if not name: # name only contains invalid characters name = "x-" + os.path.basename(repo_path) name = _gen_valid_repo(name) # If basename only contains whitespace then the # end result is name = 'x-'. return name, missing @staticmethod def _read_repo_name(repo_path): """ Read repo_name from repo_path. Returns repo_name, missing. """ repo_name_path = os.path.join(repo_path, REPO_NAME_LOC) f = None try: f = io.open( _unicode_encode(repo_name_path, encoding=_encodings['fs'], errors='strict'), mode='r', encoding=_encodings['repo.content'], errors='replace') return f.readline().strip(), False except EnvironmentError: return "x-" + os.path.basename(repo_path), True finally: if f is not None: f.close() -mike signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH/RFC] userinstall: new feature for running src_install as non root
This does not support fowners just yet as we'll need to queue/dequeue
the accounts on the fly.
X-Gentoo-Bug: 566614
X-Gentoo-Bug-URL: https://bugs.gentoo.org/566614
---
man/make.conf.5| 4
pym/portage/const.py | 1 +
pym/portage/package/ebuild/config.py | 22 +-
pym/portage/package/ebuild/doebuild.py | 9 ++---
4 files changed, 20 insertions(+), 16 deletions(-)
diff --git a/man/make.conf.5 b/man/make.conf.5
index 1d1cfeb..0c575db 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -648,6 +648,10 @@ checksum differs from the file that was originally
installed.
When portage is run as root, drop privileges to portage:portage during the
fetching of package sources.
.TP
+.B userinstall
+When portage is run as root, drop privileges to portage:portage during the
+install of package sources.
+.TP
.B userpriv
Allow portage to drop root privileges and compile packages as
portage:portage without a sandbox (unless \fIusersandbox\fR is also used).
diff --git a/pym/portage/const.py b/pym/portage/const.py
index 6c4f613..d895633 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -195,6 +195,7 @@ SUPPORTED_FEATURES = frozenset([
"unmerge-orphans",
"unprivileged",
"userfetch",
+ "userinstall",
"userpriv",
"usersandbox",
"usersync",
diff --git a/pym/portage/package/ebuild/config.py
b/pym/portage/package/ebuild/config.py
index 40aa99d..70f2276 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -1152,21 +1152,17 @@ class config(object):
"fakeroot binary is not installed.\n"),
noiselevel=-1)
if os.getuid() == 0 and not hasattr(os, "setgroups"):
- warning_shown = False
-
- if "userpriv" in self.features:
- writemsg(_("!!! FEATURES=userpriv is enabled,
but "
- "os.setgroups is not available.\n"),
noiselevel=-1)
- warning_shown = True
-
- if "userfetch" in self.features:
- writemsg(_("!!! FEATURES=userfetch is enabled,
but "
- "os.setgroups is not available.\n"),
noiselevel=-1)
+ userfeatures = {'userfetch', 'userinstall', 'userpriv'}
+ enabled = userfeatures & self.features
+ if enabled:
+ writemsg(
+ _('!!! FEATURES="%s" is enabled, but
os.setgroups is not available.\n'),
+ ' '.join(enabled), noiselevel=-1)
warning_shown = True
- if warning_shown and platform.python_implementation()
== 'PyPy':
- writemsg(_("!!! See
https://bugs.pypy.org/issue833 for details.\n"),
- noiselevel=-1)
+ if platform.python_implementation() == 'PyPy':
+ writemsg(_("!!! See
https://bugs.pypy.org/issue833 for details.\n"),
+ noiselevel=-1)
def load_best_module(self,property_string):
best_mod =
best_from_dict(property_string,self.modules,self.module_priority)
diff --git a/pym/portage/package/ebuild/doebuild.py
b/pym/portage/package/ebuild/doebuild.py
index ff8958e..0fa5c20 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -1350,6 +1350,9 @@ def _spawn_actionmap(settings):
droppriv = "userpriv" in features and \
"userpriv" not in restrict and \
secpass >= 2
+ instpriv = "userinstall" in features and \
+ "userinstall" not in restrict and \
+ secpass >= 2
fakeroot = "fakeroot" in features
@@ -1370,9 +1373,9 @@ def _spawn_actionmap(settings):
"configure":{"cmd":ebuild_sh, "args":{"droppriv":droppriv, "free":nosandbox,
"sesandbox":sesandbox, "fakeroot":0}},
"compile": {"cmd":ebuild_sh, "args":{"droppriv":droppriv, "free":nosandbox,
"sesandbox":sesandbox, "fakeroot":0}},
"test": {"cmd":ebuild_sh, "args":{"droppriv":droppriv, "free":nosandbox,
"sesandbox":sesandbox, "fakeroot":0}},
-"install": {"cmd":ebuild_sh, "args":{"droppriv":0,"free":0,
"sesandbox":sesandbox, "fakeroot":fakeroot}},
-"rpm": {"cmd":misc_sh, "args":{"droppriv":0,"free":0,
"sesandbox":0, "fakeroot":fakeroot}},
-"package": {"cmd":misc_sh, "args":{"droppriv":0,"free":0,
"sesandbox":0, "fakeroot":fakeroot}},
+"install": {"cmd":ebuild_sh, "args":{"droppriv":instpriv, "free":0,
"sesandbox":sesandbox, "fakeroot":fakeroot}},
+"rpm":
Re: [gentoo-portage-dev] [PATCH/RFC] userinstall: new feature for running src_install as non root
On 24 Nov 2015 15:58, Zac Medico wrote: > On 11/24/2015 03:11 PM, Mike Frysinger wrote: > > This does not support fowners just yet as we'll need to queue/dequeue > > the accounts on the fly. > > > > X-Gentoo-Bug: 566614 > > X-Gentoo-Bug-URL: https://bugs.gentoo.org/566614 > > --- > > man/make.conf.5| 4 > > pym/portage/const.py | 1 + > > pym/portage/package/ebuild/config.py | 22 +- > > pym/portage/package/ebuild/doebuild.py | 9 ++--- > > 4 files changed, 20 insertions(+), 16 deletions(-) > > In pym/repoman/qa_data.py you need to add userinstall to valid_restrict. > > Otherwise, looks good. suggestions for having fowners queue data ? should it just write to a file in $T and then have portage read commands out of that ? -mike signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH] ebuild: unset all funcs/vars that start with ___
Since the __* (two) namespace is reserved, and ___* (three) has rarely
(if ever) been used in ebuilds, we can nuke all funcs/vars that start
with that. It makes clean up easier for us.
---
bin/save-ebuild-env.sh | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index 31c2d60..ddef1fd 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -89,7 +89,9 @@ __save_ebuild_env() {
___eapi_has_package_manager_build_user && unset -f
package_manager_build_user
___eapi_has_package_manager_build_group && unset -f
package_manager_build_group
- unset -f $(compgen -A function ___eapi_)
+ # Clear out the triple underscore namespace as it is reserved by the PM.
+ unset -f $(compgen -A function ___)
+ unset ${!___*}
# portage config variables and variables set directly by portage
unset ACCEPT_LICENSE BAD BRACKET BUILD_PREFIX COLS \
--
2.6.2
Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels
On 12 Nov 2015 16:58, Zac Medico wrote: > On 11/12/2015 04:06 PM, Mike Frysinger wrote: > > from ebuilds/eclasses that have already stopped using __: > > __do_sed_fix () > > ___ECLASS_RECUR_MULTILIB=yes > > ___ECLASS_RECUR_TOOLCHAIN_FUNCS=yes > > __versionator_shopt_toggle () > > __versionator__test_version_compare () > > __versionator__test_version_is_at_least () > > > > grepping the tree, i see like two packages and one eclass still using __. > > both of which are trivial to convert. > > Sure, but do we really want to confuse people who might be ignorant of > this rule? Having functions disappear from the environment without > warning is very likely to cause confusion... that already happens to a degree if you happen to use a name that portage uses itself. we can add a repoman check, and if we think it comes up enough, have portage itself warn when it blows away things it didn't register. > Also, there's the > element of backward-compatibility for any __* functions in > /var/db/pkg/*/*/environment.bz2 of users' installed systems that might > be needed during pkg_prerm and pkg_postrm. the ones i highlighted are not needed for those purposes. you're right it could be a problem, i think the likelihood is low considering how infrequently these two are used, and how much we push people to use other phases instead. > >> Also note that some internals have been intentionally preserved in > >> environment.bz2. For example, __eapi6_src_install exposes the default > >> src_install implementation, which someone might examine for debugging > >> purposes. > > > > is that actually useful ? i can't see how it would be. > > Shrug, probably not (unless there's a bug in a particular > implementation, and someone wants to go back and check which > implementation was used for a particular installed package). if we want to tag that kind of metadata in a build, we should just explicitly include something like PORTAGE_VERSION. -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels
On 12 Nov 2015 21:07, Tim Harder wrote: > On 2015-11-11 14:42, Zac Medico wrote: > > Please unset all new internal function inside bin/save-ebuild-env.sh. > > Note that it already uses this line to unset functions beginning with > > ___eapi: > > > >unset -f $(compgen -A function ___eapi_) > > > > However, your __eapi functions will not be matched because they only > > begin with 2 underscores. > > Just to note another approach, pkgcore generates global and per-eapi > function lists at install time (or uses the generation scripts when > running from a checkout) so manually tracking lists of functions isn't > required. > > The main reason I set that up was because I sometimes forgot to add > functions to a similar list that was previously used. :) > > I'm not overly familiar with portage's bash code so I'm not sure if > that's feasible with its current structure. i was thinking of building lists too, but i'm not entirely sure how to tease that out of portage. -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels
On 11 Nov 2015 10:32, Michał Górny wrote: > I'm not convinced we ought to do this for EAPI < 6. It is a breaking > change after all, and as such changes the behavior of EAPI < 6 ebuilds. that is a false statement. anything not working with bash-3.2 is already broken according to the PMS. > There are some ebuilds/eclasses that have bash version checks, > and execute bash-4 code when bash-4 is available. As far as I > understand, this will effectively prohibit bash-4 code even though > BASH_VERSINFO will still indicate bash-4 is being used. no, that's not what it does. it changes behavior for code that itself has changed behavior between versions. it does not disable any newer functionality. it would have been nice to have a knob that would turn off newer functionality as well, but upstream didn't seem keen on it. -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels
On 11 Nov 2015 07:33, Ulrich Mueller wrote:
> >>>>> On Tue, 10 Nov 2015, Mike Frysinger wrote:
> > + # Set the compat level in case things change with newer ones. We must
> > not
> > + # export this into the env otherwise we might break other shell
> > scripts we
> > + # execute (e.g. ones in /usr/bin).
> > + BASH_COMPAT="${maj}.${min}"
> > +
> > + # The variable above is new to bash-4.3. For older versions, we have
> > to use
> > + # a compat knob. Further, the compat knob only exists with older
> > versions
> > + # (e.g. bash-4.3 has compat42 but not compat43). This means we only
> > need to
> > + # turn the knob with older EAPIs, and only when running newer bash
> > versions:
> > + # there is no bash-3.3 (it went 3.2 to 4.0), and when requiring
> > bash-4.2, the
> > + # var works with bash-4.3+, and you don't need to set compat to 4.2
> > when you
> > + # are already running 4.2.
> > + if __eapi_bash_3_2 && [[ ${BASH_VERSINFO[0]} -gt 3 ]] ; then
> > + shopt -s compat32
> > + fi
>
> Wouldn't this profit from an additional test for <bash-4.3? If I
> understood the upstream discussion correctly, they were thinking about
> dropping the compat* options in some future version?
my take away was that they weren't going to be adding new compat levels.
i don't think they were planning on dropping existing ones.
-mike
signature.asc
Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels
On 11 Nov 2015 13:04, Zac Medico wrote:
> On 11/11/2015 12:55 PM, Mike Frysinger wrote:
> > On 11 Nov 2015 11:42, Zac Medico wrote:
> >> On 11/10/2015 08:39 PM, Mike Frysinger wrote:
> >>> +# Set up the bash version compatibility level.
> >>> +__check_bash_version() {
> >>
> >> Please unset all new internal function inside bin/save-ebuild-env.sh.
> >> Note that it already uses this line to unset functions beginning with
> >> ___eapi:
> >>
> >>unset -f $(compgen -A function ___eapi_)
> >
> > why don't we create a new namespace for portage funcs ? something like
> > __e* ?
>
> That works for me. According to PMS, we're free to do anything we want
> as long as it begins with at least 2 underscores.
interesting. why don't we just unmap all things that begin with 2 underscores
then ? or maybe 3 ?
-mike
signature.asc
Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels
On 11 Nov 2015 11:42, Zac Medico wrote:
> On 11/10/2015 08:39 PM, Mike Frysinger wrote:
> > +# Set up the bash version compatibility level.
> > +__check_bash_version() {
>
> Please unset all new internal function inside bin/save-ebuild-env.sh.
> Note that it already uses this line to unset functions beginning with
> ___eapi:
>
>unset -f $(compgen -A function ___eapi_)
why don't we create a new namespace for portage funcs ? something like __e* ?
> However, your __eapi functions will not be matched because they only
> begin with 2 underscores.
that wasn't intentional. i'll change it to 3.
-mike
signature.asc
Description: Digital signature
[gentoo-portage-dev] [PATCH v2] ebuild: set up bash compat levels
To try and provide better stability across bash versions,
set the language compat level based on the current EAPI.
This does not ban newer features, it tells bash to use
the older bash behavior when the behavior changes across
versions.
---
bin/eapi.sh| 8
bin/ebuild.sh | 42 ++
bin/save-ebuild-env.sh | 2 +-
3 files changed, 51 insertions(+), 1 deletion(-)
diff --git a/bin/eapi.sh b/bin/eapi.sh
index 528e6f2..cd3e1a4 100644
--- a/bin/eapi.sh
+++ b/bin/eapi.sh
@@ -191,3 +191,11 @@ ___eapi_enables_failglob_in_global_scope() {
___eapi_enables_globstar() {
[[ ${1-${EAPI-0}} =~ ^(4-python|5-progress)$ ]]
}
+
+___eapi_bash_3_2() {
+ [[ ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+}
+
+___eapi_bash_4_2() {
+ [[ ${1-${EAPI-0}} =~ ^(6)$ ]]
+}
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 75a9d24..78a93f0 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -6,8 +6,50 @@
# Make sure it's before everything so we don't mess aliases that follow.
unalias -a
+# Make sure this isn't exported to scripts we execute.
+unset BASH_COMPAT
+
source "${PORTAGE_BIN_PATH}/isolated-functions.sh" || exit 1
+# Set up the bash version compatibility level. This does not disable
+# features when running with a newer version, but makes it so that when
+# bash changes behavior in an incompatible way, the older behavior is
+# used instead.
+__check_bash_version() {
+ # Figure out which min version of bash we require.
+ local maj min
+ if ___eapi_bash_3_2 ; then
+ maj=3 min=2
+ elif ___eapi_bash_4_2 ; then
+ maj=4 min=2
+ else
+ return
+ fi
+
+ # Make sure the active bash is sane.
+ if [[ ${BASH_VERSINFO[0]} -lt ${maj} ]] ||
+ [[ ${BASH_VERSINFO[0]} -eq ${maj} && ${BASH_VERSINFO[1]} -lt ${min}
]] ; then
+ die ">=bash-${maj}.${min} is required"
+ fi
+
+ # Set the compat level in case things change with newer ones. We must
not
+ # export this into the env otherwise we might break other shell
scripts we
+ # execute (e.g. ones in /usr/bin).
+ BASH_COMPAT="${maj}.${min}"
+
+ # The variable above is new to bash-4.3. For older versions, we have
to use
+ # a compat knob. Further, the compat knob only exists with older
versions
+ # (e.g. bash-4.3 has compat42 but not compat43). This means we only
need to
+ # turn the knob with older EAPIs, and only when running newer bash
versions:
+ # there is no bash-3.3 (it went 3.2 to 4.0), and when requiring
bash-4.2, the
+ # var works with bash-4.3+, and you don't need to set compat to 4.2
when you
+ # are already running 4.2.
+ if ___eapi_bash_3_2 && [[ ${BASH_VERSINFO[0]} -gt 3 ]] ; then
+ shopt -s compat32
+ fi
+}
+__check_bash_version
+
if [[ $EBUILD_PHASE != depend ]] ; then
source "${PORTAGE_BIN_PATH}/phase-functions.sh" || die
source "${PORTAGE_BIN_PATH}/save-ebuild-env.sh" || die
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index 477ed28..1120297 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -75,7 +75,7 @@ __save_ebuild_env() {
__ebuild_main __ebuild_phase __ebuild_phase_with_hooks \
__ebuild_arg_to_phase __ebuild_phase_funcs default \
__unpack_tar __unset_colors \
- __source_env_files __try_source \
+ __source_env_files __try_source __check_bash_version \
__eqaquote __eqatag \
${QA_INTERCEPTORS}
--
2.6.2
[gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels
To try and provide better stability across bash versions,
set the language compat level based on the current EAPI.
---
bin/eapi.sh | 8
bin/ebuild.sh | 39 +++
2 files changed, 47 insertions(+)
diff --git a/bin/eapi.sh b/bin/eapi.sh
index 528e6f2..b236344 100644
--- a/bin/eapi.sh
+++ b/bin/eapi.sh
@@ -191,3 +191,11 @@ ___eapi_enables_failglob_in_global_scope() {
___eapi_enables_globstar() {
[[ ${1-${EAPI-0}} =~ ^(4-python|5-progress)$ ]]
}
+
+__eapi_bash_3_2() {
+ [[ ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+}
+
+__eapi_bash_4_2() {
+ [[ ${1-${EAPI-0}} =~ ^(6)$ ]]
+}
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 75a9d24..2d09fb8 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -6,8 +6,47 @@
# Make sure it's before everything so we don't mess aliases that follow.
unalias -a
+# Make sure this isn't exported to scripts we execute.
+unset BASH_COMPAT
+
source "${PORTAGE_BIN_PATH}/isolated-functions.sh" || exit 1
+# Set up the bash version compatibility level.
+__check_bash_version() {
+ # Figure out which min version of bash we require.
+ local maj min
+ if __eapi_bash_3_2 ; then
+ maj=3 min=2
+ elif __eapi_bash_4_2 ; then
+ maj=4 min=2
+ else
+ return
+ fi
+
+ # Make sure the active bash is sane.
+ if [[ ${BASH_VERSINFO[0]} -lt ${maj} ]] ||
+ [[ ${BASH_VERSINFO[0]} -eq ${maj} && ${BASH_VERSINFO[1]} -lt ${min}
]] ; then
+ die ">=bash-${maj}.${min} is required"
+ fi
+
+ # Set the compat level in case things change with newer ones. We must
not
+ # export this into the env otherwise we might break other shell
scripts we
+ # execute (e.g. ones in /usr/bin).
+ BASH_COMPAT="${maj}.${min}"
+
+ # The variable above is new to bash-4.3. For older versions, we have
to use
+ # a compat knob. Further, the compat knob only exists with older
versions
+ # (e.g. bash-4.3 has compat42 but not compat43). This means we only
need to
+ # turn the knob with older EAPIs, and only when running newer bash
versions:
+ # there is no bash-3.3 (it went 3.2 to 4.0), and when requiring
bash-4.2, the
+ # var works with bash-4.3+, and you don't need to set compat to 4.2
when you
+ # are already running 4.2.
+ if __eapi_bash_3_2 && [[ ${BASH_VERSINFO[0]} -gt 3 ]] ; then
+ shopt -s compat32
+ fi
+}
+__check_bash_version
+
if [[ $EBUILD_PHASE != depend ]] ; then
source "${PORTAGE_BIN_PATH}/phase-functions.sh" || die
source "${PORTAGE_BIN_PATH}/save-ebuild-env.sh" || die
--
2.6.2
Re: [gentoo-portage-dev] [PATCH] xpak-helper: rewrite to rely more on argparse
On 01 Nov 2015 09:36, Zac Medico wrote:
> On 10/31/2015 10:23 PM, Mike Frysinger wrote:
> > The current code implements a lot of ad-hoc argument parsing when it
> > could simply let the argparse module do it all for it. This makes the
> > code easier to understand and extend in the process.
> > ---
> > bin/xpak-helper.py | 68
> > --
> > 1 file changed, 25 insertions(+), 43 deletions(-)
> >
> > diff --git a/bin/xpak-helper.py b/bin/xpak-helper.py
> > index 8c965ec..1b2883d 100755
> > --- a/bin/xpak-helper.py
> > +++ b/bin/xpak-helper.py
> [snip]
> > -def main(argv):
> >
> > - if argv and isinstance(argv[0], bytes):
> > - for i, x in enumerate(argv):
> > - argv[i] = portage._unicode_decode(x, errors='strict')
>
> You've dropped the _unicode_decode call.
i did on purpose. i should have mentioned that.
> In order to handle python3 with arguments containing UTF-8 characters
> (in ${PKGDIR}) and a mis-matched sys.getfilesystemencoding() value, it's
> safest to decode the arguments like chmod-lite.py does.
it seems wrong that we have incomplete coverage here.
some tools do it and some do not.
> We should create
> a function for this code which is also duplicated in install.py:
you mean portage._decode_argv ?
what if we create a new module like "commandline" that provides an
ArgumentParser interface that takes care of this for us ?
-mike
signature.asc
Description: Digital signature
[gentoo-portage-dev] [PATCH] xpak-helper: rewrite to rely more on argparse
The current code implements a lot of ad-hoc argument parsing when it
could simply let the argparse module do it all for it. This makes the
code easier to understand and extend in the process.
---
bin/xpak-helper.py | 68 --
1 file changed, 25 insertions(+), 43 deletions(-)
diff --git a/bin/xpak-helper.py b/bin/xpak-helper.py
index 8c965ec..1b2883d 100755
--- a/bin/xpak-helper.py
+++ b/bin/xpak-helper.py
@@ -2,68 +2,50 @@
# Copyright 2009-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
+"""Helper to manage xpak archives"""
+
import argparse
import sys
import portage
portage._internal_caller = True
from portage import os
-def command_recompose(args):
-
- usage = "usage: recompose \n"
- if len(args) != 2:
- sys.stderr.write(usage)
- sys.stderr.write("2 arguments are required, got %s\n" %
len(args))
- return 1
+def command_recompose(parser, opts):
+ """Create an xpak archive from a directory"""
+ binpkg_path = opts.binpkg_path
+ metadata_dir = opts.metadata_dir
- binpkg_path, metadata_dir = args
-
- if not os.path.isfile(binpkg_path):
- sys.stderr.write(usage)
- sys.stderr.write("Argument 1 is not a regular file: '%s'\n" % \
- binpkg_path)
- return 1
+ if os.path.exists(binpkg_path) and not os.path.isfile(binpkg_path):
+ parser.error('binpkg_path is not a regular file: %s' %
binpkg_path)
if not os.path.isdir(metadata_dir):
- sys.stderr.write(usage)
- sys.stderr.write("Argument 2 is not a directory: '%s'\n" % \
- metadata_dir)
- return 1
+ parser.error('metadata_dir is not a directory: %s' %
metadata_dir)
t = portage.xpak.tbz2(binpkg_path)
t.recompose(metadata_dir)
return os.EX_OK
-def main(argv):
- if argv and isinstance(argv[0], bytes):
- for i, x in enumerate(argv):
- argv[i] = portage._unicode_decode(x, errors='strict')
+def get_parser():
+ """Return the command line parser"""
+ parser = argparse.ArgumentParser(description=__doc__)
+ subparsers = parser.add_subparsers()
- valid_commands = ('recompose',)
- description = "Perform metadata operations on a binary package."
- usage = "usage: %s COMMAND [args]" % \
- os.path.basename(argv[0])
+ subparser = subparsers.add_parser('recompose',
+ help=command_recompose.__doc__)
+ subparser.set_defaults(func=command_recompose)
+ subparser.add_argument('binpkg_path', help='Path to output xpak')
+ subparser.add_argument('metadata_dir', help='Path to input dir')
- parser = argparse.ArgumentParser(description=description, usage=usage)
- options, args = parser.parse_known_args(argv[1:])
+ return parser
- if not args:
- parser.error("missing command argument")
- command = args[0]
-
- if command not in valid_commands:
- parser.error("invalid command: '%s'" % command)
-
- if command == 'recompose':
- rval = command_recompose(args[1:])
- else:
- raise AssertionError("invalid command: '%s'" % command)
+def main(argv):
+ parser = get_parser()
+ opts = parser.parse_args(argv)
+ opts.func(parser, opts)
- return rval
-if __name__ == "__main__":
- rval = main(sys.argv[:])
- sys.exit(rval)
+if __name__ == '__main__':
+ sys.exit(main(sys.argv[1:]))
--
2.5.2
[gentoo-portage-dev] [PATCH v2] runtests: create a global tempdir to hold subtest files
A lot of unittests currently leak content in /tmp when they run.
Rather than explicitly track down every failing test (which we can
do regardless of this), have the runtest runner create a global
tempdir and use that as a base for children tests. Then when the
runtest script finishes, it takes care of nuking everything.
---
v2
- make tempdir setup more robust per Zac
runtests | 48 +---
1 file changed, 33 insertions(+), 15 deletions(-)
diff --git a/runtests b/runtests
index 36243c6..89c6e14 100755
--- a/runtests
+++ b/runtests
@@ -15,8 +15,10 @@ from __future__ import print_function
import argparse
import os
-import sys
+import shutil
import subprocess
+import sys
+import tempfile
# These are the versions we fully support and require to pass tests.
@@ -91,6 +93,8 @@ $ %(prog)s pym/portage/tests/xpak/test_decodeint.py
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=epilog)
+ parser.add_argument('--keep-temp', default=False, action='store_true',
+ help='Do not delete the temporary directory when exiting')
parser.add_argument('--color', type=str, default=None,
help='Whether to use colorized output (default is auto)')
parser.add_argument('--python-versions', action='append',
@@ -116,20 +120,34 @@ def main(argv):
else:
pyversions.extend(ver.split())
- # Actually test those versions now.
- statuses = []
- for ver in pyversions:
- prog = get_python_executable(ver)
- cmd = [prog, '-b', '-Wd', 'pym/portage/tests/runTests.py'] +
args
- if os.access(prog, os.X_OK):
- print('%sTesting with Python %s...%s' %
- (colors.GOOD, ver, colors.NORMAL))
- statuses.append(subprocess.call(cmd))
- elif not ignore_missing:
- print('%sCould not find requested Python %s%s' %
- (colors.BAD, ver, colors.NORMAL))
- statuses.append(1)
- print()
+ tempdir = None
+ try:
+ # Set up a single tempdir for all the tests to use.
+ # This way we know the tests won't leak things on us.
+ tempdir = tempfile.mkdtemp(prefix='portage.runtests.')
+ os.environ['TMPDIR'] = tempdir
+
+ # Actually test those versions now.
+ statuses = []
+ for ver in pyversions:
+ prog = get_python_executable(ver)
+ cmd = [prog, '-b', '-Wd',
'pym/portage/tests/runTests.py'] + args
+ if os.access(prog, os.X_OK):
+ print('%sTesting with Python %s...%s' %
+ (colors.GOOD, ver, colors.NORMAL))
+ statuses.append(subprocess.call(cmd))
+ elif not ignore_missing:
+ print('%sCould not find requested Python %s%s' %
+ (colors.BAD, ver, colors.NORMAL))
+ statuses.append(1)
+ print()
+ finally:
+ if tempdir is not None:
+ if opts.keep_temp:
+ print('Temporary directory left behind:\n%s' %
tempdir)
+ else:
+ # Nuke our tempdir and anything that might be
under it.
+ shutil.rmtree(tempdir, True)
# Then summarize it all.
print('\nSummary:\n')
--
2.5.2
[gentoo-portage-dev] [PATCH] runtests: create a global tempdir to hold subtest files
A lot of unittests currently leak content in /tmp when they run.
Rather than explicitly track down every failing test (which we can
do regardless of this), have the runtest runner create a global
tempdir and use that as a base for children tests. Then when the
runtest script finishes, it takes care of nuking everything.
---
runtests | 46 +++---
1 file changed, 31 insertions(+), 15 deletions(-)
diff --git a/runtests b/runtests
index 36243c6..fc6de0a 100755
--- a/runtests
+++ b/runtests
@@ -15,8 +15,10 @@ from __future__ import print_function
import argparse
import os
-import sys
+import shutil
import subprocess
+import sys
+import tempfile
# These are the versions we fully support and require to pass tests.
@@ -91,6 +93,8 @@ $ %(prog)s pym/portage/tests/xpak/test_decodeint.py
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=epilog)
+ parser.add_argument('--keep-temp', default=False, action='store_true',
+ help='Do not delete the temporary directory when exiting')
parser.add_argument('--color', type=str, default=None,
help='Whether to use colorized output (default is auto)')
parser.add_argument('--python-versions', action='append',
@@ -116,20 +120,32 @@ def main(argv):
else:
pyversions.extend(ver.split())
- # Actually test those versions now.
- statuses = []
- for ver in pyversions:
- prog = get_python_executable(ver)
- cmd = [prog, '-b', '-Wd', 'pym/portage/tests/runTests.py'] +
args
- if os.access(prog, os.X_OK):
- print('%sTesting with Python %s...%s' %
- (colors.GOOD, ver, colors.NORMAL))
- statuses.append(subprocess.call(cmd))
- elif not ignore_missing:
- print('%sCould not find requested Python %s%s' %
- (colors.BAD, ver, colors.NORMAL))
- statuses.append(1)
- print()
+ try:
+ # Set up a single tempdir for all the tests to use.
+ # This way we know the tests won't leak things on us.
+ tempdir = tempfile.mkdtemp(prefix='portage.runtests.')
+ os.environ['TMPDIR'] = tempdir
+
+ # Actually test those versions now.
+ statuses = []
+ for ver in pyversions:
+ prog = get_python_executable(ver)
+ cmd = [prog, '-b', '-Wd',
'pym/portage/tests/runTests.py'] + args
+ if os.access(prog, os.X_OK):
+ print('%sTesting with Python %s...%s' %
+ (colors.GOOD, ver, colors.NORMAL))
+ statuses.append(subprocess.call(cmd))
+ elif not ignore_missing:
+ print('%sCould not find requested Python %s%s' %
+ (colors.BAD, ver, colors.NORMAL))
+ statuses.append(1)
+ print()
+ finally:
+ if opts.keep_temp:
+ print('Temporary directory left behind:\n%s' % tempdir)
+ else:
+ # Nuke our tempdir and anything that might be under it.
+ shutil.rmtree(tempdir, True)
# Then summarize it all.
print('\nSummary:\n')
--
2.5.2
Re: [gentoo-portage-dev] gentoolkit.git repository reorganized
On 22 Oct 2015 12:54, Paul Varner wrote: > Mike, I know you're busy with other stuff, but if you ever want to see a > new gentoolkit/gentoolkit-dev release, consider this your authorization > to just do it. The README.dev files state how to make releases. thanks, i think this will help a lot > Since, the tools have dwindled down in gentoolkit-dev, I do think it > does make sense to keep it in the same repo and merge the packages > together behind a USE flag. I will revert the commit, that emptied the > genttolkit-dev branch and ask mgorny to nuke the new gentoolkit-dev > repository. > > As I get time, I will work towards moving the gentoolkit-dev tools into > gentoolkit and putting them behind a USE flag in the ebuild. i'm no distutils expert, and every time i try to do something "fancy", i get frustrated by the module :). do people know of examples where you can do optional installs with a flag ? a cookbook sort of entry here would help and i could take care of merging in say ekeyword. -mike signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH 1/2] chpathtool: drop optparse compat logic
We don't support python 2.6 anymore, so drop the non-argparse logic.
---
bin/chpathtool.py | 45 +++--
1 file changed, 11 insertions(+), 34 deletions(-)
diff --git a/bin/chpathtool.py b/bin/chpathtool.py
index 842f1f4..73c7a5f 100755
--- a/bin/chpathtool.py
+++ b/bin/chpathtool.py
@@ -7,18 +7,12 @@ doc = """Helper tool for converting installed files to custom
prefixes.
In other words, eprefixy $D for Gentoo/Prefix."""
__doc__ = doc
-
+import argparse
import io
import os
import stat
import sys
-try:
- from argparse import ArgumentParser
-except ImportError:
- ArgumentParser = None
- from optparse import OptionParser
-
CONTENT_ENCODING = 'utf_8'
FS_ENCODING = 'utf_8'
@@ -152,33 +146,16 @@ def chpath_inplace_symlink(filename, st, old, new):
def main(argv):
- if ArgumentParser is not None:
- parser = ArgumentParser(description=doc)
- parser.add_argument('location', default=None,
- help='root directory (e.g. $D)')
- parser.add_argument('old', default=None,
- help='original build prefix (e.g. /)')
- parser.add_argument('new', default=None,
- help='new install prefix (e.g. $EPREFIX)')
- opts = parser.parse_args(argv)
-
- location, old, new = opts.location, opts.old, opts.new
- else:
- # Argument parsing compatibility for Python 2.6 using optparse.
- parser = OptionParser(description=doc,
- usage="usage: %prog [-h] location old new\n\n"
+ \
- " location: root directory (e.g. $D)\n" + \
- " old: original build prefix (e.g. /)\n"
+ \
- " new: new install prefix (e.g.
$EPREFIX)")
-
- (opts, args) = parser.parse_args()
-
- if len(args) != 3:
- parser.print_usage()
- parser.error("%s: error: expected 3 arguments,
got %i"
- % (__file__, len(args)))
-
- location, old, new = args[0:3]
+ parser = argparse.ArgumentParser(description=doc)
+ parser.add_argument('location', default=None,
+ help='root directory (e.g. $D)')
+ parser.add_argument('old', default=None,
+ help='original build prefix (e.g. /)')
+ parser.add_argument('new', default=None,
+ help='new install prefix (e.g. $EPREFIX)')
+ opts = parser.parse_args(argv)
+
+ location, old, new = opts.location, opts.old, opts.new
is_text_file = IsTextFile()
--
2.5.2
[gentoo-portage-dev] [PATCH 2/2] _argparse: punt the module
Since we don't support python 2.6 anymore, there's no need to wrap
argparse, so switch all the users to the standard library for it.
---
bin/binhost-snapshot | 4 ++--
bin/ebuild | 4 ++--
bin/egencache| 4 ++--
bin/fixpackages | 4 ++--
bin/glsa-check | 4 ++--
bin/install.py | 4 ++--
bin/portageq | 6 +++---
bin/quickpkg | 4 ++--
bin/xattr-helper.py | 4 ++--
bin/xpak-helper.py | 4 ++--
pym/_emerge/main.py | 4 ++--
pym/portage/_emirrordist/main.py | 4 ++--
pym/portage/emaint/main.py | 6 ++
pym/portage/sync/controller.py | 1 -
pym/portage/tests/__init__.py| 4 ++--
pym/portage/util/_argparse.py| 42
pym/repoman/argparser.py | 4 ++--
17 files changed, 31 insertions(+), 76 deletions(-)
delete mode 100644 pym/portage/util/_argparse.py
diff --git a/bin/binhost-snapshot b/bin/binhost-snapshot
index 3a34643..4130e75 100755
--- a/bin/binhost-snapshot
+++ b/bin/binhost-snapshot
@@ -2,6 +2,7 @@
# Copyright 2010-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
+import argparse
import io
import os
import sys
@@ -17,7 +18,6 @@ if
osp.isfile(osp.join(osp.dirname(osp.dirname(osp.realpath(__file__))), ".porta
sys.path.insert(0,
osp.join(osp.dirname(osp.dirname(osp.realpath(__file__))), "pym"))
import portage
portage._internal_caller = True
-from portage.util._argparse import ArgumentParser
def parse_args(argv):
prog_name = os.path.basename(argv[0])
@@ -45,7 +45,7 @@ def parse_args(argv):
"write Packages index with\n" + \
" snapshot_uri"
- parser = ArgumentParser(usage=usage)
+ parser = argparse.ArgumentParser(usage=usage)
parser.add_argument('--hardlinks',
help='create hardlinks (y or n, default is y)',
choices=('y', 'n'),
diff --git a/bin/ebuild b/bin/ebuild
index ad52ed5..59fced0 100755
--- a/bin/ebuild
+++ b/bin/ebuild
@@ -4,6 +4,7 @@
from __future__ import print_function
+import argparse
import platform
import signal
import sys
@@ -48,13 +49,12 @@ from portage import _shell_quote
from portage import _unicode_decode
from portage import _unicode_encode
from portage.const import VDB_PATH
-from portage.util._argparse import ArgumentParser
from _emerge.Package import Package
from _emerge.RootConfig import RootConfig
description = "See the ebuild(1) man page for more info"
usage = "Usage: ebuild [command] ..."
-parser = ArgumentParser(description=description, usage=usage)
+parser = argparse.ArgumentParser(description=description, usage=usage)
force_help = "When used together with the digest or manifest " + \
"command, this option forces regeneration of digests for all " + \
diff --git a/bin/egencache b/bin/egencache
index 67fca73..6407a44 100755
--- a/bin/egencache
+++ b/bin/egencache
@@ -5,6 +5,7 @@
# unicode_literals for compat with TextIOWrapper in Python 2
from __future__ import print_function, unicode_literals
+import argparse
import platform
import signal
import stat
@@ -54,7 +55,6 @@ from portage.const import TIMESTAMP_FORMAT
from portage.manifest import guessManifestFileType
from portage.package.ebuild._parallel_manifest.ManifestScheduler import
ManifestScheduler
from portage.util import cmp_sort_key, writemsg_level
-from portage.util._argparse import ArgumentParser
from portage.util._async.run_main_scheduler import run_main_scheduler
from portage.util._eventloop.global_event_loop import global_event_loop
from portage import cpv_getkey
@@ -81,7 +81,7 @@ if sys.hexversion >= 0x300:
def parse_args(args):
usage = "egencache [options] ... [atom] ..."
- parser = ArgumentParser(usage=usage)
+ parser = argparse.ArgumentParser(usage=usage)
actions = parser.add_argument_group('Actions')
actions.add_argument("--update",
diff --git a/bin/fixpackages b/bin/fixpackages
index 8a0c444..70ca9fc 100755
--- a/bin/fixpackages
+++ b/bin/fixpackages
@@ -4,6 +4,7 @@
from __future__ import print_function
+import argparse
import os
import sys
@@ -14,7 +15,6 @@ import portage
portage._internal_caller = True
from portage import os
from portage.output import EOutput
-from portage.util._argparse import ArgumentParser
from textwrap import wrap
from portage._global_updates import _global_updates
mysettings = portage.settings
@@ -25,7 +25,7 @@ description = """The fixpackages program performs package
move updates on
configuration files, installed packages, and binary packages."""
description = " ".join(description.split())
-parser = ArgumentParser(description=description)
+parser = argparse.ArgumentParser(description=description)
parser.parse_args()
if mysettings['ROOT'] !=
Re: [gentoo-portage-dev] gentoolkit.git repository reorganized
On 22 Oct 2015 00:45, Mike Frysinger wrote: > On 21 Oct 2015 16:35, Paul Varner wrote: > > On 10/20/2015 03:34 AM, Alexander Berntsen wrote: > > > On 15/10/15 19:42, Paul Varner wrote: > > > > Over the last couple of days, I have done the following: > > > > > > > 1. Migrated the gentoolkit-dev branch to its own gentoolkit-dev.git > > > > repository > > > > 2. Moved the gentoolkit branch to master on the > > > > gentoolkit.git repository > > > Why did you not just make gentoolkit master, and leave gentoolkit-dev as > > > a branch? That's certainly the common way of using git. > > > > > > > Mainly, because at this point gentoolkit and gentoolkit-dev are now > > almost completely separate code bases as well as being separate packages. > > > > They share a common ancestry and that can be seen looking through the > > commit log, but starting with gentoolkit-0.2.5, gentoolkit started > > migrating to python as the only scripting language and utilizing the > > Portage API with setuptools as the build system. The two remaining bash > > scripts are being rewritten in python and when that is complete, they > > will be completely separate code bases. > > > > gentoolkit-dev has stayed as a collection of stand-alone scripts written > > in multiple languages intended mainly for Gentoo developers. > > > > Since they really do not share any code anymore, it did not make sense > > to me keeping gentoolkit-dev as a branch and it should be in its own > > repository. > > echangelog is the only non-shell/python script, and arguably not useful > anymore. repoman itself has a changelog option, and since the move to > git, we don't commit ChangeLog entries anymore. i would just punt it. > > there's also eviewcvs written in perl, but that's also dead now that > we use git, so it should be punted. > > that really only leaves three: > - ebump - bash > - ekeyword - python > - imlate - python > > why not merge them into a single repo ? you can have a dev/ subdir > for scripts that are more developer oriented and put them behind a > USE=dev flag. another reason i think there should be one: gentoolkit-dev rarely sees releases, nor is it clear who is supposed to be making them, nor does it seem like a good use of time to have independent builds/packages. since gentoolkit is getting rolled, updates could finally go out. case in point: ekeyword was rewritten almost 2 years ago and it still hasn't seen a release. -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] gentoolkit.git repository reorganized
On 21 Oct 2015 16:35, Paul Varner wrote: > On 10/20/2015 03:34 AM, Alexander Berntsen wrote: > > On 15/10/15 19:42, Paul Varner wrote: > > > Over the last couple of days, I have done the following: > > > > > 1. Migrated the gentoolkit-dev branch to its own gentoolkit-dev.git > > > repository > > > 2. Moved the gentoolkit branch to master on the > > > gentoolkit.git repository > > Why did you not just make gentoolkit master, and leave gentoolkit-dev as > > a branch? That's certainly the common way of using git. > > > > Mainly, because at this point gentoolkit and gentoolkit-dev are now > almost completely separate code bases as well as being separate packages. > > They share a common ancestry and that can be seen looking through the > commit log, but starting with gentoolkit-0.2.5, gentoolkit started > migrating to python as the only scripting language and utilizing the > Portage API with setuptools as the build system. The two remaining bash > scripts are being rewritten in python and when that is complete, they > will be completely separate code bases. > > gentoolkit-dev has stayed as a collection of stand-alone scripts written > in multiple languages intended mainly for Gentoo developers. > > Since they really do not share any code anymore, it did not make sense > to me keeping gentoolkit-dev as a branch and it should be in its own > repository. echangelog is the only non-shell/python script, and arguably not useful anymore. repoman itself has a changelog option, and since the move to git, we don't commit ChangeLog entries anymore. i would just punt it. there's also eviewcvs written in perl, but that's also dead now that we use git, so it should be punted. that really only leaves three: - ebump - bash - ekeyword - python - imlate - python why not merge them into a single repo ? you can have a dev/ subdir for scripts that are more developer oriented and put them behind a USE=dev flag. -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS
On 11 Jun 2015 00:21, Brian Dolbec wrote: On Sat, 30 May 2015 10:36:05 -0400 Mike Frysinger wrote: so if we're happy with this implementation, i'll start a thread on gentoo-dev so people aren't caught by surprise, and we can merge this for the next release. -mike So, what's the status of this one? I don't recall many responses to the -dev thread. i'll be pushing it at some point. guess i could do it now since it doesn't depend on anything else. -mike signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH] require PORTAGE_{BIN,PYM}_PATH not be cleared
Rather than hardcode a full path everywhere as a fallback, assume the
value is always set to the right location. The current path isn't the
right place anymore already.
If it turns out we want to support this scenario, we can do it via a
bunch of bootstrapping (and symlinked) files.
---
bin/ebuild-helpers/die| 2 +-
bin/ebuild-helpers/dobin | 2 +-
bin/ebuild-helpers/doconfd| 2 +-
bin/ebuild-helpers/dodir | 2 +-
bin/ebuild-helpers/dodoc | 2 +-
bin/ebuild-helpers/doenvd | 2 +-
bin/ebuild-helpers/doexe | 2 +-
bin/ebuild-helpers/dohard | 2 +-
bin/ebuild-helpers/doheader | 2 +-
bin/ebuild-helpers/dohtml | 6 ++
bin/ebuild-helpers/doinfo | 2 +-
bin/ebuild-helpers/doinitd| 2 +-
bin/ebuild-helpers/doins | 2 +-
bin/ebuild-helpers/dolib | 2 +-
bin/ebuild-helpers/doman | 2 +-
bin/ebuild-helpers/domo | 2 +-
bin/ebuild-helpers/dosbin | 2 +-
bin/ebuild-helpers/dosed | 2 +-
bin/ebuild-helpers/dosym | 2 +-
bin/ebuild-helpers/ecompress | 2 +-
bin/ebuild-helpers/ecompressdir | 2 +-
bin/ebuild-helpers/elog | 2 +-
bin/ebuild-helpers/emake | 2 +-
bin/ebuild-helpers/fowners| 2 +-
bin/ebuild-helpers/fperms | 2 +-
bin/ebuild-helpers/keepdir| 2 +-
bin/ebuild-helpers/newins | 2 +-
bin/ebuild-helpers/portageq | 4 +---
bin/ebuild-helpers/prepall| 2 +-
bin/ebuild-helpers/prepalldocs| 2 +-
bin/ebuild-helpers/prepallinfo| 2 +-
bin/ebuild-helpers/prepallman | 2 +-
bin/ebuild-helpers/prepallstrip | 2 +-
bin/ebuild-helpers/prepinfo | 2 +-
bin/ebuild-helpers/prepman| 2 +-
bin/ebuild-helpers/prepstrip | 3 +--
bin/ebuild-helpers/unprivileged/chown | 4 ++--
bin/ebuild-helpers/xattr/install | 2 --
bin/ebuild-ipc| 4 +---
bin/ebuild.sh | 3 ---
bin/helper-functions.sh | 2 +-
bin/isolated-functions.sh | 2 +-
bin/misc-functions.sh | 2 +-
43 files changed, 43 insertions(+), 55 deletions(-)
diff --git a/bin/ebuild-helpers/die b/bin/ebuild-helpers/die
index 9869141..5c6b1d2 100755
--- a/bin/ebuild-helpers/die
+++ b/bin/ebuild-helpers/die
@@ -2,6 +2,6 @@
# Copyright 2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
-source ${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}/isolated-functions.sh
+source ${PORTAGE_BIN_PATH}/isolated-functions.sh || exit 1
die $@
exit 1
diff --git a/bin/ebuild-helpers/dobin b/bin/ebuild-helpers/dobin
index 0ba1eb0..9f4d73d 100755
--- a/bin/ebuild-helpers/dobin
+++ b/bin/ebuild-helpers/dobin
@@ -2,7 +2,7 @@
# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
-source ${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}/isolated-functions.sh
+source ${PORTAGE_BIN_PATH}/isolated-functions.sh || exit 1
if [[ $# -lt 1 ]] ; then
__helpers_die ${0##*/}: at least one argument needed
diff --git a/bin/ebuild-helpers/doconfd b/bin/ebuild-helpers/doconfd
index 1baa3ed..926c318 100755
--- a/bin/ebuild-helpers/doconfd
+++ b/bin/ebuild-helpers/doconfd
@@ -3,7 +3,7 @@
# Distributed under the terms of the GNU General Public License v2
if [[ $# -lt 1 ]] ; then
- source ${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}/isolated-functions.sh
+ source ${PORTAGE_BIN_PATH}/isolated-functions.sh || exit 1
__helpers_die ${0##*/}: at least one argument needed
exit 1
fi
diff --git a/bin/ebuild-helpers/dodir b/bin/ebuild-helpers/dodir
index e03ba9a..eed2c8b 100755
--- a/bin/ebuild-helpers/dodir
+++ b/bin/ebuild-helpers/dodir
@@ -2,7 +2,7 @@
# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
-source ${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}/isolated-functions.sh
+source ${PORTAGE_BIN_PATH}/isolated-functions.sh || exit 1
if ! ___eapi_has_prefix_variables; then
ED=${D}
diff --git a/bin/ebuild-helpers/dodoc b/bin/ebuild-helpers/dodoc
index 6ccf0a4..75029eb 100755
--- a/bin/ebuild-helpers/dodoc
+++ b/bin/ebuild-helpers/dodoc
@@ -2,7 +2,7 @@
# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
-source ${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}/isolated-functions.sh
+source ${PORTAGE_BIN_PATH}/isolated-functions.sh || exit 1
if ___eapi_dodoc_supports_-r; then
__PORTAGE_HELPER='dodoc' exec doins $@
diff --git a/bin/ebuild-helpers/doenvd b/bin/ebuild-helpers/doenvd
index 67bb934..eb31f37 100755
--- a/bin/ebuild-helpers/doenvd
+++ b/bin/ebuild-helpers/doenvd
@@ -3,7 +3,7 @@
# Distributed under the terms of the GNU General Public License v2
if [[ $# -lt
Re: [gentoo-portage-dev] [PATCH v4] xattr: centralize the various shims in one place
On 30 May 2015 16:59, Mike Frysinger wrote: Rather than each module implementing its own shim around the various methods for accessing extended attributes, start a dedicated module that exports a consistent API. ping ... are people ok with this change in API ? -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH v4] xattr: centralize the various shims in one place
On 10 Jun 2015 11:54, Zac Medico wrote: On 05/30/2015 01:59 PM, Mike Frysinger wrote: LGTM, except this one line is indented with spaces instead of tabs in vartree.py: def tar_contents(contents, root, tar, protect=None, onProgress=None, - xattr=False): + xattrs=False): i don't know if we have a standard here. sometimes it's a single tab, sometimes it's spaces to line up. i prefer the latter as that's generally what PEP8 does. -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] runtests: rewrite in python
On 30 May 2015 12:30, Brian Dolbec wrote:
On Sat, 30 May 2015 14:27:25 -0400 Mike Frysinger wrote:
+def get_python_executable(ver):
+ Find the right python executable for |ver|
+ if ver == 'pypy':
+ prog = 'pypy'
+ else:
+ prog = 'python' + ver
+ return os.path.join(EPREFIX, 'usr', 'bin', prog)
The only thing I don't like about this is it could mean more
maintenance changes in the future if others come along.
to be clear: this is not new code. this is (more or less) a straight
port from bash to python. your feedback here applies to the bash
version as well. i'd like to minimize the changes when converting
languages and then address feedback like this on top of that.
I think making the lists have more complete naming would be better
PYTHON_SUPPORTED_VERSIONS = [
'python2.7',
'python3.3',
'python3.4',
]
# The rest are just nice to have.
PYTHON_NICE_VERSIONS = [
'pypy',
'python3.5',
'foo-bar-7.6',
]
Then all that is needed in get_python_executable() is the final
path. No other future code changes are likely to be needed.
Also easier to override from the environment without editing code.
this makes the command line interface a bit more annoying. today you
can do: $ runtests --python-version '2.7 3.3'
but with this change, it'd be:
$ runtests --python-version 'python2.7 python3.3'
we could add some logic so that if the arg is composed of dots
digits, we'd blindly prefix it with python.
Well, that get's back to almost the same kind of
get_python_executable(). But I think it would be a little better than
the existing.
We could instead do a string search and include any
member with that substring. That would include python3.3 and
foo-bar-3.3 for a 3.3 cli entry. It could make for a more flexible cli
versions = []
for y in args.python_versions:
versions.extend([x for x in all_pythons if y in x])
that would perform badly if you used 2 or 3 (as you'd match minors and
such). what about this patch ?
-mike
--- a/runtests
+++ b/runtests
@@ -15,20 +15,21 @@ from __future__ import print_function
import argparse
import os
+import re
import sys
import subprocess
# These are the versions we fully support and require to pass tests.
PYTHON_SUPPORTED_VERSIONS = [
- '2.7',
- '3.3',
- '3.4',
+ 'python2.7',
+ 'python3.3',
+ 'python3.4',
]
# The rest are just nice to have.
PYTHON_NICE_VERSIONS = [
'pypy',
- '3.5',
+ 'python3.5',
]
EPREFIX = os.environ.get('PORTAGE_OVERRIDE_EPREFIX', '/')
@@ -68,10 +69,10 @@ class Colors(object):
def get_python_executable(ver):
Find the right python executable for |ver|
- if ver == 'pypy':
- prog = 'pypy'
- else:
+ if re.match(r'[0-9.]+$', ver):
prog = 'python' + ver
+ else:
+ prog = ver
return os.path.join(EPREFIX, 'usr', 'bin', prog)
signature.asc
Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] officially drop support for python 2.6, 3.1, and 3.2
On 02 Jun 2015 06:49, Alexander Berntsen wrote: On 02/06/15 06:42, Mike Frysinger wrote: the portage ebuild dropped support: python 3.1: 21 months ago python 2.6: 10 months ago python 3.2: 8 weeks ago the python team itself no longer supports these versions either and it's not possible to install other modules using them. they're already forcing you to pick 2.7 or 3.3+. You make a compelling case here. plus, i'm not talking about changing existing releases, just new ones, which means it's going to be a while before people are completely impacted. I realise that. Anything else would be stupid. at some point we have to move forward and the shims for 2.6/3.1/3.2 are holding us back now. 2.6 is obvious enough, but 3.1 3.2 are a pain due to the unicode differences. conversely, py2.7 and py3.3+ have the same unicode literal format. I agree on all points. So I think this patch would be fine to merge. I'd like Brian to chime in, but I'm a tentative +1. dol-sen but yeah, I don't see a problem in dropping them zmedico yeah dol-sen especially since gkeys is not 3.2 capable dol-sen and I'm starting to integrate it in portage zmedico those pythons are ancient -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] make PORTAGE_BIN_PATH settings relocatable
On 01 Jun 2015 11:49, Zac Medico wrote: On 06/01/2015 10:25 AM, Mike Frysinger wrote: (a) just throw an error and exit when PORTAGE_BIN_PATH is not set ... considering the current portage code points to a path where it is no longer installed, maybe that's ok ? the recent changes to make portage install copies for each python version is what broke it i think. I like this idea. I can't think of a reason why we should allow PORTAGE_BIN_PATH to be unset. i'll try this in the CrOS project where we're using Gentoo/Prefix and see how it goes. and then try it on my system for a while to further check it. otherwise i'll go with Tim's inspired idea about symlinked wrappers. -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] make PORTAGE_BIN_PATH settings relocatable
On 01 Jun 2015 02:34, Mike Frysinger wrote:
-source ${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}/isolated-functions.sh
+if [[ -z ${PORTAGE_BIN_PATH} ]] ; then
+ PORTAGE_BIN_PATH=$(dirname $(dirname $(readlink -f $0)))
+fi
+source ${PORTAGE_BIN_PATH}/isolated-functions.sh
for people who don't want to scan this whole thing, basically every file gets a
change like this.
two things to note:
(1) i did not retain the ${VAR:-def} form because bash evaluates the default
value even if it doesn't use it. so when you do something like:
VAR=val
echo ${VAR:-$(rm -rf /some/path)}
it will first run the `rm` and then expand val for the echo. the form i used
thus avoids the runtime overhead of doing dirname/readlink unnecessarily.
(2) i avoided the bash string functions like ${VAR%/*} because it splits up
into multiple statements and is not as readable imo:
PORTAGE_BIN_PATH=$(readlink -f $0)
PORTAGE_BIN_PATH=${PORTAGE_BIN_PATH%/*}
PORTAGE_BIN_PATH=${PORTAGE_BIN_PATH%/*}
or even worse:
PORTAGE_BIN_PATH=$(readlink -f $0);
PORTAGE_BIN_PATH=${PORTAGE_BIN_PATH%/*}; PORTAGE_BIN_PATH=${PORTAGE_BIN_PATH%/*}
or also bad imo as it's still not that readable:
p=$(readlink -f $0); p=${p%/*}; p=${p%/*}
PORTAGE_BIN_PATH=${p}
the dirname version might be a bit slower, but i think that's acceptable
considering this is the fallback case that should rarely be run (as the env
var should normally be set).
alternative crazy ideas:
(a) just throw an error and exit when PORTAGE_BIN_PATH is not set ...
considering the current portage code points to a path where it is no longer
installed, maybe that's ok ? the recent changes to make portage install copies
for each python version is what broke it i think.
(b) add a tool alongside it that can be `eval`-ed, although you'd still need to
do something like:
if [[ -z ${PORTAGE_BIN_PATH} ]] ; then
eval $($(dirname $(readlink -f $0))/some-tool)
fi
and i don't think that's an improvement relative to my patch (deal with quoting
bs and such).
(c) convert most things to python and use relative module imports to get at the
bootstrap module. then that one would hold all the logic.
(d) some other alternative ?
-mike
signature.asc
Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] officially drop support for python 2.6, 3.1, and 3.2
On 01 Jun 2015 23:16, Alexander Berntsen wrote: On 30/05/15 16:58, Mike Frysinger wrote: We no longer support these in the ebuild and don't want to waste time implementing compatibility shims for them. In principle I don't mind this at all. However, should we do some sort of survey? Maybe a forum poll or something to see if someone desperately needs them? the portage ebuild dropped support: python 3.1: 21 months ago python 2.6: 10 months ago python 3.2: 8 weeks ago the python team itself no longer supports these versions either and it's not possible to install other modules using them. they're already forcing you to pick 2.7 or 3.3+. plus, i'm not talking about changing existing releases, just new ones, which means it's going to be a while before people are completely impacted. at some point we have to move forward and the shims for 2.6/3.1/3.2 are holding us back now. 2.6 is obvious enough, but 3.1 3.2 are a pain due to the unicode differences. conversely, py2.7 and py3.3+ have the same unicode literal format. -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] _approx_multilib_categories: rename ia to ia64 (bug 550898)
On 31 May 2015 11:43, Zac Medico wrote: In LinkageMapELF.py, there's a fallback mapping for multilib categories, which is used by the LinkMapElf class when NEEDED.ELF.2 does not contain multilib categories due to being generated by older portage. This mapping should be consistent with the mutilib categories generated by the compute_multilib_category function which was modified by commit b50bbcb70506254d66937dac376056bbf99b3fe9. looks obvious enough ;) -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH v3] xattr: centralize the various shims in one place
On 30 May 2015 12:21, Zac Medico wrote:
On 05/30/2015 08:14 AM, Mike Frysinger wrote:
+ @classmethod
+ def list(cls, item, nofollow=False, namespace=None, _names_only=True):
+ cmd = ['getfattr', '-d', '--absolute-names', item]
All getfattr calls need to use -m- ('-' pattern means match all) in
order to list all attributes.
i use -m already. here's a bit more of that function:
def list(cls, item, nofollow=False, namespace=None, _names_only=True):
cmd = ['getfattr', '-d', '--absolute-names', item]
if nofollow:
cmd += ['-h']
cmd += ['-m', ('^%s[.]' % namespace) if namespace else '']
proc = cls._call(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
so when there's no namespace specified, it ends up doing:
getfattr -d --absolute-names /path/file -m ''
which is effectively the same thing, but i can change the '' to '-' since
that's
what the documentation suggests.
-mike
signature.asc
Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] runtests: rewrite in python
On 30 May 2015 14:42, Mike Gilbert wrote: On Sat, May 30, 2015 at 1:18 PM, Brian Dolbec wrote: On Sat, 30 May 2015 12:29:14 -0400 Mike Frysinger wrote: +from __future__ import print_function If I'm not mistaken, this is not needed for 2.7+ anymore. It is still for 2.6 though. You are mistaken. Without this, print is still treated as a statement and anything in parens is treated as a tuple. not exactly true -- (1) and (foo) are not tuples. need a comma to turn a paren into a tuple. e.g. (1,) and (foo,). -mike signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH v4] xattr: centralize the various shims in one place
Rather than each module implementing its own shim around the various
methods for accessing extended attributes, start a dedicated module
that exports a consistent API.
---
v4
- merge in recent quickpkg changes
- add a XATTRS_WORKS symbol for easy testing
- use - with -m when matching all
bin/quickpkg | 12 +-
bin/xattr-helper.py | 11 +-
pym/portage/dbapi/vartree.py | 10 +-
pym/portage/tests/util/test_xattr.py | 178 +++
pym/portage/util/_xattr.py | 228 +++
pym/portage/util/movefile.py | 100 ---
pym/portage/util/xattr.py| 20 ---
7 files changed, 441 insertions(+), 118 deletions(-)
create mode 100644 pym/portage/tests/util/test_xattr.py
create mode 100644 pym/portage/util/_xattr.py
delete mode 100644 pym/portage/util/xattr.py
diff --git a/bin/quickpkg b/bin/quickpkg
index 726abff..262fda4 100755
--- a/bin/quickpkg
+++ b/bin/quickpkg
@@ -21,8 +21,8 @@ from portage.dbapi.dep_expand import dep_expand
from portage.dep import Atom, use_reduce
from portage.exception import (AmbiguousPackageName, InvalidAtom, InvalidData,
InvalidDependString, PackageSetNotFound, PermissionDenied)
-from portage.util import ConfigProtect, ensure_dirs, shlex_split
-import portage.util.xattr as _xattr
+from portage.util import ConfigProtect, ensure_dirs, shlex_split, _xattr
+xattr = _xattr.xattr
from portage.dbapi.vartree import dblink, tar_contents
from portage.checksum import perform_md5
from portage._sets import load_default_config, SETPREFIX
@@ -36,7 +36,7 @@ def quickpkg_atom(options, infos, arg, eout):
vartree = trees[vartree]
vardb = vartree.dbapi
bintree = trees[bintree]
- xattr = 'xattr' in settings.features
+ xattrs = 'xattr' in settings.features
include_config = options.include_config == y
include_unmodified_config = options.include_unmodified_config == y
@@ -137,8 +137,8 @@ def quickpkg_atom(options, infos, arg, eout):
# The tarfile module will write pax headers holding the
# xattrs only if PAX_FORMAT is specified here.
tar = tarfile.open(binpkg_tmpfile, w:bz2,
- format=tarfile.PAX_FORMAT if xattr else
tarfile.DEFAULT_FORMAT)
- tar_contents(contents, root, tar, protect=protect,
xattr=xattr)
+ format=tarfile.PAX_FORMAT if xattrs else
tarfile.DEFAULT_FORMAT)
+ tar_contents(contents, root, tar, protect=protect,
xattrs=xattrs)
tar.close()
xpak.tbz2(binpkg_tmpfile).recompose_mem(xpdata)
finally:
@@ -238,7 +238,7 @@ def quickpkg_main(options, args, eout):
eout.eerror(No write access to '%s' % bintree.pkgdir)
return errno.EACCES
- if 'xattr' in portage.settings.features and not hasattr(_xattr,
'getxattr'):
+ if 'xattr' in portage.settings.features and not _xattr.XATTRS_WORKS:
eout.eerror(No xattr support library was found,
so xattrs will not be preserved!)
portage.settings.unlock()
diff --git a/bin/xattr-helper.py b/bin/xattr-helper.py
index 3e9b81e..19f25f9 100755
--- a/bin/xattr-helper.py
+++ b/bin/xattr-helper.py
@@ -19,16 +19,7 @@ import re
import sys
from portage.util._argparse import ArgumentParser
-
-if hasattr(os, getxattr):
-
- class xattr(object):
- get = os.getxattr
- set = os.setxattr
- list = os.listxattr
-
-else:
- import xattr
+from portage.util._xattr import xattr
_UNQUOTE_RE = re.compile(br'\\[0-7]{3}')
diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
index 62d880e..e755fde 100644
--- a/pym/portage/dbapi/vartree.py
+++ b/pym/portage/dbapi/vartree.py
@@ -35,7 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.movefile:movefile',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
- 'portage.util:xattr@_xattr',
+ 'portage.util._xattr:xattr',
'portage.util._dyn_libs.PreservedLibsRegistry:PreservedLibsRegistry',
'portage.util._dyn_libs.LinkageMapELF:LinkageMapELF@LinkageMap',
'portage.util._async.SchedulerInterface:SchedulerInterface',
@@ -5268,7 +5268,7 @@ def write_contents(contents, root, f):
f.write(line)
def tar_contents(contents, root, tar, protect=None, onProgress=None,
- xattr=False):
+ xattrs=False):
os = _os_merge
encoding = _encodings['merge']
@@ -5390,13 +5390,13 @@ def tar_contents(contents, root, tar, protect=None,
onProgress=None,
encoding=encoding,
errors='strict')
-
[gentoo-portage-dev] [PATCH] runtests: rewrite in python
The bash was getting ugly, and this allows us to add more smarts sanely
to the main script.
---
DEVELOPING | 2 +-
runtests| 156
runtests.sh | 109 --
3 files changed, 157 insertions(+), 110 deletions(-)
create mode 100755 runtests
delete mode 100755 runtests.sh
diff --git a/DEVELOPING b/DEVELOPING
index 0b0bb60..31b5594 100644
--- a/DEVELOPING
+++ b/DEVELOPING
@@ -225,7 +225,7 @@ Second create a git tag for this release:
Then create the tarball and run the tests:
./mkrelease.sh --changelog-rev v2.2.7 --tag --runtests 2.2.8
Make sure you have all supported python versions installed first
-(see PYTHON_SUPPORTED_VERSIONS in runtests.sh).
+(see PYTHON_SUPPORTED_VERSIONS in runtests).
Version bump the ebuild and verify it can re-install itself:
emerge portage
diff --git a/runtests b/runtests
new file mode 100755
index 000..d1f7b6f
--- /dev/null
+++ b/runtests
@@ -0,0 +1,156 @@
+#!/usr/bin/python
+# Copyright 2010-2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+#
+# Note: We don't want to import portage modules directly because we do things
+# like run the testsuite through multiple versions of python.
+
+Helper script to run portage unittests against different python versions.
+
+Note: Any additional arguments will be passed down directly to the underlying
+unittest runner. This lets you select specific tests to execute.
+
+
+from __future__ import print_function
+
+import argparse
+import os
+import sys
+import subprocess
+
+
+# These are the versions we fully support and require to pass tests.
+PYTHON_SUPPORTED_VERSIONS = [
+ '2.7',
+ '3.3',
+ '3.4',
+]
+# The rest are just nice to have.
+PYTHON_NICE_VERSIONS = [
+ 'pypy',
+ '3.5',
+]
+
+EPREFIX = os.environ.get('PORTAGE_OVERRIDE_EPREFIX', '/')
+
+
+class Colors(object):
+ Simple object holding color constants.
+
+ _COLORS_YES = ('y', 'yes', 'true')
+ _COLORS_NO = ('n', 'no', 'false')
+
+ GOOD = BAD = NORMAL = ''
+
+ def __init__(self, colorize=None):
+ if colorize is None:
+ nocolors = os.environ.get('NOCOLOR', 'false')
+ # Ugh, look away, for here we invert the world!
+ if nocolors in self._COLORS_YES:
+ colorize = False
+ elif nocolors in self._COLORS_NO:
+ colorize = True
+ else:
+ raise ValueError('$NOCOLORS is invalid: %s' %
nocolors)
+ else:
+ if colorize in self._COLORS_YES:
+ colorize = True
+ elif colorize in self._COLORS_NO:
+ colorize = False
+ else:
+ raise ValueError('--colors is invalid: %s' %
colorize)
+
+ if colorize:
+ self.GOOD = '\033[1;32m'
+ self.BAD = '\033[1;31m'
+ self.NORMAL = '\033[0m'
+
+
+def get_python_executable(ver):
+ Find the right python executable for |ver|
+ if ver == 'pypy':
+ prog = 'pypy'
+ else:
+ prog = 'python' + ver
+ return os.path.join(EPREFIX, 'usr', 'bin', prog)
+
+
+def get_parser():
+ Return a argument parser for this module
+ epilog = Examples:
+List all the available unittests.
+$ %(prog)s --list
+
+Run against specific versions of python.
+$ %(prog)s --python-version '2.7 3.3'
+
+Run just one unittest.
+$ %(prog)s pym/portage/tests/xpak/test_decodeint.py
+
+ parser = argparse.ArgumentParser(
+ description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog=epilog)
+ parser.add_argument('--color', type=str, default=None,
+ help='Whether to use colorized output (default is auto)')
+ parser.add_argument('--python-versions', action='append',
+ help='Versions of python to test (default is test available)')
+ return parser
+
+
+def main(argv):
+ parser = get_parser()
+ opts, args = parser.parse_known_args(argv)
+ colors = Colors(colorize=opts.color)
+
+ # Figure out all the versions we want to test.
+ if opts.python_versions is None:
+ ignore_missing = True
+ pyversions = PYTHON_SUPPORTED_VERSIONS + PYTHON_NICE_VERSIONS
+ else:
+ ignore_missing = False
+ pyversions = []
+ for ver in opts.python_versions:
+ if ver == 'supported':
+ pyversions.extend(PYTHON_SUPPORTED_VERSIONS)
+ else:
+ pyversions.extend(ver.split())
+
+ #
Re: [gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS
On 30 May 2015 11:22, Zac Medico wrote:
On 05/30/2015 07:36 AM, Mike Frysinger wrote:
On 26 May 2015 08:58, Zac Medico wrote:
On 05/26/2015 07:24 AM, Mike Frysinger wrote:
+ # Only check on 32-bit systems. Filtering by $ARCH here isn't perfect,
but
+ # it should be good enough for our needs.
+ case ${ARCH} in
+ arm|mips|ppc|sh|x86) ;;
+ *) return ;;
+ esac
Shouldn't we also enable this for 64-bit archs when multilib is enabled?
yes, but i think we should start here first. getting multilib right is
kind of
a pain. this should give us enough coverage i think to get people to start
filing bugs which implicitly covers multilib users. once the dust has
settled,
we can look at expanding the multilib coverage. although that would really
require a python implementation, and the current install hooks logic
implicitly
requires every file to be bash.
to use your pkg-config example, it installs 32bit 64bit ELFs into
/usr/bin.
we can't scan all 32bit ELFs because it would incorrectly flag ILP32 ABIs
like
x32 n32.
Our compute_multilib_category function has logic that could be used to
separate and filter them:
https://gitweb.gentoo.org/proj/portage.git/tree/pym/portage/dep/soname/multilib_category.py
that'd require a python implementation which hooks are somewhat limited by
currently ;)
-mik
signature.asc
Description: Digital signature
Re: [gentoo-portage-dev] [PATCH v3] xattr: centralize the various shims in one place
On 30 May 2015 11:14, Mike Frysinger wrote:
Rather than each module implementing its own shim around the various
methods for accessing extended attributes, start a dedicated module
that exports a consistent API.
and it looks like i just missed the new pym/portage/util/xattr.py module.
i'll have to merge that in too. but first about the API.
the current code does:
import portage.util.xattr as _xattr
and then the _xattr module provides three functions:
getxattr
listxattr
setxattr
which is kind of confusing as both pyxattr and python 3.3 (via the os module)
provide these functions, but with different signatures. for example:
os.getxattr(path, attribute, follow_symlinks=True)
xattr.getxattr(item, attribute[, nofollow=False])
i think this is why the pyxattr module has actually marked all of these as
deprecated.
mine is:
from portage.util._xattr import xattr
and then the xattr module provides the standard pyxattr functions:
xattr.get
xattr.get_all
xattr.list
xattr.remove
xattr.set
and i omit all of the confusing variants. the reason i don't put the functions
into the module itself is so that we can more xattr related functions which i've
already done with the context manager:
from portage.util._xattr import preserve_xattrs
with preserve_xattrs('/some/file'):
...
so if there's no reason to keep portage.util.xattr, i'll kill it and move the
new quickpkg usage to _xattr.
-mike
signature.asc
Description: Digital signature
Re: [gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS
On 26 May 2015 08:58, Zac Medico wrote:
On 05/26/2015 07:24 AM, Mike Frysinger wrote:
+ # Only check on 32-bit systems. Filtering by $ARCH here isn't perfect,
but
+ # it should be good enough for our needs.
+ case ${ARCH} in
+ arm|mips|ppc|sh|x86) ;;
+ *) return ;;
+ esac
Shouldn't we also enable this for 64-bit archs when multilib is enabled?
yes, but i think we should start here first. getting multilib right is kind of
a pain. this should give us enough coverage i think to get people to start
filing bugs which implicitly covers multilib users. once the dust has settled,
we can look at expanding the multilib coverage. although that would really
require a python implementation, and the current install hooks logic implicitly
requires every file to be bash.
to use your pkg-config example, it installs 32bit 64bit ELFs into /usr/bin.
we can't scan all 32bit ELFs because it would incorrectly flag ILP32 ABIs like
x32 n32.
ideally we'd scan the whole multilib dir, but that too runs into problems.
when /usr/lib is the path for both x86 multilib and non-multilib content (like
libexec stuff), we can't blindly scan reject all ELFs in there. the previous
note about ILP32 applies here too.
so if we're happy with this implementation, i'll start a thread on gentoo-dev
so
people aren't caught by surprise, and we can merge this for the next release.
-mike
signature.asc
Description: Digital signature
[gentoo-portage-dev] [PATCH] officially drop support for python 2.6, 3.1, and 3.2
We no longer support these in the ebuild and don't want to waste time
implementing compatibility shims for them.
---
.travis.yml | 2 --
DEVELOPING | 8
README | 2 +-
runtests.sh | 4 ++--
4 files changed, 7 insertions(+), 9 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 69e1659..8a86aa6 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,8 +1,6 @@
language: python
python:
-- 2.6
- 2.7
-- 3.2
- 3.3
- 3.4
- pypy
diff --git a/DEVELOPING b/DEVELOPING
index 55470d8..0b0bb60 100644
--- a/DEVELOPING
+++ b/DEVELOPING
@@ -8,10 +8,10 @@ bad habits that exist in the current codebase.
Python Version
--
-Python 2.6 is the minimum supported version, since it is the first
-version to support Python 3 syntax. All exception handling should use
-Python 3 'except' syntax, and the print function should be used instead
-of Python 2's print statement (from __future__ import print_function).
+Python 2.7 is the minimum supported version as it eases 3.x compatibility.
+All exception handling should use Python 3 'except' syntax, and the print
+function should be used instead of Python 2's print statement (use from
+__future__ import print_function everywhere).
Dependencies
diff --git a/README b/README
index 5558dde..415a962 100644
--- a/README
+++ b/README
@@ -10,7 +10,7 @@ package managers.
Dependencies
-Python and Bash should be the only hard dependencies. Python 2.6 is the
+Python and Bash should be the only hard dependencies. Python 2.7 is the
minimum supported version.
diff --git a/runtests.sh b/runtests.sh
index 651928f..a7ca101 100755
--- a/runtests.sh
+++ b/runtests.sh
@@ -3,8 +3,8 @@
# Distributed under the terms of the GNU General Public License v2
# These are the versions we care about. The rest are just nice to have.
-PYTHON_SUPPORTED_VERSIONS=2.6 2.7 3.2 3.3 3.4
-PYTHON_VERSIONS=2.6 2.7 pypy 3.1 3.2 3.3 3.4 3.5
+PYTHON_SUPPORTED_VERSIONS=2.7 3.3 3.4
+PYTHON_VERSIONS=2.7 pypy 3.3 3.4 3.5
# has to be run from portage root dir
cd ${0%/*} || exit 1
--
2.4.1
[gentoo-portage-dev] [PATCH v3] xattr: centralize the various shims in one place
Rather than each module implementing its own shim around the various
methods for accessing extended attributes, start a dedicated module
that exports a consistent API.
---
bin/xattr-helper.py | 11 +-
pym/portage/tests/util/test_xattr.py | 178
pym/portage/util/_xattr.py | 221 +++
pym/portage/util/movefile.py | 100
4 files changed, 423 insertions(+), 87 deletions(-)
create mode 100644 pym/portage/tests/util/test_xattr.py
create mode 100644 pym/portage/util/_xattr.py
diff --git a/bin/xattr-helper.py b/bin/xattr-helper.py
index 3e9b81e..19f25f9 100755
--- a/bin/xattr-helper.py
+++ b/bin/xattr-helper.py
@@ -19,16 +19,7 @@ import re
import sys
from portage.util._argparse import ArgumentParser
-
-if hasattr(os, getxattr):
-
- class xattr(object):
- get = os.getxattr
- set = os.setxattr
- list = os.listxattr
-
-else:
- import xattr
+from portage.util._xattr import xattr
_UNQUOTE_RE = re.compile(br'\\[0-7]{3}')
diff --git a/pym/portage/tests/util/test_xattr.py
b/pym/portage/tests/util/test_xattr.py
new file mode 100644
index 000..2e2564a
--- /dev/null
+++ b/pym/portage/tests/util/test_xattr.py
@@ -0,0 +1,178 @@
+# Copyright 2010-2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+Tests for the portage.util._xattr module
+
+from __future__ import print_function
+
+try:
+ # Try python-3.3 module first.
+ # pylint: disable=no-name-in-module
+ from unittest import mock
+except ImportError:
+ try:
+ # Try standalone module.
+ import mock
+ except ImportError:
+ mock = None
+
+import subprocess
+
+import portage
+from portage.tests import TestCase
+from portage.util._xattr import (xattr as _xattr, _XattrSystemCommands,
+ _XattrStub)
+
+
+orig_popen = subprocess.Popen
+def MockSubprocessPopen(stdin):
+ Helper to mock (closely) a subprocess.Popen call
+
+ The module has minor tweaks in behavior when it comes to encoding and
+ python versions, so use a real subprocess.Popen call to fake out the
+ runtime behavior. This way we don't have to also implement different
+ encodings as that gets ugly real fast.
+
+ # pylint: disable=protected-access
+ proc = orig_popen(['cat'], stdout=subprocess.PIPE,
stdin=subprocess.PIPE)
+ proc.stdin.write(portage._unicode_encode(stdin,
portage._encodings['stdio']))
+ return proc
+
+
+class SystemCommandsTest(TestCase):
+ Test _XattrSystemCommands
+
+ OUTPUT = '\n'.join((
+ '# file: /bin/ping',
+ 'security.capability=0sAQAAAgAgAAA=',
+ 'user.foo=asdf',
+ '',
+ ))
+
+ def _setUp(self):
+ if mock is None:
+ self.skipTest('need mock for testing')
+
+ return _XattrSystemCommands
+
+ def _testGetBasic(self):
+ Verify the get() behavior
+ xattr = self._setUp()
+ with mock.patch.object(subprocess, 'Popen') as call_mock:
+ # Verify basic behavior, and namespace arg works as
expected.
+ xattr.get('/some/file', 'user.foo')
+ xattr.get('/some/file', 'foo', namespace='user')
+ self.assertEqual(call_mock.call_args_list[0],
call_mock.call_args_list[1])
+
+ # Verify nofollow behavior.
+ call_mock.reset()
+ xattr.get('/some/file', 'user.foo', nofollow=True)
+ self.assertIn('-h', call_mock.call_args[0][0])
+
+ def testGetParsing(self):
+ Verify get() parses output sanely
+ xattr = self._setUp()
+ with mock.patch.object(subprocess, 'Popen') as call_mock:
+ # Verify output parsing.
+ call_mock.return_value = MockSubprocessPopen('\n'.join([
+ '# file: /some/file',
+ 'user.foo=asdf',
+ '',
+ ]))
+ call_mock.reset()
+ self.assertEqual(xattr.get('/some/file', 'user.foo'),
b'asdf')
+
+ def testGetAllBasic(self):
+ Verify the get_all() behavior
+ xattr = self._setUp()
+ with mock.patch.object(subprocess, 'Popen') as call_mock:
+ # Verify basic behavior.
+ xattr.get_all('/some/file')
+
+ # Verify nofollow behavior.
+ call_mock.reset()
+ xattr.get_all('/some/file', nofollow=True)
+ self.assertIn('-h', call_mock.call_args[0][0])
+
+ def
[gentoo-portage-dev] [PATCH] install-qa-check.d: issue warnings for 32bit ELFs not using LFS
From: Mike Frysinger vap...@chromium.org
Start issuing QA warnings when ELFs are installed using the old 32bit
file interface. This programs can fail when working with large files
(like ones more than 4GiB), but even just trying to stat a file that
happens to have a 64bit inode. It also can lead to silent corruption
when one library/app is using 64bit structures and tries to work with
another one that uses 32bit (or vice versa). This is because the API
might utilize off_t's, the library was compiled with off_t==off64_t,
and the app was compiled with off_t==off32_t.
X-Gentoo-Bug: 471102
X-Gentoo-Bug-url: https://bugs.gentoo.org/471102
---
bin/install-qa-check.d/10large-file-support | 158
1 file changed, 158 insertions(+)
create mode 100644 bin/install-qa-check.d/10large-file-support
diff --git a/bin/install-qa-check.d/10large-file-support
b/bin/install-qa-check.d/10large-file-support
new file mode 100644
index 000..34726d7
--- /dev/null
+++ b/bin/install-qa-check.d/10large-file-support
@@ -0,0 +1,158 @@
+# Detect 32-bit builds that are using legacy 32-bit file interfaces.
+# https://en.wikipedia.org/wiki/Large_file_support
+#
+# We want to make sure everyone is using the 64-bit interfaces.
+# https://bugs.gentoo.org/471102
+
+# Lists gleaned from headers and this doc:
+# http://people.redhat.com/berrange/notes/largefile.html
+# http://opengroup.org/platform/lfs.html
+SYMBOLS=(
+ # aio.h
+ aio_cancel
+ aio_error
+ aio_fsync
+ aio_read
+ aio_return
+ aio_suspend
+ aio_write
+ lio_listio
+
+ # dirent.h
+ alphasort
+ getdirentries
+ readdir
+ readdir_r
+ scandir
+ scandirat
+ versionsort
+
+ # fcntl.h
+ creat
+ fallocate
+ fopen
+ fopenat
+ freopen
+ open
+ openat
+ posix_fadvise
+ posix_fallocate
+ __open
+ __open_2
+ __openat_2
+
+ # ftw.h
+ ftw
+ nftw
+
+ # glob.h
+ glob
+ globfree
+
+ # stdio.h
+ fgetpos
+ fopen
+ freopen
+ fseeko
+ fsetpos
+ ftello
+ tmpfile
+
+ # stdlib.h
+ mkostemp
+ mkostemps
+ mkstemp
+ mkstemps
+
+ # sys/mman.h
+ mmap
+
+ # sys/resource.h
+ getrlimit
+ prlimit
+ setrlimit
+
+ # sys/sendfile.h
+ sendfile
+
+ # sys/stat.h
+ fstat
+ fstatat
+ lstat
+ stat
+ __fxstat
+ __fxstatat
+ __lxstat
+ __xstat
+
+ # sys/statfs.h
+ fstatfs
+
+ # sys/statvfs.h
+ statvfs
+ fstatvfs
+
+ # unistd.h
+ lockf
+ lseek
+ ftruncate
+ pread
+ preadv
+ pwrite
+ pwritev
+ truncate
+ __pread_chk
+)
+SYMBOLS_REGEX=$(printf '%s|' ${SYMBOLS[@]})
+# The @@ part is to workaround a limitation in pax-utils w/versioned symbols.
+SYMBOLS_REGEX=^(${SYMBOLS_REGEX%|})(@@.*)?$
+
+check_lfs() {
+ local files=$(scanelf -F '%s %p' -qyRgs -${SYMBOLS_REGEX} $@)
+
+ if [[ -n ${files} ]]; then
+ echo
+ eqawarn QA Notice: The following files were not built with LFS
support:
+ eqawarn Please see https://bugs.gentoo.org/471102 for
details.
+ eqawarn ${files}
+ echo
+ fi
+}
+
+filtered_check_lfs() {
+ if ! type -P scanelf /dev/null || has binchecks ${RESTRICT}; then
+ return
+ fi
+
+ # Only check glibc uClibc libraries. Others are presumed to use LFS
by
+ # default (e.g. musl), or it's not relevant (e.g. newlib).
+ case ${CHOST} in
+ *-gnu*|*-uclibc*) ;;
+ *) return ;;
+ esac
+
+ # Only check on 32-bit systems. Filtering by $ARCH here isn't perfect,
but
+ # it should be good enough for our needs.
+ case ${ARCH} in
+ arm|mips|ppc|sh|x86) ;;
+ *) return ;;
+ esac
+
+ # Obviously filter out C libraries themselves :).
+ # The sandbox has to capture all symbols by design.
+ case ${CATEGORY}/${PN} in
+ */glibc|\
+ */uclibc|\
+ */gcc|\
+ sys-apps/sandbox) ;;
+ *) check_lfs ${ED} ;;
+ esac
+}
+
+# Allow for people to run manually for testing/debugging.
+if [[ $# -ne 0 ]]; then
+ eqawarn() { echo * $*; }
+ check_lfs $@
+else
+ filtered_check_lfs
+fi
--
2.4.1
[gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS
From: Mike Frysinger vap...@chromium.org
Start issuing QA warnings when ELFs are installed using the old 32bit
file interface. These programs can fail out right:
* working with large files (more than 2GiB) can return EOVERFLOW
* stating files on large filesystems w/64bit inodes can fail too
It also can lead to silent corruption that is hard to debug like when
one library/app is compiled using 64bit structures and tries to work
with another one that uses 32bit (or vice versa). This is because
the API in the header utilizes off_t's or structs (like stat) that
have off_t's embedded in them. By default, off_t is defined as an
off32_t, but when the code is compiled with -D_FILE_OFFSET_BITS=64,
it is transparently changed to off64_t. That means while one side
was compiled (w/out warnings) expecting 32bit structs, the other
was compiled (w/out warnings) expecting 64bit structs. The ABI is
different, but C does not support type checking, so no one notices.
The only sane way forward is to use LFS everywhere.
X-Gentoo-Bug: 549092
X-Gentoo-Bug-url: https://bugs.gentoo.org/549092
---
v2
- do some word smithing in the commit message
bin/install-qa-check.d/10large-file-support | 158
1 file changed, 158 insertions(+)
create mode 100644 bin/install-qa-check.d/10large-file-support
diff --git a/bin/install-qa-check.d/10large-file-support
b/bin/install-qa-check.d/10large-file-support
new file mode 100644
index 000..34726d7
--- /dev/null
+++ b/bin/install-qa-check.d/10large-file-support
@@ -0,0 +1,158 @@
+# Detect 32-bit builds that are using legacy 32-bit file interfaces.
+# https://en.wikipedia.org/wiki/Large_file_support
+#
+# We want to make sure everyone is using the 64-bit interfaces.
+# https://bugs.gentoo.org/471102
+
+# Lists gleaned from headers and this doc:
+# http://people.redhat.com/berrange/notes/largefile.html
+# http://opengroup.org/platform/lfs.html
+SYMBOLS=(
+ # aio.h
+ aio_cancel
+ aio_error
+ aio_fsync
+ aio_read
+ aio_return
+ aio_suspend
+ aio_write
+ lio_listio
+
+ # dirent.h
+ alphasort
+ getdirentries
+ readdir
+ readdir_r
+ scandir
+ scandirat
+ versionsort
+
+ # fcntl.h
+ creat
+ fallocate
+ fopen
+ fopenat
+ freopen
+ open
+ openat
+ posix_fadvise
+ posix_fallocate
+ __open
+ __open_2
+ __openat_2
+
+ # ftw.h
+ ftw
+ nftw
+
+ # glob.h
+ glob
+ globfree
+
+ # stdio.h
+ fgetpos
+ fopen
+ freopen
+ fseeko
+ fsetpos
+ ftello
+ tmpfile
+
+ # stdlib.h
+ mkostemp
+ mkostemps
+ mkstemp
+ mkstemps
+
+ # sys/mman.h
+ mmap
+
+ # sys/resource.h
+ getrlimit
+ prlimit
+ setrlimit
+
+ # sys/sendfile.h
+ sendfile
+
+ # sys/stat.h
+ fstat
+ fstatat
+ lstat
+ stat
+ __fxstat
+ __fxstatat
+ __lxstat
+ __xstat
+
+ # sys/statfs.h
+ fstatfs
+
+ # sys/statvfs.h
+ statvfs
+ fstatvfs
+
+ # unistd.h
+ lockf
+ lseek
+ ftruncate
+ pread
+ preadv
+ pwrite
+ pwritev
+ truncate
+ __pread_chk
+)
+SYMBOLS_REGEX=$(printf '%s|' ${SYMBOLS[@]})
+# The @@ part is to workaround a limitation in pax-utils w/versioned symbols.
+SYMBOLS_REGEX=^(${SYMBOLS_REGEX%|})(@@.*)?$
+
+check_lfs() {
+ local files=$(scanelf -F '%s %p' -qyRgs -${SYMBOLS_REGEX} $@)
+
+ if [[ -n ${files} ]]; then
+ echo
+ eqawarn QA Notice: The following files were not built with LFS
support:
+ eqawarn Please see https://bugs.gentoo.org/471102 for
details.
+ eqawarn ${files}
+ echo
+ fi
+}
+
+filtered_check_lfs() {
+ if ! type -P scanelf /dev/null || has binchecks ${RESTRICT}; then
+ return
+ fi
+
+ # Only check glibc uClibc libraries. Others are presumed to use LFS
by
+ # default (e.g. musl), or it's not relevant (e.g. newlib).
+ case ${CHOST} in
+ *-gnu*|*-uclibc*) ;;
+ *) return ;;
+ esac
+
+ # Only check on 32-bit systems. Filtering by $ARCH here isn't perfect,
but
+ # it should be good enough for our needs.
+ case ${ARCH} in
+ arm|mips|ppc|sh|x86) ;;
+ *) return ;;
+ esac
+
+ # Obviously filter out C libraries themselves :).
+ # The sandbox has to capture all symbols by design.
+ case ${CATEGORY}/${PN} in
+ */glibc|\
+ */uclibc|\
+ */gcc|\
+ sys-apps/sandbox) ;;
+ *) check_lfs ${ED} ;;
+ esac
+}
+
+# Allow for people to run manually for testing/debugging.
+if [[ $# -ne 0 ]]; then
+ eqawarn() { echo * $*; }
+ check_lfs $@
+else
+ filtered_check_lfs
+fi
--
2.4.1
Re: [gentoo-portage-dev] [PATCH] config: skip missing profiles/repo_name warning when using layout.conf:repo-name
On 14 May 2015 19:01, Mike Frysinger wrote: On 14 May 2015 11:43, Brian Dolbec wrote: On Thu, 14 May 2015 06:50:42 -0400 Mike Frysinger wrote: The repo name is considered missing if there is no explicit repo_name file. But if it's set in layout.conf via repo-name, we can skip that. Otherwise we end up with copy paste spew. Yeah, merge please Wasn't there a bug for this? If there is pleas add the bug # to the commit message i'm not seeing one ... searched for repo-name and repo_name ... i've pushed this as-is then -mike signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] config: skip missing profiles/repo_name warning when using layout.conf:repo-name
On 14 May 2015 11:43, Brian Dolbec wrote: On Thu, 14 May 2015 06:50:42 -0400 Mike Frysinger wrote: The repo name is considered missing if there is no explicit repo_name file. But if it's set in layout.conf via repo-name, we can skip that. Otherwise we end up with copy paste spew. Yeah, merge please Wasn't there a bug for this? If there is pleas add the bug # to the commit message i'm not seeing one ... searched for repo-name and repo_name ... -mike signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH] config: skip missing profiles/repo_name warning when using layout.conf:repo-name
From: Bertrand Simonnet bsimon...@chromium.org
The repo name is considered missing if there is no explicit repo_name
file. But if it's set in layout.conf via repo-name, we can skip that.
Otherwise we end up with copy paste spew.
---
pym/portage/repository/config.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index e44b619..7401b3a 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -252,6 +252,7 @@ class RepoConfig(object):
# useful when having two copies of the same
repo enabled
# to avoid modifying profiles/repo_name in one
of them
self.name = layout_data['repo-name']
+ self.missing_repo_name = False
for value in ('allow-missing-manifest',
'allow-provide-virtual', 'cache-formats',
--
2.4.0
[gentoo-portage-dev] [PATCH] Do not interrupt on SIGCONT
From: Bertrand SIMONNET bsimon...@chromium.org SIGCONT signals should not interrupt any system calls (locking or wait pid for example). URL: http://crbug.com/417800 X-Gentoo-Bug-URL: https://bugs.gentoo.org/500436 --- pym/_emerge/Scheduler.py | 1 + 1 file changed, 1 insertion(+) diff --git a/pym/_emerge/Scheduler.py b/pym/_emerge/Scheduler.py index d6db311..6e3bf1a 100644 --- a/pym/_emerge/Scheduler.py +++ b/pym/_emerge/Scheduler.py @@ -1017,6 +1017,7 @@ class Scheduler(PollScheduler): earlier_sigterm_handler = signal.signal(signal.SIGTERM, sighandler) earlier_sigcont_handler = \ signal.signal(signal.SIGCONT, self._sigcont_handler) + signal.siginterrupt(signal.SIGCONT, False) try: rval = self._merge() -- 2.3.0
Re: [gentoo-portage-dev] [RFC] Inherit updates settings from master repositories by default
On Wednesday, February 05, 2014 19:11:12 Sebastian Luther wrote: Am 05.02.2014 09:03, schrieb Mike Frysinger: On Saturday, February 01, 2014 20:38:05 Arfrever Frehtes Taifersar Arahesis wrote: this i'm not so sure about. when you have a local overlay, portage complains when there are no masters which means most people have just blindly added masters = gentoo. but if they have packages in there using the old name (on purpose), then updates will constantly tromp on that. The old behavior was to always apply the updates from ::gentoo as long as the repo didn't have its own updates. This means it doesn't matter if the repo sets the masters = gentoo as long as it doesn't contain updates. ok, i thought it always ignored the gentoo updates dir at least, there should be one of: - one-time automatic migration of existing layout.conf files where we set updates-master = for them. How do you know if it's the user's repo or a layman repo, where layout.conf is manged by other people? you ask layman. this isn't difficult. - a warning phase where we complain if the field isn't set, and we default to current behavior. once some time has elapsed, we stop warning and we change the default. Be sure to only hit users which are really affected by the change (i.e. repos with existing updates and master repos which contain updates, which affect packages in the repo). doing it based on the current set of affected packages doens't make sense then the set of possible updates is constantly changing -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [RFC] Inherit updates settings from master repositories by default
On Wednesday, February 05, 2014 03:03:10 Mike Frysinger wrote: On Saturday, February 01, 2014 20:38:05 Arfrever wrote: If this attribute is not set explicitly, then it defaults to value of 'masters' attribute. this i'm not so sure about. when you have a local overlay, portage complains when there are no masters which means most people have just blindly added masters = gentoo. but if they have packages in there using the old name (on purpose), then updates will constantly tromp on that. if portage is already processing updates from the main gentoo-x86 tree in repos (that lack an updates/ dir) then i think it simplifies things to: - if updates/ exists in the repo and the attribute is not explicitly set, then default to instead of the value of masters and issue a warning telling the user to explicit set the attribute -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Implement --newrepo flag.
On Friday, February 14, 2014 09:31:19 David James wrote: The --newrepo flag tells emerge to recompile a package if it is now being pulled from a different repository. BUG=chromium:200417 TEST=Verify ebuilds get pulled in when repo changes. --- man/emerge.1 | 5 +++ pym/_emerge/create_depgraph_params.py | 1 + pym/_emerge/depgraph.py | 80 ++- pym/_emerge/help.py | 2 +- 4 files changed, 48 insertions(+), 40 deletions(-) looks like your client line wrapped everything which is why it's unusable. i guess you need to use `git send-email` or tell your client to disable line wrapping when sending out patches. unless gmail is your client in which case you have to use `git` ;). -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [RFC] Location of Portage bash API (outside of ebuilds)
On Saturday, February 01, 2014 21:08:11 Arfrever Frehtes Taifersar Arahesis wrote: bin/isolated-functions.sh contains at least 1 useful function, which could be exposed for external consumers (without __ prefix), but must have private name (with __ prefix) when bin/isolated-functions.sh is used in ebuild environment. who are these mysterious external consumers ? what you propose means all funcs in there now become exported API and we now have to live with that. w/out any real background, i'm very hesitant to allow that. Possible solutions: 1. Make /usr/lib/portage/bin/isolated-functions.sh magically define non-private variants of useful functions when run in non-ebuild environment. this is a no-go. we should be cutting down on internal overhead. 2. Provide /usr/bin/portage.bash, which would source isolated-functions.sh (and maybe other files) and define non-private variants of useful functions. /usr/bin/portage.bash would be easier sourceable due to PATH. 3. Provide /usr/lib/portage.bash, which would source isolated-functions.sh (and maybe other files) and define non-private variants of useful functions. 4. Another location... (I would probably prefer solution #2.) sounds like a file that should be sourced only which imo bans it from $PATH. i'm aware of the magic shell behavior where `source` searches PATH, but that's not an acceptable reason imo. easy to add something like `portageq helpers-dir` that'd give you a path and then you use that to load the various scripts directly. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000
On Monday, January 27, 2014 15:02:30 viv...@gmail.com wrote: patch install from coreutils (and then upstream changes) is not an option? that route is being pursued independently. we already have a wrapper and will have one for the foreseeable future. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Add the has function to the ebuild(5) man page.
On Wednesday 22 January 2014 16:22:35 Michael Orlitzky wrote: I WTF'ed on this for a long time before I noticed that the docs for has were sort-of contained in hasv. Might as well give has its own. please use `git send-email` when posting patches this duplicates the hasv section which is not a good thing. i rewrote hasv to refer to the has function, fixed up the commit message, and merged it. http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commitdiff;h=669ccb584f4d39b764223ae44cb2d45ac6eb3903 -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH v3] Test for read-only filesystems, bail out during preinst if there are any which will be written to and display a useful error message. Fixes bug 378869.
On Monday 20 January 2014 22:53:25 Gordon Pettey wrote: If your going to make that argumint, ewe mite a's well right the documentation in LOL-1337. Encouraging bad grammar in documentation just make's thing's harder for everybody. (1) don't top post (2) you posted nothing to support your assertion that using 's to pluralize acronyms is bad grammar (3) i posted information that shows it's an accepted form so if you don't have anything useful to post, then just refrain from doing so -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH v5] Test for read-only filesystems, bail out during preinst if there are any which will be written to and display a useful error message. Fixes bug 378869.
On Monday 20 January 2014 23:00:31 Chris Reffett wrote: + if not ro_filesystems: + return ro_filesystems + + for directory in dir_list: + for filesystem in ro_filesystems: + if filesystem == directory: + ro_filesystems_written.add(filesystem) + + return ro_filesystems_written is this just the intersection of two sets ? so all this logic could be: return ro_filesystems ^ set(dir_list) -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH v2] prepman: do not compress files =128 bytes
On Thursday 21 March 2013 20:17:30 Mike Frysinger wrote: On Thursday 21 March 2013 19:25:08 James Cloos wrote: MF == Mike Frysinger vap...@gentoo.org writes: MF realistically, there are no man pages that are under 4k. look at the MF referenced bug for more details. Oh. I didn't look closely enough to notice that it was only for man; I presumed that it would apply to /usr/share/doc/ as well the subject does say prepman ;) we'll probably do something for doc too. my focus was on man. portage-2.2.8 now implements this policy for all files given to ecompress -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH 0/3] Initial fetch() refactoring
On Sunday 19 January 2014 17:46:36 Alec Warner wrote: On Sun, Jan 19, 2014 at 2:45 PM, Alexander Berntsen wrote: On 19/01/14 22:22, Sebastian Luther wrote: The usual doc string style used in portage is: text Please use that for new functions. Also make sure you don't use spaces to indent the last . As mentioned by Mike in another thread, we should use PEP 257[0]. I will convert old code to conform to this... sometime... soon... (I promise!) So if new patches could just do that right away, that would be neat. Does pylint or pyflakes point out if you mess it up? Automation for the win. the good news is that i wrote a pylintrc module for Chromium OS to enforce docstring style. the bad news is that it doesn't work with pyflakes. https://chromium.googlesource.com/chromiumos/chromite/+/master/cros/commands/lint.py what we did there was just merge it and then have people fix things up as they went rather than try to clean it all up first. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH v3 0/4] Initial fetch() refactoring
On Sunday 19 January 2014 22:26:06 W. Trevor King wrote: W. Trevor King (4): pym/portage/package/ebuild/fetch.py: Factor out _get_checksum_failure_max_tries pym/portage/package/ebuild/fetch.py: Factor out _get_fetch_resume_size pym/portage/package/ebuild/fetch.py: Factor out _get_uris pym/portage/package/ebuild/fetch.py: Flatten conditionals in _get_fetch_resume_size no need to use the full file path. a simpler prefix is fine like: ebuild: fetch: ... -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH v3] Test for read-only filesystems, bail out during preinst if there are any which will be written to and display a useful error message. Fixes bug 378869.
On Sunday 19 January 2014 05:39:25 Alexander Berntsen wrote: On 19/01/14 10:17, Mike Frysinger wrote: prefer OSes - OS's That's just not proper English. i don't think that phrase means what you think it means. if you're wishing for English to be a standard, then you're in for a rude surprise. It makes no sense. of course it does. Please don't prefer that. If for nothing else, then to prevent me from sighing whenever I have to read it. https://en.wikipedia.org/wiki/English_plurals#Plurals_of_letters_and_abbreviations -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH v4] Test for read-only filesystems, bail out during preinst if there are any which will be written to and display a useful error message. Fixes bug 378869.
On Monday 20 January 2014 19:50:54 Chris Reffett wrote: + # Check for read-only filesystems please use full sentences. that means putting a period at the end. this applies to many additions in this patch. + try: + with io.open(/proc/mounts, mode='r', encoding=_encodings['content'], + errors='replace') as f: + roregex = re.compile(r'(\A|,)ro(\Z|,)') + for line in f: + if roregex.search(line.split( )[3].strip()) is not None: + romount = line.split( )[1].strip() + ro_filesystems.add(romount) pretty sure the body of this with block is indented one too many times -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] Signing off patches
On Saturday 18 January 2014 17:57:38 Tom Wijsman wrote: On Sat, 18 Jan 2014 08:43:12 -0800 W. Trevor King wrote: On Sat, Jan 18, 2014 at 04:02:02PM +0100, Tom Wijsman wrote: I think the idea is that you shouldn't need to refer to an external resource like the mailing list to understand the idea behind the patch, Either someone cares about the background of a patch or he/she doesn't. full details to understand the change must be in the commit message. saying go find it in the mailing list is not a workable solution. this doesn't mean you have to copy paste the entire discussion, but it does mean you have to distill things down. for topics that go on for a while, adding a tag linking to the mailing list archive is certainly OK. when it comes to tags, i only copy in what other people have bothered posting. so if someone posts their Reviewed-by or Acked-by, i'll use them. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] Bugzilla workflow
On Tuesday 14 January 2014 21:58:38 Tom Wijsman wrote: On Sun, 05 Jan 2014 15:42:48 -0800 Brian Dolbec wrote: 2) start working on a solution, a) if you have significant progress, but need more time, mark it accordingly, assign it to yourself, leave a comment, etc. Assigning it to oneself is a quite good idea as it allows to easily keep track of the bugs you are working on, otherwise you have to rely on techniques that aren't optimal; which are unfortunate. In the lists of all bugs, that can be obtained by checking out the product and/or categories; this gives a quite clear overview of who is working on what, as well as which bugs are still free. As this is still able to be done, there seems no need to assign the bug to Portage team. i disagree. dev-portage@ get's cc-ed on bugs when they're being kept abreast of developments (like PMS), or someone just wants an opinion/feedback on an issue. so there's no way to differentiate between bugs that are assigned to the portage team and bugs where the portage team's opinion is being requested. i want a query for the former and i just rely on generated bugzilla e-mails for the latter. what's wrong with using the whiteboard ? it's a free text field and you can easily construct a query that produces exactly what you want. just stick in your username in there. 3) commit the fix. Mark it as InVCS, if not already, set status to IN_PROGRESS InVCS becomes redundant; other than that, good. i don't see how it's redundant. there is no other flag that indicates things have been fixed in the git tree and the only reason the bug remains open is that a release has not yet been cut. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH v3] Test for read-only filesystems, bail out during preinst if there are any which will be written to and display a useful error message. Fixes bug 378869.
On Saturday 18 January 2014 21:00:48 Chris Reffett wrote:
--- /dev/null
+++ b/pym/portage/util/rochecker.py
@@ -0,0 +1,81 @@
+#-*- coding:utf-8 -*-
+# Copyright 2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+Methods to check whether Portage is going to write to read-only
filesystems. +Since the methods are not portable across different OSes,
each OS needs its +own method. To expand RO checking for different OSes,
add a method which +accepts a list of directories and returns a list of
mounts which need to be +remounted RW, then add elif ostype == (the
ostype value for your OS) to
prefer OSes - OS's
+get_ro_checker()
+
period at the end
+def get_ro_checker():
+
+ Uses the system type to find an appropriate method for testing whether
Portage
+ is going to write to any read-only filesystems
+
+ @return:
+ 1. A method for testing for RO filesystems appropriate to the current
system
+
not a new issue, but we really need to get away from this style and use
PEP257. someone feel like fixing that in the code base ? :)
+ if ostype == Linux:
+ return linux_ro_checker
+ else:
+ return empty_ro_checker
isn't this a glorified dict ?
_CHECKERS = {
'Linux': linux_ro_checker,
}
def get_ro_checker()
return _CHECKERS.get(ostype, empty_ro_checker)
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH 2/3] emerge: Rename --autounmask-write to --autounmask
On Saturday 18 January 2014 19:21:11 Alexander Berntsen wrote: Rename --autounmask-write to --autounmask. typically when we delete/rename an option, we give users a heads up. that means a cycle where the old option exists but emits a warning before switching to the new one. then we can delete it. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH v4] Add repoman check to warn if src_prepare/src_configure are used in EAPI 0/1 and if pkg_pretend is used in EAPI 4. Fixes bug 379491.
Acked-by: Mike Frysinger vap...@gentoo.org btw you should keep the patch summary short. if it goes long, better to move it to the main body. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH 3/3] Have repoman deprecate G2CONF for the GNOME team. (bug #482084).
On Wednesday 15 January 2014 19:07:20 Tom Wijsman wrote: +class DeprecateG2CONF(LineCheck): + repoman_check_name = 'G2CONF.deprecated' + re = re.compile(r'.*G2CONF.*') + + def check(self, num, line): + Run the check on line and return error if there is one + m = self.re.match(line) use re.search instead of re.match so you can drop the redundant .*. further, i think you want to use word boundaries. so: re = re.compile(r'\bG2CONF\b') -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH 3/3] Have repoman deprecate G2CONF for the GNOME team. (bug #482084).
btw, suggestion for commit summary/message: repoman: deprecate G2CONF Deprecate the G2CONF variable as requested by the GNOME team. URL: http://bugs.gentoo.org/482084 -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH 1/3 v2] Have repoman check if the packages to unpack rare archive formats from SRC_URI are present in DEPEND (bug #205909).
On Friday 17 January 2014 18:03:57 Tom Wijsman wrote:
---
please shorten your commit summary and move more content to the body
+getNonSystemArchiveDepends.archivers = {
it is super weird to attach to the object like this. some might even say
wrong. move it into the class definition.
def getNonSystemArchiveDepends(fetchlist, eapi):
...
ARCHIVERS = {
...
}
+ re.compile('.*\.7[zZ]$'):app-arch/p7zip,
regexes should always use raw strings. there should also be a space after the
colon in dicts. so you want:
re.compile(r'.*\.7[zZ]$'): 'app-arch/p7zip',
+ re.compile('.*\.lzma$'):app-arch/lzma-utils,
xz-utils, not lzma-utils
+ re.compile('.*\.(bz2?|tbz2)$'):app-arch/bzip2,
+ re.compile('.*\.(tar(\.(bz2?|gz|Z))?|tbz2|t[bg]z)?$'):app-arch/tar,
+ re.compile('.*\.(gz|tar\.Z|t[bg]z|[zZ])$'):app-arch/gzip,
+ re.compile('.*\.tar.xz$'):app-arch/tar,
requiring people list gzip, tar, and bzip2 is a significant policy change
(which i'm inclined to say is wrong). it needs discussion on the dev mailing
list first.
+def checkArchiveDepends(atoms, catpkg, relative_path, \
+ system_set_atoms, needed_unpack_depends, stats, fails):
you don't need the \ there because you have paren already to gather things.
+ for entry in needed_unpack_depends[catpkg]:
+ if entry not in [atom.cp for atom in atoms if atom != ||]:
+ stats['unpack.' + mytype + '.missing'] += 1
+ fails['unpack.' + mytype + '.missing'].append( \
+ relative_path + : %s is missing in %s % \
+ (entry, mytype))
i know existing style doesn't follow it, but imo we should avoid string
concatenation. it makes the code harder to read imo.
key = 'unpack.%s.missing' % mytype
stats[key] += 1
fails[key].append(...)
+def eapi_has_xz_utils(eapi):
+ return eapi not in (0, 1, 2)
i would use eapi_supports_xz_archives unless there's a pre-existing style
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Add repoman check to warn if src_prepare/src_configure are used in EAPI 0/1
On Tuesday 14 January 2014 16:15:48 Tom Wijsman wrote: On Mon, 13 Jan 2014 23:59:11 -0500 Mike Frysinger wrote: we probably should just use dev branches in the main repo, at least for people who have write access to the repo dev/$USERNAME/whatever you want To be more clear, which one? g.o.g.o, GitHub or is one of both fine? g.o.g.o r'\s*src_(configure|prepare)\s*\(\)' You can then proceed further and move the re outside: the idea was to walk a balance between simplicity and maintainability. imo, the fixed version above is the best. What about the latter improvements about the parentheses? seems fine as long as portage supports an EAPI, i see no reason to omit useful checks like this. Repeating my original question in different words: Why is it useful? people run repoman outside of the main tree. we don't really know their desire for EAPI compatibility. legacy/old portage/who knows. Chromium OS for a long time was restricted to EAPI 4 for two reasons -- it had an old portage version (and upgrading to a newer one regressed performance significantly, so we held off until we could figure out why), and it was using a really old stage3 to build the SDK (which means we needed to support upgrading an old system too). -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH 2/3] emerge: Rename --autounmask-write to --autounmask
On Sunday 19 January 2014 05:48:33 Alexander Berntsen wrote: On 19/01/14 10:20, Mike Frysinger wrote: typically when we delete/rename an option, we give users a heads up. that means a cycle where the old option exists but emits a warning before switching to the new one. then we can delete it. We should have a news item. i don't think so. i doubt this flag is in common use enough to warrant spamming every user of Gentoo. especially when the people who do use it can easily be migrated per my recommendation. similarly, people who do use this flag but miss the news would be broken. generally speaking, i think it makes a lot more sense to put the notice at the point where it matters. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Add repoman check to warn if src_prepare/src_configure are used in EAPI 0/1
On Sunday 19 January 2014 11:59:36 Mike Gilbert wrote: On Sun, Jan 19, 2014 at 4:44 AM, Mike Frysinger vap...@gentoo.org wrote: Chromium OS for a long time was restricted to EAPI 4 for two reasons -- it had an old portage version (and upgrading to a newer one regressed performance significantly, so we held off until we could figure out why) I am curious to know more about the performance regression if you can share. Is that something that got fixed, or did you disable some features (like the slot-operator stuff)? we finally tracked it down (was due to new the new FEATURES=merge-sync option. when you're installing 200 to 500 binary packages (with like 32 in parallel), that can easily choke your throughput. some systems saw really excessive latencies (which i would guess was due to their drive taking longer to process dirty blocks). but it took us some time to figure that out as we were making a large version jump and we didn't have too much time to dedicate to tracking it down. lots o bugs to fix. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Add repoman check to warn if src_prepare/src_configure are used in EAPI 0/1
On Monday 20 January 2014 01:02:31 Alec Warner wrote: Are you still doing crazy crap like disabling all of the portage locking? ;p yes no. we make sure we hold the right locks and don't do operations that otherwise cause problems. then we disable the core lock grabbing :P. the current locking logic is pessimistic and causes way too much of a slow down for us to do anything else. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] When must we write tests, when is it optional?
On Saturday 18 January 2014 01:40:04 Mike Frysinger wrote: On Friday 17 January 2014 19:10:56 Tom Wijsman wrote: Looking back on the repoman patches from both me and creffett, we haven't written tests; it's on my plan to still do that, if possible. This makes me wonder: Should writing tests be a requirement? When? i'm not sure if it should be a hard requirement, but it is strongly encouraged i would even encourage reviewers to inquire about tests, especially when the change is not entirely obvious -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Check for and report read-only filesystems
On Friday 10 January 2014 22:07:52 Chris Reffett wrote:
Attached is a patch to test if Portage is going to write to a
read-only filesystem and print out the list of filesystems that need
to be remounted RW. This leaves ${D} intact rather than having some
files moved before hitting the RO filesystem. Fixes bug 378869. Since
git.overlays.gentoo.org is down, I haven't had the chance to rebase
this against latest, but I can resubmit if it doesn't cleanly apply.
This is my first patch to the list, so I apologize if I didn't submit
correctly.
please use `git send-email` to post patches. attaching them makes things
harder to review.
--- /dev/null
+++ b/pym/portage/util/checkwriteable.py
+ with open(/proc/mounts) as procmounts:
this won't work on non-Linux systems
also, as a general style thing, unless there's a real need for the var to have
a full name, just use f or fp
+ roregex = re.compile(r'(\A|,)ro(\Z|,)')
+ for line in procmounts:
+ if roregex.search(line.split( )[3].strip())
is not None:
+ romount = line.split( )[1].strip()
+ ro_filesystems.add(romount)
ad hoc parsing of /proc/mounts isn't a good thing. at the risk of over
engineering, there should be a func that would take care of expanding the
mount paths into a list of namedcollections (one per mount). then you simply
walk it looking at its options.
+ for directory in dir_list:
+ for filesystem in ro_filesystems:
+ if re.match(filesystem, directory):
+ ro_filesystems_written.add(filesystem)
there's no need to use re and in fact you don't want to. you're looking at
wrong strings here, not regexes.
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] First draft of a portage pylint.
people have been using pyflakes of late. not that i'm against adding this.
we should have a wrapper script:
$ cat pylint
#!/bin/sh
exec pylint --rcfile ${0%/*}/pylintrc $@
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] [REV] Added support for variable expansion in source command. wrt bug #490896
cnf/make.conf.example.source_test| 7 +++ cnf/make.conf.example.source_test_after | 8 i don't think test files belong in cnf/. keep them in the same dir as the test that uses them. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Add repoman check to warn if src_prepare/src_configure are used in EAPI 0/1
On Monday 13 January 2014 19:08:30 creff...@gentoo.org wrote: From: Chris Reffett creff...@gentoo.org you might want to set ~/.gitconfig to have your name/email so that git sendemail uses the right info in the actual e-mail From: field + undefined_phases_re = re.compile(r'^\s*(src_configure|src_prepare)\s*\(\)') you use re.match, so technically the ^ is redundant i'd suggest merging the regex a little: r'^\ssrc_(configure|prepare)\s*\(\)' could check for pkg_pretend/pkg_info too (it's new to EAPI 4) the regex is naive and can match valid ebuilds. consider ones that handle $EAPI itself and will call the right funcs as necessary. can you check the tree to see if anyone is doing this ? if so, we should drop a note to the dev list and get them fixed up. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH v3] Added support for variable expansion in source command argument in make.conf
seems reasonable, although i haven't used the shlex module myself before -mike signature.asc Description: This is a digitally signed message part.
[gentoo-portage-dev] [PATCH 3/3] mkrelease: add a --runtests shortcut
This automates the release+test cycle a bit so people don't have to do
it all by hand.
---
DEVELOPING | 7 ++-
mkrelease.sh | 15 +--
2 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/DEVELOPING b/DEVELOPING
index 7aac81b..b704a50 100644
--- a/DEVELOPING
+++ b/DEVELOPING
@@ -167,11 +167,8 @@ Releases
First create a git tag for this release:
git tag v2.2.8
-Then create the tarball:
- ./mkrelease.sh --changelog-rev v2.2.7 --tag 2.2.8
-
-Unpack the tarball and run tests:
- ./runtests.sh --python-versions=supported
+Then create the tarball and run the tests:
+ ./mkrelease.sh --changelog-rev v2.2.7 --tag --runtests 2.2.8
Make sure you have all supported python versions installed first
(see PYTHON_SUPPORTED_VERSIONS in runtests.sh).
diff --git a/mkrelease.sh b/mkrelease.sh
index 50bdb3c..334b4fb 100755
--- a/mkrelease.sh
+++ b/mkrelease.sh
@@ -8,9 +8,10 @@ BRANCH=${BRANCH:-master}
USE_TAG=false
CHANGELOG_REVISION=
UPLOAD_LOCATION=
+RUNTESTS=false
usage() {
- echo Usage: ${0##*/} [--changelog-rev tree-ish] [-t|--tag]
[-u|--upload location] version
+ echo Usage: ${0##*/} [--changelog-rev tree-ish] [-t|--tag]
[-u|--upload location] [--runtests] version
exit ${1:-0}
}
@@ -19,7 +20,7 @@ die() {
usage 1
}
-ARGS=$(getopt -o htu: --long help,changelog-rev:,tag,upload: \
+ARGS=$(getopt -o htu: --long help,changelog-rev:,runtests,tag,upload: \
-n ${0##*/} -- $@)
[ $? != 0 ] die initialization error
@@ -42,6 +43,10 @@ while true; do
-h|--help)
usage
;;
+ --runtests)
+ RUNTESTS=true
+ shift
+ ;;
--)
shift
break
@@ -78,6 +83,12 @@ cp -a ${SOURCE_DIR}/{bin,cnf,doc,man,misc,pym}
${RELEASE_DIR}/ || die direc
cp
${SOURCE_DIR}/{DEVELOPING,LICENSE,Makefile,NEWS,README,RELEASE-NOTES,TEST-NOTES}
\
${RELEASE_DIR}/ || die file copy failed
+if [[ ${RUNTESTS} == true ]] ; then
+ pushd ${SOURCE_DIR} /dev/null
+ ./runtests.sh --python-versions=supported || die tests failed
+ popd /dev/null
+fi
+
rm -rf ${SOURCE_DIR} || die directory cleanup failed
echo Setting portage.VERSION
--
1.8.4.3
[gentoo-portage-dev] [PATCH 2/3] runtests: hardfail when a requested version is not found
This way people don't have to worry about runtests.sh passing for a
specific version when it wasn't found in the system.
---
runtests.sh | 6 ++
1 file changed, 6 insertions(+)
diff --git a/runtests.sh b/runtests.sh
index 3006be5..2870907 100755
--- a/runtests.sh
+++ b/runtests.sh
@@ -30,15 +30,18 @@ interrupted() {
trap interrupted SIGINT
unused_args=()
+IGNORE_MISSING_VERSIONS=true
while [ $# -gt 0 ] ; do
case $1 in
--python-versions=*)
PYTHON_VERSIONS=${1#--python-versions=}
+ IGNORE_MISSING_VERSIONS=false
;;
--python-versions)
shift
PYTHON_VERSIONS=$1
+ IGNORE_MISSING_VERSIONS=false
;;
*)
unused_args[${#unused_args[@]}]=$1
@@ -73,6 +76,9 @@ for version in ${PYTHON_VERSIONS}; do
exit_status=1
fi
echo
+ elif [[ ${IGNORE_MISSING_VERSIONS} != true ]] ; then
+ echo -e ${BAD}Could not find requested Python
${version}${NORMAL}
+ exit_status=1
fi
done
--
1.8.4.3
Re: [gentoo-portage-dev] [PATCH] Document layout.conf more (bug 395359)
extending the example isn't the way to go. see the commit i pushed. if you've got details that i missed, then rebase on top of that and send an update. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH 3/3] mkrelease: add a --runtests shortcut
On Sunday 05 January 2014 13:23:38 Brian Dolbec wrote: On Sun, 2014-01-05 at 10:50 -0500, Mike Frysinger wrote: This automates the release+test cycle a bit so people don't have to do it all by hand. I've added them to my checkout and running it now. If it all works as it should, I'll push them all with the 2.2.8 release. They do not affect portage/emerge, so shouldn't need more testing time. i've pushed all but this last one now. feel free to push it once you've verified things on your side. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Document layout.conf more (bug 395359)
On Sunday 05 January 2014 16:44:31 Sebastian Luther wrote: One thing about the 'profile-formats' key. portage-1-compat is not a valid value (see _valid_profile_formats variable), but pms is. From the use in other places it seems that portage-1-compat is like portage-1, but warns for incompatibilities with pms. portage-1-compat is used internally by portage, but you're right that it's not allowed to be set directly. i'll tweak the section a bit. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] Bugzilla workflow
On Sunday 05 January 2014 17:05:04 Sebastian Luther wrote: For now I tried to handle bugs like Zac did. That is: There always exists a tracker bug, named: [Tracker] sys-apps/portage-next version. would be nice to start adding an alias field. this way you can do: https://bugs.gentoo.org/show_bug.cgi?id=portage-2.2.8 I like this workflow because it makes it easy to see what has been fixed since the last release. The only thing I have no use for, is this InVCS keyword. I do not know what Zac used it for. Does anyone have a use for it? you can add a bug blocking the release tracker that hasn't yet been fixed in git. the InVCS tag let's people filter between the two states. Another topic is the bug status for open bugs, i.e. CONFIRMED, UNCONFIRMED, IN_PROGRESS. I've never bothered with changing them and haven't found them useful, but Brian suggested to use IN_PROGRESS at times. What are your thoughts? i don't personally care about any of those, but some people find it useful. they all make sense, so it's just a matter of people using them to keep track of things. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] Re: [PATCH] econf: update configure/config.{sub,guess} atomically to avoid races
On Friday 20 December 2013 22:07:17 Ryan Hill wrote:
On Tue, 17 Dec 2013 18:28:12 -0500 Mike Frysinger wrote:
Use $BASHPID which will be unique even in subshells.
This requires bash 4.0. I'm all for it, but haters gonna hate.
ugh. fun fact: this is used in multiple places in portage already, as well as
some eclasses.
it can be re-implemented with:
read -r pid _ /proc/self/stat; echo $pid
but that doesn't work in a subshell, so you couldn't do:
bashpid() { local pid; read -r pid _ /proc/self/stat; echo $pid; }
echo ${BASHPID:-$(bashpid)}
because now you'd be a sub-subshell. in this case you want the parent pid,
but that isn't easy to get as the 2nd field of stat is the process name, and
i'm not sure that won't have spaces (since it might be the name of the shell
script). plus it isn't terribly portable.
this seems reasonable portable though:
bashpid() { sh -c 'echo $PPID'; }
echo ${BASHPID:-$(bashpid)}
even if it does have the overhead of running another bash process. but
considering this only impacts bash-3.2, maybe we simply don't care.
-mike
signature.asc
Description: This is a digitally signed message part.
[gentoo-portage-dev] [PATCH] add a __bashpid helper for bash-4.0 versions
The $BASHPID variable is new to bash-4.0, so we need to add fallback logic
to support older versions (notably, bash-3.2).
Reported-by: Ryan Hill dirtye...@gentoo.org
---
bin/ebuild-helpers/prepstrip | 4 ++--
bin/ebuild.sh| 2 +-
bin/helper-functions.sh | 2 +-
bin/isolated-functions.sh| 9 -
bin/phase-functions.sh | 2 +-
bin/phase-helpers.sh | 2 +-
6 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/bin/ebuild-helpers/prepstrip b/bin/ebuild-helpers/prepstrip
index 9b2d47c..64ea80d 100755
--- a/bin/ebuild-helpers/prepstrip
+++ b/bin/ebuild-helpers/prepstrip
@@ -116,7 +116,7 @@ save_elf_sources() {
buildid=$(debugedit -i \
-b ${WORKDIR} \
-d ${prepstrip_sources_dir} \
- -l ${tmpdir}/sources/${x##*/}.${BASHPID} \
+ -l ${tmpdir}/sources/${x##*/}.${BASHPID:-$(__bashpid)} \
${x})
}
@@ -208,7 +208,7 @@ process_elf() {
# see if we can split strip at the same time
if [[ -n ${SPLIT_STRIP_FLAGS} ]] ; then
local shortname=${x##*/}.debug
- local
splitdebug=${tmpdir}/splitdebug/${shortname}.${BASHPID}
+ local
splitdebug=${tmpdir}/splitdebug/${shortname}.${BASHPID:-$(__bashpid)}
${already_stripped} || \
${STRIP} ${strip_flags} \
-f ${splitdebug} \
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 35f4b91..be044e0 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -468,7 +468,7 @@ if [[ -n ${QA_INTERCEPTORS} ]] ; then
fi
# Subshell/helper die support (must export for the die helper).
-export EBUILD_MASTER_PID=$BASHPID
+export EBUILD_MASTER_PID=${BASHPID:-$(__bashpid)}
trap 'exit 1' SIGTERM
if ! has $EBUILD_PHASE clean cleanrm depend \
diff --git a/bin/helper-functions.sh b/bin/helper-functions.sh
index 4a46278..b9bc74a 100644
--- a/bin/helper-functions.sh
+++ b/bin/helper-functions.sh
@@ -34,7 +34,7 @@ __multijob_init() {
}
__multijob_child_init() {
- trap 'echo ${BASHPID} $? '${mj_write_fd} EXIT
+ trap 'echo ${BASHPID:-$(__bashpid)} $? '${mj_write_fd} EXIT
trap 'exit 1' INT TERM
}
diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
index 42d9e70..b99aec7 100644
--- a/bin/isolated-functions.sh
+++ b/bin/isolated-functions.sh
@@ -98,6 +98,13 @@ nonfatal() {
PORTAGE_NONFATAL=1 $@
}
+__bashpid() {
+ # The BASHPID variable is new to bash-4.0, so add a hack for older
+ # versions. This must be used like so:
+ # ${BASHPID:-$(__bashpid)}
+ sh -c 'echo ${PPID}'
+}
+
__helpers_die() {
if ___eapi_helpers_can_die; then
die $@
@@ -216,7 +223,7 @@ die() {
[[ -n $PORTAGE_IPC_DAEMON ]] $PORTAGE_BIN_PATH/ebuild-ipc exit 1
# subshell die support
- [[ $BASHPID = $EBUILD_MASTER_PID ]] || kill -s SIGTERM
$EBUILD_MASTER_PID
+ [[ ${BASHPID:-$(__bashpid)} == ${EBUILD_MASTER_PID} ]] || kill -s
SIGTERM ${EBUILD_MASTER_PID}
exit 1
}
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index 711b721..f39a024 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -830,7 +830,7 @@ __ebuild_main() {
# setup EBUILD_MASTER_PID to refer to the current $BASHPID,
# which seems to give the best results when further
# nested subshells call die.
- export EBUILD_MASTER_PID=$BASHPID
+ export EBUILD_MASTER_PID=${BASHPID:-$(__bashpid)}
trap 'exit 1' SIGTERM
#a reasonable default for $S
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index 1a7ae03..412decb 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -469,7 +469,7 @@ unpack() {
econf() {
local x
- local pid=${BASHPID}
+ local pid=${BASHPID:-$(__bashpid)}
if ! ___eapi_has_prefix_variables; then
local EPREFIX=
--
1.8.4.3
Re: [gentoo-portage-dev] musings on config.{sub,guess} replacement living in econf (was: [PATCH] econf: update configure/config.{sub,guess} atomically to avoid races)
On Wednesday 18 December 2013 17:20:32 Greg Turner wrote:
My hunch is that the decision to put the config.{sub,guess}
replacement code in econf was intended as a quick-and-dirty way to
avoid doing the replacements, in cases where no configure script runs
in an ebuild.
it was intended as the easy answer so that all packages would get it for
free. previously, we had an eclass, and had to update every single package
to call a function in that eclass. it sucked hard. it makes sense to have it
in `econf` because that func is only used in conjunction with autotool based
packages.
Post EAPI-2, the convention that hacking on the sources in ${S} is
a no-no after src_prepare has clearly crystallized considerably (I'm
guessing the code has EAPI-[01] origins); violating that convention in
econf seems awkward.
this code existed long before EAPI was ever a thing
o It doesn't run, if, for some reason, the ebuild must invoke
configure directly rather than use econf
this happens in like 3 packages. we can suck it up.
o when econf is invoked repeatedly, it does the same
O(# of dirs in ${S}) noop over and over
true, but it hasn't been a big enough deal for us to care
In short... moving the config.{sub,guess} replacement code (but
probably not the shebang patching for reasons of expedience) to some
post-src_prepare place would probably be more elegant and pretty easy
to do.
that discussion should happen on gentoo-dev in conjunction with PMS. or file a
bug if there isn't one already.
-mike
signature.asc
Description: This is a digitally signed message part.
[gentoo-portage-dev] [PATCH] econf: update config.{sub,guess} atomically to avoid races
URL: https://bugs.gentoo.org/487478
---
bin/phase-helpers.sh | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index ec48c94..59c053a 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -512,7 +512,9 @@ econf() {
-name config.guess -o -name config.sub ')' -print0 | \
while read -r -d $'\0' x ; do
__vecho * econf: updating ${x/${WORKDIR}\/}
with ${EPREFIX}/usr/share/gnuconfig/${x##*/}
- cp -f
${EPREFIX}/usr/share/gnuconfig/${x##*/} ${x}
+ # Make sure we do this atomically incase we're
run in parallel. #487478
+ cp -f
${EPREFIX}/usr/share/gnuconfig/${x##*/} ${x}.$$
+ mv -f ${x}.$$ ${x}
done
fi
--
1.8.4.3
[gentoo-portage-dev] [PATCH] econf: update configure/config.{sub,guess} atomically to avoid races
Use $BASHPID which will be unique even in subshells.
URL: https://bugs.gentoo.org/487478
---
bin/phase-helpers.sh | 17 +++--
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index ec48c94..1a7ae03 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -469,6 +469,7 @@ unpack() {
econf() {
local x
+ local pid=${BASHPID}
if ! ___eapi_has_prefix_variables; then
local EPREFIX=
@@ -501,18 +502,22 @@ econf() {
if [[ -n $CONFIG_SHELL \
$(head -n1 $ECONF_SOURCE/configure) =~
^'#!'[[:space:]]*/bin/sh([[:space:]]|$) ]] ; then
# preserve timestamp, see bug #440304
- touch -r $ECONF_SOURCE/configure
$ECONF_SOURCE/configure._portage_tmp_.$$ || die
- sed -e 1s:^#![[:space:]]*/bin/sh:#!$CONFIG_SHELL: -i
$ECONF_SOURCE/configure || \
- die Substition of shebang in
'$ECONF_SOURCE/configure' failed
- touch -r $ECONF_SOURCE/configure._portage_tmp_.$$
$ECONF_SOURCE/configure || die
- rm -f $ECONF_SOURCE/configure._portage_tmp_.$$
+ touch -r ${ECONF_SOURCE}/configure
${ECONF_SOURCE}/configure._portage_tmp_.${pid} || die
+ sed -i \
+ -e 1s:^#![[:space:]]*/bin/sh:#!$CONFIG_SHELL:
\
+ ${ECONF_SOURCE}/configure \
+ || die Substition of shebang in
'${ECONF_SOURCE}/configure' failed
+ touch -r
${ECONF_SOURCE}/configure._portage_tmp_.${pid} ${ECONF_SOURCE}/configure ||
die
+ rm -f ${ECONF_SOURCE}/configure._portage_tmp_.${pid}
fi
if [ -e ${EPREFIX}/usr/share/gnuconfig/ ]; then
find ${WORKDIR} -type f '(' \
-name config.guess -o -name config.sub ')' -print0 | \
while read -r -d $'\0' x ; do
__vecho * econf: updating ${x/${WORKDIR}\/}
with ${EPREFIX}/usr/share/gnuconfig/${x##*/}
- cp -f
${EPREFIX}/usr/share/gnuconfig/${x##*/} ${x}
+ # Make sure we do this atomically incase we're
run in parallel. #487478
+ cp -f
${EPREFIX}/usr/share/gnuconfig/${x##*/} ${x}.${pid}
+ mv -f ${x}.${pid} ${x}
done
fi
--
1.8.4.3
Re: [gentoo-portage-dev] [PATCH] QA warning for files in /var/{cache,lib,lock,run}/ or /run/ (bug 493154)
On Wednesday 11 December 2013 03:43:12 Sebastian Luther wrote: Am 11.12.2013 08:57, schrieb Mike Frysinger: i'll send out a new version. FWIW, new patch looks fine. thanks for the review ... i pushed that version -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] QA warning for files in /var/{cache,lib,lock,run}/ or /run/ (bug 493154)
On Thursday 05 December 2013 15:57:17 sebastianlut...@gmx.de wrote:
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -242,6 +242,11 @@ install_qa_check() {
[[ -d ${ED}/$x ]] f+= $x\n
done
+ # It's ok create these directories, but not to install into them. (bug
493154)
our style uses #12345 rather than (bug 12345)
# It's ok create these directories, but not to install into them. #493154
+ for x in var/cache var/lib var/lock var/run run ; do
+ [[ -d ${ED}/$x ]] [[ $(find ${ED}/${x} -prune -empty) =
]]
f+= $x\n
the -d check doesn't handle symlinks correctly ... baselayout would install
/var/run - /run, and the -d would deref that to the real symlink. however,
`find` will not descend into the symlink, so we end up being saved by that.
we have -z for detecting empty output rather than comparing to an empty string
non-builtin vars should use braces, so that'd be ${x}. i know some of the
code in here doesn't follow that, but we should be marching in that direction
with new code.
if [[ -n $f ]] ; then
eqawarn QA Notice: This ebuild installs into the following
deprecated directories:
this warning is incorrect for these paths. we need to create a new warning
that clearly explains what is going on.
i'll send out a new version.
-mike
signature.asc
Description: This is a digitally signed message part.
[gentoo-portage-dev] [PATCH v2] QA warning for files in /var/{cache,lock,run}/ or /run/
From: Sebastian Luther sebastianlut...@gmx.de
No warning will be issued if the directories are created, but are
left empty.
URL: https://bugs.gentoo.org/493154
---
bin/misc-functions.sh | 19 ++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index bd99def..2c4d248 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -241,13 +241,30 @@ install_qa_check() {
for x in etc/app-defaults usr/man usr/info usr/X11R6 usr/doc usr/locale
; do
[[ -d ${ED}/$x ]] f+= $x\n
done
-
if [[ -n $f ]] ; then
eqawarn QA Notice: This ebuild installs into the following
deprecated directories:
eqawarn
eqawarn $f
fi
+ # It's ok create these directories, but not to install into them.
#493154
+ # TODO: We should add var/lib to this list.
+ f=
+ for x in var/cache var/lock var/run run ; do
+ if [[ ! -L ${ED}/${x} -d ${ED}/${x} ]] ; then
+ if [[ -z $(find ${ED}/${x} -prune -empty) ]] ; then
+ f+=$(cd ${ED}; find ${x} -printf ' %p\n')
+ fi
+ fi
+ done
+ if [[ -n ${f} ]] ; then
+ eqawarn QA Notice: This ebuild installs into paths that should
be created at runtime.
+ eqawarn To fix, simply do not install into these directories.
Instead, your package
+ eqawarn should create dirs on the fly at runtime as needed
via init scripts/etc...
+ eqawarn
+ eqawarn ${f}
+ fi
+
set +f
f=
for x in ${ED}etc/udev/rules.d/* ${ED}lib*/udev/rules.d/* ; do
--
1.8.4.3
Re: [gentoo-portage-dev] [PATCH] Remove ban on installing *.a and *.la files in / (bug 492542)
i don't think we want this. i've followed up on the bug. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] make.conf.5: Document PYTHON_TARGETS, bug #493180
On Tuesday 03 December 2013 04:19:27 Alexander Berntsen wrote: Using git send-email since vapier told me to. I haven't used this before, so sorry if I have done something wrong. Please let me know. looks ok from `git send-email` pov, but random comments not meant for the commit message should be below the --- line ;) as for the patch, i'm of the opinion that make.conf is not for documenting random USE_EXPAND-ed variables. this sort of thing should go into the relevant eclass, python-r1.eclass in this case. then people would read about it via `man python-r1.eclass`. there is the matter of visibility ... we could add a generic pointer to the make.conf man page discussing that there are many more variables that might impact the build and to look at the man page for the eclasses. -mike signature.asc Description: This is a digitally signed message part.
