Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command
On 12/17/2014 03:57 PM, Zac Medico wrote: > On 12/17/2014 02:32 PM, Brian Dolbec wrote: >> On Wed, 17 Dec 2014 12:30:53 -0800 > >> But what about the different sync types. How do we handle this in a >> git repo instead of an rsync one? > > I was thinking that we would only need to implement this for the rsync > repo, since that's the only one that includes the glsas. If there's some > reason to implement it for other sync types in the future, then we can > do it when the need arises. Also, at some point we might introduce a notion of submodules, similar to git submodules, and if necessary each submodule could potentially be synced using a different protocol. However, this is way beyond the present needs. For now, simply extending the rsync module to allow syncing of sub-directories should be sufficient (though I would not expose the ability to sync arbitrary directories through the user interface). -- Thanks, Zac
[gentoo-portage-dev] [RFC] add emerge --changed-deps and --binpkg-changed-deps options, analogous to --newuse and --binpkg-respect-use
Hi, The @changed-deps set is useful, but it has limitations similar to the @installed set [1], which can make it unsuitable for use when updating the whole system. In order to solve the problems of @changed-deps, we can implement two new options that are analogous to --newuse and --binpkg-respect-use, called --changed-deps and --binpkg-changed-deps. The rationale for having separate a --binpkg-* option is the same in both cases: depending on the situation, people may want different behavior for binary packages. For example, just like ---binpkg-respect-use is automatically enabled if the user has not specified --usepkgonly, we could do the same for --binpkg-changed-deps (but allow the user to explicitly override the automatic behavior). Please respond if you agree or disagree with this proposal. [1] https://bugs.gentoo.org/show_bug.cgi?id=387059 -- Thanks, Zac
Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command
On 12/17/2014 02:32 PM, Brian Dolbec wrote: > On Wed, 17 Dec 2014 12:30:53 -0800 > Zac Medico wrote: > >> Hi, >> >> For server deployments, it's common for administrators to want to use >> glsa-check for security vulnerabilities, without necessarily wanting >> to to do a full sync [1]. >> >> So, I propose that we add an 'emerge --sync-glsa' action and a >> corresponding 'emaint sync-glsa' command that will only sync the >> relevant metadata/glsa subdirectory of the relevant repository(s). >> Please respond if you agree or disagree with this proposal. >> >> [1] https://bugs.gentoo.org/show_bug.cgi?id=89641 > > I agree in principal, but you didn't say enough about your idea to be > able to say yes/no to it yet. Yeah, I need to explain the implementation ideas that were floating around in my head when I made the proposal. > I've read the bug comments. > > So, what you are saying for the above is set a very restrictive rsync > to just sync the glsa directory? > > So, with the new pluging-sync sytem in place, how do you propose we do > this? > > 1) Make a new emaint sync option which does the restriction and > appropriate sync module calls? > emaint sync --glsa & emerge --sync-glsa > > 2) emaint sync-glsa sounds like a modified copy of the sync module. >To me this is a poor idea. It would needlessly duplicate code. I was thinking that we could tag the repos containing glsas using an attribute in repos.conf, and teach the existing rsync module how to sync the glsas alone. > But what about the different sync types. How do we handle this in a > git repo instead of an rsync one? I was thinking that we would only need to implement this for the rsync repo, since that's the only one that includes the glsas. If there's some reason to implement it for other sync types in the future, then we can do it when the need arises. > > 3) if it is a separate repo, then the current emaint sync > module does not need __ANY__ modifications. > > emaint sync -r glsa <== would sync the repo named glsa no > matter what type of sync method is used. rsync, git, svn, > cvs,... > > Another advantage to a separate repo, for your use case that > they don't want the main repo to be synced, the new sync system > adds an "auto-sync" variable. "emerge --sync" will only sync > repos marked with it set to yes. So, if all they want is the > glsa's synced automatically, set only i'ts auto-sync to yes. > All other repos will have to be synced manually via the "emaint > sync --repo foo" command which does not look at the auto-sync > variable. Well, it sounds like you're introducing a new "repo" type that's different from a package repository. That's not really necessary if we take the approach that I've describe above. > Bottom line of this comes down to getting infra to create a standalone > repo for the glsa's. The current rsync tree could merge the gentoo and > glsa repos for backwards compatibility. For your use case, they set the > gentoo repo's rsync restriction to exclude the glsa's. And have the > separate glsa repo downloaded as it's own repository. With the gentoo > tree moving to git, then the git tree would not include the glsa's, so > the user would need to install the glsa tree as well. Note that we can even re-use the existing rsync tree if we take the "separate repo" approach, and simply configure the "separate repo" to sync from rsync://rsync.gentoo.org/gentoo-portage/metadata/glsa instead of rsync://rsync.gentoo.org/gentoo-portage. > Only code changes I see to portage, pkgcore (I know nothing about > paludis) are to look for the glsa's in the 2 possible locations. The > standalone glsa repo, failing that, backup to the gentoo tree. Well, is there anything wrong with the tagging the existing repo in repos.conf as I've described above? -- Thanks, Zac
Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command
On Wed, 17 Dec 2014 17:49:56 -0500 Michael Orlitzky wrote: > On 12/17/2014 05:32 PM, Brian Dolbec wrote: > > > > Only code changes I see to portage, pkgcore (I know nothing about > > paludis) are to look for the glsa's in the 2 possible locations. > > The standalone glsa repo, failing that, backup to the gentoo tree. > > > > Could we ship a GLSA overlay enabled by default, regardless of what we > do with the main repo? Then for simplicity, we update the tools to > check metadata/glsa in overlays as well. No need to special-case a > fallback location. > > glsa-check would of course want to sync the GLSA repo before running. > > yes, that is a very simple edit to the default repos.conf file adding the glsa repo. Yes, I forgot that glsa-check needed changes so it would/could be configured to sync the repo too. Pkgcore does not have the capability (yet) to read/use extended repos.conf repo attributes. So, would need more code changes. I think pkgcore does not yet handle repos.conf files... I'm not sure. -- Brian Dolbec
Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command
On 12/17/2014 05:32 PM, Brian Dolbec wrote: > > Only code changes I see to portage, pkgcore (I know nothing about > paludis) are to look for the glsa's in the 2 possible locations. The > standalone glsa repo, failing that, backup to the gentoo tree. > Could we ship a GLSA overlay enabled by default, regardless of what we do with the main repo? Then for simplicity, we update the tools to check metadata/glsa in overlays as well. No need to special-case a fallback location. glsa-check would of course want to sync the GLSA repo before running.
Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command
On Wed, 17 Dec 2014 12:30:53 -0800 Zac Medico wrote: > Hi, > > For server deployments, it's common for administrators to want to use > glsa-check for security vulnerabilities, without necessarily wanting > to to do a full sync [1]. > > So, I propose that we add an 'emerge --sync-glsa' action and a > corresponding 'emaint sync-glsa' command that will only sync the > relevant metadata/glsa subdirectory of the relevant repository(s). > Please respond if you agree or disagree with this proposal. > > [1] https://bugs.gentoo.org/show_bug.cgi?id=89641 I agree in principal, but you didn't say enough about your idea to be able to say yes/no to it yet. I've read the bug comments. So, what you are saying for the above is set a very restrictive rsync to just sync the glsa directory? So, with the new pluging-sync sytem in place, how do you propose we do this? 1) Make a new emaint sync option which does the restriction and appropriate sync module calls? emaint sync --glsa & emerge --sync-glsa 2) emaint sync-glsa sounds like a modified copy of the sync module. To me this is a poor idea. It would needlessly duplicate code. But what about the different sync types. How do we handle this in a git repo instead of an rsync one? 3) if it is a separate repo, then the current emaint sync module does not need __ANY__ modifications. emaint sync -r glsa <== would sync the repo named glsa no matter what type of sync method is used. rsync, git, svn, cvs,... Another advantage to a separate repo, for your use case that they don't want the main repo to be synced, the new sync system adds an "auto-sync" variable. "emerge --sync" will only sync repos marked with it set to yes. So, if all they want is the glsa's synced automatically, set only i'ts auto-sync to yes. All other repos will have to be synced manually via the "emaint sync --repo foo" command which does not look at the auto-sync variable. Bottom line of this comes down to getting infra to create a standalone repo for the glsa's. The current rsync tree could merge the gentoo and glsa repos for backwards compatibility. For your use case, they set the gentoo repo's rsync restriction to exclude the glsa's. And have the separate glsa repo downloaded as it's own repository. With the gentoo tree moving to git, then the git tree would not include the glsa's, so the user would need to install the glsa tree as well. Only code changes I see to portage, pkgcore (I know nothing about paludis) are to look for the glsa's in the 2 possible locations. The standalone glsa repo, failing that, backup to the gentoo tree. -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v2] bintree.py: fix str() calls for Python 2 (532784)
On Wed, 17 Dec 2014 09:59:22 -0800 Zac Medico wrote: > Avoid a UnicodeDecodeError raised when str(e) converts an exception > to bytes with Python 2. Since this file has unicode_literals enabled, > use literal unicode format strings to format messages for unicode > exceptions. However, with Python 2, an EnvironmentError exception > may contain either bytes or unicode, so use _unicode(errors="replace") > to ensure safety for EnvironmentError with all locales. > > Also, convert remaining str() calls to use _unicode() for uniform > behavior regardless of python version. > > X-Gentoo-Bug: 532784 > X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=532784 > --- > PATCH v2 replaces the _unicode_decode call with > _unicode(errors="replace"), which is required in order to force the > EnvironmentError object to be converted to a unicode string. > Yup, LGTM -- Brian Dolbec
[gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command
Hi, For server deployments, it's common for administrators to want to use glsa-check for security vulnerabilities, without necessarily wanting to to do a full sync [1]. So, I propose that we add an 'emerge --sync-glsa' action and a corresponding 'emaint sync-glsa' command that will only sync the relevant metadata/glsa subdirectory of the relevant repository(s). Please respond if you agree or disagree with this proposal. [1] https://bugs.gentoo.org/show_bug.cgi?id=89641 -- Thanks, Zac
[gentoo-portage-dev] [PATCH] Support override of default profile EAPI (532670)
Add support for a profile-default-eapi flag in the metadata/layout.conf profile-formats field. When this flag is enabled, it enables support for a profile_eapi_when_unspecified attribute which can be used to specify the default EAPI to use for profile directories that do not contain an eapi file. For example, the following settings will cause all profiles that do not contain an eapi file to default to EAPI 5 instead of EAPI 0: profile-formats = profile-default-eapi profile_eapi_when_unspecified = 5 This is convenient for organizations that maintain their own profiles in separate repositories from Gentoo, since they typically want to use EAPI 5 for all of their profiles, and this allows them to avoid having separate eapi files in each directory of their profiles. The name of the profile_eapi_when_unspecified attribute is inherited from Paludis, which has supported this attribute in exheres repositories for many years. The default EAPI setting is stored in the RepoConfig.eapi attribute, which is used as the "default" parameter for all read_corresponding_eapi_file calls. Each profile node in LocationsManager.profiles_complex now has an eapi attribute which serves to cache the computed EAPI for that node, allowing redundant read_corresponding_eapi_file calls to be eliminated. As a result, functions such as grabfile_package now have eapi and eapi_default parameters, where eapi is used to pass in a cached EAPI, and eapi_default is used to pass in a default for read_corresponding_eapi_file to use when a cached EAPI is not available. For /etc/portage/profile, the EAPI is considered to have a default value of None, which means that atoms from any supported EAPI are allowed. X-Gentoo-Bug: 532670 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=532670 --- This patch superseedes the eapi-x profile-formats patch, since feedback from all commenters on bug #532670 expressed desires to have the default EAPI expressed using a separate attribute from profile-formats. man/portage.5 | 13 ++- pym/portage/_sets/ProfilePackageSet.py | 3 +- pym/portage/_sets/profiles.py | 27 +++-- .../package/ebuild/_config/KeywordsManager.py | 6 +- .../package/ebuild/_config/LocationsManager.py | 37 -- pym/portage/package/ebuild/_config/MaskManager.py | 12 +- pym/portage/package/ebuild/_config/UseManager.py | 88 +++--- pym/portage/package/ebuild/config.py | 9 +- pym/portage/repository/config.py | 41 ++- .../tests/resolver/test_profile_default_eapi.py| 126 + pym/portage/util/__init__.py | 11 +- 11 files changed, 314 insertions(+), 59 deletions(-) create mode 100644 pym/portage/tests/resolver/test_profile_default_eapi.py diff --git a/man/portage.5 b/man/portage.5 index 88cf3bb..fbff9a6 100644 --- a/man/portage.5 +++ b/man/portage.5 @@ -254,6 +254,11 @@ use.stable.force, package.use.stable.mask and package.use.stable.force. These files behave similarly to previously supported USE configuration files, except that they only influence packages that are merged due to a stable keyword. + +If the eapi file does not exist, then the \fBEAPI\fR defaults to +\fI0\fR unless the default has been overridden by a +profile_eapi_when_unspecified setting inside \fImetadata/layout.conf\fR +of the containing repository. .TP .BR make.defaults The profile default settings for Portage. The general format is described @@ -1114,7 +1119,11 @@ The default setting for repoman's --echangelog option. The cache formats supported in the metadata tree. There is the old "pms" format and the newer/faster "md5-dict" format. Default is to detect dirs. .TP -.BR profile\-formats " = [pms|portage-1|portage-2|profile-bashrcs|profile-set]" +.BR profile_eapi_when_unspecified +The EAPI to use for profiles when unspecified. This attribute is +supported only if profile-default-eapi is included in profile-formats. +.TP +.BR profile\-formats " = [pms] [portage-1] [portage-2] [profile-bashrcs] [profile-set] [profile-default-eapi]" Control functionality available to profiles in this repo such as which files may be dirs, or the syntax available in parent files. Use "portage-2" if you're unsure. The default is "portage-1-compat" mode which is meant to be compatible @@ -1123,6 +1132,8 @@ Setting profile-bashrcs will enable the per-profile bashrc mechanism \fBpackage.bashrc\fR. Setting profile-set enables support for using the profile \fBpackages\fR file to add atoms to the @profile package set. See the profile \fBpackages\fR section for more information. +Setting profile-default-eapi enables support for the +profile_eapi_when_unspecified attribute. .RE .RE diff --git a/pym/portage/_sets/ProfilePackageSet.py b/pym/portage/_sets/ProfilePackageSet.py index c2f5fee..2fcafb6 100644 --- a/pym/portage/_sets/ProfilePackageSet.py +++ b/pym/portage/_se
[gentoo-portage-dev] [PATCH v2] bintree.py: fix str() calls for Python 2 (532784)
Avoid a UnicodeDecodeError raised when str(e) converts an exception
to bytes with Python 2. Since this file has unicode_literals enabled,
use literal unicode format strings to format messages for unicode
exceptions. However, with Python 2, an EnvironmentError exception
may contain either bytes or unicode, so use _unicode(errors="replace")
to ensure safety for EnvironmentError with all locales.
Also, convert remaining str() calls to use _unicode() for uniform
behavior regardless of python version.
X-Gentoo-Bug: 532784
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=532784
---
PATCH v2 replaces the _unicode_decode call with
_unicode(errors="replace"), which is required in order to force the
EnvironmentError object to be converted to a unicode string.
pym/portage/dbapi/bintree.py | 21 -
1 file changed, 12 insertions(+), 9 deletions(-)
diff --git a/pym/portage/dbapi/bintree.py b/pym/portage/dbapi/bintree.py
index d7c7f95..1156b66 100644
--- a/pym/portage/dbapi/bintree.py
+++ b/pym/portage/dbapi/bintree.py
@@ -391,7 +391,7 @@ class binarytree(object):
# sanity check
for atom in (origcp, newcp):
if not isjustname(atom):
- raise InvalidPackageName(str(atom))
+ raise InvalidPackageName(_unicode(atom))
mynewcat = catsplit(newcp)[0]
origmatches=self.dbapi.cp_list(origcp)
moves = 0
@@ -803,8 +803,8 @@ class binarytree(object):
d["CPV"] = mycpv
d["SLOT"] = slot
- d["MTIME"] = str(s[stat.ST_MTIME])
- d["SIZE"] = str(s.st_size)
+ d["MTIME"] = _unicode(s[stat.ST_MTIME])
+ d["SIZE"] = _unicode(s.st_size)
d.update(zip(self._pkgindex_aux_keys,
self.dbapi.aux_get(mycpv,
self._pkgindex_aux_keys)))
@@ -1024,7 +1024,11 @@ class binarytree(object):
except EnvironmentError as e:
writemsg(_("\n\n!!! Error fetching binhost
package" \
" info from '%s'\n") %
_hide_url_passwd(base_url))
- writemsg("!!! %s\n\n" % str(e))
+ # With Python 2, the EnvironmentError message
may
+ # contain bytes or unicode, so use _unicode to
ensure
+ # safety with all locales (bug #532784).
+ writemsg("!!! %s\n\n" % _unicode(e,
+ _encodings["stdio"], errors="replace"))
del e
pkgindex = None
if proc is not None:
@@ -1242,8 +1246,8 @@ class binarytree(object):
d["CPV"] = cpv
st = os.stat(pkg_path)
- d["MTIME"] = str(st[stat.ST_MTIME])
- d["SIZE"] = str(st.st_size)
+ d["MTIME"] = _unicode(st[stat.ST_MTIME])
+ d["SIZE"] = _unicode(st.st_size)
rel_path = self._pkg_paths[cpv]
# record location if it's non-default
@@ -1270,7 +1274,7 @@ class binarytree(object):
if profile_path.startswith(profiles_base):
profile_path = profile_path[len(profiles_base):]
header["PROFILE"] = profile_path
- header["VERSION"] = str(self._pkgindex_version)
+ header["VERSION"] = _unicode(self._pkgindex_version)
base_uri = self.settings.get("PORTAGE_BINHOST_HEADER_URI")
if base_uri:
header["URI"] = base_uri
@@ -1316,8 +1320,7 @@ class binarytree(object):
deps = use_reduce(deps, uselist=use,
token_class=token_class)
deps = paren_enclose(deps)
except portage.exception.InvalidDependString as e:
- writemsg("%s: %s\n" % (k, str(e)),
- noiselevel=-1)
+ writemsg("%s: %s\n" % (k, e), noiselevel=-1)
raise
metadata[k] = deps
--
2.0.4
[gentoo-portage-dev] [PATCH] bintree.py: fix str() calls for Python 2 (532784)
Avoid a UnicodeDecodeError raised when str(e) converts an exception
to bytes with Python 2. Since this file has unicode_literals enabled,
use literal unicode format strings to format messages for unicode
exceptions. However, with Python 2, an EnvironmentError exception
may contain either bytes or unicode, so use _unicode_decode to ensure
safety for EnvironmentError with all locales.
Also, convert remaining str() calls to use _unicode() for uniform
behavior regardless of python version.
X-Gentoo-Bug: 532784
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=532784
---
pym/portage/dbapi/bintree.py | 20 +++-
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git a/pym/portage/dbapi/bintree.py b/pym/portage/dbapi/bintree.py
index d7c7f95..b9098b2 100644
--- a/pym/portage/dbapi/bintree.py
+++ b/pym/portage/dbapi/bintree.py
@@ -391,7 +391,7 @@ class binarytree(object):
# sanity check
for atom in (origcp, newcp):
if not isjustname(atom):
- raise InvalidPackageName(str(atom))
+ raise InvalidPackageName(_unicode(atom))
mynewcat = catsplit(newcp)[0]
origmatches=self.dbapi.cp_list(origcp)
moves = 0
@@ -803,8 +803,8 @@ class binarytree(object):
d["CPV"] = mycpv
d["SLOT"] = slot
- d["MTIME"] = str(s[stat.ST_MTIME])
- d["SIZE"] = str(s.st_size)
+ d["MTIME"] = _unicode(s[stat.ST_MTIME])
+ d["SIZE"] = _unicode(s.st_size)
d.update(zip(self._pkgindex_aux_keys,
self.dbapi.aux_get(mycpv,
self._pkgindex_aux_keys)))
@@ -1024,7 +1024,10 @@ class binarytree(object):
except EnvironmentError as e:
writemsg(_("\n\n!!! Error fetching binhost
package" \
" info from '%s'\n") %
_hide_url_passwd(base_url))
- writemsg("!!! %s\n\n" % str(e))
+ # With Python 2, the exception message may
contain
+ # bytes or unicode, so use _unicode_decode to
ensure
+ # safety with all locales (bug #532784).
+ writemsg("!!! %s\n\n" % _unicode_decode(e))
del e
pkgindex = None
if proc is not None:
@@ -1242,8 +1245,8 @@ class binarytree(object):
d["CPV"] = cpv
st = os.stat(pkg_path)
- d["MTIME"] = str(st[stat.ST_MTIME])
- d["SIZE"] = str(st.st_size)
+ d["MTIME"] = _unicode(st[stat.ST_MTIME])
+ d["SIZE"] = _unicode(st.st_size)
rel_path = self._pkg_paths[cpv]
# record location if it's non-default
@@ -1270,7 +1273,7 @@ class binarytree(object):
if profile_path.startswith(profiles_base):
profile_path = profile_path[len(profiles_base):]
header["PROFILE"] = profile_path
- header["VERSION"] = str(self._pkgindex_version)
+ header["VERSION"] = _unicode(self._pkgindex_version)
base_uri = self.settings.get("PORTAGE_BINHOST_HEADER_URI")
if base_uri:
header["URI"] = base_uri
@@ -1316,8 +1319,7 @@ class binarytree(object):
deps = use_reduce(deps, uselist=use,
token_class=token_class)
deps = paren_enclose(deps)
except portage.exception.InvalidDependString as e:
- writemsg("%s: %s\n" % (k, str(e)),
- noiselevel=-1)
+ writemsg("%s: %s\n" % (k, e), noiselevel=-1)
raise
metadata[k] = deps
--
2.0.4
