Re: [gentoo-portage-dev] [PATCH 0/5] webrsync: support sync-openpgp-key-path (bug 661838)
On Fri, 27 Jul 2018 00:56:40 -0700 Zac Medico wrote: > Add repos.conf settings that enable sync-openpgp-key-path support for > both emerge-webrsync and emerge-delta-webrsync: > > sync-webrsync-delta = true|false > Use app-portage/emerge-delta-webrsync to minimize > bandwidth. Defaults to false. > > sync-webrsync-keep-snapshots = true|false > Keep snapshots in DISTDIR (do not delete). Defaults to false. > > sync-webrsync-verify-signature = true|false > Require the detached tarball signature to contain a good > OpenPGP signature. This uses the OpenPGP key(ring) specified by the > sync-openpgp-key-path setting. Defaults to false. > > Zac Medico (5): > emerge-webrsync: exit early for signature problem (bug 661838) > webrsync: support sync-openpgp-key-path (bug 661838) > webrsync: support sync-webrsync-keep-snapshots > emerge-delta-webrsync: exit early for signature problem (bug 661838) > webrsync: support emerge-delta-webrsync (bug 661838) > > bin/emerge-webrsync | 22 ++- > lib/portage/sync/modules/webrsync/__init__.py | 6 +- > lib/portage/sync/modules/webrsync/webrsync.py | 84 > --- > man/portage.5 | 12 > misc/emerge-delta-webrsync| 23 +++- 5 files > changed, 135 insertions(+), 12 deletions(-) > this series looks good. I'm also surprised to see nearly all the changes were on the python side. Barely any changes to the bash scripts.
[gentoo-portage-dev] [PATCH 3/5] webrsync: support sync-webrsync-keep-snapshots
Add a repos.conf sync-webrsync-keep-snapshots setting that enables
the emerge-webrsync --keep option, which keeps snapshots in DISTDIR
instead of deleting them.
---
lib/portage/sync/modules/webrsync/__init__.py | 1 +
man/portage.5 | 3 +++
2 files changed, 4 insertions(+)
diff --git a/lib/portage/sync/modules/webrsync/__init__.py
b/lib/portage/sync/modules/webrsync/__init__.py
index 1e09d1a47..118e752de 100644
--- a/lib/portage/sync/modules/webrsync/__init__.py
+++ b/lib/portage/sync/modules/webrsync/__init__.py
@@ -46,6 +46,7 @@ module_spec = {
},
'validate_config': CheckSyncConfig,
'module_specific_options': (
+ 'sync-webrsync-keep-snapshots',
'sync-webrsync-verify-signature',
),
},
diff --git a/man/portage.5 b/man/portage.5
index 8ebf980f5..4cb1b0b34 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -1128,6 +1128,9 @@ when 0. Defaults to disabled.
Require the repository to contain a signed MetaManifest and verify
it using \fBapp\-portage/gemato\fR. Defaults to no.
.TP
+.B sync\-webrsync\-keep\-snapshots = true|false
+Keep snapshots in \fBDISTDIR\fR (do not delete). Defaults to false.
+.TP
.B sync\-webrsync\-verify\-signature = true|false
Require the detached tarball signature to contain a good OpenPGP
signature. This uses the OpenPGP key(ring) specified by the
--
2.16.4
[gentoo-portage-dev] [PATCH 5/5] webrsync: support emerge-delta-webrsync (bug 661838)
Add a repos.conf sync-webrsync-delta setting that makes the webrsync
module call emerge-delta-webrsync, so that emerge-delta-webrsync users
can benefit from sync-openpgp-key-path support in the webrsync module.
Bug: https://bugs.gentoo.org/661838
---
lib/portage/sync/modules/webrsync/__init__.py | 1 +
lib/portage/sync/modules/webrsync/webrsync.py | 10 ++
man/portage.5 | 4
misc/emerge-delta-webrsync| 16 +++-
4 files changed, 30 insertions(+), 1 deletion(-)
diff --git a/lib/portage/sync/modules/webrsync/__init__.py
b/lib/portage/sync/modules/webrsync/__init__.py
index 118e752de..a413553a1 100644
--- a/lib/portage/sync/modules/webrsync/__init__.py
+++ b/lib/portage/sync/modules/webrsync/__init__.py
@@ -46,6 +46,7 @@ module_spec = {
},
'validate_config': CheckSyncConfig,
'module_specific_options': (
+ 'sync-webrsync-delta',
'sync-webrsync-keep-snapshots',
'sync-webrsync-verify-signature',
),
diff --git a/lib/portage/sync/modules/webrsync/webrsync.py
b/lib/portage/sync/modules/webrsync/webrsync.py
index 1b4c08e65..609ba0be2 100644
--- a/lib/portage/sync/modules/webrsync/webrsync.py
+++ b/lib/portage/sync/modules/webrsync/webrsync.py
@@ -34,6 +34,16 @@ class WebRsync(SyncBase):
def __init__(self):
SyncBase.__init__(self, 'emerge-webrsync',
'>=sys-apps/portage-2.3')
+ @property
+ def has_bin(self):
+ if (self._bin_command != 'emerge-delta-webrsync' and
+ self.repo.module_specific_options.get(
+ 'sync-webrsync-delta', 'false').lower() in ('true',
'yes')):
+ self._bin_command = 'emerge-delta-webrsync'
+ self.bin_command =
portage.process.find_binary(self._bin_command)
+ self.bin_pkg =
'>=app-portage/emerge-delta-webrsync-3.7.5'
+
+ return super(WebRsync, self).has_bin
def sync(self, **kwargs):
'''Sync the repository'''
diff --git a/man/portage.5 b/man/portage.5
index 4cb1b0b34..cd9d5036d 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -1128,6 +1128,10 @@ when 0. Defaults to disabled.
Require the repository to contain a signed MetaManifest and verify
it using \fBapp\-portage/gemato\fR. Defaults to no.
.TP
+.B sync\-webrsync\-delta = true|false
+Use \fBapp\-portage/emerge\-delta\-webrsync\fR to minimize bandwidth.
+Defaults to false.
+.TP
.B sync\-webrsync\-keep\-snapshots = true|false
Keep snapshots in \fBDISTDIR\fR (do not delete). Defaults to false.
.TP
diff --git a/misc/emerge-delta-webrsync b/misc/emerge-delta-webrsync
index ebaa616f9..5ade2708b 100755
--- a/misc/emerge-delta-webrsync
+++ b/misc/emerge-delta-webrsync
@@ -4,7 +4,15 @@
# Author: Brian Harring , kar...@gentoo.org originally.
# Rewritten from the old, Perl-based emerge-webrsync script
+# repos.conf configuration for use with emerge --sync and emaint sync
+# using keyring from app-crypt/openpgp-keys-gentoo-release:
+# [gentoo]
+# sync-type = webrsync
+# sync-webrsync-delta = true
+# sync-webrsync-verify-signature = true
+# sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc
#
+# Alternative (legacy) PORTAGE_GPG_DIR configuration:
# gpg key import
# KEY_ID=0x96D8BF6D
# gpg --homedir /etc/portage/gnupg --keyserver subkeys.pgp.net --recv-keys
$KEY_ID
@@ -106,7 +114,13 @@ if [[ ! -d $STATE_DIR ]]; then
exit -2
fi
-if has webrsync-gpg ${FEATURES} ; then
+if has $(__repo_attr "${repo_name}" sync-webrsync-verify-signature) true yes;
then
+ if [[ ! -d ${PORTAGE_GPG_DIR} ]]; then
+ eecho "Do not call ${argv0##*/} directly, instead call emerge
--sync or emaint sync."
+ exit 1
+ fi
+ WEBSYNC_VERIFY_SIGNATURE=1
+elif has webrsync-gpg ${FEATURES}; then
WEBSYNC_VERIFY_SIGNATURE=1
else
WEBSYNC_VERIFY_SIGNATURE=0
--
2.16.4
[gentoo-portage-dev] [PATCH 4/5] emerge-delta-webrsync: exit early for signature problem (bug 661838)
Exit early after signature verification failure, since it's
typically inappropriate to try other mirrors in this case
(it may indicate a keyring problem).
Bug: https://bugs.gentoo.org/661838
---
misc/emerge-delta-webrsync | 7 +++
1 file changed, 7 insertions(+)
diff --git a/misc/emerge-delta-webrsync b/misc/emerge-delta-webrsync
index 868c6a347..ebaa616f9 100755
--- a/misc/emerge-delta-webrsync
+++ b/misc/emerge-delta-webrsync
@@ -283,6 +283,13 @@ check_file_signature() {
fi
done <<< "${gnupg_status}"
fi
+ if [[ ${r} -ne 0 ]]; then
+ # Exit early since it's typically inappropriate
to
+ # try other mirrors in this case (it may
indicate
+ # a keyring problem).
+ eecho "signature verification failed"
+ exit 1
+ fi
else
eecho "cannot check signature: gpg binary not found"
exit 1
--
2.16.4
[gentoo-portage-dev] [PATCH 2/5] webrsync: support sync-openpgp-key-path (bug 661838)
Add repos.conf sync-webrsync-verify-signature = true|false setting that
enables sync-openpgp-key-path support like in the rsync and git sync
modules. This is disabled by default, in order to avoid interference
with legacy manual PORTAGE_GPG_DIR configuration.
When sync-webrsync-verify-signature = true is set in repos.conf,
if the PORTAGE_GPG_DIR has not been exported, emerge-webrsync will
assume that it has been called directly and it will output an error
message advising the user to instead call emerge --sync or emaint sync.
Bug: https://bugs.gentoo.org/661838
---
bin/emerge-webrsync | 15 +-
lib/portage/sync/modules/webrsync/__init__.py | 4 +-
lib/portage/sync/modules/webrsync/webrsync.py | 74 +++
man/portage.5 | 5 ++
4 files changed, 87 insertions(+), 11 deletions(-)
diff --git a/bin/emerge-webrsync b/bin/emerge-webrsync
index b135567b7..841f01614 100755
--- a/bin/emerge-webrsync
+++ b/bin/emerge-webrsync
@@ -10,7 +10,14 @@
# - all output should prob be converted to e* funcs
# - add support for ROOT
+# repos.conf configuration for use with emerge --sync and emaint sync
+# using keyring from app-crypt/openpgp-keys-gentoo-release:
+# [gentoo]
+# sync-type = webrsync
+# sync-webrsync-verify-signature = true
+# sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc
#
+# Alternative (legacy) PORTAGE_GPG_DIR configuration:
# gpg key import
# KEY_ID=0x96D8BF6D
# gpg --homedir /etc/portage/gnupg --keyserver subkeys.pgp.net --recv-keys
$KEY_ID
@@ -67,7 +74,13 @@ do_verbose=0
do_debug=0
keep=false
-if has webrsync-gpg ${FEATURES} ; then
+if has $(__repo_attr "${repo_name}" sync-webrsync-verify-signature) true yes;
then
+ if [[ ! -d ${PORTAGE_GPG_DIR} ]]; then
+ eecho "Do not call ${argv0##*/} directly, instead call emerge
--sync or emaint sync."
+ exit 1
+ fi
+ WEBSYNC_VERIFY_SIGNATURE=1
+elif has webrsync-gpg ${FEATURES}; then
WEBSYNC_VERIFY_SIGNATURE=1
else
WEBSYNC_VERIFY_SIGNATURE=0
diff --git a/lib/portage/sync/modules/webrsync/__init__.py
b/lib/portage/sync/modules/webrsync/__init__.py
index dc7def20c..1e09d1a47 100644
--- a/lib/portage/sync/modules/webrsync/__init__.py
+++ b/lib/portage/sync/modules/webrsync/__init__.py
@@ -45,7 +45,9 @@ module_spec = {
'exists and is a valid repository',
},
'validate_config': CheckSyncConfig,
- 'module_specific_options': (),
+ 'module_specific_options': (
+ 'sync-webrsync-verify-signature',
+ ),
},
}
}
diff --git a/lib/portage/sync/modules/webrsync/webrsync.py
b/lib/portage/sync/modules/webrsync/webrsync.py
index 3d79f4557..1b4c08e65 100644
--- a/lib/portage/sync/modules/webrsync/webrsync.py
+++ b/lib/portage/sync/modules/webrsync/webrsync.py
@@ -1,17 +1,25 @@
'''WebRsync module for portage'''
+import io
import logging
import portage
from portage import os
from portage.util import writemsg_level
+from portage.util.futures import asyncio
from portage.output import create_color_func
good = create_color_func("GOOD")
bad = create_color_func("BAD")
warn = create_color_func("WARN")
from portage.sync.syncbase import SyncBase
+try:
+ from gemato.exceptions import GematoException
+ import gemato.openpgp
+except ImportError:
+ gemato = None
+
class WebRsync(SyncBase):
'''WebRSync sync class'''
@@ -39,15 +47,63 @@ class WebRsync(SyncBase):
for var in ['uid', 'gid', 'groups']:
self.spawn_kwargs.pop(var, None)
- exitcode = portage.process.spawn_bash("%s" % \
- (self.bin_command),
- **self.spawn_kwargs)
- if exitcode != os.EX_OK:
- msg = "!!! emerge-webrsync error in %s" %
self.repo.location
- self.logger(self.xterm_titles, msg)
- writemsg_level(msg + "\n", level=logging.ERROR,
noiselevel=-1)
- return (exitcode, False)
- return (exitcode, True)
+ verbose = '--verbose' in self.options['emerge_config'].opts
+ quiet = '--quiet' in self.options['emerge_config'].opts
+ openpgp_env = None
+ try:
+ if self.repo.module_specific_options.get(
+ 'sync-webrsync-verify-signature',
'false').lower() in ('true', 'yes'):
+
+ if not self.repo.sync_openpgp_key_path:
+ writemsg_level("!!!
sync-openpgp-key-path is not set\n",
+ level=logging.ERROR,
noiselevel=-1)
+ return (1, False)
+
+
[gentoo-portage-dev] [PATCH 1/5] emerge-webrsync: exit early for signature problem (bug 661838)
Exit early after signature verification failure, since it's
typically inappropriate to try other mirrors in this case
(it may indicate a keyring problem).
Bug: https://bugs.gentoo.org/661838
---
bin/emerge-webrsync | 7 +++
1 file changed, 7 insertions(+)
diff --git a/bin/emerge-webrsync b/bin/emerge-webrsync
index 560dd0236..b135567b7 100755
--- a/bin/emerge-webrsync
+++ b/bin/emerge-webrsync
@@ -191,6 +191,13 @@ check_file_signature() {
fi
done <<< "${gnupg_status}"
fi
+ if [[ ${r} -ne 0 ]]; then
+ # Exit early since it's typically inappropriate
to
+ # try other mirrors in this case (it may
indicate
+ # a keyring problem).
+ eecho "signature verification failed"
+ exit 1
+ fi
else
eecho "cannot check signature: gpg binary not found"
exit 1
--
2.16.4
[gentoo-portage-dev] [PATCH 0/5] webrsync: support sync-openpgp-key-path (bug 661838)
Add repos.conf settings that enable sync-openpgp-key-path support for both emerge-webrsync and emerge-delta-webrsync: sync-webrsync-delta = true|false Use app-portage/emerge-delta-webrsync to minimize bandwidth. Defaults to false. sync-webrsync-keep-snapshots = true|false Keep snapshots in DISTDIR (do not delete). Defaults to false. sync-webrsync-verify-signature = true|false Require the detached tarball signature to contain a good OpenPGP signature. This uses the OpenPGP key(ring) specified by the sync-openpgp-key-path setting. Defaults to false. Zac Medico (5): emerge-webrsync: exit early for signature problem (bug 661838) webrsync: support sync-openpgp-key-path (bug 661838) webrsync: support sync-webrsync-keep-snapshots emerge-delta-webrsync: exit early for signature problem (bug 661838) webrsync: support emerge-delta-webrsync (bug 661838) bin/emerge-webrsync | 22 ++- lib/portage/sync/modules/webrsync/__init__.py | 6 +- lib/portage/sync/modules/webrsync/webrsync.py | 84 --- man/portage.5 | 12 misc/emerge-delta-webrsync| 23 +++- 5 files changed, 135 insertions(+), 12 deletions(-) -- 2.16.4
