Re: [gentoo-portage-dev] [PATCH 2/2] Update /usr/portage references (bug 378603)
On Sun, 5 Aug 2018 22:46:58 -0700
Zac Medico wrote:
> On 08/05/2018 09:59 PM, Ulrich Mueller wrote:
> >> On Sun, 5 Aug 2018, Zac Medico wrote:
> >
> >> --- a/cnf/make.conf.example
> >> +++ b/cnf/make.conf.example
> >> [...]
> >
> >> @@ -119,16 +119,16 @@
> >> # fetched on demand for a given build. If you would like to
> >> # selectively prune obsolete files from this directory, see
> >> # eclean from the gentoolkit package. Note that locations
> >> under -# /usr/portage are not necessarily safe for data
> >> storage. See the +# /var/db/repos/gentoo are not necessarily
> >> safe for data storage. See the # PORTDIR documentation for
> >> more information. -#DISTDIR=/usr/portage/distfiles
> >> +#DISTDIR=/var/db/repos/gentoo/distfiles
> >
> > Shouldn't this be /var/cache/distfiles ...
> >
> >> #
> >> # PKGDIR is the location of binary packages that you can have
> >> created # with '--buildpkg' or '-b' while emerging a package.
> >> This can get # up to several hundred megs, or even a few gigs.
> >> Note that -# locations under /usr/portage are not necessarily
> >> safe for data +# locations under /var/db/repos/gentoo are not
> >> necessarily safe for data # storage. See the PORTDIR
> >> documentation for more information. -#PKGDIR=/usr/portage/packages
> >> +#PKGDIR=/var/db/repos/gentoo/packages
> >
> > ... and /var/cache/binpkgs?
>
> Thanks, I've fixed the /var/cache/{distfiles,binpkgs} locations in v2.
>
> >> --- a/lib/portage/cache/flat_hash.py
> >> +++ b/lib/portage/cache/flat_hash.py
> >> @@ -144,7 +144,7 @@ class database(fs_template.FsBased):
> >># Only recurse 1 deep, in
> >> order to avoid iteration over # entries from another nested cache
> >> instance. This can # happen if the user nests an overlay inside
> >> - # /usr/portage/local as
> >> in bug #302764.
> >> +
> >> # /var/db/repos/gentoo/local as in bug #302764.
> >
> > Shouldn't a local overlay be in /var/db/repos/local, but never
> > inside of the gentoo repo?
>
> Yes, but for a long time /usr/local/portage was somewhat standard, in
> fact it's still mentioned here:
>
> https://wiki.gentoo.org/wiki/Custom_repository
>
> Nowadays, repository verification will prevent that from working...
>
But that too can be changed along with all the user install
documentation which will need to be updated as well. The new
recomended location should be /var/db/repos/local. I will be updating
layman for /var/db/repos/... as well. That is the intention of the
"repos/" subdir.
All these changes as well as the catlayst changes need to be
co-ordinated so that snapshots and portage and stages don't precede the
docs changes.
> >> --- a/man/make.conf.5
> >> +++ b/man/make.conf.5
> >> @@ -219,10 +219,10 @@ Use the \fBPORTAGE_RO_DISTDIRS\fR variable
> >> to specify one or more read-only directories containing distfiles.
> >>
> >> Note
> >> -that locations under /usr/portage are not necessarily safe for
> >> data storage. +that locations under /var/db/repos/gentoo are not
> >> necessarily safe for data storage.
> >
> > IMHO this statement should be made much stronger.
>
> Yes, we could also say something about repository verification here.
> The old defaults set a really bad example, so we needed statements
> like this to steer people in the right direction.
> --
> Thanks,
> Zac
>
pgpT1yfKTqhPZ.pgp
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH 2/2] Update /usr/portage references (bug 378603)
On 08/05/2018 10:46 PM, Zac Medico wrote: >>> --- a/lib/portage/cache/flat_hash.py >>> +++ b/lib/portage/cache/flat_hash.py >>> @@ -144,7 +144,7 @@ class database(fs_template.FsBased): >>> # Only recurse 1 deep, in order to >>> avoid iteration over >>> # entries from another nested cache >>> instance. This can >>> # happen if the user nests an overlay >>> inside >>> - # /usr/portage/local as in bug #302764. >>> + # /var/db/repos/gentoo/local as in bug >>> #302764. >> >> Shouldn't a local overlay be in /var/db/repos/local, but never inside >> of the gentoo repo? > > Yes, but for a long time /usr/local/portage was somewhat standard, in > fact it's still mentioned here: > > https://wiki.gentoo.org/wiki/Custom_repository > > Nowadays, repository verification will prevent that from working... Actually the wiki has /usr/local/portage, but the default rsync options have has --exclude=/local for a very long time. It might be safe to remove it, but I guess it's probably safest to trigger a warning message first, since some people might still be using it with rsync-verify disabled. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH 3/4] rsync: split out repo storage framework
Since there aremany ways to manage repository storage, split out a repo
storage framework. The HardlinkQuarantineRepoStorage class implements
the existing default behavior, and the InplaceRepoStorage class
implements the legacy behavior (when sync-allow-hardlinks is disabled in
repos.conf).
Each class implements RepoStorageInterface, which uses coroutine methods
since coroutines are well-suited to the I/O bound tasks that these
methods perform. The _sync_decorator is used to convert coroutine
methods to synchronous methods, for smooth integration into the
surrounding synchronous code.
Bug: https://bugs.gentoo.org/662070
---
lib/portage/repository/storage/__init__.py | 0
.../repository/storage/hardlink_quarantine.py | 95 ++
lib/portage/repository/storage/inplace.py | 49 +++
lib/portage/repository/storage/interface.py| 87
lib/portage/sync/controller.py | 1 +
lib/portage/sync/modules/rsync/rsync.py| 85 +--
lib/portage/sync/syncbase.py | 31 +++
7 files changed, 284 insertions(+), 64 deletions(-)
create mode 100644 lib/portage/repository/storage/__init__.py
create mode 100644 lib/portage/repository/storage/hardlink_quarantine.py
create mode 100644 lib/portage/repository/storage/inplace.py
create mode 100644 lib/portage/repository/storage/interface.py
diff --git a/lib/portage/repository/storage/__init__.py
b/lib/portage/repository/storage/__init__.py
new file mode 100644
index 0..e69de29bb
diff --git a/lib/portage/repository/storage/hardlink_quarantine.py
b/lib/portage/repository/storage/hardlink_quarantine.py
new file mode 100644
index 0..7e9cf4493
--- /dev/null
+++ b/lib/portage/repository/storage/hardlink_quarantine.py
@@ -0,0 +1,95 @@
+# Copyright 2018 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+from portage import os
+from portage.repository.storage.interface import (
+ RepoStorageException,
+ RepoStorageInterface,
+)
+from portage.util.futures import asyncio
+from portage.util.futures.compat_coroutine import (
+ coroutine,
+ coroutine_return,
+)
+
+from _emerge.SpawnProcess import SpawnProcess
+
+
+class HardlinkQuarantineRepoStorage(RepoStorageInterface):
+ """
+ This is the default storage module, since its quite compatible with
+ most configurations.
+
+ It's desirable to be able to create shared hardlinks between the
+ download directory and the normal repository, and this is facilitated
+ by making the download directory be a subdirectory of the normal
+ repository location (ensuring that no mountpoints are crossed).
+ Shared hardlinks are created by using the rsync --link-dest option.
+
+ Since the download is initially unverified, it is safest to save
+ it in a quarantine directory. The quarantine directory is also
+ useful for making the repository update more atomic, so that it
+ less likely that normal repository location will be observed in
+ a partially synced state.
+ """
+ def __init__(self, repo, spawn_kwargs):
+ self._user_location = repo.location
+ self._update_location = None
+ self._spawn_kwargs = spawn_kwargs
+ self._current_update = None
+
+ @coroutine
+ def _check_call(self, cmd):
+ """
+ Run cmd and raise RepoStorageException on failure.
+
+ @param cmd: command to executre
+ @type cmd: list
+ """
+ p = SpawnProcess(args=cmd, scheduler=asyncio._wrap_loop(),
**self._spawn_kwargs)
+ p.start()
+ if (yield p.async_wait()) != os.EX_OK:
+ raise RepoStorageException('command exited with status
{}: {}'.\
+ format(p.returncode, ' '.join(cmd)))
+
+ @coroutine
+ def init_update(self):
+ update_location = os.path.join(self._user_location,
'.tmp-unverified-download-quarantine')
+ yield self._check_call(['rm', '-rf', update_location])
+
+ # Use rsync --link-dest to hardlink a files into
self._update_location,
+ # since cp -l is not portable.
+ yield self._check_call(['rsync', '-a', '--link-dest',
self._user_location,
+ '--exclude',
'/{}'.format(os.path.basename(update_location)),
+ self._user_location + '/', update_location + '/'])
+
+ self._update_location = update_location
+
+ coroutine_return(self._update_location)
+
+ @property
+ def current_update(self):
+ if self._update_location is None:
+ raise RepoStorageException('current update does not
exist')
+ return self._update_location
+
+ @coroutine
+
[gentoo-portage-dev] [PATCH 4/4] Add sync-rcu support for rsync (bug 662070)
Add a boolean sync-rcu repos.conf setting that behaves as follows:
Enable read-copy-update (RCU) behavior for sync operations. The
current latest immutable version of a repository will be referenced
by a symlink found where the repository would normally be located
(see the location setting). Repository consumers should resolve
the cannonical path of this symlink before attempt to access
the repository, and all operations should be read-only, since
the repository is considered immutable. Updates occur by atomic
replacement of the symlink, which causes new consumers to use the
new immutable version, while any earlier consumers continue to
use the cannonical path that was resolved earlier. This option
requires sync-allow-hardlinks and sync-rcu-store-dir options to
be enabled, and currently also requires that sync-type is set
to rsync. This option is disabled by default, since the symlink
usage would require special handling for scenarios involving bind
mounts and chroots.
Bug: https://bugs.gentoo.org/662070
---
lib/portage/repository/config.py | 36 +++-
lib/portage/repository/storage/hardlink_rcu.py | 251 +
lib/portage/sync/syncbase.py | 4 +-
man/portage.5 | 35
4 files changed, 323 insertions(+), 3 deletions(-)
create mode 100644 lib/portage/repository/storage/hardlink_rcu.py
diff --git a/lib/portage/repository/config.py b/lib/portage/repository/config.py
index f790f9392..8cdc2a696 100644
--- a/lib/portage/repository/config.py
+++ b/lib/portage/repository/config.py
@@ -84,7 +84,7 @@ class RepoConfig(object):
'profile_formats', 'sign_commit', 'sign_manifest',
'strict_misc_digests',
'sync_depth', 'sync_hooks_only_on_change',
'sync_type', 'sync_umask', 'sync_uri', 'sync_user',
'thin_manifest',
- 'update_changelog', '_eapis_banned', '_eapis_deprecated',
+ 'update_changelog', 'user_location', '_eapis_banned',
'_eapis_deprecated',
'_masters_orig', 'module_specific_options',
'manifest_required_hashes',
'sync_allow_hardlinks',
'sync_openpgp_key_path',
@@ -93,6 +93,10 @@ class RepoConfig(object):
'sync_openpgp_key_refresh_retry_delay_exp_base',
'sync_openpgp_key_refresh_retry_delay_mult',
'sync_openpgp_key_refresh_retry_overall_timeout',
+ 'sync_rcu',
+ 'sync_rcu_store_dir',
+ 'sync_rcu_spare_snapshots',
+ 'sync_rcu_ttl_days',
)
def __init__(self, name, repo_opts, local_config=True):
@@ -198,6 +202,22 @@ class RepoConfig(object):
'sync_openpgp_key_refresh_retry_overall_timeout'):
setattr(self, k, repo_opts.get(k.replace('_', '-'),
None))
+ self.sync_rcu = repo_opts.get(
+ 'sync-rcu', 'false').lower() in ('true', 'yes')
+
+ self.sync_rcu_store_dir = repo_opts.get('sync-rcu-store-dir')
+
+ for k in ('sync-rcu-spare-snapshots', 'sync-rcu-ttl-days'):
+ v = repo_opts.get(k, '').strip() or None
+ if v:
+ try:
+ v = int(v)
+ except (OverflowError, ValueError):
+ writemsg(_("!!! Invalid %s setting for
repo"
+ " %s: %s\n") % (k, name, v),
noiselevel=-1)
+ v = None
+ setattr(self, k.replace('-', '_'), v)
+
self.module_specific_options = {}
# Not implemented.
@@ -206,9 +226,14 @@ class RepoConfig(object):
format = format.strip()
self.format = format
+ self.user_location = None
location = repo_opts.get('location')
if location is not None and location.strip():
if os.path.isdir(location) or portage._sync_mode:
+ # The user_location is required for sync-rcu
support,
+ # since it manages a symlink which resides at
that
+ # location (and realpath is irreversible).
+ self.user_location = location
location = os.path.realpath(location)
else:
location = None
@@ -542,6 +567,10 @@ class RepoConfigLoader(object):
'sync_openpgp_key_refresh_retry_delay_exp_base',
'sync_openpgp_key_refresh_retry_delay_mult',
[gentoo-portage-dev] [PATCH 1/4] Implement asyncio.iscoroutinefunction for compat_coroutine
Sometimes it's useful to test if a function is a coroutine function, so implement a version of asyncio.iscoroutinefunction that works with asyncio.coroutine as well as compat_coroutine.coroutine (since both kinds of coroutine functions behave identically for our purposes). --- lib/portage/util/futures/_asyncio/__init__.py | 14 ++ lib/portage/util/futures/compat_coroutine.py | 12 2 files changed, 26 insertions(+) diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py index faab98e47..2a637624d 100644 --- a/lib/portage/util/futures/_asyncio/__init__.py +++ b/lib/portage/util/futures/_asyncio/__init__.py @@ -36,6 +36,7 @@ except ImportError: import portage portage.proxy.lazyimport.lazyimport(globals(), 'portage.util.futures.unix_events:_PortageEventLoopPolicy', + 'portage.util.futures:compat_coroutine@_compat_coroutine', ) from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop as _AsyncioEventLoop from portage.util._eventloop.global_event_loop import ( @@ -152,6 +153,19 @@ def create_subprocess_exec(*args, **kwargs): return result +def iscoroutinefunction(func): + """ + Return True if func is a decorated coroutine function, + supporting both asyncio.coroutine and compat_coroutine since + their behavior is identical for all practical purposes. + """ + if _compat_coroutine._iscoroutinefunction(func): + return True + elif _real_asyncio is not None and _real_asyncio.iscoroutinefunction(func): + return True + return False + + class Task(Future): """ Schedule the execution of a coroutine: wrap it in a future. A task diff --git a/lib/portage/util/futures/compat_coroutine.py b/lib/portage/util/futures/compat_coroutine.py index 59fdc31b6..be305c1b5 100644 --- a/lib/portage/util/futures/compat_coroutine.py +++ b/lib/portage/util/futures/compat_coroutine.py @@ -8,6 +8,17 @@ portage.proxy.lazyimport.lazyimport(globals(), 'portage.util.futures:asyncio', ) +# A marker for iscoroutinefunction. +_is_coroutine = object() + + +def _iscoroutinefunction(func): + """ + Return True if func is a decorated coroutine function + created with the coroutine decorator for this module. + """ + return getattr(func, '_is_coroutine', None) is _is_coroutine + def coroutine(generator_func): """ @@ -34,6 +45,7 @@ def coroutine(generator_func): @functools.wraps(generator_func) def wrapped(*args, **kwargs): return _generator_future(generator_func, *args, **kwargs) + wrapped._is_coroutine = _is_coroutine return wrapped -- 2.16.4
Re: [gentoo-portage-dev] [PATCH 2/2 v2] Update /usr/portage references (bug 378603)
On Sun, 5 Aug 2018 22:32:34 -0700
Zac Medico wrote:
> Update all relevant references in docs, messages, and comments
> to refer to /var/db/repos/gentoo instead of /usr/portage. Also
> update DISTDIR and PKGDIR references to refer to the new
> /var/cache/{distfiles,binpkgs} locations.
>
> Bug: https://bugs.gentoo.org/378603
> ---
> [PATCH 2/2 v2] fixes DISTDIR and PKGDIR references to rever to the
> new /var/cache/{distfiles,binpkgs} locations
>
> cnf/make.conf.example| 12 +--
> lib/portage/__init__.py | 2 +-
> lib/portage/cache/flat_hash.py | 2 +-
> lib/portage/tests/news/test_NewsItem.py | 2 +-
> lib/portage/tests/resolver/ResolverPlayground.py | 2 +-
> lib/portage/xml/metadata.py | 4 ++--
> man/ebuild.5 | 4 ++--
> man/emerge.1 | 6 +++---
> man/make.conf.5 | 20
> +- man/portage.5|
> 26
> man/quickpkg.1 | 2 +-
> repoman/lib/repoman/__init__.py | 2 +-
> repoman/lib/repoman/checks/herds/herdbase.py | 2 +- 13 files
> changed, 43 insertions(+), 43 deletions(-)
>
> diff --git a/cnf/make.conf.example b/cnf/make.conf.example
> index 04f3a0274..c16f2afbd 100644
> --- a/cnf/make.conf.example
> +++ b/cnf/make.conf.example
> @@ -14,7 +14,7 @@
> # https://wiki.gentoo.org/wiki/Handbook:X86/Working/USE
> #
> # The available list of use flags with descriptions is in your
> portage tree. -# Use 'less' to view them: -->
> less /usr/portage/profiles/use.desc <-- +# Use 'less' to view them:
> --> less /var/db/repos/gentoo/profiles/use.desc <-- #
> # 'ufed' is an ncurses/dialog interface available in portage to make
> handling # useflags for you. 'emerge app-portage/ufed'
> @@ -111,7 +111,7 @@
> # will protect the default locations of DISTDIR and PKGDIR, but
> users are # warned that any other locations inside PORTDIR are
> not necessarily safe # for data storage.
> -#PORTDIR=/usr/portage
> +#PORTDIR=/var/db/repos/gentoo
> #
> # DISTDIR is where all of the source code tarballs will be placed for
> # emerges. After packages are built, it is safe to remove any and
> @@ -119,16 +119,16 @@
> # fetched on demand for a given build. If you would like to
> # selectively prune obsolete files from this directory, see
> # eclean from the gentoolkit package. Note that locations under
> -# /usr/portage are not necessarily safe for data storage. See the
> +# /var/db/repos/gentoo are not necessarily safe for data
> storage. See the # PORTDIR documentation for more information.
> -#DISTDIR=/usr/portage/distfiles
> +#DISTDIR=/var/cache/distfiles
> #
> # PKGDIR is the location of binary packages that you can have created
> # with '--buildpkg' or '-b' while emerging a package. This can
> get # up to several hundred megs, or even a few gigs. Note that
> -# locations under /usr/portage are not necessarily safe for data
> +# locations under /var/db/repos/gentoo are not necessarily safe
> for data # storage. See the PORTDIR documentation for more
> information. -#PKGDIR=/usr/portage/packages
> +#PKGDIR=/var/cache/binpkgs
> #
> # PORT_LOGDIR is the location where portage will store all the logs
> it # creates from each individual merge. They are stored as
> diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py
> index 166bfc700..61a240100 100644
> --- a/lib/portage/__init__.py
> +++ b/lib/portage/__init__.py
> @@ -133,7 +133,7 @@ except ImportError as e:
> sys.stderr.write("!!! Failed to complete portage imports.
> There are internal modules for\n") sys.stderr.write("!!! portage and
> failure here indicates that you have a problem with your\n")
> sys.stderr.write("!!! installation of portage. Please try a rescue
> portage located in the\n")
> - sys.stderr.write("!!! portage tree under
> '/usr/portage/sys-apps/portage/files/' (default).\n")
> + sys.stderr.write("!!! portage tree under
> '/var/db/repos/gentoo/sys-apps/portage/files/' (default).\n")
This should also get updated replacing "portage tree" with "gentoo
tree"
...
> The packages, after being created, will be placed in \fBPKGDIR\fR.
> This variable is defined in \fBmake.conf\fR(5) and defaults to
> -/usr/portage/packages.
> +/var/cache/binpkgs.
> .SH OPTIONS
> .TP
> .B
> diff --git a/repoman/lib/repoman/__init__.py
> b/repoman/lib/repoman/__init__.py index 89779b95c..d1312a267 100644
> --- a/repoman/lib/repoman/__init__.py
> +++ b/repoman/lib/repoman/__init__.py
> @@ -14,7 +14,7 @@ except ImportError as e:
> sys.stderr.write("!!! Failed to complete portage imports.
> There are internal modules for\n") sys.stderr.write("!!! portage and
> failure here indicates that you have a problem with your\n")
>
[gentoo-portage-dev] [PATCH 2/4] Add _sync_decorator module
Add functions that decorate coroutine methods and functions for
synchronous usage, allowing coroutines to smoothly blend with
synchronous code. This eliminates clutter that might otherwise
discourage the proliferation of coroutine usage for I/O bound tasks.
In the next commit, _sync_decorator will be used for smooth
integration of new classes that have coroutine methods.
Bug: https://bugs.gentoo.org/662070
---
.../tests/util/futures/test_compat_coroutine.py| 14 ++
lib/portage/util/futures/_sync_decorator.py| 54 ++
2 files changed, 68 insertions(+)
create mode 100644 lib/portage/util/futures/_sync_decorator.py
diff --git a/lib/portage/tests/util/futures/test_compat_coroutine.py
b/lib/portage/tests/util/futures/test_compat_coroutine.py
index cbc070869..2b5ae91cd 100644
--- a/lib/portage/tests/util/futures/test_compat_coroutine.py
+++ b/lib/portage/tests/util/futures/test_compat_coroutine.py
@@ -6,6 +6,7 @@ from portage.util.futures.compat_coroutine import (
coroutine,
coroutine_return,
)
+from portage.util.futures._sync_decorator import _sync_decorator, _sync_methods
from portage.tests import TestCase
@@ -157,3 +158,16 @@ class CompatCoroutineTestCase(TestCase):
loop.run_until_complete(asyncio.wait([writer, reader]))
self.assertEqual(reader.result(), values)
+
+ # Test decoration of coroutine methods and functions for
+ # synchronous usage, allowing coroutines to smoothly
+ # blend with synchronous code.
+ sync_cubby = _sync_methods(cubby, loop=loop)
+ sync_reader = _sync_decorator(reader_coroutine, loop=loop)
+ writer = asyncio.ensure_future(writer_coroutine(cubby, values,
None), loop=loop)
+ self.assertEqual(sync_reader(cubby, None), values)
+ self.assertTrue(writer.done())
+
+ for i in range(3):
+ sync_cubby.write(i)
+ self.assertEqual(sync_cubby.read(), i)
diff --git a/lib/portage/util/futures/_sync_decorator.py
b/lib/portage/util/futures/_sync_decorator.py
new file mode 100644
index 0..02a0963a7
--- /dev/null
+++ b/lib/portage/util/futures/_sync_decorator.py
@@ -0,0 +1,54 @@
+# Copyright 2018 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import functools
+
+import portage
+portage.proxy.lazyimport.lazyimport(globals(),
+ 'portage.util.futures:asyncio',
+)
+
+
+def _sync_decorator(func, loop=None):
+ """
+ Decorate an asynchronous function (either a corouting function or a
+ function that returns a Future) with a wrapper that runs the function
+ synchronously.
+ """
+ loop = asyncio._wrap_loop(loop)
+ @functools.wraps(func)
+ def wrapper(*args, **kwargs):
+ return loop.run_until_complete(func(*args, **kwargs))
+ return wrapper
+
+
+def _sync_methods(obj, loop=None):
+ """
+ For use with synchronous code that needs to interact with an object
+ that has coroutine methods, this function generates a proxy which
+ conveniently converts coroutine methods into synchronous methods.
+ This allows coroutines to smoothly blend with synchronous
+ code, eliminating clutter that might otherwise discourage the
+ proliferation of coroutine usage for I/O bound tasks.
+ """
+ loop = asyncio._wrap_loop(loop)
+ return _ObjectAttrWrapper(obj,
+ lambda attr: _sync_decorator(attr, loop=loop)
+ if asyncio.iscoroutinefunction(attr) else attr)
+
+
+class _ObjectAttrWrapper(portage.proxy.objectproxy.ObjectProxy):
+
+ __slots__ = ('_obj', '_attr_wrapper')
+
+ def __init__(self, obj, attr_wrapper):
+ object.__setattr__(self, '_obj', obj)
+ object.__setattr__(self, '_attr_wrapper', attr_wrapper)
+
+ def __getattribute__(self, attr):
+ obj = object.__getattribute__(self, '_obj')
+ attr_wrapper = object.__getattribute__(self, '_attr_wrapper')
+ return attr_wrapper(getattr(obj, attr))
+
+ def _get_target(self):
+ return object.__getattribute__(self, '_obj')
--
2.16.4
[gentoo-portage-dev] [PATCH 0/4] Add sync-rcu support for rsync (bug 662070)
Add a boolean sync-rcu repos.conf setting that behaves as follows: sync-rcu = yes|no Enable read-copy-update (RCU) behavior for sync operations. The current latest immutable version of a repository will be referenced by a symlink found where the repository would normally be located (see the location setting). Repository consumers should resolve the cannonical path of this symlink before attempt to access the repository, and all operations should be read-only, since the repository is considered immutable. Updates occur by atomic replacement of the symlink, which causes new consumers to use the new immutable version, while any earlier consumers continue to use the cannonical path that was resolved earlier. This option requires sync-allow-hardlinks and sync-rcu-store-dir options to be enabled, and currently also requires that sync-type is set to rsync. This option is disabled by default, since the symlink usage would require special handling for scenarios involving bind mounts and chroots. sync-rcu-store-dir Directory path reserved for sync-rcu storage. This directory must have a unique value for each repository (do not set it in the DEFAULT section). This directory must not contain any other files or directories aside from those that are created automatically when sync-rcu is enabled. sync-rcu-spare-snapshots = 1 Number of spare snapshots for sync-rcu to retain with expired ttl. This protects the previous latest snapshot from being removed immediately after a new version becomes available, since it might still be used by running processes. sync-rcu-ttl-days = 7 Number of days for sync-rcu to retain previous immutable snapshots of a repository. After the ttl of a particular snapshot has expired, it will be remove automatically (the latest snapshot is exempt, and sync-rcu-spare-snapshots configures the number of previous snapshots that are exempt). If the ttl is set too low, then a snapshot could expire while it is in use by a running process. Zac Medico (4): Implement asyncio.iscoroutinefunction for compat_coroutine Add _sync_decorator module rsync: split out repo storage framework Add sync-rcu support for rsync (bug 662070) lib/portage/repository/config.py | 36 ++- lib/portage/repository/storage/__init__.py | 0 .../repository/storage/hardlink_quarantine.py | 95 lib/portage/repository/storage/hardlink_rcu.py | 251 + lib/portage/repository/storage/inplace.py | 49 lib/portage/repository/storage/interface.py| 87 +++ lib/portage/sync/controller.py | 1 + lib/portage/sync/modules/rsync/rsync.py| 85 ++- lib/portage/sync/syncbase.py | 33 +++ .../tests/util/futures/test_compat_coroutine.py| 14 ++ lib/portage/util/futures/_asyncio/__init__.py | 14 ++ lib/portage/util/futures/_sync_decorator.py| 54 + lib/portage/util/futures/compat_coroutine.py | 12 + man/portage.5 | 35 +++ 14 files changed, 700 insertions(+), 66 deletions(-) create mode 100644 lib/portage/repository/storage/__init__.py create mode 100644 lib/portage/repository/storage/hardlink_quarantine.py create mode 100644 lib/portage/repository/storage/hardlink_rcu.py create mode 100644 lib/portage/repository/storage/inplace.py create mode 100644 lib/portage/repository/storage/interface.py create mode 100644 lib/portage/util/futures/_sync_decorator.py -- 2.16.4
Re: [gentoo-portage-dev] [PATCH 2/2] Update /usr/portage references (bug 378603)
On 06/08/18 19:35, Zac Medico wrote: > On 08/06/2018 12:30 AM, Brian Dolbec wrote: >> All these changes as well as the catlayst changes need to be >> co-ordinated so that snapshots and portage and stages don't precede the >> docs changes. > I suppose all of the docs changes can be made in advance, with mention > of both old a new locations. These commands can be used to check the > local configuration: > > portageq get_repo_path / gentoo > portageq distdir > portageq pkgdir Sounds like a plan .. do we need a tracker bug?! :P Let's do this! :D MJE signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH 2/2] Update /usr/portage references (bug 378603)
On 08/06/2018 02:50 PM, M. J. Everitt wrote: > On 06/08/18 19:35, Zac Medico wrote: >> On 08/06/2018 12:30 AM, Brian Dolbec wrote: >>> All these changes as well as the catlayst changes need to be >>> co-ordinated so that snapshots and portage and stages don't precede the >>> docs changes. >> I suppose all of the docs changes can be made in advance, with mention >> of both old a new locations. These commands can be used to check the >> local configuration: >> >> portageq get_repo_path / gentoo >> portageq distdir >> portageq pkgdir > Sounds like a plan .. do we need a tracker bug?! :P > > Let's do this! :D Ok, tracker created: https://bugs.gentoo.org/662982 -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [PATCH] systemd.eclass: set BDEPEND for EAPI 7
> On Mon, 06 Aug 2018, Vadim A Misbakh-Soloviov wrote:
> Anyway, I think, it is possible to add something like
> "EAPI=${EAPI:-0}" somewhere at the top of eclass, to don't call
> "${EAPI:-0}" each time when EAPI variable is needed.
No, that is not possible. Changing EAPI in an eclass would make the
inheriting ebuild invalid. See PMS section 7.3.1.
Ulrich
Re: [gentoo-dev] [PATCH] systemd.eclass: set BDEPEND for EAPI 7
On Mon, Aug 6, 2018 at 3:35 PM Vadim A. Misbakh-Soloviov
wrote:
>
> В письме от понедельник, 6 августа 2018 г. 22:13:49 MSK пользователь Ulrich
> Mueller написал:
> > > On Mon, 6 Aug 2018, Mike Gilbert wrote:
> > > -DEPEND="virtual/pkgconfig"
> > > +if [[ ${EAPI} == [0123456] ]]; then
> >
> > This should use ${EAPI:-0} because for EAPI 0 the variable can be
> > empty.
> >
> > > + DEPEND="virtual/pkgconfig"
> > > +else
> > > + BDEPEND="virtual/pkgconfig"
> > > +fi
>
> And how about "-le"/"-lt"/"-ge"/"-gt"/"-eq" syntax?
> Or even ((EAPI<7))?
> Are they forbidden to use in eclasses?
If I recall correctly, EAPI values are not required to be numeric, and
are not required to increase in any predictable manner. They only do
so by convention.
> Anyway, I think, it is possible to add something like "EAPI=${EAPI:-0}"
> somewhere at the top of eclass, to don't call "${EAPI:-0}" each time when EAPI
> variable is needed.
Re-assigning EAPI within an eclass seems like a bad idea to me.
A better solution would be to ban ebuilds with no defined EAPI, but
that's outside the scope of this patch.
Re: [gentoo-dev] [PATCH] systemd.eclass: set BDEPEND for EAPI 7
On Mon, Aug 6, 2018 at 3:51 PM, Mike Gilbert wrote:
> On Mon, Aug 6, 2018 at 3:35 PM Vadim A. Misbakh-Soloviov
> wrote:
> >
> > В письме от понедельник, 6 августа 2018 г. 22:13:49 MSK пользователь
> Ulrich
> > Mueller написал:
> > > > On Mon, 6 Aug 2018, Mike Gilbert wrote:
> > > > -DEPEND="virtual/pkgconfig"
> > > > +if [[ ${EAPI} == [0123456] ]]; then
> > >
> > > This should use ${EAPI:-0} because for EAPI 0 the variable can be
> > > empty.
> > >
> > > > + DEPEND="virtual/pkgconfig"
> > > > +else
> > > > + BDEPEND="virtual/pkgconfig"
> > > > +fi
> >
> > And how about "-le"/"-lt"/"-ge"/"-gt"/"-eq" syntax?
> > Or even ((EAPI<7))?
> > Are they forbidden to use in eclasses?
>
> If I recall correctly, EAPI values are not required to be numeric, and
> are not required to increase in any predictable manner. They only do
> so by convention.
>
They do not even do so by convention; there are numerous EAPIs in the wild
that are non-numeric.
-A
>
> > Anyway, I think, it is possible to add something like "EAPI=${EAPI:-0}"
> > somewhere at the top of eclass, to don't call "${EAPI:-0}" each time
> when EAPI
> > variable is needed.
>
> Re-assigning EAPI within an eclass seems like a bad idea to me.
>
> A better solution would be to ban ebuilds with no defined EAPI, but
> that's outside the scope of this patch.
>
>
[gentoo-dev] Re: commit 50078fbbb39667734 for linux-info.eclass
Hi, just an acknowledge for the moment that kernel project has seen the request and is currently looking into this. Thanks. -- Regards, Thomas Deutschmann / Gentoo Linux Developer C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [PATCH] systemd.eclass: set BDEPEND for EAPI 7
В письме от понедельник, 6 августа 2018 г. 22:13:49 MSK пользователь Ulrich
Mueller написал:
> > On Mon, 6 Aug 2018, Mike Gilbert wrote:
> > -DEPEND="virtual/pkgconfig"
> > +if [[ ${EAPI} == [0123456] ]]; then
>
> This should use ${EAPI:-0} because for EAPI 0 the variable can be
> empty.
>
> > + DEPEND="virtual/pkgconfig"
> > +else
> > + BDEPEND="virtual/pkgconfig"
> > +fi
And how about "-le"/"-lt"/"-ge"/"-gt"/"-eq" syntax?
Or even ((EAPI<7))?
Are they forbidden to use in eclasses?
Anyway, I think, it is possible to add something like "EAPI=${EAPI:-0}"
somewhere at the top of eclass, to don't call "${EAPI:-0}" each time when EAPI
variable is needed.
[gentoo-dev] Re: News item review v2: Migration required for OpenSSH with LDAP
Changes: * Incorporated suggestions by Peter Stuge * Package sys-auth/sakcl added --- Title: Migration required for OpenSSH with LDAP Author: Thomas Deutschmann Posted: 2018-08-xx Revision: 1 News-Item-Format: 2.0 Display-If-Installed: net-misc/openssh If your sshd authenticates against LDAP, you have to migrate your current setup to a new one using sshd's "AuthorizedKeysCommand" option and a wrapper provided by packages like the new sys-auth/ssh-ldap-pubkey or sys-auth/sakcl because beginning with net-misc/openssh-7.7_p1, deprecated OpenSSH-LPK patch set is deprecated and no longer applies. We have created a short migration guide in the Wiki [1] for more details. [1] https://wiki.gentoo.org/wiki/SSH/LDAP_migration --- -- Regards, Thomas Deutschmann / Gentoo Linux Developer C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] News item review: OpenSSH LDAP support
Hi Thomas, I suggest some improvements.. Thomas Deutschmann wrote: > Title: OpenSSH LDAP support Perhaps qualify this a bit, e.g. "Migration required for OpenSSH with LDAP" > When your sshd authenticates against LDAP, you have to migrate your s,When,If, > current setup to a new one using sshd's "AuthorizedKeysCommand" option and > use s, use,, > a wrapper provided by packages like the new sys-auth/ssh-ldap-pubkey > because beginning with net-misc/openssh-7.7_p1, deprecated OpenSSH-LPK > patch set no longer applies. Maybe "because beginning with net-misc/openssh-7.7_p1 the OpenSSH-LPK patch set is deprecated and no longer applies." Thanks a lot! //Peter
[gentoo-dev] Re: News item review v3: Migration required for OpenSSH with LDAP
Changes: * Incorporated suggestions by Peter Stuge * Package sys-auth/sakcl added * Last sentence corrected --- Title: Migration required for OpenSSH with LDAP Author: Thomas Deutschmann Posted: 2018-08-xx Revision: 1 News-Item-Format: 2.0 Display-If-Installed: net-misc/openssh If your sshd authenticates against LDAP, you have to migrate your current setup to a new one using sshd's "AuthorizedKeysCommand" option and a wrapper provided by packages like the new sys-auth/ssh-ldap-pubkey or sys-auth/sakcl because beginning with net-misc/openssh-7.7_p1, OpenSSH-LPK patch set is deprecated and no longer applies. We have created a short migration guide in the Wiki [1] for more details. [1] https://wiki.gentoo.org/wiki/SSH/LDAP_migration --- -- Regards, Thomas Deutschmann / Gentoo Linux Developer C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [PATCH] systemd.eclass: set BDEPEND for EAPI 7
On 08/06/2018 10:09 PM, Alec Warner wrote:
>
> They do not even do so by convention; there are numerous EAPIs in the
> wild that are non-numeric.
Then this line
if [[ ${EAPI} == [0123456] ]]; then
is a short-term solution, right?
--
Toralf
PGP 23217DA7 9B888F45
signature.asc
Description: OpenPGP digital signature
[gentoo-dev] commit 50078fbbb39667734 for linux-info.eclass
hi all, like someone else I do build out of sources kernels, since commit 50078fbbb3966773401c1fc59838c1e5952c1318 it has been impossible to build some modules from gentoo repo. See bug https://bugs.gentoo.org/662772 reported by another user with the same problem. Please revert the commit, after that we should discuss here an appropriate fix, with this statement I'm offering to test it for vanilla kernel, out of source builds. thank you, Francesco Riosa
Re: [gentoo-portage-dev] [PATCH 2/2] Update /usr/portage references (bug 378603)
> On Mon, 6 Aug 2018, Brian Dolbec wrote: > But that too can be changed along with all the user install > documentation which will need to be updated as well. The new > recomended location should be /var/db/repos/local. I will be updating > layman for /var/db/repos/... as well. That is the intention of the > "repos/" subdir. What is the exact plan for layman? IIUC, only the storage dir should be changed to /var/db/repos, but the rest (cache, installed, make_conf) would stay under /var/lib/layman? > All these changes as well as the catlayst changes need to be > co-ordinated so that snapshots and portage and stages don't precede > the docs changes. +1 Ulrich
[gentoo-dev] [PATCH] systemd.eclass: set BDEPEND for EAPI 7
---
eclass/systemd.eclass | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/eclass/systemd.eclass b/eclass/systemd.eclass
index 72f4845efc45..b822f54f8d06 100644
--- a/eclass/systemd.eclass
+++ b/eclass/systemd.eclass
@@ -26,11 +26,15 @@
inherit toolchain-funcs
case ${EAPI:-0} in
- 0|1|2|3|4|5|6) ;;
+ 0|1|2|3|4|5|6|7) ;;
*) die "${ECLASS}.eclass API in EAPI ${EAPI} not yet established."
esac
-DEPEND="virtual/pkgconfig"
+if [[ ${EAPI} == [0123456] ]]; then
+ DEPEND="virtual/pkgconfig"
+else
+ BDEPEND="virtual/pkgconfig"
+fi
# @FUNCTION: _systemd_get_dir
# @USAGE:
--
2.18.0
Re: [gentoo-dev] [PATCH] udev.eclass: return unprefixed udevdir
On Fri, Jun 8, 2018 at 9:54 AM Guilherme Amadio wrote: > > Hi, > > This is a simple fix, but I'd like it to be reviewed. > > Without this get_udevdir returns the prefixed directory, leading > to double prefix on udev related packages (e.g. fuse-common). > > Even if not used in prefix, udev related packages are usually pulled > in as dependencies of other packages (xrootd for fuse). > > Cheers, > -Guilherme Looks good to me.
[gentoo-dev] Re: commit 50078fbbb39667734 for linux-info.eclass
Il 06/08/2018 17:13, Francesco Riosa ha scritto:
...
> Please revert the commit, after that we should discuss here an
> appropriate fix, with this statement I'm offering to test it for vanilla
> kernel, out of source builds.
tried to fix it and it seem to work, so I retract the revert request and
instead ask for review of this patch:
From de7bec54c51361ae2aecbfa06a9cf3300a338dca Mon Sep 17 00:00:00 2001
From: Francesco Riosa
Date: Mon, 6 Aug 2018 19:18:05 +0200
Subject: [PATCH] linux-info.eclass: CONFIG_LOCALVERSION for out of source
builds
Closes: https://bugs.gentoo.org/662772
---
eclass/linux-info.eclass | 26 +-
1 file changed, 13 insertions(+), 13 deletions(-)
diff --git a/eclass/linux-info.eclass b/eclass/linux-info.eclass
index b158e345d16..98ec0ac8dab 100644
--- a/eclass/linux-info.eclass
+++ b/eclass/linux-info.eclass
@@ -548,13 +548,22 @@ get_version() {
return 1
fi
+ [ -d "${OUTPUT_DIR}" ] && KV_OUT_DIR="${OUTPUT_DIR}"
+ if [ -n "${KV_OUT_DIR}" ];
+ then
+ qeinfo "Found kernel object directory:"
+ qeinfo " ${KV_OUT_DIR}"
+ fi
+ # and if we STILL have not got it, then we better just set it to KV_DIR
+ KV_OUT_DIR="${KV_OUT_DIR:-${KV_DIR}}"
+
# Grab the kernel release from the output directory.
# TODO: we MUST detect kernel.release being out of date, and
'return 1' from
# this function.
- if [ -s "${KV_DIR}"/include/config/kernel.release ]; then
- KV_LOCAL=$(<"${KV_DIR}"/include/config/kernel.release)
- elif [ -s "${KV_DIR}"/.kernelrelease ]; then
- KV_LOCAL=$(<"${KV_DIR}"/.kernelrelease)
+ if [ -s "${KV_OUT_DIR}"/include/config/kernel.release ]; then
+ KV_LOCAL=$(<"${KV_OUT_DIR}"/include/config/kernel.release)
+ elif [ -s "${KV_OUT_DIR}"/.kernelrelease ]; then
+ KV_LOCAL=$(<"${KV_OUT_DIR}"/.kernelrelease)
else
KV_LOCAL=
fi
@@ -586,15 +595,6 @@ get_version() {
done
fi
- [ -d "${OUTPUT_DIR}" ] && KV_OUT_DIR="${OUTPUT_DIR}"
- if [ -n "${KV_OUT_DIR}" ];
- then
- qeinfo "Found kernel object directory:"
- qeinfo " ${KV_OUT_DIR}"
- fi
- # and if we STILL have not got it, then we better just set it to KV_DIR
- KV_OUT_DIR="${KV_OUT_DIR:-${KV_DIR}}"
-
# And we should set KV_FULL to the full expanded version
KV_FULL="${KV_MAJOR}.${KV_MINOR}.${KV_PATCH}${KV_EXTRA}${KV_LOCAL}"
--
2.18.0
From de7bec54c51361ae2aecbfa06a9cf3300a338dca Mon Sep 17 00:00:00 2001
From: Francesco Riosa
Date: Mon, 6 Aug 2018 19:18:05 +0200
Subject: [PATCH] linux-info.eclass: CONFIG_LOCALVERSION for out of source
builds
Closes: https://bugs.gentoo.org/662772
---
eclass/linux-info.eclass | 26 +-
1 file changed, 13 insertions(+), 13 deletions(-)
diff --git a/eclass/linux-info.eclass b/eclass/linux-info.eclass
index b158e345d16..98ec0ac8dab 100644
--- a/eclass/linux-info.eclass
+++ b/eclass/linux-info.eclass
@@ -548,13 +548,22 @@ get_version() {
return 1
fi
+ [ -d "${OUTPUT_DIR}" ] && KV_OUT_DIR="${OUTPUT_DIR}"
+ if [ -n "${KV_OUT_DIR}" ];
+ then
+ qeinfo "Found kernel object directory:"
+ qeinfo "${KV_OUT_DIR}"
+ fi
+ # and if we STILL have not got it, then we better just set it to KV_DIR
+ KV_OUT_DIR="${KV_OUT_DIR:-${KV_DIR}}"
+
# Grab the kernel release from the output directory.
# TODO: we MUST detect kernel.release being out of date, and 'return 1' from
# this function.
- if [ -s "${KV_DIR}"/include/config/kernel.release ]; then
- KV_LOCAL=$(<"${KV_DIR}"/include/config/kernel.release)
- elif [ -s "${KV_DIR}"/.kernelrelease ]; then
- KV_LOCAL=$(<"${KV_DIR}"/.kernelrelease)
+ if [ -s "${KV_OUT_DIR}"/include/config/kernel.release ]; then
+ KV_LOCAL=$(<"${KV_OUT_DIR}"/include/config/kernel.release)
+ elif [ -s "${KV_OUT_DIR}"/.kernelrelease ]; then
+ KV_LOCAL=$(<"${KV_OUT_DIR}"/.kernelrelease)
else
KV_LOCAL=
fi
@@ -586,15 +595,6 @@ get_version() {
done
fi
- [ -d "${OUTPUT_DIR}" ] && KV_OUT_DIR="${OUTPUT_DIR}"
- if [ -n "${KV_OUT_DIR}" ];
- then
- qeinfo "Found kernel object directory:"
- qeinfo "${KV_OUT_DIR}"
- fi
- # and if we STILL have not got it, then we better just set it to KV_DIR
- KV_OUT_DIR="${KV_OUT_DIR:-${KV_DIR}}"
-
# And we should set KV_FULL to the full expanded version
KV_FULL="${KV_MAJOR}.${KV_MINOR}.${KV_PATCH}${KV_EXTRA}${KV_LOCAL}"
--
2.18.0
Re: [gentoo-portage-dev] [PATCH 2/2] Update /usr/portage references (bug 378603)
On 08/06/2018 12:30 AM, Brian Dolbec wrote: > All these changes as well as the catlayst changes need to be > co-ordinated so that snapshots and portage and stages don't precede the > docs changes. I suppose all of the docs changes can be made in advance, with mention of both old a new locations. These commands can be used to check the local configuration: portageq get_repo_path / gentoo portageq distdir portageq pkgdir -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH 2/2 v2] Update /usr/portage references (bug 378603)
On 08/06/2018 12:44 AM, Brian Dolbec wrote:
> On Sun, 5 Aug 2018 22:32:34 -0700
> Zac Medico wrote:
>> a/repoman/lib/repoman/checks/herds/herdbase.py
>> b/repoman/lib/repoman/checks/herds/herdbase.py index
>> ebe6a19b4..1e7c0b27c 100644 ---
>> a/repoman/lib/repoman/checks/herds/herdbase.py +++
>> b/repoman/lib/repoman/checks/herds/herdbase.py @@ -119,7 +119,7 @@
>> def get_herd_base(repoman_settings):
>> if __name__ == '__main__':
>> -h = make_herd_base('/usr/portage/metadata/herds.xml')
>> +h = make_herd_base('/var/db/repos/gentoo/metadata/herds.xml')
>>
>> assert(h.known_herd('sound'))
>> assert(not h.known_herd('media-sound'))
>
> Has it been long enough since herds are removed from gentoo that we
> can drop the herds code completey? but otherwise the change itself is
> fine.
Yeah, let's remove it:
https://github.com/gentoo/portage/pull/353
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-dev] [PATCH] systemd.eclass: set BDEPEND for EAPI 7
> On Mon, 6 Aug 2018, Mike Gilbert wrote:
> -DEPEND="virtual/pkgconfig"
> +if [[ ${EAPI} == [0123456] ]]; then
This should use ${EAPI:-0} because for EAPI 0 the variable can be
empty.
> + DEPEND="virtual/pkgconfig"
> +else
> + BDEPEND="virtual/pkgconfig"
> +fi
