[gentoo-portage-dev] [PATCH 4/4] rsync: add key refresh retry (bug 649276)

[Zac Medico]

Since key refresh is prone to failure, retry using exponential
backoff with random jitter. This adds the following sync-openpgp-*
configuration settings:

sync-openpgp-key-refresh-retry-count = 40

  Maximum number of times to retry key refresh if it fails.  Between
  each  key  refresh attempt, there is an exponential delay with a
  constant multiplier and a uniform random multiplier between 0 and 1.

sync-openpgp-key-refresh-retry-delay-exp-base = 2

  The base of the exponential expression. The exponent is the number
  of previous refresh attempts.

sync-openpgp-key-refresh-retry-delay-max = 60

  Maximum  delay between each retry attempt, in units of seconds. This
  places a limit on the length of the exponential delay.

sync-openpgp-key-refresh-retry-delay-mult = 4

  Multiplier for the exponential delay.

sync-openpgp-key-refresh-retry-overall-timeout = 1200

  Combined time limit for all refresh attempts, in units of seconds.

Bug: https://bugs.gentoo.org/649276
---
 cnf/repos.conf  |  5 ++
 man/portage.5   | 19 
 pym/portage/repository/config.py| 22 +
 pym/portage/sync/modules/rsync/rsync.py | 16 ++-
 pym/portage/sync/syncbase.py| 85 -
 5 files changed, 144 insertions(+), 3 deletions(-)

diff --git a/cnf/repos.conf b/cnf/repos.conf
index 984ecd220..5759b8b43 100644
--- a/cnf/repos.conf
+++ b/cnf/repos.conf
@@ -9,6 +9,11 @@ auto-sync = yes
 sync-rsync-verify-metamanifest = yes
 sync-rsync-verify-max-age = 24
 sync-openpgp-key-path = 
/var/lib/gentoo/gkeys/keyrings/gentoo/release/pubring.gpg
+sync-openpgp-key-refresh-retry-count = 40
+sync-openpgp-key-refresh-retry-overall-timeout = 1200
+sync-openpgp-key-refresh-retry-delay-exp-base = 2
+sync-openpgp-key-refresh-retry-delay-max = 60
+sync-openpgp-key-refresh-retry-delay-mult = 4
 
 # for daily squashfs snapshots
 #sync-type = squashdelta
diff --git a/man/portage.5 b/man/portage.5
index 549c51c73..d3e258a43 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -1081,6 +1081,25 @@ only for protocols supporting cryptographic 
verification, provided
 that the respective verification option is enabled. If unset, the user's
 keyring is used.
 .TP
+.B sync\-openpgp\-key\-refresh\-retry\-count = 40
+Maximum number of times to retry key refresh if it fails. Between each
+key refresh attempt, there is an exponential delay with a constant
+multiplier and a uniform random multiplier between 0 and 1.
+.TP
+.B sync\-openpgp\-key\-refresh\-retry\-delay\-exp\-base = 2
+The base of the exponential expression. The exponent is the number of
+previous refresh attempts.
+.TP
+.B sync\-openpgp\-key\-refresh\-retry\-delay\-max = 60
+Maximum delay between each retry attempt, in units of seconds. This
+places a limit on the length of the exponential delay.
+.TP
+.B sync\-openpgp\-key\-refresh\-retry\-delay\-mult = 4
+Multiplier for the exponential delay.
+.TP
+.B sync\-openpgp\-key\-refresh\-retry\-overall\-timeout = 1200
+Combined time limit for all refresh attempts, in units of seconds.
+.TP
 .B sync-rsync-vcs-ignore = true|false
 Ignore vcs directories that may be present in the repository. It is the
 user's responsibility to set sync-rsync-extra-opts to protect vcs
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index b5db4855f..1d897bb90 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -87,6 +87,11 @@ class RepoConfig(object):
'update_changelog', '_eapis_banned', '_eapis_deprecated',
'_masters_orig', 'module_specific_options', 
'manifest_required_hashes',
'sync_openpgp_key_path',
+   'sync_openpgp_key_refresh_retry_count',
+   'sync_openpgp_key_refresh_retry_delay_max',
+   'sync_openpgp_key_refresh_retry_delay_exp_base',
+   'sync_openpgp_key_refresh_retry_delay_mult',
+   'sync_openpgp_key_refresh_retry_overall_timeout',
)
 
def __init__(self, name, repo_opts, local_config=True):
@@ -186,6 +191,13 @@ class RepoConfig(object):
self.sync_openpgp_key_path = repo_opts.get(
'sync-openpgp-key-path', None)
 
+   for k in ('sync_openpgp_key_refresh_retry_count',
+   'sync_openpgp_key_refresh_retry_delay_max',
+   'sync_openpgp_key_refresh_retry_delay_exp_base',
+   'sync_openpgp_key_refresh_retry_delay_mult',
+   'sync_openpgp_key_refresh_retry_overall_timeout'):
+   setattr(self, k, repo_opts.get(k.replace('_', '-'), 
None))
+
self.module_specific_options = {}
 
# Not implemented.
@@ -523,6 +535,11 @@ class RepoConfigLoader(object):
'force', 'masters', 
'priority', 'strict_misc_digests',
 

[gentoo-portage-dev] [PATCH 3/4] Add retry decorator (API inspired by tenacity)

[Zac Medico]

This decorator will be useful for retrying asynchronous
operations, such as gpg key refresh (bug 649276). The
API is inspired by tenacity, but is simpler. Only
asynchronous functions (like @asyncio.coroutine functions)
are supported. In order to retry a synchronous function,
first convert it to an asynchronous function as follows:

asynchronous_func = functools.partial(
loop.run_in_executor, None, synchronous_func)

Bug: https://bugs.gentoo.org/649276
See: https://github.com/jd/tenacity
---
 pym/portage/tests/util/futures/test_retry.py | 147 ++
 pym/portage/util/futures/futures.py  |   6 +
 pym/portage/util/futures/retry.py| 178 +++
 3 files changed, 331 insertions(+)
 create mode 100644 pym/portage/tests/util/futures/test_retry.py
 create mode 100644 pym/portage/util/futures/retry.py

diff --git a/pym/portage/tests/util/futures/test_retry.py 
b/pym/portage/tests/util/futures/test_retry.py
new file mode 100644
index 0..7641e4e92
--- /dev/null
+++ b/pym/portage/tests/util/futures/test_retry.py
@@ -0,0 +1,147 @@
+# Copyright 2018 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import functools
+
+try:
+   import threading
+except ImportError:
+   import dummy_threading as threading
+
+from portage.tests import TestCase
+from portage.util._eventloop.global_event_loop import global_event_loop
+from portage.util.backoff import RandomExponentialBackoff
+from portage.util.futures.futures import TimeoutError
+from portage.util.futures.retry import retry
+from portage.util.futures.wait import wait
+from portage.util.monotonic import monotonic
+
+
+class SucceedLaterException(Exception):
+   pass
+
+
+class SucceedLater(object):
+   """
+   A callable object that succeeds some duration of time has passed.
+   """
+   def __init__(self, duration):
+   self._succeed_time = monotonic() + duration
+
+   def __call__(self):
+   remaining = self._succeed_time - monotonic()
+   if remaining > 0:
+   raise SucceedLaterException('time until success: {} 
seconds'.format(remaining))
+   return 'success'
+
+
+class SucceedNeverException(Exception):
+   pass
+
+
+class SucceedNever(object):
+   """
+   A callable object that never succeeds.
+   """
+   def __call__(self):
+   raise SucceedNeverException('expected failure')
+
+
+class HangForever(object):
+   """
+   A callable object that sleeps forever.
+   """
+   def __call__(self):
+   threading.Event().wait()
+
+
+class RetryTestCase(TestCase):
+   def testSucceedLater(self):
+   loop = global_event_loop()
+   func = SucceedLater(1)
+   func_coroutine = functools.partial(loop.run_in_executor, None, 
func)
+   decorator = retry(try_max=,
+   delay_func=RandomExponentialBackoff(multiplier=0.1, 
base=2))
+   decorated_func = decorator(func_coroutine)
+   result = loop.run_until_complete(decorated_func())
+   self.assertEqual(result, 'success')
+
+   def testSucceedNever(self):
+   loop = global_event_loop()
+   func = SucceedNever()
+   func_coroutine = functools.partial(loop.run_in_executor, None, 
func)
+   decorator = retry(try_max=4, try_timeout=None,
+   delay_func=RandomExponentialBackoff(multiplier=0.1, 
base=2))
+   decorated_func = decorator(func_coroutine)
+   done, pending = 
loop.run_until_complete(wait([decorated_func()]))
+   self.assertEqual(len(done), 1)
+   self.assertTrue(isinstance(done[0].exception().__cause__, 
SucceedNeverException))
+
+   def testSucceedNeverReraise(self):
+   loop = global_event_loop()
+   func = SucceedNever()
+   func_coroutine = functools.partial(loop.run_in_executor, None, 
func)
+   decorator = retry(reraise=True, try_max=4, try_timeout=None,
+   delay_func=RandomExponentialBackoff(multiplier=0.1, 
base=2))
+   decorated_func = decorator(func_coroutine)
+   done, pending = 
loop.run_until_complete(wait([decorated_func()]))
+   self.assertEqual(len(done), 1)
+   self.assertTrue(isinstance(done[0].exception(), 
SucceedNeverException))
+
+   def testHangForever(self):
+   loop = global_event_loop()
+   func = HangForever()
+   func_coroutine = functools.partial(loop.run_in_executor, None, 
func)
+   decorator = retry(try_max=2, try_timeout=0.1,
+   delay_func=RandomExponentialBackoff(multiplier=0.1, 
base=2))
+   decorated_func = decorator(func_coroutine)
+   done, pending = 

[gentoo-portage-dev] [PATCH 2/4] Add ExponentialBackoff and RandomExponentialBackoff

[Zac Medico]

This will be useful as parameters for retry decorators.
---
 pym/portage/util/backoff.py | 48 +
 1 file changed, 48 insertions(+)
 create mode 100644 pym/portage/util/backoff.py

diff --git a/pym/portage/util/backoff.py b/pym/portage/util/backoff.py
new file mode 100644
index 0..d8854337c
--- /dev/null
+++ b/pym/portage/util/backoff.py
@@ -0,0 +1,48 @@
+# Copyright 2018 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import random
+import sys
+
+
+class ExponentialBackoff(object):
+   """
+   An object that when called with number of previous tries, calculates
+   an exponential delay for the next try.
+   """
+   def __init__(self, multiplier=1, base=2, limit=sys.maxsize):
+   """
+   @param multiplier: constant multiplier
+   @type multiplier: int or float
+   @param base: maximum number of tries
+   @type base: int or float
+   @param limit: maximum number of seconds to delay
+   @type limit: int or float
+   """
+   self._multiplier = multiplier
+   self._base = base
+   self._limit = limit
+
+   def __call__(self, tries):
+   """
+   Given a number of previous tries, calculate the amount of time
+   to delay the next try.
+
+   @param tries: number of previous tries
+   @type tries: int
+   @return: amount of time to delay the next try
+   @rtype: int
+   """
+   try:
+   return min(self._limit, self._multiplier * (self._base 
** tries))
+   except OverflowError:
+   return self._limit
+
+
+class RandomExponentialBackoff(ExponentialBackoff):
+   """
+   Equivalent to ExponentialBackoff, with an extra multiplier that uses
+   a random distribution between 0 and 1.
+   """
+   def __call__(self, tries):
+   return random.random() * super(RandomExponentialBackoff, 
self).__call__(tries)
-- 
2.13.6

[gentoo-portage-dev] [PATCH 1/4] Add ForkExecutor (bug 649588)

[Zac Medico]

This is useful for asynchronous operations that we might
need to cancel if they take too long, since (concurrent.
futures.ProcessPoolExecutor tasks are not cancellable).
This ability to cancel tasks makes this executor useful
as an alternative to portage.exception.AlarmSignal.

Also add an asyncio-compatible EventLoop.run_in_executor
method the uses ForkExecutor as the default executor,
which will later be used to implement the corresponding
asyncio.AbstractEventLoop run_in_executor method.

Bug: https://bugs.gentoo.org/649588
---
 pym/portage/util/_eventloop/EventLoop.py  |  45 -
 pym/portage/util/futures/executor/__init__.py |   0
 pym/portage/util/futures/executor/fork.py | 130 ++
 3 files changed, 174 insertions(+), 1 deletion(-)
 create mode 100644 pym/portage/util/futures/executor/__init__.py
 create mode 100644 pym/portage/util/futures/executor/fork.py

diff --git a/pym/portage/util/_eventloop/EventLoop.py 
b/pym/portage/util/_eventloop/EventLoop.py
index f472a3dae..1574a6837 100644
--- a/pym/portage/util/_eventloop/EventLoop.py
+++ b/pym/portage/util/_eventloop/EventLoop.py
@@ -24,6 +24,7 @@ except ImportError:
 import portage
 portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.futures.futures:_EventLoopFuture',
+   'portage.util.futures.executor.fork:ForkExecutor',
 )
 
 from portage import OrderedDict
@@ -122,6 +123,7 @@ class EventLoop(object):
self._idle_callbacks = OrderedDict()
self._timeout_handlers = {}
self._timeout_interval = None
+   self._default_executor = None
 
self._poll_obj = None
try:
@@ -721,6 +723,46 @@ class EventLoop(object):
return self._handle(self.timeout_add(
delay * 1000, self._call_soon_callback(callback, 
args)), self)
 
+   def run_in_executor(self, executor, func, *args):
+   """
+   Arrange for a func to be called in the specified executor.
+
+   The executor argument should be an Executor instance. The 
default
+   executor is used if executor is None.
+
+   Use functools.partial to pass keywords to the *func*.
+
+   @param executor: executor
+   @type executor: concurrent.futures.Executor or None
+   @param func: a function to call
+   @type func: callable
+   @return: a Future
+   @rtype: asyncio.Future (or compatible)
+   """
+   if executor is None:
+   executor = self._default_executor
+   if executor is None:
+   executor = ForkExecutor(loop=self)
+   self._default_executor = executor
+   return executor.submit(func, *args)
+
+   def close(self):
+   """Close the event loop.
+
+   This clears the queues and shuts down the executor,
+   and waits for it to finish.
+   """
+   executor = self._default_executor
+   if executor is not None:
+   self._default_executor = None
+   executor.shutdown(wait=True)
+
+   if self._poll_obj is not None:
+   close = getattr(self._poll_obj, 'close')
+   if close is not None:
+   close()
+   self._poll_obj = None
+
 
 _can_poll_device = None
 
@@ -782,10 +824,11 @@ class _epoll_adapter(object):
that is associated with an epoll instance will close automatically when
it is garbage collected, so it's not necessary to close it explicitly.
"""
-   __slots__ = ('_epoll_obj',)
+   __slots__ = ('_epoll_obj', 'close')
 
def __init__(self, epoll_obj):
self._epoll_obj = epoll_obj
+   self.close = epoll_obj.close
 
def register(self, fd, *args):
self._epoll_obj.register(fd, *args)
diff --git a/pym/portage/util/futures/executor/__init__.py 
b/pym/portage/util/futures/executor/__init__.py
new file mode 100644
index 0..e69de29bb
diff --git a/pym/portage/util/futures/executor/fork.py 
b/pym/portage/util/futures/executor/fork.py
new file mode 100644
index 0..9cd1db2ca
--- /dev/null
+++ b/pym/portage/util/futures/executor/fork.py
@@ -0,0 +1,130 @@
+# Copyright 2018 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import collections
+import functools
+import multiprocessing
+import os
+import sys
+import traceback
+
+from portage.util._async.AsyncFunction import AsyncFunction
+from portage.util._eventloop.global_event_loop import global_event_loop
+
+
+class ForkExecutor(object):
+   """
+   An implementation of concurrent.futures.Executor that forks a
+   new process for each task, with support for cancellation of tasks.
+

[gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

[Zac Medico]

Since key refresh is prone to failure, retry using exponential
backoff with random jitter. This adds the following sync-openpgp-*
configuration settings:

sync-openpgp-key-refresh-retry-count = 40

  Maximum number of times to retry key refresh if it fails.  Between
  each  key  refresh attempt, there is an exponential delay with a
  constant multiplier and a uniform random multiplier between 0 and 1.

sync-openpgp-key-refresh-retry-delay-exp-base = 2

  The base of the exponential expression. The exponent is the number
  of previous refresh attempts.

sync-openpgp-key-refresh-retry-delay-max = 60

  Maximum  delay between each retry attempt, in units of seconds. This
  places a limit on the length of the exponential delay.

sync-openpgp-key-refresh-retry-delay-mult = 4

  Multiplier for the exponential delay.

sync-openpgp-key-refresh-retry-overall-timeout = 1200

  Combined time limit for all refresh attempts, in units of seconds.

Bug: https://bugs.gentoo.org/649276

Zac Medico (4):
  Add ForkExecutor (bug 649588)
  Add ExponentialBackoff and RandomExponentialBackoff
  Add retry decorator (API inspired by tenacity)
  rsync: add key refresh retry (bug 649276)

 cnf/repos.conf|   5 +
 man/portage.5 |  19 +++
 pym/portage/repository/config.py  |  22 
 pym/portage/sync/modules/rsync/rsync.py   |  16 ++-
 pym/portage/sync/syncbase.py  |  85 +++-
 pym/portage/tests/util/futures/test_retry.py  | 147 +
 pym/portage/util/_eventloop/EventLoop.py  |  45 ++-
 pym/portage/util/backoff.py   |  48 +++
 pym/portage/util/futures/executor/__init__.py |   0
 pym/portage/util/futures/executor/fork.py | 130 +++
 pym/portage/util/futures/futures.py   |   6 +
 pym/portage/util/futures/retry.py | 178 ++
 12 files changed, 697 insertions(+), 4 deletions(-)
 create mode 100644 pym/portage/tests/util/futures/test_retry.py
 create mode 100644 pym/portage/util/backoff.py
 create mode 100644 pym/portage/util/futures/executor/__init__.py
 create mode 100644 pym/portage/util/futures/executor/fork.py
 create mode 100644 pym/portage/util/futures/retry.py

-- 
2.13.6

Re: [gentoo-dev] rfc: empty directories in ${D}

[Michał Górny]

W dniu sob, 31.03.2018 o godzinie 20∶46 +0100, użytkownik Andrey Utkin
napisał:
> On Thu, Mar 29, 2018 at 11:57:06AM -0400, Alec Warner wrote:
> > On Thu, Mar 29, 2018 at 11:47 AM, Michael Orlitzky  wrote:
> > 
> > > On 03/29/2018 11:28 AM, Alec Warner wrote:
> > > > 
> > > > Is there any particular reason we need to remove them?
> > > > 
> > > 
> > > The PMS says that empty directories are undefined, so the portage
> > > behavior of installing them and leaving them alone leads to
> > > incompatibilities. Ebuilds rely on the portage behavior, and if another
> > > PM (within its rights) deletes them, then the package breaks with the
> > > non-portage PM.
> > > 
> > > 
> > 
> > So we could simply change the PMS to keep the empty directories?
> > 
> > Why is removing them *better* is my question.
> 
> Right, I am not aware why PMS has left this explicitly undefined. Have
> read through https://bugs.gentoo.org/644366 but there's no hint on why,
> too. I appreciate mjo's proposal. I think it would be good for ebuild
> maintainer to have a switch "empty dirs are ok by default". The
> disagreement seems to be based on a prejudice and distrust towards
> upstreams' build systems.

You can accuse developers of 'prejudice and distrust', or you can
believe that they actually have some experience and knowledge to prove
the point. But if you start with prejudice towards Gentoo developers,
then I suppose there's no point in arguing further.

-- 
Best regards,
Michał Górny

[gentoo-portage-dev] [PATCH] revdep-rebuild.sh: use awk instead of gawk

[Zac Medico]

From: Hadrien Lacour 

---
 bin/revdep-rebuild.sh | 28 +++-
 1 file changed, 19 insertions(+), 9 deletions(-)

diff --git a/bin/revdep-rebuild.sh b/bin/revdep-rebuild.sh
index 633701e..5fecf97 100755
--- a/bin/revdep-rebuild.sh
+++ b/bin/revdep-rebuild.sh
@@ -235,9 +235,19 @@ countdown() {
 # Replace whitespace with linebreaks, normalize repeated '/' chars, and sort -u
 # (If any libs have whitespace in their filenames, someone needs punishment.)
 clean_var() {
-   gawk 'BEGIN {RS="[[:space:]]"}
-/-\*/ {exit}
-/[^[:space:]]/ {gsub(/\/\/+/, "/"); print}' | sort -u
+   awk '
+   BEGIN {FS = "[[:space:]]"}
+
+   {
+   for(i = 1; i <= NF; ++i) {
+   if($i ~ /-\*/)
+   exit
+   else if($i){
+   gsub(/\/\/+/, "/", $i)
+   print $i
+   }
+   }
+   }' | sort -u
 }
 ##
 # Exit and optionally output to sterr
@@ -805,8 +815,8 @@ main_checks() {
# Look for symbol not defined errors
if grep -vF "${LD_LIBRARY_MASK:=$'\a'}" 
<<< "$ldd_output" |
grep -q -E 'symbol .* not 
defined'; then
-   message=$(gawk '/symbol .* not 
defined/ {NF--; print $0}' <<< "$ldd_output")
-   broken_lib=$(gawk '/symbol .* 
not defined/ {print $NF}' <<< "$ldd_output" | \
+   message=$(awk '/symbol .* not 
defined/ {ORS = FS; for(i = 1; i < NF; ++i) print $i; printf "\n"}' <<< 
"$ldd_output")
+   broken_lib=$(awk '/symbol .* 
not defined/ {print $NF}' <<< "$ldd_output" | \
sed 's/[()]//g')
echo "obj $broken_lib" >> 
"$BROKEN_FILE"
echo_v "  broken $broken_lib 
($message)"
@@ -820,7 +830,7 @@ main_checks() {
*)
if grep -vF 
"${LD_LIBRARY_MASK:=$'\a'}" <<< "$ldd_output" |
grep -q -F 'undefined 
symbol:'; then
-   message=$(gawk 
'/undefined symbol:/ {print $3}' <<< "$ldd_output")
+   message=$(awk 
'/undefined symbol:/ {print $3}' <<< "$ldd_output")

message="${message//$'\n'/ }"
echo "obj $target_file" 
>> "$BROKEN_FILE"
echo_v "  broken 
$target_file (undefined symbols(s): $message)"
@@ -835,7 +845,7 @@ main_checks() {
la_broken=""
la_lib=""
for depend in $(
-   gawk -F"[=']" '/^dependency_libs/{
+   awk -F"[=']" '/^dependency_libs/{
print $3
}' "$target_file"
); do
@@ -876,7 +886,7 @@ main_checks() {
done < <(
# Regexify LD_LIBRARY_MASK. Exclude it from the 
search.
LD_LIBRARY_MASK="${LD_LIBRARY_MASK//$'\n'/|}"
-   gawk -v ldmask="(${LD_LIBRARY_MASK//./.})" '
+   awk -v ldmask="(${LD_LIBRARY_MASK//./.})" '
/no version information available/ && 
$0 !~ ldmask {
gsub(/[()]/, "", $NF)
if (seen[$NF]++)  next
@@ -1068,7 +1078,7 @@ show_unowned_files() {
ewarn "The broken files are:"
while read filename junk; do
[[ $junk = *none* ]] && ewarn "  $filename"
-   done < "$OWNERS_FILE" | gawk '!s[$0]++' # (omit dupes)
+   done < "$OWNERS_FILE" | awk '!s[$0]++' # (omit dupes)
fi
 }
 
-- 
2.13.6

Re: [gentoo-dev] rfc: empty directories in ${D}

[Andrey Utkin]

On Thu, Mar 29, 2018 at 11:57:06AM -0400, Alec Warner wrote:
> On Thu, Mar 29, 2018 at 11:47 AM, Michael Orlitzky  wrote:
> 
> > On 03/29/2018 11:28 AM, Alec Warner wrote:
> > >
> > > Is there any particular reason we need to remove them?
> > >
> >
> > The PMS says that empty directories are undefined, so the portage
> > behavior of installing them and leaving them alone leads to
> > incompatibilities. Ebuilds rely on the portage behavior, and if another
> > PM (within its rights) deletes them, then the package breaks with the
> > non-portage PM.
> >
> >
> So we could simply change the PMS to keep the empty directories?
> 
> Why is removing them *better* is my question.

Right, I am not aware why PMS has left this explicitly undefined. Have
read through https://bugs.gentoo.org/644366 but there's no hint on why,
too. I appreciate mjo's proposal. I think it would be good for ebuild
maintainer to have a switch "empty dirs are ok by default". The
disagreement seems to be based on a prejudice and distrust towards
upstreams' build systems.


signature.asc
Description: Digital signature

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[James Le Cuirot]

On Sat, 31 Mar 2018 09:39:47 + (UTC)
Farid BENAMROUCHE  wrote:

> interresting aproach.
> this could work. however, i can see a few limitations:
> - you must be root.

Actually you don't if you add -r to unshare, which gives you what is
sometimes called fakeroot. Obviously you still can't modify the files
if they are really owned by root but that's true of any solution.

> - this is specific to linux as of today.

True and I am only interested in Linux but I like to play nice. Other
platforms could potentially still briefly bind mount but it wouldn't be
isolated from the other processes so it wouldn't be entirely safe. Safe
enough though? You'd need to weigh this up against how many people use
ROOT!=/ on other platforms. Not many at all, I imagine.

> - if you want to hide the mechanism, i don't see how without doing
> the same portage modifications as in my solution.

You could handle this in the eclass functions but as you pointed out,
many things call chown/chgrp directly. Usage by ebuilds themselves can
be addressed but if a build system calls these then eclass functions
will not help. What would work is adding some identically-named
wrappers to the PATH.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer


pgpt1qN6u7M80.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[Farid BENAMROUCHE]

interresting aproach.
this could work. however, i can see a few limitations:
- you must be root.
- this is specific to linux as of today.
- if you want to hide the mechanism, i don't see how without doing the same 
portage modifications as in my solution.


but this is maybe worth investigating. my solution isn't perfect too, I admit.




En date de : Ven 30.3.18, James Le Cuirot  a écrit :

 Objet: Re: [gentoo-dev] Re : Modification proposal for user/group creation 
when ROOT!="/"
 À: gentoo-dev@lists.gentoo.org
 Date: Vendredi 30 mars 2018, 21h56
 
 On Fri, 30 Mar 2018 20:47:20
 +0100
 James Le Cuirot 
 wrote:
 
 > On Fri, 30 Mar
 2018 20:23:49 +0100
 > James Le Cuirot
 
 wrote:
 > 
 > > I did
 just have a lightbulb moment though. I've been playing
 with
 > > unshare recently and I
 wondered if we could leverage it here.
 >
 > 
 > >   $ sudo unshare -m /bin/sh
 -c "mount --bind /mnt/somewhere/etc /etc &&
 groupadd foo"
 > >   groupadd:
 Cannot determine your user name.  
 > 
 > Aha! I was trying to do this against an
 NFS share for a system with a
 > different
 architecture. If I use a local mount with a compatible
 > architecture, it actually does work.
 I'll explore this some more.
 
 Figured it out! The system I was doing this
 against has an ancient
 glibc (long story)
 with an old nsswitch.conf. I replaced this file with
 a newer one and it all started working. Do you
 agree this could be the
 way forwards?
 
 -- 
 James Le Cuirot (chewi)
 Gentoo Linux
 Developer