[gentoo-portage-dev] [PATCH] _wrap_loop: Prevent redundant AsyncioEventLoop instances
Ultimately the loop arguments that necessitate the _wrap_loop function can be removed, because our aim since bug 761538 should be to eliminate them. Meanwhile, we don't want _wrap_loop to return redundant AsyncioEventLoop instances if we can easily prevent it. Therefore, use _safe_loop(create=False) to look up the AsyncioEventLoop instance associated with the current thread, and avoid creating redundant instances. This serves to guard against garbage collection of AsyncioEventLoop instances which may have _coroutine_exithandlers added by the atexit_register function since commit cb0c09d8cecb from bug 937740. If _safe_loop(create=False) fails to associate a loop with the current thread, raise an AssertionError for portage internal API consumers. It's not known whether external API consumers will trigger this case, so if it happens then emit a UserWarning and return a temporary AsyncioEventLoop instance. Fixes: cb0c09d8cecb ("Support coroutine exitfuncs for non-main loops") Bug: https://bugs.gentoo.org/938127 Bug: https://bugs.gentoo.org/937740 Bug: https://bugs.gentoo.org/761538 Closes: https://github.com/gentoo/portage/pull/1372 Signed-off-by: Zac Medico --- lib/portage/util/futures/_asyncio/__init__.py | 43 --- 1 file changed, 38 insertions(+), 5 deletions(-) diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py index 8805e35756..c960d03630 100644 --- a/lib/portage/util/futures/_asyncio/__init__.py +++ b/lib/portage/util/futures/_asyncio/__init__.py @@ -26,6 +26,7 @@ __all__ = ( import sys import types +import warnings import weakref import asyncio as _real_asyncio @@ -46,6 +47,7 @@ from asyncio import ( ) import threading +from typing import Optional import portage @@ -251,11 +253,35 @@ def _wrap_loop(loop=None): # The default loop returned by _wrap_loop should be consistent # with global_event_loop, in order to avoid accidental registration # of callbacks with a loop that is not intended to run. -loop = loop or _safe_loop() -return loop if hasattr(loop, "_asyncio_wrapper") else _AsyncioEventLoop(loop=loop) +if hasattr(loop, "_asyncio_wrapper"): +return loop + +# This returns a running loop if it exists, and otherwise returns +# a loop associated with the current thread. +safe_loop = _safe_loop(create=loop is None) +if safe_loop is not None and (loop is None or safe_loop._loop is loop): +return safe_loop + +if safe_loop is None: +msg = f"_wrap_loop argument '{loop}' not associated with thread '{threading.get_ident()}'" +else: +msg = f"_wrap_loop argument '{loop}' different frome loop '{safe_loop._loop}' already associated with thread '{threading.get_ident()}'" + +if portage._internal_caller: +raise AssertionError(msg) +# It's not known whether external API consumers will trigger this case, +# so if it happens then emit a UserWarning before returning a temporary +# AsyncioEventLoop instance. +warnings.warn(msg, UserWarning, stacklevel=2) -def _safe_loop(): +# We could possibly add a weak reference in _thread_weakrefs.loops when +# safe_loop is None, but if safe_loop is not None, then there is a +# collision in _thread_weakrefs.loops that would need to be resolved. +return _AsyncioEventLoop(loop=loop) + + +def _safe_loop(create: Optional[bool] = True) -> Optional[_AsyncioEventLoop]: """ Return an event loop that's safe to use within the current context. For portage internal callers or external API consumers calling from @@ -276,8 +302,13 @@ def _safe_loop(): are added to a WeakValueDictionary, and closed via an atexit hook if they still exist during exit for the current pid. -@rtype: asyncio.AbstractEventLoop (or compatible) -@return: event loop instance +@type create: bool +@param create: Create a loop by default if a loop is not already associated +with the current thread. If create is False, then return None if a loop +is not already associated with the current thread. +@rtype: AsyncioEventLoop or None +@return: event loop instance, or None if the create parameter is False and +a loop is not already associated with the current thread. """ loop = _get_running_loop() if loop is not None: @@ -292,6 +323,8 @@ def _safe_loop(): try: loop = _thread_weakrefs.loops[thread_key] except KeyError: +if not create: +return None try: try: _loop = _real_asyncio.get_running_loop() -- 2.44.2
[gentoo-portage-dev] [PATCH] Support coroutine exitfuncs for non-main loops
Since an API consumer can cause loops to be instantiated for non-main threads, support coroutine exitfuncs for each loop. The included Socks5ServerAtExitThreadedTestCase calls get_socks5_proxy from a non-main thread, and demonstrates that coroutine exitfuncs for the resulting non-main loop will reliably stop the socks5 proxy via atexit hook. The _thread_weakrefs_atexit function will now make a temporary adjustment to _thread_weakrefs.loops so that a loop is associated with the current thread when it is closing. Also, the _get_running_loop function will now store weak references to all _AsyncioEventLoop instances it creates, since each has a _coroutine_exithandlers attribute that can be modified by atexit_register calls. Bug: https://bugs.gentoo.org/937740 Closes: https://github.com/gentoo/portage/pull/1368 Signed-off-by: Zac Medico --- lib/portage/process.py| 11 +++-- lib/portage/tests/util/test_socks5.py | 38 - .../util/_eventloop/asyncio_event_loop.py | 15 ++- lib/portage/util/futures/_asyncio/__init__.py | 41 +-- 4 files changed, 76 insertions(+), 29 deletions(-) diff --git a/lib/portage/process.py b/lib/portage/process.py index 23e2507b53..38adebda66 100644 --- a/lib/portage/process.py +++ b/lib/portage/process.py @@ -194,7 +194,6 @@ def spawn_fakeroot(mycommand, fakeroot_state=None, opt_name=None, **keywords): _exithandlers = [] -_coroutine_exithandlers = [] def atexit_register(func, *args, **kargs): @@ -205,7 +204,9 @@ def atexit_register(func, *args, **kargs): # The internal asyncio wrapper module would trigger a circular import # if used here. if _asyncio.iscoroutinefunction(func): -_coroutine_exithandlers.append((func, args, kargs)) +# Add this coroutine function to the exit handlers for the loop +# which is associated with the current thread. +global_event_loop()._coroutine_exithandlers.append((func, args, kargs)) else: _exithandlers.append((func, args, kargs)) @@ -238,13 +239,17 @@ async def run_coroutine_exitfuncs(): """ This is the same as run_exitfuncs but it uses asyncio.iscoroutinefunction to check which functions to run. It is called by the AsyncioEventLoop -_close_main method just before the loop is closed. +_close method just before the loop is closed. If the loop is explicitly closed before exit, then that will cause run_coroutine_exitfuncs to run before run_exitfuncs. Otherwise, a run_exitfuncs hook will close it, causing run_coroutine_exitfuncs to be called via run_exitfuncs. """ +# The _thread_weakrefs_atexit function makes an adjustment to ensure +# that global_event_loop() returns the correct loop when it is closing, +# regardless of which thread the loop was initially associated with. +_coroutine_exithandlers = global_event_loop()._coroutine_exithandlers tasks = [] while _coroutine_exithandlers: func, targs, kargs = _coroutine_exithandlers.pop() diff --git a/lib/portage/tests/util/test_socks5.py b/lib/portage/tests/util/test_socks5.py index 35f919d970..078e3b1a23 100644 --- a/lib/portage/tests/util/test_socks5.py +++ b/lib/portage/tests/util/test_socks5.py @@ -194,17 +194,17 @@ class Socks5ServerTestCase(TestCase): asyncio.run(self._test_socks5_proxy()) async def _test_socks5_proxy(self): -loop = asyncio.get_running_loop() +loop = global_event_loop() host = "127.0.0.1" content = b"Hello World!" path = "/index.html" proxy = None tempdir = tempfile.mkdtemp() -previous_exithandlers = portage.process._coroutine_exithandlers +previous_exithandlers = loop._coroutine_exithandlers try: -portage.process._coroutine_exithandlers = [] +loop._coroutine_exithandlers = [] with AsyncHTTPServer(host, {path: content}, loop) as server: settings = { "PORTAGE_TMPDIR": tempdir, @@ -227,11 +227,11 @@ class Socks5ServerTestCase(TestCase): finally: try: # Also run_coroutine_exitfuncs to test atexit hook cleanup. -self.assertNotEqual(portage.process._coroutine_exithandlers, []) +self.assertNotEqual(loop._coroutine_exithandlers, []) await portage.process.run_coroutine_exitfuncs() -self.assertEqual(portage.process._coroutine_exithandlers, []) +self.assertEqual(loop._coroutine_exithandlers, []) finally: -portage.process._coroutine_exithandlers = previous_exithandlers +loop._coroutine_exithandlers = previous_exithandlers shutil.rmtree(tempdir) @@ -284,6 +284,8 @@ class Socks5ServerAtExitTestCase(TestCase): so this test uses python -c to ensure that atexit hooks will work. """ +_threaded = False +
[gentoo-portage-dev] [PATCH] run_exitfuncs: Support loop close via hook
Handle the case where the loop has not been explicitly closed before exit. In this case, run_exitfuncs previously tried to call coroutine functions as though they were normal functions, which obvously would not behave correctly. Solve this problem by storing the coroutine functions in a separate _coroutine_exithandlers list that belongs exclusively to the run_coroutine_exitfuncs function, so that it is safe to close the loop and call run_coroutine_exitfuncs from inside a run_exitfuncs hook. A _thread_weakrefs_atexit hook already exists that will close weakly referenced loops. The _thread_weakrefs_atexit hook is now fixed to release its lock when it closes a loop, since the same lock may need to be re-acquired when run_coroutine_exitfuncs runs. The included test case demonstrates that run_exitfuncs will run via an atexit hook and correctly terminate the socks5 proxy in a standalone program using the portage API (like eclean). Due to a deadlock that will occur if an _exit_function atexit hook from the multiprocessing module executes before run_exitfuncs, a portage.process._atexit_register_run_exitfuncs() function needs to be called in order to re-order the hooks after the first process has been started via the multiprocessing module. The natural place to call this is in the ForkProcess class, using a global variable to trigger the call just once. Fixes: c3ebdbb42e72 ("elog/mod_custom: Spawn processes in background") Bug: https://bugs.gentoo.org/937384 Closes: https://github.com/gentoo/portage/pull/1366 Signed-off-by: Zac Medico --- lib/portage/process.py| 37 +-- lib/portage/tests/__init__.py | 19 ++ lib/portage/tests/util/test_socks5.py | 66 +-- lib/portage/util/_async/ForkProcess.py| 8 +++ lib/portage/util/futures/_asyncio/__init__.py | 24 --- 5 files changed, 133 insertions(+), 21 deletions(-) diff --git a/lib/portage/process.py b/lib/portage/process.py index 6e4e0d7162..23e2507b53 100644 --- a/lib/portage/process.py +++ b/lib/portage/process.py @@ -3,6 +3,7 @@ # Distributed under the terms of the GNU General Public License v2 +import asyncio as _asyncio import atexit import errno import fcntl @@ -193,6 +194,7 @@ def spawn_fakeroot(mycommand, fakeroot_state=None, opt_name=None, **keywords): _exithandlers = [] +_coroutine_exithandlers = [] def atexit_register(func, *args, **kargs): @@ -200,7 +202,12 @@ def atexit_register(func, *args, **kargs): what is registered. For example, when portage restarts itself via os.execv, the atexit module does not work so we have to do it manually by calling the run_exitfuncs() function in this module.""" -_exithandlers.append((func, args, kargs)) +# The internal asyncio wrapper module would trigger a circular import +# if used here. +if _asyncio.iscoroutinefunction(func): +_coroutine_exithandlers.append((func, args, kargs)) +else: +_exithandlers.append((func, args, kargs)) def run_exitfuncs(): @@ -232,12 +239,16 @@ async def run_coroutine_exitfuncs(): This is the same as run_exitfuncs but it uses asyncio.iscoroutinefunction to check which functions to run. It is called by the AsyncioEventLoop _close_main method just before the loop is closed. + +If the loop is explicitly closed before exit, then that will cause +run_coroutine_exitfuncs to run before run_exitfuncs. Otherwise, a +run_exitfuncs hook will close it, causing run_coroutine_exitfuncs to be +called via run_exitfuncs. """ tasks = [] -for index, (func, targs, kargs) in reversed(list(enumerate(_exithandlers))): -if asyncio.iscoroutinefunction(func): -del _exithandlers[index] -tasks.append(asyncio.ensure_future(func(*targs, **kargs))) +while _coroutine_exithandlers: +func, targs, kargs = _coroutine_exithandlers.pop() +tasks.append(asyncio.ensure_future(func(*targs, **kargs))) tracebacks = [] exc_info = None for task in tasks: @@ -255,7 +266,21 @@ async def run_coroutine_exitfuncs(): raise exc_info[1].with_traceback(exc_info[2]) -atexit.register(run_exitfuncs) +def _atexit_register_run_exitfuncs(): +""" +Register the run_exitfuncs atexit hook. If this hook is not called +before the multiprocessing module's _exit_function, then there will +be a deadlock. In order to prevent the deadlock, this function must +be called in order to re-order the hooks after the first process has +been started via the multiprocessing module. The natural place to +call this is in the ForkProcess class, though it should also be +called once before, in case the ForkProcess class is never called. +""" +atexit.unregister(run_exitfuncs) +atexit.register(run_exitfuncs) + + +_atexit_register_run_exitfuncs() # It used to be necessary for API consumers to remove pids from spawned_pids, # since otherwise it wou
[gentoo-portage-dev] [PATCH v2] Pass through MAKEFLAGS and exclude from environment.bz2
Allow the MAKEFLAGS environment variable to pass through, in case a centralized GNU Make POSIX Jobserver is available. In order to prevent persistence of this variable in environment.bz2, exclude it when the __save_ebuild_env --exclude-init-phases argument is given. Ultimately we may want to add support for portage to parse MAKEFLAGS and use it to allocate job tokens in various circumstances. For example, emerge could allocate a job token for each job started for emerge --jobs. This would remove a job token from the pool that is available to nested make calls, but is reasonable because nested make calls will execute jobs serially when no jobserver tokens remain. Bug: https://bugs.gentoo.org/692576 Signed-off-by: Zac Medico --- [PATCH v2] documents MAKEFLAGS in make.conf.5 and disables an automatic MAKEOPTS setting if it would conflict with MAKEFLAGS. bin/save-ebuild-env.sh | 5 - lib/portage/__init__.py| 3 ++- lib/portage/package/ebuild/_config/special_env_vars.py | 3 ++- lib/portage/package/ebuild/doebuild.py | 3 ++- man/make.conf.5| 10 +- 5 files changed, 19 insertions(+), 5 deletions(-) diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh index 3a2560aabf..6943e59b0b 100644 --- a/bin/save-ebuild-env.sh +++ b/bin/save-ebuild-env.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Copyright 1999-2021 Gentoo Authors +# Copyright 1999-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # @FUNCTION: __save_ebuild_env @@ -24,6 +24,9 @@ __save_ebuild_env() { unset PYTHONPATH fi fi + + # Discard stale GNU Make POSIX Jobserver flags. + unset MAKEFLAGS fi # misc variables inherited from the calling environment diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py index 21bf993170..10c303477f 100644 --- a/lib/portage/__init__.py +++ b/lib/portage/__init__.py @@ -1,4 +1,4 @@ -# Copyright 1998-2023 Gentoo Authors +# Copyright 1998-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # pylint: disable=ungrouped-imports @@ -657,6 +657,7 @@ def create_trees( # environment to apply to the config that's associated # with ROOT != "/", so pass a nearly empty dict for the env parameter. env_sequence = ( +"MAKEFLAGS", "PATH", "PORTAGE_GRPNAME", "PORTAGE_REPOSITORIES", diff --git a/lib/portage/package/ebuild/_config/special_env_vars.py b/lib/portage/package/ebuild/_config/special_env_vars.py index 1a66192c96..55e5111ef0 100644 --- a/lib/portage/package/ebuild/_config/special_env_vars.py +++ b/lib/portage/package/ebuild/_config/special_env_vars.py @@ -1,4 +1,4 @@ -# Copyright 2010-2021 Gentoo Authors +# Copyright 2010-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 __all__ = ( @@ -112,6 +112,7 @@ environ_whitelist = frozenset( "FEATURES", "FILESDIR", "HOME", +"MAKEFLAGS", "MERGE_TYPE", "NOCOLOR", "NO_COLOR", diff --git a/lib/portage/package/ebuild/doebuild.py b/lib/portage/package/ebuild/doebuild.py index 403836b80b..bb888e7ae4 100644 --- a/lib/portage/package/ebuild/doebuild.py +++ b/lib/portage/package/ebuild/doebuild.py @@ -598,7 +598,8 @@ def doebuild_environment( ) mysettings.features.remove(feature) -if "MAKEOPTS" not in mysettings: +# MAKEOPTS conflicts with MAKEFLAGS, so skip this if MAKEFLAGS exists. +if "MAKEOPTS" not in mysettings and "MAKEFLAGS" not in mysettings: nproc = get_cpu_count() if nproc: mysettings["MAKEOPTS"] = "-j%d" % (nproc) diff --git a/man/make.conf.5 b/man/make.conf.5 index 21ae09e574..9af563a5c8 100644 --- a/man/make.conf.5 +++ b/man/make.conf.5 @@ -1,4 +1,4 @@ -.TH "MAKE.CONF" "5" "Jan 2024" "Portage @VERSION@" "Portage" +.TH "MAKE.CONF" "5" "Aug 2024" "Portage @VERSION@" "Portage" .SH "NAME" make.conf \- custom settings for Portage .SH "SYNOPSIS" @@ -948,6 +948,14 @@ Setting this and other *FLAGS variables arbitrarily may cause compile or runtime failures. Bug reports submitted when nonstandard values are enabled for these flags may be closed as INVALID. .TP +.B MAKEFLAGS +Use this variable instead of \fBMAKEOPTS\fR if you want to inject a +centralized job server for make. In this case \fBMAKEOPTS\fR should be +unset or else it can cause the make jobserver mode to reset. Include +a \fB--jobserver-auth=fifo:PATH\fR flag to specify the path of the +centralized jobserver fifo, which needs to be readable and writable by +the portage group when userpriv is enabled. +.TP .B MAKEOPTS Use this variable if you want to use parallel make. For example, if you hav
[gentoo-portage-dev] [PATCH] Pass through MAKEFLAGS and exclude from environment.bz2
Allow the MAKEFLAGS environment variable to pass through, in case a centralized GNU Make POSIX Jobserver is available. In order to prevent persistence of this variable in environment.bz2, exclude it when the __save_ebuild_env --exclude-init-phases argument is given. Ultimately we may want to add support for portage to parse MAKEFLAGS and use it to allocate job tokens in various circumstances. For example, emerge could allocate a job token for each job started for emerge --jobs. This would remove a job token from the pool that is available to nested make calls, but is reasonable because nested make calls will execute jobs serially when no jobserver tokens remain. Bug: https://bugs.gentoo.org/692576 Closes: https://github.com/gentoo/portage/pull/1364 Signed-off-by: Zac Medico --- bin/save-ebuild-env.sh | 5 - lib/portage/__init__.py| 3 ++- lib/portage/package/ebuild/_config/special_env_vars.py | 3 ++- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh index 3a2560aabf..6943e59b0b 100644 --- a/bin/save-ebuild-env.sh +++ b/bin/save-ebuild-env.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Copyright 1999-2021 Gentoo Authors +# Copyright 1999-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # @FUNCTION: __save_ebuild_env @@ -24,6 +24,9 @@ __save_ebuild_env() { unset PYTHONPATH fi fi + + # Discard stale GNU Make POSIX Jobserver flags. + unset MAKEFLAGS fi # misc variables inherited from the calling environment diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py index 21bf993170..10c303477f 100644 --- a/lib/portage/__init__.py +++ b/lib/portage/__init__.py @@ -1,4 +1,4 @@ -# Copyright 1998-2023 Gentoo Authors +# Copyright 1998-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # pylint: disable=ungrouped-imports @@ -657,6 +657,7 @@ def create_trees( # environment to apply to the config that's associated # with ROOT != "/", so pass a nearly empty dict for the env parameter. env_sequence = ( +"MAKEFLAGS", "PATH", "PORTAGE_GRPNAME", "PORTAGE_REPOSITORIES", diff --git a/lib/portage/package/ebuild/_config/special_env_vars.py b/lib/portage/package/ebuild/_config/special_env_vars.py index 1a66192c96..55e5111ef0 100644 --- a/lib/portage/package/ebuild/_config/special_env_vars.py +++ b/lib/portage/package/ebuild/_config/special_env_vars.py @@ -1,4 +1,4 @@ -# Copyright 2010-2021 Gentoo Authors +# Copyright 2010-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 __all__ = ( @@ -112,6 +112,7 @@ environ_whitelist = frozenset( "FEATURES", "FILESDIR", "HOME", +"MAKEFLAGS", "MERGE_TYPE", "NOCOLOR", "NO_COLOR", -- 2.44.2
[gentoo-portage-dev] Re: [PATCH v2] _EbuildFetcherProcess: Handle SIGTERM
On 7/28/24 18:35, Zac Medico wrote: @@ -201,20 +226,40 @@ def _userpriv_test_write_file(settings, file_path): if rval is not None: return rval +# Optimize away the spawn when privileges do not need to be dropped. +if not ( +"userfetch" in settings.features +and os.getuid() == 0 +and portage_gid +and portage_uid +and hasattr(os, "setgroups") +): +rval = os.access(os.path.dirname(file_path), os.W_OK) +_userpriv_test_write_file_cache[file_path] = rval +return rval + +# Optimize away the spawn if we can detect a portage group write +# permission bit, but if this optimization fails then continue with +# the spawn for ACL support. +st = os.stat(os.path.dirname(file_path)) +if st.st_gid == int(portage_gid) and stat.S_IMODE(st.st_mode) & 0o020: +_userpriv_test_write_file_cache[file_path] = True +return True + It turned out that all of the above could be reduced to this because os.access works in any case that _want_userfetch returns False: # Optimize away the spawn when privileges do not need to be dropped. if not _want_userfetch(settings): rval = os.access(os.path.dirname(file_path), os.W_OK) _userpriv_test_write_file_cache[file_path] = rval return rval -- Thanks, Zac OpenPGP_signature.asc Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH v2] _EbuildFetcherProcess: Handle SIGTERM
Fix _EbuildFetcherProcess to handle SIGTERM, so that FETCHCOMMAND processes will not be left running in the background: * Convert the fetch function to an async_fetch coroutine function so that it can use asyncio.CancelledError handlers to terminate running processes. * Use multiprocessing.active_children() to detect and terminate any processes that asyncio.CancelledError handlers did not have an opportunity to terminate because the exception arrived too soon after fork/spawn. * Add unit test to verify that a child process is correctly killed when EbuildFetcher is cancelled, with short timeout in case it takes some time for the process to disappear. Bug: https://bugs.gentoo.org/936273 Signed-off-by: Zac Medico --- Changes in [PATCH v2]: * Use multiple multiprocessing.active_children() loops to allow some more time for children to respond to SIGTERM. * Fix event loop recursion in _userpriv_test_write_file. * Optimize away the spawn in _userpriv_test_write_file if a portage group write permission bit is detected. lib/_emerge/EbuildFetcher.py | 68 --- lib/portage/package/ebuild/fetch.py| 116 ++--- lib/portage/tests/ebuild/test_fetch.py | 100 - lib/portage/tests/util/test_socks5.py | 16 +++- 4 files changed, 268 insertions(+), 32 deletions(-) diff --git a/lib/_emerge/EbuildFetcher.py b/lib/_emerge/EbuildFetcher.py index 81d4b1054b..994271236c 100644 --- a/lib/_emerge/EbuildFetcher.py +++ b/lib/_emerge/EbuildFetcher.py @@ -4,6 +4,8 @@ import copy import functools import io +import multiprocessing +import signal import sys import portage @@ -17,11 +19,12 @@ from portage.package.ebuild.fetch import ( _check_distfile, _drop_privs_userfetch, _want_userfetch, -fetch, +async_fetch, ) from portage.util._async.AsyncTaskFuture import AsyncTaskFuture from portage.util._async.ForkProcess import ForkProcess from portage.util._pty import _create_pty_or_pipe +from portage.util.futures import asyncio from _emerge.CompositeTask import CompositeTask @@ -34,6 +37,7 @@ class EbuildFetcher(CompositeTask): "logfile", "pkg", "prefetch", +"pre_exec", "_fetcher_proc", ) @@ -253,6 +257,7 @@ class _EbuildFetcherProcess(ForkProcess): self._get_manifest(), self._uri_map, self.fetchonly, +self.pre_exec, ) ForkProcess._start(self) @@ -263,7 +268,10 @@ class _EbuildFetcherProcess(ForkProcess): self._settings = None @staticmethod -def _target(settings, manifest, uri_map, fetchonly): +def _target(settings, manifest, uri_map, fetchonly, pre_exec): +if pre_exec is not None: +pre_exec() + # Force consistent color output, in case we are capturing fetch # output through a normal pipe due to unavailability of ptys. portage.output.havecolor = settings.get("NOCOLOR") not in ("yes", "true") @@ -273,17 +281,53 @@ class _EbuildFetcherProcess(ForkProcess): if _want_userfetch(settings): _drop_privs_userfetch(settings) -rval = 1 allow_missing = manifest.allow_missing or "digest" in settings.features -if fetch( -uri_map, -settings, -fetchonly=fetchonly, -digests=copy.deepcopy(manifest.getTypeDigests("DIST")), -allow_missing_digests=allow_missing, -): -rval = os.EX_OK -return rval + +async def main(): +loop = asyncio.get_event_loop() +task = asyncio.ensure_future( +async_fetch( +uri_map, +settings, +fetchonly=fetchonly, +digests=copy.deepcopy(manifest.getTypeDigests("DIST")), +allow_missing_digests=allow_missing, +) +) + +def sigterm_handler(signum, _frame): +loop.call_soon_threadsafe(task.cancel) +signal.signal(signal.SIGTERM, signal.SIG_IGN) + +signal.signal(signal.SIGTERM, sigterm_handler) +try: +await task +except asyncio.CancelledError: +# If asyncio.CancelledError arrives too soon after fork/spawn +# then handers will not have an opportunity to terminate +# the corresponding process, so clean up after this race. +for proc in multiprocessing.active_children(): +proc.terminate() + +# Use a non-zero timeout only for the first join because +# later joins are delayed by the first join. +timeout = 0.25 +for proc in multiprocessing.active_children(): +proc.join(timeout) +timeout = 0 + +for proc in multiprocessing.active_children():
Re: [gentoo-portage-dev] [PATCH] _EbuildFetcherProcess: Handle SIGTERM
On 7/22/24 23:40, Joey Pabalinas wrote: On Mon, Jul 22, 2024 at 09:40:48PM GMT, Zac Medico wrote: Fix _EbuildFetcherProcess to handle SIGTERM, so that FETCHCOMMAND processes will not be left running in the background: * Convert the fetch function to an async_fetch coroutine function so that it can use asyncio.CancelledError handlers to terminate running processes. * Use multiprocessing.active_children() to detect and terminate any processes that asyncio.CancelledError handlers did not have an opportunity to terminate because the exception arrived too soon after fork/spawn. * Add unit test to verify that a child process is correctly killed when EbuildFetcher is cancelled, with short timeout in case it takes some time for the process to disappear. Bug: https://bugs.gentoo.org/936273 Closes: https://github.com/gentoo/portage/pull/1361 Signed-off-by: Zac Medico Looks good. Reviewed-by: Joey Pabalinas Thanks! There's now an update in https://github.com/gentoo/portage/pull/1361 to use multiple multiprocessing.active_children() loops in order to allow the children a little more time to terminate. -- Thanks, Zac OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] _EbuildFetcherProcess: Handle SIGTERM
On Mon, Jul 22, 2024 at 09:40:48PM GMT, Zac Medico wrote: > Fix _EbuildFetcherProcess to handle SIGTERM, so that FETCHCOMMAND > processes will not be left running in the background: > > * Convert the fetch function to an async_fetch coroutine function > so that it can use asyncio.CancelledError handlers to terminate > running processes. > > * Use multiprocessing.active_children() to detect and terminate > any processes that asyncio.CancelledError handlers did not have > an opportunity to terminate because the exception arrived too > soon after fork/spawn. > > * Add unit test to verify that a child process is correctly > killed when EbuildFetcher is cancelled, with short timeout in > case it takes some time for the process to disappear. > > Bug: https://bugs.gentoo.org/936273 > Closes: https://github.com/gentoo/portage/pull/1361 > Signed-off-by: Zac Medico Looks good. Reviewed-by: Joey Pabalinas -- Cheers, Joey Pabalinas signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] _EbuildFetcherProcess: Handle SIGTERM
Fix _EbuildFetcherProcess to handle SIGTERM, so that FETCHCOMMAND processes will not be left running in the background: * Convert the fetch function to an async_fetch coroutine function so that it can use asyncio.CancelledError handlers to terminate running processes. * Use multiprocessing.active_children() to detect and terminate any processes that asyncio.CancelledError handlers did not have an opportunity to terminate because the exception arrived too soon after fork/spawn. * Add unit test to verify that a child process is correctly killed when EbuildFetcher is cancelled, with short timeout in case it takes some time for the process to disappear. Bug: https://bugs.gentoo.org/936273 Closes: https://github.com/gentoo/portage/pull/1361 Signed-off-by: Zac Medico --- lib/_emerge/EbuildFetcher.py | 58 +++--- lib/portage/package/ebuild/fetch.py| 97 ++-- lib/portage/tests/ebuild/test_fetch.py | 100 - lib/portage/tests/util/test_socks5.py | 16 +++- 4 files changed, 245 insertions(+), 26 deletions(-) diff --git a/lib/_emerge/EbuildFetcher.py b/lib/_emerge/EbuildFetcher.py index 81d4b1054b..ece33dd630 100644 --- a/lib/_emerge/EbuildFetcher.py +++ b/lib/_emerge/EbuildFetcher.py @@ -4,6 +4,8 @@ import copy import functools import io +import multiprocessing +import signal import sys import portage @@ -17,11 +19,12 @@ from portage.package.ebuild.fetch import ( _check_distfile, _drop_privs_userfetch, _want_userfetch, -fetch, +async_fetch, ) from portage.util._async.AsyncTaskFuture import AsyncTaskFuture from portage.util._async.ForkProcess import ForkProcess from portage.util._pty import _create_pty_or_pipe +from portage.util.futures import asyncio from _emerge.CompositeTask import CompositeTask @@ -34,6 +37,7 @@ class EbuildFetcher(CompositeTask): "logfile", "pkg", "prefetch", +"pre_exec", "_fetcher_proc", ) @@ -253,6 +257,7 @@ class _EbuildFetcherProcess(ForkProcess): self._get_manifest(), self._uri_map, self.fetchonly, +self.pre_exec, ) ForkProcess._start(self) @@ -263,7 +268,10 @@ class _EbuildFetcherProcess(ForkProcess): self._settings = None @staticmethod -def _target(settings, manifest, uri_map, fetchonly): +def _target(settings, manifest, uri_map, fetchonly, pre_exec): +if pre_exec is not None: +pre_exec() + # Force consistent color output, in case we are capturing fetch # output through a normal pipe due to unavailability of ptys. portage.output.havecolor = settings.get("NOCOLOR") not in ("yes", "true") @@ -273,17 +281,43 @@ class _EbuildFetcherProcess(ForkProcess): if _want_userfetch(settings): _drop_privs_userfetch(settings) -rval = 1 allow_missing = manifest.allow_missing or "digest" in settings.features -if fetch( -uri_map, -settings, -fetchonly=fetchonly, -digests=copy.deepcopy(manifest.getTypeDigests("DIST")), -allow_missing_digests=allow_missing, -): -rval = os.EX_OK -return rval + +async def main(): +loop = asyncio.get_event_loop() +task = asyncio.ensure_future( +async_fetch( +uri_map, +settings, +fetchonly=fetchonly, +digests=copy.deepcopy(manifest.getTypeDigests("DIST")), +allow_missing_digests=allow_missing, +) +) + +def sigterm_handler(signum, _frame): +loop.call_soon_threadsafe(task.cancel) +signal.signal(signal.SIGTERM, signal.SIG_IGN) + +signal.signal(signal.SIGTERM, sigterm_handler) +try: +await task +except asyncio.CancelledError: +# If asyncio.CancelledError arrives too soon after fork/spawn +# then handers will not have an opportunity to termintate +# the corresponding process, so clean up after this race. +for proc in multiprocessing.active_children(): +proc.terminate() +proc.join(0) +if proc.exitcode is None: +proc.kill() +proc.join(0) +signal.signal(signal.SIGTERM, signal.SIG_DFL) +os.kill(os.getpid(), signal.SIGTERM) + +return os.EX_OK if task.result() else 1 + +return asyncio.run(main()) def _get_ebuild_path(self): if self.ebuild_path is not None: diff --git a/lib/portage/package/ebuild/fetch.py b/lib/portage/package/ebuild/fetch.py index ed40cf6ede..124b2b7056 100644 --- a/lib/portage/package/ebuild/fetch.py +++ b/lib/portage/
Re: [gentoo-portage-dev] [PATCH 0/2] Minor documentation fixups
Bryan Gardiner writes: > Just fixes for a couple of small documentation issues I found. Applied - thanks! > > Bryan Gardiner (2): > depgraph.py: Fix the URL for the changed-deps wiki page > ebuild.5: Escape carats which are special to groff > > lib/_emerge/depgraph.py | 2 +- > man/ebuild.5| 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-)
[gentoo-portage-dev] [PATCH 1/2] depgraph.py: Fix the URL for the changed-deps wiki page
Signed-off-by: Bryan Gardiner --- lib/_emerge/depgraph.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py index 6853ec491..9673d85f8 100644 --- a/lib/_emerge/depgraph.py +++ b/lib/_emerge/depgraph.py @@ -1181,7 +1181,7 @@ class depgraph: "NOTE: Refer to the following page for more information about dependency", " change(s) without revision bump:", "", -" https://wiki.gentoo.org/wiki/Project:Portage/Changed_Deps";, +" https://wiki.gentoo.org/wiki/Project:Portage/Changed_dependencies";, "", " In order to suppress reports about dependency changes, add", " --changed-deps-report=n to the EMERGE_DEFAULT_OPTS variable in", -- 2.43.2
[gentoo-portage-dev] [PATCH 2/2] ebuild.5: Escape carats which are special to groff
Raw carats are converted to circumflex characters which are not searchable, so they need to be escaped. See groff_char(7). Signed-off-by: Bryan Gardiner --- man/ebuild.5 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/man/ebuild.5 b/man/ebuild.5 index a32ba4828..36d65eb18 100644 --- a/man/ebuild.5 +++ b/man/ebuild.5 @@ -734,7 +734,7 @@ If flag1 enabled then flag2 enabled flag1? ( flag2 ) If flag1 disabled then flag2 enabled !flag1? ( flag2 ) If flag1 disabled then flag2 disabled !flag1? ( !flag2 ) Must enable any one or more (inclusive or) || ( flag1 flag2 flag3 ) -Must enable exactly one but not more (exclusive or)^^ ( flag1 flag2 flag3 ) +Must enable exactly one but not more (exclusive or)\(ha\(ha ( flag1 flag2 flag3 ) May enable at most one (EAPI 5 or later) ?? ( flag1 flag2 flag3 ) .TE .TP -- 2.43.2
[gentoo-portage-dev] [PATCH 0/2] Minor documentation fixups
Just fixes for a couple of small documentation issues I found. Bryan Gardiner (2): depgraph.py: Fix the URL for the changed-deps wiki page ebuild.5: Escape carats which are special to groff lib/_emerge/depgraph.py | 2 +- man/ebuild.5| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- 2.43.2
[gentoo-portage-dev] [PATCH 1/2] Add get_repo_revision_history function and repo_revisions file
The history of synced revisions is provided by a new get_repo_revision_history function and corresponding /var/lib/portage/repo_revisions file, with history limit currently capped at 25 revisions. If a change is detected and the current process has permission to update the repo_revisions file, then the file will be updated with any newly detected revisions. For volatile repos the revisions may be unordered, which makes them unusable for the purposes of the revision history, so the revisions of volatile repos are not tracked. This functions detects revisions which are not yet visible to the current process due to the sync-rcu option. The emaint revisions --purgerepos and --purgeallrepos options allow revisions for some or all repos to be easily purged from the history. For example, the emerge-webrsync script uses this emaint commmand to purge the revision history of the gentoo repo when the emerge-webrsync --revert option is used to roll back to a previous snapshot: emaint revisions --purgerepos="${repo_name}" Bug: https://bugs.gentoo.org/924772 Signed-off-by: Zac Medico --- bin/emerge-webrsync | 3 +- lib/portage/const.py | 1 + lib/portage/emaint/modules/meson.build| 1 + .../emaint/modules/revisions/__init__.py | 36 + .../emaint/modules/revisions/meson.build | 8 ++ .../emaint/modules/revisions/revisions.py | 95 + lib/portage/sync/controller.py| 8 +- lib/portage/sync/meson.build | 1 + lib/portage/sync/revision_history.py | 133 ++ lib/portage/tests/sync/test_sync_local.py | 75 +- man/emaint.1 | 18 ++- man/portage.5 | 17 ++- 12 files changed, 388 insertions(+), 8 deletions(-) create mode 100644 lib/portage/emaint/modules/revisions/__init__.py create mode 100644 lib/portage/emaint/modules/revisions/meson.build create mode 100644 lib/portage/emaint/modules/revisions/revisions.py create mode 100644 lib/portage/sync/revision_history.py diff --git a/bin/emerge-webrsync b/bin/emerge-webrsync index 99da05543a..caa4986da2 100755 --- a/bin/emerge-webrsync +++ b/bin/emerge-webrsync @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Copyright 1999-2023 Gentoo Authors +# Copyright 1999-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # Author: Karl Trygve Kalleberg # Rewritten from the old, Perl-based emerge-webrsync script @@ -732,6 +732,7 @@ main() { [[ ${do_debug} -eq 1 ]] && set -x if [[ -n ${revert_date} ]] ; then + emaint revisions --purgerepos="${repo_name}" do_snapshot 1 "${revert_date}" else do_latest_snapshot diff --git a/lib/portage/const.py b/lib/portage/const.py index 2154213b7b..c9a71009a7 100644 --- a/lib/portage/const.py +++ b/lib/portage/const.py @@ -51,6 +51,7 @@ PRIVATE_PATH = "var/lib/portage" WORLD_FILE = f"{PRIVATE_PATH}/world" WORLD_SETS_FILE = f"{PRIVATE_PATH}/world_sets" CONFIG_MEMORY_FILE = f"{PRIVATE_PATH}/config" +REPO_REVISIONS = f"{PRIVATE_PATH}/repo_revisions" NEWS_LIB_PATH = "var/lib/gentoo" # these variables get EPREFIX prepended automagically when they are diff --git a/lib/portage/emaint/modules/meson.build b/lib/portage/emaint/modules/meson.build index 48f4f77d83..33b396be94 100644 --- a/lib/portage/emaint/modules/meson.build +++ b/lib/portage/emaint/modules/meson.build @@ -12,5 +12,6 @@ subdir('logs') subdir('merges') subdir('move') subdir('resume') +subdir('revisions') subdir('sync') subdir('world') diff --git a/lib/portage/emaint/modules/revisions/__init__.py b/lib/portage/emaint/modules/revisions/__init__.py new file mode 100644 index 00..c51cbb4bf3 --- /dev/null +++ b/lib/portage/emaint/modules/revisions/__init__.py @@ -0,0 +1,36 @@ +# Copyright 2024 Gentoo Authors +# Distributed under the terms of the GNU General Public License v2 + +doc = """Purge repo_revisions history file.""" +__doc__ = doc + + +module_spec = { +"name": "revisions", +"description": doc, +"provides": { +"purgerevisions": { +"name": "revisions", +"sourcefile": "revisions", +"class": "PurgeRevisions", +"description": "Purge repo_revisions history", +"functions": ["purgeallrepos", "purgerepos"], +"func_desc": { +"repo": { +"long": "--purgerepos", +"help": "(revisions module only): --purgerepos Purge revisions for the specified repo(s)", +"status": "Purging %s", +"action": "store", +"func": "purgerepos", +}, +"allrepos": { +"long": "--purgeallrepos", +"help": "(revisions module only): --purgeallrepos Purge revisions for all repos", +
[gentoo-portage-dev] [PATCH 2/2] bintree: Add REPO_REVISIONS to package index header
As a means for binhost clients to select source repo revisions which are consistent with binhosts, inject REPO_REVISIONS from a package into the index header, using a history of synced revisions to guarantee forward progress. This queries the relevant repos to check if any new revisions have appeared in the absence of a proper sync operation. Bug: https://bugs.gentoo.org/924772 Signed-off-by: Zac Medico --- lib/portage/dbapi/bintree.py | 66 - lib/portage/tests/sync/test_sync_local.py | 71 +++ 2 files changed, 123 insertions(+), 14 deletions(-) diff --git a/lib/portage/dbapi/bintree.py b/lib/portage/dbapi/bintree.py index 7bc1f60f6d..fbf60e74eb 100644 --- a/lib/portage/dbapi/bintree.py +++ b/lib/portage/dbapi/bintree.py @@ -48,6 +48,7 @@ from portage.exception import ( from portage.localization import _ from portage.output import colorize from portage.package.ebuild.profile_iuse import iter_iuse_vars +from portage.sync.revision_history import get_repo_revision_history from portage.util import ensure_dirs from portage.util.file_copy import copyfile from portage.util.futures import asyncio @@ -62,6 +63,7 @@ from portage import _unicode_encode import codecs import errno import io +import json import re import stat import subprocess @@ -134,13 +136,19 @@ class bindbapi(fakedbapi): "USE", "_mtime_", } +# Keys required only when initially adding a package. +self._init_aux_keys = { +"REPO_REVISIONS", +} self._aux_cache = {} self._aux_cache_slot_dict_cache = None @property def _aux_cache_slot_dict(self): if self._aux_cache_slot_dict_cache is None: -self._aux_cache_slot_dict_cache = slot_dict_class(self._aux_cache_keys) +self._aux_cache_slot_dict_cache = slot_dict_class( +chain(self._aux_cache_keys, self._init_aux_keys) +) return self._aux_cache_slot_dict_cache def __getstate__(self): @@ -1791,6 +1799,10 @@ class binarytree: pkgindex = self._new_pkgindex() d = self._inject_file(pkgindex, cpv, full_path) +repo_revisions = d.get("REPO_REVISIONS") +if repo_revisions: +repo_revisions = json.loads(repo_revisions) +self._inject_repo_revisions(pkgindex.header, repo_revisions) self._update_pkgindex_header(pkgindex.header) self._pkgindex_write(pkgindex) @@ -1872,7 +1884,7 @@ class binarytree: @return: package metadata """ if keys is None: -keys = self.dbapi._aux_cache_keys +keys = chain(self.dbapi._aux_cache_keys, self.dbapi._init_aux_keys) metadata = self.dbapi._aux_cache_slot_dict() else: metadata = {} @@ -1916,6 +1928,56 @@ class binarytree: return metadata +def _inject_repo_revisions(self, header, repo_revisions): +""" +Inject REPO_REVISIONS from a package into the index header, +using a history of synced revisions to guarantee forward +progress. This queries the relevant repos to check if any +new revisions have appeared in the absence of a proper sync +operation. + +This does not expose REPO_REVISIONS that do not appear in +the sync history, since such revisions suggest that the +package was not built locally, and in this case its +REPO_REVISIONS are not intended to be exposed. +""" +synced_repo_revisions = get_repo_revision_history( +self.settings["EROOT"], +[self.settings.repositories[repo_name] for repo_name in repo_revisions], +) +header_repo_revisions = ( +json.loads(header["REPO_REVISIONS"]) if header.get("REPO_REVISIONS") else {} +) +for repo_name, repo_revision in repo_revisions.items(): +rev_list = synced_repo_revisions.get(repo_name, []) +header_rev = header_repo_revisions.get(repo_name) +if not rev_list or header_rev in (repo_revision, rev_list[0]): +continue +try: +header_rev_index = ( +None if header_rev is None else rev_list.index(header_rev) +) +except ValueError: +header_rev_index = None +try: +repo_revision_index = rev_list.index(repo_revision) +except ValueError: +repo_revision_index = None +if repo_revision_index is not None and ( +header_rev_index is None or repo_revision_index < header_rev_index +): +# There is forward progress when repo_revision is more recent +# than header_rev or header_rev was not found in the history. +# Do not expose repo_revision here if it does not appear in +
[gentoo-portage-dev] [PATCH 0/2] Add REPO_REVISIONS to package index header
As a means for binhost clients to select source repo revisions which are consistent with binhosts, inject REPO_REVISIONS from a package into the index header, using a history of synced revisions to guarantee forward progress. This queries the relevant repos to check if any new revisions have appeared in the absence of a proper sync operation. The history of synced revisions is provided by a new get_repo_revision_history function and corresponding /var/lib/portage/repo_revisions file, with history limit currently capped at 25 revisions. If a change is detected and the current process has permission to update the repo_revisions file, then the file will be updated with any newly detected revisions. For volatile repos the revisions may be unordered, which makes them unusable for the purposes of the revision history, so the revisions of volatile repos are not tracked. This functions detects revisions which are not yet visible to the current process due to the sync-rcu option. The emaint revisions --purgerepos and --purgeallrepos options allow revisions for some or all repos to be easily purged from the history. For example, the emerge-webrsync script uses this emaint commmand to purge the revision history of the gentoo repo when the emerge-webrsync --revert option is used to roll back to a previous snapshot: emaint revisions --purgerepos="${repo_name}" This series can also be reviewed at https://github.com/gentoo/portage/pull/1306. Bug: https://bugs.gentoo.org/924772 Signed-off-by: Zac Medico Zac Medico (2): Add get_repo_revision_history function and repo_revisions file bintree: Add REPO_REVISIONS to package index header bin/emerge-webrsync | 3 +- lib/portage/const.py | 1 + lib/portage/dbapi/bintree.py | 66 +++- lib/portage/emaint/modules/meson.build| 1 + .../emaint/modules/revisions/__init__.py | 36 + .../emaint/modules/revisions/meson.build | 8 + .../emaint/modules/revisions/revisions.py | 95 lib/portage/sync/controller.py| 8 +- lib/portage/sync/meson.build | 1 + lib/portage/sync/revision_history.py | 133 lib/portage/tests/sync/test_sync_local.py | 146 -- man/emaint.1 | 18 ++- man/portage.5 | 17 +- 13 files changed, 511 insertions(+), 22 deletions(-) create mode 100644 lib/portage/emaint/modules/revisions/__init__.py create mode 100644 lib/portage/emaint/modules/revisions/meson.build create mode 100644 lib/portage/emaint/modules/revisions/revisions.py create mode 100644 lib/portage/sync/revision_history.py -- 2.41.0
[gentoo-portage-dev] [PATCH 9/9] EbuildPhase: async_check_locale
Change config.environ() check_locale calls to async_check_locale calls in the EbuildPhase _async_start method in order to eliminate synchronous waiting for child processes in the main event loop thread. Bug: https://bugs.gentoo.org/923841 Signed-off-by: Zac Medico --- lib/_emerge/EbuildMetadataPhase.py| 4 +++ lib/_emerge/EbuildPhase.py| 28 ++- lib/portage/package/ebuild/config.py | 26 - lib/portage/util/futures/_asyncio/__init__.py | 9 ++ lib/portage/util/locale.py| 28 +-- 5 files changed, 70 insertions(+), 25 deletions(-) diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py index 9fcdabe840..90a3ea05aa 100644 --- a/lib/_emerge/EbuildMetadataPhase.py +++ b/lib/_emerge/EbuildMetadataPhase.py @@ -8,6 +8,7 @@ import portage portage.proxy.lazyimport.lazyimport( globals(), +"_emerge.EbuildPhase:_setup_locale", "portage.package.ebuild._metadata_invalid:eapi_invalid", ) from portage import os @@ -83,6 +84,9 @@ class EbuildMetadataPhase(SubProcess): settings.setcpv(self.cpv) settings.configdict["pkg"]["EAPI"] = parsed_eapi +# This requires above setcpv and EAPI setup. +await _setup_locale(self.settings) + debug = settings.get("PORTAGE_DEBUG") == "1" master_fd = None slave_fd = None diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py index c81bf54a81..c8caf73722 100644 --- a/lib/_emerge/EbuildPhase.py +++ b/lib/_emerge/EbuildPhase.py @@ -1,4 +1,4 @@ -# Copyright 1999-2021 Gentoo Authors +# Copyright 1999-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 import functools @@ -24,6 +24,7 @@ from portage.package.ebuild.prepare_build_dirs import ( _prepare_fake_distdir, _prepare_fake_filesdir, ) +from portage.eapi import _get_eapi_attrs from portage.util import writemsg, ensure_dirs from portage.util._async.AsyncTaskFuture import AsyncTaskFuture from portage.util._async.BuildLogger import BuildLogger @@ -54,12 +55,34 @@ portage.proxy.lazyimport.lazyimport( + "_post_src_install_write_metadata," + "_preinst_bsdflags", "portage.util.futures.unix_events:_set_nonblocking", +"portage.util.locale:async_check_locale,split_LC_ALL", ) from portage import os from portage import _encodings from portage import _unicode_encode +async def _setup_locale(settings): +eapi_attrs = _get_eapi_attrs(settings["EAPI"]) +if eapi_attrs.posixish_locale: +split_LC_ALL(settings) +settings["LC_COLLATE"] = "C" +# check_locale() returns None when check can not be executed. +if await async_check_locale(silent=True, env=settings.environ()) is False: +# try another locale +for l in ("C.UTF-8", "en_US.UTF-8", "en_GB.UTF-8", "C"): +settings["LC_CTYPE"] = l +if await async_check_locale(silent=True, env=settings.environ()): +# TODO: output the following only once +# writemsg( +# _("!!! LC_CTYPE unsupported, using %s instead\n") +# % self.settings["LC_CTYPE"] +# ) +break +else: +raise AssertionError("C locale did not pass the test!") + + class EbuildPhase(CompositeTask): __slots__ = ("actionmap", "fd_pipes", "phase", "settings") + ("_ebuild_lock",) @@ -94,6 +117,9 @@ class EbuildPhase(CompositeTask): self._start_task(AsyncTaskFuture(future=future), self._async_start_exit) async def _async_start(self): + +await _setup_locale(self.settings) + need_builddir = self.phase not in EbuildProcess._phases_without_builddir if need_builddir: diff --git a/lib/portage/package/ebuild/config.py b/lib/portage/package/ebuild/config.py index d7b0ca5676..35c77486ec 100644 --- a/lib/portage/package/ebuild/config.py +++ b/lib/portage/package/ebuild/config.py @@ -29,7 +29,6 @@ portage.proxy.lazyimport.lazyimport( "portage.dbapi.vartree:vartree", "portage.package.ebuild.doebuild:_phase_func_map", "portage.util.compression_probe:_compressors", -"portage.util.locale:check_locale,split_LC_ALL", ) from portage import bsd_chflags, load_mod, os, selinux, _unicode_decode from portage.const import ( @@ -3368,20 +3367,17 @@ class config: mydict["EBUILD_PHASE_FUNC"] = phase_func if eapi_attrs.posixish_locale: -split_LC_ALL(mydict) -mydict["LC_COLLATE"] = "C" -# check_locale() returns None when check can not be executed. -if check_locale(silent=True, env=mydict) is False: -# try another locale -for l in ("C.UTF-8", "en_US.UTF-8", "en_GB.UTF-8", "C"): -mydict["LC_CTYPE"] = l -if check_locale(
[gentoo-portage-dev] [PATCH 8/9] EbuildMetadataPhase: Migrate to _async_start
Bug: https://bugs.gentoo.org/923841 Signed-off-by: Zac Medico --- lib/_emerge/EbuildMetadataPhase.py | 17 ++--- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py index 784712e8cb..9fcdabe840 100644 --- a/lib/_emerge/EbuildMetadataPhase.py +++ b/lib/_emerge/EbuildMetadataPhase.py @@ -46,6 +46,12 @@ class EbuildMetadataPhase(SubProcess): _files_dict = slot_dict_class(_file_names, prefix="") def _start(self): +asyncio.ensure_future( +self._async_start(), loop=self.scheduler +).add_done_callback(self._async_start_done) +self._registered = True + +async def _async_start(self): ebuild_path = self.ebuild_hash.location with open( @@ -116,7 +122,6 @@ class EbuildMetadataPhase(SubProcess): self._raw_metadata = [] files.ebuild = master_fd self.scheduler.add_reader(files.ebuild, self._output_handler) -self._registered = True retval = portage.doebuild( ebuild_path, @@ -150,16 +155,6 @@ class EbuildMetadataPhase(SubProcess): self._proc = retval -asyncio.ensure_future( -self._async_start(), loop=self.scheduler -).add_done_callback(self._async_start_done) - -async def _async_start(self): -# Call async check_locale here for bug 923841, but code -# also needs to migrate from _start to here, including -# the self.deallocate_config set_result call. -pass - def _async_start_done(self, future): future.cancelled() or future.result() if self._was_cancelled(): -- 2.41.0
[gentoo-portage-dev] [PATCH 7/9] actions: disable pytest-xdist for spawn start-method (workers crash)
Bug: https://bugs.gentoo.org/924416 Signed-off-by: Zac Medico --- .github/workflows/ci.yml | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5bffd97206..762999b7cc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -90,5 +90,8 @@ jobs: - name: Run tests for ${{ matrix.python-version }} run: | [[ "${{ matrix.start-method }}" == "spawn" ]] && export PORTAGE_MULTIPROCESSING_START_METHOD=spawn - export PYTEST_ADDOPTS="-vv -ra -l -o console_output_style=count -n $(nproc) --dist=worksteal" + # spawn start-method crashes pytest-xdist workers (bug 924416) + [[ "${{ matrix.start-method }}" == "spawn" ]] && \ +export PYTEST_ADDOPTS="-vv -ra -l -o console_output_style=count" || \ +export PYTEST_ADDOPTS="-vv -ra -l -o console_output_style=count -n $(nproc) --dist=worksteal" meson test -C /tmp/build --verbose -- 2.41.0
[gentoo-portage-dev] [PATCH 6/9] async_aux_get: Use EbuildMetadataPhase deallocate_config future
For the portdbapi async_aux_get method, there is not a very good place to store a config pool, so instead use asyncio.Lock to manage access to the portdbapi doebuild_settings attribute when using the main event loop in the main thread. For other threads, clone a config instance since we do not have a thread-safe config pool. This cloning is expensive, but since portage internals do not trigger this case, it suffices for now (an AssertionError ensures that internals do not trigger it). For the main event loop running in the main thread, performance with the asyncio.Lock should not be significantly different to performance prior to commit c95fc64abf96, since check_locale results are typically cached and before there was only a single shared doebuild_settings instance with access serialized via the EbuildMetadataPhase _start method. Update async_aux_get callers to use asyncio.ensure_future on the returned coroutine when needed, since it used to return a future instead of a coroutine, and sometimes a future is needed for add_done_callback usage. In the portdbapi async_fetch_map method, fix a broken reference to "future" which should have been "aux_get_future", an error discovered while testing this patch. Bug: https://bugs.gentoo.org/924319 Fixes: c95fc64abf96 ("EbuildPhase: async_check_locale") Signed-off-by: Zac Medico --- lib/portage/_emirrordist/FetchIterator.py | 10 +- lib/portage/dbapi/porttree.py | 129 +++--- lib/portage/tests/update/test_move_ent.py | 3 + 3 files changed, 97 insertions(+), 45 deletions(-) diff --git a/lib/portage/_emirrordist/FetchIterator.py b/lib/portage/_emirrordist/FetchIterator.py index eaf3e53596..e4fdd092af 100644 --- a/lib/portage/_emirrordist/FetchIterator.py +++ b/lib/portage/_emirrordist/FetchIterator.py @@ -1,4 +1,4 @@ -# Copyright 2013-2018 Gentoo Foundation +# Copyright 2013-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 import threading @@ -14,6 +14,7 @@ from portage.exception import PortageException, PortageKeyError from portage.package.ebuild.fetch import DistfileName from portage.util._async.AsyncTaskFuture import AsyncTaskFuture from portage.util._async.TaskScheduler import TaskScheduler +from portage.util.futures import asyncio from portage.util.futures.iter_completed import iter_gather from .FetchTask import FetchTask from _emerge.CompositeTask import CompositeTask @@ -276,8 +277,11 @@ def _async_fetch_tasks(config, hash_filter, repo_config, digests_future, cpv, lo result.set_result(fetch_tasks) def future_generator(): -yield config.portdb.async_aux_get( -cpv, ("RESTRICT",), myrepo=repo_config.name, loop=loop +yield asyncio.ensure_future( +config.portdb.async_aux_get( +cpv, ("RESTRICT",), myrepo=repo_config.name, loop=loop +), +loop, ) yield config.portdb.async_fetch_map(cpv, mytree=repo_config.location, loop=loop) diff --git a/lib/portage/dbapi/porttree.py b/lib/portage/dbapi/porttree.py index 61d431f917..4eebe1183f 100644 --- a/lib/portage/dbapi/porttree.py +++ b/lib/portage/dbapi/porttree.py @@ -1,4 +1,4 @@ -# Copyright 1998-2021 Gentoo Authors +# Copyright 1998-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 __all__ = ["close_portdbapi_caches", "FetchlistDict", "portagetree", "portdbapi"] @@ -41,7 +41,9 @@ from portage.util.futures import asyncio from portage.util.futures.iter_completed import iter_gather from _emerge.EbuildMetadataPhase import EbuildMetadataPhase +import contextlib import os as _os +import threading import traceback import warnings import errno @@ -239,6 +241,7 @@ class portdbapi(dbapi): # this purpose because doebuild makes many changes to the config # instance that is passed in. self.doebuild_settings = config(clone=self.settings) +self._doebuild_settings_lock = asyncio.Lock() self.depcachedir = os.path.realpath(self.settings.depcachedir) if os.environ.get("SANDBOX_ON") == "1": @@ -356,6 +359,17 @@ class portdbapi(dbapi): self._better_cache = None self._broken_ebuilds = set() +def __getstate__(self): +state = self.__dict__.copy() +# These attributes are not picklable, so they are automatically +# regenerated after unpickling. +state["_doebuild_settings_lock"] = None +return state + +def __setstate__(self, state): +self.__dict__.update(state) +self._doebuild_settings_lock = asyncio.Lock() + def _set_porttrees(self, porttrees): """ Consumers, such as emirrordist, may modify the porttrees attribute in @@ -669,7 +683,7 @@ class portdbapi(dbapi): self.async_aux_get(mycpv, mylist, mytree=mytree, myrepo=myrepo, loop=loop) ) -def async_aux_get(self, mycpv, mylist, mytree=None, myrepo=None, loop=None): +
[gentoo-portage-dev] [PATCH 5/9] asyncio: Wrap asyncio.Lock for python 3.9 compat
Wrap asyncio.Lock for compatibility with python 3.9 where the deprecated loop parameter is required in order to avoid "got Future attached to a different loop" errors. The pordbapi async_aux_get method can use asyncio.Lock to serialize access to its doebuild_settings attribute in order to prevent issues like bug 924319. Bug: https://bugs.gentoo.org/924319 Signed-off-by: Zac Medico --- lib/portage/util/futures/_asyncio/__init__.py | 17 + 1 file changed, 17 insertions(+) diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py index 8f1b8e8275..b6481c281e 100644 --- a/lib/portage/util/futures/_asyncio/__init__.py +++ b/lib/portage/util/futures/_asyncio/__init__.py @@ -9,6 +9,7 @@ __all__ = ( "CancelledError", "Future", "InvalidStateError", +"Lock", "TimeoutError", "get_child_watcher", "get_event_loop", @@ -22,6 +23,7 @@ __all__ = ( "wait_for", ) +import sys import types import weakref @@ -35,6 +37,7 @@ from asyncio import ( FIRST_EXCEPTION, Future, InvalidStateError, +Lock as _Lock, shield, TimeoutError, wait_for, @@ -159,6 +162,20 @@ def iscoroutinefunction(func): return False +class Lock(_Lock): +""" +Inject loop parameter for python3.9 or less in order to avoid +"got Future attached to a different loop" errors. +""" + +def __init__(self, **kwargs): +if sys.version_info >= (3, 10): +kwargs.pop("loop", None) +elif "loop" not in kwargs: +kwargs["loop"] = _safe_loop()._loop +super().__init__(**kwargs) + + class Task(Future): """ Schedule the execution of a coroutine: wrap it in a future. A task -- 2.41.0
[gentoo-portage-dev] [PATCH 4/9] ResolverPlayground: Use egencache to create manifests
Make the ResolverPlayground _create_ebuild_manifests method call egencache --jobs, which reliably triggers the KeyError from bug 924319 for multiple tests: lib/portage/tests/bin/test_doins.py::DoIns::testDoInsFallback Exception in callback EbuildMetadataPhase._async_start_done() handle: )> Traceback (most recent call last): File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "lib/_emerge/EbuildMetadataPhase.py", line 154, in _async_start_done future.cancelled() or future.result() ^^^ File "lib/_emerge/EbuildMetadataPhase.py", line 130, in _async_start retval = portage.doebuild( ^ File "lib/portage/package/ebuild/doebuild.py", line 1030, in doebuild doebuild_environment( File "lib/portage/package/ebuild/doebuild.py", line 519, in doebuild_environment eapi = mysettings.configdict["pkg"]["EAPI"] File "lib/portage/util/__init__.py", line 1684, in __getitem__ return UserDict.__getitem__(self, item_key) File "lib/portage/cache/mappings.py", line 175, in __getitem__ return self.data[key] ~^ KeyError: 'EAPI' Bug: https://bugs.gentoo.org/924319 Signed-off-by: Zac Medico --- lib/portage/tests/dbapi/test_portdb_cache.py | 4 +- .../tests/resolver/ResolverPlayground.py | 38 ++- 2 files changed, 23 insertions(+), 19 deletions(-) diff --git a/lib/portage/tests/dbapi/test_portdb_cache.py b/lib/portage/tests/dbapi/test_portdb_cache.py index c7c6913b49..c24a4f2098 100644 --- a/lib/portage/tests/dbapi/test_portdb_cache.py +++ b/lib/portage/tests/dbapi/test_portdb_cache.py @@ -1,6 +1,7 @@ -# Copyright 2012-2023 Gentoo Authors +# Copyright 2012-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 +import shutil import subprocess import sys import textwrap @@ -63,6 +64,7 @@ class PortdbCacheTestCase(TestCase): python_cmd = (portage_python, "-b", "-Wd", "-c") test_commands = ( +(lambda: shutil.rmtree(md5_cache_dir) or True,), (lambda: not os.path.exists(pms_cache_dir),), (lambda: not os.path.exists(md5_cache_dir),), python_cmd diff --git a/lib/portage/tests/resolver/ResolverPlayground.py b/lib/portage/tests/resolver/ResolverPlayground.py index 2d26012873..75c86b615c 100644 --- a/lib/portage/tests/resolver/ResolverPlayground.py +++ b/lib/portage/tests/resolver/ResolverPlayground.py @@ -3,6 +3,7 @@ import bz2 import fnmatch +import subprocess import tempfile import portage @@ -18,8 +19,6 @@ from portage.const import ( from portage.process import find_binary from portage.dep import Atom, _repo_separator from portage.dbapi.bintree import binarytree -from portage.package.ebuild.config import config -from portage.package.ebuild.digestgen import digestgen from portage._sets import load_default_config from portage._sets.base import InternalPackageSet from portage.tests import cnf_path @@ -323,22 +322,25 @@ class ResolverPlayground: f.write(misc_content) def _create_ebuild_manifests(self, ebuilds): -tmpsettings = config(clone=self.settings) -tmpsettings["PORTAGE_QUIET"] = "1" -for cpv in ebuilds: -a = Atom("=" + cpv, allow_repo=True) -repo = a.repo -if repo is None: -repo = "test_repo" - -repo_dir = self._get_repo_dir(repo) -ebuild_dir = os.path.join(repo_dir, a.cp) -ebuild_path = os.path.join(ebuild_dir, a.cpv.split("/")[1] + ".ebuild") - -portdb = self.trees[self.eroot]["porttree"].dbapi -tmpsettings["O"] = ebuild_dir -if not digestgen(mysettings=tmpsettings, myportdb=portdb): -raise AssertionError(f"digest creation failed for {ebuild_path}") +for repo_name in self._repositories: +if repo_name == "DEFAULT": +continue +egencache_cmd = [ +"egencache", +f"--repo={repo_name}", +"--update", +"--update-manifests", +"--sign-manifests=n", +"--strict-manifests=n", + f"--repositories-configuration={self.settings['PORTAGE_REPOSITORIES']}", +f"--jobs={portage.util.cpuinfo.get_cpu_count()}", +] +result = subprocess.run( +egencache_cmd, +env=self.settings.environ(), +) +if result.returncode != os.EX_OK: +raise AssertionError(f"command failed: {egencache_cmd}") def _create_binpkgs(self, binpkgs): # When using BUILD_ID, there can be multiple instances for the -- 2.41.0
[gentoo-portage-dev] [PATCH 3/9] MetadataRegen: Use EbuildMetadataPhase deallocate_config
For EbuildMetadataPhase consumers like MetadataRegen and depgraph, store a pool of config instances in a config_pool list, and return instaces to the list when the deallocate_config future is done. Bug: https://bugs.gentoo.org/924319 Fixes: c95fc64abf96 ("EbuildPhase: async_check_locale") Signed-off-by: Zac Medico --- lib/_emerge/MetadataRegen.py | 16 ++-- lib/_emerge/depgraph.py | 11 +++ 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/lib/_emerge/MetadataRegen.py b/lib/_emerge/MetadataRegen.py index d29722b94c..538a94b450 100644 --- a/lib/_emerge/MetadataRegen.py +++ b/lib/_emerge/MetadataRegen.py @@ -1,4 +1,4 @@ -# Copyright 1999-2020 Gentoo Authors +# Copyright 1999-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 from _emerge.EbuildMetadataPhase import EbuildMetadataPhase @@ -44,6 +44,7 @@ class MetadataRegen(AsyncScheduler): valid_pkgs = self._valid_pkgs cp_set = self._cp_set consumer = self._consumer +config_pool = [] portage.writemsg_stdout("Regenerating cache entries...\n") for cp in self._cp_iter: @@ -73,12 +74,23 @@ class MetadataRegen(AsyncScheduler): consumer(cpv, repo_path, metadata, ebuild_hash, True) continue +if config_pool: +settings = config_pool.pop() +else: +settings = portage.config(clone=portdb.settings) + +deallocate_config = self.scheduler.create_future() +deallocate_config.add_done_callback( +lambda future: config_pool.append(future.result()) +) + yield EbuildMetadataPhase( cpv=cpv, ebuild_hash=ebuild_hash, portdb=portdb, repo_path=repo_path, -settings=portdb.doebuild_settings, +settings=settings, +deallocate_config=deallocate_config, write_auxdb=self._write_auxdb, ) diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py index 1674fa289e..70b83ee1f4 100644 --- a/lib/_emerge/depgraph.py +++ b/lib/_emerge/depgraph.py @@ -754,6 +754,7 @@ class depgraph: def _dynamic_deps_preload(self, fake_vartree): portdb = fake_vartree._portdb +config_pool = [] for pkg in fake_vartree.dbapi: self._spinner_update() self._dynamic_config._package_tracker.add_installed_pkg(pkg) @@ -768,12 +769,22 @@ class depgraph: if metadata is not None: fake_vartree.dynamic_deps_preload(pkg, metadata) else: +if config_pool: +settings = config_pool.pop() +else: +settings = portage.config(clone=portdb.settings) + +deallocate_config = portdb._event_loop.create_future() +deallocate_config.add_done_callback( +lambda future: config_pool.append(future.result()) +) proc = EbuildMetadataPhase( cpv=pkg.cpv, ebuild_hash=ebuild_hash, portdb=portdb, repo_path=repo_path, settings=portdb.doebuild_settings, +deallocate_config=deallocate_config, ) proc.addExitListener(self._dynamic_deps_proc_exit(pkg, fake_vartree)) yield proc -- 2.41.0
[gentoo-portage-dev] [PATCH 2/9] EbuildMetadataPhase: Add deallocate_config future
Use a deallocate_config future to release self.settings when it is no longer needed. It's necessary to manage concurrency since commit c95fc64abf96 because mutation of self.settings is no longer limited to the EbuildMetadataPhase _start method, where exclusive access was guaranteed within the main thread. Add support to the isAlive() method to detect when the EbuildMetadataPhase has started but the pid is not yet available (due to async_check_locale usage from commit c95fc64abf96). This can be used to check if an EbuildMetadataPhase instance has been successfully started so that it can be relied upon to set the result of the deallocate_config future. Bug: https://bugs.gentoo.org/924319 Signed-off-by: Zac Medico --- lib/_emerge/EbuildMetadataPhase.py | 36 ++ lib/_emerge/SubProcess.py | 5 - 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py index f4f685e81c..784712e8cb 100644 --- a/lib/_emerge/EbuildMetadataPhase.py +++ b/lib/_emerge/EbuildMetadataPhase.py @@ -14,6 +14,7 @@ from portage import os from portage import _encodings from portage import _unicode_decode from portage import _unicode_encode +from portage.util.futures import asyncio import fcntl @@ -33,6 +34,7 @@ class EbuildMetadataPhase(SubProcess): "portdb", "repo_path", "settings", +"deallocate_config", "write_auxdb", ) + ( "_eapi", @@ -127,6 +129,15 @@ class EbuildMetadataPhase(SubProcess): returnproc=True, ) settings.pop("PORTAGE_PIPE_FD", None) +# At this point we can return settings to the caller +# since we never use it for anything more than an +# eapi_invalid call after this, and eapi_invalid is +# insensitive to concurrent modifications. +if ( +self.deallocate_config is not None +and not self.deallocate_config.cancelled() +): +self.deallocate_config.set_result(settings) os.close(slave_fd) null_input.close() @@ -139,6 +150,31 @@ class EbuildMetadataPhase(SubProcess): self._proc = retval +asyncio.ensure_future( +self._async_start(), loop=self.scheduler +).add_done_callback(self._async_start_done) + +async def _async_start(self): +# Call async check_locale here for bug 923841, but code +# also needs to migrate from _start to here, including +# the self.deallocate_config set_result call. +pass + +def _async_start_done(self, future): +future.cancelled() or future.result() +if self._was_cancelled(): +pass +elif future.cancelled(): +self.cancel() +self._was_cancelled() + +if self.deallocate_config is not None and not self.deallocate_config.done(): +self.deallocate_config.set_result(self.settings) + +if self.returncode is not None: +self._unregister() +self.wait() + def _output_handler(self): while True: buf = self._read_buf(self._files.ebuild) diff --git a/lib/_emerge/SubProcess.py b/lib/_emerge/SubProcess.py index 029bbc3f44..057e0adc24 100644 --- a/lib/_emerge/SubProcess.py +++ b/lib/_emerge/SubProcess.py @@ -18,9 +18,12 @@ class SubProcess(AbstractPollTask): # we've sent a kill signal to our subprocess. _cancel_timeout = 1 # seconds +def isAlive(self): +return (self._registered or self.pid is not None) and self.returncode is None + @property def pid(self): -return self._proc.pid +return None if self._proc is None else self._proc.pid def _poll(self): # Simply rely on _async_waitpid_cb to set the returncode. -- 2.41.0
[gentoo-portage-dev] [PATCH 1/9] anydbm: Pickle support for multiprocessing spawn
The egencache usage in ResolverPlayground that was used to trigger bug 924319 triggered a pickling error for AuxdbTestCase.test_anydbm with the multiprocessing spawn start method, so fix the anydbm cache module to omit the unpicklable database object from pickled state, and regenerate it after unpickling. Bug: https://bugs.gentoo.org/924319 Signed-off-by: Zac Medico --- lib/portage/cache/anydbm.py | 17 - lib/portage/tests/dbapi/test_auxdb.py | 4 +--- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/lib/portage/cache/anydbm.py b/lib/portage/cache/anydbm.py index 94a270a483..ad7042ae41 100644 --- a/lib/portage/cache/anydbm.py +++ b/lib/portage/cache/anydbm.py @@ -1,4 +1,4 @@ -# Copyright 2005-2020 Gentoo Authors +# Copyright 2005-2024 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # Author(s): Brian Harring (ferri...@gentoo.org) @@ -67,6 +67,21 @@ class database(fs_template.FsBased): raise cache_errors.InitializationError(self.__class__, e) self._ensure_access(self._db_path) +def __getstate__(self): +state = self.__dict__.copy() +# These attributes are not picklable, so they are automatically +# regenerated after unpickling. +state["_database__db"] = None +return state + +def __setstate__(self, state): +self.__dict__.update(state) +mode = "w" +if dbm.whichdb(self._db_path) in ("dbm.gnu", "gdbm"): +# Allow multiple concurrent writers (see bug #53607). +mode += "u" +self.__db = dbm.open(self._db_path, mode, self._perms) + def iteritems(self): # dbm doesn't implement items() for k in self.__db.keys(): diff --git a/lib/portage/tests/dbapi/test_auxdb.py b/lib/portage/tests/dbapi/test_auxdb.py index 0de0123a5f..aac6ce361c 100644 --- a/lib/portage/tests/dbapi/test_auxdb.py +++ b/lib/portage/tests/dbapi/test_auxdb.py @@ -16,9 +16,7 @@ class AuxdbTestCase(TestCase): from portage.cache.anydbm import database except ImportError: self.skipTest("dbm import failed") -self._test_mod( -"portage.cache.anydbm.database", multiproc=False, picklable=False -) +self._test_mod("portage.cache.anydbm.database", multiproc=False, picklable=True) def test_flat_hash_md5(self): self._test_mod("portage.cache.flat_hash.md5_database") -- 2.41.0
[gentoo-portage-dev] [PATCH 0/9] EbuildPhase/EbuildMetadataPhase: async check_locale
Change config.environ() check_locale calls to async_check_locale calls in the EbuildPhase/EbuildMetadataPhase _async_start method in order to eliminate synchronous waiting for child processes in the main event loop thread. Note that this series of changes causes access to the portdbapi doebuild_settings attribute to no longer be serialized via the EbuildMetadataPhase _start_method. As a result, exclusive access to config instances needs to be guaranteed in some other way to avoid triggering problems (see bug 924319), such as by maintaining a config pool or by serializing access via an asyncio.Lock instance. This series can also be reviewed at https://github.com/gentoo/portage/pull/1267. Bug: https://bugs.gentoo.org/923841 Bug: https://bugs.gentoo.org/924319 Signed-off-by: Zac Medico Zac Medico (9): anydbm: Pickle support for multiprocessing spawn EbuildMetadataPhase: Add deallocate_config future MetadataRegen: Use EbuildMetadataPhase deallocate_config ResolverPlayground: Use egencache to create manifests asyncio: Wrap asyncio.Lock for python 3.9 compat async_aux_get: Use EbuildMetadataPhase deallocate_config future actions: disable pytest-xdist for spawn start-method (workers crash) EbuildMetadataPhase: Migrate to _async_start EbuildPhase: async_check_locale .github/workflows/ci.yml | 5 +- lib/_emerge/EbuildMetadataPhase.py| 37 - lib/_emerge/EbuildPhase.py| 28 +++- lib/_emerge/MetadataRegen.py | 16 ++- lib/_emerge/SubProcess.py | 5 +- lib/_emerge/depgraph.py | 11 ++ lib/portage/_emirrordist/FetchIterator.py | 10 +- lib/portage/cache/anydbm.py | 17 ++- lib/portage/dbapi/porttree.py | 129 -- lib/portage/package/ebuild/config.py | 26 ++-- lib/portage/tests/dbapi/test_auxdb.py | 4 +- lib/portage/tests/dbapi/test_portdb_cache.py | 4 +- .../tests/resolver/ResolverPlayground.py | 38 +++--- lib/portage/tests/update/test_move_ent.py | 3 + lib/portage/util/futures/_asyncio/__init__.py | 26 lib/portage/util/locale.py| 28 ++-- 16 files changed, 289 insertions(+), 98 deletions(-) -- 2.41.0
[gentoo-portage-dev] [PATCH v2] process.spawn: Add returnproc parameter
In order to migrate away from unsafe os.fork() usage in threaded processes (https://github.com/python/cpython/issues/84559), add a returnproc parameter that is similar to returnpid, which causes spawn to return a single Process object instead of a list of pids. The Process API is a subset of asyncio.subprocess.Process. The returnproc parameter conflicts with the logfile parameter, since the caller is expected to use the fd_pipes parameter to implement logging (this was also true for the returnpid parameter). In the future, spawn will return objects of a different type but with a compatible interface to Process, in order to encapsulate implementation-dependent objects like multiprocessing.Process which are designed to manage the process lifecycle and need to persist until it exits. Trigger a UserWarning when the returnpid parameter is used, in order to encourage migration to returnproc (do not use DeprecationWarning since it is hidden by default). This warning will be temporarily suppressed for portage internals, until they finish migrating to returnproc. There are probably very few if any external consumers of spawn with the returnpid parameter, so it seems safe to move quickly with this deprecation. Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico --- [PATCH v2] Make returnproc return a singlar Process (for comparison, returnpid never returned a list container more than a single pid). Also, returnproc explicitly conflicts with the logfile parameter, since callers are expected to use the fd_pipes parameter to implement logging (this was also true for returnpid). lib/portage/process.py| 84 +++ lib/portage/tests/process/meson.build | 1 + .../tests/process/test_spawn_returnproc.py| 39 + 3 files changed, 124 insertions(+) create mode 100644 lib/portage/tests/process/test_spawn_returnproc.py diff --git a/lib/portage/process.py b/lib/portage/process.py index 71750a715f..6ec52efc4a 100644 --- a/lib/portage/process.py +++ b/lib/portage/process.py @@ -15,6 +15,7 @@ import subprocess import sys import traceback import os as _os +import warnings from dataclasses import dataclass from functools import lru_cache @@ -27,6 +28,7 @@ import portage portage.proxy.lazyimport.lazyimport( globals(), +"portage.util._eventloop.global_event_loop:global_event_loop", "portage.util:dump_traceback,writemsg,writemsg_level", ) @@ -277,12 +279,78 @@ def calc_env_stats(env) -> EnvStats: env_too_large_warnings = 0 +class Process: +""" +An object that wraps OS processes created by spawn. +In the future, spawn will return objects of a different type +but with a compatible interface to this class, in order +to encapsulate implementation-dependent objects like +multiprocessing.Process which are designed to manage +the process lifecycle and need to persist until it exits. +""" + +def __init__(self, pid): +self.pid = pid +self.returncode = None +self._exit_waiters = [] + +def __repr__(self): +return f"<{self.__class__.__name__} {self.pid}>" + +async def wait(self): +""" +Wait for the child process to terminate. + +Set and return the returncode attribute. +""" +if self.returncode is not None: +return self.returncode + +loop = global_event_loop() +if not self._exit_waiters: +loop._asyncio_child_watcher.add_child_handler(self.pid, self._child_handler) +waiter = loop.create_future() +self._exit_waiters.append(waiter) +return await waiter + +def _child_handler(self, pid, returncode): +if pid != self.pid: +raise AssertionError(f"expected pid {self.pid}, got {pid}") +self.returncode = returncode + +for waiter in self._exit_waiters: +if not waiter.cancelled(): +waiter.set_result(returncode) +self._exit_waiters = None + +def send_signal(self, sig): +"""Send a signal to the process.""" +if self.returncode is not None: +# Skip signalling a process that we know has already died. +return + +try: +os.kill(self.pid, sig) +except ProcessLookupError: +# Suppress the race condition error; bpo-40550. +pass + +def terminate(self): +"""Terminate the process with SIGTERM""" +self.send_signal(signal.SIGTERM) + +def kill(self): +"""Kill the process with SIGKILL""" +self.send_signal(signal.SIGKILL) + + def spawn( mycommand, env=None, opt_name=None, fd_pipes=None, returnpid=False, +returnproc=False, uid=None, gid=None, groups=None, @@ -315,6 +383,9 @@ def spawn( @param returnpid: Return the Process IDs for a successful spawn. NOTE: This requires the caller clean up all the PIDs, otherwise spawn will clean them.
[gentoo-portage-dev] [PATCH] process.spawn: Add returnproc parameter
In order to migrate away from unsafe os.fork() usage in threaded processes (https://github.com/python/cpython/issues/84559), add a returnproc parameter that is similar to returnpid, which causes spawn to return Process objects instead of pids. The Process API is a subset of asyncio.subprocess.Process. In the future, spawn will return objects of a different type but with a compatible interface to Process, in order to encapsulate implementation-dependent objects like multiprocessing.Process which are designed to manage the process lifecycle and need to persist until it exits. Trigger a UserWarning when the returnpid parameter is used, in order to encourage migration to returnproc (do not use DeprecationWarning since it is hidden by default). This warning will be temporarily suppressed for portage internals, until they finish migrating to returnproc. There are probably very few if any external consumers of spawn with the returnpid parameter, so it seems safe to move quickly with this deprecation. Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico --- lib/portage/process.py| 79 +++ lib/portage/tests/process/meson.build | 1 + .../tests/process/test_spawn_returnproc.py| 39 + 3 files changed, 119 insertions(+) create mode 100644 lib/portage/tests/process/test_spawn_returnproc.py diff --git a/lib/portage/process.py b/lib/portage/process.py index 71750a715f..74953b3e22 100644 --- a/lib/portage/process.py +++ b/lib/portage/process.py @@ -15,6 +15,7 @@ import subprocess import sys import traceback import os as _os +import warnings from dataclasses import dataclass from functools import lru_cache @@ -27,6 +28,7 @@ import portage portage.proxy.lazyimport.lazyimport( globals(), +"portage.util._eventloop.global_event_loop:global_event_loop", "portage.util:dump_traceback,writemsg,writemsg_level", ) @@ -277,12 +279,78 @@ def calc_env_stats(env) -> EnvStats: env_too_large_warnings = 0 +class Process: +""" +An object that wraps OS processes created by spawn. +In the future, spawn will return objects of a different type +but with a compatible interface to this class, in order +to encapsulate implementation-dependent objects like +multiprocessing.Process which are designed to manage +the process lifecycle and need to persist until it exits. +""" + +def __init__(self, pid): +self.pid = pid +self.returncode = None +self._exit_waiters = [] + +def __repr__(self): +return f"<{self.__class__.__name__} {self.pid}>" + +async def wait(self): +""" +Wait for the child process to terminate. + +Set and return the returncode attribute. +""" +if self.returncode is not None: +return self.returncode + +loop = global_event_loop() +if not self._exit_waiters: +loop._asyncio_child_watcher.add_child_handler(self.pid, self._child_handler) +waiter = loop.create_future() +self._exit_waiters.append(waiter) +return await waiter + +def _child_handler(self, pid, returncode): +if pid != self.pid: +raise AssertionError(f"expected pid {self.pid}, got {pid}") +self.returncode = returncode + +for waiter in self._exit_waiters: +if not waiter.cancelled(): +waiter.set_result(returncode) +self._exit_waiters = None + +def send_signal(self, sig): +"""Send a signal to the process.""" +if self.returncode is not None: +# Skip signalling a process that we know has already died. +return + +try: +os.kill(self.pid, sig) +except ProcessLookupError: +# Suppress the race condition error; bpo-40550. +pass + +def terminate(self): +"""Terminate the process with SIGTERM""" +self.send_signal(signal.SIGTERM) + +def kill(self): +"""Kill the process with SIGKILL""" +self.send_signal(signal.SIGKILL) + + def spawn( mycommand, env=None, opt_name=None, fd_pipes=None, returnpid=False, +returnproc=False, uid=None, gid=None, groups=None, @@ -315,6 +383,9 @@ def spawn( @param returnpid: Return the Process IDs for a successful spawn. NOTE: This requires the caller clean up all the PIDs, otherwise spawn will clean them. @type returnpid: Boolean +@param returnproc: Return the Process objects for a successful spawn. +NOTE: This requires the caller clean up all the PIDs, otherwise spawn will clean them. +@type returnproc: Boolean @param uid: User ID to spawn as; useful for dropping privilages @type uid: Integer @param gid: Group ID to spawn as; useful for dropping privilages @@ -491,7 +562,15 @@ def spawn( # If the caller wants to handle cleaning up the processes, we tell # it about all processes that
[gentoo-portage-dev] [PATCH v2] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
Stop implicitly forcing `NO_COLOR` and `NOCOLOR` in ebuild environment. This is undesired for several reasons: 1. It makes it impossible to control color for emerge output independently of command output, e.g. when one goes to a pty while the other goes to logs. 2. It makes it impossible to get color output in logs when running `emerge --jobs ...`. 3. Forcing `NO_COLOR=1` turns out to cause random test failures, and while fixing them is commendable, it is a pain for arch testing and it is currently blocking stabilization requests. With the new approach, the color output in programs is consistent between using ``--jobs`` or ``--quiet-build``, and not. Therefore, both cases generate uniform logs. In order to obtain logs free of color codes, one can either filter them via `ansifilter(1)` (as the manpages already recommend) or explicitly set `NO_COLOR`. Furthermore, one can combine color-free build output (for clean logs) with colorful emerge output by using: NO_COLOR=true emerge --color y ... Bug: https://bugs.gentoo.org/918515 Signed-off-by: Michał Górny --- NEWS | 5 + bin/ebuild | 4 lib/_emerge/AbstractEbuildProcess.py | 6 -- lib/_emerge/actions.py | 5 - lib/portage/tests/resolver/ResolverPlayground.py | 3 --- 5 files changed, 5 insertions(+), 18 deletions(-) Changes in v2: add a NEWS entry diff --git a/NEWS b/NEWS index f1305744f..eb5eb4ab0 100644 --- a/NEWS +++ b/NEWS @@ -9,6 +9,11 @@ Release notes take the form of the following optional categories: portage-3.0.56 (UNRELEASED) -- +Breaking changes: +* ebuild: no longer implicitly set NOCOLOR and NO_COLOR when color output + is disabled for emerge itself. In order to obtain color-free logs, please + either explicitly set NO_COLOR environment variable or use e.g. ansifilter. + Features: * sync: git: Add sync-git-verify-max-age-days option. diff --git a/bin/ebuild b/bin/ebuild index cbefb5816..69db474c0 100755 --- a/bin/ebuild +++ b/bin/ebuild @@ -132,10 +132,6 @@ def main(): or not sys.stdout.isatty() ): portage.output.nocolor() -portage.settings.unlock() -portage.settings["NO_COLOR"] = "true" -portage.settings.backup_changes("NO_COLOR") -portage.settings.lock() apply_priorities(portage.settings) diff --git a/lib/_emerge/AbstractEbuildProcess.py b/lib/_emerge/AbstractEbuildProcess.py index 96d91b5da..3a8a18b5f 100644 --- a/lib/_emerge/AbstractEbuildProcess.py +++ b/lib/_emerge/AbstractEbuildProcess.py @@ -80,12 +80,6 @@ class AbstractEbuildProcess(SpawnProcess): self._async_wait() return -if self.background: -# Automatically prevent color codes from showing up in logs, -# since we're not displaying to a terminal anyway. -self.settings["NOCOLOR"] = "true" -self.settings["NO_COLOR"] = "true" - start_ipc_daemon = False if self._enable_ipc_daemon: self.settings.pop("PORTAGE_EBUILD_EXIT_FILE", None) diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py index 37264114e..dbd9707a8 100644 --- a/lib/_emerge/actions.py +++ b/lib/_emerge/actions.py @@ -2749,15 +2749,10 @@ def adjust_config(myopts, settings): if "--color" in myopts: if "y" == myopts["--color"]: portage.output.havecolor = 1 -settings["NO_COLOR"] = "" else: portage.output.havecolor = 0 -settings["NO_COLOR"] = "true" -settings.backup_changes("NO_COLOR") elif settings.get("TERM") == "dumb" or not sys.stdout.isatty(): portage.output.havecolor = 0 -settings["NO_COLOR"] = "true" -settings.backup_changes("NO_COLOR") if "--pkg-format" in myopts: settings["PORTAGE_BINPKG_FORMAT"] = myopts["--pkg-format"] diff --git a/lib/portage/tests/resolver/ResolverPlayground.py b/lib/portage/tests/resolver/ResolverPlayground.py index 475c4aaac..115800a60 100644 --- a/lib/portage/tests/resolver/ResolverPlayground.py +++ b/lib/portage/tests/resolver/ResolverPlayground.py @@ -591,9 +591,6 @@ class ResolverPlayground: "PORTAGE_TMPDIR": os.path.join(self.eroot, "var/tmp"), } -if portage.util.no_color(os.environ): -make_conf["NO_COLOR"] = os.environ["NO_COLOR"] - # Pass along PORTAGE_USERNAME and PORTAGE_GRPNAME since they # need to be inherited by ebuild subprocesses. if "PORTAGE_USERNAME" in os.environ: -- 2.43.0
Re: [gentoo-portage-dev] [PATCH] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
On Sun, 2023-11-26 at 18:33 +0100, Ulrich Mueller wrote: > > > > > > On Sun, 26 Nov 2023, Michał Górny wrote: > > > 3. Forcing `NO_COLOR=1` turns out to cause random test failures, > > and while fixing them is commendable, it is a pain for arch testing > > and it is currently blocking stabilization requests. > > I'd argue that test suites that fail because of NO_COLOR are broken. > IIUC, these failures will show up for users who set NO_COLOR=1 in their > make.conf or in the environment? Sure, they are broken. However, as I said I'd rather fix them in my spare time than have to fix them to unblock a security stablereq because implicit Portage logic blocks all AT work on a minor issue that's non- trivial to fix. > > > With the new approach, the color output in programs is consistent > > between using ``--jobs`` or ``--quiet-build``, and not. Therefore, > > both cases generate uniform logs. In order to obtain logs free of color > > codes, one can either filter them via `ansifilter(1)` (as the manpages > > already recommend) or explicitly set `NO_COLOR`. > > Will this still guarantee that NO_COLOR=1 from the environment is always > passed on to the build system? > > Otherwise, it would break ebuild-mode in some configurations. > Yes, I've tested it and it worked. Both for passing from external env and via make.conf. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
> On Sun, 26 Nov 2023, Michał Górny wrote: > 3. Forcing `NO_COLOR=1` turns out to cause random test failures, >and while fixing them is commendable, it is a pain for arch testing >and it is currently blocking stabilization requests. I'd argue that test suites that fail because of NO_COLOR are broken. IIUC, these failures will show up for users who set NO_COLOR=1 in their make.conf or in the environment? > With the new approach, the color output in programs is consistent > between using ``--jobs`` or ``--quiet-build``, and not. Therefore, > both cases generate uniform logs. In order to obtain logs free of color > codes, one can either filter them via `ansifilter(1)` (as the manpages > already recommend) or explicitly set `NO_COLOR`. Will this still guarantee that NO_COLOR=1 from the environment is always passed on to the build system? Otherwise, it would break ebuild-mode in some configurations. Ulrich signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
Stop implicitly forcing `NO_COLOR` and `NOCOLOR` in ebuild environment. This is undesired for several reasons: 1. It makes it impossible to control color for emerge output independently of command output, e.g. when one goes to a pty while the other goes to logs. 2. It makes it impossible to get color output in logs when running `emerge --jobs ...`. 3. Forcing `NO_COLOR=1` turns out to cause random test failures, and while fixing them is commendable, it is a pain for arch testing and it is currently blocking stabilization requests. With the new approach, the color output in programs is consistent between using ``--jobs`` or ``--quiet-build``, and not. Therefore, both cases generate uniform logs. In order to obtain logs free of color codes, one can either filter them via `ansifilter(1)` (as the manpages already recommend) or explicitly set `NO_COLOR`. Furthermore, one can combine color-free build output (for clean logs) with colorful emerge output by using: NO_COLOR=true emerge --color y ... Bug: https://bugs.gentoo.org/918515 --- bin/ebuild | 4 lib/_emerge/AbstractEbuildProcess.py | 6 -- lib/_emerge/actions.py | 5 - lib/portage/tests/resolver/ResolverPlayground.py | 3 --- 4 files changed, 18 deletions(-) diff --git a/bin/ebuild b/bin/ebuild index cbefb5816..69db474c0 100755 --- a/bin/ebuild +++ b/bin/ebuild @@ -132,10 +132,6 @@ def main(): or not sys.stdout.isatty() ): portage.output.nocolor() -portage.settings.unlock() -portage.settings["NO_COLOR"] = "true" -portage.settings.backup_changes("NO_COLOR") -portage.settings.lock() apply_priorities(portage.settings) diff --git a/lib/_emerge/AbstractEbuildProcess.py b/lib/_emerge/AbstractEbuildProcess.py index 96d91b5da..3a8a18b5f 100644 --- a/lib/_emerge/AbstractEbuildProcess.py +++ b/lib/_emerge/AbstractEbuildProcess.py @@ -80,12 +80,6 @@ class AbstractEbuildProcess(SpawnProcess): self._async_wait() return -if self.background: -# Automatically prevent color codes from showing up in logs, -# since we're not displaying to a terminal anyway. -self.settings["NOCOLOR"] = "true" -self.settings["NO_COLOR"] = "true" - start_ipc_daemon = False if self._enable_ipc_daemon: self.settings.pop("PORTAGE_EBUILD_EXIT_FILE", None) diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py index 37264114e..dbd9707a8 100644 --- a/lib/_emerge/actions.py +++ b/lib/_emerge/actions.py @@ -2749,15 +2749,10 @@ def adjust_config(myopts, settings): if "--color" in myopts: if "y" == myopts["--color"]: portage.output.havecolor = 1 -settings["NO_COLOR"] = "" else: portage.output.havecolor = 0 -settings["NO_COLOR"] = "true" -settings.backup_changes("NO_COLOR") elif settings.get("TERM") == "dumb" or not sys.stdout.isatty(): portage.output.havecolor = 0 -settings["NO_COLOR"] = "true" -settings.backup_changes("NO_COLOR") if "--pkg-format" in myopts: settings["PORTAGE_BINPKG_FORMAT"] = myopts["--pkg-format"] diff --git a/lib/portage/tests/resolver/ResolverPlayground.py b/lib/portage/tests/resolver/ResolverPlayground.py index 475c4aaac..115800a60 100644 --- a/lib/portage/tests/resolver/ResolverPlayground.py +++ b/lib/portage/tests/resolver/ResolverPlayground.py @@ -591,9 +591,6 @@ class ResolverPlayground: "PORTAGE_TMPDIR": os.path.join(self.eroot, "var/tmp"), } -if portage.util.no_color(os.environ): -make_conf["NO_COLOR"] = os.environ["NO_COLOR"] - # Pass along PORTAGE_USERNAME and PORTAGE_GRPNAME since they # need to be inherited by ebuild subprocesses. if "PORTAGE_USERNAME" in os.environ: -- 2.43.0
[gentoo-portage-dev] [PATCH 3/3] prepman: Deprecate it
According to PMS, calling prepman from ebuilds was never allowed. Last usage in Gentoo repository has been removed on 2019-04-14. Signed-off-by: Ulrich Müller --- bin/ebuild-helpers/prepman | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/bin/ebuild-helpers/prepman b/bin/ebuild-helpers/prepman index 871fd589b..0c0a7e812 100755 --- a/bin/ebuild-helpers/prepman +++ b/bin/ebuild-helpers/prepman @@ -1,8 +1,12 @@ #!/usr/bin/env bash -# Copyright 1999-2018 Gentoo Foundation +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # Note: this really does nothing these days. It's going to be banned # when the last consumers are gone. +source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1 + +eqawarn "QA Notice: '${0##*/}' is not allowed in ebuild scope" + exit 0 -- 2.40.0
[gentoo-portage-dev] [PATCH 2/3] prepinfo: Deprecate it
According to PMS, calling prepinfo from ebuilds was never allowed. Last usage in Gentoo repository has been removed on 2018-11-22. Bug: https://bugs.gentoo.org/899898 Signed-off-by: Ulrich Müller --- bin/ebuild-helpers/prepinfo | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/bin/ebuild-helpers/prepinfo b/bin/ebuild-helpers/prepinfo index 14fa38226..af5ae9f08 100755 --- a/bin/ebuild-helpers/prepinfo +++ b/bin/ebuild-helpers/prepinfo @@ -1,9 +1,11 @@ #!/usr/bin/env bash -# Copyright 1999-2022 Gentoo Authors +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1 +eqawarn "QA Notice: '${0##*/}' is not allowed in ebuild scope" + if ! ___eapi_has_prefix_variables ; then ED=${D} fi -- 2.40.0
[gentoo-portage-dev] [PATCH 1/3] misc-functions.sh: Inline prepinfo code in install_qa_check()
Add a deprecation warning. Presumably, subdirs of /usr/share/info have never been used by any package except app-editors/emacs. Remove duplicate rm statement at the end of the function. Bug: https://bugs.gentoo.org/899898 Signed-off-by: Ulrich Müller --- bin/misc-functions.sh | 24 +++- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh index 55c4afe73..235823afc 100755 --- a/bin/misc-functions.sh +++ b/bin/misc-functions.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Copyright 1999-2018 Gentoo Authors +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # # Miscellaneous shell functions that make use of the ebuild env but don't need @@ -143,7 +143,24 @@ install_qa_check() { chflags -R nosunlnk,nouunlnk "${ED}" 2>/dev/null fi - [[ -d ${ED%/}/usr/share/info ]] && prepinfo + if [[ -d ${ED%/}/usr/share/info ]]; then + # Portage regenerates this on the installed system. + rm -f "${ED%/}"/usr/share/info/dir{,.info}{,.Z,.gz,.bz2,.lzma,.lz,.xz,.zst} \ + || die "rm failed" + # Recurse into subdirs. Remove this code after 2023-12-31. #899898 + while read -r -d '' x; do + ( shopt -s failglob; : "${x}"/.keepinfodir* ) 2>/dev/null \ + && continue + for f in "${x}"/dir{,.info}{,.Z,.gz,.bz2,.lzma,.lz,.xz,.zst}; do + if [[ -e ${f} ]]; then + eqawarn "QA Notice: Removing Info directory file '${f}'." + eqawarn "Relying on this behavior is deprecated and may" + eqawarn "cause file collisions in future." + rm -f "${f}" || die "rm failed" + fi + done + done < <(find "${ED%/}"/usr/share/info -mindepth 1 -type d -print0) + fi # If binpkg-docompress is enabled, apply compression before creating # the binary package. @@ -251,9 +268,6 @@ install_qa_check() { prepallstrip fi fi - - # Portage regenerates this on the installed system. - rm -f "${ED%/}"/usr/share/info/dir{,.Z,.gz,.bz2,.lzma,.lz,.xz,.zst} || die "rm failed!" } __dyn_instprep() { -- 2.40.0
Re: [gentoo-portage-dev] [PATCH] Remove obsolete FEATURES=force-prefix
Sounds fine to me too. On 22-01-2023 09:16:53 +, James Le Cuirot wrote: > On Sun, 2023-01-22 at 09:18 +0100, Ulrich Müller wrote: > > Signed-off-by: Ulrich Müller > > --- > > bin/dohtml.py| 6 ++ > > bin/eapi.sh | 4 ++-- > > lib/portage/const.py | 3 +-- > > lib/portage/package/ebuild/config.py | 11 +++ > > man/make.conf.5 | 8 +--- > > 5 files changed, 9 insertions(+), 23 deletions(-) > > +1 -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] Remove obsolete FEATURES=force-prefix
On Sun, 2023-01-22 at 09:18 +0100, Ulrich Müller wrote: > Signed-off-by: Ulrich Müller > --- > bin/dohtml.py| 6 ++ > bin/eapi.sh | 4 ++-- > lib/portage/const.py | 3 +-- > lib/portage/package/ebuild/config.py | 11 +++ > man/make.conf.5 | 8 +--- > 5 files changed, 9 insertions(+), 23 deletions(-) +1 signature.asc Description: This is a digitally signed message part
[gentoo-portage-dev] [PATCH] Remove obsolete FEATURES=force-prefix
Signed-off-by: Ulrich Müller --- bin/dohtml.py| 6 ++ bin/eapi.sh | 4 ++-- lib/portage/const.py | 3 +-- lib/portage/package/ebuild/config.py | 11 +++ man/make.conf.5 | 8 +--- 5 files changed, 9 insertions(+), 23 deletions(-) diff --git a/bin/dohtml.py b/bin/dohtml.py index 4ebee7779..d1f0e3140 100755 --- a/bin/dohtml.py +++ b/bin/dohtml.py @@ -1,5 +1,5 @@ #!/usr/bin/env python -# Copyright 1999-2020 Gentoo Authors +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # @@ -146,9 +146,7 @@ class OptionsClass: self.PF = os.environ["PF"] if self.PF: self.PF = normalize_path(self.PF) -if "force-prefix" not in os.environ.get( -"FEATURES", "" -).split() and os.environ.get("EAPI", "0") in ("0", "1", "2"): +if os.environ.get("EAPI", "0") in ("0", "1", "2"): self.ED = os.environ.get("D", "") else: self.ED = os.environ.get("ED", "") diff --git a/bin/eapi.sh b/bin/eapi.sh index d90c3cd2e..ae815f3a6 100644 --- a/bin/eapi.sh +++ b/bin/eapi.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Copyright 2012-2022 Gentoo Authors +# Copyright 2012-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # PHASES @@ -27,7 +27,7 @@ ___eapi_has_S_WORKDIR_fallback() { # VARIABLES ___eapi_has_prefix_variables() { - [[ ! ${1-${EAPI-0}} =~ ^(0|1|2)$ || " ${FEATURES} " == *" force-prefix "* ]] + [[ ! ${1-${EAPI-0}} =~ ^(0|1|2)$ ]] } ___eapi_has_BROOT() { diff --git a/lib/portage/const.py b/lib/portage/const.py index a6f7a2242..99206fe2c 100644 --- a/lib/portage/const.py +++ b/lib/portage/const.py @@ -1,5 +1,5 @@ # portage: Constants -# Copyright 1998-2021 Gentoo Authors +# Copyright 1998-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 import os @@ -156,7 +156,6 @@ SUPPORTED_FEATURES = frozenset( "fakeroot", "fixlafiles", "force-mirror", -"force-prefix", "getbinpkg", "gpg-keepalive", "icecream", diff --git a/lib/portage/package/ebuild/config.py b/lib/portage/package/ebuild/config.py index 23b524235..f66bf6485 100644 --- a/lib/portage/package/ebuild/config.py +++ b/lib/portage/package/ebuild/config.py @@ -1,4 +1,4 @@ -# Copyright 2010-2021 Gentoo Authors +# Copyright 2010-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 __all__ = [ @@ -3347,14 +3347,9 @@ class config: if not (src_like_phase and eapi_attrs.broot): mydict.pop("BROOT", None) -# Prefix variables are supported beginning with EAPI 3, or when -# force-prefix is in FEATURES, since older EAPIs would otherwise be -# useless with prefix configurations. This brings compatibility with -# the prefix branch of portage, which also supports EPREFIX for all -# EAPIs (for obvious reasons). +# Prefix variables are supported beginning with EAPI 3. if phase == "depend" or ( -"force-prefix" not in self.features -and eapi is not None +eapi is not None and not eapi_supports_prefix(eapi) ): mydict.pop("ED", None) diff --git a/man/make.conf.5 b/man/make.conf.5 index 34c3cd5b5..05832ce1c 100644 --- a/man/make.conf.5 +++ b/man/make.conf.5 @@ -1,4 +1,4 @@ -.TH "MAKE.CONF" "5" "Nov 2021" "Portage VERSION" "Portage" +.TH "MAKE.CONF" "5" "Jan 2023" "Portage VERSION" "Portage" .SH "NAME" make.conf \- custom settings for Portage .SH "SYNOPSIS" @@ -522,12 +522,6 @@ fixes (order of flags, duplicated entries, ...) Only fetch files from configured mirrors, ignoring \fBSRC_URI\fR, except when \fImirror\fR is in the \fBebuild\fR(5) \fBRESTRICT\fR variable. .TP -.B force\-prefix -Enable prefix support for all ebuilds, regardless of EAPI, since older EAPIs -would otherwise be useless with prefix configurations. This brings -compatibility with the prefix branch of portage, which also supports EPREFIX -for all EAPIs (for obvious reasons). -.TP .B getbinpkg Force emerges to always try to fetch files from the \fIPORTAGE_BINHOST\fR. See \fBmake.conf\fR(5) for more information. -- 2.39.1
Re: [gentoo-portage-dev] [PATCH 2/2] Add caching to _slot_operator_check_reverse_dependencies
On 11/24/22 19:36, Pin-yen Lin wrote: Add lru_cache to speed up the running time of "Calculating dependencies". In a ChromeOS use case, this patch decreases the running time from 311s to 197s with almost no memory usage increase. Signed-off-by: Pin-yen Lin --- lib/_emerge/depgraph.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py index ce6cabcc1..9649bb2a8 100644 --- a/lib/_emerge/depgraph.py +++ b/lib/_emerge/depgraph.py @@ -2240,6 +2240,7 @@ class depgraph: return None +@functools.lru_cache(maxsize=100) def _slot_operator_check_reverse_dependencies( self, existing_pkg, candidate_pkg, replacement_parent=None ): Merged. Thank you! https://gitweb.gentoo.org/proj/portage.git/commit/?id=0c42cc962e1926ecbdc83d903a2804f9e037f2a9 https://gitweb.gentoo.org/proj/portage.git/commit/?id=839ab46be1777e5886da28b98b53a462b992c5bf -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 2/2] Add caching to _slot_operator_check_reverse_dependencies
On Thu, Nov 24, 2022 at 10:36 PM Pin-yen Lin wrote: > > Add lru_cache to speed up the running time of "Calculating > dependencies". > > In a ChromeOS use case, this patch decreases the running time from > 311s to 197s with almost no memory usage increase. > > Signed-off-by: Pin-yen Lin Thank you! With recent subslot rebuilds (icu, boost, poppler), I measure an improvement of 19%! Benchmark 1: emerge @world -vuNDp Time (mean ± σ): 42.668 s ± 0.555 s[User: 42.095 s, System: 0.315 s] Range (min … max): 41.572 s … 43.342 s10 runs Benchmark 2: emerge @world -vuNDp Time (mean ± σ): 35.991 s ± 0.154 s[User: 35.409 s, System: 0.332 s] Range (min … max): 35.831 s … 36.306 s10 runs Summary 'emerge @world -vuNDp' ran 1.19 ± 0.02 times faster than 'emerge @world -vuNDp'
Re: [gentoo-portage-dev] sys-kernel/vanilla-sources not supported any more?
On Fri, Nov 25, 2022 at 5:50 AM Joakim Tjernlund wrote: > > I have noticed sys-kernel/vanilla-sources lagging behind for a while now and > I wonder if this pkg is obsolete? I think you sent this message to the wrong place. This list is for discussion of Portage development, not packages in the gentoo repository.
[gentoo-portage-dev] sys-kernel/vanilla-sources not supported any more?
I have noticed sys-kernel/vanilla-sources lagging behind for a while now and I wonder if this pkg is obsolete? Jocke
[gentoo-portage-dev] [PATCH 2/2] Add caching to _slot_operator_check_reverse_dependencies
Add lru_cache to speed up the running time of "Calculating dependencies". In a ChromeOS use case, this patch decreases the running time from 311s to 197s with almost no memory usage increase. Signed-off-by: Pin-yen Lin --- lib/_emerge/depgraph.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py index ce6cabcc1..9649bb2a8 100644 --- a/lib/_emerge/depgraph.py +++ b/lib/_emerge/depgraph.py @@ -2240,6 +2240,7 @@ class depgraph: return None +@functools.lru_cache(maxsize=100) def _slot_operator_check_reverse_dependencies( self, existing_pkg, candidate_pkg, replacement_parent=None ): -- 2.38.1.584.g0f3c55d4c2-goog
[gentoo-portage-dev] [PATCH 1/2] Move and rename check_reverse_dependencies
Move the function outside the original scope and rename the function to _slot_operator_check_reverse_dependencies to accommodate the coding style. Signed-off-by: Pin-yen Lin --- lib/_emerge/depgraph.py | 232 +--- 1 file changed, 119 insertions(+), 113 deletions(-) diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py index c976f8205..ce6cabcc1 100644 --- a/lib/_emerge/depgraph.py +++ b/lib/_emerge/depgraph.py @@ -2240,6 +2240,117 @@ class depgraph: return None +def _slot_operator_check_reverse_dependencies( +self, existing_pkg, candidate_pkg, replacement_parent=None +): +""" +Check if candidate_pkg satisfies all of existing_pkg's non- +slot operator parents. +""" +debug = "--debug" in self._frozen_config.myopts +built_slot_operator_parents = set() +for parent, atom in self._dynamic_config._parent_atoms.get( +existing_pkg, [] +): +if atom.soname or atom.slot_operator_built: +built_slot_operator_parents.add(parent) + +for parent, atom in self._dynamic_config._parent_atoms.get( +existing_pkg, [] +): +if isinstance(parent, Package): +if parent in built_slot_operator_parents: +if hasattr(atom, "_orig_atom"): +# If atom is the result of virtual expansion, then +# derefrence it to _orig_atom so that it will be correctly +# handled as a built slot operator dependency when +# appropriate (see bug 764764). +atom = atom._orig_atom +# This parent may need to be rebuilt, therefore +# discard its soname and built slot operator +# dependency components which are not necessarily +# relevant. +if atom.soname: +continue +elif atom.package and atom.slot_operator_built: +# This discards the slot/subslot component. +atom = atom.with_slot("=") + +if replacement_parent is not None and ( +replacement_parent.slot_atom == parent.slot_atom +or replacement_parent.cpv == parent.cpv +): +# This parent is irrelevant because we intend to +# replace it with replacement_parent. +continue + +if any( +pkg is not parent +and (pkg.slot_atom == parent.slot_atom or pkg.cpv == parent.cpv) +for pkg in self._dynamic_config._package_tracker.match( +parent.root, Atom(parent.cp) +) +): +# This parent may need to be eliminated due to a +# slot conflict, so its dependencies aren't +# necessarily relevant. +continue + +if not self._too_deep( +parent.depth +) and not self._frozen_config.excluded_pkgs.findAtomForPackage( +parent, modified_use=self._pkg_use_enabled(parent) +): +# Check for common reasons that the parent's +# dependency might be irrelevant. +if self._upgrade_available(parent): +# This parent could be replaced by +# an upgrade (bug 584626). +continue +if parent.installed and self._in_blocker_conflict(parent): +# This parent could be uninstalled in order +# to solve a blocker conflict (bug 612772). +continue +if self._dynamic_config.digraph.has_edge(parent, existing_pkg): +# There is a direct circular dependency between +# parent and existing_pkg. This type of +# relationship tends to prevent updates +# of packages (bug 612874). Since candidate_pkg +# is available, we risk a missed update if we +# don't try to eliminate this parent from the +# graph. Therefore, we give candidate_pkg a +# chance, and assume that it will be masked +# by backtracking if necessary. +continue + +atom_set = InternalPackageSet(initial_atoms=(atom,), allow_repo=True) +if not atom_set.findAtomForPackage( +candidate_pkg, modified_use=self._pkg_use_enabled(candidate_pkg) +): +if debug: +parent_atoms = [] +
Re: [gentoo-portage-dev] Fixing EAPI 8 --enable-static once and for all
On Sat, Nov 19, 2022 at 3:33 AM Ulrich Mueller wrote: > > > On Sat, 19 Nov 2022, David Seifert wrote: > > > With this extensive analysis, I believe this patch to be safe. > > Still looks like an incompatible change, so it will need an EAPI bump. > > EAPI 9 feature bug is here: https://bugs.gentoo.org/815169 I support this patch to fix Portage, regardless of what PMS says. Coding to the spec doesn't make sense if the spec is broken.
Re: [gentoo-portage-dev] Fixing EAPI 8 --enable-static once and for all
On Sat, 2022-11-19 at 09:33 +0100, Ulrich Mueller wrote: > > > > > > On Sat, 19 Nov 2022, David Seifert wrote: > > > With this extensive analysis, I believe this patch to be safe. > > Still looks like an incompatible change, so it will need an EAPI bump. > > EAPI 9 feature bug is here: https://bugs.gentoo.org/815169 I dare say we've done worse retroactive changes. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] Fixing EAPI 8 --enable-static once and for all
> On Sat, 19 Nov 2022, David Seifert wrote: > With this extensive analysis, I believe this patch to be safe. Still looks like an incompatible change, so it will need an EAPI bump. EAPI 9 feature bug is here: https://bugs.gentoo.org/815169 signature.asc Description: PGP signature
[gentoo-portage-dev] Fixing EAPI 8 --enable-static once and for all
Attached a patch to finally fix https://bugs.gentoo.org/814380. When I opened the bug, it was clear that I intended to only disable static libtool archive building. Unfortunately, I hadn't done the necessary due diligence, and it turns out how we detect the "--enable-static" flag is overly broad and leads to a sizable number of false positives in EAPI 8. I have 937 EAPI 8 packages on my machine, and in 0 cases does my patch now produce static libraries where none were produced before, while it successfully reduces the annoying QA warnings on packages that don't use libtool or use the flag for different purposes: 1. packages that have an "off-target"/false-positive flag triggering the current logic: dev-libs/nspr-4.35 --enable-static-rtl Use the MSVC static runtime library sys-fs/lvm2-2.03.16 --enable-static_linkuse this to link the tools to their libraries 2. packages with a real --{dis,en}able-static flag, but with semantics that don't actually disable building of static-libs: dev-util/strace-5.19 --enable-static link strace statically sys-process/htop-3.2.1 --enable-static build a static htop binary [default=no] media-video/mkvtoolnix-71.1.0 --enable-static make a static build of the applications (no) --enable-static-qt link to static versions of the Qt library (no) dev-util/strace has a $(use_enable static) option, and the other two are default disabled. Hence in all of these cases, the patch does not change the build output. 3. packages with a --{dis,en}able-static flag that controls building of static libraries, but with a slightly different --help output that doesn't trigger the updated glob anymore: dev-libs/icu-72.1 --enable-static build static libraries default=no dev-libs/nettle-3.8.1 --disable-staticDo not build any static library net-print/cups-2.4.2 --enable-static install static libraries All of these packages pass a $(use_enable static-libs static) option, hence none of them would be affected by the missing --disable-static. With this extensive analysis, I believe this patch to be safe.
[gentoo-portage-dev] [PATCH] Make EAPI 8 `--disable-static` logic libtool-specific
* The intention has always been to only target `configure` scripts that use libtool, not just any script with a `--disable-static*` option. * libtool has been using the same `configure` format for at least the past 15 years (going back to libtool 1.5.22): 1. shared and static libraries enabled (the main use case): --enable-shared[=PKGS] build shared libraries [default=yes] --enable-static[=PKGS] build static libraries [default=yes] 2. shared libraries enabled and static libraries disabled: --enable-static[=PKGS] build static libraries [default=no] --enable-shared[=PKGS] build shared libraries [default=yes] 3. shared libraries disabled and static libraries enabled: --enable-shared[=PKGS] build shared libraries [default=no] --enable-static[=PKGS] build static libraries [default=yes] Bug: https://bugs.gentoo.org/814380 --- bin/phase-helpers.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh index 2217e5a0b..6f691f6ff 100644 --- a/bin/phase-helpers.sh +++ b/bin/phase-helpers.sh @@ -655,8 +655,8 @@ econf() { fi if ___eapi_econf_passes_--disable-static; then - if [[ ${conf_help} == *--disable-static* || \ - ${conf_help} == *--enable-static* ]]; then + if [[ ${conf_help} == *--enable-shared\[=PKGS\]* && + ${conf_help} == *--enable-static\[=PKGS\]* ]]; then conf_args+=( --disable-static ) fi fi -- 2.38.1
Re: [gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
On 28-07-2022 20:11:24 +0200, Fabian Groffen wrote: > On 28-07-2022 19:57:50 +0200, Arfrever Frehtes Taifersar Arahesis wrote: > > But this code starts with 'if [[ -z ${USERLAND} ]]', so it should not > > be run when USERLAND="GNU". > > > > USERLAND="GNU" is set in make.defaults, which is parsed by Portage > > early (at Python side) and these values are passed by Portage to > > ebuild.sh subprocess. > > Hmmm, that would be nice, features/prefix/make.defaults includes setting > that. > > Makes me still wonder why we do this, is it a fallback for when the > profiles aren't available (no tree yet) or something? > > It exports XARGS, which only used in the ___parallel_xargs function. > That function in turn is used in 2 places. So all this stuff is done to > add -r to xargs if xargs isn't BSD xargs. Iow --no-run-if-empty, which > is cool, but apparently not strictly necessary (fugly warning I guess at > worst). We're already checking xargs to support --max-procs=, so we > could in the same go check for support of --no-run-if-empty, and drop > the whole XARGS thing and declutter the code quite a lot :) Sorry, I grepped wrong. XARGS is used throughout the code, so it needs to be initialised in isolated-functions.sh. It could be initialised by simply probing instead of relying on uname though. As for USERLAND, emerge-webrsync and delta-webrsync appear to use it, but they retrieve it from portageq, so I guess USERLAND is not really used in Portage itself. Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
On 28-07-2022 19:57:50 +0200, Arfrever Frehtes Taifersar Arahesis wrote: > 2022-07-28 17:47 UTCに、Fabian Groffen は書いた: > > bin/isolated-functions.sh does the following bit: > > > > if [[ -z ${USERLAND} ]] ; then > >case $(uname -s) in > >*BSD|DragonFly) > >export USERLAND="BSD" > >;; > >*) > >export USERLAND="GNU" > >;; > >esac > > fi > > > > (after which it uses USERLAND for a single purpose, to export XARGS to > > either GNU '[g]xargs -r', or BSD 'xargs') > > > > This bit is problematic for Prefix, because Prefix may run on *BSD, but > > use USERLAND=GNU. > > But this code starts with 'if [[ -z ${USERLAND} ]]', so it should not > be run when USERLAND="GNU". > > USERLAND="GNU" is set in make.defaults, which is parsed by Portage > early (at Python side) and these values are passed by Portage to > ebuild.sh subprocess. Hmmm, that would be nice, features/prefix/make.defaults includes setting that. Makes me still wonder why we do this, is it a fallback for when the profiles aren't available (no tree yet) or something? It exports XARGS, which only used in the ___parallel_xargs function. That function in turn is used in 2 places. So all this stuff is done to add -r to xargs if xargs isn't BSD xargs. Iow --no-run-if-empty, which is cool, but apparently not strictly necessary (fugly warning I guess at worst). We're already checking xargs to support --max-procs=, so we could in the same go check for support of --no-run-if-empty, and drop the whole XARGS thing and declutter the code quite a lot :) Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
2022-07-28 17:47 UTCに、Fabian Groffen は書いた: > bin/isolated-functions.sh does the following bit: > > if [[ -z ${USERLAND} ]] ; then >case $(uname -s) in >*BSD|DragonFly) >export USERLAND="BSD" >;; >*) >export USERLAND="GNU" >;; >esac > fi > > (after which it uses USERLAND for a single purpose, to export XARGS to > either GNU '[g]xargs -r', or BSD 'xargs') > > This bit is problematic for Prefix, because Prefix may run on *BSD, but > use USERLAND=GNU. But this code starts with 'if [[ -z ${USERLAND} ]]', so it should not be run when USERLAND="GNU". USERLAND="GNU" is set in make.defaults, which is parsed by Portage early (at Python side) and these values are passed by Portage to ebuild.sh subprocess.
[gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
Hi, bin/isolated-functions.sh does the following bit: if [[ -z ${USERLAND} ]] ; then case $(uname -s) in *BSD|DragonFly) export USERLAND="BSD" ;; *) export USERLAND="GNU" ;; esac fi (after which it uses USERLAND for a single purpose, to export XARGS to either GNU '[g]xargs -r', or BSD 'xargs') This bit is problematic for Prefix, because Prefix may run on *BSD, but use USERLAND=GNU. The problem is theoretical at this point, occasionally people come in and want to use Prefix on *BSD again. Now I think Gentoo/BSD is theoretical too. It officially is no longer maintained[1]. So, question here is, do we want to retain this bit of historical compatibility work (also check things like gsed wrapper), or can it go, basically removing a potential problem for Gentoo Prefix? [1] https://wiki.gentoo.org/wiki/Gentoo_FreeBSD -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
2022-07-26 07:03 UTCに、Florian Schmaus は書いた: > But then I wondered if "make SHELL=$BROOT/bin/sh" wouldn't override > explicitly set SHELL values in Makefiles. Assume a package has > > SHELL = /bin/zsh > > in one of its Makefiles. Then emake would reset this to 'sh'. Which > appears like it could cause build issues. > > If this is the case, then I am not sure what we can do about it. It > appears fragile, if not impossible, to ask 'make' which value for SHELL > it would assume, so that emake could adjust the path. Another option > could be that affected packages define a variable in their ebuild, e.g. > EMAKE_SHELL="zsh", which emake could extend with BROOT before passing > the resulting value as SHELL to make. If there was such package, it could just override SHELL on emake invocation: src_compile() { emake SHELL="${BROOT}/bin/zsh" # Or: emake SHELL="$(type -P zsh)" }
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
On 26-07-2022 09:20:58 +0200, Fabian Groffen wrote: > On 26-07-2022 09:03:18 +0200, Florian Schmaus wrote: > > On 26.07.22 05:00, Sam James wrote: > > >> On 25 Jul 2022, at 16:28, Fabian Groffen wrote: > > >> > > >> bin/ebuild-helpers/emake: force SHELL to be set > > >> > [snip] > > >> > > >> diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake > > >> index 60718a2e4..21da85845 100755 > > >> --- a/bin/ebuild-helpers/emake > > >> +++ b/bin/ebuild-helpers/emake > > >> @@ -12,7 +12,7 @@ > > >> source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1 > > >> > > >> cmd=( > > >> -${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE} > > >> +${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" > > >> ${EXTRA_EMAKE} > > >> ) > > >> > > >> if [[ ${PORTAGE_QUIET} != 1 ]] ; then > > >> > > > > > > I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh > > > to avoid using > > > an ancient host sh? > > > > I was about to write the same (also using EPREFIX, but EBROOT seems what > > you want, as you figured). > > > > But then I wondered if "make SHELL=$BROOT/bin/sh" wouldn't override > > explicitly set SHELL values in Makefiles. Assume a package has > > > > SHELL = /bin/zsh > > > > in one of its Makefiles. Then emake would reset this to 'sh'. Which > > appears like it could cause build issues. > > > > If this is the case, then I am not sure what we can do about it. It > > appears fragile, if not impossible, to ask 'make' which value for SHELL > > it would assume, so that emake could adjust the path. Another option > > could be that affected packages define a variable in their ebuild, e.g. > > EMAKE_SHELL="zsh", which emake could extend with BROOT before passing > > the resulting value as SHELL to make. > > So, I can also envision we drop this patch, and I see if I can patch > make(1) to use $EPREFIX/bin/sh instead of /bin/sh by default. Not sure, > but this would retain the behaviour Portage is doing now for non-Prefix, > and would get the behaviour we want in Prefix. > > On an alternative note, there is CONFIG_SHELL (used for setting which shell > to use with configure), which I think in many cases bleeds through to > make, but should there be a MAKE_SHELL perhaps as well? Then the > default would be pretty much ok. > > (We never ran into any problems forcing SHELL to bash in Prefix, but > perhaps that's not a representative test for the whole of Gentoo.) I've been looking around in make, and it seems we can set a default shell there, would be pretty simple, I guess. It doesn't solve, however, a bigger problem, which is what make's source also mentions, lots of Makefiles having SHELL=/bin/sh. In other words, setting a default in make makes no difference in preventing it from using /bin/sh (which is the main aim here). Therefore, I would like to consider the possibility to override SHELL this way via emake, as it seems like the fix we need (for Prefix at least) afterall. Overriding should be safe, I think. SHELL=/bin/zsh makes little sense to me, SHELL=/bin/csh would be more of a thing, but is there any package out there that needs it? And if it does, wouldn't it be acceptable to just handle that in that package? It would require a dep on that shell anyway (so you could consider it a kludgy sanity check as well). I think using bash like the original patch did isn't quite nice, it should use sh instead. However, it cannot hardcode EPREFIX/bin/sh without ensuring that binary exists, else we break bootstraps. So I was thinking: how about something like SHELL=$(type -P sh)? To be completely safe here, it could become an if such that if there's no sh, it falls back to ${BASH}. Would this approach be acceptable? Should it be perhaps conditional based on whether an offset prefix is active? Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] phase-functions: make ED, EROOT read-only
On 25-07-2022 19:43:21 -0400, Mike Gilbert wrote: > On Mon, Jul 25, 2022 at 1:03 PM Fabian Groffen wrote: > > > > bin/phase-functions.sh: make ED and EROOT read-only too > > > > Like D, make ED and EROOT read-only vars. > > Makes sense. https://github.com/gentoo/portage/pull/870 Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] misc-functions: Prefix fixes
On 25-07-2022 19:42:51 -0400, Mike Gilbert wrote: > On Mon, Jul 25, 2022 at 12:47 PM Fabian Groffen wrote: > > > > bin/misc-functions.sh: some Prefix fixes > > > > - ED needs not to exist, whereas D does, so ensure we check for that, > > and create ED if absent, necessary for further checks to succeed > > - use EPREFIX in INSTALL_MASK > > Seems good to me. https://github.com/gentoo/portage/pull/870 -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] world-writable: tune for Prefix
On 25-07-2022 19:39:38 -0400, Mike Gilbert wrote: > On Mon, Jul 25, 2022 at 12:41 PM Fabian Groffen wrote: > > > > bin/install-qa-check.d/90world-writable: include EPREFIX in reports > > > > It is much less confusing and consistent to report full paths including > > the leading EPREFIX. > > Makes sense to me. https://github.com/gentoo/portage/pull/869 thanks -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] multilib-strict: fix for Prefix
On 25-07-2022 19:38:57 -0400, Mike Gilbert wrote: > On Mon, Jul 25, 2022 at 12:26 PM Fabian Groffen wrote: > > > > bin/install-qa-check.d/80multilib-strict: use file/find from Prefix > > > > diff --git a/bin/install-qa-check.d/80multilib-strict > > b/bin/install-qa-check.d/80multilib-strict > > index afd223250..3db4ecce3 100644 > > --- a/bin/install-qa-check.d/80multilib-strict > > +++ b/bin/install-qa-check.d/80multilib-strict > > @@ -1,7 +1,7 @@ > > # Strict multilib directory checks > > multilib_strict_check() { > > if has multilib-strict ${FEATURES} && \ > > - [[ -x /usr/bin/file && -x /usr/bin/find ]] && \ > > + [[ -x ${EPREFIX}/usr/bin/file && -x ${EPREFIX}/usr/bin/find ]] > > && \ > >[[ -n ${MULTILIB_STRICT_DIRS} && -n ${MULTILIB_STRICT_DENY} ]] > > then > > rm -f "${T}/multilib-strict.log" > > Seems good to me. https://github.com/gentoo/portage/pull/868 Thanks -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets
On 26-07-2022 04:01:37 +0100, Sam James wrote: > > > > On 26 Jul 2022, at 00:33, Mike Gilbert wrote: > > > > On Mon, Jul 25, 2022 at 11:38 AM Fabian Groffen wrote: > >> > >> bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects > >> > >> Check for dylib on Darwin, so on everything else. > >> > >> Signed-off-by: Fabian Groffen > >> > >> diff --git a/bin/install-qa-check.d/80libraries > >> b/bin/install-qa-check.d/80libraries > >> index 8dc35bb87..a477ec9cb 100644 > >> --- a/bin/install-qa-check.d/80libraries > >> +++ b/bin/install-qa-check.d/80libraries > >> @@ -140,7 +140,9 @@ lib_check() { > >>local abort="no" > >>local a s > >>for a in "${ED%/}"/usr/lib*/*.a ; do > >> - s=${a%.a}.so > >> + [[ ${CHOST} == *-darwin* ]] \ > >> + && s=${a%.a}.dylib \ > >> + || s=${a%.a}.so > > > > I would find this much easier to read if you converted it to an > > if/else statement instead of chaining && and ||. > > Yes, please. https://github.com/gentoo/portage/pull/867 Thanks -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
On 25-07-2022 19:29:35 -0400, Mike Gilbert wrote: > Could you please create a PR at https://github.com/gentoo/portage so > that the CI system can test the changes for this patch series? https://github.com/gentoo/portage/pull/866 Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
On 25-07-2022 19:50:41 +0200, Ulrich Mueller wrote: > > On Mon, 25 Jul 2022, Fabian Groffen wrote: > > > @@ -50,6 +51,16 @@ if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" > > ]]; then > > __helpers_die "${helper} used with \${D} or \${ED}" > > exit 1 > > fi > > +if [[ -n ${EPREFIX} && \ > > + ${_E_INSDESTTREE_#${EPREFIX}} != "${_E_INSDESTTREE_}" ]]; > > The semicolon is redundant. removed, thanks -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
On 26-07-2022 09:03:18 +0200, Florian Schmaus wrote: > On 26.07.22 05:00, Sam James wrote: > >> On 25 Jul 2022, at 16:28, Fabian Groffen wrote: > >> > >> bin/ebuild-helpers/emake: force SHELL to be set > >> [snip] > >> > >> diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake > >> index 60718a2e4..21da85845 100755 > >> --- a/bin/ebuild-helpers/emake > >> +++ b/bin/ebuild-helpers/emake > >> @@ -12,7 +12,7 @@ > >> source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1 > >> > >> cmd=( > >> - ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE} > >> + ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE} > >> ) > >> > >> if [[ ${PORTAGE_QUIET} != 1 ]] ; then > >> > > > > I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh to > > avoid using > > an ancient host sh? > > I was about to write the same (also using EPREFIX, but EBROOT seems what > you want, as you figured). > > But then I wondered if "make SHELL=$BROOT/bin/sh" wouldn't override > explicitly set SHELL values in Makefiles. Assume a package has > > SHELL = /bin/zsh > > in one of its Makefiles. Then emake would reset this to 'sh'. Which > appears like it could cause build issues. > > If this is the case, then I am not sure what we can do about it. It > appears fragile, if not impossible, to ask 'make' which value for SHELL > it would assume, so that emake could adjust the path. Another option > could be that affected packages define a variable in their ebuild, e.g. > EMAKE_SHELL="zsh", which emake could extend with BROOT before passing > the resulting value as SHELL to make. So, I can also envision we drop this patch, and I see if I can patch make(1) to use $EPREFIX/bin/sh instead of /bin/sh by default. Not sure, but this would retain the behaviour Portage is doing now for non-Prefix, and would get the behaviour we want in Prefix. On an alternative note, there is CONFIG_SHELL (used for setting which shell to use with configure), which I think in many cases bleeds through to make, but should there be a MAKE_SHELL perhaps as well? Then the default would be pretty much ok. (We never ran into any problems forcing SHELL to bash in Prefix, but perhaps that's not a representative test for the whole of Gentoo.) Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
On 26.07.22 05:00, Sam James wrote: On 25 Jul 2022, at 16:28, Fabian Groffen wrote: bin/ebuild-helpers/emake: force SHELL to be set On Prefix systems /bin/sh can be anything, including very ancient. So ensure we're running with bash, since that's what Gentoo Linux is expecting /bin/sh to be (by default, at least). Provide a fallback for the (near impossible) case that we use a bash that doesn't set BASH, or when we don't use bash at all. This is not expected, though, as we explicitly require bash throughout all Portage, so we don't really care about using a non-Prefixed one, for this really shouldn't happen. Signed-off-by: Fabian Groffen diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake index 60718a2e4..21da85845 100755 --- a/bin/ebuild-helpers/emake +++ b/bin/ebuild-helpers/emake @@ -12,7 +12,7 @@ source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1 cmd=( - ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE} + ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE} ) if [[ ${PORTAGE_QUIET} != 1 ]] ; then I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh to avoid using an ancient host sh? I was about to write the same (also using EPREFIX, but EBROOT seems what you want, as you figured). But then I wondered if "make SHELL=$BROOT/bin/sh" wouldn't override explicitly set SHELL values in Makefiles. Assume a package has SHELL = /bin/zsh in one of its Makefiles. Then emake would reset this to 'sh'. Which appears like it could cause build issues. If this is the case, then I am not sure what we can do about it. It appears fragile, if not impossible, to ask 'make' which value for SHELL it would assume, so that emake could adjust the path. Another option could be that affected packages define a variable in their ebuild, e.g. EMAKE_SHELL="zsh", which emake could extend with BROOT before passing the resulting value as SHELL to make. - Flow
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
> On 26 Jul 2022, at 04:00, Sam James wrote: > > > >> On 25 Jul 2022, at 16:28, Fabian Groffen wrote: >> >> bin/ebuild-helpers/emake: force SHELL to be set >> >> On Prefix systems /bin/sh can be anything, including very ancient. So >> ensure we're running with bash, since that's what Gentoo Linux is >> expecting /bin/sh to be (by default, at least). >> >> Provide a fallback for the (near impossible) case that we use a bash >> that doesn't set BASH, or when we don't use bash at all. This is not >> expected, though, as we explicitly require bash throughout all Portage, >> so we don't really care about using a non-Prefixed one, for this really >> shouldn't happen. >> >> Signed-off-by: Fabian Groffen >> >> diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake >> index 60718a2e4..21da85845 100755 >> --- a/bin/ebuild-helpers/emake >> +++ b/bin/ebuild-helpers/emake >> @@ -12,7 +12,7 @@ >> source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1 >> >> cmd=( >> -${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE} >> +${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE} >> ) >> >> if [[ ${PORTAGE_QUIET} != 1 ]] ; then >> > > I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh to > avoid using > an ancient host sh? > Sorry, ${BROOT}, I guess. > I might use Bash for Portage but my /bin/sh is dash usually. > >> -- >> Fabian Groffen >> Gentoo on a different level signature.asc Description: Message signed with OpenPGP
Re: [gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets
> On 26 Jul 2022, at 00:33, Mike Gilbert wrote: > > On Mon, Jul 25, 2022 at 11:38 AM Fabian Groffen wrote: >> >> bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects >> >> Check for dylib on Darwin, so on everything else. >> >> Signed-off-by: Fabian Groffen >> >> diff --git a/bin/install-qa-check.d/80libraries >> b/bin/install-qa-check.d/80libraries >> index 8dc35bb87..a477ec9cb 100644 >> --- a/bin/install-qa-check.d/80libraries >> +++ b/bin/install-qa-check.d/80libraries >> @@ -140,7 +140,9 @@ lib_check() { >>local abort="no" >>local a s >>for a in "${ED%/}"/usr/lib*/*.a ; do >> - s=${a%.a}.so >> + [[ ${CHOST} == *-darwin* ]] \ >> + && s=${a%.a}.dylib \ >> + || s=${a%.a}.so > > I would find this much easier to read if you converted it to an > if/else statement instead of chaining && and ||. Yes, please. signature.asc Description: Message signed with OpenPGP
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
> On 25 Jul 2022, at 16:28, Fabian Groffen wrote: > > bin/ebuild-helpers/emake: force SHELL to be set > > On Prefix systems /bin/sh can be anything, including very ancient. So > ensure we're running with bash, since that's what Gentoo Linux is > expecting /bin/sh to be (by default, at least). > > Provide a fallback for the (near impossible) case that we use a bash > that doesn't set BASH, or when we don't use bash at all. This is not > expected, though, as we explicitly require bash throughout all Portage, > so we don't really care about using a non-Prefixed one, for this really > shouldn't happen. > > Signed-off-by: Fabian Groffen > > diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake > index 60718a2e4..21da85845 100755 > --- a/bin/ebuild-helpers/emake > +++ b/bin/ebuild-helpers/emake > @@ -12,7 +12,7 @@ > source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1 > > cmd=( > - ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE} > + ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE} > ) > > if [[ ${PORTAGE_QUIET} != 1 ]] ; then > I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh to avoid using an ancient host sh? I might use Bash for Portage but my /bin/sh is dash usually. > -- > Fabian Groffen > Gentoo on a different level signature.asc Description: Message signed with OpenPGP
Re: [gentoo-portage-dev] [PATCH] phase-functions: make ED, EROOT read-only
On Mon, Jul 25, 2022 at 1:03 PM Fabian Groffen wrote: > > bin/phase-functions.sh: make ED and EROOT read-only too > > Like D, make ED and EROOT read-only vars. Makes sense.
Re: [gentoo-portage-dev] [PATCH] misc-functions: Prefix fixes
On Mon, Jul 25, 2022 at 12:47 PM Fabian Groffen wrote: > > bin/misc-functions.sh: some Prefix fixes > > - ED needs not to exist, whereas D does, so ensure we check for that, > and create ED if absent, necessary for further checks to succeed > - use EPREFIX in INSTALL_MASK Seems good to me.
Re: [gentoo-portage-dev] [PATCH] world-writable: tune for Prefix
On Mon, Jul 25, 2022 at 12:41 PM Fabian Groffen wrote: > > bin/install-qa-check.d/90world-writable: include EPREFIX in reports > > It is much less confusing and consistent to report full paths including > the leading EPREFIX. Makes sense to me.
Re: [gentoo-portage-dev] [PATCH] multilib-strict: fix for Prefix
On Mon, Jul 25, 2022 at 12:26 PM Fabian Groffen wrote: > > bin/install-qa-check.d/80multilib-strict: use file/find from Prefix > > diff --git a/bin/install-qa-check.d/80multilib-strict > b/bin/install-qa-check.d/80multilib-strict > index afd223250..3db4ecce3 100644 > --- a/bin/install-qa-check.d/80multilib-strict > +++ b/bin/install-qa-check.d/80multilib-strict > @@ -1,7 +1,7 @@ > # Strict multilib directory checks > multilib_strict_check() { > if has multilib-strict ${FEATURES} && \ > - [[ -x /usr/bin/file && -x /usr/bin/find ]] && \ > + [[ -x ${EPREFIX}/usr/bin/file && -x ${EPREFIX}/usr/bin/find ]] && \ >[[ -n ${MULTILIB_STRICT_DIRS} && -n ${MULTILIB_STRICT_DENY} ]] > then > rm -f "${T}/multilib-strict.log" Seems good to me.
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
On Mon, Jul 25, 2022 at 11:28 AM Fabian Groffen wrote: > > bin/ebuild-helpers/emake: force SHELL to be set > > On Prefix systems /bin/sh can be anything, including very ancient. So > ensure we're running with bash, since that's what Gentoo Linux is > expecting /bin/sh to be (by default, at least). > > Provide a fallback for the (near impossible) case that we use a bash > that doesn't set BASH, or when we don't use bash at all. This is not > expected, though, as we explicitly require bash throughout all Portage, > so we don't really care about using a non-Prefixed one, for this really > shouldn't happen. I'm a little on the fence about this: in theory, Makefiles should use POSIX-compatible shell commands unless the author explicitly chooses to use bash. I guess we can get away with this since ebuilds always require bash anyway.
Re: [gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets
On Mon, Jul 25, 2022 at 11:38 AM Fabian Groffen wrote: > > bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects > > Check for dylib on Darwin, so on everything else. > > Signed-off-by: Fabian Groffen > > diff --git a/bin/install-qa-check.d/80libraries > b/bin/install-qa-check.d/80libraries > index 8dc35bb87..a477ec9cb 100644 > --- a/bin/install-qa-check.d/80libraries > +++ b/bin/install-qa-check.d/80libraries > @@ -140,7 +140,9 @@ lib_check() { > local abort="no" > local a s > for a in "${ED%/}"/usr/lib*/*.a ; do > - s=${a%.a}.so > + [[ ${CHOST} == *-darwin* ]] \ > + && s=${a%.a}.dylib \ > + || s=${a%.a}.so I would find this much easier to read if you converted it to an if/else statement instead of chaining && and ||.
Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
Could you please create a PR at https://github.com/gentoo/portage so that the CI system can test the changes for this patch series?
Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
> On Mon, 25 Jul 2022, Fabian Groffen wrote: > @@ -50,6 +51,16 @@ if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" ]]; > then > __helpers_die "${helper} used with \${D} or \${ED}" > exit 1 > fi > +if [[ -n ${EPREFIX} && \ > + ${_E_INSDESTTREE_#${EPREFIX}} != "${_E_INSDESTTREE_}" ]]; The semicolon is redundant. > +then > + __vecho "---" 1>&2 > + __vecho "You should not use \${EPREFIX} with helpers." 1>&2 > + __vecho " --> ${_E_INSDESTTREE_}" 1>&2 > + __vecho "---" 1>&2 > + __helpers_die "${helper} used with \${EPREFIX}" > + exit 1 > +fi signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] phase-functions: make ED, EROOT read-only
bin/phase-functions.sh: make ED and EROOT read-only too Like D, make ED and EROOT read-only vars. Signed-off-by: Fabian Groffen diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh index ccf7eeea7..212b19fc1 100644 --- a/bin/phase-functions.sh +++ b/bin/phase-functions.sh @@ -12,7 +12,7 @@ PORTAGE_READONLY_METADATA="BDEPEND DEFINED_PHASES DEPEND DESCRIPTION PDEPEND RDEPEND REPOSITORY RESTRICT SLOT SRC_URI" PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE EBUILD_PHASE_FUNC \ - EBUILD_SH_ARGS EMERGE_FROM FILESDIR MERGE_TYPE \ + EBUILD_SH_ARGS ED EMERGE_FROM EROOT FILESDIR MERGE_TYPE \ PM_EBUILD_HOOK_DIR \ PORTAGE_ACTUAL_DISTDIR PORTAGE_ARCHLIST PORTAGE_BASHRC \ PORTAGE_BINPKG_FILE PORTAGE_BINPKG_TAR_OPTS PORTAGE_BINPKG_TMPFILE \ -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] misc-functions: Prefix fixes
bin/misc-functions.sh: some Prefix fixes - ED needs not to exist, whereas D does, so ensure we check for that, and create ED if absent, necessary for further checks to succeed - use EPREFIX in INSTALL_MASK Signed-off-by: Fabian Groffen diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh index c8bac08e7..8fcc23588 100755 --- a/bin/misc-functions.sh +++ b/bin/misc-functions.sh @@ -19,6 +19,8 @@ source "${PORTAGE_BIN_PATH}/ebuild.sh" || exit 1 install_symlink_html_docs() { if ! ___eapi_has_prefix_variables; then local ED=${D} + else + [[ ! -d ${ED} && -d ${D} ]] && dodir / fi cd "${ED}" || die "cd failed" # Symlink the html documentation (if DOC_SYMLINKS_DIR is set in make.conf) @@ -83,7 +87,7 @@ install_qa_check() { local EPREFIX= ED=${D} fi - cd "${ED}" || die "cd failed" + cd "${D}" || die "cd failed" # Collect the paths for QA checks, highest prio first. paths=( @@ -367,7 +718,7 @@ preinst_mask() { local f x for f in man info doc; do if has no${f} ${FEATURES}; then - INSTALL_MASK+=" /usr/share/${f}" + INSTALL_MASK+=" ${EPREFIX}/usr/share/${f}" fi done -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] world-writable: tune for Prefix
bin/install-qa-check.d/90world-writable: include EPREFIX in reports It is much less confusing and consistent to report full paths including the leading EPREFIX. Signed-off-by: Fabian Groffen diff --git a/bin/install-qa-check.d/90world-writable b/bin/install-qa-check.d/90world-writable index 820683bd6..90b961a86 100644 --- a/bin/install-qa-check.d/90world-writable +++ b/bin/install-qa-check.d/90world-writable @@ -2,7 +2,7 @@ world_writable_check() { # Now we look for all world writable files. - local unsafe_files=$(find "${ED}" -type f -perm -2 | sed -e "s:^${ED}:/:") + local unsafe_files=$(find "${ED}" -type f -perm -2 | sed -e "s:^${D}:/:") local OLDIFS x prev_shopts=$- OLDIFS=$IFS @@ -19,7 +21,7 @@ world_writable_check() { eqawarn fi - local unsafe_files=$(find "${ED}" -type f '(' -perm -2002 -o -perm -4002 ')' | sed -e "s:^${ED}:/:") + local unsafe_files=$(find "${ED}" -type f '(' -perm -2002 -o -perm -4002 ')' | sed -e "s:^${D}:/:") if [[ -n ${unsafe_files} ]] ; then eqawarn "QA Notice: Unsafe files detected (set*id and world writable)" -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] multilib-strict: fix for Prefix
bin/install-qa-check.d/80multilib-strict: use file/find from Prefix diff --git a/bin/install-qa-check.d/80multilib-strict b/bin/install-qa-check.d/80multilib-strict index afd223250..3db4ecce3 100644 --- a/bin/install-qa-check.d/80multilib-strict +++ b/bin/install-qa-check.d/80multilib-strict @@ -1,7 +1,7 @@ # Strict multilib directory checks multilib_strict_check() { if has multilib-strict ${FEATURES} && \ - [[ -x /usr/bin/file && -x /usr/bin/find ]] && \ + [[ -x ${EPREFIX}/usr/bin/file && -x ${EPREFIX}/usr/bin/find ]] && \ [[ -n ${MULTILIB_STRICT_DIRS} && -n ${MULTILIB_STRICT_DENY} ]] then rm -f "${T}/multilib-strict.log" -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets
bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects Check for dylib on Darwin, so on everything else. Signed-off-by: Fabian Groffen diff --git a/bin/install-qa-check.d/80libraries b/bin/install-qa-check.d/80libraries index 8dc35bb87..a477ec9cb 100644 --- a/bin/install-qa-check.d/80libraries +++ b/bin/install-qa-check.d/80libraries @@ -140,7 +140,9 @@ lib_check() { local abort="no" local a s for a in "${ED%/}"/usr/lib*/*.a ; do - s=${a%.a}.so + [[ ${CHOST} == *-darwin* ]] \ + && s=${a%.a}.dylib \ + || s=${a%.a}.so if [[ ! -e ${s} ]] ; then s=${s%usr/*}${s##*/usr/} if [[ -e ${s} ]] ; then -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
bin/ebuild-helpers/emake: force SHELL to be set On Prefix systems /bin/sh can be anything, including very ancient. So ensure we're running with bash, since that's what Gentoo Linux is expecting /bin/sh to be (by default, at least). Provide a fallback for the (near impossible) case that we use a bash that doesn't set BASH, or when we don't use bash at all. This is not expected, though, as we explicitly require bash throughout all Portage, so we don't really care about using a non-Prefixed one, for this really shouldn't happen. Signed-off-by: Fabian Groffen diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake index 60718a2e4..21da85845 100755 --- a/bin/ebuild-helpers/emake +++ b/bin/ebuild-helpers/emake @@ -12,7 +12,7 @@ source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1 cmd=( - ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE} + ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE} ) if [[ ${PORTAGE_QUIET} != 1 ]] ; then -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] usage of /bin/bash in shebangs
On 25-07-2022 08:09:02 -0700, Zac Medico wrote: > On 7/24/22 23:17, Fabian Groffen wrote: > > On 24-07-2022 13:58:31 -0700, Zac Medico wrote: > >> On 7/24/22 12:29, Fabian Groffen wrote: > >>> Hi, > >>> > >>> Quick question, I noticed that portage uses /bin/bash hardcoded in > >>> shebang of scripts, while it uses /usr/bin/env python for python > >>> executable files. > >>> > >>> Is there anything against using /usr/bin/env bash for shell scripts? > >>> Changing this would help for Prefix. > >> > >> > >> I can't think of any reason not to do this. > > > > Do you want to see a patch, or is it OK to push this myself? > > It's OK to just push this yourself. Thanks! Pushed, thanks, b716e3591 -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
bin/ebuild-helpers/doins: fix D check, add EPREFIX check ED = D/EPREFIX, so checking for ED includes EPREFIX, which when this is absent fails to check for D. Simply check for D instead, which catches both the case for D and ED. Add check for usage of EPREFIX, like for using D with helpers. Signed-off-by: Fabian Groffen diff --git a/bin/ebuild-helpers/doins b/bin/ebuild-helpers/doins index 24fe48121..4315a038f 100755 --- a/bin/ebuild-helpers/doins +++ b/bin/ebuild-helpers/doins @@ -42,7 +42,7 @@ if ! ___eapi_has_prefix_variables; then export ED="${D}" fi -if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" ]]; then +if [[ ${_E_INSDESTTREE_#${D}} != "${_E_INSDESTTREE_}" ]]; then __vecho "---" 1>&2 __vecho "You should not use \${D} or \${ED} with helpers." 1>&2 __vecho " --> ${_E_INSDESTTREE_}" 1>&2 @@ -50,6 +51,16 @@ if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" ]]; then __helpers_die "${helper} used with \${D} or \${ED}" exit 1 fi +if [[ -n ${EPREFIX} && \ + ${_E_INSDESTTREE_#${EPREFIX}} != "${_E_INSDESTTREE_}" ]]; +then + __vecho "---" 1>&2 + __vecho "You should not use \${EPREFIX} with helpers." 1>&2 + __vecho " --> ${_E_INSDESTTREE_}" 1>&2 + __vecho "---" 1>&2 + __helpers_die "${helper} used with \${EPREFIX}" + exit 1 +fi if ___eapi_doins_and_newins_preserve_symlinks; then DOINS_ARGS+=( --preserve_symlinks ) -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] usage of /bin/bash in shebangs
On 7/24/22 23:17, Fabian Groffen wrote: On 24-07-2022 13:58:31 -0700, Zac Medico wrote: On 7/24/22 12:29, Fabian Groffen wrote: Hi, Quick question, I noticed that portage uses /bin/bash hardcoded in shebang of scripts, while it uses /usr/bin/env python for python executable files. Is there anything against using /usr/bin/env bash for shell scripts? Changing this would help for Prefix. I can't think of any reason not to do this. Do you want to see a patch, or is it OK to push this myself? It's OK to just push this yourself. Thanks! -- Thanks, Zac OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-portage-dev] usage of /bin/bash in shebangs
On 24-07-2022 13:58:31 -0700, Zac Medico wrote: > On 7/24/22 12:29, Fabian Groffen wrote: > > Hi, > > > > Quick question, I noticed that portage uses /bin/bash hardcoded in > > shebang of scripts, while it uses /usr/bin/env python for python > > executable files. > > > > Is there anything against using /usr/bin/env bash for shell scripts? > > Changing this would help for Prefix. > > > I can't think of any reason not to do this. Do you want to see a patch, or is it OK to push this myself? Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] usage of /bin/bash in shebangs
On 7/24/22 12:29, Fabian Groffen wrote: Hi, Quick question, I noticed that portage uses /bin/bash hardcoded in shebang of scripts, while it uses /usr/bin/env python for python executable files. Is there anything against using /usr/bin/env bash for shell scripts? Changing this would help for Prefix. I can't think of any reason not to do this. -- Thanks, Zac OpenPGP_signature Description: OpenPGP digital signature
[gentoo-portage-dev] usage of /bin/bash in shebangs
Hi, Quick question, I noticed that portage uses /bin/bash hardcoded in shebang of scripts, while it uses /usr/bin/env python for python executable files. Is there anything against using /usr/bin/env bash for shell scripts? Changing this would help for Prefix. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 16:35:11 +0200, Michał Górny wrote: > This would require a new EAPI. We don't really want more Portage- > specific hacks that are going to break for everyone not using Portage or > the very specific Portage version. > > I'm not saying that it's not doable but I'm not convinced the added > complexity is really worth the effort, especially given that this looks > like a very special corner case. In the end, fixing Melpa is the right > thing to do. > > For a start, you'd have to ensure that the "normalizer script" (or its > dependencies, if you put it in the repo) is available at the time of > fetching. This pretty much goes back to the problem of "fetch > dependencies", and requires a major design change in Portage that > reduces separation between fetching and installing that we have now. > I mean, right now Portage pretty much assumes that you can do > a `--fetchonly` with no extra packages necessary. > > The "normalizer" wouldn't be trivial either. In the end, we're talking > about getting 100% consistent results on all platforms, over > a reasonably long timeframe. Oh, what a chain of complexity. Yeah, I'll go try fixing melpa first. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, 2022-05-16 at 19:37 +0200, Markus Walter wrote: > Hello all, > > is it possible to do the following: after fetching a distfile portage runs > an external normaliser program specified in an ebuild before checking the > hash? > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. > This would require a new EAPI. We don't really want more Portage- specific hacks that are going to break for everyone not using Portage or the very specific Portage version. I'm not saying that it's not doable but I'm not convinced the added complexity is really worth the effort, especially given that this looks like a very special corner case. In the end, fixing Melpa is the right thing to do. For a start, you'd have to ensure that the "normalizer script" (or its dependencies, if you put it in the repo) is available at the time of fetching. This pretty much goes back to the problem of "fetch dependencies", and requires a major design change in Portage that reduces separation between fetching and installing that we have now. I mean, right now Portage pretty much assumes that you can do a `--fetchonly` with no extra packages necessary. The "normalizer" wouldn't be trivial either. In the end, we're talking about getting 100% consistent results on all platforms, over a reasonably long timeframe. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, May 17, 2022 at 8:32 AM Markus Walter wrote: > > On Tue, 17 May 2022 14:14:57 +0200, > Rich Freeman wrote: > > > > On Mon, May 16, 2022 at 1:37 PM Markus Walter > > wrote: > > > > > > My use case is the following: I would like to improve the gs-elpa program > > > and provide a precomputed overlay for melpa. However the melpa distfiles > > > are > > > rebuilt everyday and cause checksum failures. However the only thing > > > changing are the timestamps. Hence if a normaliser program could simply > > > set > > > all timestamps to some predefined value (say 1.1.1970) then this problem > > > should vanish. > > > > > > > Wouldn't a simpler solution be to just have an ebuild setting that > > tells the package manager to not check the timestamp? > > The timestamps are inside archive files thus changing the overall file > hash. This happens during distfile download, where some more sophisticated > replace all timestamps function would be necessary than just ignoring one > timestamp. Ah, apologies. Totally missed that. Yeah, obviously if the timestamps INSIDE the archive are changing your only solution is to edit the contents. I thought it was just the external timestamp changing (I'm not sure if portage even checks that, or if wget changes the mtime on files it downloads). -- Rich
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 14:14:57 +0200, Rich Freeman wrote: > > On Mon, May 16, 2022 at 1:37 PM Markus Walter > wrote: > > > > My use case is the following: I would like to improve the gs-elpa program > > and provide a precomputed overlay for melpa. However the melpa distfiles are > > rebuilt everyday and cause checksum failures. However the only thing > > changing are the timestamps. Hence if a normaliser program could simply set > > all timestamps to some predefined value (say 1.1.1970) then this problem > > should vanish. > > > > Wouldn't a simpler solution be to just have an ebuild setting that > tells the package manager to not check the timestamp? The timestamps are inside archive files thus changing the overall file hash. This happens during distfile download, where some more sophisticated replace all timestamps function would be necessary than just ignoring one timestamp. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, May 16, 2022 at 1:37 PM Markus Walter wrote: > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. > Wouldn't a simpler solution be to just have an ebuild setting that tells the package manager to not check the timestamp? -- Rich
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 05:39:27 +0200, Robin H. Johnson wrote: > I don't know what 'gs-elpa' & 'Melpa' are, but maybe talking to upstream > would be good here, and improving that behavior. That's actually not too bad a plan. Thanks for this somewhat obvious suggestion, which I however failed to see. I'll try that. Although reproducibility is not a tenet there, they may take a patch which improves the situation. Melpa is the largest (not totally sure) repository for Emacs add-ons. And gs-elpa is the g-sorcery integration for Gentoo. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 05:21:02 +0200, Zac Medico wrote: > The only usable hook that we currently have for this is FETCHCOMMAND and > RESUMCOMMAND in make.conf. You can replace them with a script that does the > normal thing and then sets the timestamp. The default values are found in > /usr/share/portage/config/make.globals. Thanks for pointing this out, but I fear this will not easily solve the problem for new users. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, May 16, 2022 at 07:37:40PM +0200, Markus Walter wrote: > Hello all, > > is it possible to do the following: after fetching a distfile portage runs > an external normaliser program specified in an ebuild before checking the > hash? > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. I don't know what 'gs-elpa' & 'Melpa' are, but maybe talking to upstream would be good here, and improving that behavior. If the file contents or non-timestamp metadata change, absolutely the timestamps should change. But otherwise, the timestamp should NOT change. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature
Re: [gentoo-portage-dev] Normaliser function for distfiles
On 5/16/22 10:37, Markus Walter wrote: Hello all, is it possible to do the following: after fetching a distfile portage runs an external normaliser program specified in an ebuild before checking the hash? My use case is the following: I would like to improve the gs-elpa program and provide a precomputed overlay for melpa. However the melpa distfiles are rebuilt everyday and cause checksum failures. However the only thing changing are the timestamps. Hence if a normaliser program could simply set all timestamps to some predefined value (say 1.1.1970) then this problem should vanish. Thanks in advance Markus The only usable hook that we currently have for this is FETCHCOMMAND and RESUMCOMMAND in make.conf. You can replace them with a script that does the normal thing and then sets the timestamp. The default values are found in /usr/share/portage/config/make.globals. -- Thanks, Zac OpenPGP_signature Description: OpenPGP digital signature
[gentoo-portage-dev] Normaliser function for distfiles
Hello all, is it possible to do the following: after fetching a distfile portage runs an external normaliser program specified in an ebuild before checking the hash? My use case is the following: I would like to improve the gs-elpa program and provide a precomputed overlay for melpa. However the melpa distfiles are rebuilt everyday and cause checksum failures. However the only thing changing are the timestamps. Hence if a normaliser program could simply set all timestamps to some predefined value (say 1.1.1970) then this problem should vanish. Thanks in advance Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
[gentoo-portage-dev] [PATCH] MergeProcess: propagate mtimedb["ldpath"] to parent process (bug 836375)
Use an instance of multiprocessing.Pipe to propagate mtimedb["ldpath"] from the MergeProcess child process to the parent process. This fixes env_update calls to avoid unnecessary regeneration of ld.so.cache in cases where mtimedb["ldpath"] has not changed since the last call to env_update. Bug: https://bugs.gentoo.org/836375 --- lib/portage/dbapi/_MergeProcess.py | 18 ++ lib/portage/dbapi/vartree.py | 10 ++ 2 files changed, 28 insertions(+) diff --git a/lib/portage/dbapi/_MergeProcess.py b/lib/portage/dbapi/_MergeProcess.py index db3f3b105..667a5bf20 100644 --- a/lib/portage/dbapi/_MergeProcess.py +++ b/lib/portage/dbapi/_MergeProcess.py @@ -2,6 +2,7 @@ # Distributed under the terms of the GNU General Public License v2 import io +import multiprocessing import platform import fcntl @@ -38,6 +39,7 @@ class MergeProcess(ForkProcess): "_dblink", "_elog_keys", "_locked_vdb", +"_mtime_reader", ) def _start(self): @@ -113,6 +115,15 @@ class MergeProcess(ForkProcess): self._elog_reader_fd = None return False +def _mtime_handler(self): +try: +mtimes = self._mtime_reader.recv() +except EOFError: +pass +else: +self.prev_mtimes.clear() +self.prev_mtimes.update(mtimes) + def _spawn(self, args, fd_pipes, **kwargs): """ Extend the superclass _spawn method to perform some pre-fork and @@ -127,6 +138,11 @@ class MergeProcess(ForkProcess): fcntl.fcntl(elog_reader_fd, fcntl.F_GETFL) | os.O_NONBLOCK, ) +mtime_reader, mtime_writer = multiprocessing.Pipe(duplex=False) +fd_pipes[mtime_writer.fileno()] = mtime_writer.fileno() +self.scheduler.add_reader(mtime_reader.fileno(), self._mtime_handler) +self._mtime_reader = mtime_reader + blockers = None if self.blockers is not None: # Query blockers in the main process, since closing @@ -142,6 +158,7 @@ class MergeProcess(ForkProcess): vartree=self.vartree, blockers=blockers, pipe=elog_writer_fd, +mtime_pipe=mtime_writer, ) fd_pipes[elog_writer_fd] = elog_writer_fd self.scheduler.add_reader(elog_reader_fd, self._elog_output_handler) @@ -160,6 +177,7 @@ class MergeProcess(ForkProcess): self._elog_reader_fd = elog_reader_fd pids = super(MergeProcess, self)._spawn(args, fd_pipes, **kwargs) os.close(elog_writer_fd) +mtime_writer.close() self._buf = "" self._elog_keys = set() # Discard messages which will be collected by the subprocess, diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py index 602913862..a95d60691 100644 --- a/lib/portage/dbapi/vartree.py +++ b/lib/portage/dbapi/vartree.py @@ -1806,6 +1806,7 @@ class dblink: blockers=None, scheduler=None, pipe=None, +mtime_pipe=None, ): """ Creates a DBlink object for a given CPV. @@ -1862,6 +1863,7 @@ class dblink: self._device_path_map = {} self._hardlink_merge_map = {} self._hash_key = (self._eroot, self.mycpv) +self._mtime_pipe = mtime_pipe self._protect_obj = None self._pipe = pipe self._postinst_failure = False @@ -2618,6 +2620,7 @@ class dblink: writemsg_level=self._display_merge, vardbapi=self.vartree.dbapi, ) +self._send_mtimes(ldpath_mtimes) unmerge_with_replacement = preserve_paths is not None if not unmerge_with_replacement: @@ -4243,6 +4246,12 @@ class dblink: def _emerge_log(self, msg): emergelog(False, msg) +def _send_mtimes(self, mtimes): +if self._mtime_pipe is None: +return + +self._mtime_pipe.send(mtimes) + def treewalk( self, srcroot, @@ -5274,6 +5283,7 @@ class dblink: writemsg_level=self._display_merge, vardbapi=self.vartree.dbapi, ) +self._send_mtimes(prev_mtimes) # For gcc upgrades, preserved libs have to be removed after the # the library path has been updated. -- 2.35.1
Re: [gentoo-portage-dev] Changing the VDB format
On 4/11/2022 15:20, Sid Spry wrote: > On Mon, Apr 11, 2022, at 3:02 PM, Joshua Kinard wrote: >> >> I think json is the best format for storing the data on-disk. It's intended >> to be a data serialization format to convert data from a non-specific memory >> format to a storable on-disk format and back again, so this is a perfect use >> for it. > > Can we avoid adding another format? I find json very hard to edit by hand, > it's > good at storing lots of data in a quasi-textual format, but is strict enough > to be > obnoxious to work with. I sympathize with you here on this, json is not a format geared for editing by hand... :: looks disapprovingly at net-misc/kea :: > Can the files not be concatenated? Doing so is similar to the tar suggestion, > but would keep everything very portage-like. Have the contents assigned to > variables. I am betting someone tried this at the start but settled on the > current > scheme. Does anyone know why? (This would have to be done in bash syntax > I assume.) > > Alternatively, I think the tar suggestion is quite elegant. There's streaming > decompressors you can use from python. It adds an extra step to modify but > that could be handled transparently by a dev mode. In dev mode, leave the > files > after extraction and do not re-extract, for release mode replace the archive > with > what is on disk. Out of curiosity, what are you doing that requires manual editing of the VDB data? That data isn't, in normal scenarios, supposed to be arbitrarily edited. Throwing out a good optimization for what sounds like a niche corner-case doesn't seem like a great plan. Given json's malleable format, and especially Matt's example script, converting from json data to another format that's more conducive to manual editing in rare circumstances, then converting back, is not impossible. -- Joshua Kinard Gentoo/MIPS ku...@gentoo.org rsa6144/5C63F4E3F5C6C943 2015-04-27 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic