[gentoo-portage-dev] [PATCH] emerge: Default enable soname dependencies (bug 687956)

2021-11-28 Thread Zac Medico
Default emerge --ignore-soname-deps=n, in order to enable
soname dependencies by default. As always, soname dependencies
remain inapplicable in the absence of the --usepkgonly option
(or --getbinpkgonly). Therefore, this change only affects
commands that specify --usepkgonly or --getbinpkgonly.

Bug: https://bugs.gentoo.org/687956
Signed-off-by: Zac Medico 
---
 lib/_emerge/create_depgraph_params.py | 2 +-
 man/emerge.1  | 7 ---
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/lib/_emerge/create_depgraph_params.py 
b/lib/_emerge/create_depgraph_params.py
index 11c3e3736..95c4c2035 100644
--- a/lib/_emerge/create_depgraph_params.py
+++ b/lib/_emerge/create_depgraph_params.py
@@ -104,7 +104,7 @@ def create_depgraph_params(myopts, myaction):
 if ignore_built_slot_operator_deps is not None:
 myparams["ignore_built_slot_operator_deps"] = 
ignore_built_slot_operator_deps
 
-myparams["ignore_soname_deps"] = myopts.get("--ignore-soname-deps", "y")
+myparams["ignore_soname_deps"] = myopts.get("--ignore-soname-deps", "n")
 
 dynamic_deps = myopts.get("--dynamic-deps", "y") != "n" and "--nodeps" not 
in myopts
 if dynamic_deps:
diff --git a/man/emerge.1 b/man/emerge.1
index 8f6d12925..ff565b46f 100644
--- a/man/emerge.1
+++ b/man/emerge.1
@@ -639,9 +639,10 @@ supported beginning with \fBEAPI 5\fR.
 .TP
 .BR "\-\-ignore\-soname\-deps < y | n >"
 Ignore the soname dependencies of binary and installed packages. This
-option is enabled by default, since soname dependencies are relatively
-new, and the required metadata is not guaranteed to exist for binary and
-installed packages built with older versions of portage. Also, soname
+option may be useful when working with binary or installed packages
+that lack appropriate soname dependency metadata because they were built
+with a package manager that does not support soname dependencies (perhaps
+an older version of portage). Soname
 dependencies will be automatically ignored for dependency calculations
 that can pull unbuilt ebuilds into the dependency graph, since unbuilt
 ebuilds do not have any soname dependency metadata, making it impossible
-- 
2.32.0




Re: [gentoo-dev] [PATCH] go-module.eclass: Add GO_OPTIONAL flag

2021-11-28 Thread Zac Medico

On 11/21/21 02:57, Florian Schmaus wrote:

Following the pattern found in other eclasses, add GO_OPTIONAL to the
go-module eclass. This allows to inherit the eclass without pulling
its dependencies. See, e.g., bug #775779 for the motivation.

Signed-off-by: Florian Schmaus 
---
  eclass/go-module.eclass | 31 ++-
  1 file changed, 22 insertions(+), 9 deletions(-)

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
index 3ad8542a28ae..c9eb90ac62ea 100644
--- a/eclass/go-module.eclass
+++ b/eclass/go-module.eclass
@@ -1,4 +1,4 @@
-# Copyright 2019-2020 Gentoo Authors
+# Copyright 2019-2021 Gentoo Authors
  # Distributed under the terms of the GNU General Public License v2
  
  # @ECLASS: go-module.eclass

@@ -55,13 +55,17 @@ if [[ -z ${_GO_MODULE} ]]; then
  
  _GO_MODULE=1
  
-BDEPEND=">=dev-lang/go-1.12"

+if [[ ! ${GO_OPTIONAL} ]]; then
+   BDEPEND=">=dev-lang/go-1.12"
  
-# Workaround for pkgcheck false positive: https://github.com/pkgcore/pkgcheck/issues/214

-# MissingUnpackerDep: version ...: missing BDEPEND="app-arch/unzip"
-# Added here rather than to each affected package, so it can be cleaned up just
-# once when pkgcheck is improved.
-BDEPEND+=" app-arch/unzip"
+   # Workaround for pkgcheck false positive: 
https://github.com/pkgcore/pkgcheck/issues/214
+   # MissingUnpackerDep: version ...: missing BDEPEND="app-arch/unzip"
+   # Added here rather than to each affected package, so it can be cleaned 
up just
+   # once when pkgcheck is improved.
+   BDEPEND+=" app-arch/unzip"
+
+   EXPORT_FUNCTIONS src_unpack
+fi
  
  # Force go to build in module mode.

  # In this mode the GOPATH environment variable is ignored.
@@ -83,8 +87,6 @@ QA_FLAGS_IGNORED='.*'
  # Go packages should not be stripped with strip(1).
  RESTRICT+=" strip"
  
-EXPORT_FUNCTIONS src_unpack

-
  # @ECLASS-VARIABLE: EGO_SUM
  # @DESCRIPTION:
  # This is an array based on the go.sum content from inside the target package.
@@ -147,6 +149,17 @@ EXPORT_FUNCTIONS src_unpack
  # directory structure.
  declare -A -g _GOMODULE_GOSUM_REVERSE_MAP
  
+# @ECLASS-VARIABLE: GO_OPTIONAL

+# @DEFAULT_UNSET
+# @PRE_INHERIT
+# @DESCRIPTION:
+# If set to a non-null value before inherit, then the Go part of the
+# ebuild will be considered optional. No dependencies will be added and
+# no phase functions will be exported.
+#
+# If you enable GO_OPTIONAL, you have to set BDEPEND on >=dev-lang/go-1.12
+# for your package and call go-module_src_unpack manually.
+
  # @FUNCTION: go-module_set_globals
  # @DESCRIPTION:
  # Convert the information in EGO_SUM for other usage in the ebuild.



How about if we also add a GO_DEPEND variable, so that eclasshi 
consumers can do something like BDEPEND="go? ( ${GO_DEPEND} )" ?

--
Thanks,
Zac


OpenPGP_signature
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH] Install example repo.postsync.d script into sharedir

2021-11-08 Thread Zac Medico

On 11/4/21 02:50, Daniel Cordero wrote:

The sysconfdir is for host specific configuration files, and this
example script makes no host specific change (it is not enabled by
default).

Install the script under portage's sharedir, from where administrators
can copy it into sysconfdir, if needed.

Signed-off-by: Daniel Cordero 
---
  setup.py | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/setup.py b/setup.py
index 549fff650..fe0265c64 100755
--- a/setup.py
+++ b/setup.py
@@ -833,12 +833,12 @@ setup(
  ["$portage_setsdir", ["cnf/sets/portage.conf"]],
  ["$docdir", ["NEWS", "RELEASE-NOTES"]],
  ["$portage_base/bin", ["bin/deprecated-path"]],
-["$sysconfdir/portage/repo.postsync.d", 
["cnf/repo.postsync.d/example"]],
+["$portage_confdir/repo.postsync.d", 
["cnf/repo.postsync.d/example"]],
  ],
  [
  ("etc", "cnf", ("etc-update.conf", "dispatch-conf.conf")),
  ("etc/logrotate.d", "cnf/logrotate.d", ("elog-save-summary",)),
-("etc/portage/repo.postsync.d", "cnf/repo.postsync.d", 
("example",)),
+("share/portage/config/repo.postsync.d", "cnf/repo.postsync.d", 
("example",)),
  (
  "share/portage/config",
  "cnf",



Merged, thanks!

https://gitweb.gentoo.org/proj/portage.git/commit/?id=b7dc908a99e564de6f10174b1489028d939b917f
--
Thanks,
Zac



[gentoo-portage-dev] Re: [PATCH] EbuildIpc.communicate: lockfile PermissionDenied retry

2021-11-06 Thread Zac Medico

On 11/3/21 20:09, Zac Medico wrote:

The lockfile function is expected to raise PermissionDenied if
the (root) parent process holds the lock, so retry in this case.

Bug: https://bugs.gentoo.org/468990
Signed-off-by: Zac Medico 
---
  bin/ebuild-ipc.py | 11 ++-
  1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/bin/ebuild-ipc.py b/bin/ebuild-ipc.py
index 4999c043a..6eaa658a2 100755
--- a/bin/ebuild-ipc.py
+++ b/bin/ebuild-ipc.py
@@ -158,7 +158,16 @@ class EbuildIpc:
  # Make locks quiet since unintended locking messages displayed on
  # stdout could corrupt the intended output of this program.
  portage.locks._quiet = True
-lock_obj = portage.locks.lockfile(self.ipc_lock_file, unlinkfile=True)
+# Acquire lock with PermissionDenied retry for bug #468990.
+for _ in range(1000):
+try:
+lock_obj = portage.locks.lockfile(self.ipc_lock_file, 
unlinkfile=True)
+except portage.exception.PermissionDenied:
+time.sleep(0.1)
+else:
+break
+else:
+raise portage.exception.PermissionDenied(self.ipc_lock_file)
  
  try:

  return self._communicate(args)



Withdrawn in favor of this lockfile permission race fix:

https://archives.gentoo.org/gentoo-portage-dev/message/88916062415d9f692091dfb947f1bda2
--
Thanks,
Zac


OpenPGP_signature
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] EbuildIpcDaemon: fix lock permission race

2021-11-06 Thread Zac Medico
Move ipc files to a .ipc subdirectory, with a setgid bit to
prevent a lockfile group permission race. The lockfile function
uses an appropriate open call with mode argument so that the
lockfile is created atomically with both group ownership and
group write bit.

Bug: https://bugs.gentoo.org/468990
Signed-off-by: Zac Medico 
---
 bin/ebuild-ipc.py| 6 +++---
 bin/phase-functions.sh   | 4 ++--
 lib/_emerge/AbstractEbuildProcess.py | 4 ++--
 lib/_emerge/EbuildIpcDaemon.py   | 2 +-
 lib/portage/package/ebuild/prepare_build_dirs.py | 6 ++
 lib/portage/tests/ebuild/test_doebuild_spawn.py  | 1 +
 lib/portage/tests/ebuild/test_ipc_daemon.py  | 6 +++---
 7 files changed, 18 insertions(+), 11 deletions(-)

diff --git a/bin/ebuild-ipc.py b/bin/ebuild-ipc.py
index 4999c043a..c24ba6f58 100755
--- a/bin/ebuild-ipc.py
+++ b/bin/ebuild-ipc.py
@@ -138,9 +138,9 @@ class EbuildIpc:
 
 def __init__(self):
 self.fifo_dir = os.environ["PORTAGE_BUILDDIR"]
-self.ipc_in_fifo = os.path.join(self.fifo_dir, ".ipc_in")
-self.ipc_out_fifo = os.path.join(self.fifo_dir, ".ipc_out")
-self.ipc_lock_file = os.path.join(self.fifo_dir, ".ipc_lock")
+self.ipc_in_fifo = os.path.join(self.fifo_dir, ".ipc", "in")
+self.ipc_out_fifo = os.path.join(self.fifo_dir, ".ipc", "out")
+self.ipc_lock_file = os.path.join(self.fifo_dir, ".ipc", "lock")
 
 def _daemon_is_alive(self):
 try:
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index d3221993d..5eb031805 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -291,10 +291,10 @@ __dyn_clean() {
rm -f 
"$PORTAGE_BUILDDIR"/.{ebuild_changed,logid,pretended,setuped,unpacked,prepared} 
\

"$PORTAGE_BUILDDIR"/.{configured,compiled,tested,packaged,instprepped} \
"$PORTAGE_BUILDDIR"/.die_hooks \
-   "$PORTAGE_BUILDDIR"/.ipc_{in,out,lock} \
"$PORTAGE_BUILDDIR"/.exit_status
 
-   rm -rf "${PORTAGE_BUILDDIR}/build-info"
+   rm -rf "${PORTAGE_BUILDDIR}/build-info" \
+   "${PORTAGE_BUILDDIR}/.ipc"
rm -rf "${WORKDIR}"
rm -f "${PORTAGE_BUILDDIR}/files"
fi
diff --git a/lib/_emerge/AbstractEbuildProcess.py 
b/lib/_emerge/AbstractEbuildProcess.py
index 1b4e7759f..6d89d40f0 100644
--- a/lib/_emerge/AbstractEbuildProcess.py
+++ b/lib/_emerge/AbstractEbuildProcess.py
@@ -249,8 +249,8 @@ class AbstractEbuildProcess(SpawnProcess):
 
 def _init_ipc_fifos(self):
 
-input_fifo = os.path.join(self.settings["PORTAGE_BUILDDIR"], ".ipc_in")
-output_fifo = os.path.join(self.settings["PORTAGE_BUILDDIR"], 
".ipc_out")
+input_fifo = os.path.join(self.settings["PORTAGE_BUILDDIR"], ".ipc", 
"in")
+output_fifo = os.path.join(self.settings["PORTAGE_BUILDDIR"], ".ipc", 
"out")
 
 for p in (input_fifo, output_fifo):
 
diff --git a/lib/_emerge/EbuildIpcDaemon.py b/lib/_emerge/EbuildIpcDaemon.py
index ee6fd7658..78594ff0a 100644
--- a/lib/_emerge/EbuildIpcDaemon.py
+++ b/lib/_emerge/EbuildIpcDaemon.py
@@ -81,7 +81,7 @@ class EbuildIpcDaemon(FifoIpcDaemon):
 # write something to the pipe just before we close it, and in that
 # case the write will be lost. Therefore, try for a non-blocking
 # lock, and only re-open the pipe if the lock is acquired.
-lock_filename = os.path.join(os.path.dirname(self.input_fifo), 
".ipc_lock")
+lock_filename = os.path.join(os.path.dirname(self.input_fifo), 
"lock")
 try:
 lock_obj = lockfile(lock_filename, unlinkfile=True, 
flags=os.O_NONBLOCK)
 except TryAgain:
diff --git a/lib/portage/package/ebuild/prepare_build_dirs.py 
b/lib/portage/package/ebuild/prepare_build_dirs.py
index 659198905..9d2474fd8 100644
--- a/lib/portage/package/ebuild/prepare_build_dirs.py
+++ b/lib/portage/package/ebuild/prepare_build_dirs.py
@@ -102,6 +102,12 @@ def prepare_build_dirs(myroot=None, settings=None, 
cleanup=False):
 apply_secpass_permissions(
 mysettings[dir_key], uid=portage_uid, gid=portage_gid
 )
+# The setgid bit prevents a lockfile group permission race for bug 
#468990.
+ensure_dirs(
+os.path.join(mysettings["PORTAGE_BUILDDIR"], ".ipc"),
+gid=portage_gid,
+mode=0o2770,
+)
 except PermissionDenied as e:
 writemsg(_("Permission Denied: %s\n") %

[gentoo-portage-dev] [PATCH] fetch: enable resume for digestgen case

2021-11-04 Thread Zac Medico
Enable resume for the digestgen case (no digests available), when
the temporary file exceeds PORTAGE_FETCH_RESUME_MIN_SIZE. This
fixes a case which caused the ebuild digest command to skip the
download and fail with a message like this:

!!! File b'/var/cache/distfiles/foo.tar.xz' doesn't exist, can't update Manifest

Bug: https://bugs.gentoo.org/821571
Signed-off-by: Zac Medico 
---
 lib/portage/package/ebuild/fetch.py | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/lib/portage/package/ebuild/fetch.py 
b/lib/portage/package/ebuild/fetch.py
index 8c64362c2..2d3625800 100644
--- a/lib/portage/package/ebuild/fetch.py
+++ b/lib/portage/package/ebuild/fetch.py
@@ -1485,10 +1485,11 @@ def fetch(
 except EnvironmentError:
 pass
 elif not orig_digests:
-# We don't have a digest, but the file exists.  We must
-# assume that it is fully downloaded.
+# We don't have a digest, and the temporary file 
exists.
 if not force:
-continue
+# Try to resume this download when full
+# download has not been explicitly forced.
+fetched = 1
 else:
 if (
 mydigests[myfile].get("size") is not None
-- 
2.32.0




[gentoo-portage-dev] [PATCH] EbuildIpc.communicate: lockfile PermissionDenied retry

2021-11-03 Thread Zac Medico
The lockfile function is expected to raise PermissionDenied if
the (root) parent process holds the lock, so retry in this case.

Bug: https://bugs.gentoo.org/468990
Signed-off-by: Zac Medico 
---
 bin/ebuild-ipc.py | 11 ++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/bin/ebuild-ipc.py b/bin/ebuild-ipc.py
index 4999c043a..6eaa658a2 100755
--- a/bin/ebuild-ipc.py
+++ b/bin/ebuild-ipc.py
@@ -158,7 +158,16 @@ class EbuildIpc:
 # Make locks quiet since unintended locking messages displayed on
 # stdout could corrupt the intended output of this program.
 portage.locks._quiet = True
-lock_obj = portage.locks.lockfile(self.ipc_lock_file, unlinkfile=True)
+# Acquire lock with PermissionDenied retry for bug #468990.
+for _ in range(1000):
+try:
+lock_obj = portage.locks.lockfile(self.ipc_lock_file, 
unlinkfile=True)
+except portage.exception.PermissionDenied:
+time.sleep(0.1)
+else:
+break
+else:
+raise portage.exception.PermissionDenied(self.ipc_lock_file)
 
 try:
 return self._communicate(args)
-- 
2.32.0




Re: [gentoo-portage-dev] [PATCH] bin/estrip: avoid copying directories in FEATURES=installsources

2021-07-17 Thread Zac Medico
On 7/17/21 12:59 PM, Sergei Trofimovich wrote:
> Initially problem is noticed on gcc-11 as a full ${WORKDIR} syncing
> into /usr/src/debug. It happens because `debug.sources` sometimes
> contains directory. For example on bash-5 it has:
> 
> $ grep -zv '/<[^/>]*>$' debug.sources | LANG=C sort -z -u  | sed -e 
> 's/\x00/\n/g'
> bash-5.0/
> bash-5.0/alias.c
> ...
> 
> This causes syncing object files, config.log, final binaries
> and other unexpected data. The change avoids syncking paths
> that end with '/'.
> 
> Signed-off-by: Sergei Trofimovich 
> ---
>  bin/estrip | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/bin/estrip b/bin/estrip
> index 7ef1ec35c..6cca0d04b 100755
> --- a/bin/estrip
> +++ b/bin/estrip
> @@ -464,7 +464,10 @@ if [[ -s ${tmpdir}/debug.sources ]] && \
>  then
>   __vecho "installsources: rsyncing source files"
>   [[ -d ${D%/}/${prepstrip_sources_dir#/} ]] || mkdir -p 
> "${D%/}/${prepstrip_sources_dir#/}"
> + # skip installation of ".../" (system headers? why inner slashes 
> are forbidden?)
> + # skip syncing of ".../foo/" (complete directories)
>   grep -zv '/<[^/>]*>$' "${tmpdir}"/debug.sources | \
> + grep -zv '/$' | \
>   (cd "${WORKDIR}"; LANG=C sort -z -u | \
>   rsync -tL0 --chmod=ugo-st,a+r,go-w,Da+x,Fa-x --files-from=- 
> "${WORKDIR}/" "${D%/}/${prepstrip_sources_dir#/}/" )
>  
> 

Looks good. Merged with both grep calls combined via grep -e. Thanks!

https://gitweb.gentoo.org/proj/portage.git/commit/?id=e083c8bf20d8488d329e3dccd643c28429e6fe30
-- 
Thanks,
Zac



OpenPGP_signature
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH] man/make.conf.5: remove mention of zlib USE flag

2021-07-04 Thread Zac Medico
On 6/28/21 2:56 PM, Thymo van Beers wrote:
> Both sys-devel/binutils and sys-devel/gdb are built with system zlib by
> default for some time now. This commit removes the mention of USE=zlib to 
> avoid
> confusion.
> 
> Signed-off-by: Thymo van Beers 
> ---
>  man/make.conf.5 | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/man/make.conf.5 b/man/make.conf.5
> index 1c72109ad..db742fdb5 100644
> --- a/man/make.conf.5
> +++ b/man/make.conf.5
> @@ -378,8 +378,7 @@ redundant on\-the\-fly compression.  The resulting file 
> will be called
>  .TP
>  .B compressdebug
>  Compress the debug sections in the split debug files with zlib to save
> -space.  Make sure you have built both binutils and gdb with USE=zlib
> -support for this to work.  See \fBsplitdebug\fR for general split debug
> +space.  See \fBsplitdebug\fR for general split debug
>  information (upon which this feature depends).
>  .TP
>  .B config\-protect\-if\-modified
> 

Merged thanks!

https://gitweb.gentoo.org/proj/portage.git/commit/?id=ee944c4fd76af4f7dffb756e9ed0303cb9606112
-- 
Thanks,
Zac



[gentoo-portage-dev] Re: [PATCH v2] Add @unsatisfied-deps package set (bug 248026)

2021-06-19 Thread Zac Medico
On 6/18/21 8:29 PM, Zac Medico wrote:
> On 6/18/21 8:13 PM, Zac Medico wrote:
>> On 6/18/21 6:01 PM, Zac Medico wrote:
>>> If emerge --depclean fails to resolve any dependencies, then it will
>>> now suggest emerge @unsatisfied-deps as the simplest possible
>>> solution, and will also suggest to unmerge @unavailable where
>>> appropriate at the end:
>>>
>>> $ emerge --depclean
>>>
>>> Calculating dependencies... done!
>>>  * Dependencies could not be completely resolved due to
>>>  * the following required packages not being installed:
>>>  *
>>>  *   virtual/cdrtools pulled in by:
>>>  * app-cdr/cdrdao-1.2.4
>>>  *
>>>  * Have you forgotten to resolve unsatisfied dependencies prior to
>>>  * depclean? The simplest possible command for this purpose is as
>>>  * follows:
>>>  *
>>>  *   emerge @unsatisfied-deps
>>
>> It turns out that @unsatisfied-deps is often unsuitable here because it
>> pulls in a bunch of @installed packages, when you really want to use
>> @world as the source of truth.
> 
> The underlying reason is the same as the reason that we've never used
> @installed updates as a substitute for @world updates. It just doesn't
> work, because @installed is polluted in comparison to @world.
> 

My plan is it introduce an @unsatisfied-world set which is equivalent to
@unsatisfied-deps but filters out any non @world packages.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] Re: [PATCH v2] Add @unsatisfied-deps package set (bug 248026)

2021-06-18 Thread Zac Medico
On 6/18/21 8:13 PM, Zac Medico wrote:
> On 6/18/21 6:01 PM, Zac Medico wrote:
>> If emerge --depclean fails to resolve any dependencies, then it will
>> now suggest emerge @unsatisfied-deps as the simplest possible
>> solution, and will also suggest to unmerge @unavailable where
>> appropriate at the end:
>>
>> $ emerge --depclean
>>
>> Calculating dependencies... done!
>>  * Dependencies could not be completely resolved due to
>>  * the following required packages not being installed:
>>  *
>>  *   virtual/cdrtools pulled in by:
>>  * app-cdr/cdrdao-1.2.4
>>  *
>>  * Have you forgotten to resolve unsatisfied dependencies prior to
>>  * depclean? The simplest possible command for this purpose is as
>>  * follows:
>>  *
>>  *   emerge @unsatisfied-deps
> 
> It turns out that @unsatisfied-deps is often unsuitable here because it
> pulls in a bunch of @installed packages, when you really want to use
> @world as the source of truth.

The underlying reason is the same as the reason that we've never used
@installed updates as a substitute for @world updates. It just doesn't
work, because @installed is polluted in comparison to @world.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] Re: [PATCH v2] Add @unsatisfied-deps package set (bug 248026)

2021-06-18 Thread Zac Medico
On 6/18/21 6:01 PM, Zac Medico wrote:
> If emerge --depclean fails to resolve any dependencies, then it will
> now suggest emerge @unsatisfied-deps as the simplest possible
> solution, and will also suggest to unmerge @unavailable where
> appropriate at the end:
> 
> $ emerge --depclean
> 
> Calculating dependencies... done!
>  * Dependencies could not be completely resolved due to
>  * the following required packages not being installed:
>  *
>  *   virtual/cdrtools pulled in by:
>  * app-cdr/cdrdao-1.2.4
>  *
>  * Have you forgotten to resolve unsatisfied dependencies prior to
>  * depclean? The simplest possible command for this purpose is as
>  * follows:
>  *
>  *   emerge @unsatisfied-deps

It turns out that @unsatisfied-deps is often unsuitable here because it
pulls in a bunch of @installed packages, when you really want to use
@world as the source of truth.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH v2] Add @unsatisfied-deps package set (bug 248026)

2021-06-18 Thread Zac Medico
If emerge --depclean fails to resolve any dependencies, then it will
now suggest emerge @unsatisfied-deps as the simplest possible
solution, and will also suggest to unmerge @unavailable where
appropriate at the end:

$ emerge --depclean

Calculating dependencies... done!
 * Dependencies could not be completely resolved due to
 * the following required packages not being installed:
 *
 *   virtual/cdrtools pulled in by:
 * app-cdr/cdrdao-1.2.4
 *
 * Have you forgotten to resolve unsatisfied dependencies prior to
 * depclean? The simplest possible command for this purpose is as
 * follows:
 *
 *   emerge @unsatisfied-deps
 *
 * The most comprehensive possible update command is this:
 *
 *   emerge --update --newuse --deep --with-bdeps=y @world
 *
 * Note that the --with-bdeps=y option is not required in many
 * situations. Refer to the emerge manual page (run `man emerge`)
 * for more information about --with-bdeps.
 *
 * Also, note that it may be necessary to manually uninstall
 * packages that no longer exist in the repository, since it may not
 * be possible to satisfy their dependencies. The simplest possible
 * command for this purpose is as follows, but be careful to examine
 * the resulting package list carefully:
 *
 *   emerge --ask --unmerge @unavailable
 *

Bug: https://bugs.gentoo.org/248026
Signed-off-by: Zac Medico 
---
[PATCH v2] Update --depclean message to suggest @unsatisfied-deps
   and unmerge @unavailable where appropriate.

 cnf/sets/portage.conf |  5 +++
 doc/config/sets.docbook   |  7 
 lib/_emerge/actions.py| 19 +++--
 lib/portage/_sets/__init__.py |  4 ++
 lib/portage/_sets/dbapi.py| 73 ++-
 5 files changed, 104 insertions(+), 4 deletions(-)

diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
index c4ad2efca..2bf38e414 100644
--- a/cnf/sets/portage.conf
+++ b/cnf/sets/portage.conf
@@ -115,3 +115,8 @@ class = portage.sets.dbapi.ChangedDepsSet
 class = portage.sets.dbapi.VariableSet
 variable = INHERITED
 includes = golang-base golang-build golang-vcs golang-vcs-snapshot go-module
+
+# Package set which contains all installed packages having one or more
+# unsatisfied runtime dependencies.
+[unsatisfied-deps]
+class = portage.sets.dbapi.UnsatisfiedDepsSet
diff --git a/doc/config/sets.docbook b/doc/config/sets.docbook
index eba98f468..015ec0c05 100644
--- a/doc/config/sets.docbook
+++ b/doc/config/sets.docbook
@@ -610,6 +610,13 @@



+   
+   portage.sets.dbapi.UnsatisfiedDepsSet
+   
+   Package set which contains all installed packages
+   having one or more unsatisfied runtime dependencies.
+   
+   



diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 1946f49df..ba2592bba 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -1000,11 +1000,18 @@ def _calc_depclean(settings, trees, ldpath_mtimes,
msg.append("%s" % (parent,))
msg.append("")
msg.extend(textwrap.wrap(
-   "Have you forgotten to do a complete update 
prior " + \
-   "to depclean? The most comprehensive command 
for this " + \
+   "Have you forgotten to resolve unsatisfied 
dependencies prior "
+   "to depclean? The simplest possible command for 
this "
"purpose is as follows:", 65
))
msg.append("")
+   msg.append("  " + \
+   good("emerge @unsatisfied-deps"))
+   msg.append("")
+   msg.extend(textwrap.wrap(
+   "The most comprehensive possible update command 
is this:", 65
+   ))
+   msg.append("")
msg.append("  " + \
good("emerge --update --newuse --deep 
--with-bdeps=y @world"))
msg.append("")
@@ -1018,8 +1025,14 @@ def _calc_depclean(settings, trees, ldpath_mtimes,
msg.extend(textwrap.wrap(
"Also, note that it may be necessary to 
manually uninstall " + \
"packages that no longer exist in the 
repository, since " + \
-   "it may not be possible to satisfy their 
dependencies.", 65
+   "it may not be possible to satisfy their 
dependencies."
+

[gentoo-portage-dev] [PATCH] Add @unsatisfied-deps package set (bug 248026)

2021-06-18 Thread Zac Medico
Bug: https://bugs.gentoo.org/248026
Signed-off-by: Zac Medico 
---
 cnf/sets/portage.conf  |  5 +++
 doc/config/sets.docbook|  7 
 lib/portage/_sets/dbapi.py | 73 +-
 3 files changed, 84 insertions(+), 1 deletion(-)

diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
index c4ad2efca..2bf38e414 100644
--- a/cnf/sets/portage.conf
+++ b/cnf/sets/portage.conf
@@ -115,3 +115,8 @@ class = portage.sets.dbapi.ChangedDepsSet
 class = portage.sets.dbapi.VariableSet
 variable = INHERITED
 includes = golang-base golang-build golang-vcs golang-vcs-snapshot go-module
+
+# Package set which contains all installed packages having one or more
+# unsatisfied runtime dependencies.
+[unsatisfied-deps]
+class = portage.sets.dbapi.UnsatisfiedDepsSet
diff --git a/doc/config/sets.docbook b/doc/config/sets.docbook
index eba98f468..1aa7e8aa1 100644
--- a/doc/config/sets.docbook
+++ b/doc/config/sets.docbook
@@ -610,6 +610,13 @@



+   
+   portage.sets.dbapi.UnsatisfiedDepsSet
+   
+   Package set which contains all installed packages packages
+   having one or more unsatisfied runtime dependencies.
+   
+   



diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
index 8e1f19979..8e436980c 100644
--- a/lib/portage/_sets/dbapi.py
+++ b/lib/portage/_sets/dbapi.py
@@ -14,8 +14,16 @@ from portage._sets.base import PackageSet
 from portage._sets import SetConfigError, get_boolean
 import portage
 
+from _emerge.Package import Package
+from _emerge.RootConfig import RootConfig
+
 __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
-   "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
+   "EverythingSet",
+   "OwnerSet",
+   "SubslotChangedSet",
+   "UnsatisfiedDepsSet",
+   "VariableSet",
+]
 
 class EverythingSet(PackageSet):
_operations = ["merge"]
@@ -303,6 +311,69 @@ class UnavailableBinaries(EverythingSet):
 
singleBuilder = classmethod(singleBuilder)
 
+class UnsatisfiedDepsSet(PackageSet):
+
+   _operations = ["merge", "unmerge"]
+
+   description = (
+   "Package set which contains all installed packages packages "
+   "having one or more unsatisfied runtime dependencies."
+   )
+
+   def __init__(self, vardb=None):
+   super().__init__()
+   self._vardb = vardb
+
+   def load(self):
+   vardb = self._vardb
+   trees = {
+   vardb.settings["EROOT"]: {
+   "porttree": vardb.vartree,
+   "vartree": vardb.vartree,
+   }
+   }
+   root_config = RootConfig(vardb.settings, 
trees[vardb.settings["EROOT"]], None)
+   atoms = []
+   for pkg_str in vardb.cpv_all():
+   try:
+   metadata = dict(
+   zip(
+   vardb._aux_cache_keys,
+   vardb.aux_get(pkg_str, 
vardb._aux_cache_keys),
+   )
+   )
+   except KeyError:
+   continue
+   pkg = Package(
+   built=True,
+   cpv=pkg_str,
+   installed=True,
+   metadata=metadata,
+   root_config=root_config,
+   type_name="installed",
+   )
+
+   runtime_deps = " ".join(pkg._metadata[k] for k in 
Package._runtime_keys)
+
+   success, unsatisfied_deps = portage.dep_check(
+   runtime_deps,
+   None,
+   vardb.settings,
+   myuse=pkg.use.enabled,
+   myroot=vardb.settings["EROOT"],
+   trees=trees,
+   )
+
+   if not (success and not unsatisfied_deps):
+   atoms.append(pkg.slot_atom)
+
+   self._setAtoms(atoms)
+
+   def singleBuilder(cls, options, settings, trees):
+   return cls(vardb=trees["vartree"].dbapi)
+
+   singleBuilder = classmethod(singleBuilder)
+
 class CategorySet(PackageSet):
_operations = ["merge", "unmerge"]
 
-- 
2.26.2




Re: [gentoo-portage-dev] Installed pkg list in dependency order ?

2021-06-14 Thread Zac Medico
On 6/14/21 8:16 AM, Joakim Tjernlund wrote:
> Hi 
> 
> I need to generate a pkg list from an uptodate gentoo system too feed this 
> list into qmerge
> so qmerge can install these pkgs in correct order.
> 
> Tried emerge -e but that list is not in dep order, seems to be just a list 
> with pkgs to rebuild.
> 
>  Jocke 
> 

The problem with emerge -e is that is pull in the whole kitchen sink, so
all of the circular dependencies will be mixed into that order. What you
want is a smaller calculation that only updates the intended packages.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-dev] Re: [PATCH 0/2] go-module.eclass cleanups

2021-05-25 Thread Zac Medico
On 5/21/21 8:45 AM, William Hubbs wrote:
> This is an improvement to my previous patch. It is a patch series now
> because there are two separate changes:
> 
> - GOPROXY is exported in go-module_set_globals since it is not needed if
>   EGO_SUM is not set in the ebuild.
> 
> - go-module_setup_proxy is added. This function sets up the go proxy
>   directory so that dependencies can be read from it.
> 
> William Hubbs (2):
>   go-module.eclass: fix GOPROXY export
>   go-module.eclass: add go-module_setup_proxy function
> 
>  eclass/go-module.eclass | 47 +++--
>  1 file changed, 45 insertions(+), 2 deletions(-)
> 

This series looks great to me! Thank you!
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] Re: [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

2021-05-21 Thread Zac Medico
On 5/20/21 9:03 AM, William Hubbs wrote:
> On Wed, May 19, 2021 at 01:57:38PM -0700, Zac Medico wrote:
>> On 5/19/21 1:45 PM, Zac Medico wrote:
>>>> +# @FUNCTION: go-module_setup_proxy
>>>> +# @DESCRIPTION:
>>>> +# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
>>>> +# this function in src_unpack.
>>>> +# It sets up the go module proxy in the appropriate location and exports
>>>> +# the GOPROXY environment variable so that go calls will be able to
>>>> +# locate the proxy directory.
>>>> +go-module_setup_proxy() {
>>>> +  # shellcheck disable=SC2120
>>>> +  debug-print-function "${FUNCNAME}" "$@"
>>>> +
>>>> +  if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
>>>> +  die "go-module_set_globals must be called in global scope"
>>>> +  fi
>>>> +
>>>> +  local goproxy_dir="${T}/go-proxy"
>>>> +  mkdir -p "${goproxy_dir}" || die
>>>> +
>>>> +  # For each Golang module distfile, look up where it's supposed to go and
>>>> +  # symlink it into place.
>>>> +  local f
>>>> +  local goproxy_mod_dir
>>>> +  for f in ${A}; do
>>>> +  goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
>>>> +  if [[ -n "${goproxy_mod_path}" ]]; then
>>>> +  debug-print-function "Populating go proxy for 
>>>> ${goproxy_mod_path}"
>>>> +  # Build symlink hierarchy
>>>> +  goproxy_mod_dir=$( dirname 
>>>> "${goproxy_dir}"/"${goproxy_mod_path}" )
>>>> +  mkdir -p "${goproxy_mod_dir}" || die
>>>> +  ln -sf "${DISTDIR}"/"${f}" 
>>>> "${goproxy_dir}/${goproxy_mod_path}" ||
>>>> +  die "Failed to ln"
>>>> +  local v=${goproxy_mod_path}
>>>> +  v="${v%.mod}"
>>>> +  v="${v%.zip}"
>>>> +  v="${v//*\/}"
>>>> +  _go-module_gosum_synthesize_files "${goproxy_mod_dir}" 
>>>> "${v}"
>>>> +  fi
>>>> +  done
>>>> +  export GOPROXY="file://${goproxy_dir}"
>>>> +
>>>> +  # Validate the gosum now
>>>> +  _go-module_src_unpack_verify_gosum
>>>> +}
>>>> +
>>>>  # @FUNCTION: go-module_src_unpack
>>>>  # @DESCRIPTION:
>>>>  # If EGO_SUM is set, unpack the base tarball(s) and set up the
>>>>
>>>
>>> The go-module_setup_proxy function solves bug 790851 nicely, since
>>> sys-cluster/k3s ebuilds can call that instead of go-module_src_unpack.
>>
>> I do have one criticism of the go-module_setup_proxy, which is that it
>> relies on the side-effect of the GOPROXY export for its operation. We
>> can instead echo the GOPROXY value to stdout and force the caller to
>> export it themselves, and provide a convenience wrapper function which
>> works based on side-effects.
> 
> You really only need GOPROXY if you use EGO_SUM in your ebuild, so I
> could probably export GOPROXY in go-module_set_globals.
> 
> William
> 

Yes, go-module_set_globals is a nice place to consolidate all of the
exports. Great idea!
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-dev] Re: [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

2021-05-19 Thread Zac Medico
On 5/19/21 1:45 PM, Zac Medico wrote:
>> +# @FUNCTION: go-module_setup_proxy
>> +# @DESCRIPTION:
>> +# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
>> +# this function in src_unpack.
>> +# It sets up the go module proxy in the appropriate location and exports
>> +# the GOPROXY environment variable so that go calls will be able to
>> +# locate the proxy directory.
>> +go-module_setup_proxy() {
>> +# shellcheck disable=SC2120
>> +debug-print-function "${FUNCNAME}" "$@"
>> +
>> +if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
>> +die "go-module_set_globals must be called in global scope"
>> +fi
>> +
>> +local goproxy_dir="${T}/go-proxy"
>> +mkdir -p "${goproxy_dir}" || die
>> +
>> +# For each Golang module distfile, look up where it's supposed to go and
>> +# symlink it into place.
>> +local f
>> +local goproxy_mod_dir
>> +for f in ${A}; do
>> +goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
>> +if [[ -n "${goproxy_mod_path}" ]]; then
>> +debug-print-function "Populating go proxy for 
>> ${goproxy_mod_path}"
>> +# Build symlink hierarchy
>> +goproxy_mod_dir=$( dirname 
>> "${goproxy_dir}"/"${goproxy_mod_path}" )
>> +mkdir -p "${goproxy_mod_dir}" || die
>> +ln -sf "${DISTDIR}"/"${f}" 
>> "${goproxy_dir}/${goproxy_mod_path}" ||
>> +die "Failed to ln"
>> +local v=${goproxy_mod_path}
>> +v="${v%.mod}"
>> +v="${v%.zip}"
>> +v="${v//*\/}"
>> +_go-module_gosum_synthesize_files "${goproxy_mod_dir}" 
>> "${v}"
>> +fi
>> +done
>> +export GOPROXY="file://${goproxy_dir}"
>> +
>> +# Validate the gosum now
>> +_go-module_src_unpack_verify_gosum
>> +}
>> +
>>  # @FUNCTION: go-module_src_unpack
>>  # @DESCRIPTION:
>>  # If EGO_SUM is set, unpack the base tarball(s) and set up the
>>
> 
> The go-module_setup_proxy function solves bug 790851 nicely, since
> sys-cluster/k3s ebuilds can call that instead of go-module_src_unpack.

I do have one criticism of the go-module_setup_proxy, which is that it
relies on the side-effect of the GOPROXY export for its operation. We
can instead echo the GOPROXY value to stdout and force the caller to
export it themselves, and provide a convenience wrapper function which
works based on side-effects.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-dev] Re: [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

2021-05-19 Thread Zac Medico
On 5/19/21 12:48 PM, William Hubbs wrote:
> If an ebuild uses EGO_SUM and needs to define a custom src_unpack phase,
> these functions will make that easier.
> 
> go-module_setup_proxy is used to create a local file proxy of the
> dependencies listed in EGO_SUM and go-module_filter_proxy is used to
> create a new ${A} with the EGO_SUM_SRC_URI values removed.
> 
> Signed-off-by: William Hubbs 
> ---
>  eclass/go-module.eclass | 69 +
>  1 file changed, 69 insertions(+)
> 
> diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> index c9a7ab12eaf..80e1f711215 100644
> --- a/eclass/go-module.eclass
> +++ b/eclass/go-module.eclass
> @@ -236,6 +236,75 @@ go-module_set_globals() {
>   _GO_MODULE_SET_GLOBALS_CALLED=1
>  }
>  
> +# @FUNCTION: go-module_filter_proxy
> +# @DESCRIPTION:
> +# If your ebuild redefines src_unpack and uses EGO_SUM, use the return
> +# value of this function in place of ${A} in your src_unpack function.
> +# It filters the EGO_SUM_SRC_URI values out of SRC_URI.
> +go-module_filter_proxy() {
> + # shellcheck disable=SC2120
> + debug-print-function "${FUNCNAME}" "$@"
> +
> + if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
> + die "go-module_set_globals must be called in global scope"
> + fi
> +
> + # Skip golang modules and add the rest of the entries to a local
> + # array.
> + local f
> + local -a src
> + for f in ${A}; do
> + if [[ -z ${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]} ]]; then
> + src+=("${f}")
> + fi
> + done
> + echo "${src[@]}"
> +}

While this go-module_filter_proxy might seem like a convenient way for
an ebuild to list distfiles which are not derived from EGO_SUM, I think
it's not the responsibility of go-module.eclass to provide this
information, therefore it's probably better to force the ebuild
developer to provide their own mechanism to track those files if needed,
rather than add a superfluous function to the eclass.

> +# @FUNCTION: go-module_setup_proxy
> +# @DESCRIPTION:
> +# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
> +# this function in src_unpack.
> +# It sets up the go module proxy in the appropriate location and exports
> +# the GOPROXY environment variable so that go calls will be able to
> +# locate the proxy directory.
> +go-module_setup_proxy() {
> + # shellcheck disable=SC2120
> + debug-print-function "${FUNCNAME}" "$@"
> +
> + if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
> + die "go-module_set_globals must be called in global scope"
> + fi
> +
> + local goproxy_dir="${T}/go-proxy"
> + mkdir -p "${goproxy_dir}" || die
> +
> + # For each Golang module distfile, look up where it's supposed to go and
> + # symlink it into place.
> + local f
> + local goproxy_mod_dir
> + for f in ${A}; do
> + goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
> + if [[ -n "${goproxy_mod_path}" ]]; then
> + debug-print-function "Populating go proxy for 
> ${goproxy_mod_path}"
> + # Build symlink hierarchy
> + goproxy_mod_dir=$( dirname 
> "${goproxy_dir}"/"${goproxy_mod_path}" )
> + mkdir -p "${goproxy_mod_dir}" || die
> + ln -sf "${DISTDIR}"/"${f}" 
> "${goproxy_dir}/${goproxy_mod_path}" ||
> + die "Failed to ln"
> + local v=${goproxy_mod_path}
> + v="${v%.mod}"
> + v="${v%.zip}"
> + v="${v//*\/}"
> + _go-module_gosum_synthesize_files "${goproxy_mod_dir}" 
> "${v}"
> + fi
> + done
> + export GOPROXY="file://${goproxy_dir}"
> +
> + # Validate the gosum now
> + _go-module_src_unpack_verify_gosum
> +}
> +
>  # @FUNCTION: go-module_src_unpack
>  # @DESCRIPTION:
>  # If EGO_SUM is set, unpack the base tarball(s) and set up the
> 

The go-module_setup_proxy function solves bug 790851 nicely, since
sys-cluster/k3s ebuilds can call that instead of go-module_src_unpack.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH v2 2/2] ebuild.5: Add eapply documentation

2021-04-24 Thread Zac Medico
On 4/10/21 6:23 PM, Nekun wrote:
> Signed-off-by: Nekun 
> ---
>  man/ebuild.5 | 20 
>  1 file changed, 20 insertions(+)
Thank you! I've merged these and noted it on the bug here:

https://bugs.gentoo.org/698244#c1

Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] emerge: use parse_intermixed_args when available (bug 784566)

2021-04-20 Thread Zac Medico
The included unit test case previously failed with this error:

  emerge: error: unrecognized arguments: dev-libs/A

Bug: https://bugs.gentoo.org/784566
Signed-off-by: Zac Medico 
---
 lib/_emerge/main.py |  2 +-
 lib/portage/tests/emerge/test_simple.py | 10 +-
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/main.py b/lib/_emerge/main.py
index 31e690584..191be6479 100644
--- a/lib/_emerge/main.py
+++ b/lib/_emerge/main.py
@@ -834,7 +834,7 @@ def parse_opts(tmpcmdline, silent=False):
 
tmpcmdline = insert_optional_args(tmpcmdline)
 
-   myoptions = parser.parse_args(args=tmpcmdline)
+   myoptions = getattr(parser, "parse_intermixed_args", 
parser.parse_args)(args=tmpcmdline)
 
if myoptions.alert in true_y:
myoptions.alert = True
diff --git a/lib/portage/tests/emerge/test_simple.py 
b/lib/portage/tests/emerge/test_simple.py
index 6e282337f..5b110407f 100644
--- a/lib/portage/tests/emerge/test_simple.py
+++ b/lib/portage/tests/emerge/test_simple.py
@@ -1,6 +1,7 @@
 # Copyright 2011-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
+import argparse
 import subprocess
 
 import portage
@@ -289,7 +290,14 @@ call_has_and_best_version() {
port=binhost_server.server_port,
path=binhost_remote_path)
 
-   test_commands = (
+   test_commands = ()
+
+   if hasattr(argparse.ArgumentParser, "parse_intermixed_args"):
+   test_commands += (
+   emerge_cmd + ("--oneshot", "dev-libs/A", "-v", 
"dev-libs/A"),
+   )
+
+   test_commands += (
emerge_cmd + ("--usepkgonly", "--root", cross_root, 
"--quickpkg-direct=y", "--quickpkg-direct-root", "/", "dev-libs/A"),
emerge_cmd + ("--usepkgonly", "--quickpkg-direct=y", 
"--quickpkg-direct-root", cross_root, "dev-libs/A"),
env_update_cmd,
-- 
2.26.2




Re: [gentoo-portage-dev] profile-formats not respected ?

2021-04-20 Thread Zac Medico
On 4/20/21 5:03 AM, Joakim Tjernlund wrote:
> On Mon, 2021-04-19 at 14:10 -0700, Zac Medico wrote:
>> On 4/19/21 6:36 AM, Joakim Tjernlund wrote:
>>> I got an embedded ppc32 system which I build in a QEMU user chroot and I 
>>> cannot
>>> make the profile-formats = portage-2 profile-bashrcs profile-set in my own 
>>> profiles layout.conf
>>> work for me.
>>> Seems like portage just ignores this setting and I cannot understand why.
>>> Any pointers?
>>>
>>>  Jocke
>>>
>>
>> Hopefully this command will provide a clue:
>>
>> python -c 'import portage;
>> print(portage.settings._locations_manager.profiles_complex)'
> 
> Got some progress. I got a profile.bashrc: 
> .../my-overlay/profiles/cusfpv3/profile.bashrc
> where I setup PKG_INSTALL_MASK/INSTALL_MASK and this does not work in
> portage-3.0.18. I can echo the vars. and see them set but portage ignores 
> these settings.
> This system was previously using a very old portage, 2.3.76 and that was fine.
> 
> 
> Another strange thing is my own set, i need to keep that at 
> .../my-overlay/{sets, sets.conf}
> In our amd64 DE profile I can have my sets at 
> .../my-overlay/profiles/infinera/sets , why is that?

A setting like this in /etc/portage/sets.conf would do it:

[my-overlay-profile-sets]
class = portage.sets.files.StaticFileSet
multiset = true
directory = /my-overlay/profiles/infinera/sets
world-candidate = true

> Anyhow, here is my profile printout:
> ./print-port.py 
> (_profile_node(location='/usr/portage/profiles/base', 
> portage1_directories=True, user_config=False, 
> profile_formats=('portage-1-compat',), eapi='5', allow_build_id=False,
> show_deprecated_warning=False), 
> _profile_node(location='/usr/portage/profiles/default/linux', 
> portage1_directories=True, user_config=False, 
> profile_formats=('portage-1-compat',),
> eapi='5', allow_build_id=False, show_deprecated_warning=False), 
> _profile_node(location='/usr/portage/profiles/arch/base', 
> portage1_directories=True, user_config=False,
> profile_formats=('portage-1-compat',), eapi='5', allow_build_id=False, 
> show_deprecated_warning=False), 
> _profile_node(location='/usr/portage/profiles/arch/powerpc',
> portage1_directories=True, user_config=False, 
> profile_formats=('portage-1-compat',), eapi='5', allow_build_id=False, 
> show_deprecated_warning=False),
> _profile_node(location='/usr/portage/profiles/default/linux/powerpc', 
> portage1_directories=True, user_config=False, 
> profile_formats=('portage-1-compat',), eapi='5',
> allow_build_id=False, show_deprecated_warning=False), 
> _profile_node(location='/usr/portage/profiles/arch/base', 
> portage1_directories=True, user_config=False,
> profile_formats=('portage-1-compat',), eapi='5', allow_build_id=False, 
> show_deprecated_warning=False), 
> _profile_node(location='/usr/portage/profiles/arch/powerpc',
> portage1_directories=True, user_config=False, 
> profile_formats=('portage-1-compat',), eapi='5', allow_build_id=False, 
> show_deprecated_warning=False),
> _profile_node(location='/usr/portage/profiles/arch/powerpc/ppc32', 
> portage1_directories=True, user_config=False, 
> profile_formats=('portage-1-compat',), eapi='5',
> allow_build_id=False, show_deprecated_warning=False), 
> _profile_node(location='/usr/portage/profiles/default/linux/powerpc/ppc32', 
> portage1_directories=True, user_config=False,
> profile_formats=('portage-1-compat',), eapi='5', allow_build_id=False, 
> show_deprecated_warning=False), 
> _profile_node(location='/usr/portage/profiles/releases',
> portage1_directories=True, user_config=False, 
> profile_formats=('portage-1-compat',), eapi='5', allow_build_id=False, 
> show_deprecated_warning=False),
> _profile_node(location='/usr/portage/profiles/releases/17.0', 
> portage1_directories=True, user_config=False, 
> profile_formats=('portage-1-compat',), eapi='5', allow_build_id=False,
> show_deprecated_warning=False), 
> _profile_node(location='/usr/portage/profiles/default/linux/powerpc/ppc32/17.0',
>  portage1_directories=True, user_config=False,
> profile_formats=('portage-1-compat',), eapi='5', allow_build_id=False, 
> show_deprecated_warning=True), 
> _profile_node(location='/usr/local/portage/tmv3-target-
> overlay/profiles/cusfpv3', portage1_directories=True, user_config=False, 
> profile_formats=('profile-bashrcs', 'portage-2', 'profile-set'), eapi='5', 
> allow_build_id=False,
> show_deprecated_warning=True), _profile_node(location='/etc/portage/profile', 
> portage1_directories=True, user_config=True, 
> profile_formats=('profile-bashrcs', 'profile-set'),
> eapi=None, allow_build_id=True, show_deprecated_warning=False))
> 

Are these effective profile_formats now consistent with your
expectations, and do they now behave as you would expect?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] profile-formats not respected ?

2021-04-19 Thread Zac Medico
On 4/19/21 6:36 AM, Joakim Tjernlund wrote:
> I got an embedded ppc32 system which I build in a QEMU user chroot and I 
> cannot
> make the profile-formats = portage-2 profile-bashrcs profile-set in my own 
> profiles layout.conf
> work for me.
> Seems like portage just ignores this setting and I cannot understand why.
> Any pointers?
> 
>  Jocke
> 

Hopefully this command will provide a clue:

python -c 'import portage;
print(portage.settings._locations_manager.profiles_complex)'
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] make.conf.5: Sugest PORTAGE_LOG_FILTER_FILE_CMD supervisor for cat fallback (bug 781854)

2021-04-09 Thread Zac Medico
If PORTAGE_LOG_FILTER_FILE_CMD fails after exec, then output
will be lost. Therefore, suggest to use bash as a supervisor,
with fallback to cat.

Bug: https://bugs.gentoo.org/781854
Signed-off-by: Zac Medico 
---
 man/make.conf.5 | 15 +--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/man/make.conf.5 b/man/make.conf.5
index 8d551c95e..8a4a2ae30 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -1,4 +1,4 @@
-.TH "MAKE.CONF" "5" "Feb 2021" "Portage VERSION" "Portage"
+.TH "MAKE.CONF" "5" "Apr 2021" "Portage VERSION" "Portage"
 .SH "NAME"
 make.conf \- custom settings for Portage
 .SH "SYNOPSIS"
@@ -998,7 +998,18 @@ will set idle io priority. For more information about 
ionice, see
 .B PORTAGE_LOG_FILTER_FILE_CMD
 This variable specifies a command that filters build log output to a
 log file. In order to filter ANSI escape codes from build logs,
-\fBansifilter\fR(1) is a convenient setting for this variable.
+\fBansifilter\fR(1) is a convenient setting for this variable. Generally,
+PORTAGE_LOG_FILTER_FILE_CMD should include a supervisor that falls back
+to cat if the real filter command fails after exec. For example, a
+supervisor is needed for ansifilter, in case it fails after exec due to
+a problem resolving libstdc++ during a gcc upgrade.
+.br
+.I Example:
+.nf
+# Use bash as a supervisor, for fallback to cat if asifilter fails
+# after exec due to a problem resolving libstdc++ during a gcc upgrade.
+PORTAGE_LOG_FILTER_FILE_CMD="bash -c \\"ansifilter; exec cat\\""
+.fi
 .TP
 .B PORTAGE_LOGDIR
 This variable defines the directory in which per\-ebuild logs are kept.
-- 
2.26.2




Re: [gentoo-portage-dev] Implement new userpatch feature in existing eclass?

2021-04-04 Thread Zac Medico
On 4/4/21 10:54 AM, Nekun wrote:
> Hi all.
> 
> Recently, I start working on optional atom specifiers feature in
> userpatch facility: if package directory name starts with percent sign,
> following word threated as a regular Portage atom, e.g
> "/etc/portage/patches/sys-kernel/%<=gentoo-sources-5.4" ==
> "<=sys-kernel/gentoo-sources-5.4". This might be very useful in cases
> when patches applied to minor updates, but major update breaks it (e.g.,
> in Linux kernel), so I want to specify smth like "=gentoo-sources-5.4*".
> I added new command in portageq to match two atoms and call it from
> eapply_user function in phase-function.sh, in same manner as
> has_version/best_version are called it. But recently I found that
> eapply_user implemented in Portage only in EAPI 6, and there is its
> predecessor, epatch_user, implemented in epatch.eclass. So, ebuilds with
> EAPI<6 (I found 4463 in last gentoo snapshot) will ignore new "atomic"
> patch directories. Obviously, this is rather confusing, unacceptable
> behaviour.
> 
> Can I patch epatch.eclass in gentoo repository to implement new
> userpatch facility for older EAPIs? I guess that EAPI version is
> considered as stable, unchangeable behaviour of all functions, but in
> other side, this feature doesn't changes anything existing: old
> userpatch semantics preserves and order of applying
> (${P}-${PR},${P},${PN}) not changed, seeking for atoms added at tail.
> 

Today, I count only 2445, or 8.3% of ebuilds have EAPI 5.

I imagine that the migration is moving along, since we deprecated EAPI 5
on 2020-11-26 here:

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b2e281bb698eb93704e1987dc4df1cf2dd3c2cff
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] Implement new userpatch feature in existing eclass?

2021-04-04 Thread Zac Medico
On 4/4/21 10:54 AM, Nekun wrote:
> Hi all.
> 
> Recently, I start working on optional atom specifiers feature in
> userpatch facility: if package directory name starts with percent sign,
> following word threated as a regular Portage atom, e.g
> "/etc/portage/patches/sys-kernel/%<=gentoo-sources-5.4" ==
> "<=sys-kernel/gentoo-sources-5.4". This might be very useful in cases
> when patches applied to minor updates, but major update breaks it (e.g.,
> in Linux kernel), so I want to specify smth like "=gentoo-sources-5.4*".
> I added new command in portageq to match two atoms and call it from
> eapply_user function in phase-function.sh, in same manner as
> has_version/best_version are called it. But recently I found that
> eapply_user implemented in Portage only in EAPI 6, and there is its
> predecessor, epatch_user, implemented in epatch.eclass. So, ebuilds with
> EAPI<6 (I found 4463 in last gentoo snapshot) will ignore new "atomic"
> patch directories. Obviously, this is rather confusing, unacceptable
> behaviour.
> 
> Can I patch epatch.eclass in gentoo repository to implement new
> userpatch facility for older EAPIs? I guess that EAPI version is
> considered as stable, unchangeable behaviour of all functions, but in
> other side, this feature doesn't changes anything existing: old
> userpatch semantics preserves and order of applying
> (${P}-${PR},${P},${PN}) not changed, seeking for atoms added at tail.
> 

Today, I count only 2445, or 8.3% of ebuilds have EAPI 5.

I imagine that the migration is moving along, since we deprecated EAPI 5
on 2020-11-26 here:

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b2e281bb698eb93704e1987dc4df1cf2dd3c2cff

-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH] Use atomic_ofstream as Context Manager i.e., with-statement contexts

2021-03-16 Thread Zac Medico
On 3/8/21 11:25 PM, Florian Schmaus wrote:
> With [1: e93e6d65fa1c] atomic_ofstream became a Context Manager. This
> commit transforms three further call sites of atomic_ofstream() to use
> with-statement contexts for easier readability and increased
> robustness against resource leaks.
> 
> 1: e93e6d65fa1ca75f676a227f7918f8b6d747425c
>Make atomic_ofstream a Context Manager
> 
> Signed-off-by: Florian Schmaus 
> ---
>  lib/_emerge/BlockerCache.py|  6 +++---
>  lib/portage/dbapi/_VdbMetadataDelta.py | 11 +--
>  lib/portage/dbapi/vartree.py   |  6 +++---

Merged. Thank you!

https://gitweb.gentoo.org/proj/portage.git/commit/?id=8879cef9006d2277453aaee407c234a2d1bc47ba
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH] Mark EAPIs "4-python" and "5-progress" as deprecated

2021-03-06 Thread Zac Medico
On 3/4/21 11:35 AM, Matt Turner wrote:
> Signed-off-by: Matt Turner 
> ---
> I've asked Arfrever multiple times if these are still used anywhere, and
> he seemingly has not responded intentionally.
> 
> According to https://bugs.gentoo.org/174536#c27 these EAPIs were only
> used in Arfrever's personal overlay, and even in 2012 there were
> questions about why they were supported in portage.
> 
> The "Progress Overlay" does contain ebuilds using these EAPIs but it has
> not been updated since 2018 and doesn't look like it is useful at this
> point.
> 
>  lib/portage/__init__.py | 8 
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py
> index 24c9d8b89..184db6ae2 100644
> --- a/lib/portage/__init__.py
> +++ b/lib/portage/__init__.py
> @@ -465,16 +465,16 @@ def abssymlink(symlink, target=None):
>  _doebuild_manifest_exempt_depend = 0
>  
>  _testing_eapis = frozenset([
> - "4-python",
> - "5-progress",
>  ])
>  _deprecated_eapis = frozenset([
> + "3_pre1",
> + "3_pre2",
>   "4_pre1",
> + "4-python",
>   "4-slot-abi",
> - "3_pre2",
> - "3_pre1",
>   "5_pre1",
>   "5_pre2",
> + "5-progress",
>   "6_pre1",
>   "7_pre1",
>  ])
> 

Merged, thanks!

https://gitweb.gentoo.org/proj/portage.git/commit/?id=67cf9c2b05042de37f36f5b6840c450128a065bd
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH] Use asyncio.subprocess.Process directly

2021-03-06 Thread Zac Medico
On 3/4/21 11:24 AM, Matt Turner wrote:
> With no need to support Python 2, we can remove our private
> implementation.
> 
> Signed-off-by: Matt Turner 
> ---
> I don't know how to test this. I intentionally broke the return value of
> create_subprocess_exec and didn't see any bad results.
> 
>  lib/portage/util/futures/_asyncio/__init__.py |   8 +-
>  lib/portage/util/futures/_asyncio/process.py  | 116 --
>  2 files changed, 4 insertions(+), 120 deletions(-)
>  delete mode 100644 lib/portage/util/futures/_asyncio/process.py

Merged, thanks!

https://gitweb.gentoo.org/proj/portage.git/commit/?id=1e843f853a9afe82d599e6ab09064147ddc1d271


> diff --git a/lib/portage/util/futures/_asyncio/__init__.py 
> b/lib/portage/util/futures/_asyncio/__init__.py
> index 5590963f1..207e7205d 100644
> --- a/lib/portage/util/futures/_asyncio/__init__.py
> +++ b/lib/portage/util/futures/_asyncio/__init__.py
> @@ -25,6 +25,7 @@ import types
>  import weakref
>  
>  import asyncio as _real_asyncio
> +from asyncio.subprocess import Process
>  
>  try:
>   import threading
> @@ -138,7 +138,7 @@ def create_subprocess_exec(*args, **kwargs):
>  
>   result = loop.create_future()
>  
> - result.set_result(_Process(subprocess.Popen(
> + result.set_result(Process(subprocess.Popen(
>   args,
>   stdin=kwargs.pop('stdin', None),
>   stdout=kwargs.pop('stdout', None),

The above area is actually no longer used, since we should always have a
_AsyncioEventLoop instance here, and we can remove the EventLoop class now.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH 3/3] lib: Remove outdated Python 2 comments

2021-03-06 Thread Zac Medico
On 3/4/21 11:23 AM, Matt Turner wrote:
> Fixes: 788c0e8bb ("Remove from __future__ import unicode_literals")
> Signed-off-by: Matt Turner 
> ---
>  bin/egencache   | 2 --
>  lib/_emerge/Package.py  | 9 -
>  lib/_emerge/Scheduler.py| 2 --
>  lib/_emerge/UseFlagDisplay.py   | 2 --
>  lib/_emerge/resolver/output.py  | 2 --
>  lib/portage/cache/flat_hash.py  | 3 ---
>  lib/portage/tests/unicode/test_string_format.py | 9 -
>  lib/portage/util/digraph.py | 3 ---
>  8 files changed, 32 deletions(-)

Series looks good. Merged. Thanks!

https://gitweb.gentoo.org/proj/portage.git/commit/?id=9003c5201c6503ddad9237bcffbc6f775567661b
https://gitweb.gentoo.org/proj/portage.git/commit/?id=af100c65ebf7fd84307a84819602a934ebb0741c
https://gitweb.gentoo.org/proj/portage.git/commit/?id=21c6c0c1088ded78397594bfd78102361f8d837b
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-dev] Package up for grabs: dev-python/python-backoff

2021-03-02 Thread Zac Medico
We have and ebuild for this package up for grabs:

https://pypi.org/project/backoff/

No open bugs and no reverse dependencies:

https://bugs.gentoo.org/buglist.cgi?quicksearch=python-backoff
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] emerge: make --binpkg-respect-use=y imply --autounmask-use=n

2021-03-01 Thread Zac Medico
If --binpkg-respect-use=y is given explicitly, then it implies
--autounmask-use=n, because these options naturally oppose
eachother.

Bug: https://bugs.gentoo.org/773469
Signed-off-by: Zac Medico 
---
 lib/_emerge/create_depgraph_params.py   | 22 ++---
 lib/portage/tests/resolver/test_useflags.py | 20 +--
 man/emerge.1| 11 +++
 3 files changed, 32 insertions(+), 21 deletions(-)

diff --git a/lib/_emerge/create_depgraph_params.py 
b/lib/_emerge/create_depgraph_params.py
index 25dd2a1b4..267600fb6 100644
--- a/lib/_emerge/create_depgraph_params.py
+++ b/lib/_emerge/create_depgraph_params.py
@@ -41,12 +41,22 @@ def create_depgraph_params(myopts, myaction):
# binpkg_changed_deps: reject binary packages with outdated deps
myparams = {"recurse" : True}
 
+   binpkg_respect_use = myopts.get("--binpkg-respect-use")
+   if binpkg_respect_use is not None:
+   myparams["binpkg_respect_use"] = binpkg_respect_use
+   elif "--usepkgonly" not in myopts:
+   # If --binpkg-respect-use is not explicitly specified, we enable
+   # the behavior automatically (like requested in bug #297549), as
+   # long as it doesn't strongly conflict with other options that
+   # have been specified.
+   myparams["binpkg_respect_use"] = "auto"
+
autounmask_keep_keywords = myopts.get("--autounmask-keep-keywords")
autounmask_keep_masks = myopts.get("--autounmask-keep-masks")
 
autounmask = myopts.get("--autounmask")
autounmask_license = myopts.get('--autounmask-license', 'y' if 
autounmask is True else 'n')
-   autounmask_use = myopts.get('--autounmask-use')
+   autounmask_use = 'n' if myparams.get('binpkg_respect_use') == 'y' else 
myopts.get('--autounmask-use')
if autounmask == 'n':
autounmask = False
else:
@@ -153,16 +163,6 @@ def create_depgraph_params(myopts, myaction):
'--update' in myopts:
myparams['rebuilt_binaries'] = True
 
-   binpkg_respect_use = myopts.get('--binpkg-respect-use')
-   if binpkg_respect_use is not None:
-   myparams['binpkg_respect_use'] = binpkg_respect_use
-   elif '--usepkgonly' not in myopts:
-   # If --binpkg-respect-use is not explicitly specified, we enable
-   # the behavior automatically (like requested in bug #297549), as
-   # long as it doesn't strongly conflict with other options that
-   # have been specified.
-   myparams['binpkg_respect_use'] = 'auto'
-
binpkg_changed_deps = myopts.get('--binpkg-changed-deps')
if binpkg_changed_deps is not None:
myparams['binpkg_changed_deps'] = binpkg_changed_deps
diff --git a/lib/portage/tests/resolver/test_useflags.py 
b/lib/portage/tests/resolver/test_useflags.py
index d66da0866..b799e62ff 100644
--- a/lib/portage/tests/resolver/test_useflags.py
+++ b/lib/portage/tests/resolver/test_useflags.py
@@ -46,15 +46,23 @@ class UseFlagsTestCase(TestCase):
success = True,
mergelist = ["dev-libs/A-1"]),
 
-   # In the unit test case for bug 773469, the 
--autounmask-backtrack option
-   # is needed in order to trigger the 
--binpkg-respect-use=y behavior that
-   # appears confusingly similar to --binpkg-respect-use=n 
behavior.
+   # For bug 773469, we wanted --binpkg-respect-use=y to 
trigger a
+   # slot collision. Instead, a combination of default 
--autounmask-use
+   # combined with --autounmask-backtrack=y from 
EMERGE_DEFAULT_OPTS
+   # triggered this behavior which appeared confusingly 
similar to
+   #--binpkg-respect-use=n behavior.
+   #ResolverPlaygroundTestCase(
+   #   ["dev-libs/C", "dev-libs/D"],
+   #   options={"--usepkg": True, 
"--binpkg-respect-use": "y", "--autounmask-backtrack": "y"},
+   #   success=True,
+   #   use_changes={"dev-libs/C-1": {"abi_x86_32": 
True}},
+   #   mergelist=["[binary]dev-libs/C-1", 
"[binary]dev-libs/D-1"],
ResolverPlaygroundTestCase(
["dev-libs/C", "dev-libs/D"],
options={"--usepkg": True, 
"--binpkg-respect-use": "y", "--autounmask-backtrack": "y"},
-   success=True,
-

[gentoo-portage-dev] Planning to publish portage releases on pypi

2021-02-28 Thread Zac Medico
I'd like to begin publishing portage releases on pypi here:

https://pypi.org/project/portage

It won't allow me to create the the project, so I've opened this ticket
to claim it:

https://github.com/pypa/pypi-support/issues/934
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH v3] emirrordist: add --content-db option required for content-hash layout (bug 756778)

2021-02-26 Thread Zac Medico
Add a --content-db option which is required for the content-hash
layout because its file listings return content digests instead of
distfile names.

The content db serves to translate content digests to distfiles
names, and distfiles names to content digests. All keys have a
prefix separated by a colon. For digest keys, the prefix is the
hash algorithm name. For filename keys, the prefix is "filename".

The value associated with a digest key is a set of file names. The
value associated with a distfile key is a set of content revisions.
Each content revision is expressed as a dictionary of digests which
is suitable for construction of a DistfileName instance.

Bug: https://bugs.gentoo.org/756778
Signed-off-by: Zac Medico 
---
[PATCH v3] changed the value associated with a digest key is a set
of file name, and fixed ContentDB.remove to preserved independent
references to identical content (like removing one of multiple
hardlinks).

 lib/portage/_emirrordist/Config.py   |   8 +-
 lib/portage/_emirrordist/ContentDB.py| 178 +++
 lib/portage/_emirrordist/DeletionIterator.py |  25 ++-
 lib/portage/_emirrordist/DeletionTask.py |   8 +
 lib/portage/_emirrordist/FetchTask.py|   5 +-
 lib/portage/_emirrordist/main.py |  15 +-
 lib/portage/package/ebuild/fetch.py  |   8 +-
 lib/portage/tests/ebuild/test_fetch.py   |  14 ++
 man/emirrordist.1|   6 +-
 9 files changed, 256 insertions(+), 11 deletions(-)
 create mode 100644 lib/portage/_emirrordist/ContentDB.py

diff --git a/lib/portage/_emirrordist/Config.py 
b/lib/portage/_emirrordist/Config.py
index 4bee4f45e..cfe944040 100644
--- a/lib/portage/_emirrordist/Config.py
+++ b/lib/portage/_emirrordist/Config.py
@@ -1,4 +1,4 @@
-# Copyright 2013-2020 Gentoo Authors
+# Copyright 2013-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import copy
@@ -10,6 +10,7 @@ import time
 from portage import os
 from portage.package.ebuild.fetch import MirrorLayoutConfig
 from portage.util import grabdict, grablines
+from .ContentDB import ContentDB
 
 class Config:
def __init__(self, options, portdb, event_loop):
@@ -65,6 +66,11 @@ class Config:
self.distfiles_db = self._open_shelve(
options.distfiles_db, 'distfiles')
 
+   self.content_db = None
+   if options.content_db is not None:
+   self.content_db = ContentDB(self._open_shelve(
+   options.content_db, 'content'))
+
self.deletion_db = None
if options.deletion_db is not None:
self.deletion_db = self._open_shelve(
diff --git a/lib/portage/_emirrordist/ContentDB.py 
b/lib/portage/_emirrordist/ContentDB.py
new file mode 100644
index 0..7084cecff
--- /dev/null
+++ b/lib/portage/_emirrordist/ContentDB.py
@@ -0,0 +1,178 @@
+# Copyright 2021 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+import logging
+import operator
+import shelve
+import typing
+
+from portage.package.ebuild.fetch import DistfileName
+
+
+class ContentDB:
+   """
+   The content db serves to translate content digests to distfiles
+   names, and distfiles names to content digests. All keys have a
+   prefix separated by a colon. For digest keys, the prefix is the
+   hash algorithm name. For filename keys, the prefix is "filename".
+
+   The value associated with a digest key is a set of file names. The
+   value associated with a distfile key is a set of content revisions.
+   Each content revision is expressed as a dictionary of digests which
+   is suitable for construction of a DistfileName instance.
+   """
+
+   def __init__(self, shelve_instance: shelve.Shelf):
+   self._shelve = shelve_instance
+
+   def add(self, filename: DistfileName):
+   """
+   Add file name and digests.
+
+   @param filename: file name with digests attribute
+   """
+   distfile_str = str(filename)
+   distfile_key = "filename:{}".format(distfile_str)
+   for k, v in filename.digests.items():
+   if k != "size":
+   digest_key = "{}:{}".format(k, v).lower()
+   try:
+   digest_files = self._shelve[digest_key]
+   except KeyError:
+   digest_files = set()
+   digest_files.add(distfile_str)
+   self._shelve[digest_key] = digest_files
+   try:
+   content_revisions = self._shelve[distfile_key]
+  

[gentoo-portage-dev] [PATCH] make.globals: make FEATURES=-binpkg-multi-instance sticky for existing installs

2021-02-26 Thread Zac Medico
Add a _compat_upgrade.binpkg_multi_instance script that the ebuild
can call in pkg_preinst in order to maintain a backward-compatible
FEATURES=-binpkg-multi-instance default on existing installs where
enabling binpkg-multi-instance could cause disruption.

Bug: https://bugs.gentoo.org/772785
Signed-off-by: Zac Medico 
---
 .../_compat_upgrade/binpkg_multi_instance.py  | 33 +++
 1 file changed, 33 insertions(+)
 create mode 100644 lib/portage/_compat_upgrade/binpkg_multi_instance.py

diff --git a/lib/portage/_compat_upgrade/binpkg_multi_instance.py 
b/lib/portage/_compat_upgrade/binpkg_multi_instance.py
new file mode 100644
index 0..bfeb72baa
--- /dev/null
+++ b/lib/portage/_compat_upgrade/binpkg_multi_instance.py
@@ -0,0 +1,33 @@
+# Copyright 2021 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+import portage
+from portage import os
+from portage.const import GLOBAL_CONFIG_PATH
+
+COMPAT_FEATURES = 'FEATURES="${FEATURES} -binpkg-multi-instance"'
+
+
+def main():
+   """
+   If the current installation is still has binpkg-multi-instance
+   disabled, then patch make.globals inside ${ED} to maintain backward
+   compatibility. This is intended to be called from the ebuild as
+   follows:
+
+   pkg_preinst() {
+   python_setup
+   env -u FEATURES
+   
PYTHONPATH="${D%/}$(python_get_sitedir)${PYTHONPATH:+:${PYTHONPATH}}" \
+   "${PYTHON}" -m 
portage._compat_upgrade.binpkg_multi_instance || die
+   }
+   """
+   if 'binpkg-multi-instance' not in portage.settings.features:
+   portage.output.EOutput().einfo('Setting make.globals default {} 
for backward compatibility'.format(COMPAT_FEATURES))
+   config_path = os.path.join(os.environ['ED'], 
GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals')
+   with open(config_path, 'at') as f:
+   f.write("{}\n".format(COMPAT_FEATURES))
+
+
+if __name__ == '__main__':
+   main()
-- 
2.26.2




[gentoo-portage-dev] [PATCH v2] emirrordist: add --content-db option required for content-hash layout (bug 756778)

2021-02-26 Thread Zac Medico
Add a --content-db option which is required for the content-hash
layout because its file listings return content digests instead of
distfile names.

The content db serves to translate content digests to distfiles
names, and distfiles names to content digests. All keys have a
prefix separated by a colon. For digest keys, the prefix is the
hash algorithm name. For filename keys, the prefix is "filename".

The value associated with a digest key is a plain filename. The
value associated with a distfile key is a set of content revisions.
Each content revision is expressed as a dictionary of digests which
is suitable for construction of a DistfileName instance.

Bug: https://bugs.gentoo.org/756778
Signed-off-by: Zac Medico 
---
[PATCH v2] Split out ContentDB class and associate distfile key
with a set of content revisions, where each content revision is
expressed as a dictionary of digests.

 lib/portage/_emirrordist/Config.py   |   8 +-
 lib/portage/_emirrordist/ContentDB.py| 158 +++
 lib/portage/_emirrordist/DeletionIterator.py |  25 ++-
 lib/portage/_emirrordist/DeletionTask.py |   8 +
 lib/portage/_emirrordist/FetchTask.py|   5 +-
 lib/portage/_emirrordist/main.py |  15 +-
 lib/portage/tests/ebuild/test_fetch.py   |  14 ++
 man/emirrordist.1|   6 +-
 8 files changed, 232 insertions(+), 7 deletions(-)
 create mode 100644 lib/portage/_emirrordist/ContentDB.py

diff --git a/lib/portage/_emirrordist/Config.py 
b/lib/portage/_emirrordist/Config.py
index 4bee4f45e..cfe944040 100644
--- a/lib/portage/_emirrordist/Config.py
+++ b/lib/portage/_emirrordist/Config.py
@@ -1,4 +1,4 @@
-# Copyright 2013-2020 Gentoo Authors
+# Copyright 2013-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import copy
@@ -10,6 +10,7 @@ import time
 from portage import os
 from portage.package.ebuild.fetch import MirrorLayoutConfig
 from portage.util import grabdict, grablines
+from .ContentDB import ContentDB
 
 class Config:
def __init__(self, options, portdb, event_loop):
@@ -65,6 +66,11 @@ class Config:
self.distfiles_db = self._open_shelve(
options.distfiles_db, 'distfiles')
 
+   self.content_db = None
+   if options.content_db is not None:
+   self.content_db = ContentDB(self._open_shelve(
+   options.content_db, 'content'))
+
self.deletion_db = None
if options.deletion_db is not None:
self.deletion_db = self._open_shelve(
diff --git a/lib/portage/_emirrordist/ContentDB.py 
b/lib/portage/_emirrordist/ContentDB.py
new file mode 100644
index 0..60e6ef39d
--- /dev/null
+++ b/lib/portage/_emirrordist/ContentDB.py
@@ -0,0 +1,158 @@
+# Copyright 2021 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+import logging
+import operator
+import shelve
+import typing
+
+from portage.package.ebuild.fetch import DistfileName
+
+
+class ContentDB:
+   """
+   The content db serves to translate content digests to distfiles
+   names, and distfiles names to content digests. All keys have a
+   prefix separated by a colon. For digest keys, the prefix is the
+   hash algorithm name. For filename keys, the prefix is "filename".
+
+   The value associated with a digest key is a plain filename. The
+   value associated with a distfile key is a set of content revisions.
+   Each content revision is expressed as a dictionary of digests which
+   is suitable for construction of a DistfileName instance.
+   """
+
+   def __init__(self, shelve_instance: shelve.Shelf):
+   self._shelve = shelve_instance
+
+   def add(self, filename: DistfileName):
+   """
+   Add file name and digests.
+
+   @param filename: file name with digests attribute
+   """
+   distfile_str = str(filename)
+   distfile_key = "filename:{}".format(distfile_str)
+   for k, v in filename.digests.items():
+   if k != "size":
+   self._shelve["{}:{}".format(k, v).lower()] = 
distfile_str
+   try:
+   content_revisions = self._shelve[distfile_key]
+   except KeyError:
+   content_revisions = set()
+
+   revision_key = tuple(
+   sorted(
+   (
+   (algo.lower(), 
filename.digests[algo].lower())
+   for algo in filename.digests
+   

[gentoo-portage-dev] [PATCH] emirrordist: add --content-db option required for content-hash layout (bug 756778)

2021-02-24 Thread Zac Medico
Add a --content-db option which is required for the content-hash
layout because its file listings return content digests instead of
distfile names.

The content db includes a reverse mapping, for use during garbage
collection. All keys have a prefix separated by a colon. For digest
keys, the prefix is the hash algorithm name. For filename keys,
the prefix is "filename". The values for digest keys are plain
filenames, and the values for distfile keys are dictionaries
of digests suitable for construction of DistfileName instances.

Bug: https://bugs.gentoo.org/756778
Signed-off-by: Zac Medico 
---
 lib/portage/_emirrordist/Config.py   |  7 +++-
 lib/portage/_emirrordist/DeletionIterator.py | 38 ++--
 lib/portage/_emirrordist/DeletionTask.py | 26 ++
 lib/portage/_emirrordist/FetchTask.py| 16 -
 lib/portage/_emirrordist/main.py | 15 +++-
 lib/portage/tests/ebuild/test_fetch.py   | 14 
 man/emirrordist.1|  6 +++-
 7 files changed, 116 insertions(+), 6 deletions(-)

diff --git a/lib/portage/_emirrordist/Config.py 
b/lib/portage/_emirrordist/Config.py
index 4bee4f45e..53f6582fe 100644
--- a/lib/portage/_emirrordist/Config.py
+++ b/lib/portage/_emirrordist/Config.py
@@ -1,4 +1,4 @@
-# Copyright 2013-2020 Gentoo Authors
+# Copyright 2013-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import copy
@@ -65,6 +65,11 @@ class Config:
self.distfiles_db = self._open_shelve(
options.distfiles_db, 'distfiles')
 
+   self.content_db = None
+   if options.content_db is not None:
+   self.content_db = self._open_shelve(
+   options.content_db, 'content')
+
self.deletion_db = None
if options.deletion_db is not None:
self.deletion_db = self._open_shelve(
diff --git a/lib/portage/_emirrordist/DeletionIterator.py 
b/lib/portage/_emirrordist/DeletionIterator.py
index 08985ed6c..24fb096bf 100644
--- a/lib/portage/_emirrordist/DeletionIterator.py
+++ b/lib/portage/_emirrordist/DeletionIterator.py
@@ -1,10 +1,12 @@
-# Copyright 2013-2019 Gentoo Authors
+# Copyright 2013-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import logging
 import stat
+import typing
 
 from portage import os
+from portage.package.ebuild.fetch import DistfileName
 from .DeletionTask import DeletionTask
 
 class DeletionIterator:
@@ -12,6 +14,37 @@ class DeletionIterator:
def __init__(self, config):
self._config = config
 
+   def _map_filename(self, filename: typing.Union[str, DistfileName]) -> 
typing.Union[str, DistfileName]:
+   """
+   Map a filename listed by the layout get_filenames method,
+   translating it from a content digest to a distfile name.
+   If filename is already a distfile name, then it will pass
+   through unchanged.
+
+   @param filename: A filename listed by layout get_filenames
+   @return: The distfile name, mapped from the corresponding
+   content digest when necessary
+   """
+   if not isinstance(filename, DistfileName):
+   if self._config.content_db is not None:
+   distfile_key = "filename:{}".format(filename)
+   try:
+   digests = 
self._config.content_db[distfile_key]
+   except KeyError:
+   pass
+   else:
+   return DistfileName(filename, 
digests=digests)
+   return DistfileName(filename)
+   if filename.digests and self._config.content_db is not None:
+   for k, v in filename.digests.items():
+   digest_key = "{}:{}".format(k, v).lower()
+   try:
+   distfile_str = 
self._config.content_db[digest_key]
+   except KeyError:
+   continue
+   return DistfileName(distfile_str, digests={k:v})
+   return filename
+
def __iter__(self):
distdir = self._config.options.distfiles
file_owners = self._config.file_owners
@@ -22,7 +55,8 @@ class DeletionIterator:
start_time = self._config.start_time
distfiles_set = set()
for layout in self._config.layouts:
-   distfiles_set.update(layout.get_filenames(distdir))
+

Re: [gentoo-dev] New project: binhost

2021-02-24 Thread Zac Medico
On 2/24/21 2:29 AM, Zac Medico wrote:
> For example, for 3 USE flags, up to 8 combinations will be indexed:
> 
> IUSE="a b c installsources splitdebug"
> SRC_URI="
>   !a? !b? !c? ( mirror://binhost/24fe6bd377 )
>   !a? !b?  c? ( mirror://binhost/fbe14cbb02 )
>   !a?  b? !c? ( mirror://binhost/1dfff1f2ac )
>   !a?  b?  c? ( mirror://binhost/ae60f2940d )
>a? !b? !c? ( mirror://binhost/2976e1acc0 )
>a? !b?  c? ( mirror://binhost/f4809db70c )
>a?  b? !c? ( mirror://binhost/ecd08466cf )
>a?  b?  c? ( mirror://binhost/0c00f33b2e )
>   installsources? (
> !a? !b? !c? ( mirror://binhost/063a14d6c7 )
> !a? !b?  c? ( mirror://binhost/f67c311625 )
> !a?  b? !c? ( mirror://binhost/1dfff1f2ac )
> !a?  b?  c? ( mirror://binhost/17a673e16a )
>  a? !b? !c? ( mirror://binhost/914d1cecfe )
>  a? !b?  c? ( mirror://binhost/ca18d86a2b )
>  a?  b? !c? ( mirror://binhost/6bce13471a )
>  a?  b?  c? ( mirror://binhost/3a6bdcd228 )
>   )
>   splitdebug? (
> !a? !b? !c? ( mirror://binhost/29b2f38c41 )
> !a? !b?  c? ( mirror://binhost/8adc9bef51 )
> !a?  b? !c? ( mirror://binhost/954d2ce484 )
> !a?  b?  c? ( mirror://binhost/32a614aaca )
>  a? !b? !c? ( mirror://binhost/3548a2302d )
>  a? !b?  c? ( mirror://binhost/e0c02cdc88 )
>  a?  b? !c? ( mirror://binhost/f9cbd3c181 )
>  a?  b?  c? ( mirror://binhost/31d4c03474 )
>   )
> "
> 
> For installsources, we can automate deduplication, so that we can
> distribute the same file content for multiple USE combinations when
> appropriate. If all of the combinations have identical content, then it
> will look like this:
> 
>   installsources? (
> !a? !b? !c? ( mirror://binhost/063a14d6c7 )
> !a? !b?  c? ( mirror://binhost/063a14d6c7 )
> !a?  b? !c? ( mirror://binhost/063a14d6c7 )
> !a?  b?  c? ( mirror://binhost/063a14d6c7 )
>  a? !b? !c? ( mirror://binhost/063a14d6c7 )
>  a? !b?  c? ( mirror://binhost/063a14d6c7 )
>  a?  b? !c? ( mirror://binhost/063a14d6c7 )
>  a?  b?  c? ( mirror://binhost/063a14d6c7 )
>   )
> 
> In order to ensure that an ebin is not selected for a USE combination
> that has not been built yet, combinations for existing builds will be
> listed in REQUIRED_USE, like this:
> 
> REQUIRED_USE="
> || (
>   ( !a !b !c )
>   ( !a !b  c )
>   ( !a  b !c )
>   ( !a  b  c )
>   (  a !b !c )
>   (  a !b  c )
>   (  a  b !c )
>   (  a  b  c )
> )
> "

In https://bugs.gentoo.org/772380 I'm planning to implement a script
that will take an existing $PKGDIR as input, and generate an "ebin"
binhost as output. It will automatically split out pre-built content
bundles for installsources and splitdebug as shown above. If there is
more than one build for a particular package version and USE
combination, then the script will choose the package instance with
latest BUILD_TIME metadata (in alignment with
FEATURES=binpkg-multi-instance).
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] New project: binhost

2021-02-24 Thread Zac Medico
On 2/23/21 12:33 PM, Zac Medico wrote:
> On 2/23/21 12:05 PM, Zac Medico wrote:
>> On 2/23/21 11:46 AM, Zac Medico wrote:
>>> On 2/20/21 8:17 PM, Zac Medico wrote:
>>>> IUSE_RUNTIME will obviously introduce conditionals in binary package
>>>> dependencies, but we should welcome these conditionals because they will
>>>> provide useful flexibility.
>>>>
>>>> I think IUSE_RUNTIME will be a very nice feature to have for project
>>>> binhost, since it will allow binary package dependencies to behave with
>>>> flexibility that more closely resembles the flexibility of ebuild
>>>> dependencies.
>>>
>>> We can borrow paludis's notion of pbins [1] to generate ebuilds which
>>> install pre-built content, and the generated ebuilds could have USE
>>> flags that behave similarly to IUSE_RUNTIME in the sense that changes to
>>> USE flags will result in different builds of pre-built content being
>>> installed. A content-hash distfiles layout [2] could serve as a
>>> convenient way to store separate builds of pre-built content for
>>> multiple combinations of USE flags, and a generated ebuild would index
>>> the build by USE flag combination.
>>>
>>> Also, for the generated ebuilds, we can generate USE flags to include
>>> separate SRC_URI downloads for pre-built content to support things like
>>> FEATURES=split-debug and FEATURES=install-sources.
>>
>> Note that all of this can use existing EAPI features, since everything new
>> would be implemented in an ebuild generator that generates a single
>> ebuild to index pre-built content from multiple binary package builds.
>>
>>> [1] https://paludis.exherbo.org/overview/pbins.html
>>> [2] https://bugs.gentoo.org/756778
>>
> 
> For generated ebuilds, we'll have a choice to model things as USE flags
> or sub-packages. For example, content from FEATURES=split-debug and
> FEATURES=install-sources content might make more sense to model as
> sub-packages than USE flags. It makes more sense to generate a
> sub-package when there is a need for the sub-package to have USE flags.
> For example, a split-debug sub-package can have USE flags which index
> pre-built content from builds for multiple USE flag combinations.
> Similar USE flags could be useful for an install-sources sub-package if
> the source code has patches which are conditional on USE flags.

Since the generated ebuilds are inspired by pbins, we might call them
"ebins". Once we have designed an ebin generation process that we're
happy with, we should consider making it part of an EAPI, so that
package managers can generate "ebins" on request. The EAPI should
include ways to split out files and distribute them separately based on
USE flags and/or sub-packages, so that binhost users can easily filter
which files are installed based on USE configuration.

We can automatically map user's splitdebug and installsources FEATURES
settings into USE settings, in the same way that FEATURES=test
automatically maps to USE=test.

Each generated ebuild (ebin) will use its SRC_URI metadata to index
builds for each USE flag combination for which a build exists. For
example, for 3 USE flags, up to 8 combinations will be indexed:

IUSE="a b c installsources splitdebug"
SRC_URI="
  !a? !b? !c? ( mirror://binhost/24fe6bd377 )
  !a? !b?  c? ( mirror://binhost/fbe14cbb02 )
  !a?  b? !c? ( mirror://binhost/1dfff1f2ac )
  !a?  b?  c? ( mirror://binhost/ae60f2940d )
   a? !b? !c? ( mirror://binhost/2976e1acc0 )
   a? !b?  c? ( mirror://binhost/f4809db70c )
   a?  b? !c? ( mirror://binhost/ecd08466cf )
   a?  b?  c? ( mirror://binhost/0c00f33b2e )
  installsources? (
!a? !b? !c? ( mirror://binhost/063a14d6c7 )
!a? !b?  c? ( mirror://binhost/f67c311625 )
!a?  b? !c? ( mirror://binhost/1dfff1f2ac )
!a?  b?  c? ( mirror://binhost/17a673e16a )
 a? !b? !c? ( mirror://binhost/914d1cecfe )
 a? !b?  c? ( mirror://binhost/ca18d86a2b )
 a?  b? !c? ( mirror://binhost/6bce13471a )
 a?  b?  c? ( mirror://binhost/3a6bdcd228 )
  )
  splitdebug? (
!a? !b? !c? ( mirror://binhost/29b2f38c41 )
!a? !b?  c? ( mirror://binhost/8adc9bef51 )
!a?  b? !c? ( mirror://binhost/954d2ce484 )
!a?  b?  c? ( mirror://binhost/32a614aaca )
 a? !b? !c? ( mirror://binhost/3548a2302d )
 a? !b?  c? ( mirror://binhost/e0c02cdc88 )
 a?  b? !c? ( mirror://binhost/f9cbd3c181 )
 a?  b?  c? ( mirror://binhost/31d4c03474 )
  )
"

For installsources, we can automate deduplication, so that we can
distribute the same file content for multiple USE combinations when
appropriate. If all of the combinations have identical content, then it
will look like this:

  installsources? (
!

[gentoo-portage-dev] Re: [PATCH] repoman: revert preserve_old_lib deprecation (bug 480244)

2021-02-24 Thread Zac Medico
On 2/23/21 3:41 PM, Zac Medico wrote:
> Repoman should not report that preserve_old_lib is deprecated,
> since preserve-libs is not covered by PMS.
> This reverts commit 49cbc17bf7b99be586e158c1bd588cfe91dfe58c.
> 
> Bug: https://bugs.gentoo.org/480244
> Bug: https://bugs.gentoo.org/692486
> Signed-off-by: Zac Medico 
> ---
>  repoman/cnf/linechecks/linechecks.yaml| 2 +-
>  .../lib/repoman/modules/linechecks/deprecated/deprecated.py   | 4 ++--
>  2 files changed, 3 insertions(+), 3 deletions(-)

Due to controversy, I would like to withdraw this patch, and call upon
interested parties to help decide an appropriate course of action or
inaction for https://bugs.gentoo.org/480244.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] repoman: revert preserve_old_lib deprecation (bug 480244)

2021-02-23 Thread Zac Medico
Repoman should not report that preserve_old_lib is deprecated,
since preserve-libs is not covered by PMS.
This reverts commit 49cbc17bf7b99be586e158c1bd588cfe91dfe58c.

Bug: https://bugs.gentoo.org/480244
Bug: https://bugs.gentoo.org/692486
Signed-off-by: Zac Medico 
---
 repoman/cnf/linechecks/linechecks.yaml| 2 +-
 .../lib/repoman/modules/linechecks/deprecated/deprecated.py   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/repoman/cnf/linechecks/linechecks.yaml 
b/repoman/cnf/linechecks/linechecks.yaml
index 2182b467a..7c040e2b2 100644
--- a/repoman/cnf/linechecks/linechecks.yaml
+++ b/repoman/cnf/linechecks/linechecks.yaml
@@ -25,7 +25,7 @@ errors:
 DEPRECATED_BINDNOW_FLAGS: 'Deprecated bindnow-flags call'
 EAPI_DEFINED_AFTER_INHERIT: 'EAPI defined after inherit'
 NO_AS_NEEDED: 'Upstream asneeded linking bug (no-as-needed)'
-PRESERVE_OLD_LIB: 'Ebuild calls deprecated preserve_old_lib'
+PRESERVE_OLD_LIB: 'Upstream ABI change workaround on line: %d'
 BUILT_WITH_USE: 'built_with_use'
 NO_OFFSET_WITH_HELPERS: 'Helper function is used with D, ROOT, ED, EROOT 
or EPREFIX'
 USEQ_ERROR: 'Ebuild calls deprecated useq function'
diff --git a/repoman/lib/repoman/modules/linechecks/deprecated/deprecated.py 
b/repoman/lib/repoman/modules/linechecks/deprecated/deprecated.py
index d1a590f1d..ad488c53a 100644
--- a/repoman/lib/repoman/modules/linechecks/deprecated/deprecated.py
+++ b/repoman/lib/repoman/modules/linechecks/deprecated/deprecated.py
@@ -19,8 +19,8 @@ class DeprecatedHasq(LineCheck):
 
 
 class PreserveOldLib(LineCheck):
-   """Check for calls to the deprecated preserve_old_lib function."""
-   repoman_check_name = 'ebuild.minorsyn'
+   """Check for calls to the preserve_old_lib function."""
+   repoman_check_name = 'upstream.workaround'
re = re.compile(r'.*preserve_old_lib')
error = 'PRESERVE_OLD_LIB'
 
-- 
2.26.2




Re: [gentoo-dev] New project: binhost

2021-02-23 Thread Zac Medico
On 2/23/21 12:05 PM, Zac Medico wrote:
> On 2/23/21 11:46 AM, Zac Medico wrote:
>> On 2/20/21 8:17 PM, Zac Medico wrote:
>>> IUSE_RUNTIME will obviously introduce conditionals in binary package
>>> dependencies, but we should welcome these conditionals because they will
>>> provide useful flexibility.
>>>
>>> I think IUSE_RUNTIME will be a very nice feature to have for project
>>> binhost, since it will allow binary package dependencies to behave with
>>> flexibility that more closely resembles the flexibility of ebuild
>>> dependencies.
>>
>> We can borrow paludis's notion of pbins [1] to generate ebuilds which
>> install pre-built content, and the generated ebuilds could have USE
>> flags that behave similarly to IUSE_RUNTIME in the sense that changes to
>> USE flags will result in different builds of pre-built content being
>> installed. A content-hash distfiles layout [2] could serve as a
>> convenient way to store separate builds of pre-built content for
>> multiple combinations of USE flags, and a generated ebuild would index
>> the build by USE flag combination.
>>
>> Also, for the generated ebuilds, we can generate USE flags to include
>> separate SRC_URI downloads for pre-built content to support things like
>> FEATURES=split-debug and FEATURES=install-sources.
> 
> Note that all of this can use existing EAPI features, since everything new
> would be implemented in an ebuild generator that generates a single
> ebuild to index pre-built content from multiple binary package builds.
> 
>> [1] https://paludis.exherbo.org/overview/pbins.html
>> [2] https://bugs.gentoo.org/756778
> 

For generated ebuilds, we'll have a choice to model things as USE flags
or sub-packages. For example, content from FEATURES=split-debug and
FEATURES=install-sources content might make more sense to model as
sub-packages than USE flags. It makes more sense to generate a
sub-package when there is a need for the sub-package to have USE flags.
For example, a split-debug sub-package can have USE flags which index
pre-built content from builds for multiple USE flag combinations.
Similar USE flags could be useful for an install-sources sub-package if
the source code has patches which are conditional on USE flags.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] New project: binhost

2021-02-23 Thread Zac Medico
On 2/23/21 11:46 AM, Zac Medico wrote:
> On 2/20/21 8:17 PM, Zac Medico wrote:
>> On 2/13/21 4:53 PM, Zac Medico wrote:
>>> On 2/13/21 4:37 PM, Zac Medico wrote:
>>>> On 2/11/21 1:17 AM, Michał Górny wrote:
>>>>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>>>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>>>>
>>>>>>> Hi all, 
>>>>>>>
>>>>>>> I'm announcing a new project here - "binhost"
>>>>>>>
>>>>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>>>>> precompiled packages for a subset of configurations, via central
>>>>>>> binary package hosting. Currently we are still in the conceptual
>>>>>>> planning stage. "
>>>>>>>
>>>>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>>>>
>>>>>>> If you're interested in helping out, feel free to add yourself on the
>>>>>>> wiki page.
>>>>>>>
>>>>>>> Note that I see actually *building* the packages not as the central
>>>>>>> point of the project (that could be e.g. a side effect of a
>>>>>>> tinderbox). I'm more concerned about
>>>>>>> * what configurations should we use
>>>>>>> * what portage features are still needed or need improvements (e.g.
>>>>>>> binpkg signing and verification)
>>>>>>> * how should hosting look like
>>>>>>> * and how we can test this on a limited scale before it goes "into
>>>>>>> production"
>>>>>>> * ...
>>>>>>>
>>>>>>> Comments, ideas, flamebaits? :D
>>>>>>>
>>>>>>> Cheers, 
>>>>>>> Andreas
>>>>>>>
>>>>>>
>>>>>> It would be great to improve portage speed with handling binpkgs. I
>>>>>> already have my own binhost for a couple of Gentoo systems and even
>>>>>> though these systems don't have to compile anything themselves,
>>>>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>>>>> installation time. Arch Linux' pacman only takes a fraction of this
>>>>>> time for the very same task.
>>>>>> I know that I compare apples with pears here but even reducing the
>>>>>> current portage time by 50% would be a huge improvement.
>>>>>
>>>>> Is that really a problem?  For me, Portage takes about an hour just to
>>>>> do the dependency processing these days.  In fact, building from sources
>>>>> is now faster than dependency calculations.
>>>>
>>>> The ratio of these times is dependent on the complexity of the
>>>> dependencies involved, and so is the answer to your question.
>>>
>>> Also, in the context of binary packages, dependencies calculations are
>>> generally simpler for binary packages because their USE conditionals and
>>> slot-operator := dependencies are frozen in a particular state. This
>>> dramatically reduces the search space involved in dependency calculation.
>>
>> IUSE_RUNTIME will obviously introduce conditionals in binary package
>> dependencies, but we should welcome these conditionals because they will
>> provide useful flexibility.
>>
>> I think IUSE_RUNTIME will be a very nice feature to have for project
>> binhost, since it will allow binary package dependencies to behave with
>> flexibility that more closely resembles the flexibility of ebuild
>> dependencies.
> 
> We can borrow paludis's notion of pbins [1] to generate ebuilds which
> install pre-built content, and the generated ebuilds could have USE
> flags that behave similarly to IUSE_RUNTIME in the sense that changes to
> USE flags will result in different builds of pre-built content being
> installed. A content-hash distfiles layout [2] could serve as a
> convenient way to store separate builds of pre-built content for
> multiple combinations of USE flags, and a generated ebuild would index
> the build by USE flag combination.
> 
> Also, for the generated ebuilds, we can generate USE flags to include
> separate SRC_URI downloads for pre-built content to support things like
> FEATURES=split-debug and FEATURES=install-sources.

Note that all of this can existing EAPI features, since everything new
would be implemented in an ebuild generator that generates a single
ebuild to index pre-built content from multiple binary package builds.

> [1] https://paludis.exherbo.org/overview/pbins.html
> [2] https://bugs.gentoo.org/756778

-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] New project: binhost

2021-02-23 Thread Zac Medico
On 2/13/21 5:51 PM, Zac Medico wrote:
> On 2/10/21 11:11 AM, Rich Freeman wrote:
>> On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel  
>> wrote:
>>>
>>> * what portage features are still needed or need improvements (e.g. binpkg
>>> signing and verification)
>>> * how should hosting look like
>>
>> Some ideas for portage enhancements:
>>
>> 1.  Ability to fetch binary packages from some kind of repo.
> 
> The old PORTAGE_BINHOST functionality has been replaced with a
> binrepos.conf file that's very similar to repos.conf:
> 
> https://bugs.gentoo.org/668334
> 
> It doesn't have explicit support for multiple local binary package
> repositories yet, but somebody got it working with src-uri set to a
> file:/ uri as described in comments on this bug:
> 
> https://bugs.gentoo.org/768957
> 
>> 2.  Ability to have multiple binary packages co-exist in a repo (local
>> or remote) with different build attributes (arch, USE, CFLAGS,
>> DEPENDS, whatever).
> 
> We can now enable FEATURES=binpkg-multi-instance by default now that
> this bug is fixed:
> 
> https://bugs.gentoo.org/571126
> 
>> 3.  Ability to pick the most appropriate binary packages to use based
>> on user preferences (with a mix of hard and soft preferences).
> 
> Current package selection logic for binary packages is basically the
> same as for ebuilds. These are the notable differences:
> 
> 1) Binary packages are sorted in descending order by (version, mtime),
> so then most recent builds are preferred when the versions are identical.
> 
> 2) The --binpkg-respect-use option rejects binary packages what would
> need to be rebuilt in order to match local USE settings.
> 
>> One idea I've had around how #2-3 might be implemented is:
>> 1.  Binary packages already contain data on how they were built (USE
>> flags, dependencies, etc).  Place this in a file using a deterministic
>> sorting/etc order so that two builds with the same settings will have
>> the same results.
> 
> This would only be needed to multi-profile binhosts that provide a
> variety of configurations for the same package.
> 
> Features like this are not necessary if the binhost only intends to
> provide packages for a single profile.
> 
>> 2.  Generate a hash of the file contents - this can go in the filename
>> so that the file can co-exist with other files, and be located
>> assuming you have a full matching set of metadata.
> 
> For FEATURES=binpkg-multi-instance we currently use an integer BUILD_ID
> ensure that file names are unique.
> 
>> 3.  Start dropping attributes from the file based on a list of
>> priorities and generate additional hashes.  Create symlinked files to
>> the original file using these hashes (overwriting or not existing
>> symlinks based on policy).  This allows the binary package to be found
>> using either an exact set of attributes or a subset of higher-priority
>> attributes.  This is analogous to shared object symlinking.
>> 4.  The package manager will look for a binary package first using the
>> user's full config, and then by dropping optional elements of the
>> config (so maybe it does the search without CFLAGs, then without USE
>> flags).  Eventually it aborts based on user prefs (maybe the user only
>> wants an exact match, or is willing to accept alternate CFLAGs but not
>> USE flags, or maybe anything for the arch is selected> 5.  As always the 
>> final selected binary package still gets evaluated
>> like any other binary package to ensure it is usable.
>>
>> Such a system can identify whether a potentially usable file exists
>> using only filename, cutting down on fetching.  In the interests of
>> avoiding useless fetches we would only carry step 3 reasonably far -
>> packages would have to match based on architecture and any dynamic
>> linking requirements.  So we wouldn't generate hashes that didn't
>> include at least those minimums, and the package manager wouldn't
>> search for them.
>>
>> Obviously you could do more (if you have 5 combinations of use flags,
>> look for the set that matches most closely).  That couldn't be done
>> using hashes alone in an efficient way.  You could have a small
>> manifest file alongside the binary package that could be fetched
>> separately if the package manager wants to narrow things down and
>> fetch a few of those to narrow it down further.
> 
> All of the above is oriented toward multi-profile binhosts, so we'll
> have to do a cost/benefit analysis to determine whether it's worth the
> effort to introduce the complexity that multi-profile binhosts add.

This idea to borrow paludis's notion of pbins could work well for a
multi-profile binhost:

https://archives.gentoo.org/gentoo-dev/message/1aea0de0ff588c67bf652f4c1f8ef304
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] New project: binhost

2021-02-23 Thread Zac Medico
On 2/20/21 8:17 PM, Zac Medico wrote:
> On 2/13/21 4:53 PM, Zac Medico wrote:
>> On 2/13/21 4:37 PM, Zac Medico wrote:
>>> On 2/11/21 1:17 AM, Michał Górny wrote:
>>>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>>>
>>>>>> Hi all, 
>>>>>>
>>>>>> I'm announcing a new project here - "binhost"
>>>>>>
>>>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>>>> precompiled packages for a subset of configurations, via central
>>>>>> binary package hosting. Currently we are still in the conceptual
>>>>>> planning stage. "
>>>>>>
>>>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>>>
>>>>>> If you're interested in helping out, feel free to add yourself on the
>>>>>> wiki page.
>>>>>>
>>>>>> Note that I see actually *building* the packages not as the central
>>>>>> point of the project (that could be e.g. a side effect of a
>>>>>> tinderbox). I'm more concerned about
>>>>>> * what configurations should we use
>>>>>> * what portage features are still needed or need improvements (e.g.
>>>>>> binpkg signing and verification)
>>>>>> * how should hosting look like
>>>>>> * and how we can test this on a limited scale before it goes "into
>>>>>> production"
>>>>>> * ...
>>>>>>
>>>>>> Comments, ideas, flamebaits? :D
>>>>>>
>>>>>> Cheers, 
>>>>>> Andreas
>>>>>>
>>>>>
>>>>> It would be great to improve portage speed with handling binpkgs. I
>>>>> already have my own binhost for a couple of Gentoo systems and even
>>>>> though these systems don't have to compile anything themselves,
>>>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>>>> installation time. Arch Linux' pacman only takes a fraction of this
>>>>> time for the very same task.
>>>>> I know that I compare apples with pears here but even reducing the
>>>>> current portage time by 50% would be a huge improvement.
>>>>
>>>> Is that really a problem?  For me, Portage takes about an hour just to
>>>> do the dependency processing these days.  In fact, building from sources
>>>> is now faster than dependency calculations.
>>>
>>> The ratio of these times is dependent on the complexity of the
>>> dependencies involved, and so is the answer to your question.
>>
>> Also, in the context of binary packages, dependencies calculations are
>> generally simpler for binary packages because their USE conditionals and
>> slot-operator := dependencies are frozen in a particular state. This
>> dramatically reduces the search space involved in dependency calculation.
> 
> IUSE_RUNTIME will obviously introduce conditionals in binary package
> dependencies, but we should welcome these conditionals because they will
> provide useful flexibility.
> 
> I think IUSE_RUNTIME will be a very nice feature to have for project
> binhost, since it will allow binary package dependencies to behave with
> flexibility that more closely resembles the flexibility of ebuild
> dependencies.

We can borrow paludis's notion of pbins [1] to generate ebuilds which
install pre-built content, and the generated ebuilds could have USE
flags that behave similarly to IUSE_RUNTIME in the sense that changes to
USE flags will result in different builds of pre-built content being
installed. A content-hash distfiles layout [2] could serve as a
convenient way to store separate builds of pre-built content for
multiple combinations of USE flags, and a generated ebuild would index
the build by USE flag combination.

Also, for the generated ebuilds, we can generate USE flags to include
separate SRC_URI downloads for pre-built content to support things like
FEATURES=split-debug and FEATURES=install-sources.

[1] https://paludis.exherbo.org/overview/pbins.html
[2] https://bugs.gentoo.org/756778
-- 
Thanks,
Zac





signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] Add content-hash distfiles layout (bug 756778)

2021-02-21 Thread Zac Medico
From: Daniel Robbins 

The content-hash layout is identical to the filename-hash layout,
except for these two differences:

1) A content digest is used instead of a filename digest.
2) The final element of the path returned from the get_path method
   corresponds to the complete content digest. The path is a
   function of the content digest alone.

Motivations to use the content-hash layout instead of the
filename-hash layout may include:

1) Since the file path is independent of the file name, file
name collisions cannot occur. This makes the content-hash
layout suitable for storage of multiple types of files (not
only gentoo distfiles). For example, it can be used to store
distfiles for multiple linux distros within the same tree,
with automatic deduplication based on content digest. This
layout can be used to store and distribute practically anything
(including binary packages for example).

2) Allows multiple revisions for the same distfiles name. An
existing distfile can be updated, and if a user still has an
older copy of an ebuild repository (or an overlay), then a user
can successfully fetch a desired revision of the distfile as
long as it has not been purged from the mirror.

3) File integrity data is integrated into the layout itself,
making it very simple to verify the integrity of any file that
it contains. The only tool required is an implementation of
the chosen hash algorithm.

Bug: https://bugs.gentoo.org/756778
Signed-off-by: Zac Medico 
---
 lib/portage/package/ebuild/fetch.py| 160 +++--
 lib/portage/tests/ebuild/test_fetch.py |  40 ++-
 2 files changed, 184 insertions(+), 16 deletions(-)

diff --git a/lib/portage/package/ebuild/fetch.py 
b/lib/portage/package/ebuild/fetch.py
index e0fecaf23..7d2ef93bf 100644
--- a/lib/portage/package/ebuild/fetch.py
+++ b/lib/portage/package/ebuild/fetch.py
@@ -344,6 +344,31 @@ _size_suffix_map = {
 }
 
 
+class DistfileName(str):
+   def __new__(cls, s, digests=None):
+   return str.__new__(cls, s)
+
+   def __init__(self, s, digests=None):
+   super().__init__()
+   self.digests = {} if digests is None else digests
+
+   def digests_equal(self, other):
+   """
+   Test if digests compare equal to those of another instance.
+   """
+   if not isinstance(other, DistfileName):
+   return False
+   matches = []
+   for algo, digest in self.digests.items():
+   other_digest = other.digests.get(algo)
+   if other_digest is not None:
+   if other_digest == digest:
+   matches.append(algo)
+   else:
+   return False
+   return bool(matches)
+
+
 class FlatLayout:
def get_path(self, filename):
return filename
@@ -413,6 +438,90 @@ class FilenameHashLayout:
return False
 
 
+class ContentHashLayout(FilenameHashLayout):
+   """
+   The content-hash layout is identical to the filename-hash layout,
+   except for these two differences:
+
+   1) A content digest is used instead of a filename digest.
+   2) The final element of the path returned from the get_path method
+  corresponds to the complete content digest. The path is a
+  function of the content digest alone.
+
+   Motivations to use the content-hash layout instead of the
+   filename-hash layout may include:
+
+   1) Since the file path is independent of the file name, file
+   name collisions cannot occur. This makes the content-hash
+   layout suitable for storage of multiple types of files (not
+   only gentoo distfiles). For example, it can be used to store
+   distfiles for multiple linux distros within the same tree,
+   with automatic deduplication based on content digest. This
+   layout can be used to store and distribute practically anything
+   (including binary packages for example).
+
+   2) Allows multiple revisions for the same distfiles name. An
+   existing distfile can be updated, and if a user still has an
+   older copy of an ebuild repository (or an overlay), then a user
+   can successfully fetch a desired revision of the distfile as
+   long as it has not been purged from the mirror.
+
+   3) File integrity data is integrated into the layout itself,
+   making it very simple to verify the integrity of any file that
+   it contains. The only tool required is an implementation of
+   the chosen hash algorithm.
+   """
+
+   def get_path(self, filename):
+   """
+   For content-hash, the path is a function of the content digest 
alone.
+   The final element of the path re

Re: [gentoo-dev] New project: binhost

2021-02-20 Thread Zac Medico
On 2/13/21 4:53 PM, Zac Medico wrote:
> On 2/13/21 4:37 PM, Zac Medico wrote:
>> On 2/11/21 1:17 AM, Michał Górny wrote:
>>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>>
>>>>> Hi all, 
>>>>>
>>>>> I'm announcing a new project here - "binhost"
>>>>>
>>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>>> precompiled packages for a subset of configurations, via central
>>>>> binary package hosting. Currently we are still in the conceptual
>>>>> planning stage. "
>>>>>
>>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>>
>>>>> If you're interested in helping out, feel free to add yourself on the
>>>>> wiki page.
>>>>>
>>>>> Note that I see actually *building* the packages not as the central
>>>>> point of the project (that could be e.g. a side effect of a
>>>>> tinderbox). I'm more concerned about
>>>>> * what configurations should we use
>>>>> * what portage features are still needed or need improvements (e.g.
>>>>> binpkg signing and verification)
>>>>> * how should hosting look like
>>>>> * and how we can test this on a limited scale before it goes "into
>>>>> production"
>>>>> * ...
>>>>>
>>>>> Comments, ideas, flamebaits? :D
>>>>>
>>>>> Cheers, 
>>>>> Andreas
>>>>>
>>>>
>>>> It would be great to improve portage speed with handling binpkgs. I
>>>> already have my own binhost for a couple of Gentoo systems and even
>>>> though these systems don't have to compile anything themselves,
>>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>>> installation time. Arch Linux' pacman only takes a fraction of this
>>>> time for the very same task.
>>>> I know that I compare apples with pears here but even reducing the
>>>> current portage time by 50% would be a huge improvement.
>>>
>>> Is that really a problem?  For me, Portage takes about an hour just to
>>> do the dependency processing these days.  In fact, building from sources
>>> is now faster than dependency calculations.
>>
>> The ratio of these times is dependent on the complexity of the
>> dependencies involved, and so is the answer to your question.
> 
> Also, in the context of binary packages, dependencies calculations are
> generally simpler for binary packages because their USE conditionals and
> slot-operator := dependencies are frozen in a particular state. This
> dramatically reduces the search space involved in dependency calculation.

IUSE_RUNTIME will obviously introduce conditionals in binary package
dependencies, but we should welcome these conditionals because they will
provide useful flexibility.

I think IUSE_RUNTIME will be a very nice feature to have for project
binhost, since it will allow binary package dependencies to behave with
flexibility that more closely resembles the flexibility of ebuild
dependencies.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] make.defaults: prevent USE="${USE} ..." misbehavior

2021-02-18 Thread Zac Medico
Discard parent profile USE from the expand_map variable before
it is used to evaluate a child profile. This prevents accidents
triggered by USE="${USE} ..." settngs at the top of make.defaults
which caused parent profile USE to override parent profile
package.use settings.

Bug: https://bugs.gentoo.org/771549
Signed-off-by: Zac Medico 
---
 lib/portage/package/ebuild/config.py | 13 ++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/lib/portage/package/ebuild/config.py 
b/lib/portage/package/ebuild/config.py
index e5ec681af..638c72959 100644
--- a/lib/portage/package/ebuild/config.py
+++ b/lib/portage/package/ebuild/config.py
@@ -612,9 +612,16 @@ class config:
 
mygcfg = {}
if profiles_complex:
-   mygcfg_dlists = 
[getconfig(os.path.join(x.location, "make.defaults"),
-   tolerant=tolerant, expand=expand_map, 
recursive=x.portage1_directories)
-   for x in profiles_complex]
+   mygcfg_dlists = []
+   for x in profiles_complex:
+   # Prevent accidents triggered by 
USE="${USE} ..." settings
+   # at the top of make.defaults which 
caused parent profile
+   # USE to override parent profile 
package.use settings.
+   expand_map.pop("USE", None)
+   mygcfg_dlists.append(
+   
getconfig(os.path.join(x.location, "make.defaults"),
+   tolerant=tolerant, 
expand=expand_map,
+   
recursive=x.portage1_directories))
self._make_defaults = mygcfg_dlists
mygcfg = stack_dicts(mygcfg_dlists,
incrementals=self.incrementals)
-- 
2.26.2




Re: [gentoo-dev] New project: binhost

2021-02-13 Thread Zac Medico
On 2/10/21 11:11 AM, Rich Freeman wrote:
> On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel  
> wrote:
>>
>> * what portage features are still needed or need improvements (e.g. binpkg
>> signing and verification)
>> * how should hosting look like
> 
> Some ideas for portage enhancements:
> 
> 1.  Ability to fetch binary packages from some kind of repo.

The old PORTAGE_BINHOST functionality has been replaced with a
binrepos.conf file that's very similar to repos.conf:

https://bugs.gentoo.org/668334

It doesn't have explicit support for multiple local binary package
repositories yet, but somebody got it working with src-uri set to a
file:/ uri as described in comments on this bug:

https://bugs.gentoo.org/768957

> 2.  Ability to have multiple binary packages co-exist in a repo (local
> or remote) with different build attributes (arch, USE, CFLAGS,
> DEPENDS, whatever).

We can now enable FEATURES=binpkg-multi-instance by default now that
this bug is fixed:

https://bugs.gentoo.org/571126

> 3.  Ability to pick the most appropriate binary packages to use based
> on user preferences (with a mix of hard and soft preferences).

Current package selection logic for binary packages is basically the
same as for ebuilds. These are the notable differences:

1) Binary packages are sorted in descending order by (version, mtime),
so then most recent builds are preferred when the versions are identical.

2) The --binpkg-respect-use option rejects binary packages what would
need to be rebuilt in order to match local USE settings.

> One idea I've had around how #2-3 might be implemented is:
> 1.  Binary packages already contain data on how they were built (USE
> flags, dependencies, etc).  Place this in a file using a deterministic
> sorting/etc order so that two builds with the same settings will have
> the same results.

This would only be needed to multi-profile binhosts that provide a
variety of configurations for the same package.

Features like this are not necessary if the binhost only intends to
provide packages for a single profile.

> 2.  Generate a hash of the file contents - this can go in the filename
> so that the file can co-exist with other files, and be located
> assuming you have a full matching set of metadata.

For FEATURES=binpkg-multi-instance we currently use an integer BUILD_ID
ensure that file names are unique.

> 3.  Start dropping attributes from the file based on a list of
> priorities and generate additional hashes.  Create symlinked files to
> the original file using these hashes (overwriting or not existing
> symlinks based on policy).  This allows the binary package to be found
> using either an exact set of attributes or a subset of higher-priority
> attributes.  This is analogous to shared object symlinking.
> 4.  The package manager will look for a binary package first using the
> user's full config, and then by dropping optional elements of the
> config (so maybe it does the search without CFLAGs, then without USE
> flags).  Eventually it aborts based on user prefs (maybe the user only
> wants an exact match, or is willing to accept alternate CFLAGs but not
> USE flags, or maybe anything for the arch is selected> 5.  As always the 
> final selected binary package still gets evaluated
> like any other binary package to ensure it is usable.
> 
> Such a system can identify whether a potentially usable file exists
> using only filename, cutting down on fetching.  In the interests of
> avoiding useless fetches we would only carry step 3 reasonably far -
> packages would have to match based on architecture and any dynamic
> linking requirements.  So we wouldn't generate hashes that didn't
> include at least those minimums, and the package manager wouldn't
> search for them.
> 
> Obviously you could do more (if you have 5 combinations of use flags,
> look for the set that matches most closely).  That couldn't be done
> using hashes alone in an efficient way.  You could have a small
> manifest file alongside the binary package that could be fetched
> separately if the package manager wants to narrow things down and
> fetch a few of those to narrow it down further.

All of the above is oriented toward multi-profile binhosts, so we'll
have to do a cost/benefit analysis to determine whether it's worth the
effort to introduce the complexity that multi-profile binhosts add.

> Or you could skip the hash searching and just fetch all the manifests
> for a particular package/arch and just search all of those, but that
> is more data to transfer just to do a query.  A metadata cache of some
> kind of might be another solution.  Content hashes would probably
> still be useful just to allow co-existence of alternate builds.

This also relates to the centralized Packages file that's currently used
to distribute the package metadata for all packages in a binhost. We can
make it scale better if we split out a separate index per package, not
unlike a pypi simple index:

https://pypi.org/simple/
-- 

Re: [gentoo-dev] New project: binhost

2021-02-13 Thread Zac Medico
On 2/13/21 4:37 PM, Zac Medico wrote:
> On 2/11/21 1:17 AM, Michał Górny wrote:
>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>
>>>> Hi all, 
>>>>
>>>> I'm announcing a new project here - "binhost"
>>>>
>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>> precompiled packages for a subset of configurations, via central
>>>> binary package hosting. Currently we are still in the conceptual
>>>> planning stage. "
>>>>
>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>
>>>> If you're interested in helping out, feel free to add yourself on the
>>>> wiki page.
>>>>
>>>> Note that I see actually *building* the packages not as the central
>>>> point of the project (that could be e.g. a side effect of a
>>>> tinderbox). I'm more concerned about
>>>> * what configurations should we use
>>>> * what portage features are still needed or need improvements (e.g.
>>>> binpkg signing and verification)
>>>> * how should hosting look like
>>>> * and how we can test this on a limited scale before it goes "into
>>>> production"
>>>> * ...
>>>>
>>>> Comments, ideas, flamebaits? :D
>>>>
>>>> Cheers, 
>>>> Andreas
>>>>
>>>
>>> It would be great to improve portage speed with handling binpkgs. I
>>> already have my own binhost for a couple of Gentoo systems and even
>>> though these systems don't have to compile anything themselves,
>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>> installation time. Arch Linux' pacman only takes a fraction of this
>>> time for the very same task.
>>> I know that I compare apples with pears here but even reducing the
>>> current portage time by 50% would be a huge improvement.
>>
>> Is that really a problem?  For me, Portage takes about an hour just to
>> do the dependency processing these days.  In fact, building from sources
>> is now faster than dependency calculations.
> 
> The ratio of these times is dependent on the complexity of the
> dependencies involved, and so is the answer to your question.

Also, in the context of binary packages, dependencies calculations are
generally simpler for binary packages because their USE conditionals and
slot-operator := dependencies are frozen in a particular state. This
dramatically reduces the search space involved in dependency calculation.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] New project: binhost

2021-02-13 Thread Zac Medico
On 2/11/21 1:17 AM, Michał Górny wrote:
> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>
>>> Hi all, 
>>>
>>> I'm announcing a new project here - "binhost"
>>>
>>> "The Gentoo Binhost project aims to provide readily installable,
>>> precompiled packages for a subset of configurations, via central
>>> binary package hosting. Currently we are still in the conceptual
>>> planning stage. "
>>>
>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>
>>> If you're interested in helping out, feel free to add yourself on the
>>> wiki page.
>>>
>>> Note that I see actually *building* the packages not as the central
>>> point of the project (that could be e.g. a side effect of a
>>> tinderbox). I'm more concerned about
>>> * what configurations should we use
>>> * what portage features are still needed or need improvements (e.g.
>>> binpkg signing and verification)
>>> * how should hosting look like
>>> * and how we can test this on a limited scale before it goes "into
>>> production"
>>> * ...
>>>
>>> Comments, ideas, flamebaits? :D
>>>
>>> Cheers, 
>>> Andreas
>>>
>>
>> It would be great to improve portage speed with handling binpkgs. I
>> already have my own binhost for a couple of Gentoo systems and even
>> though these systems don't have to compile anything themselves,
>> installing ~100 to ~200 binpkgs takes way more than an hour of
>> installation time. Arch Linux' pacman only takes a fraction of this
>> time for the very same task.
>> I know that I compare apples with pears here but even reducing the
>> current portage time by 50% would be a huge improvement.
> 
> Is that really a problem?  For me, Portage takes about an hour just to
> do the dependency processing these days.  In fact, building from sources
> is now faster than dependency calculations.

The ratio of these times is dependent on the complexity of the
dependencies involved, and so is the answer to your question.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] New project: binhost

2021-02-13 Thread Zac Medico
On 2/10/21 10:51 AM, Lars Wendler wrote:
> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
> 
>> Hi all, 
>>
>> I'm announcing a new project here - "binhost"
>>
>> "The Gentoo Binhost project aims to provide readily installable,
>> precompiled packages for a subset of configurations, via central
>> binary package hosting. Currently we are still in the conceptual
>> planning stage. "
>>
>> https://wiki.gentoo.org/wiki/Project:Binhost
>>
>> If you're interested in helping out, feel free to add yourself on the
>> wiki page.
>>
>> Note that I see actually *building* the packages not as the central
>> point of the project (that could be e.g. a side effect of a
>> tinderbox). I'm more concerned about
>> * what configurations should we use
>> * what portage features are still needed or need improvements (e.g.
>> binpkg signing and verification)
>> * how should hosting look like
>> * and how we can test this on a limited scale before it goes "into
>> production"
>> * ...
>>
>> Comments, ideas, flamebaits? :D
>>
>> Cheers, 
>> Andreas
>>
> 
> It would be great to improve portage speed with handling binpkgs. I
> already have my own binhost for a couple of Gentoo systems and even
> though these systems don't have to compile anything themselves,
> installing ~100 to ~200 binpkgs takes way more than an hour of
> installation time. Arch Linux' pacman only takes a fraction of this
> time for the very same task.
> I know that I compare apples with pears here but even reducing the
> current portage time by 50% would be a huge improvement.

In order to maximize throughput, we have FEATURES="parallel-install"
that for example allows one package's files to be merged while a
pkg_{pre,post}{inst,rm} phase is running for an unrelated package that
is in the merge queue.

There's also FEATURES="-ebuild-locks" which allows privileged
pkg_{pre,post}{inst,rm} phases to run concurrently with privileged
phases of unrelated packages in the merge queue.

Ultimately, pkg_{pre,post}{inst,rm} phases could be the limiting factor
here. I portage, we should eliminate calls to these phases when
DEFINED_PHASES metadata indicated that they're not defined.

Also, FEATURES=preserve-libs introduces considerable overhead because it
regenerates LinkageMapELF data structures for every merge. It would be
much more efficient if it could incrementally update these data structures.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] portage.getpid: call os.getpid() lazily

2021-01-30 Thread Zac Medico
Call os.getpid() lazily, which eliminates getpid calls when possible
after os.fork() in the portage.process module.

Bug: https://bugs.gentoo.org/767913
Signed-off-by: Zac Medico 
---
 lib/portage/__init__.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py
index 3c9f78497..24c9d8b89 100644
--- a/lib/portage/__init__.py
+++ b/lib/portage/__init__.py
@@ -375,7 +375,7 @@ _sync_mode = False
 class _ForkWatcher:
@staticmethod
def hook(_ForkWatcher):
-   _ForkWatcher.current_pid = _os.getpid()
+   _ForkWatcher.current_pid = None
# Force instantiation of a new event loop policy as a workaround
# for https://bugs.python.org/issue22087.
asyncio.set_event_loop_policy(None)
@@ -388,6 +388,8 @@ def getpid():
"""
Cached version of os.getpid(). ForkProcess updates the cache.
"""
+   if _ForkWatcher.current_pid is None:
+   _ForkWatcher.current_pid = _os.getpid()
return _ForkWatcher.current_pid
 
 def _get_stdin():
-- 
2.26.2




[gentoo-portage-dev] [PATCH] emerge: disable --autounmask-license by default

2021-01-30 Thread Zac Medico
Disable --autounmask-license by default, in order to limit user
exposure to risks associated with package.license changes.
The changes that this option suggests are only intended to be
accepted when a user has made a conscious decision to accept
the corresponding license(s). Creation of package.license
changes introduces a risk that users may erroneously accept the
changes due to some kind of accident or misunderstanding,
rather than due to conscious decisions about licenses.
These risks provide motivation to disable --autounmask-license
by default. The --autounmask-use option will remain as the
only autounmask option that is still enabled by default.

The unit tests demonstrate interactions between --autounmask
and --autounmask-license options. The --autounmask option
enables --autounmask-license unless --autounmask-license=n
has been specified. If --autounmask=n is used to disable
autounmask, then --autounmask-license=y has no effect.

Bug: https://bugs.gentoo.org/766773
Signed-off-by: Zac Medico 
---
 lib/_emerge/create_depgraph_params.py |  8 +++---
 lib/portage/tests/resolver/test_autounmask.py | 25 +--
 man/emerge.1  | 11 +++-
 3 files changed, 31 insertions(+), 13 deletions(-)

diff --git a/lib/_emerge/create_depgraph_params.py 
b/lib/_emerge/create_depgraph_params.py
index 0d0e07b9c..25dd2a1b4 100644
--- a/lib/_emerge/create_depgraph_params.py
+++ b/lib/_emerge/create_depgraph_params.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import logging
@@ -45,7 +45,7 @@ def create_depgraph_params(myopts, myaction):
autounmask_keep_masks = myopts.get("--autounmask-keep-masks")
 
autounmask = myopts.get("--autounmask")
-   autounmask_license = myopts.get('--autounmask-license')
+   autounmask_license = myopts.get('--autounmask-license', 'y' if 
autounmask is True else 'n')
autounmask_use = myopts.get('--autounmask-use')
if autounmask == 'n':
autounmask = False
@@ -53,7 +53,7 @@ def create_depgraph_params(myopts, myaction):
if autounmask is None:
if autounmask_use in (None, 'y'):
autounmask = True
-   elif autounmask_license in (None, 'y'):
+   if autounmask_license in ('y',):
autounmask = True
 
# Do not enable package.accept_keywords or package.mask
@@ -67,7 +67,7 @@ def create_depgraph_params(myopts, myaction):
 
myparams['autounmask'] = autounmask
myparams['autounmask_keep_use'] = True if autounmask_use == 'n' else 
False
-   myparams['autounmask_keep_license'] = True if autounmask_license == 'n' 
else False
+   myparams['autounmask_keep_license'] = False if autounmask_license == 
'y' else True
myparams['autounmask_keep_keywords'] = False if 
autounmask_keep_keywords in (None, 'n') else True
myparams['autounmask_keep_masks'] = False if autounmask_keep_masks in 
(None, 'n') else True
 
diff --git a/lib/portage/tests/resolver/test_autounmask.py 
b/lib/portage/tests/resolver/test_autounmask.py
index a3bf0ff94..86ae4bbf6 100644
--- a/lib/portage/tests/resolver/test_autounmask.py
+++ b/lib/portage/tests/resolver/test_autounmask.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2019 Gentoo Authors
+# Copyright 2010-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from portage.tests import TestCase
@@ -440,13 +440,34 @@ class AutounmaskTestCase(TestCase):
mergelist=["dev-libs/A-1"],
license_changes={ "dev-libs/A-1": 
set(["TEST"]) }),
 
-   # Test default --autounmask-license
+   # Test that --autounmask enables 
--autounmask-license
ResolverPlaygroundTestCase(
["=dev-libs/A-1"],
+   options={"--autounmask": True},
success=False,
mergelist=["dev-libs/A-1"],
license_changes={ "dev-libs/A-1": 
set(["TEST"]) }),
 
+   # Test that --autounmask-license is not enabled 
by default
+   ResolverPlaygroundTestCase(
+   ["=dev-libs/A-1"],
+   success=False,
+   ),
+
+   # Test that --autounmask does not override 
--autounmask-license=n
+   ResolverPlaygroundTestCase(
+  

[gentoo-portage-dev] [PATCH] emaint --fix merges: add -y, --yes option

2021-01-23 Thread Zac Medico
Since the emaint --fix merges uses emerge --ask, add
a -y, --yes option to use --ask=n instead.

Bug: https://bugs.gentoo.org/766767
Signed-off-by: Zac Medico 
---
 lib/portage/emaint/modules/merges/__init__.py | 14 --
 lib/portage/emaint/modules/merges/merges.py   | 11 +++
 man/emaint.1  |  6 +-
 3 files changed, 24 insertions(+), 7 deletions(-)

diff --git a/lib/portage/emaint/modules/merges/__init__.py 
b/lib/portage/emaint/modules/merges/__init__.py
index 89aa758a0..449f39dce 100644
--- a/lib/portage/emaint/modules/merges/__init__.py
+++ b/lib/portage/emaint/modules/merges/__init__.py
@@ -1,4 +1,4 @@
-# Copyright 2005-2014 Gentoo Foundation
+# Copyright 2005-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 doc = """Scan for failed merges and fix them."""
@@ -26,7 +26,17 @@ module_spec = {
'action': 'store_true',
'func': 'purge'
}
-   }
+   },
+   'opt_desc': {
+   'yes': {
+   "short": "-y",
+   "long": "--yes",
+   "help": ("(merges submodule only): Do 
not prompt for "
+   "emerge invocations"),
+   "action": "store_true",
+   "dest": "yes",
+   }
+   },
}
}
 }
diff --git a/lib/portage/emaint/modules/merges/merges.py 
b/lib/portage/emaint/modules/merges/merges.py
index 775dc59d2..d60916f1e 100644
--- a/lib/portage/emaint/modules/merges/merges.py
+++ b/lib/portage/emaint/modules/merges/merges.py
@@ -1,4 +1,4 @@
-# Copyright 2005-2020 Gentoo Authors
+# Copyright 2005-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import portage
@@ -186,7 +186,7 @@ class MergesHandler:
pkg_atoms.add(pkg_atom)
 
 
-   def _emerge_pkg_atoms(self, module_output, pkg_atoms):
+   def _emerge_pkg_atoms(self, module_output, pkg_atoms, yes=False):
"""
Emerge the specified packages atoms.
 
@@ -194,6 +194,8 @@ class MergesHandler:
@type module_output: Class
@param pkg_atoms: packages atoms to emerge
@type pkg_atoms: set
+   @param yes: do not prompt for emerge invocations
+   @type yes: bool
@rtype: list
@return: List of results
"""
@@ -206,7 +208,7 @@ class MergesHandler:
portage._python_interpreter,
'-b',
os.path.join(EPREFIX or '/', 'usr', 'bin', 'emerge'),
-   '--ask',
+   '--ask=n' if yes else '--ask',
'--quiet',
'--oneshot',
'--complete-graph=y'
@@ -265,7 +267,8 @@ class MergesHandler:
errors.append(', '.join(sorted(failed_pkgs)))
return (False, errors)
self._remove_failed_dirs(failed_pkgs)
-   results = self._emerge_pkg_atoms(module_output, pkg_atoms)
+   results = self._emerge_pkg_atoms(module_output, pkg_atoms,
+   yes=kwargs.get('options', {}).get("yes", False))
# list any new failed merges
for pkg in sorted(self._scan()):
results.append("'%s' still found as a failed merge." % 
pkg)
diff --git a/man/emaint.1 b/man/emaint.1
index d244756e9..c9f8ab939 100644
--- a/man/emaint.1
+++ b/man/emaint.1
@@ -1,4 +1,4 @@
-.TH "EMAINT" "1" "Jan 2017" "Portage VERSION" "Portage"
+.TH "EMAINT" "1" "Jan 2021" "Portage VERSION" "Portage"
 .SH NAME
 emaint \- performs package management related system health checks and 
maintenance
 .SH SYNOPSIS
@@ -82,6 +82,10 @@ OPTION
 .BR \-t \ \fINUM\fR,\  \-\-time \ \fINUM\fR
 Changes the minimum age \fINUM\fR (in days) of the logs to be listed or
 deleted.
+.SH OPTIONS merges command only
+.TP
+.BR \-y ", " \-\-yes
+Do not prompt for emerge invocations.
 .SH OPTIONS sync command only
 .TP
 .BR \-a ", " \-\-auto
-- 
2.26.2




[gentoo-portage-dev] [PATCH] binarytree.move_ent: copy on write for package move

2021-01-19 Thread Zac Medico
Copy on write when applying package moves, and silently
skip package moves when the same move has already been
applied to the same build of the package. Since the old
package instance is preserved, it avoids the problem
of having enries for deleted packages remain in the
package index. We can simply assume that the package
will be deleted by eclean-pkg when its time comes.

Bug: https://bugs.gentoo.org/766012
Signed-off-by: Zac Medico 
---
 lib/portage/dbapi/bintree.py  | 40 ---
 lib/portage/emaint/modules/move/move.py   | 13 ++--
 lib/portage/tests/update/test_move_ent.py |  7 ++--
 3 files changed, 44 insertions(+), 16 deletions(-)

diff --git a/lib/portage/dbapi/bintree.py b/lib/portage/dbapi/bintree.py
index 180e48c3b..76fca5523 100644
--- a/lib/portage/dbapi/bintree.py
+++ b/lib/portage/dbapi/bintree.py
@@ -31,6 +31,7 @@ from portage.exception import AlarmSignal, 
InvalidPackageName, \
ParseError, PortageException
 from portage.localization import _
 from portage.package.ebuild.profile_iuse import iter_iuse_vars
+from portage.util.file_copy import copyfile
 from portage.util.futures import asyncio
 from portage.util.futures.compat_coroutine import coroutine
 from portage.util.futures.executor.fork import ForkExecutor
@@ -483,6 +484,17 @@ class binarytree:
myoldpkg = catsplit(mycpv)[1]
mynewpkg = catsplit(mynewcpv)[1]
 
+   # If this update has already been applied to the same
+   # package build then silently continue.
+   applied = False
+   for maybe_applied in 
self.dbapi.match('={}'.format(mynewcpv)):
+   if maybe_applied.build_time == mycpv.build_time:
+   applied = True
+   break
+
+   if applied:
+   continue
+
if (mynewpkg != myoldpkg) and 
self.dbapi.cpv_exists(mynewcpv):
writemsg(_("!!! Cannot update binary: 
Destination exists.\n"),
noiselevel=-1)
@@ -513,24 +525,30 @@ class binarytree:
mydata[_unicode_encode(mynewpkg + 
'.ebuild',

encoding=_encodings['repo.content'])] = ebuild_data
 
-   mytbz2.recompose_mem(portage.xpak.xpak_mem(mydata))
-
-   self.dbapi.cpv_remove(mycpv)
-   del self._pkg_paths[self.dbapi._instance_key(mycpv)]
metadata = self.dbapi._aux_cache_slot_dict()
for k in self.dbapi._aux_cache_keys:
v = mydata.get(_unicode_encode(k))
if v is not None:
v = _unicode_decode(v)
metadata[k] = " ".join(v.split())
+
+   # Create a copy of the old version of the package and
+   # apply the update to it. Leave behind the old version,
+   # assuming that it will be deleted by eclean-pkg when 
its
+   # time comes.
mynewcpv = _pkg_str(mynewcpv, metadata=metadata, 
db=self.dbapi)
-   new_path = self.getname(mynewcpv)
-   self._pkg_paths[
-   self.dbapi._instance_key(mynewcpv)] = 
new_path[len(self.pkgdir)+1:]
-   if new_path != tbz2path:
-   self._ensure_dir(os.path.dirname(new_path))
-   _movefile(tbz2path, new_path, 
mysettings=self.settings)
-   self.inject(mynewcpv)
+   update_path = self.getname(mynewcpv, allocate_new=True) 
+ ".partial"
+   self._ensure_dir(os.path.dirname(update_path))
+   update_path_lock = None
+   try:
+   update_path_lock = lockfile(update_path, 
wantnewlockfile=True)
+   copyfile(tbz2path, update_path)
+   mytbz2 = portage.xpak.tbz2(update_path)
+   
mytbz2.recompose_mem(portage.xpak.xpak_mem(mydata))
+   self.inject(mynewcpv, filename=update_path)
+   finally:
+   if update_path_lock is not None:
+   unlockfile(update_path_lock)
 
return moves
 
diff --git a/lib/portage/emaint/modules/move/move.py 
b/lib/portage/emaint/modules/move/move.py
index 8fc3269ca..2a95e99c4 100644
--- a/lib/portage/emaint/modules/move/move.py
+++ b/lib/portage/emaint/modules/move/move.py
@@ -1,4 +1,4 @@
-# Copyright 2005-2020 Gentoo 

Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set

2021-01-18 Thread Zac Medico
On 1/18/21 8:42 PM, Alec Warner wrote:
> On Mon, Jan 18, 2021 at 8:09 PM Zac Medico  wrote:
>>
>> On 1/18/21 6:07 PM, Alec Warner wrote:
>>> On Fri, Jan 15, 2021 at 6:47 PM Matt Turner  wrote:
>>>>
>>>> This set is the upgradable packages for which the highest visible
>>>> version has a different subslot than the currently installed version.
>>>>
>>>> The primary purpose of this feature is for use in catalyst builds. We
>>>> update the "seed" stage3 before using it to build a new stage1.
>>>>
>>>> Updating the entire stage is expensive and unnecessary (since we're
>>>> going to build the latest packages in stage1 and then rebuild everything
>>>> in stage3).
>>>>
>>>> What we definitely do need to update in the original stage3 however, is
>>>> any package that would trigger a subslot rebuild.
>>>>
>>>> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
>>>> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
>>>> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
>>>> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
>>>> will be built and libmpfr.so.6 included into the stage1. Since the old
>>>> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
>>>> subsequent stage builds.
>>>>
>>>> Our current options to update the seed are too large a hammer (e.g.,
>>>> "--update --deep --newuse @world" or "--update --deep --newuse
>>>> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
>>>> updating seed stages for no gain beyond updating only packages for whom
>>>> the subslot has changed.
>>>>
>>>> With this set, catalyst will likely use
>>>>
>>>> emerge @changed-subslot --ignore-built-slot-operator-deps y
>>>>
>>>> to update the seed stage.
>>>>
>>>> Thank you to Zac Medico for showing me how to do this.
>>>>
>>>> Bug: https://bugs.gentoo.org/739004
>>>> Signed-off-by: Matt Turner 
>>>> ---
>>>>  cnf/sets/portage.conf  |  5 +
>>>>  lib/portage/_sets/dbapi.py | 39 +-
>>>>  2 files changed, 43 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
>>>> index 22f0fa3a5..5651a9c53 100644
>>>> --- a/cnf/sets/portage.conf
>>>> +++ b/cnf/sets/portage.conf
>>>> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
>>>>  [rebuilt-binaries]
>>>>  class = portage.sets.dbapi.RebuiltBinaries
>>>>
>>>> +# Installed packages for which the subslot of the highest visible ebuild
>>>> +# version is different than the currently installed version.
>>>> +[changed-subslot]
>>>> +class = portage.sets.dbapi.SubslotChangedSet
>>>> +
>>>>  # Installed packages for which the highest visible ebuild
>>>>  # version is lower than the currently installed version.
>>>>  [downgrade]
>>>> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
>>>> index 52367c4a6..46ba5c17d 100644
>>>> --- a/lib/portage/_sets/dbapi.py
>>>> +++ b/lib/portage/_sets/dbapi.py
>>>> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
>>>>  import portage
>>>>
>>>>  __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
>>>> -   "EverythingSet", "OwnerSet", "VariableSet"]
>>>> +   "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
>>>>
>>>>  class EverythingSet(PackageSet):
>>>> _operations = ["merge"]
>>>> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
>>>>
>>>> singleBuilder = classmethod(singleBuilder)
>>>>
>>>> +class SubslotChangedSet(PackageSet):
>>>> +
>>>> +   _operations = ["merge", "unmerge"]
>>>> +
>>>> +   description = "Package set which contains all packages " + \
>>>> +   "for which the subslot of the highest visible ebuild is " 
>>>> + \
>&g

Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set

2021-01-18 Thread Zac Medico
On 1/18/21 6:07 PM, Alec Warner wrote:
> On Fri, Jan 15, 2021 at 6:47 PM Matt Turner  wrote:
>>
>> This set is the upgradable packages for which the highest visible
>> version has a different subslot than the currently installed version.
>>
>> The primary purpose of this feature is for use in catalyst builds. We
>> update the "seed" stage3 before using it to build a new stage1.
>>
>> Updating the entire stage is expensive and unnecessary (since we're
>> going to build the latest packages in stage1 and then rebuild everything
>> in stage3).
>>
>> What we definitely do need to update in the original stage3 however, is
>> any package that would trigger a subslot rebuild.
>>
>> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
>> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
>> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
>> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
>> will be built and libmpfr.so.6 included into the stage1. Since the old
>> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
>> subsequent stage builds.
>>
>> Our current options to update the seed are too large a hammer (e.g.,
>> "--update --deep --newuse @world" or "--update --deep --newuse
>> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
>> updating seed stages for no gain beyond updating only packages for whom
>> the subslot has changed.
>>
>> With this set, catalyst will likely use
>>
>> emerge @changed-subslot --ignore-built-slot-operator-deps y
>>
>> to update the seed stage.
>>
>> Thank you to Zac Medico for showing me how to do this.
>>
>> Bug: https://bugs.gentoo.org/739004
>> Signed-off-by: Matt Turner 
>> ---
>>  cnf/sets/portage.conf  |  5 +
>>  lib/portage/_sets/dbapi.py | 39 +-
>>  2 files changed, 43 insertions(+), 1 deletion(-)
>>
>> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
>> index 22f0fa3a5..5651a9c53 100644
>> --- a/cnf/sets/portage.conf
>> +++ b/cnf/sets/portage.conf
>> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
>>  [rebuilt-binaries]
>>  class = portage.sets.dbapi.RebuiltBinaries
>>
>> +# Installed packages for which the subslot of the highest visible ebuild
>> +# version is different than the currently installed version.
>> +[changed-subslot]
>> +class = portage.sets.dbapi.SubslotChangedSet
>> +
>>  # Installed packages for which the highest visible ebuild
>>  # version is lower than the currently installed version.
>>  [downgrade]
>> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
>> index 52367c4a6..46ba5c17d 100644
>> --- a/lib/portage/_sets/dbapi.py
>> +++ b/lib/portage/_sets/dbapi.py
>> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
>>  import portage
>>
>>  __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
>> -   "EverythingSet", "OwnerSet", "VariableSet"]
>> +   "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
>>
>>  class EverythingSet(PackageSet):
>> _operations = ["merge"]
>> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
>>
>> singleBuilder = classmethod(singleBuilder)
>>
>> +class SubslotChangedSet(PackageSet):
>> +
>> +   _operations = ["merge", "unmerge"]
>> +
>> +   description = "Package set which contains all packages " + \
>> +   "for which the subslot of the highest visible ebuild is " + \
>> +   "different than the currently installed version."
> 
> description = ("string1",
>"string2",
>"string3")
> 
> vs concat + \ for line continuation?
> 
> -A

We also got flak on irc about the classmethod(singleBuilder) usage as
opposed to @classmethod. This package set is formatted exactly like
others in the file, so it's just a copy / paste thing.

On the topic of python formatting, maybe we should use something like
https://github.com/psf/black to automate it?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set

2021-01-17 Thread Zac Medico
On 1/15/21 6:47 PM, Matt Turner wrote:
> This set is the upgradable packages for which the highest visible
> version has a different subslot than the currently installed version.
> 
> The primary purpose of this feature is for use in catalyst builds. We
> update the "seed" stage3 before using it to build a new stage1.
> 
> Updating the entire stage is expensive and unnecessary (since we're
> going to build the latest packages in stage1 and then rebuild everything
> in stage3).
> 
> What we definitely do need to update in the original stage3 however, is
> any package that would trigger a subslot rebuild.
> 
> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
> will be built and libmpfr.so.6 included into the stage1. Since the old
> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
> subsequent stage builds.
> 
> Our current options to update the seed are too large a hammer (e.g.,
> "--update --deep --newuse @world" or "--update --deep --newuse
> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
> updating seed stages for no gain beyond updating only packages for whom
> the subslot has changed.
> 
> With this set, catalyst will likely use
> 
>   emerge @changed-subslot --ignore-built-slot-operator-deps y
> 
> to update the seed stage.
> 
> Thank you to Zac Medico for showing me how to do this.
> 
> Bug: https://bugs.gentoo.org/739004
> Signed-off-by: Matt Turner 
> ---
>  cnf/sets/portage.conf  |  5 +
>  lib/portage/_sets/dbapi.py | 39 +-
>  2 files changed, 43 insertions(+), 1 deletion(-)

Look good. Thanks! Merged:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=6c70596d6d6382160490bb16ef559ccfdd982fae
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] AsyncioEventLoop: wrap child watcher for thread safety (bug 764905)

2021-01-10 Thread Zac Medico
Use a child watcher wrapper to deliver the callbacks via the
call_soon_threadsafe method, since documentation for the asycio
AbstractChildWatcher class says that callbacks must be thread
safe.

Bug: https://bugs.gentoo.org/764905
Signed-off-by: Zac Medico 
---
 .../util/_eventloop/asyncio_event_loop.py | 30 ++-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/lib/portage/util/_eventloop/asyncio_event_loop.py 
b/lib/portage/util/_eventloop/asyncio_event_loop.py
index 4d7047ae8..b77728088 100644
--- a/lib/portage/util/_eventloop/asyncio_event_loop.py
+++ b/lib/portage/util/_eventloop/asyncio_event_loop.py
@@ -6,6 +6,7 @@ import signal
 
 import asyncio as _real_asyncio
 from asyncio.events import AbstractEventLoop as _AbstractEventLoop
+from asyncio.unix_events import AbstractChildWatcher as _AbstractChildWatcher
 
 import portage
 
@@ -47,6 +48,7 @@ class AsyncioEventLoop(_AbstractEventLoop):
self.set_debug = loop.set_debug
self.get_debug = loop.get_debug
self._wakeup_fd = -1
+   self._child_watcher = None
 
if portage._internal_caller:

loop.set_exception_handler(self._internal_caller_exception_handler)
@@ -87,7 +89,9 @@ class AsyncioEventLoop(_AbstractEventLoop):
@rtype: asyncio.AbstractChildWatcher
@return: the internal event loop's AbstractChildWatcher 
interface
"""
-   return _real_asyncio.get_child_watcher()
+   if self._child_watcher is None:
+   self._child_watcher = 
_ChildWatcherThreadSafetyWrapper(self, _real_asyncio.get_child_watcher())
+   return self._child_watcher
 
@property
def _asyncio_wrapper(self):
@@ -126,3 +130,27 @@ class AsyncioEventLoop(_AbstractEventLoop):
except ValueError:
# This is intended to fail when not called in 
the main thread.
pass
+
+
+class _ChildWatcherThreadSafetyWrapper(_AbstractChildWatcher):
+   def __init__(self, loop, real_watcher):
+   self._loop = loop
+   self._real_watcher = real_watcher
+
+   def close(self):
+   pass
+
+   def __enter__(self):
+   return self
+
+   def __exit__(self, a, b, c):
+   pass
+
+   def _child_exit(self, pid, status, callback, *args):
+   self._loop.call_soon_threadsafe(callback, pid, status, *args)
+
+   def add_child_handler(self, pid, callback, *args):
+   self._real_watcher.add_child_handler(pid, self._child_exit, 
callback, *args)
+
+   def remove_child_handler(self, pid):
+   return self._real_watcher.remove_child_handler(pid)
-- 
2.26.2




[gentoo-portage-dev] [PATCH] check_reverse_dependencies: dereference virtual expansions (bug 764764)

2021-01-10 Thread Zac Medico
If an atom is the result of virtual expansion, then derefrence it to
_orig_atom in check_reverse_dependencies so that it will be correctly
handled as a built slot operator dependency when appropriate. This
solves a case triggered in bug 764764 where a virtual expansion from
virtual/dist-kernel:0/5.10.5= to =virtual/dist-kernel-5.10.5 prevented
the atom from being handled as a built slot operator dependency, which
prevented rebuilds from being triggered.

Bug: https://bugs.gentoo.org/764764
Signed-off-by: Zac Medico 
---
 lib/_emerge/depgraph.py   | 22 +++
 .../test_slot_operator_reverse_deps.py|  2 +-
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index f3e834a60..2bf04406f 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -2073,6 +2073,12 @@ class depgraph:
for parent, atom in 
self._dynamic_config._parent_atoms.get(existing_pkg, []):
if isinstance(parent, Package):
if parent in 
built_slot_operator_parents:
+   if hasattr(atom, '_orig_atom'):
+   # If atom is the result 
of virtual expansion, then
+   # derefrence it to 
_orig_atom so that it will be correctly
+   # handled as a built 
slot operator dependency when
+   # appropriate (see bug 
764764).
+   atom = atom._orig_atom
# This parent may need to be 
rebuilt, therefore
# discard its soname and built 
slot operator
# dependency components which 
are not necessarily
@@ -2131,6 +2137,22 @@ class depgraph:
allow_repo=True)
if not 
atom_set.findAtomForPackage(candidate_pkg,

modified_use=self._pkg_use_enabled(candidate_pkg)):
+   if debug:
+   parent_atoms = []
+   for other_parent, other_atom in 
self._dynamic_config._parent_atoms.get(existing_pkg, []):
+   if other_parent is 
parent:
+   
parent_atoms.append(other_atom)
+   msg = (
+   "",
+   "",
+   
"check_reverse_dependencies:",
+   "   candidate package 
does not match atom '%s': %s" % (atom, candidate_pkg),
+   "   parent: %s" % 
parent,
+   "   parent atoms: %s" % 
" ".join(parent_atoms),
+   "",
+   )
+   writemsg_level("\n".join(msg),
+   noiselevel=-1, 
level=logging.DEBUG)
return False
return True
 
diff --git a/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py 
b/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py
index 6e7214043..ef884f8ca 100644
--- a/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py
+++ b/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py
@@ -284,7 +284,7 @@ class SlotOperatorReverseDepsVirtualTestCase(TestCase):
["@world"],
options = {"--update": True, "--deep": True},
success = True,
-   mergelist = []
+   mergelist = ['sys-kernel/gentoo-kernel-5.10.6', 
'virtual/dist-kernel-5.10.6', 'app-emulation/virtualbox-modules-6.1.16-r1', 
'x11-drivers/nvidia-drivers-460.32.03']
),
)
 
-- 
2.26.2




Re: [gentoo-portage-dev] [PATCH gentoolkit] gentoolkit: Remove gentoolkit.test.cmp

2021-01-09 Thread Zac Medico
On 1/4/21 10:22 AM, Matt Turner wrote:
> Signed-off-by: Matt Turner 
> ---
> Seems to only be used by duplicated unit tests? I guess this might have
> been useful when Python 2 was still supported?
> 
>  pym/gentoolkit/test/__init__.py  | 23 ---
>  pym/gentoolkit/test/test_atom.py | 15 ---
>  pym/gentoolkit/test/test_cpv.py  | 15 ---
>  3 files changed, 53 deletions(-)

Looks good.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH gentoolkit] equery: Remove 'changes' subcommand

2021-01-09 Thread Zac Medico
On 1/4/21 10:44 AM, Matt Turner wrote:
> ChangeLogs have been gone from gentoo.git since the beginning, and
> Council agreed in 2016 to allow Infra to decide whether to distribute
> them through rsync, which they have decided not to do [1].
> 
> [1] https://projects.gentoo.org/council/meeting-logs/20160410-summary.txt
> 
> Signed-off-by: Matt Turner 
> ---
>  pym/gentoolkit/equery/__init__.py   |   1 -
>  pym/gentoolkit/equery/changes.py| 184 
>  pym/gentoolkit/helpers.py   | 173 --
>  pym/gentoolkit/test/equery/test_init.py |   1 -
>  pym/gentoolkit/test/test_helpers.py |  47 --
>  5 files changed, 406 deletions(-)
>  delete mode 100644 pym/gentoolkit/equery/changes.py

Looks good. We should also be able to remove the egencache
--update-changelogs option.

-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] Re: [PATCH] global_event_loop: return running loop for current thread

2021-01-04 Thread Zac Medico
On 1/4/21 1:06 AM, Zac Medico wrote:
> Like asyncio.get_event_loop(), return the running loop for the
> current thread if there is one, and otherwise construct a new
> one if needed. This allows the _safe_loop function to become
> synonymous with the global_event_loop function.
> 
> Bug: https://bugs.gentoo.org/763339
> Signed-off-by: Zac Medico 
> ---
>  .../util/_eventloop/global_event_loop.py  | 28 ++-
>  lib/portage/util/futures/_asyncio/__init__.py | 22 ++-
>  2 files changed, 17 insertions(+), 33 deletions(-)

For the case of "loop running in non-main thread" of API consumer, this
change makes portage compatible with PEP 492 coroutines with async and
await syntax. Portage internals can safely begin using async / await
syntax instead of compat_coroutine.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] global_event_loop: return running loop for current thread

2021-01-04 Thread Zac Medico
Like asyncio.get_event_loop(), return the running loop for the
current thread if there is one, and otherwise construct a new
one if needed. This allows the _safe_loop function to become
synonymous with the global_event_loop function.

Bug: https://bugs.gentoo.org/763339
Signed-off-by: Zac Medico 
---
 .../util/_eventloop/global_event_loop.py  | 28 ++-
 lib/portage/util/futures/_asyncio/__init__.py | 22 ++-
 2 files changed, 17 insertions(+), 33 deletions(-)

diff --git a/lib/portage/util/_eventloop/global_event_loop.py 
b/lib/portage/util/_eventloop/global_event_loop.py
index 413011178..bf314dc34 100644
--- a/lib/portage/util/_eventloop/global_event_loop.py
+++ b/lib/portage/util/_eventloop/global_event_loop.py
@@ -1,28 +1,4 @@
-# Copyright 2012-2020 Gentoo Authors
+# Copyright 2012-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
-import portage
-from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop
-
-_instances = {}
-
-
-def global_event_loop():
-   """
-   Get a global EventLoop (or compatible object) instance which
-   belongs exclusively to the current process.
-   """
-
-   pid = portage.getpid()
-   instance = _instances.get(pid)
-   if instance is not None:
-   return instance
-
-   constructor = AsyncioEventLoop
-
-   # Use the _asyncio_wrapper attribute, so that unit tests can compare
-   # the reference to one retured from _wrap_loop(), since they should
-   # not close the loop if it refers to a global event loop.
-   instance = constructor()._asyncio_wrapper
-   _instances[pid] = instance
-   return instance
+from portage.util.futures._asyncio import _safe_loop as global_event_loop
diff --git a/lib/portage/util/futures/_asyncio/__init__.py 
b/lib/portage/util/futures/_asyncio/__init__.py
index d39f31786..ab1468d43 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -1,4 +1,4 @@
-# Copyright 2018-2020 Gentoo Authors
+# Copyright 2018-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 __all__ = (
@@ -37,9 +37,6 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.futures:compat_coroutine@_compat_coroutine',
 )
 from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop as 
_AsyncioEventLoop
-from portage.util._eventloop.global_event_loop import (
-   global_event_loop as _global_event_loop,
-)
 # pylint: disable=redefined-builtin
 from portage.util.futures.futures import (
CancelledError,
@@ -238,7 +235,7 @@ def _wrap_loop(loop=None):
# The default loop returned by _wrap_loop should be consistent
# with global_event_loop, in order to avoid accidental registration
# of callbacks with a loop that is not intended to run.
-   loop = loop or _global_event_loop()
+   loop = loop or _safe_loop()
return (loop if hasattr(loop, '_asyncio_wrapper')
else _AsyncioEventLoop(loop=loop))
 
@@ -267,8 +264,9 @@ def _safe_loop():
@rtype: asyncio.AbstractEventLoop (or compatible)
@return: event loop instance
"""
-   if portage._internal_caller or threading.current_thread() is 
threading.main_thread():
-   return _global_event_loop()
+   loop = _get_running_loop()
+   if loop is not None:
+   return loop
 
thread_key = threading.get_ident()
with _thread_weakrefs.lock:
@@ -286,6 +284,16 @@ def _safe_loop():
return loop
 
 
+def _get_running_loop():
+   with _thread_weakrefs.lock:
+   if _thread_weakrefs.pid == portage.getpid():
+   try:
+   loop = 
_thread_weakrefs.loops[threading.get_ident()]
+   except KeyError:
+   return None
+   return loop if loop.is_running() else None
+
+
 def _thread_weakrefs_atexit():
with _thread_weakrefs.lock:
if _thread_weakrefs.pid == portage.getpid():
-- 
2.26.2




Re: [gentoo-portage-dev] [PATCH gentoolkit] eclean: Add --changed-iuse flag

2021-01-03 Thread Zac Medico
On 1/2/21 4:08 PM, Matt Turner wrote:
> Allows binpkgs to be deleted if they are not usable due to IUSE changes.
> ---
> Just kind of spitballing. I'm not sure about what USE flags we should
> ignore or whether it should be configurable, etc. On one hand, deleting
> binpkgs that don't have a newly added PYTHON_TARGET option might make
> sense if your binhost is configured to rebuild the package. On the
> other, you probably don't want to throw out amd64 binpkgs because
> abi_riscv_* was added.

The special case for abi_* flags is ugly. Why not do emerge emerge
--changed-use, and ignore changed IUSE for flags that aren't enabled?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH gentoolkit] bin: Add merge-driver-ekeyword

2020-12-31 Thread Zac Medico
On 12/31/20 11:47 AM, Matt Turner wrote:
> Since the KEYWORDS=... assignment is a single line, git struggles to
> handle conflicts. When rebasing a series of commits that modify the
> KEYWORDS=... it's usually easier to throw them away and reapply on the
> new tree than it is to manually handle conflicts during the rebase.
> 
> git allows a 'merge driver' program to handle conflicts; this program
> handles conflicts in the KEYWORDS=... assignment. E.g., given an ebuild
> with these keywords:
> 
> KEYWORDS="~alpha amd64 arm arm64 ~hppa ppc ppc64 x86"
> 
> One developer drops the ~alpha keyword and pushes to gentoo.git, and
> another developer stabilizes hppa. Without this merge driver, git
> requires the second developer to manually resolve the conflict which is
> tedious and prone to mistakes when rebasing a long series of patches.
> With the custom merge driver, it automatically resolves the conflict.
> 
> To use the merge driver, configure your gentoo.git as such:
> 
> gentoo.git/.git/config:
> 
>   [merge "keywords"]
>   name = KEYWORDS merge driver
>   driver = merge-driver-ekeyword %O %A %B %P
> 
> gentoo.git/.git/info/attributes:
> 
>   *.ebuild merge=keywords
> 
> Signed-off-by: Matt Turner 
> ---
> v3: Address Zac's feedback: use tempfile.TemporaryDirectory

Looks great!
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH 1/2] Make atomic_ofstream a Context Manager

2020-12-30 Thread Zac Medico
On 12/18/20 10:46 AM, Florian Schmaus wrote:
> This allows using a "with statement" together with instances of
> atomic_ofstream. Allowing for more readable, less error prone and
> shorter code.
> 
> Signed-off-by: Florian Schmaus 

Thanks, this is much better! Merged both patches:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=e93e6d65fa1ca75f676a227f7918f8b6d747425c
https://gitweb.gentoo.org/proj/portage.git/commit/?id=1574ae127b270739c4293271c959d1d981684906
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH gentoolkit] bin: Add merge-driver-ekeyword

2020-12-28 Thread Zac Medico
On 12/28/20 5:09 PM, Zac Medico wrote:
> On 12/28/20 3:15 PM, Matt Turner wrote:
>> +def apply_keyword_changes(ebuild: str, pathname: str,
>> +  changes: List[Tuple[Optional[str],
>> +  Optional[str]]]) -> int:
>> +result: int = 0
>> +
>> +# ekeyword will only modify files named *.ebuild, so make a symlink
>> +ebuild_symlink: str = os.path.basename(pathname)
>> +os.symlink(ebuild, ebuild_symlink)
> 
> Are we sure that the current working directory is an entirely safe place
> to create this symlink? A simple fix would be to use
> tempfile.TemporaryDirectory to create a temporary directory to hold the
> symlink. Or, we could change ekeyword to assume that an argument is an
> ebuild if os.path.isfile(arg) succeeds.
> 
>> +for removals, additions in changes:
>> +args = []
>> +for rem in removals:
>> +# Drop leading '~' and '-' characters and prepend '^'
>> +i = 1 if rem[0] in ('~', '-') else 0
>> +args.append('^' + rem[i:])
>> +if additions:
>> +args.extend(additions)
>> +args.append(ebuild_symlink)
>> +
>> +result = ekeyword.main(args)

Another option is to bypass the ekeyword.main function, like this:

try:
   ekeyword.process_ebuild(pathname, list(map(ekeyword.arg_to_op, args))
except Exception:
   result = 1
   traceback.print_exc()
else:
   result = 0


>> +if result != 0:
>> +break
>> +
>> +os.remove(ebuild_symlink)
>> +return result
> 
> 


-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH v2] ekeyword: remove .ebuild file suffix requirement (bug 762331)

2020-12-28 Thread Zac Medico
We'd like to use ekeyword in a git merge driver implementation, but the
files that the driver will pass to ekeyword do not necessarily have a
.ebuild suffix. Therefore, it would be handy to be able to distinguish
ebuild arguments some other way. If the ignorable_arg(arg) function
returns True and os.path.isfile(arg) returns True, then simply assume
that the argument is an ebuild.

Bug: https://bugs.gentoo.org/762331
Signed-off-by: Zac Medico 
---
[PATCH v2] fix to respect the ignorable_arg function

 pym/gentoolkit/ekeyword/ekeyword.py | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/pym/gentoolkit/ekeyword/ekeyword.py 
b/pym/gentoolkit/ekeyword/ekeyword.py
index 4e57c09..eeceed4 100755
--- a/pym/gentoolkit/ekeyword/ekeyword.py
+++ b/pym/gentoolkit/ekeyword/ekeyword.py
@@ -244,7 +244,7 @@ def process_content(ebuild, data, ops, arch_status=None, 
verbose=0,
pass
else:
# Chop the full path and the .ebuild suffix.
-   disp_name = os.path.basename(ebuild)[:-7]
+   disp_name, _, _ = os.path.basename(ebuild).partition('.ebuild')
def logit(msg):
print('%s: %s' % (disp_name, msg))
 
@@ -395,7 +395,9 @@ def args_to_work(args, arch_status=None, _repo=None, 
quiet=0):
last_todo_arches = []
 
for arg in args:
-   if arg.endswith('.ebuild'):
+   if ignorable_arg(arg, quiet=quiet):
+   pass
+   elif os.path.isfile(arg):
if not todo_arches:
todo_arches = last_todo_arches
work.append([arg, todo_arches])
@@ -405,7 +407,7 @@ def args_to_work(args, arch_status=None, _repo=None, 
quiet=0):
op = arg_to_op(arg)
if not arch_status or op.arch in arch_status:
todo_arches.append(op)
-   elif not ignorable_arg(arg, quiet=quiet):
+   else:
raise ValueError('unknown arch/argument: %s' % 
arg)
 
if todo_arches:
@@ -475,6 +477,7 @@ def main(argv):
opts.style = 'color-inline'
 
arch_status = load_profile_data()
+   print(arch_status)
try:
work = args_to_work(work_args, arch_status=arch_status, 
quiet=opts.quiet)
except ValueError as e:
-- 
2.26.2




[gentoo-portage-dev] [PATCH] ekeyword: remove .ebuild file suffix requirement (bug 762331)

2020-12-28 Thread Zac Medico
We'd like to use ekeyword in a git merge driver implementation, but the
files that the driver will pass to ekeyword do not necessarily have a
.ebuild suffix. Therefore, it would be handy to be able to distinguish
ebuild arguments some other way.

Bug: https://bugs.gentoo.org/762331
Signed-off-by: Zac Medico 
---
 pym/gentoolkit/ekeyword/ekeyword.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pym/gentoolkit/ekeyword/ekeyword.py 
b/pym/gentoolkit/ekeyword/ekeyword.py
index 4e57c09..665eee5 100755
--- a/pym/gentoolkit/ekeyword/ekeyword.py
+++ b/pym/gentoolkit/ekeyword/ekeyword.py
@@ -244,7 +244,7 @@ def process_content(ebuild, data, ops, arch_status=None, 
verbose=0,
pass
else:
# Chop the full path and the .ebuild suffix.
-   disp_name = os.path.basename(ebuild)[:-7]
+   disp_name, _, _ = os.path.basename(ebuild).partition('.ebuild')
def logit(msg):
print('%s: %s' % (disp_name, msg))
 
@@ -395,7 +395,7 @@ def args_to_work(args, arch_status=None, _repo=None, 
quiet=0):
last_todo_arches = []
 
for arg in args:
-   if arg.endswith('.ebuild'):
+   if os.path.isfile(arg):
if not todo_arches:
todo_arches = last_todo_arches
work.append([arg, todo_arches])
-- 
2.26.2




Re: [gentoo-portage-dev] [PATCH gentoolkit] bin: Add merge-driver-ekeyword

2020-12-28 Thread Zac Medico
On 12/28/20 3:15 PM, Matt Turner wrote:
> +def apply_keyword_changes(ebuild: str, pathname: str,
> +  changes: List[Tuple[Optional[str],
> +  Optional[str]]]) -> int:
> +result: int = 0
> +
> +# ekeyword will only modify files named *.ebuild, so make a symlink
> +ebuild_symlink: str = os.path.basename(pathname)
> +os.symlink(ebuild, ebuild_symlink)

Are we sure that the current working directory is an entirely safe place
to create this symlink? A simple fix would be to use
tempfile.TemporaryDirectory to create a temporary directory to hold the
symlink. Or, we could change ekeyword to assume that an argument is an
ebuild if os.path.isfile(arg) succeeds.

> +for removals, additions in changes:
> +args = []
> +for rem in removals:
> +# Drop leading '~' and '-' characters and prepend '^'
> +i = 1 if rem[0] in ('~', '-') else 0
> +args.append('^' + rem[i:])
> +if additions:
> +args.extend(additions)
> +args.append(ebuild_symlink)
> +
> +result = ekeyword.main(args)
> +if result != 0:
> +break
> +
> +os.remove(ebuild_symlink)
> +return result


-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH] Drop Python 2 comatibility in extension modules

2020-12-24 Thread Zac Medico
On 12/24/20 10:14 AM, Mike Gilbert wrote:
> Signed-off-by: Mike Gilbert 
> ---
>  src/portage_util_file_copy_reflink_linux.c | 10 +-
>  src/portage_util_libc.c| 10 +-
>  2 files changed, 2 insertions(+), 18 deletions(-)

Looks good. Please merge.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH] Adjust mangling of "arch" value from scanelf output

2020-12-23 Thread Zac Medico
On 12/23/20 7:34 PM, Mike Gilbert wrote:
> scanelf may generate output that looks like this:
> 
> ```
> UNKNOWN_TYPE;lib/firmware/ath10k/WCN3990/hw1.0/wlanmdsp.mbn;;  -  ;
> EM_ARM;lib/firmware/mediatek/mt8183/scp.img;;  -  ;
> ...
> ```
> 
> Previously, we removed the first 3 characters of the first field and
> stored this as the "arch" in NEEDED.ELF.2. This unintentionally
> changes "UNKNOWN_TYPE" to "NOWN_TYPE".
> 
> Instead, let's just remove the string "EM_" from the front.
> 
> Signed-off-by: Mike Gilbert 
> ---
>  bin/misc-functions.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
> index c2a16cbe0..d7009d7eb 100755
> --- a/bin/misc-functions.sh
> +++ b/bin/misc-functions.sh
> @@ -194,7 +194,7 @@ install_qa_check() {
>   fi
>  
>   echo "${obj} ${needed}" >> 
> "${PORTAGE_BUILDDIR}"/build-info/NEEDED
> - echo "${arch:3};${obj};${soname};${rpath};${needed}" >> 
> "${PORTAGE_BUILDDIR}"/build-info/NEEDED.ELF.2
> + echo "${arch#EM_};${obj};${soname};${rpath};${needed}" 
> >> "${PORTAGE_BUILDDIR}"/build-info/NEEDED.ELF.2
>   done }
>  
>   [ -n "${QA_SONAME_NO_SYMLINK}" ] && \
> 

Look good. This won't cause any problems for portage since these files
are outside of the known multilib categories.
-- 
Thanks,
Zac





signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] [PATCH gentoolkit 1/4] Remove imports from __future__

2020-12-20 Thread Zac Medico
On 12/20/20 2:10 PM, Matt Turner wrote:
> gentoolkit supports only Python 3.6+ now, so these are not used.
> 
> Signed-off-by: Matt Turner 

The whole series LGTM (including "Remove unused sys imports" patch 5/4).
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH v4] Use default asyncio event loop implementation in API consumer threads

2020-12-06 Thread Zac Medico
Make the _safe_loop function return an AsyncioEventLoop instance,
so that the default asyncio event loop implementation will be used
in API consumer threads. This is possible because the underlying
asyncio.get_event_loop() function returns a separate  event loop for
each thread. The AsyncioEventLoop _run_until_complete method will
now appropriately handle a ValueError from signal.set_wakeup_fd(-1)
if it is not called in the main thread.

For external API consumers calling from a non-main thread, an
asyncio loop must be registered for the current thread, or else an
error will be raised like this:

  RuntimeError: There is no current event loop in thread 'Thread-1'.

In order to avoid this RuntimeError, the external API consumer
is responsible for setting an event loop and managing its lifecycle.
This code will set an event loop for the current thread:

  asyncio.set_event_loop(asyncio.new_event_loop())

In order to avoid a ResourceWarning, the caller should also close
the corresponding loop before the current thread terminates.

Bug: https://bugs.gentoo.org/758755
Signed-off-by: Zac Medico 
---
[PATCH v4] treat external API consumers the same as interal callers
if they call from the main thread, and document asyncio loop
lifecycle management now required for external API consumers
calling from a non-main thread

 .../util/_eventloop/asyncio_event_loop.py |  6 -
 lib/portage/util/futures/_asyncio/__init__.py | 26 ++-
 2 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/lib/portage/util/_eventloop/asyncio_event_loop.py 
b/lib/portage/util/_eventloop/asyncio_event_loop.py
index 836f1c30a..4d7047ae8 100644
--- a/lib/portage/util/_eventloop/asyncio_event_loop.py
+++ b/lib/portage/util/_eventloop/asyncio_event_loop.py
@@ -121,4 +121,8 @@ class AsyncioEventLoop(_AbstractEventLoop):
try:
return self._loop.run_until_complete(future)
finally:
-   self._wakeup_fd = signal.set_wakeup_fd(-1)
+   try:
+   self._wakeup_fd = signal.set_wakeup_fd(-1)
+   except ValueError:
+   # This is intended to fail when not called in 
the main thread.
+   pass
diff --git a/lib/portage/util/futures/_asyncio/__init__.py 
b/lib/portage/util/futures/_asyncio/__init__.py
index a902ad895..0b35c6daf 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -34,7 +34,6 @@ import portage
 portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.futures.unix_events:_PortageEventLoopPolicy',
'portage.util.futures:compat_coroutine@_compat_coroutine',
-   'portage.util._eventloop.EventLoop:EventLoop@_EventLoop',
 )
 from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop as 
_AsyncioEventLoop
 from portage.util._eventloop.global_event_loop import (
@@ -246,14 +245,27 @@ def _wrap_loop(loop=None):
 def _safe_loop():
"""
Return an event loop that's safe to use within the current context.
-   For portage internal callers, this returns a globally shared event
-   loop instance. For external API consumers, this constructs a
-   temporary event loop instance that's safe to use in a non-main
-   thread (it does not override the global SIGCHLD handler).
+   For portage internal callers or external API consumers calling from
+   the main thread, this returns a globally shared event loop instance.
+
+   For external API consumers calling from a non-main thread, an
+   asyncio loop must be registered for the current thread, or else an
+   error will be raised like this:
+
+ RuntimeError: There is no current event loop in thread 'Thread-1'.
+
+   In order to avoid this RuntimeError, the external API consumer
+   is responsible for setting an event loop and managing its lifecycle.
+   This code will set an event loop for the current thread:
+
+ asyncio.set_event_loop(asyncio.new_event_loop())
+
+   In order to avoid a ResourceWarning, the caller should also close the
+   corresponding loop before the current thread terminates.
 
@rtype: asyncio.AbstractEventLoop (or compatible)
@return: event loop instance
"""
-   if portage._internal_caller:
+   if portage._internal_caller or threading.current_thread() is 
threading.main_thread():
return _global_event_loop()
-   return _EventLoop(main=False)
+   return _AsyncioEventLoop()
-- 
2.26.2




[gentoo-portage-dev] Re: [PATCH v3] Use default asyncio event loop implementation in API consumer threads

2020-12-06 Thread Zac Medico
Accidentally encrypted the last email. Here's an unencrypted version.

On 12/6/20 2:14 PM, Zac Medico wrote:
> Make the _safe_loop function return an AsyncioEventLoop instance,
> so that the default asyncio event loop implementation will be used
> in API consumer threads. This is possible because the underlying
> asyncio.get_event_loop() function returns a new event loop for
> each thread. The AsyncioEventLoop _run_until_complete method will
> now appropriately handle a ValueError from signal.set_wakeup_fd(-1)
> if it is not called in the main thread.
> 
> Bug: https://bugs.gentoo.org/758755
> Signed-off-by: Zac Medico 
> ---
> [PATCH v3] fixed AsyncioEventLoop _run_until_complete method to
> handle ValueError from signal.set_wakeup_fd(-1)
> 
>  lib/portage/util/_eventloop/asyncio_event_loop.py | 6 +-
>  lib/portage/util/futures/_asyncio/__init__.py | 3 +--
>  2 files changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/lib/portage/util/_eventloop/asyncio_event_loop.py 
> b/lib/portage/util/_eventloop/asyncio_event_loop.py
> index 836f1c30a..4d7047ae8 100644
> --- a/lib/portage/util/_eventloop/asyncio_event_loop.py
> +++ b/lib/portage/util/_eventloop/asyncio_event_loop.py
> @@ -121,4 +121,8 @@ class AsyncioEventLoop(_AbstractEventLoop):
>   try:
>   return self._loop.run_until_complete(future)
>   finally:
> - self._wakeup_fd = signal.set_wakeup_fd(-1)
> + try:
> + self._wakeup_fd = signal.set_wakeup_fd(-1)
> + except ValueError:
> + # This is intended to fail when not called in 
> the main thread.
> + pass
> diff --git a/lib/portage/util/futures/_asyncio/__init__.py 
> b/lib/portage/util/futures/_asyncio/__init__.py
> index a902ad895..12013be00 100644
> --- a/lib/portage/util/futures/_asyncio/__init__.py
> +++ b/lib/portage/util/futures/_asyncio/__init__.py
> @@ -34,7 +34,6 @@ import portage
>  portage.proxy.lazyimport.lazyimport(globals(),
>   'portage.util.futures.unix_events:_PortageEventLoopPolicy',
>   'portage.util.futures:compat_coroutine@_compat_coroutine',
> - 'portage.util._eventloop.EventLoop:EventLoop@_EventLoop',
>  )
>  from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop as 
> _AsyncioEventLoop
>  from portage.util._eventloop.global_event_loop import (
> @@ -256,4 +255,4 @@ def _safe_loop():
>   """
>   if portage._internal_caller:
>   return _global_event_loop()
> - return _EventLoop(main=False)
> + return _AsyncioEventLoop()
> 

This fails if an event loop has not been created for the current thread:

  File "/usr/lib/python3.8/asyncio/events.py", line 639, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-1'.

However, if we automatically instantiate a loop for the current thread
then we will be responsible for closing it as well, or else we'll
eventually see a ResourceWarning like this:

/usr/lib/python3.8/asyncio/base_events.py:654: ResourceWarning: unclosed
event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
  _warn(f"unclosed event loop {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

So, I think it's probably best if we force the API consumer to manage
the lifecycle of an asyncio loop for each thread that it uses to call
the portage API.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] Re: [PATCH v3] Use default asyncio event loop implementation in API consumer threads

2020-12-06 Thread Zac Medico
On 12/6/20 2:14 PM, Zac Medico wrote:
> Make the _safe_loop function return an AsyncioEventLoop instance,
> so that the default asyncio event loop implementation will be used
> in API consumer threads. This is possible because the underlying
> asyncio.get_event_loop() function returns a new event loop for
> each thread. The AsyncioEventLoop _run_until_complete method will
> now appropriately handle a ValueError from signal.set_wakeup_fd(-1)
> if it is not called in the main thread.
> 
> Bug: https://bugs.gentoo.org/758755
> Signed-off-by: Zac Medico 
> ---
> [PATCH v3] fixed AsyncioEventLoop _run_until_complete method to
> handle ValueError from signal.set_wakeup_fd(-1)
> 
>  lib/portage/util/_eventloop/asyncio_event_loop.py | 6 +-
>  lib/portage/util/futures/_asyncio/__init__.py | 3 +--
>  2 files changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/lib/portage/util/_eventloop/asyncio_event_loop.py 
> b/lib/portage/util/_eventloop/asyncio_event_loop.py
> index 836f1c30a..4d7047ae8 100644
> --- a/lib/portage/util/_eventloop/asyncio_event_loop.py
> +++ b/lib/portage/util/_eventloop/asyncio_event_loop.py
> @@ -121,4 +121,8 @@ class AsyncioEventLoop(_AbstractEventLoop):
>   try:
>   return self._loop.run_until_complete(future)
>   finally:
> - self._wakeup_fd = signal.set_wakeup_fd(-1)
> + try:
> + self._wakeup_fd = signal.set_wakeup_fd(-1)
> + except ValueError:
> + # This is intended to fail when not called in 
> the main thread.
> + pass
> diff --git a/lib/portage/util/futures/_asyncio/__init__.py 
> b/lib/portage/util/futures/_asyncio/__init__.py
> index a902ad895..12013be00 100644
> --- a/lib/portage/util/futures/_asyncio/__init__.py
> +++ b/lib/portage/util/futures/_asyncio/__init__.py
> @@ -34,7 +34,6 @@ import portage
>  portage.proxy.lazyimport.lazyimport(globals(),
>   'portage.util.futures.unix_events:_PortageEventLoopPolicy',
>   'portage.util.futures:compat_coroutine@_compat_coroutine',
> - 'portage.util._eventloop.EventLoop:EventLoop@_EventLoop',
>  )
>  from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop as 
> _AsyncioEventLoop
>  from portage.util._eventloop.global_event_loop import (
> @@ -256,4 +255,4 @@ def _safe_loop():
>   """
>   if portage._internal_caller:
>   return _global_event_loop()
> - return _EventLoop(main=False)
> + return _AsyncioEventLoop()
> 

This fails if an event loop has not been created for the current thread:

  File "/usr/lib/python3.8/asyncio/events.py", line 639, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-1'.

However, if we automatically instantiate a loop for the current thread
then we will be responsible for closing it as well, or else we'll
eventually see a ResourceWarning like this:

/usr/lib/python3.8/asyncio/base_events.py:654: ResourceWarning: unclosed
event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
  _warn(f"unclosed event loop {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

So, I think it's probably best if we force the API consumer to manage
the lifecycle of an asyncio loop for each thread that it uses to call
the portage API.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH v3] Use default asyncio event loop implementation in API consumer threads

2020-12-06 Thread Zac Medico
Make the _safe_loop function return an AsyncioEventLoop instance,
so that the default asyncio event loop implementation will be used
in API consumer threads. This is possible because the underlying
asyncio.get_event_loop() function returns a new event loop for
each thread. The AsyncioEventLoop _run_until_complete method will
now appropriately handle a ValueError from signal.set_wakeup_fd(-1)
if it is not called in the main thread.

Bug: https://bugs.gentoo.org/758755
Signed-off-by: Zac Medico 
---
[PATCH v3] fixed AsyncioEventLoop _run_until_complete method to
handle ValueError from signal.set_wakeup_fd(-1)

 lib/portage/util/_eventloop/asyncio_event_loop.py | 6 +-
 lib/portage/util/futures/_asyncio/__init__.py | 3 +--
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/lib/portage/util/_eventloop/asyncio_event_loop.py 
b/lib/portage/util/_eventloop/asyncio_event_loop.py
index 836f1c30a..4d7047ae8 100644
--- a/lib/portage/util/_eventloop/asyncio_event_loop.py
+++ b/lib/portage/util/_eventloop/asyncio_event_loop.py
@@ -121,4 +121,8 @@ class AsyncioEventLoop(_AbstractEventLoop):
try:
return self._loop.run_until_complete(future)
finally:
-   self._wakeup_fd = signal.set_wakeup_fd(-1)
+   try:
+   self._wakeup_fd = signal.set_wakeup_fd(-1)
+   except ValueError:
+   # This is intended to fail when not called in 
the main thread.
+   pass
diff --git a/lib/portage/util/futures/_asyncio/__init__.py 
b/lib/portage/util/futures/_asyncio/__init__.py
index a902ad895..12013be00 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -34,7 +34,6 @@ import portage
 portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.futures.unix_events:_PortageEventLoopPolicy',
'portage.util.futures:compat_coroutine@_compat_coroutine',
-   'portage.util._eventloop.EventLoop:EventLoop@_EventLoop',
 )
 from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop as 
_AsyncioEventLoop
 from portage.util._eventloop.global_event_loop import (
@@ -256,4 +255,4 @@ def _safe_loop():
"""
if portage._internal_caller:
return _global_event_loop()
-   return _EventLoop(main=False)
+   return _AsyncioEventLoop()
-- 
2.26.2




[gentoo-portage-dev] [PATCH v2] Use default asyncio event loop implementation in API consumer threads

2020-12-06 Thread Zac Medico
Make the _safe_loop function return an AsyncioEventLoop instance,
so that the default asyncio event loop implementation will be used
in API consumer threads. This is possible because the underlying
asyncio.get_event_loop() function returns a new event loop for
each thread.

Bug: https://bugs.gentoo.org/758755
Signed-off-by: Zac Medico 
---
[PATCH v2] fixed _safe_loop function to return a new
AsyncioEventLoop per thread

 lib/portage/util/futures/_asyncio/__init__.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py 
b/lib/portage/util/futures/_asyncio/__init__.py
index a902ad895..12013be00 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -34,7 +34,6 @@ import portage
 portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.futures.unix_events:_PortageEventLoopPolicy',
'portage.util.futures:compat_coroutine@_compat_coroutine',
-   'portage.util._eventloop.EventLoop:EventLoop@_EventLoop',
 )
 from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop as 
_AsyncioEventLoop
 from portage.util._eventloop.global_event_loop import (
@@ -256,4 +255,4 @@ def _safe_loop():
"""
if portage._internal_caller:
return _global_event_loop()
-   return _EventLoop(main=False)
+   return _AsyncioEventLoop()
-- 
2.26.2




[gentoo-portage-dev] Re: [PATCH] Use default asyncio event loop implementation in API consumer threads

2020-12-06 Thread Zac Medico
On 12/6/20 1:46 AM, Zac Medico wrote:
> Make the _safe_loop function an alias for the global_event_loop
> function, so that the default asyncio event loop implementation
> will be used in API consumer threads. This is possible because
> global_event_loop has been fixed (bug 758740) to always use
> AsyncioEventLoop, and that uses asyncio.get_event_loop() which
> returns a new event loop for each thread.

I think we may still need a separate _safe_loop function here,
since global_event_loop returns a separate loop per pid, but
_safe_loop needs to return a separate loop per thread.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] Use default asyncio event loop implementation in API consumer threads

2020-12-06 Thread Zac Medico
Make the _safe_loop function an alias for the global_event_loop
function, so that the default asyncio event loop implementation
will be used in API consumer threads. This is possible because
global_event_loop has been fixed (bug 758740) to always use
AsyncioEventLoop, and that uses asyncio.get_event_loop() which
returns a new event loop for each thread.

Bug: https://bugs.gentoo.org/758755
Signed-off-by: Zac Medico 
---
 lib/portage/util/futures/_asyncio/__init__.py | 17 +
 1 file changed, 1 insertion(+), 16 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py 
b/lib/portage/util/futures/_asyncio/__init__.py
index a902ad895..ce3685709 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -39,6 +39,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
 from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop as 
_AsyncioEventLoop
 from portage.util._eventloop.global_event_loop import (
global_event_loop as _global_event_loop,
+   global_event_loop as _safe_loop,
 )
 # pylint: disable=redefined-builtin
 from portage.util.futures.futures import (
@@ -241,19 +242,3 @@ def _wrap_loop(loop=None):
loop = loop or _global_event_loop()
return (loop if hasattr(loop, '_asyncio_wrapper')
else _AsyncioEventLoop(loop=loop))
-
-
-def _safe_loop():
-   """
-   Return an event loop that's safe to use within the current context.
-   For portage internal callers, this returns a globally shared event
-   loop instance. For external API consumers, this constructs a
-   temporary event loop instance that's safe to use in a non-main
-   thread (it does not override the global SIGCHLD handler).
-
-   @rtype: asyncio.AbstractEventLoop (or compatible)
-   @return: event loop instance
-   """
-   if portage._internal_caller:
-   return _global_event_loop()
-   return _EventLoop(main=False)
-- 
2.26.2




[gentoo-portage-dev] [PATCH] Use default asyncio event loop implementation in child processes

2020-12-06 Thread Zac Medico
Use the default asyncio event loop implementation in child
processes, instead of portage's internal EventLoop. After
fork, instantiate a new asyncio.DefaultEventLoopPolicy as
a workaround for https://bugs.python.org/issue22087, which
is necessary for RetryTestCase to succeed.

Bug: https://bugs.gentoo.org/758740
Signed-off-by: Zac Medico 
---
 lib/portage/__init__.py  | 4 
 lib/portage/util/_eventloop/global_event_loop.py | 7 ---
 2 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py
index 4d4b590a8..2b821e81a 100644
--- a/lib/portage/__init__.py
+++ b/lib/portage/__init__.py
@@ -9,6 +9,7 @@ VERSION = "HEAD"
 # ===
 
 try:
+   import asyncio
import sys
import errno
if not hasattr(errno, 'ESTALE'):
@@ -373,6 +374,9 @@ class _ForkWatcher:
@staticmethod
def hook(_ForkWatcher):
_ForkWatcher.current_pid = _os.getpid()
+   # Force instantiation of a new event loop as a workaround for
+   # https://bugs.python.org/issue22087.
+   asyncio.set_event_loop_policy(asyncio.DefaultEventLoopPolicy())
 
 _ForkWatcher.hook(_ForkWatcher)
 
diff --git a/lib/portage/util/_eventloop/global_event_loop.py 
b/lib/portage/util/_eventloop/global_event_loop.py
index 21a1d1970..413011178 100644
--- a/lib/portage/util/_eventloop/global_event_loop.py
+++ b/lib/portage/util/_eventloop/global_event_loop.py
@@ -2,11 +2,8 @@
 # Distributed under the terms of the GNU General Public License v2
 
 import portage
-from .EventLoop import EventLoop
 from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop
 
-
-_MAIN_PID = portage.getpid()
 _instances = {}
 
 
@@ -22,10 +19,6 @@ def global_event_loop():
return instance
 
constructor = AsyncioEventLoop
-   # If the default constructor doesn't support multiprocessing,
-   # then multiprocessing constructor is used in subprocesses.
-   if not constructor.supports_multiprocessing and pid != _MAIN_PID:
-   constructor = EventLoop
 
# Use the _asyncio_wrapper attribute, so that unit tests can compare
# the reference to one retured from _wrap_loop(), since they should
-- 
2.26.2




[gentoo-portage-dev] [PATCH] _get_lock_fn: support multiprocessing spawn start method (bug 758230)

2020-12-04 Thread Zac Medico
Ensure that _get_lock_fn arguments to multiprocessing.Process will
successfully pickle, as required by the spawn start method, which
is the default for macOS since Python 3.8.

Since file descriptors are not inherited unless the fork start
method is used, the subprocess should only try to close an
inherited file descriptor for the fork start method.

Bug: https://bugs.gentoo.org/758230
Signed-off-by: Zac Medico 
---
 lib/portage/locks.py | 36 +++-
 1 file changed, 23 insertions(+), 13 deletions(-)

diff --git a/lib/portage/locks.py b/lib/portage/locks.py
index 1073343be..193045c03 100644
--- a/lib/portage/locks.py
+++ b/lib/portage/locks.py
@@ -44,18 +44,7 @@ def _get_lock_fn():
if _lock_fn is not None:
return _lock_fn
 
-   def _test_lock(fd, lock_path):
-   os.close(fd)
-   try:
-   with open(lock_path, 'a') as f:
-   fcntl.lockf(f.fileno(), 
fcntl.LOCK_EX|fcntl.LOCK_NB)
-   except EnvironmentError as e:
-   if e.errno == errno.EAGAIN:
-   # Parent process holds lock, as expected.
-   sys.exit(0)
 
-   # Something went wrong.
-   sys.exit(1)
 
fd, lock_path = tempfile.mkstemp()
try:
@@ -64,8 +53,16 @@ def _get_lock_fn():
except EnvironmentError:
pass
else:
-   proc = multiprocessing.Process(target=_test_lock,
-   args=(fd, lock_path))
+   proc = multiprocessing.Process(
+   target=_subprocess_test_lock,
+   args=(
+   # Since file descriptors are not 
inherited unless the fork start
+   # method is used, the subprocess should 
only try to close an
+   # inherited file descriptor for the 
fork start method.
+   fd if 
multiprocessing.get_start_method() == "fork" else None,
+   lock_path,
+   ),
+   )
proc.start()
proc.join()
if proc.exitcode == os.EX_OK:
@@ -80,6 +77,19 @@ def _get_lock_fn():
_lock_fn = fcntl.flock
return _lock_fn
 
+def _subprocess_test_lock(fd, lock_path):
+   if fd is not None:
+   os.close(fd)
+   try:
+   with open(lock_path, 'a') as f:
+   fcntl.lockf(f.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB)
+   except EnvironmentError as e:
+   if e.errno == errno.EAGAIN:
+   # Parent process holds lock, as expected.
+   sys.exit(0)
+
+   # Something went wrong.
+   sys.exit(1)
 
 _open_fds = {}
 _open_inodes = {}
-- 
2.26.2




[gentoo-portage-dev] [PATCH] find_smallest_cycle: don't merge satisfied PDEPEND too early

2020-12-02 Thread Zac Medico
After PDEPENDs have been neglected by the find_smallest_cycle function,
do not try to merge them too early if they are already satisfied by
an installed package. This fixes incorrect merge order for PDEPEND
cycles involving xorg-server and xorg-drivers, which was triggered
by commit 5095c2023595a75e2848f1ad3dbe25b5fb451a44 because it gave
PDEPEND higher priority than satisfied buildtime dependencies.

Fixes: 5095c2023595 ("find_smallest_cycle: enhance search prioritization")
Reported-by: josef64 in #gentoo-portage
Bug: https://bugs.gentoo.org/754903
Signed-off-by: Zac Medico 
---
 lib/_emerge/DepPrioritySatisfiedRange.py  |  1 +
 lib/_emerge/depgraph.py   |  8 +++---
 .../tests/resolver/test_merge_order.py| 27 +--
 3 files changed, 30 insertions(+), 6 deletions(-)

diff --git a/lib/_emerge/DepPrioritySatisfiedRange.py 
b/lib/_emerge/DepPrioritySatisfiedRange.py
index fb0d7db4e..f546590e0 100644
--- a/lib/_emerge/DepPrioritySatisfiedRange.py
+++ b/lib/_emerge/DepPrioritySatisfiedRange.py
@@ -93,6 +93,7 @@ class DepPrioritySatisfiedRange:
ignore_medium  = _ignore_runtime
ignore_medium_soft = _ignore_satisfied_buildtime_slot_op
ignore_medium_post = _ignore_runtime_post
+   ignore_medium_post_satisifed = _ignore_satisfied_runtime_post
ignore_soft= _ignore_optional
 
 
diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 1271bda3e..0450291d4 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -8052,18 +8052,18 @@ class depgraph:

(selected_nodes[0],), noiselevel=-1)
 
if selected_nodes and ignore_priority is not None:
-   # Try to merge ignored medium_post deps as soon 
as possible
+   # Try to merge neglected medium_post deps as 
soon as possible
# if they're not satisfied by installed 
packages.
for node in selected_nodes:
children = 
set(mygraph.child_nodes(node))
-   soft = children.difference(
+   medium_post_satisifed = 
children.difference(
mygraph.child_nodes(node,
ignore_priority = \
-   
DepPrioritySatisfiedRange.ignore_soft))
+   
DepPrioritySatisfiedRange.ignore_medium_post_satisifed))
medium_post = children.difference(
mygraph.child_nodes(node,

ignore_priority=DepPrioritySatisfiedRange.ignore_medium_post))
-   medium_post -= soft
+   medium_post -= medium_post_satisifed
for child in medium_post:
if child in selected_nodes:
continue
diff --git a/lib/portage/tests/resolver/test_merge_order.py 
b/lib/portage/tests/resolver/test_merge_order.py
index f81fd2f6f..0510a0636 100644
--- a/lib/portage/tests/resolver/test_merge_order.py
+++ b/lib/portage/tests/resolver/test_merge_order.py
@@ -217,12 +217,23 @@ class MergeOrderTestCase(TestCase):
"IUSE" : "X +encode",
"RDEPEND" : "|| ( 
>=media-video/ffmpeg-0.6.90_rc0-r2[X=,encode=] 
>=media-video/libav-0.6.90_rc[X=,encode=] )",
},
+   "x11-base/xorg-drivers-1.20-r2": {
+   "EAPI": "7",
+   "IUSE": "+video_cards_fbdev",
+   "PDEPEND": "x11-base/xorg-server 
video_cards_fbdev? ( x11-drivers/xf86-video-fbdev )",
+   },
"x11-base/xorg-server-1.14.1" : {
"EAPI" : "5",
"SLOT": "0/1.14.1",
"DEPEND" : "media-libs/mesa",
"RDEPEND" : "media-libs/mesa",
+   "PDEPEND": "x11-base/xorg-drivers",
},
+   "x11-drivers/xf86-video-fbdev-0.5.0-r1": {
+   "EAPI": "7",
+   "DEPEND": "x11-base/xorg-server:=",
+

[gentoo-portage-dev] Re: [PATCH] Updates for portage-3.0.11 release

2020-12-02 Thread Zac Medico
On 12/2/20 12:18 AM, Zac Medico wrote:
> Signed-off-by: Zac Medico 
> ---
>  RELEASE-NOTES | 6 ++
>  setup.py  | 2 +-
>  2 files changed, 7 insertions(+), 1 deletion(-)

Please ignore.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] Updates for portage-3.0.11 release

2020-12-02 Thread Zac Medico
Signed-off-by: Zac Medico 
---
 RELEASE-NOTES | 6 ++
 setup.py  | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/RELEASE-NOTES b/RELEASE-NOTES
index 61c3c3d16..7fff83017 100644
--- a/RELEASE-NOTES
+++ b/RELEASE-NOTES
@@ -1,6 +1,12 @@
 Release Notes; upgrade information mainly.
 Features/major bugfixes are listed in NEWS
 
+portage-3.0.11
+==
+* Bug Fixes:
+- Bug 756961 handle dev-lang/rust[system-bootstrap] dependency cycle
+- Bug 757306 backtracking: fix virtual choices for circular deps
+
 portage-3.0.10
 ==
 * Bug Fixes:
diff --git a/setup.py b/setup.py
index 443b94422..95a53307d 100755
--- a/setup.py
+++ b/setup.py
@@ -655,7 +655,7 @@ class build_ext(_build_ext):
 
 setup(
name = 'portage',
-   version = '3.0.10',
+   version = '3.0.11',
url = 'https://wiki.gentoo.org/wiki/Project:Portage',
author = 'Gentoo Portage Development Team',
author_email = 'dev-port...@gentoo.org',
-- 
2.26.2




[gentoo-portage-dev] [PATCH] Allow a package to replace its own buildtime dependency

2020-11-28 Thread Zac Medico
If a package has a buildtime dependency on a previous version that
it will replace, then do not treat it as a slot conflict. This
solves inappropriate behavior for dev-lang/rust[system-bootstrap].

This requires adjustments to package selection logic in several
locations, in order to ensure that an installed package instance
will be selected to satisfy a buildtime dependency when
appropriate. Dependencies of the installed package will be
entirely ignored, but that has already been the case when using
installed package to break cycles, as discussed in bug 199856.

Bug: https://bugs.gentoo.org/756961
Signed-off-by: Zac Medico 
---
 lib/_emerge/depgraph.py   | 68 ++
 lib/portage/dep/dep_check.py  | 24 ---
 .../resolver/test_circular_choices_rust.py| 69 +++
 3 files changed, 139 insertions(+), 22 deletions(-)
 create mode 100644 lib/portage/tests/resolver/test_circular_choices_rust.py

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index d10474ab3..1271bda3e 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -85,6 +85,8 @@ from _emerge.resolver.output import Display, 
format_unmatched_atom
 
 # Exposes a depgraph interface to dep_check.
 _dep_check_graph_interface = 
collections.namedtuple('_dep_check_graph_interface',(
+   # Checks if parent package will replace child.
+   'will_replace_child',
# Indicates a removal action, like depclean or prune.
'removal_action',
# Checks if update is desirable for a given package.
@@ -507,6 +509,7 @@ class _dynamic_depgraph_config:
# Track missed updates caused by solved conflicts.
self._conflict_missed_update = collections.defaultdict(dict)
dep_check_iface = _dep_check_graph_interface(
+   will_replace_child=depgraph._will_replace_child,
removal_action="remove" in myparams,
want_update_pkg=depgraph._want_update_pkg,
)
@@ -3104,6 +3107,22 @@ class depgraph:

self._frozen_config.myopts,

modified_use=self._pkg_use_enabled(pkg))),

level=logging.DEBUG, noiselevel=-1)
+   elif (pkg.installed and myparent and
+   pkg.root == myparent.root and
+   pkg.slot_atom == myparent.slot_atom):
+   # If the parent package is replacing 
the child package then
+   # there's no slot conflict. Since the 
child will be replaced,
+   # do not add it to the graph. No 
attempt will be made to
+   # satisfy its dependencies, which is 
unsafe if it has any
+   # missing dependencies, as discussed in 
bug 199856.
+   if debug:
+   writemsg_level(
+   "%s%s %s\n" % ("Replace 
Child:".ljust(15),
+   pkg, 
pkg_use_display(pkg,
+   
self._frozen_config.myopts,
+   
modified_use=self._pkg_use_enabled(pkg))),
+   level=logging.DEBUG, 
noiselevel=-1)
+   return 1
 
else:
if debug:
@@ -5877,6 +5896,27 @@ class depgraph:
(arg_atoms or update) and
not self._too_deep(depth))
 
+   def _will_replace_child(self, parent, root, atom):
+   """
+   Check if a given parent package will replace a child package
+   for the given root and atom.
+
+   @param parent: parent package
+   @type parent: Package
+   @param root: child root
+   @type root: str
+   @param atom: child atom
+   @type atom: Atom
+   @rtype: Package
+   @return: child package to replace, or None
+   """
+   if parent.root != root or parent.cp != atom.cp:
+   return None
+   for child in 
self._iter_match_pkgs(self._frozen_config.roots[root], "installed", atom):
+   if parent.slot_atom == child.slot_atom:
+   return child
+   return None
+
def _too_deep(self, depth):
""

[gentoo-portage-dev] [PATCH] backtracking: fix virtual choices for circular deps (bug 757306)

2020-11-27 Thread Zac Medico
Fix virtual choices to be consistent with circular dependency
backtracking choices.

Fixes: f78a91e44e3e ("backtracking: adjust || preference to break dependency 
cycles")
Bug: https://bugs.gentoo.org/757306
Signed-off-by: Zac Medico 
---
 lib/portage/dep/dep_check.py | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dep/dep_check.py b/lib/portage/dep/dep_check.py
index 60c8ec6d0..b89d5d651 100644
--- a/lib/portage/dep/dep_check.py
+++ b/lib/portage/dep/dep_check.py
@@ -356,6 +356,7 @@ def dep_zapdeps(unreduced, reduced, myroot, use_binaries=0, 
trees=None,
 
# Alias the trees we'll be checking availability against
parent   = trees[myroot].get("parent")
+   virt_parent = trees[myroot].get("virt_parent")
priority = trees[myroot].get("priority")
graph_db = trees[myroot].get("graph_db")
graph= trees[myroot].get("graph")
@@ -596,8 +597,10 @@ def dep_zapdeps(unreduced, reduced, myroot, 
use_binaries=0, trees=None,
if 
match_from_list(atom, cpv_slot_list):
circular_atom = 
atom
break
-   else:
-   for circular_child in 
circular_dependency.get(parent, []):
+   if circular_atom is None and 
circular_dependency is not None:
+   for circular_child in itertools.chain(
+   
circular_dependency.get(parent, []),
+   
circular_dependency.get(virt_parent, [])):
for atom in 
atoms:
if not 
atom.blocker and atom.match(circular_child):

circular_atom = atom
-- 
2.26.2




[gentoo-portage-dev] [PATCH v2] find_smallest_cycle: enhance search prioritization

2020-11-21 Thread Zac Medico
Enhance the find_smallest_cycle function to prioritize its
search so that it will minimize the use of installed packages
to break cycles. When installed packages must be used to
break cycles, it will now prefer to do this for runtime
dependencies over buildtime dependencies, since it's
preferable to build against latest versions of buildtime
dependencies whenever possible. This should solve some cases
of bug 199856 which have been triggered by unsafe reliance
on installed packages to break cycles.

The included unit test case demonstrates correct merge order
for a dependency calculation involving 6 independent cycles.
This test case fails with the master branch, due to a buildtime
dependency cycle of 3 packages being merged earlier than cycles
of 2 packages. We can generalize this to say that the master
branch may use an installed package to break an arbitrarily
sized cycle in a somewhat random location, even though that
cycle may be composed of smaller independent cycles which
would be safer to break individually.

Bug: https://bugs.gentoo.org/754903
Signed-off-by: Zac Medico 
---
[PATCH v2]
* Add a unit test case which demonstrates a significant flaw
  in the master branch.
* Sort nodes in find_smallest_cycle, for deterministic results. 

 lib/_emerge/DepPriorityNormalRange.py |  2 +
 lib/_emerge/DepPrioritySatisfiedRange.py  | 52 ++-
 lib/_emerge/depgraph.py   | 43 +--
 .../tests/resolver/test_merge_order.py| 10 
 4 files changed, 66 insertions(+), 41 deletions(-)

diff --git a/lib/_emerge/DepPriorityNormalRange.py 
b/lib/_emerge/DepPriorityNormalRange.py
index 5f3f3da70..10f205a3b 100644
--- a/lib/_emerge/DepPriorityNormalRange.py
+++ b/lib/_emerge/DepPriorityNormalRange.py
@@ -14,6 +14,7 @@ class DepPriorityNormalRange:
"""
MEDIUM  = 3
MEDIUM_SOFT = 2
+   MEDIUM_POST = 2
SOFT= 1
NONE= 0
 
@@ -37,6 +38,7 @@ class DepPriorityNormalRange:
 
ignore_medium  = _ignore_runtime
ignore_medium_soft = _ignore_runtime_post
+   ignore_medium_post = _ignore_runtime_post
ignore_soft= _ignore_optional
 
 DepPriorityNormalRange.ignore_priority = (
diff --git a/lib/_emerge/DepPrioritySatisfiedRange.py 
b/lib/_emerge/DepPrioritySatisfiedRange.py
index e056e676f..df2439f1f 100644
--- a/lib/_emerge/DepPrioritySatisfiedRange.py
+++ b/lib/_emerge/DepPrioritySatisfiedRange.py
@@ -8,17 +8,18 @@ class DepPrioritySatisfiedRange:
 
not satisfied and buildtimeHARD
not satisfied and runtime  7   MEDIUM
-   not satisfied and runtime_post 6   MEDIUM_SOFT
-   satisfied and buildtime_slot_op5   SOFT
-   satisfied and buildtime4   SOFT
-   satisfied and runtime  3   SOFT
-   satisfied and runtime_post 2   SOFT
+   satisfied and buildtime_slot_op6   MEDIUM_SOFT
+   satisfied and buildtime5   MEDIUM_SOFT
+   satisfied and runtime  4   MEDIUM_SOFT
+   runtime_post   3   MEDIUM_POST
+   satisfied and runtime_post 2   MEDIUM_POST
optional   1   SOFT
(none of the above)0   NONE
"""
MEDIUM  = 7
MEDIUM_SOFT = 6
-   SOFT= 5
+   MEDIUM_POST = 3
+   SOFT= 1
NONE= 0
 
@classmethod
@@ -37,15 +38,23 @@ class DepPrioritySatisfiedRange:
return False
return bool(priority.runtime_post)
 
+   @classmethod
+   def _ignore_runtime_post(cls, priority):
+   if priority.__class__ is not DepPriority:
+   return False
+   return bool(priority.optional or priority.runtime_post)
+
@classmethod
def _ignore_satisfied_runtime(cls, priority):
if priority.__class__ is not DepPriority:
return False
if priority.optional:
return True
-   if not priority.satisfied:
+   if priority.buildtime:
return False
-   return not priority.buildtime
+   if not priority.runtime:
+   return True
+   return bool(priority.satisfied)
 
@classmethod
def _ignore_satisfied_buildtime(cls, priority):
@@ -61,37 +70,32 @@ class DepPrioritySatisfiedRange:
def _ignore_satisfied_buildtime_slot_op(cls, priority):
if priority.__class__ is not DepPriority:
return False
-   return bool(priority.optional or \
-   priority.satisfied)
-
-   @classmethod
-   def _ignore_runtime_post(cls, priority

[gentoo-portage-dev] [PATCH] find_smallest_cycle: enhance search prioritization

2020-11-19 Thread Zac Medico
Enhance the find_smallest_cycle function to prioritize its
search so that it will minimize the use of installed packages
to break cycles. When installed packages must be used to
break cycles, it will now prefer to do this for runtime
dependencies over buildtime dependencies, since it's
preferable to build against latest versions of buildtime
dependencies whenever possible. This should solve some cases
of bug 199856 which have been triggered by unsafe reliance
on installed packages to break cycles.

Bug: https://bugs.gentoo.org/754903
Signed-off-by: Zac Medico 
---
 lib/_emerge/DepPriorityNormalRange.py|  2 +
 lib/_emerge/DepPrioritySatisfiedRange.py | 52 +---
 lib/_emerge/depgraph.py  | 41 +++
 3 files changed, 54 insertions(+), 41 deletions(-)

diff --git a/lib/_emerge/DepPriorityNormalRange.py 
b/lib/_emerge/DepPriorityNormalRange.py
index 5f3f3da70..10f205a3b 100644
--- a/lib/_emerge/DepPriorityNormalRange.py
+++ b/lib/_emerge/DepPriorityNormalRange.py
@@ -14,6 +14,7 @@ class DepPriorityNormalRange:
"""
MEDIUM  = 3
MEDIUM_SOFT = 2
+   MEDIUM_POST = 2
SOFT= 1
NONE= 0
 
@@ -37,6 +38,7 @@ class DepPriorityNormalRange:
 
ignore_medium  = _ignore_runtime
ignore_medium_soft = _ignore_runtime_post
+   ignore_medium_post = _ignore_runtime_post
ignore_soft= _ignore_optional
 
 DepPriorityNormalRange.ignore_priority = (
diff --git a/lib/_emerge/DepPrioritySatisfiedRange.py 
b/lib/_emerge/DepPrioritySatisfiedRange.py
index e056e676f..df2439f1f 100644
--- a/lib/_emerge/DepPrioritySatisfiedRange.py
+++ b/lib/_emerge/DepPrioritySatisfiedRange.py
@@ -8,17 +8,18 @@ class DepPrioritySatisfiedRange:
 
not satisfied and buildtimeHARD
not satisfied and runtime  7   MEDIUM
-   not satisfied and runtime_post 6   MEDIUM_SOFT
-   satisfied and buildtime_slot_op5   SOFT
-   satisfied and buildtime4   SOFT
-   satisfied and runtime  3   SOFT
-   satisfied and runtime_post 2   SOFT
+   satisfied and buildtime_slot_op6   MEDIUM_SOFT
+   satisfied and buildtime5   MEDIUM_SOFT
+   satisfied and runtime  4   MEDIUM_SOFT
+   runtime_post   3   MEDIUM_POST
+   satisfied and runtime_post 2   MEDIUM_POST
optional   1   SOFT
(none of the above)0   NONE
"""
MEDIUM  = 7
MEDIUM_SOFT = 6
-   SOFT= 5
+   MEDIUM_POST = 3
+   SOFT= 1
NONE= 0
 
@classmethod
@@ -37,15 +38,23 @@ class DepPrioritySatisfiedRange:
return False
return bool(priority.runtime_post)
 
+   @classmethod
+   def _ignore_runtime_post(cls, priority):
+   if priority.__class__ is not DepPriority:
+   return False
+   return bool(priority.optional or priority.runtime_post)
+
@classmethod
def _ignore_satisfied_runtime(cls, priority):
if priority.__class__ is not DepPriority:
return False
if priority.optional:
return True
-   if not priority.satisfied:
+   if priority.buildtime:
return False
-   return not priority.buildtime
+   if not priority.runtime:
+   return True
+   return bool(priority.satisfied)
 
@classmethod
def _ignore_satisfied_buildtime(cls, priority):
@@ -61,37 +70,32 @@ class DepPrioritySatisfiedRange:
def _ignore_satisfied_buildtime_slot_op(cls, priority):
if priority.__class__ is not DepPriority:
return False
-   return bool(priority.optional or \
-   priority.satisfied)
-
-   @classmethod
-   def _ignore_runtime_post(cls, priority):
-   if priority.__class__ is not DepPriority:
-   return False
-   return bool(priority.optional or \
-   priority.satisfied or \
-   priority.runtime_post)
+   if priority.optional:
+   return True
+   if priority.satisfied:
+   return True
+   return not priority.buildtime and not priority.runtime
 
@classmethod
def _ignore_runtime(cls, priority):
if priority.__class__ is not DepPriority:
return False
-   return bool(priority.satisfied or \
-   priority.optional or \
-   

[gentoo-portage-dev] [PATCH] emerge: Disable profile deprecation warning inheritance (bug 753497)

2020-11-08 Thread Zac Medico
According to PMS, a deprecated profile warning is not inherited. Since
the current profile node may have been inherited by a user profile
node, the deprecation warning may be relevant even if it is not a
top-level profile node. Therefore, consider the deprecated warning
to be irrelevant when the current profile node belongs to the same
repo as the previous profile node.

Bug: https://bugs.gentoo.org/753497
Signed-off-by: Zac Medico 
---
 .../ebuild/_config/LocationsManager.py| 30 ++-
 .../ebuild/deprecated_profile_check.py|  9 +++---
 2 files changed, 27 insertions(+), 12 deletions(-)

diff --git a/lib/portage/package/ebuild/_config/LocationsManager.py 
b/lib/portage/package/ebuild/_config/LocationsManager.py
index b90b9227c..28740b355 100644
--- a/lib/portage/package/ebuild/_config/LocationsManager.py
+++ b/lib/portage/package/ebuild/_config/LocationsManager.py
@@ -30,7 +30,9 @@ _PORTAGE1_DIRECTORIES = frozenset([
 
 _profile_node = collections.namedtuple('_profile_node',
('location', 'portage1_directories', 'user_config',
-   'profile_formats', 'eapi', 'allow_build_id'))
+   'profile_formats', 'eapi', 'allow_build_id',
+   'show_deprecated_warning',
+))
 
 _allow_parent_colon = frozenset(
["portage-2"])
@@ -132,7 +134,7 @@ class LocationsManager:
if self.profile_path:
try:

self._addProfile(os.path.realpath(self.profile_path),
-   repositories, known_repos)
+   repositories, known_repos, ())
except ParseError as e:
if not portage._sync_mode:
writemsg(_("!!! Unable to parse 
profile: '%s'\n") % self.profile_path, noiselevel=-1)
@@ -154,7 +156,9 @@ class LocationsManager:
('profile-bashrcs', 'profile-set'),
read_corresponding_eapi_file(
custom_prof + os.sep, default=None),
-   True))
+   True,
+   show_deprecated_warning=False,
+   ))
del custom_prof
 
self.profiles = tuple(self.profiles)
@@ -167,7 +171,7 @@ class LocationsManager:
noiselevel=-1)
raise DirectoryNotFound(var)
 
-   def _addProfile(self, currentPath, repositories, known_repos):
+   def _addProfile(self, currentPath, repositories, known_repos, 
previous_repos):
current_abs_path = os.path.abspath(currentPath)
allow_directories = True
allow_parent_colon = True
@@ -176,8 +180,8 @@ class LocationsManager:
current_formats = ()
eapi = None
 
-   intersecting_repos = [x for x in known_repos
-   if current_abs_path.startswith(x[0])]
+   intersecting_repos = tuple(x for x in known_repos
+   if current_abs_path.startswith(x[0]))
if intersecting_repos:
# Handle nested repositories. The longest path
# will be the correct one.
@@ -214,6 +218,14 @@ class LocationsManager:
for x in layout_data['profile-formats'])
current_formats = tuple(layout_data['profile-formats'])
 
+   # According to PMS, a deprecated profile warning is not 
inherited. Since
+   # the current profile node may have been inherited by a user 
profile
+   # node, the deprecation warning may be relevant even if it is 
not a
+   # top-level profile node. Therefore, consider the deprecated 
warning
+   # to be irrelevant when the current profile node belongs to the 
same
+   # repo as the previous profile node.
+   show_deprecated_warning = (not previous_repos or
+   tuple(x[0] for x in previous_repos) != tuple(x[0] for x 
in intersecting_repos))
 
if compat_mode:
offenders = 
_PORTAGE1_DIRECTORIES.intersection(os.listdir(currentPath))
@@ -256,7 +268,7 @@ class LocationsManager:
parentPath = 
os.path.realpath(parentPath)
 
if exists_raise_eaccess(parentPath):
-   self._addProfile(parentPath, 
repositories, known_repos)
+   self._addProfile(parentPath, 
repositories, known_repos, intersecting_repos)
else:
raise ParseError(
_("Parent '%s' not found: 
'%s'") 

[gentoo-portage-dev] [PATCH] make.conf: expand special *ROOT variables (bug 752147)

2020-11-01 Thread Zac Medico
Bug: https://bugs.gentoo.org/752147
Signed-off-by: Zac Medico 
---
 lib/portage/package/ebuild/config.py | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/portage/package/ebuild/config.py 
b/lib/portage/package/ebuild/config.py
index a09fdbced..3be8f9f6d 100644
--- a/lib/portage/package/ebuild/config.py
+++ b/lib/portage/package/ebuild/config.py
@@ -401,9 +401,14 @@ class config:
expand_map = env_d.copy()
self._expand_map = expand_map
 
-   # Allow make.globals to set default paths relative to 
${EPREFIX}.
+   # Allow make.globals and make.conf to set default paths 
relative to vars like ${EPREFIX}.
+   expand_map["BROOT"] = broot
expand_map["EPREFIX"] = eprefix
+   expand_map["EROOT"] = eroot
+   expand_map["ESYSROOT"] = esysroot
expand_map["PORTAGE_CONFIGROOT"] = config_root
+   expand_map["ROOT"] = target_root
+   expand_map["SYSROOT"] = sysroot
 
if portage._not_installed:
make_globals_path = 
os.path.join(PORTAGE_BASE_PATH, "cnf", "make.globals")
-- 
2.26.2




[gentoo-portage-dev] [PATCH] emerge: add --quickpkg-direct-root option

2020-11-01 Thread Zac Medico
Specify the root to use as the --quickpkg-direct package source. This
root is assumed to be immutable during the entire emerge operation.
The default is set to "/".

Bug: https://bugs.gentoo.org/752066
Signed-off-by: Zac Medico 
---
 lib/_emerge/actions.py  | 19 ---
 lib/_emerge/depgraph.py | 11 +--
 lib/_emerge/main.py |  5 +
 lib/portage/tests/emerge/test_simple.py |  3 ++-
 man/emerge.1| 10 --
 5 files changed, 40 insertions(+), 8 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 5e8a46957..2e155899c 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -49,7 +49,7 @@ from portage.package.ebuild._ipc.QueryCommand import 
QueryCommand
 from portage.package.ebuild.fetch import _hide_url_passwd
 from portage._sets import load_default_config, SETPREFIX
 from portage._sets.base import InternalPackageSet
-from portage.util import cmp_sort_key, writemsg, varexpand, \
+from portage.util import cmp_sort_key, normalize_path, writemsg, varexpand, \
writemsg_level, writemsg_stdout
 from portage.util.digraph import digraph
 from portage.util.SlotObject import SlotObject
@@ -106,13 +106,26 @@ def action_build(emerge_config, trees=DeprecationWarning,
# before we get here, so warn if they're not (bug #267103).
chk_updated_cfg_files(settings['EROOT'], ['/etc/portage'])
 
+   quickpkg_root = normalize_path(os.path.abspath(
+   emerge_config.opts.get('--quickpkg-direct-root',
+   
emerge_config.running_config.settings['ROOT']))).rstrip(os.path.sep) + 
os.path.sep
quickpkg_direct = ("--usepkg" in emerge_config.opts and
emerge_config.opts.get('--quickpkg-direct', 'n') == 'y' and
-   emerge_config.target_config is not emerge_config.running_config)
+   emerge_config.target_config.settings['ROOT'] != quickpkg_root)
if '--getbinpkg' in emerge_config.opts or quickpkg_direct:
kwargs = {}
if quickpkg_direct:
-   kwargs['add_repos'] = 
(emerge_config.running_config.trees['vartree'].dbapi,)
+   if quickpkg_root == emerge_config.running_config.root:
+   quickpkg_vardb = 
emerge_config.running_config.trees['vartree'].dbapi
+   else:
+   quickpkg_settings = portage.config(
+   
config_root=emerge_config.target_config.settings['PORTAGE_CONFIGROOT'],
+   target_root=quickpkg_root,
+   
env=emerge_config.target_config.settings.backupenv,
+   
sysroot=emerge_config.target_config.settings['SYSROOT'],
+   
eprefix=emerge_config.target_config.settings['EPREFIX'])
+   quickpkg_vardb = 
portage.vartree(settings=quickpkg_settings).dbapi
+   kwargs['add_repos'] = (quickpkg_vardb,)
 
try:
emerge_config.target_config.trees['bintree'].populate(
diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 0bb0352e7..898cf6c1a 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -41,7 +41,7 @@ from portage._sets import SETPREFIX
 from portage._sets.base import InternalPackageSet
 from portage.util import ConfigProtect, shlex_split, new_protect_filename
 from portage.util import cmp_sort_key, writemsg, writemsg_stdout
-from portage.util import ensure_dirs
+from portage.util import ensure_dirs, normalize_path
 from portage.util import writemsg_level, write_atomic
 from portage.util.digraph import digraph
 from portage.util.futures import asyncio
@@ -4567,8 +4567,15 @@ class depgraph:
self._dynamic_config._skip_restart = True
return False, myfavorites
 
+   # Since --quickpkg-direct assumes that --quickpkg-direct-root is
+   # immutable, assert that there are no merged or unmerge tasks
+   # for --quickpkg-direct-root.
+   quickpkg_root = normalize_path(os.path.abspath(
+   self._frozen_config.myopts.get('--quickpkg-direct-root',
+   
self._frozen_config._running_root.settings['ROOT']))).rstrip(os.path.sep) + 
os.path.sep
if (self._frozen_config.myopts.get('--quickpkg-direct', 'n') == 
'y' and
-   self._frozen_config.target_root is not 
self._frozen_config._running_root):
+   self._frozen_config.settings['ROOT'] != quickpkg_root 
and
+   self._frozen_config._running_root.settings['ROOT'] == 
quickpkg_root):
running_root = self._frozen_config._running_root.root

[gentoo-portage-dev] [PATCH] pid-sandbox: Forward SIGTSTP and SIGCONT (bug 704498)

2020-10-28 Thread Zac Medico
For correct operation of Ctrl+Z, forward SIGTSTP and SIGCONT
to all sandboxed pids.

Fixes: 37e4dc5ae842 ("pid-sandbox: pid-ns-init setsid support (bug 675870)")
Bug: https://bugs.gentoo.org/704498
Signed-off-by: Zac Medico 
---
 bin/pid-ns-init | 24 +++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/bin/pid-ns-init b/bin/pid-ns-init
index 3a218a5df..e410dd028 100644
--- a/bin/pid-ns-init
+++ b/bin/pid-ns-init
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-# Copyright 2018-2019 Gentoo Authors
+# Copyright 2018-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import errno
@@ -19,6 +19,11 @@ KILL_SIGNALS = (
signal.SIGHUP,
 )
 
+SIGTSTP_SIGCONT = (
+   signal.SIGTSTP,
+   signal.SIGCONT,
+)
+
 
 def forward_kill_signal(pid, signum, frame):
if pid == 0:
@@ -28,6 +33,18 @@ def forward_kill_signal(pid, signum, frame):
os.kill(pid, signum)
 
 
+def forward_sigtstp_sigcont(pid, signum, frame):
+   handler = None
+   if pid == 0:
+   # Temporarily disable the handler in order to prevent it from
+   # being called recursively, since the signal will also be sent
+   # to the current process.
+   handler = signal.signal(signum, signal.SIG_DFL)
+   os.kill(pid, signum)
+   if handler is not None:
+   signal.signal(signum, handler)
+
+
 def preexec_fn(uid, gid, groups, umask):
if gid is not None:
os.setgid(gid)
@@ -97,6 +114,11 @@ def main(argv):
for signum in KILL_SIGNALS:
signal.signal(signum, sig_handler)
 
+   # For correct operation of Ctrl+Z, forward SIGTSTP and SIGCONT.
+   sigtstp_sigcont_handler = functools.partial(forward_sigtstp_sigcont, 0 
if setsid else main_child_pid)
+   for signum in SIGTSTP_SIGCONT:
+   signal.signal(signum, sigtstp_sigcont_handler)
+
# wait for child processes
while True:
try:
-- 
2.26.2




[gentoo-portage-dev] [PATCH v2] emerge: enable parallel-fetch during pkg_pretend (bug 710432)

2020-09-20 Thread Zac Medico
Execute pkg_pretend phases in a coroutine while parallel-fetch
is running concurrently. When it's time to execute the pkg_pretend
phase for a remote binary package, use a Scheduler _get_prefetcher
method to get a running prefetcher if available, and otherwise
start a new fetcher.

Since pkg_pretend phases now run inside of the --keep-going retry
loop, --keep-going is now able to recover from pkg_pretend
failures, which fixes bug 404157.

Bug: https://bugs.gentoo.org/404157
Bug: https://bugs.gentoo.org/710432
Signed-off-by: Zac Medico 
---
[PATCH v2] records failed packages for correct interaction with
emerge --keep-going, which fixes bug 404157

 lib/_emerge/Scheduler.py | 142 +++
 1 file changed, 99 insertions(+), 43 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index a69421288..465f928a0 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -25,6 +25,7 @@ from portage._sets import SETPREFIX
 from portage._sets.base import InternalPackageSet
 from portage.util import ensure_dirs, writemsg, writemsg_level
 from portage.util.futures import asyncio
+from portage.util.futures.compat_coroutine import coroutine, coroutine_return
 from portage.util.SlotObject import SlotObject
 from portage.util._async.SchedulerInterface import SchedulerInterface
 from portage.package.ebuild.digestcheck import digestcheck
@@ -766,7 +767,8 @@ class Scheduler(PollScheduler):
 
return prefetcher
 
-   def _run_pkg_pretend(self):
+   @coroutine
+   def _run_pkg_pretend(self, loop=None):
"""
Since pkg_pretend output may be important, this method sends all
output directly to stdout (regardless of options like --quiet or
@@ -774,7 +776,7 @@ class Scheduler(PollScheduler):
"""
 
failures = 0
-   sched_iface = self._sched_iface
+   sched_iface = loop = asyncio._wrap_loop(loop or 
self._sched_iface)
 
for x in self._mergelist:
if not isinstance(x, Package):
@@ -789,18 +791,28 @@ class Scheduler(PollScheduler):
if "pretend" not in x.defined_phases:
continue
 
-   out_str =">>> Running pre-merge checks for " + 
colorize("INFORM", x.cpv) + "\n"
-   portage.util.writemsg_stdout(out_str, noiselevel=-1)
+   out_str = "Running pre-merge checks for " + 
colorize("INFORM", x.cpv)
+   self._status_msg(out_str)
 
root_config = x.root_config
-   settings = self.pkgsettings[root_config.root]
+   settings = self._allocate_config(root_config.root)
settings.setcpv(x)
+   if not x.built:
+   # Get required SRC_URI metadata (it's not 
cached in x.metadata
+   # because some packages have an extremely large 
SRC_URI value).
+   portdb = root_config.trees["porttree"].dbapi
+   (settings.configdict["pkg"]["SRC_URI"],) = 
yield portdb.async_aux_get(
+   x.cpv, ["SRC_URI"], myrepo=x.repo, 
loop=loop
+   )
 
# setcpv/package.env allows for per-package 
PORTAGE_TMPDIR so we
# have to validate it for each package
rval = _check_temp_dir(settings)
if rval != os.EX_OK:
-   return rval
+   failures += 1
+   self._record_pkg_failure(x, settings, FAILURE)
+   self._deallocate_config(settings)
+   continue
 
build_dir_path = os.path.join(
os.path.realpath(settings["PORTAGE_TMPDIR"]),
@@ -809,7 +821,7 @@ class Scheduler(PollScheduler):
settings["PORTAGE_BUILDDIR"] = build_dir_path
build_dir = EbuildBuildDir(scheduler=sched_iface,
settings=settings)
-   sched_iface.run_until_complete(build_dir.async_lock())
+   yield build_dir.async_lock()
current_task = None
 
try:
@@ -835,7 +847,7 @@ class Scheduler(PollScheduler):
phase='clean', 
scheduler=sched_iface, settings=settings)
current_task = clean_phase
clean_phase.start()
-  

[gentoo-portage-dev] [PATCH] emerge: enable parallel-fetch during pkg_pretend (bug 710432)

2020-09-19 Thread Zac Medico
Execute pkg_pretend phases in a coroutine while parallel-fetch
is running concurrently. When it's time to execute the pkg_pretend
phase for a remote binary package, use a Scheduler _get_prefetcher
method to get a running prefetcher if available, and otherwise
start a new fetcher.

Bug: https://bugs.gentoo.org/710432
Signed-off-by: Zac Medico 
---
 lib/_emerge/Scheduler.py | 94 +---
 1 file changed, 58 insertions(+), 36 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index a69421288..20884986f 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -25,6 +25,7 @@ from portage._sets import SETPREFIX
 from portage._sets.base import InternalPackageSet
 from portage.util import ensure_dirs, writemsg, writemsg_level
 from portage.util.futures import asyncio
+from portage.util.futures.compat_coroutine import coroutine, coroutine_return
 from portage.util.SlotObject import SlotObject
 from portage.util._async.SchedulerInterface import SchedulerInterface
 from portage.package.ebuild.digestcheck import digestcheck
@@ -766,7 +767,8 @@ class Scheduler(PollScheduler):
 
return prefetcher
 
-   def _run_pkg_pretend(self):
+   @coroutine
+   def _run_pkg_pretend(self, loop=None):
"""
Since pkg_pretend output may be important, this method sends all
output directly to stdout (regardless of options like --quiet or
@@ -774,7 +776,7 @@ class Scheduler(PollScheduler):
"""
 
failures = 0
-   sched_iface = self._sched_iface
+   sched_iface = loop = asyncio._wrap_loop(loop or 
self._sched_iface)
 
for x in self._mergelist:
if not isinstance(x, Package):
@@ -795,12 +797,18 @@ class Scheduler(PollScheduler):
root_config = x.root_config
settings = self.pkgsettings[root_config.root]
settings.setcpv(x)
+   if not x.built:
+   # Get required SRC_URI metadata (it's not 
cached in x.metadata
+   # because some packages have an extremely large 
SRC_URI value).
+   portdb = root_config.trees["porttree"].dbapi
+   settings.configdict["pkg"]["SRC_URI"], = (yield 
portdb.async_aux_get(
+   x.cpv, ["SRC_URI"], myrepo=x.repo, 
loop=loop))
 
# setcpv/package.env allows for per-package 
PORTAGE_TMPDIR so we
# have to validate it for each package
rval = _check_temp_dir(settings)
if rval != os.EX_OK:
-   return rval
+   coroutine_return(rval)
 
build_dir_path = os.path.join(
os.path.realpath(settings["PORTAGE_TMPDIR"]),
@@ -809,7 +817,7 @@ class Scheduler(PollScheduler):
settings["PORTAGE_BUILDDIR"] = build_dir_path
build_dir = EbuildBuildDir(scheduler=sched_iface,
settings=settings)
-   sched_iface.run_until_complete(build_dir.async_lock())
+   yield build_dir.async_lock()
current_task = None
 
try:
@@ -835,7 +843,7 @@ class Scheduler(PollScheduler):
phase='clean', 
scheduler=sched_iface, settings=settings)
current_task = clean_phase
clean_phase.start()
-   clean_phase.wait()
+   yield clean_phase.async_wait()
 
if x.built:
tree = "bintree"
@@ -845,10 +853,11 @@ class Scheduler(PollScheduler):
# Display fetch on stdout, so that it's 
always clear what
# is consuming time here.
if bintree.isremote(x.cpv):
-   fetcher = BinpkgFetcher(pkg=x,
-   scheduler=sched_iface)
-   fetcher.start()
-   if fetcher.wait() != os.EX_OK:
+   fetcher = 
self._get_prefetcher(x)
+   if fetcher is None:
+   fetcher = 
BinpkgFetcher(pkg=x, scheduler=loop)
+   

[gentoo-portage-dev] [PATCH] _slot_confict_backtrack: group similar missed updates (bug 743115)

2020-09-19 Thread Zac Medico
When a slot conflict occurs due to a missed update, and some other
similar update(s) are available, add the similar update(s) to the
runtime package mask for the same backtracking choice. This reduces
minimum number of backtrack tries required to solve the test case
for bug 743115 from 7 to 4, where the difference of 3 corresponds
to the number of other similar setuptools updates available.

Bug: https://bugs.gentoo.org/743115
Signed-off-by: Zac Medico 
---
 lib/_emerge/depgraph.py   | 25 ---
 lib/_emerge/resolver/backtracking.py  |  7 +++---
 .../test_slot_operator_missed_update.py   |  2 +-
 3 files changed, 26 insertions(+), 8 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 7281d8692..2a840b2ca 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -1795,15 +1795,32 @@ class depgraph:

self._dynamic_config._parent_atoms.get(to_be_masked, set())
conflict_atoms = set(parent_atom for parent_atom in 
all_parents \
if parent_atom not in parent_atoms)
-   backtrack_data.append((to_be_masked, conflict_atoms))
+
+   similar_pkgs = []
+   if conflict_atoms:
+   # If the conflict has been triggered by a 
missed update, then
+   # we can avoid excessive backtracking if we 
detect similar missed
+   # updates and mask them as part of the same 
backtracking choice.
+   for similar_pkg in 
self._iter_similar_available(to_be_masked, slot_atom):
+   if similar_pkg in conflict_pkgs:
+   continue
+   similar_conflict_atoms = []
+   for parent_atom in conflict_atoms:
+   parent, atom = parent_atom
+   if not atom.match(similar_pkg):
+   
similar_conflict_atoms.append(parent_atom)
+   if similar_conflict_atoms:
+   
similar_pkgs.append((similar_pkg, set(similar_conflict_atoms)))
+   similar_pkgs.append((to_be_masked, conflict_atoms))
+   backtrack_data.append(tuple(similar_pkgs))
 
# Prefer choices that minimize conflict atoms. This is intended
# to take precedence over the earlier package version sort. The
# package version sort is still needed or else choices for the
# testOverlapSlotConflict method of 
VirtualMinimizeChildrenTestCase
# become non-deterministic.
-   backtrack_data.sort(key=lambda item: len(item[1]))
-   to_be_masked = backtrack_data[-1][0]
+   backtrack_data.sort(key=lambda similar_pkgs: max(len(item[1]) 
for item in similar_pkgs))
+   to_be_masked = [item[0] for item in backtrack_data[-1]]
 
self._dynamic_config._backtrack_infos.setdefault(
"slot conflict", []).append(backtrack_data)
@@ -1814,7 +1831,7 @@ class depgraph:
"",
"backtracking due to slot conflict:",
"   first package:  %s" % existing_node,
-   "  package to mask: %s" % to_be_masked,
+   "  package(s) to mask: %s" % str(to_be_masked),
"  slot: %s" % slot_atom,
"   parents: %s" % ", ".join(
"(%s, '%s')" % (ppkg, atom) for ppkg, 
atom in all_parents
diff --git a/lib/_emerge/resolver/backtracking.py 
b/lib/_emerge/resolver/backtracking.py
index bc3fb3206..ca94623ac 100644
--- a/lib/_emerge/resolver/backtracking.py
+++ b/lib/_emerge/resolver/backtracking.py
@@ -166,13 +166,14 @@ class Backtracker:
self._feedback_slot_conflict(conflicts_data[0])
 
def _feedback_slot_conflict(self, conflict_data):
-   for pkg, parent_atoms in conflict_data:
+   for similar_pkgs in conflict_data:
new_node = copy.deepcopy(self._current_node)
new_node.depth += 1
new_node.mask_steps += 1
new_node.terminal = False
-   new_node.parameter.runtime_pkg_mask.setdefault(
-   pkg, {})["slot conflict"] = parent_atoms
+   for pkg, parent_atoms in similar_pkgs:
+   

[gentoo-portage-dev] Re: [PATCH] _slot_confict_backtrack: minimize conflict atoms (bug 743631)

2020-09-19 Thread Zac Medico
On 9/19/20 2:40 PM, Zac Medico wrote:
> diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
> index 3f864aefc..40e7d1325 100644
> --- a/lib/_emerge/depgraph.py
> +++ b/lib/_emerge/depgraph.py
> @@ -1797,6 +1797,12 @@ class depgraph:
>   if parent_atom not in parent_atoms)
>   backtrack_data.append((to_be_masked, conflict_atoms))
>  
> + # Prefer choices that minimize conflict atoms. This is intended
> + # to take precedence over the earlier package version sort. The
> + # package version sort is still needed or else the
> + # testOverlapSlotConflict method of 
> VirtualMinimizeChildrenTestCase
> + # will not succeed reliably with the default backtrack limit.

Updated this comment to indicate that the package version sort sort is actually
needed for deterministic results:

# Prefer choices that minimize conflict atoms. This is intended
# to take precedence over the earlier package version sort. The
# package version sort is still needed or else choices for the
# testOverlapSlotConflict method of 
VirtualMinimizeChildrenTestCase
# become non-deterministic.

-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] _slot_confict_backtrack: minimize conflict atoms (bug 743631)

2020-09-19 Thread Zac Medico
Prefer choices that minimize conflict atoms, so that choices
which satisfy all parents are preferred. This reduces the
minimum necessary backtrack tries from 21 to 7 for the unit
test related to bug 743115.

Bug: https://bugs.gentoo.org/743115
Bug: https://bugs.gentoo.org/743631
Signed-off-by: Zac Medico 
---
 lib/_emerge/depgraph.py | 6 ++
 .../tests/resolver/test_slot_operator_missed_update.py  | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 3f864aefc..40e7d1325 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -1797,6 +1797,12 @@ class depgraph:
if parent_atom not in parent_atoms)
backtrack_data.append((to_be_masked, conflict_atoms))
 
+   # Prefer choices that minimize conflict atoms. This is intended
+   # to take precedence over the earlier package version sort. The
+   # package version sort is still needed or else the
+   # testOverlapSlotConflict method of 
VirtualMinimizeChildrenTestCase
+   # will not succeed reliably with the default backtrack limit.
+   backtrack_data.sort(key=lambda item: len(item[1]))
to_be_masked = backtrack_data[-1][0]
 
self._dynamic_config._backtrack_infos.setdefault(
diff --git a/lib/portage/tests/resolver/test_slot_operator_missed_update.py 
b/lib/portage/tests/resolver/test_slot_operator_missed_update.py
index fce012f62..1ea701003 100644
--- a/lib/portage/tests/resolver/test_slot_operator_missed_update.py
+++ b/lib/portage/tests/resolver/test_slot_operator_missed_update.py
@@ -90,7 +90,7 @@ class BacktrackMissedUpdateTestCase(TestCase):
# Bug 743115: missed updates trigger excessive 
backtracking
ResolverPlaygroundTestCase(
[">=dev-python/pypy3-7.3.2_rc", "@world"],
-   options={"--update": True, "--deep": True, 
"--backtrack": 25},
+   options={"--update": True, "--deep": True, 
"--backtrack": 10},
success=True,
mergelist=[
"dev-python/pypy3-7.3.2_rc2_p37-r1",
-- 
2.25.3




  1   2   3   4   5   6   7   8   9   10   >