[gentoo-portage-dev] [PATCH] emerge: add --quickpkg-direct option

2019-11-25 Thread Zac Medico
Enable use of installed packages directly as binary
packages. This is similar to using binary packages produced by
quickpkg(1), but installed packages are used directly as though
they are binary packages. This option only works in combination
with the --root=DIR option, and it comes with the caveat that
packages are only allowed to be installed into the root that
is specified by the --root=DIR option. The other root which
serves as a source of packages is assumed to be immutabled
during the entire operation (similar to --buildpkgonly mode).

Default behavior for handling of protected configuration files
is controlled by the QUICKPKG_DEFAULT_OPTS variable. When a
configuration file is not included because it is protected, an
ewarn message is logged.

Suggested use cases:

* Install packages from a buildtime container into an empty root,
  in order to create a minimal runtime container (which need not
  include a package manager). In a multi-stage Dockerfile, install
  runtime files to an empty directory in the build stage, and in
  the final stage use COPY to populate a container with the
  contents of that directory. For greater efficiency, use buildah
  to install directly into a mounted container, avoiding the COPY
  step. Use the emerge --usepkgonly and --ignore-soname-deps=n
  options to account for soname dependencies, allowing implicit
  system dependencies such as glibc to be automatically pulled
  into the runtime image.

* Enable a live usb, iso, or pxe image to act as a binary
  installer that uses packages installed in the live image as a
  source of binary packages.

Bug: https://bugs.gentoo.org/699986
Signed-off-by: Zac Medico 
---
 lib/_emerge/Binpkg.py   |  59 +++--
 lib/_emerge/Scheduler.py|   5 +-
 lib/_emerge/actions.py  |  34 +++-
 lib/_emerge/depgraph.py |  19 
 lib/_emerge/main.py |   5 ++
 lib/portage/dbapi/__init__.py   |   3 +-
 lib/portage/dbapi/bintree.py| 110 ++--
 lib/portage/dbapi/vartree.py| 110 +++-
 lib/portage/tests/emerge/test_simple.py |   1 +
 man/emerge.1|  17 
 10 files changed, 326 insertions(+), 37 deletions(-)

diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
index f9cffa26d..b5a69f8e7 100644
--- a/lib/_emerge/Binpkg.py
+++ b/lib/_emerge/Binpkg.py
@@ -7,7 +7,6 @@ import _emerge.emergelog
 from _emerge.EbuildPhase import EbuildPhase
 from _emerge.BinpkgFetcher import BinpkgFetcher
 from _emerge.BinpkgEnvExtractor import BinpkgEnvExtractor
-from _emerge.BinpkgExtractorAsync import BinpkgExtractorAsync
 from _emerge.CompositeTask import CompositeTask
 from _emerge.BinpkgVerifier import BinpkgVerifier
 from _emerge.EbuildMerge import EbuildMerge
@@ -16,6 +15,7 @@ from _emerge.SpawnProcess import SpawnProcess
 from portage.eapi import eapi_exports_replace_vars
 from portage.util import ensure_dirs
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
+from portage.util.futures.compat_coroutine import coroutine
 import portage
 from portage import os
 from portage import shutil
@@ -135,11 +135,14 @@ class Binpkg(CompositeTask):
 
pkg = self.pkg
pkg_count = self.pkg_count
-   fetcher = BinpkgFetcher(background=self.background,
-   logfile=self.settings.get("PORTAGE_LOG_FILE"), 
pkg=self.pkg,
-   pretend=self.opts.pretend, scheduler=self.scheduler)
+   fetcher = None
 
if self.opts.getbinpkg and self._bintree.isremote(pkg.cpv):
+
+   fetcher = BinpkgFetcher(background=self.background,
+   logfile=self.settings.get("PORTAGE_LOG_FILE"), 
pkg=self.pkg,
+   pretend=self.opts.pretend, 
scheduler=self.scheduler)
+
msg = " --- (%s of %s) Fetching Binary (%s::%s)" %\
(pkg_count.curval, pkg_count.maxval, pkg.cpv,
fetcher.pkg_path)
@@ -160,7 +163,7 @@ class Binpkg(CompositeTask):
 
# The fetcher only has a returncode when
# --getbinpkg is enabled.
-   if fetcher.returncode is not None:
+   if fetcher is not None:
self._fetched_pkg = fetcher.pkg_path
if self._default_exit(fetcher) != os.EX_OK:

self._async_unlock_builddir(returncode=self.returncode)
@@ -209,7 +212,8 @@ class Binpkg(CompositeTask):
 
# This gives bashrc users an opportunity to do various things
# such as remove binary packages after they're installed.
-   self.settings["PORTAGE_BINPKG_FILE"] = pkg_path
+   if pkg_path is not None:
+   self.settings["PORTAGE_BINPKG_FILE"] = pkg_path
self._pkg_path

Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.

2019-11-25 Thread Zac Medico
On 11/25/19 5:03 AM, Ulrich Müller wrote:
> We generally try to have verbose build logs, e.g., by calling
> configure with --disable-silent-rules. Silencing patch contradicts
> this, and will suppress reporting of fuzz factors.
> 
> Note that the eapply specification in PMS calls patch without -s:
> https://projects.gentoo.org/pms/7/pms.html#x1-127001r1
> Traditionally, the -s option wasn't used by epatch either.
> 
> Bug: https://bugs.gentoo.org/674562
> Signed-off-by: Ulrich Müller 
> ---
>  bin/phase-helpers.sh | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
> index b53d39650..60f8d3243 100644
> --- a/bin/phase-helpers.sh
> +++ b/bin/phase-helpers.sh
> @@ -993,10 +993,9 @@ if ___eapi_has_eapply; then
>   ebegin "${prefix:-Applying }${f##*/}"
>   # -p1 as a sane default
>   # -f to avoid interactivity
> - # -s to silence progress output
>   # -g0 to guarantee no VCS interaction
>   # --no-backup-if-mismatch not to pollute the sources
> - ${patch_cmd} -p1 -f -s -g0 --no-backup-if-mismatch \
> + ${patch_cmd} -p1 -f -g0 --no-backup-if-mismatch \
>   "${patch_options[@]}" < "${f}"
>   failed=${?}
>   if ! eend "${failed}"; then
> 

Looks good. Please merge.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.

2019-11-25 Thread Ulrich Müller
We generally try to have verbose build logs, e.g., by calling
configure with --disable-silent-rules. Silencing patch contradicts
this, and will suppress reporting of fuzz factors.

Note that the eapply specification in PMS calls patch without -s:
https://projects.gentoo.org/pms/7/pms.html#x1-127001r1
Traditionally, the -s option wasn't used by epatch either.

Bug: https://bugs.gentoo.org/674562
Signed-off-by: Ulrich Müller 
---
 bin/phase-helpers.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index b53d39650..60f8d3243 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -993,10 +993,9 @@ if ___eapi_has_eapply; then
ebegin "${prefix:-Applying }${f##*/}"
# -p1 as a sane default
# -f to avoid interactivity
-   # -s to silence progress output
# -g0 to guarantee no VCS interaction
# --no-backup-if-mismatch not to pollute the sources
-   ${patch_cmd} -p1 -f -s -g0 --no-backup-if-mismatch \
+   ${patch_cmd} -p1 -f -g0 --no-backup-if-mismatch \
"${patch_options[@]}" < "${f}"
failed=${?}
if ! eend "${failed}"; then
-- 
2.24.0


signature.asc
Description: PGP signature