Re: [gentoo-portage-dev] [PATCH 2/4] portage.eapi: use tuple instead of str for namedtuple definition

2022-03-08 Thread Alec Warner
Looks merge-able.

On Wed, Feb 23, 2022 at 8:15 PM Matt Turner  wrote:
>
> From: "Wolfgang E. Sanyer" 
>
> Reviewed-by: Matt Turner 
> Signed-off-by: Wolfgang E. Sanyer 
> ---
>  lib/portage/eapi.py | 52 -
>  1 file changed, 33 insertions(+), 19 deletions(-)
>
> diff --git a/lib/portage/eapi.py b/lib/portage/eapi.py
> index adee87d00..18069b04b 100644
> --- a/lib/portage/eapi.py
> +++ b/lib/portage/eapi.py
> @@ -288,25 +288,39 @@ def eapi_has_sysroot(eapi):
>
>  _eapi_attrs = collections.namedtuple(
>  "_eapi_attrs",
> -"allows_package_provided "
> -"bdepend "
> -"broot "
> -"dots_in_PN dots_in_use_flags "
> -"exports_AA "
> -"exports_EBUILD_PHASE_FUNC "
> -"exports_ECLASSDIR "
> -"exports_KV "
> -"exports_merge_type "
> -"exports_PORTDIR "
> -"exports_replace_vars "
> -"feature_flag_test "
> -"idepend iuse_defaults iuse_effective posixish_locale "
> -"path_variables_end_with_trailing_slash "
> -"prefix "
> -"repo_deps required_use required_use_at_most_one_of "
> -"selective_src_uri_restriction slot_operator slot_deps "
> -"src_uri_arrows strong_blocks use_deps use_dep_defaults "
> -"empty_groups_always_true sysroot",
> +(
> +"allows_package_provided",
> +"bdepend",
> +"broot",
> +"dots_in_PN",
> +"dots_in_use_flags",
> +"exports_AA",
> +"exports_EBUILD_PHASE_FUNC",
> +"exports_ECLASSDIR",
> +"exports_KV",
> +"exports_merge_type",
> +"exports_PORTDIR",
> +"exports_replace_vars",
> +"feature_flag_test",
> +"idepend",
> +"iuse_defaults",
> +"iuse_effective",
> +"posixish_locale",
> +"path_variables_end_with_trailing_slash",
> +"prefix",
> +"repo_deps",
> +"required_use",
> +"required_use_at_most_one_of",
> +"selective_src_uri_restriction",
> +"slot_operator",
> +"slot_deps",
> +"src_uri_arrows",
> +"strong_blocks",
> +"use_deps",
> +"use_dep_defaults",
> +"empty_groups_always_true",
> +"sysroot",
> +),
>  )
>
>
> --
> 2.34.1
>
>



Re: [gentoo-portage-dev] [PATCH 1/4] portage.dep.Atom: Clean up __new__ parameters

2022-03-08 Thread Alec Warner
maybe *unused_args, **unused_kwargs, unsure on the style guide for
that (normally its _)

But feel free to merge as-is.

-A

On Wed, Feb 23, 2022 at 8:15 PM Matt Turner  wrote:
>
> From: "Wolfgang E. Sanyer" 
>
> Reviewed-by: Matt Turner 
> Signed-off-by: Wolfgang E. Sanyer 
> ---
>  lib/portage/dep/__init__.py | 12 +---
>  1 file changed, 1 insertion(+), 11 deletions(-)
>
> diff --git a/lib/portage/dep/__init__.py b/lib/portage/dep/__init__.py
> index 3b3577025..13c0f4ef7 100644
> --- a/lib/portage/dep/__init__.py
> +++ b/lib/portage/dep/__init__.py
> @@ -1489,17 +1489,7 @@ class Atom(str):
>  def __init__(self, forbid_overlap=False):
>  self.overlap = self._overlap(forbid=forbid_overlap)
>
> -def __new__(
> -cls,
> -s,
> -unevaluated_atom=None,
> -allow_wildcard=False,
> -allow_repo=None,
> -_use=None,
> -eapi=None,
> -is_valid_flag=None,
> -allow_build_id=None,
> -):
> +def __new__(cls, s, *args, **kwargs):
>  return str.__new__(cls, s)
>
>  def __init__(
> --
> 2.34.1
>
>



Re: [gentoo-portage-dev] [PATCH] estrip: rework hard link logic in save_elf_debug

2021-10-27 Thread Alec Warner
On Mon, Oct 25, 2021 at 10:10 AM Mike Gilbert  wrote:
>
> GDB loads debug files based on the file name given in the .gnu_debuglink
> section, prepended with /usr/lib/debug/${dirname}, where dirname is the
> absolute path to the parent directory of the binary being executed.
>
> For each unique inode as input, we need a link to the debug file with
> the GNU debuglink as its basename. A link to the debug file should exist
> for each directory in which the input inode exists.
>
> The debug link names should be based on the .gnu_debuglink value instead
> of the name of the file we are processing as input.
>
> The .gnu_debuglink value is based on the name of the first link
> processed for each inode. We save this value as a symlink, and then read
> it back as we process subsequent links.
>
> For example, given the following input:
>
> INODE PATH
> 1 /usr/bin/git
> 1 /usr/libexec/git-core/git-add
> 2 /usr/bin/git-shell
> 2 /usr/libexec/git-core/git-shell
>
> We generate the following inodes for the debug files:
>
> INODE DEBUGLINK
> 3 git.debug
> 4 git-shell.debug
>
> We should generate the following links:
>
> INODE PATH
> 3 /usr/lib/debug/usr/bin/git.debug
> 3 /usr/lib/debug/usr/libexec/git-core/git.debug
> 4 /usr/bin/debug/usr/bin/git-shell.debug
> 4 /usr/bin/debug/usr/libexec/git-core/git-shell.debug
>
> The previous code would have generated this broken output:
>
> INODE PATH
> 3 /usr/lib/debug/usr/bin/git.debug
> 3 /usr/lib/debug/usr/libexec/git-core/git-add.debug (*)
> 4 /usr/bin/debug/usr/bin/git-shell.debug
> 4 /usr/bin/debug/usr/libexec/git-core/git-shell.debug
>
> (*) This link has the wrong name.
>
> Bug: https://bugs.gentoo.org/820107
> Signed-off-by: Mike Gilbert 
> ---
>  bin/estrip | 31 ++-
>  1 file changed, 22 insertions(+), 9 deletions(-)
>
> diff --git a/bin/estrip b/bin/estrip
> index abe523fff..8134020fd 100755
> --- a/bin/estrip
> +++ b/bin/estrip
> @@ -201,17 +201,29 @@ save_elf_debug() {
> local x=$1
> local inode_debug=$2
> local splitdebug=$3
> -   local d_noslash=${D%/}
> -   local y=${ED%/}/usr/lib/debug/${x:${#d_noslash}}.debug
> +
> +   local x_basename=${x##*/}
> +   local x_dirname=${x%/*}
> +   local y_dirname=${ED%/}/usr/lib/debug/${x_dirname#${D%/}/}
> +   local y_basename y

Can we do better than X and Y here?

>
> # dont save debug info twice
> [[ ${x} == *".debug" ]] && return 0
>
> -   mkdir -p "${y%/*}"
> +   mkdir -p "${y_dirname}" || die
> +
> +   if [[ -L ${inode_debug} ]] ; then
> +   y_basename=$(readlink "${inode_debug}") || die
> +   y_basename=${y_basename##*/}
> +   y=${y_dirname}/${y_basename}
>
> -   if [ -f "${inode_debug}" ] ; then
> -   ln "${inode_debug}" "${y}" || die "ln failed unexpectedly"
> +   if [[ ! -e ${y} ]]; then
> +   ln -L "${inode_debug}" "${y}" || die
> +   fi
> else
> +   y_basename=${x_basename}.debug
> +   y=${y_dirname}/${y_basename}
> +
> if [[ -n ${splitdebug} ]] ; then
> mv "${splitdebug}" "${y}"
> else
> @@ -222,11 +234,12 @@ save_elf_debug() {
> fi
> # Only do the following if the debug file was
> # successfully created (see bug #446774).
> -   if [ $? -eq 0 ] ; then
> +   if [[ $? -eq 0 ]] ; then
> local args="a-x,o-w"
> [[ -g ${x} || -u ${x} ]] && args+=",go-r"
> chmod ${args} "${y}"
> -   ln "${y}" "${inode_debug}" || die "ln failed 
> unexpectedly"
> +   # symlink so we can read the name back.
> +   ln -s "${y}" "${inode_debug}" || die
> fi
> fi
>
> @@ -239,8 +252,8 @@ save_elf_debug() {
> local 
> buildid_dir="${ED%/}/usr/lib/debug/.build-id/${buildid:0:2}"
> local buildid_file="${buildid_dir}/${buildid:2}"
> mkdir -p "${buildid_dir}"
> -   [ -L "${buildid_file}".debug ] || ln -s 
> "../../${x:$((${#d_noslash} + 1))}.debug" "${buildid_file}.debug"
> -   [ -L "${buildid_file}" ] || ln -s "/${x:$((${#d_noslash} + 
> 1))}" "${buildid_file}"
> +   [[ -L "${buildid_file}".debug ]] || ln -s 
> "../../${x#${D%/}/}.debug" "${buildid_file}.debug"
> +   [[ -L "${buildid_file}" ]] || ln -s 
> "../../../../../${x#${D%/}/}" "${buildid_file}"
> fi
>  }
>
> --
> 2.33.1
>
>



Re: [gentoo-portage-dev] [RFC] LTS branch of Portage

2021-10-19 Thread Alec Warner
On Mon, Oct 18, 2021 at 4:25 PM Francesco Riosa  wrote:
>
>
> Il giorno mar 5 ott 2021 alle ore 10:31 Michał Górny  ha 
> scritto:
>>
>> Hi, everyone.
>>
>> I've been thinking about this for some time already, and the recent
>> FILESDIR mess seems to confirm it: I'd like to start a more stable LTS
>> branch of Portage.
>>
>> Roughly, the idea is that:
>>
>> - master becomes 3.1.x, and primary development happens there
>>
>> - 3.0.x becomes the LTS branch and only major bugfixes are backported
>> there
>>
>> As things settle down in the future, master would become 3.2.x, 3.1.x
>> would become LTS, 3.0.x will be discontinued and so on.
>>
>> WDYT?
>
>
> Sorry but portage is too strictly related to the ebuilds in tree, recent 
> removal of EAPI=5 from most eclasses underlined that.
> Or to put id differently if you want a LTS portage you also need a certain 
> number of "protected" eclasses and ebuilds
> It seems a lot of (very appreciated but don't count on me) work

I think this is backwards a bit. The idea is to backport things from
the main (development) branch to the LTS branch such that the tree
continues to work for both; no?

This seems mostly related to "what is a bugfix and will be backported
into LTS" and "what is a feature and is not backportable for LTS" in
terms of what the tree will rely on.

-A

>
>
>>
>>
>> --
>> Best regards,
>> Michał Górny
>>
>>
>>



Re: [gentoo-portage-dev] [PATCH] lib/_emerge/actions.py: warn on missing /run

2021-10-07 Thread Alec Warner
On Wed, Oct 6, 2021 at 8:20 PM Sam James  wrote:
>
> Newer versions of build-docbook-catalog use
> /run/lock. This exposed that we weren't
> asking users to mount /run in the handbook.
>
> Check if it exists and warn if it doesn't.
>
> This should primarily (exclusively?) be a
> problem in chroots given an init system
> should be creating this.
>
> Bug: https://bugs.gentoo.org/816303
> Signed-off-by: Sam James 
> ---
>  lib/_emerge/actions.py | 34 ++
>  1 file changed, 22 insertions(+), 12 deletions(-)
>
> diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
> index 05a115250..1b40bebd3 100644
> --- a/lib/_emerge/actions.py
> +++ b/lib/_emerge/actions.py
> @@ -2986,17 +2986,26 @@ def validate_ebuild_environment(trees):
>  check_locale()
>
>
> -def check_procfs():
> -procfs_path = "/proc"
> -if platform.system() not in ("Linux",) or os.path.ismount(procfs_path):
> -return os.EX_OK
> -msg = "It seems that %s is not mounted. You have been warned." % 
> procfs_path
> -writemsg_level(
> -"".join("!!! %s\n" % l for l in textwrap.wrap(msg, 70)),
> -level=logging.ERROR,
> -noiselevel=-1,
> -)
> -return 1

Please add a docstring (""" comment) to the function describing why
it's doing what its doing.

> +def check_mounted_fs():
> +paths = {"/proc": False, "/run": False}

So on Linux, proc is necessary. Isn't /run necessary regardless of OS?
Will docbook build on prefix, or on BSD without /run (I'm guessing
not?)

We might go with something like:

def check_mounted_fs():
  """Cheeky comment explaining why we probably need these mounts."""
required_paths = ['/run']
if platform.system() == 'Linux':
  required_paths.append('/proc')

missing_mounts = False
for path in required_paths:
  if not os.path.ismount(path):
msg = 
writemsg = ...
missing_mounts = True
  # else we do nothing, the mount is where we want it, etc.

return missing_mounts # true if mounts are missing, false otherwise.

> +
> +for path in paths.keys():
> +if platform.system() not in ("Linux",) or os.path.ismount(path):
> +paths[path] = True
> +continue
> +
> +msg = "It seems that %s is not mounted. You have been warned." % path
> +writemsg_level(
> +"".join("!!! %s\n" % l for l in textwrap.wrap(msg, 70)),
> +level=logging.ERROR,
> +noiselevel=-1,
> +)

Duncan covered this already, but I will +1 his concern about better
messaging being an option ;)

> +
> +# Were any of the mounts we were looking for missing?
> +if False in paths.values():
> +return 1

So a couple of nitpicks here. This is a common sort of "collect"
pattern; and I'd use python's any() / all() helpers for a more
pythonic experience.

E.g.

items = [some, list, of, items]
results = []
for j in items:
  results.append(some_computation(j))

# Did any of the computations succeed?
return any(results) # at least 1 True
# Did all of the computations succeed?
return all(results) # all True
# Did none of the computations succeed?
return not any(results) # all False

So in this example you have elected to store the results in a dict,
and we want to return true if all the results are true:

return all(paths.values())

But see also earlier notes above.

> +
> +return os.EX_OK

This isn't shell, so return True or False (not 0 or 1.) (but see above comment.)

>
>
>  def config_protect_check(trees):
> @@ -3474,7 +3483,8 @@ def run_action(emerge_config):
>  repo_name_check(emerge_config.trees)
>  repo_name_duplicate_check(emerge_config.trees)
>  config_protect_check(emerge_config.trees)
> -check_procfs()
> +
> +check_mounted_fs()
>
>  for mytrees in emerge_config.trees.values():
>  mydb = mytrees["porttree"].dbapi
> --
> 2.33.0
>
>



Re: [gentoo-portage-dev] [RFC] LTS branch of Portage

2021-10-05 Thread Alec Warner
On Tue, Oct 5, 2021 at 1:31 AM Michał Górny  wrote:
>
> Hi, everyone.
>
> I've been thinking about this for some time already, and the recent
> FILESDIR mess seems to confirm it: I'd like to start a more stable LTS
> branch of Portage.
>
> Roughly, the idea is that:
>
> - master becomes 3.1.x, and primary development happens there
>
> - 3.0.x becomes the LTS branch and only major bugfixes are backported
> there
>
> As things settle down in the future, master would become 3.2.x, 3.1.x
> would become LTS, 3.0.x will be discontinued and so on.

I'd appreciate a brief overview of how we would expect users to
install each branch.

E.g. one way might be:

sys-apps/portage- # dev branch
sys-apps/portage-3.1.x # LTS branch
sys-apps/portage-3.0.x # old branch; unmaintained

-A

>
> WDYT?
>
> --
> Best regards,
> Michał Górny
>
>
>



Re: [gentoo-portage-dev] [PATCH 2/2] Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg

2021-10-04 Thread Alec Warner
On Sat, Oct 2, 2021 at 1:11 PM Sam James  wrote:
>
> This is part of a series of fixes for the linked bug (failure
> to preserve libraries in some situations).
>
> When unpacking a binpkg to be installed, we should check
> for the existence of PROVIDES if we're installing any
> dynamic libraries. If PROVIDES does not exist in that case,
> this suggests that e.g. scanelf malfunctioned or some corruption occurred.
>
> Bug: https://bugs.gentoo.org/811462
> Signed-off-by: Sam James 
> ---
>  lib/_emerge/Binpkg.py | 28 +++-
>  1 file changed, 27 insertions(+), 1 deletion(-)
>
> diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
> index c7dde69bd..9b876f354 100644
> --- a/lib/_emerge/Binpkg.py
> +++ b/lib/_emerge/Binpkg.py
> @@ -2,7 +2,7 @@
>  # Distributed under the terms of the GNU General Public License v2
>
>  import functools
> -
> +import glob
>  import _emerge.emergelog
>  from _emerge.EbuildPhase import EbuildPhase
>  from _emerge.BinpkgFetcher import BinpkgFetcher
> @@ -13,6 +13,7 @@ from _emerge.EbuildMerge import EbuildMerge
>  from _emerge.EbuildBuildDir import EbuildBuildDir
>  from _emerge.SpawnProcess import SpawnProcess
>  from portage.eapi import eapi_exports_replace_vars
> +from portage.output import colorize
>  from portage.util import ensure_dirs
>  from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
>  import portage
> @@ -425,6 +426,31 @@ class Binpkg(CompositeTask):
>  self._async_unlock_builddir(returncode=self.returncode)
>  return
>
> +# Before anything else, let's do an integrity check.
> +(provides,) = self._bintree.dbapi.aux_get(self.pkg.cpv, ["PROVIDES"])
> +if not provides:
> +# Let's check if we've got inconsistent results.
> +# If we're installing dynamic libraries (.so files), we should
> +# really have a PROVIDES.
> +# (This is a complementary check at the point of ingestion for 
> the
> +# creation check in doebuild.py)
> +# Note: we could check a non-empty PROVIDES against the list of 
> .sos,
> +# but this doesn't gain us anything. We're interested in failure
> +# to properly parse the installed files at all, which should 
> really
> +# be a global problem (e.g. bug #811462)
> +installed_dynlibs = glob.glob(
> +self.settings["D"] + "/**/*.so", recursive=True
> +)
> +if installed_dynlibs:
> +self._writemsg_level(
> +colorize(
> +"BAD",
> +"!!! Error! Installing dynamic libraries (.so) with 
> blank PROVIDES!",
> +),
> +noiselevel=-1,
> +level=logging.ERROR,
> +)
> +

Here you have written the same code (and long comment block) as the
last patch...I'd take that as an argument to extract it into a
function (check_provides or whatever) and not copy and paste the code
around.

-A

>  try:
>  with io.open(
>  _unicode_encode(
> --
> 2.33.0
>
>



Re: [gentoo-portage-dev] [PATCH 1/2] doebuild.py: check for inconsistent PROVIDES/image post-src_install

2021-10-04 Thread Alec Warner
On Sat, Oct 2, 2021 at 1:11 PM Sam James  wrote:
>
> This is part of a series of fixes for the linked bug (failure
> to preserve libraries in some situations).
>
> At the point of installation (even if not merging), we need
> to detect inconsistent metadata: PROVIDES should be populated
> if we're installing any dynamic libraries. This suggests that
> e.g. scanelf malfunctioned or some corruption occurred.
>
> Bug: https://bugs.gentoo.org/811462
> Signed-off-by: Sam James 
> ---
>  lib/portage/package/ebuild/doebuild.py | 24 +++-
>  1 file changed, 23 insertions(+), 1 deletion(-)
>
> diff --git a/lib/portage/package/ebuild/doebuild.py 
> b/lib/portage/package/ebuild/doebuild.py
> index 9650a8444..dc3fe3d97 100644
> --- a/lib/portage/package/ebuild/doebuild.py
> +++ b/lib/portage/package/ebuild/doebuild.py
> @@ -3,6 +3,7 @@
>
>  __all__ = ["doebuild", "doebuild_environment", "spawn", "spawnebuild"]
>
> +import glob
>  import grp
>  import gzip
>  import errno
> @@ -3079,7 +3080,7 @@ def _post_src_install_soname_symlinks(mysettings, out):
>  ) as f:
>  f.write(soname_deps.requires)
>
> -if soname_deps.provides is not None:
> +if soname_deps.provides:

The previous code checked if soname_deps.provides was None (or not.)
You have changed it to check if soname_deps.provides is true-ish (or not.)

Why did you change it?

>  with io.open(
>  _unicode_encode(
>  os.path.join(build_info_dir, "PROVIDES"),
> @@ -3091,6 +3092,27 @@ def _post_src_install_soname_symlinks(mysettings, out):
>  errors="strict",
>  ) as f:
>  f.write(soname_deps.provides)
> +else:
> +# Let's check if we've got inconsistent results.
> +# If we're installing dynamic libraries (.so files), we should
> +# really have a PROVIDES.
> +# (This is a complementary check at the point of creation for the
> +# ingestion check in Binpkg.py)
> +# Note: we could check a non-empty PROVIDES against the list of .sos,
> +# but this doesn't gain us anything. We're interested in failure
> +# to properly parse the installed files at all, which should really
> +# be a global problem (e.g. bug #811462)
> +installed_dynlibs = glob.glob(image_dir + "/**/*.so", recursive=True)
> +
> +if installed_dynlibs:
> +self._writemsg_level(
> +colorize(
> +"BAD",
> +"!!! Error! Installing dynamic libraries (.so) with 
> blank PROVIDES!",
> +),
> +noiselevel=-1,
> +level=logging.ERROR,
> +)
>
>  if unrecognized_elf_files:
>  qa_msg = ["QA Notice: Unrecognized ELF file(s):"]
> --
> 2.33.0
>
>



Re: [gentoo-portage-dev] Performance tuning and parallelisation

2021-08-27 Thread Alec Warner
On Thu, Aug 26, 2021 at 4:03 AM Ed W  wrote:
>
> Hi All
>
> Consider this a tentative first email to test the water, but I have started 
> to look at performance
> of particularly the install phase of the emerge utility and I could use some 
> guidance on where to go
> next

To clarify; the 'install' phase installs the package into ${D}. The
'qmerge' phase is the phase that merges to the livefs.

>
> Firstly, to define the "problem": I have found gentoo to be a great base for 
> building custom
> distributions and I use it to build a small embedded distro which runs on a 
> couple of different
> architectures. (Essentially just a "ROOT=/something emerge $some_packages"). 
> However, I use some
> packaging around binpackages to avoid uncessary rebuilds, and this highlights 
> that "building" a
> complete install using only binary packages rarely gets over a load of 1. Can 
> we do better than
> this? Seems to be highly serialised on the install phase of copying the files 
> to the disk?

In terms of parallelism it's not safe to run multiple phase functions
simultaneously. This is a problem in theory and occasionally in
practice (recently discussed in #gentoo-dev.)
The phase functions run arbitrary code that modifies the livefs (as
pre / post install and rm can touch $ROOT.) As an example we observed
recently; font ebuilds will generate font related metadata. If 2
ebuilds try to generate the metadata at the same time; they can race
and cause unexpected results. Sometimes this is caught in the ebuild
(e.g. they wrote code like rebuild_indexes || die and the indexer
returned non-zero) but can simply result in silent data corruption
instead; particularly if the races go undetected.

>
> (Note I use parallel build and parallel-install flags, plus --jobs=N. If 
> there is code to compile
> then load will shoot up, but simply installing binpackages struggles to get 
> the load over about
> 0.7-1.1, so presumably single threaded in all parts?)
>
>
> Now, this is particularly noticeable where I cheated to build my arm install 
> and just used qemu
> user-mode on an amd64 host (rather than using cross-compile). Here it's very 
> noticeable that the
> install/merge phase of the build is consuming much/most of the install time.
>
> eg, random example (under qemu user mode)

I think perhaps a simpler test is to use qmerge (from portage-utils)?
If you can use emerge (e.g. in --pretend mode) to generate a package
list to merge; you can simply merge them with qmerge. I suspect qmerge
will both (a) be faster and (b) be less safe than emerge; as emerge is
doing a bunch of extra work you may or may not care about. You can
also consider running N qmerge's (again less sure how safe this is; as
the writes by qmerge may be racy.) Note again that this speed may not
come for free and you may end up with a corrupt image afterwards.

I'm not sure if folks are running qmerge in production like this
(maybe others on the list have experience.)

>
> # time ROOT=/tmp/timetest emerge -1k --nodeps openssl
>
> >>> Emerging binary (1 of 1) dev-libs/openssl-1.1.1k-r1::gentoo for 
> >>> /tmp/timetest/
> ...
> real0m30.145s
> user0m29.066s
> sys0m1.685s
>
>
> Running the same on the native host is about 5-6sec, (and I find this ratio 
> fairly consistent for
> qemu usermode, about 5-6x slower than native)
>
> If I pick another package with fewer files, then I will see this 5-6 secs 
> drop, suggesting (without
> offering proof) that the bulk of the time here is some "per file" processing.
>
> Note this machine is a 12 core AMD ryzen 3900x with SSDs that bench around 
> the 4GB/s+. So really 5-6
> seconds to install a few files is relatively "slow". Random benchmark on this 
> machine might be that
> I can backup 4.5GB of chroot with tar+zstd in about 4 seconds.
>
>
> So the question is: I assume that further parallelisation of the install 
> phase will be difficult,
> therefore the low hanging fruit here seems to be the install/merge phase and 
> why there seems to be
> quite a bit of CPU "per file installed"? Can anyone give me a leg up on how I 
> could benchmark this
> further and look for the hotspot? Perhaps someone understand the architecture 
> of this point more
> intimately and could point at whether there are opportunities to do some of 
> the processing on mass,
> rather than per file?
>
> I'm not really a python guru, but interested to poke further to see where the 
> time is going.
>
>
> Many thanks
>
> Ed W
>
>
>
>
>



Re: [gentoo-portage-dev] In what phase are file "merged"?

2021-08-06 Thread Alec Warner
On Thu, Aug 5, 2021 at 9:23 PM Ulrich Mueller  wrote:
>
> > On Fri, 06 Aug 2021, Joakim Tjernlund wrote:
>
> > On Wed, 2021-06-23 at 13:33 +0200, Michał Górny wrote:
> >> On Wed, 2021-06-23 at 12:40 +0200, Ulrich Mueller wrote:
> >> > I don't think that the ebuild can rely on any particular status of
> >> > the new package in pkg_*rm (of the old package), or the status of
> >> > the old package in pkg_*inst (of the new package).
> >>
> >> I would even say that it can't rely on the particular status of the
> >> old package in any case, if it's meant to be removed.  In particular,
> >> its dependencies can be unmerged before the package itself.
>
> > Stubled ove this mail again and noticed "its dependencies can be
> > unmerged before the package itself" stmt. That does not make sense to
> > me. Deps should be unmerged after any pkg that depends on them?
>
> A popular workflow is "emerge -c -p" followed by "emerge -C" on entries
> of the list shown. IIUC emerge -C doesn't do any dependency resolution,
> therefore ebuilds cannot rely on any removal order.

Not quite sure I follow. Let's assume I have A -> B -> C.

Is it legal for A to call a binary packaged in A in A's pkg_prerm?
If yes, then B and C have to be on the livefs at least until A's
pkg_prerm has run; right?; otherwise if we unmerged B or C before then
we might break A's binaries?

-A

>
> Ulrich



Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set

2021-01-18 Thread Alec Warner
On Mon, Jan 18, 2021 at 8:09 PM Zac Medico  wrote:
>
> On 1/18/21 6:07 PM, Alec Warner wrote:
> > On Fri, Jan 15, 2021 at 6:47 PM Matt Turner  wrote:
> >>
> >> This set is the upgradable packages for which the highest visible
> >> version has a different subslot than the currently installed version.
> >>
> >> The primary purpose of this feature is for use in catalyst builds. We
> >> update the "seed" stage3 before using it to build a new stage1.
> >>
> >> Updating the entire stage is expensive and unnecessary (since we're
> >> going to build the latest packages in stage1 and then rebuild everything
> >> in stage3).
> >>
> >> What we definitely do need to update in the original stage3 however, is
> >> any package that would trigger a subslot rebuild.
> >>
> >> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
> >> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
> >> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
> >> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
> >> will be built and libmpfr.so.6 included into the stage1. Since the old
> >> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
> >> subsequent stage builds.
> >>
> >> Our current options to update the seed are too large a hammer (e.g.,
> >> "--update --deep --newuse @world" or "--update --deep --newuse
> >> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
> >> updating seed stages for no gain beyond updating only packages for whom
> >> the subslot has changed.
> >>
> >> With this set, catalyst will likely use
> >>
> >> emerge @changed-subslot --ignore-built-slot-operator-deps y
> >>
> >> to update the seed stage.
> >>
> >> Thank you to Zac Medico for showing me how to do this.
> >>
> >> Bug: https://bugs.gentoo.org/739004
> >> Signed-off-by: Matt Turner 
> >> ---
> >>  cnf/sets/portage.conf  |  5 +
> >>  lib/portage/_sets/dbapi.py | 39 +-
> >>  2 files changed, 43 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
> >> index 22f0fa3a5..5651a9c53 100644
> >> --- a/cnf/sets/portage.conf
> >> +++ b/cnf/sets/portage.conf
> >> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
> >>  [rebuilt-binaries]
> >>  class = portage.sets.dbapi.RebuiltBinaries
> >>
> >> +# Installed packages for which the subslot of the highest visible ebuild
> >> +# version is different than the currently installed version.
> >> +[changed-subslot]
> >> +class = portage.sets.dbapi.SubslotChangedSet
> >> +
> >>  # Installed packages for which the highest visible ebuild
> >>  # version is lower than the currently installed version.
> >>  [downgrade]
> >> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
> >> index 52367c4a6..46ba5c17d 100644
> >> --- a/lib/portage/_sets/dbapi.py
> >> +++ b/lib/portage/_sets/dbapi.py
> >> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
> >>  import portage
> >>
> >>  __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
> >> -   "EverythingSet", "OwnerSet", "VariableSet"]
> >> +   "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
> >>
> >>  class EverythingSet(PackageSet):
> >> _operations = ["merge"]
> >> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
> >>
> >> singleBuilder = classmethod(singleBuilder)
> >>
> >> +class SubslotChangedSet(PackageSet):
> >> +
> >> +   _operations = ["merge", "unmerge"]
> >> +
> >> +   description = "Package set which contains all packages " + \
> >> +   "for which the subslot of the highest visible ebuild is " 
> >> + \
> >> +   "different than the currently installed version."
> >
> > description = ("string1",
> >"string2",
> >"string3")
> >
> > vs concat + \ for line continuation?
> &g

Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set

2021-01-18 Thread Alec Warner
On Fri, Jan 15, 2021 at 6:47 PM Matt Turner  wrote:
>
> This set is the upgradable packages for which the highest visible
> version has a different subslot than the currently installed version.
>
> The primary purpose of this feature is for use in catalyst builds. We
> update the "seed" stage3 before using it to build a new stage1.
>
> Updating the entire stage is expensive and unnecessary (since we're
> going to build the latest packages in stage1 and then rebuild everything
> in stage3).
>
> What we definitely do need to update in the original stage3 however, is
> any package that would trigger a subslot rebuild.
>
> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
> will be built and libmpfr.so.6 included into the stage1. Since the old
> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
> subsequent stage builds.
>
> Our current options to update the seed are too large a hammer (e.g.,
> "--update --deep --newuse @world" or "--update --deep --newuse
> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
> updating seed stages for no gain beyond updating only packages for whom
> the subslot has changed.
>
> With this set, catalyst will likely use
>
> emerge @changed-subslot --ignore-built-slot-operator-deps y
>
> to update the seed stage.
>
> Thank you to Zac Medico for showing me how to do this.
>
> Bug: https://bugs.gentoo.org/739004
> Signed-off-by: Matt Turner 
> ---
>  cnf/sets/portage.conf  |  5 +
>  lib/portage/_sets/dbapi.py | 39 +-
>  2 files changed, 43 insertions(+), 1 deletion(-)
>
> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
> index 22f0fa3a5..5651a9c53 100644
> --- a/cnf/sets/portage.conf
> +++ b/cnf/sets/portage.conf
> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
>  [rebuilt-binaries]
>  class = portage.sets.dbapi.RebuiltBinaries
>
> +# Installed packages for which the subslot of the highest visible ebuild
> +# version is different than the currently installed version.
> +[changed-subslot]
> +class = portage.sets.dbapi.SubslotChangedSet
> +
>  # Installed packages for which the highest visible ebuild
>  # version is lower than the currently installed version.
>  [downgrade]
> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
> index 52367c4a6..46ba5c17d 100644
> --- a/lib/portage/_sets/dbapi.py
> +++ b/lib/portage/_sets/dbapi.py
> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
>  import portage
>
>  __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
> -   "EverythingSet", "OwnerSet", "VariableSet"]
> +   "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
>
>  class EverythingSet(PackageSet):
> _operations = ["merge"]
> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
>
> singleBuilder = classmethod(singleBuilder)
>
> +class SubslotChangedSet(PackageSet):
> +
> +   _operations = ["merge", "unmerge"]
> +
> +   description = "Package set which contains all packages " + \
> +   "for which the subslot of the highest visible ebuild is " + \
> +   "different than the currently installed version."

description = ("string1",
   "string2",
   "string3")

vs concat + \ for line continuation?

-A

> +
> +   def __init__(self, portdb=None, vardb=None):
> +   super(SubslotChangedSet, self).__init__()
> +   self._portdb = portdb
> +   self._vardb = vardb
> +
> +   def load(self):
> +   atoms = []
> +   xmatch = self._portdb.xmatch
> +   xmatch_level = "bestmatch-visible"
> +   cp_list = self._vardb.cp_list
> +   pkg_str = self._vardb._pkg_str
> +   for cp in self._vardb.cp_all():
> +   for cpv in cp_list(cp):
> +   pkg = pkg_str(cpv, None)
> +   slot_atom = "%s:%s" % (pkg.cp, pkg.slot)
> +   ebuild = xmatch(xmatch_level, slot_atom)
> +   if not ebuild:
> +   continue
> +   if pkg.sub_slot != ebuild.sub_slot:
> +   atoms.append(slot_atom)
> +
> +   self._setAtoms(atoms)
> +
> +   def singleBuilder(cls, options, settings, trees):
> +   return cls(portdb=trees["porttree"].dbapi,
> +   vardb=trees["vartree"].dbapi)
> +
> +   singleBuilder = classmethod(singleBuilder)
> +
>  class DowngradeSet(PackageSet):
>
> _operations = ["merge", "unmerge"]
> --
> 2.26.2
>
>



[gentoo-portage-dev] pylint progress

2020-07-29 Thread Alec Warner
Hi,

Recently I've begun to run pylint on the portage codebase. You can see some
recent PRs on this[0][1][2]. Most of the linter errors I've fixed are what
I consider 'fairly trivial'. In general I'm happy to disable errors (or
instances of errors) in addition to resolving them. You can see some of
this in https://github.com/gentoo/portage/pull/593/files where we just
blanket disable a bunch of very numerous messages (either because I don't
expect to ever fix them, or because they are more work that I think we
should do at the moment.)

So I see essentially a few choices:
 - (a) Do we fix errors in certain classes and run the linter as part of
CI. This means that once we resolve errors of a certain class, we can
enable that class of check and prevent new occurrences.
 - (b) Do we just avoid running the linter as part of CI (perhaps because
we are not far enough along on this journey) and focus our efforts to get
to a point where we can do (a) without spamming CI?
 - (c) What messages should we bother not fixing at all so we can just not
work on those classes of error?

As an example of (c); I believe 'protected-access' is likely intrusive to
fix and pointless for portage (cat is out of the bag) where as a message
like W0612(unused-variable) is plausibly something we should fix
throughout the codebase. Similarly I think "E0602(undefined-variable)" is
often a bug in how pylint treats our lazy initialized variables, so most of
these are false positives.

I think getting these messages recorded will also enable other people
besides me to fix them (I know b-man said he was interested.) Curious to
hear thought on this.

-A

[0]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=49794e185350f0f9c8099ff84507873685b2b0f1
[1]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=550727af87d5f646617e0c19a3f3300c8117e7f5
[2]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=be20b37180f709ab0e451e4e07b6e82ac3a87b56


Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-09 Thread Alec Warner
On Thu, Jul 9, 2020 at 2:06 PM Chun-Yu Shei  wrote:

> Hmm, that's strange... it seems to have made it to the list archives:
> https://archives.gentoo.org/gentoo-portage-dev/message/a4db905a64e3c1f6d88c4876e8291a65
>
> (but it is entirely possible that I used "git send-email" incorrectly)
>

Ahhh it's visible there; I'll blame gMail ;)

-A


>
> On Thu, Jul 9, 2020 at 2:04 PM Alec Warner  wrote:
>
>>
>>
>> On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei  wrote:
>>
>>> Awesome!  Here's a patch that adds @lru_cache to use_reduce, vercmp, and
>>> catpkgsplit.  use_reduce was split into 2 functions, with the outer one
>>> converting lists/sets to tuples so they can be hashed and creating a
>>> copy of the returned list (since the caller seems to modify it
>>> sometimes).  I tried to select cache sizes that minimized memory use
>>> increase,
>>> while still providing about the same speedup compared to a cache with
>>> unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases
>>> from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the
>>> RES column in htop increases from 280 MB -> 290 MB.
>>>
>>> "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while
>>> max observed RES value actually decreases from 228 MB -> 214 MB (similar
>>> values observed across a few before/after runs).
>>>
>>> Here are the cache hit stats, max observed RES memory, and runtime in
>>> seconds  for various sizes in the update case.  Caching for each
>>> function was tested independently (only 1 function with caching enabled
>>> at a time):
>>>
>>> catpkgsplit:
>>> CacheInfo(hits=133, misses=21419, maxsize=None, currsize=21419)
>>> 270 MB
>>> 39.217
>>>
>>> CacheInfo(hits=1218900, misses=24905, maxsize=1, currsize=1)
>>> 271 MB
>>> 39.112
>>>
>>> CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000)
>>> 271 MB
>>> 39.217
>>>
>>> CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500)
>>> 269 MB
>>> 39.438
>>>
>>> CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000)
>>> 271 MB
>>> 39.348
>>>
>>> CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100)
>>> 271 MB
>>> 39.487
>>>
>>>
>>> use_reduce:
>>> CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561)
>>> 407 MB
>>> 35.77
>>>
>>> CacheInfo(hits=45186, misses=18800, maxsize=1, currsize=1)
>>> 353 MB
>>> 35.52
>>>
>>> CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000)
>>> 335 MB
>>> 35.31
>>>
>>> CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500)
>>> 318 MB
>>> 35.85
>>>
>>> CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000)
>>> 301 MB
>>> 36.39
>>>
>>> CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100)
>>> 299 MB
>>> 37.175
>>>
>>>
>>> I didn't bother collecting detailed stats for vercmp, since the
>>> inputs/outputs are quite small and don't cause much memory increase.
>>> Please let me know if there are any other suggestions/improvements (and
>>> thanks Sid for the lru_cache suggestion!).
>>>
>>
>> I don't see a patch attached; can you link to it?
>>
>> -A
>>
>>
>>>
>>> Thanks,
>>> Chun-Yu
>>>
>>>
>>>
>>>


Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-09 Thread Alec Warner
On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei  wrote:

> Awesome!  Here's a patch that adds @lru_cache to use_reduce, vercmp, and
> catpkgsplit.  use_reduce was split into 2 functions, with the outer one
> converting lists/sets to tuples so they can be hashed and creating a
> copy of the returned list (since the caller seems to modify it
> sometimes).  I tried to select cache sizes that minimized memory use
> increase,
> while still providing about the same speedup compared to a cache with
> unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases
> from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the
> RES column in htop increases from 280 MB -> 290 MB.
>
> "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while
> max observed RES value actually decreases from 228 MB -> 214 MB (similar
> values observed across a few before/after runs).
>
> Here are the cache hit stats, max observed RES memory, and runtime in
> seconds  for various sizes in the update case.  Caching for each
> function was tested independently (only 1 function with caching enabled
> at a time):
>
> catpkgsplit:
> CacheInfo(hits=133, misses=21419, maxsize=None, currsize=21419)
> 270 MB
> 39.217
>
> CacheInfo(hits=1218900, misses=24905, maxsize=1, currsize=1)
> 271 MB
> 39.112
>
> CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000)
> 271 MB
> 39.217
>
> CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500)
> 269 MB
> 39.438
>
> CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000)
> 271 MB
> 39.348
>
> CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100)
> 271 MB
> 39.487
>
>
> use_reduce:
> CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561)
> 407 MB
> 35.77
>
> CacheInfo(hits=45186, misses=18800, maxsize=1, currsize=1)
> 353 MB
> 35.52
>
> CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000)
> 335 MB
> 35.31
>
> CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500)
> 318 MB
> 35.85
>
> CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000)
> 301 MB
> 36.39
>
> CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100)
> 299 MB
> 37.175
>
>
> I didn't bother collecting detailed stats for vercmp, since the
> inputs/outputs are quite small and don't cause much memory increase.
> Please let me know if there are any other suggestions/improvements (and
> thanks Sid for the lru_cache suggestion!).
>

I don't see a patch attached; can you link to it?

-A


>
> Thanks,
> Chun-Yu
>
>
>
>


Re: [gentoo-portage-dev] [PATCH] config.environ: delay export of A and AA (bug 720180)

2020-05-26 Thread Alec Warner
On Tue, May 26, 2020 at 9:46 AM Zac Medico  wrote:

> On 5/26/20 1:43 AM, Alec Warner wrote:
> > On Mon, May 25, 2020 at 9:34 PM Zac Medico  > <mailto:zmed...@gentoo.org>> wrote:
> >
> > Since variables like A and AA can contain extremely large values
> which
> > may trigger E2BIG errors during attempts to execute subprocesses,
> delay
> > export until the last moment, and unexport when appropriate.
> >
> >
> > So I think if you want to do this because PMS says:
> >  AA should not be visible in EAPI > 3.
> >  A should only be visible in src_*, pkg_nofetch.
> >
> > That part of the patch makes sense to me. The part that is confusing to
> > me is the 'delay' part; can you explain that further? When you say
> > "delay until the last moment" what do you mean by that and what value is
> > it delivering?
>
> If we export an environment variable which contains an extremely large
> value, then there's a vulnerability in execve which causes it to fail
> with an E2BIG error. Since A and AA values can easily grow large enough
> to trigger this vulnerability, portage can protect itself from execve
> failures by delaying the export until the moment that it hands control
> to the ebuild phase.
>

> > Is it simply that we don't export these variables on the python side,
> > and we only use them in the shell portion?
>
> That's correct. Here's a test case which demonstrates the E2BIG error,
> and shows that 'export -n A' can suppress it:
>

I've run similar tests, but I'm less excited by this work around. I think
its reasonable to work toward eventually not exporting A and AA (the latter
already done in EAPIs > 3). Then when ebuilds encounter problems, we can
tell authors "Upgrade to EAPI X" (where X is >3 or >=8; in the potential
case of A.) Having a hard limit on A for EAPIs 0-7 and a hard limit on AA
for EAPIs 0-3 seems perfectly fine and we should expect authors to refactor
(as texlive did) in order to meet the existing API limitations.

Basically unless there are more practical use cases today, the delaying of
export seems like a feature no one will use and added complexity. I dunno
how large the go-mod use cases are yet; but I myself have not seen any with
this many deps.

-A


>
> $ A=$(dd if=/dev/zero bs=1M count=1 | tr '\0' ' ')
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB, 10 MiB) copied, 0.086557 s, 121 MB/s
> $ echo ${#A}
> 10485760
> $ export A
> $ ls
> bash: /bin/ls: Argument list too long
> $ export -n A
> $ /bin/echo hello world
> hello world


> --
> Thanks,
> Zac
>
>


Re: [gentoo-portage-dev] [PATCH] config.environ: delay export of A and AA (bug 720180)

2020-05-26 Thread Alec Warner
On Mon, May 25, 2020 at 9:34 PM Zac Medico  wrote:

> Since variables like A and AA can contain extremely large values which
> may trigger E2BIG errors during attempts to execute subprocesses, delay
> export until the last moment, and unexport when appropriate.
>

So I think if you want to do this because PMS says:
 AA should not be visible in EAPI > 3.
 A should only be visible in src_*, pkg_nofetch.

That part of the patch makes sense to me. The part that is confusing to me
is the 'delay' part; can you explain that further? When you say "delay
until the last moment" what do you mean by that and what value is it
delivering?

Is it simply that we don't export these variables on the python side, and
we only use them in the shell portion?

-A


> Bug: https://bugs.gentoo.org/720180
> Signed-off-by: Zac Medico 
> ---
>  bin/eapi.sh   |  9 +
>  bin/isolated-functions.sh | 34 
>  bin/phase-functions.sh| 13 +++
>  .../ebuild/_config/special_env_vars.py|  7 +++-
>  lib/portage/package/ebuild/config.py  | 39 ++-
>  5 files changed, 91 insertions(+), 11 deletions(-)
>
> diff --git a/bin/eapi.sh b/bin/eapi.sh
> index 29dfb008c..f56468e4a 100644
> --- a/bin/eapi.sh
> +++ b/bin/eapi.sh
> @@ -26,6 +26,15 @@ ___eapi_has_S_WORKDIR_fallback() {
>
>  # VARIABLES
>
> +___eapi_exports_A() {
> +   # https://bugs.gentoo.org/721088
> +   true
> +}
> +
> +___eapi_exports_AA() {
> +   [[ ${1-${EAPI-0}} =~ ^(0|1|2|3)$ ]]
> +}
> +
>  ___eapi_has_prefix_variables() {
> [[ ! ${1-${EAPI-0}} =~ ^(0|1|2)$ || " ${FEATURES} " == *"
> force-prefix "* ]]
>  }
> diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
> index fde684013..973450d86 100644
> --- a/bin/isolated-functions.sh
> +++ b/bin/isolated-functions.sh
> @@ -107,6 +107,39 @@ __bashpid() {
> sh -c 'echo ${PPID}'
>  }
>
> +# @FUNCTION: ___eapi_vars_export
> +# @DESCRIPTION:
> +# Export variables for the current EAPI. Calls to this function should
> +# be delayed until the last moment, since exporting these variables
> +# may trigger E2BIG errors suring attempts to execute subprocesses.
> +___eapi_vars_export() {
> +   source "${T}/environment.unexported" || die
> +
> +   if ___eapi_exports_A; then
> +   export A
> +   fi
> +
> +   if ___eapi_exports_AA; then
> +   export AA
> +   fi
> +}
> +
> +# @FUNCTION: ___eapi_vars_unexport
> +# @DESCRIPTION:
> +# Unexport variables that were exported for the current EAPI. This
> +# function should be called after an ebuild phase, in order to unexport
> +# variables that may trigger E2BIG errors during attempts to execute
> +# subprocesses.
> +___eapi_vars_unexport() {
> +   if ___eapi_exports_A; then
> +   export -n A
> +   fi
> +
> +   if ___eapi_exports_AA; then
> +   export -n AA
> +   fi
> +}
> +
>  __helpers_die() {
> if ___eapi_helpers_can_die && [[ ${PORTAGE_NONFATAL} != 1 ]]; then
> die "$@"
> @@ -122,6 +155,7 @@ die() {
>
> set +x # tracing only produces useless noise here
> local IFS=$' \t\n'
> +   ___eapi_vars_unexport
>
> if ___eapi_die_can_respect_nonfatal && [[ $1 == -n ]]; then
> shift
> diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
> index 90e622e75..df2c0d8de 100644
> --- a/bin/phase-functions.sh
> +++ b/bin/phase-functions.sh
> @@ -146,6 +146,7 @@ __filter_readonly_variables() {
> fi
> fi
>
> +   ___eapi_vars_unexport
> "${PORTAGE_PYTHON:-/usr/bin/python}"
> "${PORTAGE_BIN_PATH}"/filter-bash-environment.py "${filtered_vars}" || die
> "filter-bash-environment.py failed"
>  }
>
> @@ -212,6 +213,7 @@ __ebuild_phase() {
>
>  __ebuild_phase_with_hooks() {
> local x phase_name=${1}
> +   ___eapi_vars_export
> for x in {pre_,,post_}${phase_name} ; do
> __ebuild_phase ${x}
> done
> @@ -223,6 +225,7 @@ __dyn_pretend() {
> __vecho ">>> Remove '$PORTAGE_BUILDDIR/.pretended' to
> force pretend."
> return 0
> fi
> +   ___eapi_vars_export
> __ebuild_phase pre_pkg_pretend
> __ebuild_phase pkg_pretend
> >> "$PORTAGE_BUILDDIR/.pretended" || \
> @@ -236,6 +239,7 @@ __dyn_setup() {
> __vecho ">>> Remove '$PORTAGE_BUILDDIR/.setuped' to force
> setup."
> return 0
> fi
> +   ___eapi_vars_export
> __ebuild_phase pre_pkg_setup
> __ebuild_phase pkg_setup
> >> "$PORTAGE_BUILDDIR/.setuped" || \
> @@ -252,6 +256,7 @@ __dyn_unpack() {
> install -m${PORTAGE_WORKDIR_MODE:-0700} -d "${WORKDIR}" ||
> die "Failed to create dir '${WORKDIR}'"
> fi
> cd "${WORKDIR}" || die "Directory change failed: \`cd
> '${WORKDIR}'\`"
> +   ___eapi_vars_export
> __ebuild_phase 

Re: [gentoo-portage-dev] Need backup mentor for FUSE-based sandbox project

2020-04-25 Thread Alec Warner
On Thu, Apr 23, 2020 at 12:09 PM Michał Górny  wrote:

> Hi, everyone.
>
> It seems that we *urgently* (read: in 6 days) need to find backup
> mentors for this year's GSoC projects.  I'm mentoring the project to
> develop a FUSE-based sandbox alternative that's going to work reliably
> with more packages than the LD_PRELOAD hack [1].
>
> Would anyone volunteer to be the backup maintainer for this?
>
> [1]
> https://summerofcode.withgoogle.com/dashboard/organization/5749981018849280/proposal/5572241732927488/


I get a 403 for that.

https://wiki.gentoo.org/wiki/Google_Summer_of_Code/2020/Ideas#FUSE-powered_sandbox
is
your writeup; I assume the above is supposed to be a link to the student's
proposal?


>
>
> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-portage-dev] precisions on installed packages' dependencies

2020-03-28 Thread Alec Warner
On Fri, Mar 27, 2020 at 7:00 AM  wrote:

>
> - Alec Warner  a écrit :
> > On Tue, Mar 24, 2020 at 11:31 AM  wrote:
> > > However, I still doubt that only storing the soname dependencies is
> enough.
> > > Consider package A (that cannot be recompiled) that depends on package
> B
> > > which provides lib L.so.
> > > B is recompiled with different use flags, which put different
> > > functionalities in L.so.
> > > The dependencies of A are still satisfied (B is installed, L.so is
> > > available), but since the content of L.so changed, A cannot execute
> anymore.
> > > Hypothetically, can this scenario occur?
> > > Can this scenario occur in practice?
> > > Is there a way in emerge/portage to avoid it?
>
>
> > > You have far more experience than me on this, and it would be nice for
> me
> > > to know what I'm up against.
> >
> > A lot of this has to do with the specifics of how package managers manage
> > system state, as well as various quirks of subsets of the tree. For
> > example, a perl upgrade (X->Y) will often break perl modules who expect
> > perl-X, but get perl-Y. So one fix is to try to keep perl-X installed (so
> > we SLOT perl and have N perls installed.) Then you need to decide which
> > version of perl to build things against (X or Y, or both?) We took this
> > tactic in the python ecosystem; but perl is not slotted in Gentoo, and so
> > upgrading perl breaks all perl modules. There is a tool
> > (gentoo-perl-cleaner) that will walk the deptree and fix all of these
> > broken packages that you run after an upgrade.
> >
> > I'm not sure it's strictly avoidable. You could build perl-Y, then
> rebuild
> > all perl-modules against perl-Y, then merge the entire result to the
> > livefs. This will reduce the breakage time but likely not eliminate it;
> > plus it seems hard to implement in practice without modern filesystem
> tools
> > (overlayfs, btrfs, zfs or similar tech to make it atomic.) It also
> doesn't
> > account for executing code. What happens to perl-X code that is executing
> > when you unmerge perl-X? The short answer is that code might break and
> > 'proper' management means you should restart services after an upgrade
> > (something Gentoo doesn't typically do; but is common in Debian for
> > example.)
>
> Many thanks for this answer.
> To sum up what I understood, the problem is not really the dependencies,
> but which recompilation (and service restart) are triggered with an update.
>

Yes and no. Assume a spherical package manager that could detect
everything, we basically need to do the following for a perl X -> Y upgrade.

1 User triggers X - Y upgrade.
2 build perl-Y, but don't merge it to the livefs.
3 numerate all deps of perl-Y, build those (but don't merge them to the
livefs, but they need to build against perl-Y, not perl-X)
4 with atomic_transaction:
  4a Merge perl-Y to the livefs
  4b Merge perl-Y's dependencies to the live fs
5 restart anything that is running perl-X, so that it runs with perl-Y
6 unmerge perl-X

In practice we cannot always do 3 or 4 or 5. We will miss dependencies (due
to missing depgraph information) both in the package depgraph as well as
the service depgraph.

So in practice we do:
1 user triggers X -Y upgrade.
2 build perl-Y, merge it to the livefs, unmerge perl-X
3 run gentoo-perl-cleaner to upgrade the dependencies broken by the X - Y
upgrade (via slot deps, or whatever mechanism, it can vary.)
4 restart anything that is running perl-X

And just accept that from 2-4..well stuff will be broken. We can minimize
the time by building binpkgs ahead of time for example, and merging the
binpkgs in a parallel. Note that 5 and 4 are the same problem in both
lists. Note that per the guide linked below, sometimes 2-3 can be done 'at
once', although again practically speaking "During Perl upgrade, packages
that depend on Perl may become unavailable." even if they are all handled
by emerge, because there are many race conditions in the order that
packages get merged to the livefs.


> In gentoo, there is the ":=" slot operator (and others similar) in
> dependencies that trigger the recompilation when the dependency's slot
> change, but this is the only existing mechanism.
> And this is why every time perl changes, the compilation of its modules is
> not triggered and they are most probably broken.
> Correct?
>

They are supposed to be triggered:
https://wiki.gentoo.org/wiki/Perl#Upgrading_.28major_version.29 for example
says this. But this is up to how callers run the tool. For example gentoo
infra executes emerge via puppet, and we never execute emerge -uDNav
--with-bdeps=y --backtrack=100 --autounmask-keep-masks=y

Re: [gentoo-portage-dev] precisions on installed packages' dependencies

2020-03-26 Thread Alec Warner
On Tue, Mar 24, 2020 at 11:31 AM  wrote:

>
> - Zac Medico  a écrit :
>
> > > The goal of my tool is to have correct manipulation of package
> dependencies, and in particular here, I focus on the packages that are
> installed but not in the portage tree/a local overlay anymore (the problem
> does not occur for other packages).
> > > It seems that installed packages do not store which are the actual cpv
> they depend on. Correct?
> >
> > Right, because unfortunately that's something that changes over time.
> >
> > Also, we may not be able to pin it down at any given moment if we have
> > inconsistent || preferences as described here:
> >
> >
> https://archives.gentoo.org/gentoo-dev/message/550d3859dea6d0fb0b39064628992634
>
> Hmm, I think I see what you mean.
> Storing the cpvs that was used during solving the package's dependencies
> would be too restrictive, since two different packages could provide the
> exact same functionalities/libraries.
> And so, during a system update, only looking at the cpv dependencies would
> trigger useless recompilation of the packages that depend on the updated
> packages.
> Correct?
>
> Btw, my tool's solver does not have the problem discussed in the thread
> you're mentioning: atom order in lists has no influence in my solver.
> Would fixing the inconsistent || preferences make storing cpvs for
> installed packages more realistic?
>
>
> > > Also, I wanted to use the ebuild tool to install/uninstall package,
> which is not possible with this solution apparently.
> >
> > Why not? Would the preserve-libs feature solve your problem?
>
> ... I'm sorry, I wasn't aware of this feature.
> It would definitively solve the issue (except, as described in the bug
> 459038, when external tools remove libs).
>
> This discussion is very interesting!
> If I take this double layer of dependencies, I have to check how this
> influences the theory underlying my tool.
>
> However, I still doubt that only storing the soname dependencies is enough.
> Consider package A (that cannot be recompiled) that depends on package B
> which provides lib L.so.
> B is recompiled with different use flags, which put different
> functionalities in L.so.
> The dependencies of A are still satisfied (B is installed, L.so is
> available), but since the content of L.so changed, A cannot execute anymore.
> Hypothetically, can this scenario occur?
> Can this scenario occur in practice?
> Is there a way in emerge/portage to avoid it?
>

>
> > Well, there are a lot of upgrades that can't be performed without
> > temporarily breaking something, so the "forbid broken dependencies" idea
> > doesn't sound feasible to me.
>
> Could you tell me about several instances of such needed dependency
> breakage?
> You have far more experience than me on this, and it would be nice for me
> to know what I'm up against.
>

A lot of this has to do with the specifics of how package managers manage
system state, as well as various quirks of subsets of the tree. For
example, a perl upgrade (X->Y) will often break perl modules who expect
perl-X, but get perl-Y. So one fix is to try to keep perl-X installed (so
we SLOT perl and have N perls installed.) Then you need to decide which
version of perl to build things against (X or Y, or both?) We took this
tactic in the python ecosystem; but perl is not slotted in Gentoo, and so
upgrading perl breaks all perl modules. There is a tool
(gentoo-perl-cleaner) that will walk the deptree and fix all of these
broken packages that you run after an upgrade.

I'm not sure it's strictly avoidable. You could build perl-Y, then rebuild
all perl-modules against perl-Y, then merge the entire result to the
livefs. This will reduce the breakage time but likely not eliminate it;
plus it seems hard to implement in practice without modern filesystem tools
(overlayfs, btrfs, zfs or similar tech to make it atomic.) It also doesn't
account for executing code. What happens to perl-X code that is executing
when you unmerge perl-X? The short answer is that code might break and
'proper' management means you should restart services after an upgrade
(something Gentoo doesn't typically do; but is common in Debian for
example.)

-A


> Many thanks!
> Michael
>
>


Re: [gentoo-portage-dev] [PATCH] repoman.modules.vcs.git.changes: reindex (bug 712106)

2020-03-11 Thread Alec Warner
On Wed, Mar 11, 2020 at 12:16 AM Zac Medico  wrote:

> For files returned by git diff-index, call git update-index in order
> to ensure that the index reflects the state on disk. This will prevent
> incorrect assumptions in cases where the index is missing or stale for
> some reason. Since repoman uses this information to decide when to
> update copyright header dates, this can prevent spurious copyright
> header updates.
>
> Signed-off-by: Zac Medico 
> Bug: https://bugs.gentoo.org/712106
> ---
>  repoman/lib/repoman/modules/vcs/git/changes.py | 15 ---
>  1 file changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/repoman/lib/repoman/modules/vcs/git/changes.py
> b/repoman/lib/repoman/modules/vcs/git/changes.py
> index 7e9ac1eb5..ebf770d53 100644
> --- a/repoman/lib/repoman/modules/vcs/git/changes.py
> +++ b/repoman/lib/repoman/modules/vcs/git/changes.py
> @@ -29,8 +29,14 @@ class Changes(ChangesBase):
> '''
> super(Changes, self).__init__(options, repo_settings)
>
> -   def _scan(self):
> -   '''VCS type scan function, looks for all detectable
> changes'''
> +   def _scan(self, _reindex=True):
>

Why the underscore prefix?

-A


> +   '''
> +   VCS type scan function, looks for all detectable changes
> +
> +   @param _reindex: ensure that the git index reflects the
> state on
> +   disk for files returned by git diff-index
> +   @type _reindex: bool
> +   '''
> with repoman_popen(
> "git diff-index --name-only "
> "--relative --diff-filter=M HEAD") as f:
> @@ -51,6 +57,9 @@ class Changes(ChangesBase):
> removed = f.readlines()
> self.removed = ["./" + elem[:-1] for elem in removed]
> del removed
> +   if _reindex and (self.changed or self.new or self.removed):
> +   self.update_index([], self.changed + self.new +
> self.removed)
> +   self._scan(_reindex=False)
>
> @property
> def unadded(self):
> @@ -91,7 +100,7 @@ class Changes(ChangesBase):
> # of the working tree.
> myfiles = mymanifests + myupdates
> myfiles.sort()
> -   update_index_cmd = ["git", "update-index"]
> +   update_index_cmd = ["git", "update-index", "--add",
> "--remove"]
> update_index_cmd.extend(f.lstrip("./") for f in myfiles)
> if self.options.pretend:
> print("(%s)" % (" ".join(update_index_cmd),))
> --
> 2.24.1
>
>
>


Re: [gentoo-portage-dev] [PATCH v2] fetch: Support GLEP 75 mirror structure

2019-10-03 Thread Alec Warner
On Thu, Oct 3, 2019 at 9:37 AM Michał Górny  wrote:

> Add a support for the subset of GLEP 75 needed by Gentoo Infra.  This
> includes fetching and parsing layout.conf, and support for flat layout
> and filename-hash layout with cutoffs being multiplies of 4.
>
> Bug: https://bugs.gentoo.org/646898
> Signed-off-by: Michał Górny 
> ---
>  lib/portage/package/ebuild/fetch.py | 139 +++-
>  1 file changed, 135 insertions(+), 4 deletions(-)
>
> Changes in v2: switched to a more classy layout to make the code
> reusable in emirrordist.
>
> diff --git a/lib/portage/package/ebuild/fetch.py
> b/lib/portage/package/ebuild/fetch.py
> index 227bf45ae..18e3d390a 100644
> --- a/lib/portage/package/ebuild/fetch.py
> +++ b/lib/portage/package/ebuild/fetch.py
> @@ -7,12 +7,15 @@ __all__ = ['fetch']
>
>  import errno
>  import io
> +import itertools
> +import json
>  import logging
>  import random
>  import re
>  import stat
>  import sys
>  import tempfile
> +import time
>
>  from collections import OrderedDict
>
> @@ -27,14 +30,17 @@ portage.proxy.lazyimport.lazyimport(globals(),
> 'portage.package.ebuild.doebuild:doebuild_environment,' + \
> '_doebuild_spawn',
> 'portage.package.ebuild.prepare_build_dirs:prepare_build_dirs',
> +
>  'portage.util.configparser:SafeConfigParser,read_configs,NoOptionError',
> +   'portage.util._urlopen:urlopen',
>  )
>
>  from portage import os, selinux, shutil, _encodings, \
> _movefile, _shell_quote, _unicode_encode
>  from portage.checksum import (get_valid_checksum_keys, perform_md5,
> verify_all,
> -   _filter_unaccelarated_hashes, _hash_filter, _apply_hash_filter)
> +   _filter_unaccelarated_hashes, _hash_filter, _apply_hash_filter,
> +   checksum_str)
>  from portage.const import BASH_BINARY, CUSTOM_MIRRORS_FILE, \
> -   GLOBAL_CONFIG_PATH
> +   GLOBAL_CONFIG_PATH, CACHE_PATH
>  from portage.data import portage_gid, portage_uid, secpass,
> userpriv_groups
>  from portage.exception import FileNotFound, OperationNotPermitted, \
> PortageException, TryAgain
> @@ -253,6 +259,130 @@ _size_suffix_map = {
> 'Y' : 80,
>  }
>
> +
> +class FlatLayout(object):
> +   def get_path(self, filename):
> +   return filename
> +
> +
> +class FilenameHashLayout(object):
> +   def __init__(self, algo, cutoffs):
> +   self.algo = algo
> +   self.cutoffs = [int(x) for x in cutoffs.split(':')]
> +
> +   def get_path(self, filename):
> +   fnhash = checksum_str(filename.encode('utf8'), self.algo)
> +   ret = ''
> +   for c in self.cutoffs:
> +   assert c % 4 == 0
>

I'm not quite sure what this assert is doing. I'm not super in favor of
asserts (I'd rather see an exception like raise FooError("..."), but if you
are going to use it please use something like:

assert c %4 == 0, "Some description of why we put this assert here so if it
fires we can do something useful."

+   c = c // 4
> +   ret += fnhash[:c] + '/'
> +   fnhash = fnhash[c:]
> +   return ret + filename
> +
> +
> +class MirrorLayoutConfig(object):
> +   """
> +   Class to read layout.conf from a mirror.
> +   """
> +
> +   def __init__(self):
> +   self.structure = ()
> +
> +   def read_from_file(self, f):
> +   cp = SafeConfigParser()
> +   read_configs(cp, [f])
> +   vals = []
> +   for i in itertools.count():
> +   try:
> +   vals.append(tuple(cp.get('structure', '%d'
> % i).split()))
> +   except NoOptionError:
> +   break
> +   self.structure = tuple(vals)
> +
> +   def serialize(self):
> +   return self.structure
> +
> +   def deserialize(self, data):
> +   self.structure = data
> +
> +   @staticmethod
> +   def validate_structure(val):
> +   if val == ('flat',):
> +   return True
> +   if val[0] == 'filename-hash' and len(val) == 3:
> +   if val[1] not in get_valid_checksum_keys():
> +   return False
> +   # validate cutoffs
> +   for c in val[2].split(':'):
> +   try:
> +   c = int(c)
> +   except ValueError:
> +   break
> +   else:
> +   if c % 4 != 0:
> +   break
> +   else:
> +   return True
> +   return False
> +   return False
> +
> +   def get_best_supported_layout(self):
> +   

Re: [gentoo-portage-dev] [PATCH] fetch: Support GLEP 75 mirror structure

2019-10-03 Thread Alec Warner
On Thu, Oct 3, 2019 at 7:52 AM Michał Górny  wrote:

> Add a support for the subset of GLEP 75 needed by Gentoo Infra.  This
> includes fetching and parsing layout.conf, and support for flat layout
> and filename-hash layout with cutoffs being multiplies of 4.
>
> Bug: https://bugs.gentoo.org/646898
> Signed-off-by: Michał Górny 
> ---
>  lib/portage/package/ebuild/fetch.py | 113 +++-
>  1 file changed, 109 insertions(+), 4 deletions(-)
>
> diff --git a/lib/portage/package/ebuild/fetch.py
> b/lib/portage/package/ebuild/fetch.py
> index 227bf45ae..692efcc01 100644
> --- a/lib/portage/package/ebuild/fetch.py
> +++ b/lib/portage/package/ebuild/fetch.py
> @@ -7,12 +7,15 @@ __all__ = ['fetch']
>
>  import errno
>  import io
> +import itertools
> +import json
>  import logging
>  import random
>  import re
>  import stat
>  import sys
>  import tempfile
> +import time
>
>  from collections import OrderedDict
>
> @@ -27,14 +30,17 @@ portage.proxy.lazyimport.lazyimport(globals(),
> 'portage.package.ebuild.doebuild:doebuild_environment,' + \
> '_doebuild_spawn',
> 'portage.package.ebuild.prepare_build_dirs:prepare_build_dirs',
> +
>  'portage.util.configparser:SafeConfigParser,read_configs,NoOptionError',
> +   'portage.util._urlopen:urlopen',
>  )
>
>  from portage import os, selinux, shutil, _encodings, \
> _movefile, _shell_quote, _unicode_encode
>  from portage.checksum import (get_valid_checksum_keys, perform_md5,
> verify_all,
> -   _filter_unaccelarated_hashes, _hash_filter, _apply_hash_filter)
> +   _filter_unaccelarated_hashes, _hash_filter, _apply_hash_filter,
> +   checksum_str)
>  from portage.const import BASH_BINARY, CUSTOM_MIRRORS_FILE, \
> -   GLOBAL_CONFIG_PATH
> +   GLOBAL_CONFIG_PATH, CACHE_PATH
>  from portage.data import portage_gid, portage_uid, secpass,
> userpriv_groups
>  from portage.exception import FileNotFound, OperationNotPermitted, \
> PortageException, TryAgain
> @@ -253,6 +259,104 @@ _size_suffix_map = {
> 'Y' : 80,
>  }
>
> +
> +def filename_hash_path(filename, algo, cutoffs):
> +   """
> +   Get directory path for filename in filename-hash mirror structure.
> +
> +   @param filename: Filename to fetch
> +   @param algo: Hash algorithm
> +   @param cutoffs: Cutoff values (n:n...)
> +   @return: Directory path
> +   """
> +
> +   fnhash = checksum_str(filename.encode('utf8'), algo)
> +   ret = ''
> +   for c in cutoffs.split(':'):
> +   c = int(c) // 4
> +   ret += fnhash[:c] + '/'
>

When making a path, please use os.path.join()


> +   fnhash = fnhash[c:]
> +   return ret
> +
> +
> +def get_mirror_url(mirror_url, filename, eroot):
> +   """
> +   Get correct fetch URL for a given file, accounting for mirror
> +   layout configuration.
> +
> +   @param mirror_url: Base URL to the mirror (without '/distfiles')
> +   @param filename: Filename to fetch
> +   @param eroot: EROOT to use for the cache file
> +   @return: Full URL to fetch
> +   """
>
+
> +   cache_file = os.path.join(eroot, CACHE_PATH,
> 'mirror-metadata.json')
> +   try:
> +   with open(cache_file, 'r') as f:
> +   cache = json.load(f)
> +   except (IOError, ValueError):
> +   cache = {}
>

I'm a bit worried that we are opening this cache file off of disk every
time we call get_mirror_url(). Can we just cache the contents in memory
between calls; or even better pass the cache in as argument rather than it
be contained in get_mirror_url?


> +
> +   ts, layout = cache.get(mirror_url, (0, None))
> +   # refresh at least daily
> +   if ts < time.time() - 86400:
> +   # the default
> +   layout = ('flat',)
> +
> +   try:
> +   f = urlopen(mirror_url + '/distfiles/layout.conf')
> +   try:
> +   data = io.StringIO(f.read().decode('utf8'))
> +   finally:
> +   f.close()
> +   cp = SafeConfigParser()
> +   read_configs(cp, [data])
> +
> +   for i in itertools.count():
> +   try:
> +   val = tuple(cp.get('structure',
> '%d' % i).split())
> +   if val == ('flat',):
> +   pass
> +   elif val[0] == 'filename-hash' and
> len(val) == 3:
> +   if val[1] not in
> get_valid_checksum_keys():
> +   continue
> +   # validate cutoffs
> +   cutoffs_good = False
> +  

Re: [gentoo-portage-dev] [PATCH] Use RTNETLINK to configure the loopback interface

2019-08-27 Thread Alec Warner
unit tests?

-A

On Mon, Aug 26, 2019 at 8:03 PM Mike Gilbert  wrote:

> This eliminates the dependency on iproute2 on Linux.
>
> Signed-off-by: Mike Gilbert 
> ---
>  lib/portage/process.py  | 25 --
>  lib/portage/util/netlink.py | 91 +
>  2 files changed, 100 insertions(+), 16 deletions(-)
>  create mode 100644 lib/portage/util/netlink.py
>
> diff --git a/lib/portage/process.py b/lib/portage/process.py
> index 2a2cbd972..bb4462c7f 100644
> --- a/lib/portage/process.py
> +++ b/lib/portage/process.py
> @@ -10,7 +10,6 @@ import multiprocessing
>  import platform
>  import signal
>  import socket
> -import struct
>  import subprocess
>  import sys
>  import traceback
> @@ -489,17 +488,6 @@ def _configure_loopback_interface():
> Configure the loopback interface.
> """
>
> -   IFF_UP = 0x1
> -   ifreq = struct.pack('16sh', b'lo', IFF_UP)
> -   SIOCSIFFLAGS = 0x8914
> -
> -   sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, 0)
> -   try:
> -   fcntl.ioctl(sock, SIOCSIFFLAGS, ifreq)
> -   except IOError as e:
> -   writemsg("Unable to enable loopback interface: %s\n" %
> e.strerror, noiselevel=-1)
> -   sock.close()
> -
> # We add some additional addresses to work around odd behavior in
> glibc's
> # getaddrinfo() implementation when the AI_ADDRCONFIG flag is set.
> #
> @@ -514,12 +502,17 @@ def _configure_loopback_interface():
> # Bug: https://bugs.gentoo.org/690758
> # Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=12377#c13
>
> +   from portage.util.netlink import RtNetlink
> +
> try:
> -   subprocess.call(['ip', 'address', 'add', '10.0.0.1/8',
> 'dev', 'lo'])
> -   if _has_ipv6():
> -   subprocess.call(['ip', 'address', 'add',
> 'fd00::1/8', 'dev', 'lo'])
> +   with RtNetlink() as rtnl:
> +   ifindex = rtnl.get_link_ifindex(b'lo')
> +   rtnl.set_link_up(ifindex)
> +   rtnl.add_address(ifindex, socket.AF_INET,
> '10.0.0.1', 8)
> +   if _has_ipv6():
> +   rtnl.add_address(ifindex, socket.AF_INET6,
> 'fd::1', 8)
> except EnvironmentError as e:
> -   writemsg("Error calling 'ip': %s\n" % e.strerror,
> noiselevel=-1)
> +   writemsg("Unable to configure loopback interface: %s\n" %
> e.strerror, noiselevel=-1)
>
>  def _exec(binary, mycommand, opt_name, fd_pipes,
> env, gid, groups, uid, umask, cwd,
> diff --git a/lib/portage/util/netlink.py b/lib/portage/util/netlink.py
> new file mode 100644
> index 0..5b18b8f95
> --- /dev/null
> +++ b/lib/portage/util/netlink.py
> @@ -0,0 +1,91 @@
> +# Copyright 2019 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +from io import BytesIO
> +from os import strerror
> +from struct import Struct
> +
> +from socket import (
> +   AF_NETLINK, AF_UNSPEC,
> +   MSG_PEEK, MSG_TRUNC,
> +   NETLINK_ROUTE,
> +   SOCK_DGRAM,
> +   inet_pton, socket,
> +)
> +
> +IFA_LOCAL = 2
> +IFF_UP = 0x1
> +IFLA_IFNAME = 3
> +NLMSG_ERROR = 2
> +RTM_NEWLINK = 16
> +RTM_GETLINK = 18
> +RTM_NEWADDR = 20
> +NLM_F_REQUEST = 0x1
> +NLM_F_ACK = 0x4
> +NLM_F_EXCL = 0x200
> +NLM_F_CREATE = 0x400
> +
> +nlmsghdr = Struct('=IHHII')
> +nlmsgerr = Struct('i')
> +rtattr = Struct('HH')
> +ifinfomsg = Struct('BHiII')
> +ifaddrmsg = Struct('i')
> +
> +def create_nlmsg(nlmsg_type, nlmsg_flags, nlmsg_seq, nlmsg_pid, data):
> +   nlmsg_len = nlmsghdr.size + len(data)
> +   return nlmsghdr.pack(nlmsg_len, nlmsg_type, nlmsg_flags,
> nlmsg_seq, nlmsg_pid) + data
> +
> +def create_rtattr(rta_type, data):
> +   rta_len = rtattr.size + len(data)
> +   return rtattr.pack(rta_len, rta_type) + data
> +
> +def parse_message(msg):
> +   buf = BytesIO(msg)
> +   hdr = nlmsghdr.unpack(buf.read(nlmsghdr.size))
> +   if hdr[1] == NLMSG_ERROR:
> +   err = nlmsgerr.unpack(buf.read(nlmsgerr.size))
> +   error = -err[0]
> +   if error != 0:
> +   raise OSError(error, strerror(error))
> +   elif hdr[1] == RTM_NEWLINK:
> +   return ifinfomsg.unpack(buf.read(ifinfomsg.size))
> +
> +class RtNetlink:
> +   def __init__(self):
> +   self.sock = socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE)
> +   self.addr = (0, 0)
> +   self.sock.bind(self.addr)
> +
> +   def __enter__(self):
> +   return self
> +
> +   def __exit__(self, exc_type, exc_value, traceback):
> +   self.sock.close()
> +
> +   def send_message(self, msg):
> +   self.sock.sendto(msg, self.addr)
> +   # Messages are variable length, but 128 is enough for the
> the ones we care about.
> +   resp = self.sock.recv(128)
> +   

Re: [gentoo-portage-dev] [PATCH] bindbapi: add reentrant lock method (bug 685236)

2019-05-08 Thread Alec Warner
On Wed, May 8, 2019 at 2:05 PM Zac Medico  wrote:

> On 5/7/19 1:01 PM, Zac Medico wrote:
> > On 5/7/19 7:55 AM, Alec Warner wrote:
> >
> >> Also curious why we are not implementing enter and exit so we can avoid
> >> unbalanced pairs by using context managers.
> >>
> >> e.g. in match(), we could likely write:
> >>
> >> with self.dbapi.lock():
> >>   # try to match
> >>   update_pkgindex = self._populate_local()
> >>   if update_pkgindex:
> >> self._pkgindex_write(update_pkgindex)
> >>
> >> If the block raises an exception, __exit__ is still called, so we can
> >> eliminate the finally: blocks and replace them with a careful __exit__
> >> implementation?
> >>
> >> -A
> >
> > Yeah, the reference counted lock class could have a method that returns
> > a context manager which increments/decrements the lock count. The
> > context manager can be implemented with the contextlib.contextmanager
> > decorator like this:
> >
> > @contextlib.contextmanager
> > def contextmanager(self):
> > self.lock()
> > try:
> > yield self
> > finally:
> > self.unlock()
> >
>
> Since we really don't want asynchronous code to block waiting for a
> lock, we really should use a (python3.5+ only) asynchronous context
> manager:
>
>
> https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with
>
> Given that python2.7 is scheduled for retirement in 2020
> (https://pythonclock.org/), maybe it's time to drop support for
> python3.4 and earlier.
>

I haven't used that stuff yet, so I will defer to you on its use; same with
python support.

-A


> --
> Thanks,
> Zac
>
>


Re: [gentoo-portage-dev] [PATCH] bindbapi: add reentrant lock method (bug 685236)

2019-05-07 Thread Alec Warner
On Mon, May 6, 2019 at 8:15 PM Zac Medico  wrote:

> Add a reentrant lock/unlock method which is useful for things like
> emaint binhost and eclean-pkg. The vardbapi class already provides
> lock/unlock methods that behave the same way.
>

Curious why we are not doing more encapsulation here.

I'd suggest a new class type in locks.py:

class RefCountedLock(object): ...
# This implements the Lock() / Unlock() interface, as well as perhaps
__enter__ and __exit__.
# It uses a lockfile at path and a reference count.

Then in bintree's init, set self.lock to
locks.refCountedLock(self.bintree._pkgindex_file)

def lock(self): # on bintree
  self._lock.Lock()

def unlock(self):
  self._lock.Unlock()

Also curious why we are not implementing enter and exit so we can avoid
unbalanced pairs by using context managers.

e.g. in match(), we could likely write:

with self.dbapi.lock():
  # try to match
  update_pkgindex = self._populate_local()
  if update_pkgindex:
self._pkgindex_write(update_pkgindex)

If the block raises an exception, __exit__ is still called, so we can
eliminate the finally: blocks and replace them with a careful __exit__
implementation?

-A


>
> Bug: https://bugs.gentoo.org/685236
> Signed-off-by: Zac Medico 
> ---
>  lib/portage/dbapi/bintree.py  | 46 ++-
>  lib/portage/emaint/modules/binhost/binhost.py |  9 ++--
>  2 files changed, 38 insertions(+), 17 deletions(-)
>
> diff --git a/lib/portage/dbapi/bintree.py b/lib/portage/dbapi/bintree.py
> index 9c2d877e7..707958858 100644
> --- a/lib/portage/dbapi/bintree.py
> +++ b/lib/portage/dbapi/bintree.py
> @@ -1,4 +1,4 @@
> -# Copyright 1998-2018 Gentoo Foundation
> +# Copyright 1998-2019 Gentoo Authors
>  # Distributed under the terms of the GNU General Public License v2
>
>  from __future__ import unicode_literals
> @@ -94,6 +94,8 @@ class bindbapi(fakedbapi):
> ])
> self._aux_cache_slot_dict =
> slot_dict_class(self._aux_cache_keys)
> self._aux_cache = {}
> +   self._lock = None
> +   self._lock_count = 0
>
> @property
> def writable(self):
> @@ -106,6 +108,34 @@ class bindbapi(fakedbapi):
> """
> return os.access(first_existing(self.bintree.pkgdir),
> os.W_OK)
>
> +   def lock(self):
> +   """
> +   Acquire a reentrant lock, blocking, for cooperation with
> concurrent
> +   processes.
> +   """
> +   if self._lock_count <= 0:
> +   if self._lock is not None:
> +   raise AssertionError("already locked")
> +   # At least the parent needs to exist for the lock
> file.
> +   ensure_dirs(self.bintree.pkgdir)
> +   self._lock = lockfile(self.bintree._pkgindex_file,
> wantnewlockfile=True)
> +
> +   self._lock_count += 1
> +
> +   def unlock(self):
> +   """
> +   Release a lock, decrementing the recursion level. Each
> unlock() call
> +   must be matched with a prior lock() call, or else an
> AssertionError
> +   will be raised if unlock() is called while not locked.
> +   """
> +   if self._lock_count <= 1:
> +   if self._lock is None:
> +   raise AssertionError("not locked")
> +   unlockfile(self._lock)
> +   self._lock = None
> +
> +   self._lock_count -= 1
> +
> def match(self, *pargs, **kwargs):
> if self.bintree and not self.bintree.populated:
> self.bintree.populate()
> @@ -545,16 +575,13 @@ class binarytree(object):
> # that changes made by a concurrent
> process cannot be lost. This
> # case is avoided when possible, in order
> to minimize lock
> # contention.
> -   pkgindex_lock = None
> +   self.dbapi.lock()
> try:
> -   pkgindex_lock =
> lockfile(self._pkgindex_file,
> -   wantnewlockfile=True)
> update_pkgindex =
> self._populate_local()
> if update_pkgindex:
>
> self._pkgindex_write(update_pkgindex)
> finally:
> -   if pkgindex_lock:
> -   unlockfile(pkgindex_lock)
> +   self.dbapi.unlock()
>
> if getbinpkgs:
> if not
> self.settings.get("PORTAGE_BINHOST"):
> @@ -1110,10 +1137,8 @@ class binarytree(object):
>
> # Reread the 

Re: [gentoo-portage-dev] [PATCH 0/4] Rename PORT_LOGDIR{,_CLEAN} variables to PORTAGE_LOGDIR{,_CLEAN} Bug 668538

2018-12-17 Thread Alec Warner
On Mon, Dec 17, 2018 at 10:51 AM Michał Górny  wrote:

> On Mon, 2018-12-17 at 15:44 +, M. J. Everitt wrote:
> > On 17/12/18 12:54, Michał Górny wrote:
> > > > Not only this, but as noted, unless you know the man pages for
> portage and
> > > > make.conf in order to recite them in your sleep, they are confusing
> for
> > > > users, as they do not necessarily follow an obvious pattern, and it
> wasn't
> > > > until I was attempting to debug something that I noticed that despite
> > > > believing I had the correct settings in my make.conf (set over a
> period of
> > > > YEARS) they were in fact completely useless, and it wasn't until I
> had to
> > > > spend time with somebody debugging WTF was happening, that this
> particular
> > > > issue even became apparent...
> > >
> > > I don't see how this is an argument for anything.  You have to read
> > > the manual in order to know that such variable exists and what it
> does.
> > > Or, well, technically you don't since it's provided in
> make.conf.example
> > > already where you only need to uncomment it.
> > >
> > > Either way, the variable name is trivial.  Even if you don't follow
> > > the usual pattern of uncommenting it from make.conf.example or copying
> > > from the manual, remembering it for the time needed to retype shoudln't
> > > be a problem.
> > >
> > > So, is this a solution to a real problem?  Or is it merely a half-
> > > thought-out partial change that's going to require people to update
> > > their configuration for no long-term benefit?  And then they will have
> > > to update it again when someone decides to take another variable for
> > > a spin.
> > >
> >
> > In the case you hadn't noticed, clearly you haven't .. the change is
> > backwards compatible.. that has already been thought out.
> >
> > But you haven't actually looked at the patch have you, Michal ?
> >
>
> I did look at it.  However, that doesn't change what I said.  Being
> 'backwards compatible' does not change the fact that the old variable
> becomes deprecated now.  Ergo, users are expected to eventually switch
> to the new one.
>

So if we rewrote the patches to not deprecate the old one but to support
both; how would that strike you?

-A


>
> Even if you don't care beyond changing this now and forgetting about it
> afterwards, someone eventually will have to clean up the old variable
> and actively force people to update.


> --
> Best regards,
> Michał Górny
>


Re: [gentoo-portage-dev] Re: [RFC] gpkg format proposal v2

2018-11-12 Thread Alec Warner
On Mon, Nov 12, 2018 at 3:24 PM Ulrich Mueller  wrote:

> > On Mon, 12 Nov 2018, Michał Górny wrote:
>
> >> Also, what would be wrong with ar? It's a standard POSIX tool, and
> >> should be available everywhere.
>
> > The original post says what's wrong with ar.  Please be more specific
> > if you disagree with it.
>
> AFAICS, the arguments are that ar would be obscure, and that the LSB
> considers it deprecated. I don't find either of them convincing.
> Since when do we care about the LSB?
>

I assert that it doesn't matter which tool we pick, so we have arbitrarily
chosen tar because we like it.

If you have a basis for preferring ar over tar; I'd love to hear it. I only
brought it up because I know debian uses it.

-A


>
> Ulrich
>


Re: [gentoo-portage-dev] Re: [RFC] Improving Gentoo package format

2018-11-11 Thread Alec Warner
On Sun, Nov 11, 2018 at 2:21 PM Rich Freeman  wrote:

> On Sun, Nov 11, 2018 at 1:02 PM M. J. Everitt  wrote:
> >
> > If you can really present a decent argument for replicating the
> > functionality of other distros like Debian, Arch, Ubuntu etc then let's
> > here it. For now, the strength of Gentoo is being able to fully
> customise a
> > system to your own requirements, not being trapped by some distro
> > maintainer's arbitrary choices. Play to your USP's and strengths rather
> > than chasing rainbows ..
> >
>
> Why do we support binary packages at all?  Simple: compiling packages
> is expensive, and if you happen to already have them compiled, fully
> customized to your own requirements, then there is no point in
> recompiling them.  You're just spending a ton of resources to build
> the exact same files you already have.
>
> The only change I'm suggesting is that portage could take all the
> configuration you're already supplying, and then optionally go see if
> somebody you trust has already built the package that meets your
> requirements.  If so, then it would be downloaded and installed,
> otherwise it would just compile from source.
>

> You get the exact same files installed on your system either way.
>

I think this conversation is a bit off track. I'm not saying this isn't a
great idea, but I think its very orthogonal to the binpkg format itself.

For example, the binhost pkg index file can contain this metadata and
portage can be designed to fetch the binpkg index metadata and do matching
(afaik it already does this; it just needs extending with more metadata.)
The binpkg format itself seems not too relevant to this.

-A


>
> --
> Rich
>
>


Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format

2018-11-11 Thread Alec Warner
On Sun, Nov 11, 2018 at 3:29 AM Michał Górny  wrote:

> On Sat, 2018-11-10 at 09:37 -0500, Alec Warner wrote:
> > On Sat, Nov 10, 2018 at 8:09 AM Michał Górny  wrote:
> >
> > > Hi, everyone.
> > >
> > > The Gentoo's tbz2/xpak package format is quite old.  We've made a few
> > > incompatible changes in the past (most notably, allowing non-bzip2
> > > compression and multi-instance naming) but the core design stayed
> > > the same.  I think we should consider changing it, for the reasons
> > > outlined below.
> > >
> > > The rough format description can be found in xpak(5).  Basically, it's
> > > a regular compressed tarball with binary metadata blob appended
> > > to the end.  As such, it looks like a regular compressed tarball
> > > to the compression tools (with some ignored junk at the end).
> > > The metadata is entirely custom format and needs dedicated tools
> > > to manipulate.
> > >
> > >
> > > The current format has a few advantages whose preserving would probably
> > > be worthwhile:
> > >
> > > + The binary package is a single flat file.
> > >
> > > + It is reasonably compatible with regular compressed tarball,
> > > so the users can unpack it using standard tools (except for metadata).
> > >
> > > + The metadata is uncompressed and can be quickly found without
> touching
> > > the compressed data.
> > >
> > > + The metadata can be updated (e.g. as result of pkgmove) without
> > > touching the compressed data.
> > >
> > >
> > > However, it has a few disadvantages as well:
> > >
> > > - The metadata is entirely custom binary format, requiring dedicated
> > > tools to read or edit.
> > >
> > > - The metadata format is relying on customary behavior of compression
> > > tools that ignore junk following the compressed data.
> > >
> >
> > I agree this is a problem in theory, but I haven't seen it as a problem
> in
> > practice. Have you observed any problems around this setup?
>
> Historically one of the parallel compressor variants did not support
> this.
>
> > >
> > > - By placing the metadata at the end of file, we make it rather hard to
> > > read the metadata from remote location (via FTP, HTTP) without fetching
> > > the whole file.  [NB: it's technically possible but probably not worth
> > > the effort]
> >
> >
> > > - By requiring the custom format to be at the end of file, we make it
> > > impossible to trivially cover it with a OpenPGP signature without
> > > introducing another custom format.
> > >
> >
> > Its trivial to cover with a detached sig, no?
> >
> >
> > >
> > > - While the format might allow for some extensibility, it's rather
> > > evolutionary dead end.
> > >
> >
> > I'm not even sure how to quantify this, it just sounds like your
> subjective
> > opinion (which is fine, but its not factual.)
> >
> >
> > >
> > >
> > > I think the key points of the new format should be:
> > >
> > > 1. It should reuse common file formats as much as possible, with
> > > inventing as little custom code as possible.
> > >
> > > 2. It should allow for easy introspection and editing by users without
> > > dedicated tools.
> > >
> >
> > So I'm less confident in the editing use cases; do users edit their
> binpkgs
> > on a regular basis?
>
> It's useful for debugging stuff.  I had to use hexedit on xpak
> in the past.  Believe me, it's nowhere close to pleasant.
>

> > > 3. The metadata should allow for lookup without fetching the whole
> > > binary package.
> > >
> > > 4. The format should allow for some extensions without having to
> > > reinvent the wheel every time.
> > >
> > > 5. It would be nice to preserve the existing advantages.
> > >
> > >
> > > My proposal
> > > ===
> > >
> > > Basic format
> > > 
> > > The base of the format is a regular compressed tarball.  There's no
> junk
> > > appended to it but the metadata is stored inside it as
> > > /var/db/pkg/${PF}.  The contents are as compatible with the actual vdb
> > > format as possible.
> > >
> >
> > Just to clarify, you are suggesting we store the metadata inside the
> > contents of the binary package itself (e.g. where the other file

Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format

2018-11-10 Thread Alec Warner
On Sat, Nov 10, 2018 at 8:09 AM Michał Górny  wrote:

> Hi, everyone.
>
> The Gentoo's tbz2/xpak package format is quite old.  We've made a few
> incompatible changes in the past (most notably, allowing non-bzip2
> compression and multi-instance naming) but the core design stayed
> the same.  I think we should consider changing it, for the reasons
> outlined below.
>
> The rough format description can be found in xpak(5).  Basically, it's
> a regular compressed tarball with binary metadata blob appended
> to the end.  As such, it looks like a regular compressed tarball
> to the compression tools (with some ignored junk at the end).
> The metadata is entirely custom format and needs dedicated tools
> to manipulate.
>
>
> The current format has a few advantages whose preserving would probably
> be worthwhile:
>
> + The binary package is a single flat file.
>
> + It is reasonably compatible with regular compressed tarball,
> so the users can unpack it using standard tools (except for metadata).
>
> + The metadata is uncompressed and can be quickly found without touching
> the compressed data.
>
> + The metadata can be updated (e.g. as result of pkgmove) without
> touching the compressed data.
>
>
> However, it has a few disadvantages as well:
>
> - The metadata is entirely custom binary format, requiring dedicated
> tools to read or edit.
>
> - The metadata format is relying on customary behavior of compression
> tools that ignore junk following the compressed data.
>

I agree this is a problem in theory, but I haven't seen it as a problem in
practice. Have you observed any problems around this setup?


>
> - By placing the metadata at the end of file, we make it rather hard to
> read the metadata from remote location (via FTP, HTTP) without fetching
> the whole file.  [NB: it's technically possible but probably not worth
> the effort]


> - By requiring the custom format to be at the end of file, we make it
> impossible to trivially cover it with a OpenPGP signature without
> introducing another custom format.
>

Its trivial to cover with a detached sig, no?


>
> - While the format might allow for some extensibility, it's rather
> evolutionary dead end.
>

I'm not even sure how to quantify this, it just sounds like your subjective
opinion (which is fine, but its not factual.)


>
>
> I think the key points of the new format should be:
>
> 1. It should reuse common file formats as much as possible, with
> inventing as little custom code as possible.
>
> 2. It should allow for easy introspection and editing by users without
> dedicated tools.
>

So I'm less confident in the editing use cases; do users edit their binpkgs
on a regular basis?


>
> 3. The metadata should allow for lookup without fetching the whole
> binary package.
>
> 4. The format should allow for some extensions without having to
> reinvent the wheel every time.
>
> 5. It would be nice to preserve the existing advantages.
>
>
> My proposal
> ===
>
> Basic format
> 
> The base of the format is a regular compressed tarball.  There's no junk
> appended to it but the metadata is stored inside it as
> /var/db/pkg/${PF}.  The contents are as compatible with the actual vdb
> format as possible.
>

Just to clarify, you are suggesting we store the metadata inside the
contents of the binary package itself (e.g. where the other files that get
merged to the liveFS are?) What about collisions?

E.g. I install 'machine-images/gentoo-disk-image-1.2.3' on a machine that
already has 'machine-images/gentoo-disk-image-1.2.3' installed, won't it
overwrite files in the VDB at qmerge time?


>
> This has the following advantages:
>
> + Binary package is still stored as a single file.
>
> + It uses a standard compressed .tar format, with minimal customization.
>
> + The user can easily inspect and modify the packages with standard
> tools (tar and the compressor).
>
> + If we can maintain reasonable level of vdb compatibility, the user can
> even emergency-install a package without causing too much hassle (as it
> will be recorded in vdb); ideally Portage would detect this vdb entry
> and support fixing the install afterwards.
>

I'm not certain this is really desired.


>
>
> Optimizing for easy recognition
> ---
> In order to make it possible for magic-based tools such as file(1) to
> easily distinguish Gentoo binary packages from regular tarballs, we
> could (ab)use the volume label field, e.g. use:
>
>   $ tar -V 'gpkg: app-foo/bar-1' -c ...
>
> This will add a volume label as the first file entry inside the tarball,
> which does not affect extracting but can be trivially matched via magic
> rules.
>
> Note: this is meant to be used as a method for fast binary package
> recognition; I don't think we should reject (hand-modified) binary
> packages that lack this label.
>
>
> Optimizing for metadata reading/manipulation performance
> 
> The main problem with 

Re: [gentoo-portage-dev] Objections to renaming pym directory to lib?

2018-07-18 Thread Alec Warner
On Wed, Jul 18, 2018 at 4:36 PM, Michał Górny  wrote:

> W dniu śro, 18.07.2018 o godzinie 13∶51 -0400, użytkownik Alec Warner
> napisał:
> > On Wed, Jul 18, 2018 at 2:54 AM, Brian Dolbec  wrote:
> >
> > > On Tue, 17 Jul 2018 13:28:05 -0700
> > > Zac Medico  wrote:
> > >
> > > > Are there any objections to renaming the pym directory to lib [1]?
> > > > Note that the git log --follow option makes this kind of rename
> > > > fairly painless.
> > > >
> > > > [1] https://github.com/gentoo/portage/pull/343
> > >
> > > is fine with me
> > >
> >
> > Are we leaving symlinks; or we think (non-portage) tools won't need to
> care
> > about this name change?
> >
>
> It affects only the source tree; it doesn't affect installed Portage.
>

Ah, then go for it ;)


>
> --
> Best regards,
> Michał Górny
>


Re: [gentoo-portage-dev] Objections to renaming pym directory to lib?

2018-07-18 Thread Alec Warner
On Wed, Jul 18, 2018 at 2:54 AM, Brian Dolbec  wrote:

> On Tue, 17 Jul 2018 13:28:05 -0700
> Zac Medico  wrote:
>
> > Are there any objections to renaming the pym directory to lib [1]?
> > Note that the git log --follow option makes this kind of rename
> > fairly painless.
> >
> > [1] https://github.com/gentoo/portage/pull/343
>
> is fine with me
>

Are we leaving symlinks; or we think (non-portage) tools won't need to care
about this name change?

-A


Re: [gentoo-portage-dev] [PATCH] global_event_loop: use asyncio event loop (bug 654390)

2018-05-08 Thread Alec Warner
lgtm.

-A

On Tue, May 8, 2018 at 11:43 AM, Brian Dolbec  wrote:

> On Mon,  7 May 2018 03:18:32 -0700
> Zac Medico  wrote:
>
> > + # This attribute it used by _wrap_loop to detect if
> > the
>
> typo, s/it/is
>
> this asynio stuff is mostly beyond my familiarity... but all looks ok,
> and I know you ran the tests on it...
>
> should be good to merge unless anyone else objects...
>
> --
> Brian Dolbec 
>
>
>


Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

2018-04-01 Thread Alec Warner
On Sun, Apr 1, 2018 at 2:29 PM, Michał Górny  wrote:

> W dniu nie, 01.04.2018 o godzinie 08∶59 -0700, użytkownik Zac Medico
> napisał:
> > On 04/01/2018 03:57 AM, Michał Górny wrote:
> > > W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico
> > > napisał:
> > > > Since key refresh is prone to failure, retry using exponential
> > > > backoff with random jitter. This adds the following sync-openpgp-*
> > > > configuration settings:
> > > >
> > > > sync-openpgp-key-refresh-retry-count = 40
> > > >
> > > >   Maximum number of times to retry key refresh if it fails.  Between
> > > >   each  key  refresh attempt, there is an exponential delay with a
> > > >   constant multiplier and a uniform random multiplier between 0 and
> 1.
> > > >
> > > > sync-openpgp-key-refresh-retry-delay-exp-base = 2
> > > >
> > > >   The base of the exponential expression. The exponent is the number
> > > >   of previous refresh attempts.
> > > >
> > > > sync-openpgp-key-refresh-retry-delay-max = 60
> > > >
> > > >   Maximum  delay between each retry attempt, in units of seconds.
> This
> > > >   places a limit on the length of the exponential delay.
> > > >
> > > > sync-openpgp-key-refresh-retry-delay-mult = 4
> > > >
> > > >   Multiplier for the exponential delay.
> > > >
> > > > sync-openpgp-key-refresh-retry-overall-timeout = 1200
> > > >
> > > >   Combined time limit for all refresh attempts, in units of seconds.
> > > >
> > > > Bug: https://bugs.gentoo.org/649276
> > > >
> > > > Zac Medico (4):
> > > >   Add ForkExecutor (bug 649588)
> > > >   Add ExponentialBackoff and RandomExponentialBackoff
> > > >   Add retry decorator (API inspired by tenacity)
> > > >   rsync: add key refresh retry (bug 649276)
> > > >
> > > >  cnf/repos.conf|   5 +
> > > >  man/portage.5 |  19 +++
> > > >  pym/portage/repository/config.py  |  22 
> > > >  pym/portage/sync/modules/rsync/rsync.py   |  16 ++-
> > > >  pym/portage/sync/syncbase.py  |  85 +++-
> > > >  pym/portage/tests/util/futures/test_retry.py  | 147
> +
> > > >  pym/portage/util/_eventloop/EventLoop.py  |  45 ++-
> > > >  pym/portage/util/backoff.py   |  48 +++
> > > >  pym/portage/util/futures/executor/__init__.py |   0
> > > >  pym/portage/util/futures/executor/fork.py | 130
> +++
> > > >  pym/portage/util/futures/futures.py   |   6 +
> > > >  pym/portage/util/futures/retry.py | 178
> ++
> > > >  12 files changed, 697 insertions(+), 4 deletions(-)
> > > >  create mode 100644 pym/portage/tests/util/futures/test_retry.py
> > > >  create mode 100644 pym/portage/util/backoff.py
> > > >  create mode 100644 pym/portage/util/futures/executor/__init__.py
> > > >  create mode 100644 pym/portage/util/futures/executor/fork.py
> > > >  create mode 100644 pym/portage/util/futures/retry.py
> > > >
> > >
> > > This essentially looks like ~700 lines of code to try to workaround
> > > broken networking. I would rather try to do that using 5 lines of code
> > > but that's just me, and my programs aren't enterprise quality. I just
> > > hope it actually solves as many problems as it's going to introduce.
> >
> > The vast majority of this code is generic and reusable, and I do intend
> > to reuse it. For example, the executor support will be an essential
> > piece for the asyncio.AbstractEventLoop for bug 649588.
>
> Sure it is and sure you will. But tell me: who is going to maintain it
> all? Because as far as I can see, we're still dealing with a bus factor
> of one and all you're doing is making it worse. More code, more
> complexity, more creeping featurism and hacks.
>

I'll split this reply into two sorts of points. One is a brief point about
this patchset. The other is a larger point about the project and its
direction.

About this patchset:

I don't mind this code because its not domain specific and I can read it
and understand it. Many apps need async ways to run code, and anyone who
has used the futures API will find this code familiar. I reviewed it in
about 10 minutes. I'm not heavily invested with the async-wagon that Zac is
driving, but I also see how its an area where performance can be improved
by doing more stuff at once using an async operations. I think if you have
more generic concerns with the async frameworks I'd love to see some
discussion about them (particularly around how we can improve perf by using
tighter algorithms; often the 'perf' boost is just higher CPU utilization
and driving multi-core systems which results in better walltime perf, but
not necessarily cpu-perf.)

Its fine to disagree here. I think the project has some requirements around
minimized dependencies, so the choice is either to pull in some kind of
retry-lib and add a dep, or add this code. Its also not immediately clear
if the 3rd party deps would still not 

Re: [gentoo-portage-dev] [PATCH 3/4] Add retry decorator (API inspired by tenacity)

2018-04-01 Thread Alec Warner
also lgtm.

On Sat, Mar 31, 2018 at 10:46 PM, Zac Medico  wrote:

> This decorator will be useful for retrying asynchronous
> operations, such as gpg key refresh (bug 649276). The
> API is inspired by tenacity, but is simpler. Only
> asynchronous functions (like @asyncio.coroutine functions)
> are supported. In order to retry a synchronous function,
> first convert it to an asynchronous function as follows:
>
> asynchronous_func = functools.partial(
> loop.run_in_executor, None, synchronous_func)
>
> Bug: https://bugs.gentoo.org/649276
> See: https://github.com/jd/tenacity
> ---
>  pym/portage/tests/util/futures/test_retry.py | 147 ++
>  pym/portage/util/futures/futures.py  |   6 +
>  pym/portage/util/futures/retry.py| 178
> +++
>  3 files changed, 331 insertions(+)
>  create mode 100644 pym/portage/tests/util/futures/test_retry.py
>  create mode 100644 pym/portage/util/futures/retry.py
>
> diff --git a/pym/portage/tests/util/futures/test_retry.py
> b/pym/portage/tests/util/futures/test_retry.py
> new file mode 100644
> index 0..7641e4e92
> --- /dev/null
> +++ b/pym/portage/tests/util/futures/test_retry.py
> @@ -0,0 +1,147 @@
> +# Copyright 2018 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import functools
> +
> +try:
> +   import threading
> +except ImportError:
> +   import dummy_threading as threading
> +
> +from portage.tests import TestCase
> +from portage.util._eventloop.global_event_loop import global_event_loop
> +from portage.util.backoff import RandomExponentialBackoff
> +from portage.util.futures.futures import TimeoutError
> +from portage.util.futures.retry import retry
> +from portage.util.futures.wait import wait
> +from portage.util.monotonic import monotonic
> +
> +
> +class SucceedLaterException(Exception):
> +   pass
> +
> +
> +class SucceedLater(object):
> +   """
> +   A callable object that succeeds some duration of time has passed.
> +   """
> +   def __init__(self, duration):
> +   self._succeed_time = monotonic() + duration
> +
> +   def __call__(self):
> +   remaining = self._succeed_time - monotonic()
> +   if remaining > 0:
> +   raise SucceedLaterException('time until success:
> {} seconds'.format(remaining))
> +   return 'success'
> +
> +
> +class SucceedNeverException(Exception):
> +   pass
> +
> +
> +class SucceedNever(object):
> +   """
> +   A callable object that never succeeds.
> +   """
> +   def __call__(self):
> +   raise SucceedNeverException('expected failure')
> +
> +
> +class HangForever(object):
> +   """
> +   A callable object that sleeps forever.
> +   """
> +   def __call__(self):
> +   threading.Event().wait()
> +
> +
> +class RetryTestCase(TestCase):
> +   def testSucceedLater(self):
> +   loop = global_event_loop()
> +   func = SucceedLater(1)
> +   func_coroutine = functools.partial(loop.run_in_executor,
> None, func)
> +   decorator = retry(try_max=,
> +   delay_func=RandomExponentialBackoff(multiplier=0.1,
> base=2))
> +   decorated_func = decorator(func_coroutine)
> +   result = loop.run_until_complete(decorated_func())
> +   self.assertEqual(result, 'success')
> +
> +   def testSucceedNever(self):
> +   loop = global_event_loop()
> +   func = SucceedNever()
> +   func_coroutine = functools.partial(loop.run_in_executor,
> None, func)
> +   decorator = retry(try_max=4, try_timeout=None,
> +   delay_func=RandomExponentialBackoff(multiplier=0.1,
> base=2))
> +   decorated_func = decorator(func_coroutine)
> +   done, pending = loop.run_until_complete(wait([
> decorated_func()]))
> +   self.assertEqual(len(done), 1)
> +   self.assertTrue(isinstance(done[0].exception().__cause__,
> SucceedNeverException))
> +
> +   def testSucceedNeverReraise(self):
> +   loop = global_event_loop()
> +   func = SucceedNever()
> +   func_coroutine = functools.partial(loop.run_in_executor,
> None, func)
> +   decorator = retry(reraise=True, try_max=4,
> try_timeout=None,
> +   delay_func=RandomExponentialBackoff(multiplier=0.1,
> base=2))
> +   decorated_func = decorator(func_coroutine)
> +   done, pending = loop.run_until_complete(wait([
> decorated_func()]))
> +   self.assertEqual(len(done), 1)
> +   self.assertTrue(isinstance(done[0].exception(),
> SucceedNeverException))
> +
> +   def testHangForever(self):
> +   loop = global_event_loop()
> +   func = HangForever()
> +   

Re: [gentoo-portage-dev] [PATCH 1/4] Add ForkExecutor (bug 649588)

2018-04-01 Thread Alec Warner
lgtm.

On Sat, Mar 31, 2018 at 10:46 PM, Zac Medico  wrote:

> This is useful for asynchronous operations that we might
> need to cancel if they take too long, since (concurrent.
> futures.ProcessPoolExecutor tasks are not cancellable).
> This ability to cancel tasks makes this executor useful
> as an alternative to portage.exception.AlarmSignal.
>
> Also add an asyncio-compatible EventLoop.run_in_executor
> method the uses ForkExecutor as the default executor,
> which will later be used to implement the corresponding
> asyncio.AbstractEventLoop run_in_executor method.
>
> Bug: https://bugs.gentoo.org/649588
> ---
>  pym/portage/util/_eventloop/EventLoop.py  |  45 -
>  pym/portage/util/futures/executor/__init__.py |   0
>  pym/portage/util/futures/executor/fork.py | 130
> ++
>  3 files changed, 174 insertions(+), 1 deletion(-)
>  create mode 100644 pym/portage/util/futures/executor/__init__.py
>  create mode 100644 pym/portage/util/futures/executor/fork.py
>
> diff --git a/pym/portage/util/_eventloop/EventLoop.py
> b/pym/portage/util/_eventloop/EventLoop.py
> index f472a3dae..1574a6837 100644
> --- a/pym/portage/util/_eventloop/EventLoop.py
> +++ b/pym/portage/util/_eventloop/EventLoop.py
> @@ -24,6 +24,7 @@ except ImportError:
>  import portage
>  portage.proxy.lazyimport.lazyimport(globals(),
> 'portage.util.futures.futures:_EventLoopFuture',
> +   'portage.util.futures.executor.fork:ForkExecutor',
>  )
>
>  from portage import OrderedDict
> @@ -122,6 +123,7 @@ class EventLoop(object):
> self._idle_callbacks = OrderedDict()
> self._timeout_handlers = {}
> self._timeout_interval = None
> +   self._default_executor = None
>
> self._poll_obj = None
> try:
> @@ -721,6 +723,46 @@ class EventLoop(object):
> return self._handle(self.timeout_add(
> delay * 1000, self._call_soon_callback(callback,
> args)), self)
>
> +   def run_in_executor(self, executor, func, *args):
> +   """
> +   Arrange for a func to be called in the specified executor.
> +
> +   The executor argument should be an Executor instance. The
> default
> +   executor is used if executor is None.
> +
> +   Use functools.partial to pass keywords to the *func*.
> +
> +   @param executor: executor
> +   @type executor: concurrent.futures.Executor or None
> +   @param func: a function to call
> +   @type func: callable
> +   @return: a Future
> +   @rtype: asyncio.Future (or compatible)
> +   """
> +   if executor is None:
> +   executor = self._default_executor
> +   if executor is None:
> +   executor = ForkExecutor(loop=self)
> +   self._default_executor = executor
> +   return executor.submit(func, *args)
> +
> +   def close(self):
> +   """Close the event loop.
> +
> +   This clears the queues and shuts down the executor,
> +   and waits for it to finish.
> +   """
> +   executor = self._default_executor
> +   if executor is not None:
> +   self._default_executor = None
> +   executor.shutdown(wait=True)
> +
> +   if self._poll_obj is not None:
> +   close = getattr(self._poll_obj, 'close')
> +   if close is not None:
> +   close()
> +   self._poll_obj = None
> +
>
>  _can_poll_device = None
>
> @@ -782,10 +824,11 @@ class _epoll_adapter(object):
> that is associated with an epoll instance will close automatically
> when
> it is garbage collected, so it's not necessary to close it
> explicitly.
> """
> -   __slots__ = ('_epoll_obj',)
> +   __slots__ = ('_epoll_obj', 'close')
>
> def __init__(self, epoll_obj):
> self._epoll_obj = epoll_obj
> +   self.close = epoll_obj.close
>
> def register(self, fd, *args):
> self._epoll_obj.register(fd, *args)
> diff --git a/pym/portage/util/futures/executor/__init__.py
> b/pym/portage/util/futures/executor/__init__.py
> new file mode 100644
> index 0..e69de29bb
> diff --git a/pym/portage/util/futures/executor/fork.py
> b/pym/portage/util/futures/executor/fork.py
> new file mode 100644
> index 0..9cd1db2ca
> --- /dev/null
> +++ b/pym/portage/util/futures/executor/fork.py
> @@ -0,0 +1,130 @@
> +# Copyright 2018 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import collections
> +import functools
> +import multiprocessing
> +import os
> +import sys
> +import traceback
> +
> +from 

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

2018-03-15 Thread Alec Warner
On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny  wrote:

> Allow INSTALL_MASK patterns to start with '-' to indicate that
> a specific match is to be excluded from being masked. In this case,
> the last matching pattern determines whether the file is actually
> filtered out or kept.
> ---
>  pym/portage/dbapi/vartree.py | 10 ++
>  1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> index 21904edca..16c246b11 100644
> --- a/pym/portage/dbapi/vartree.py
> +++ b/pym/portage/dbapi/vartree.py
> @@ -3692,19 +3692,21 @@ class dblink(object):
> def _is_install_masked(self, relative_path):
> ret = False
> for pattern in self.settings.install_mask:
>
+   # if pattern starts with -, possibly exclude this
> path
> +   pat_res = not pattern.startswith('-')
> +   if not pat_res:
> +   pattern = pattern[1:]
>

Maybe consider:

pattern = pattern[1:] if pattern.startswith('-') else pattern

I'm not super keen on this pattern in python, but it seems doable here.


> # absolute path pattern
> if pattern.startswith('/'):
> # match either exact path or one of parent
> dirs
> # the latter is done via matching pattern/*
> if (fnmatch.fnmatch(relative_path,
> pattern[1:])
> or
> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
> -   ret = True
> -   break
> +   ret = pat_res
> # filename
> else:
> if 
> fnmatch.fnmatch(os.path.basename(relative_path),
> pattern):
> -   ret = True
> -   break
> +   ret = pat_res
> return ret
>
> def treewalk(self, srcroot, destroot, inforoot, myebuild,
> cleanup=0,
> --
> 2.16.2
>
>
>


Re: [gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

2018-02-26 Thread Alec Warner
On Mon, Feb 26, 2018 at 1:09 PM, Zac Medico  wrote:

> On 02/26/2018 04:29 AM, Michał Górny wrote:
> > W dniu nie, 25.02.2018 o godzinie 17∶50 -0800, użytkownik Zac Medico
> > napisał:
> >> Add async_aux_get method that returns a Future and otherwise
> >> behaves identically to aux_get. Use async_aux_get to implement
> >> the synchronous aux_get method.
> >>
> >> Bug: https://bugs.gentoo.org/648790
> >> ---
> >>  pym/portage/dbapi/porttree.py | 91 ++
> +++--
> >>  1 file changed, 70 insertions(+), 21 deletions(-)
> >>
> >
> > What's the exact use case? What's the gain, in numbers?
>
> I'm planning to use the for repoman, since it commonly has to generate
> metadata for multiple packages a once, when dealing with ebuild or
> eclass modifications.
>
> The gain in numbers is that it scales linearly with the number of
> available cores.
>

Daniel, any data on real world gains from the Funtoo implementation?


>
> > This seems like a lot of added complexity.
>
> It doesn't really add complexity, asynchronous code or this sort is
> already used throughout portage internals. Asynchronous methods like
> these are quite standard today, thanks to asyncio.
>
> > With Portage being practically unmaintainable,
>
> Any reasonably complex codebase appears to be unmaintainable to those
> who don't maintain it.
>

I mean I'm generally with mgorny, but I don't think this change is that
complicated.

The 'hard to maintain' stuff for me comes from:

1) really long functions (e.g >100 lines) with poor naming (x, my*
variables, etc.)
2) portions of the codebase with no tests (more than I'd like...)
3) The random spawl and layout of some components that just makes it
annoying to modify specific bits.

While I agree that these are all a problem (and I've tried at various times
to tilt at these windmills to no avail)
I don't think this makes any of those worse.


>
> > I'm against any changes that make things worse without
> > clearly defined major benefit.
>
> Transitioning to asynchronous interfaces like this is unavoidable as we
> move into the realm of asyncio compatibility.
>

So to me this implies we will be adding even more asyncIO to try to improve
the performance of portage by utilizing more cores?

-A


> --
> Thanks,
> Zac
>
>


Re: [gentoo-portage-dev] [PATCH 2/2] Add iter_completed convenience function (bug 648790)

2018-02-25 Thread Alec Warner
On Sun, Feb 25, 2018 at 8:50 PM, Zac Medico  wrote:

> The iter_completed function is similar to asyncio.as_completed, but
> takes an iterator of futures as input, and includes support for
> max_jobs and max_load parameters. The default values for max_jobs
> and max_load correspond to multiprocessing.cpu_count().
>
> Example usage for async_aux_get:
>
>   import portage
>   from portage.util.futures.iter_completed import iter_completed
>
>   portdb = portage.portdb
>   future_cpv = {}
>

I'm not sure I grasp the purpose of this dict, can't we just modify the
async aux get to return the cpv from the future?


>
>   def future_generator():
> for cpv in portdb.cp_list('sys-apps/portage'):
>   future = portdb.async_aux_get(cpv, portage.auxdbkeys)
>   future_cpv[id(future)] = cpv
>   yield future
>
>
for cpv in portdb.cp_list('...'):
   yield portdb.async_aux_get(cpv, portage.auxdbkeys)


>   for future in iter_completed(future_generator()):
> cpv = future_cpv.pop(id(future))
> try:
>   result = future.result()
> except KeyError as e:
>   # aux_get failed
>   print('error:', cpv, e)
> else:
>   print(cpv, result)
>

for future in iter_completed(future_generator()):
  try:
cpv, result = future.result()
  except KeyError as e:
print('error', cpv, e)


Or do we expect callers to need other things to key off of in this API?

-A


> See: https://docs.python.org/3/library/asyncio-task.html#
> asyncio.as_completed
> Bug: https://bugs.gentoo.org/648790
> ---
>  .../tests/util/futures/test_iter_completed.py  | 50 
>  pym/portage/util/_async/FuturePollTask.py  | 27 ++
>  pym/portage/util/futures/iter_completed.py | 63 ++
>  pym/portage/util/futures/wait.py   | 95
> ++
>  4 files changed, 235 insertions(+)
>  create mode 100644 pym/portage/tests/util/futures/test_iter_completed.py
>  create mode 100644 pym/portage/util/_async/FuturePollTask.py
>  create mode 100644 pym/portage/util/futures/iter_completed.py
>  create mode 100644 pym/portage/util/futures/wait.py
>
> diff --git a/pym/portage/tests/util/futures/test_iter_completed.py
> b/pym/portage/tests/util/futures/test_iter_completed.py
> new file mode 100644
> index 0..6607d871c
> --- /dev/null
> +++ b/pym/portage/tests/util/futures/test_iter_completed.py
> @@ -0,0 +1,50 @@
> +# Copyright 2018 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import time
> +from portage.tests import TestCase
> +from portage.util._async.ForkProcess import ForkProcess
> +from portage.util._eventloop.global_event_loop import global_event_loop
> +from portage.util.futures.iter_completed import iter_completed
> +
> +
> +class SleepProcess(ForkProcess):
> +   __slots__ = ('future', 'seconds')
> +   def _start(self):
> +   self.addExitListener(self._future_done)
> +   ForkProcess._start(self)
> +
> +   def _future_done(self, task):
> +   self.future.set_result(self.seconds)
> +
> +   def _run(self):
> +   time.sleep(self.seconds)
> +
> +
> +class IterCompletedTestCase(TestCase):
> +
> +   def testIterCompleted(self):
> +
> +   # Mark this as todo, since we don't want to fail if heavy
> system
> +   # load causes the tasks to finish in an unexpected order.
> +   self.todo = True
> +
> +   loop = global_event_loop()
> +   tasks = [
> +   SleepProcess(seconds=0.200),
> +   SleepProcess(seconds=0.100),
> +   SleepProcess(seconds=0.001),
> +   ]
> +
> +   expected_order = sorted(task.seconds for task in tasks)
> +
> +   def future_generator():
> +   for task in tasks:
> +   task.future = loop.create_future()
> +   task.scheduler = loop
> +   task.start()
> +   yield task.future
> +
> +   for seconds, future in zip(expected_order,
> iter_completed(future_generator(),
> +   max_jobs=None, max_load=None, loop=loop)):
> +   self.assertEqual(seconds, future.result())
> diff --git a/pym/portage/util/_async/FuturePollTask.py
> b/pym/portage/util/_async/FuturePollTask.py
> new file mode 100644
> index 0..6b7cdf7d5
> --- /dev/null
> +++ b/pym/portage/util/_async/FuturePollTask.py
> @@ -0,0 +1,27 @@
> +# Copyright 2018 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import os
> +import signal
> +
> +from _emerge.AbstractPollTask import AbstractPollTask
> +
> +
> +class FuturePollTask(AbstractPollTask):
> +   """
> +   Wraps a Future in an AsynchronousTask, which is useful for
> +   scheduling with TaskScheduler.
> +   """
> +

Re: [gentoo-portage-dev] [PATCH v2] filter-bash-environment.py: use buffered input, raw bytes (bug 647654)

2018-02-15 Thread Alec Warner
On Thu, Feb 15, 2018 at 12:49 AM, Zac Medico  wrote:

> Use sys.stdin.buffer instead of sys.stdin.buffer.raw, for buffered input.
> Also use raw bytes instead of unicode strings, in order to avoid making
> assumptions about character encodings, and also to avoid overhead from
> unicode decoding/encoding.
>
> Since the % operator does not support bytes operands in python3.4, use
> the + operator to format strings of bytes.
>
> Bug: https://bugs.gentoo.org/647654
> ---
> [PATCH v2] changes:
> * don't use % operator with bytes operands, for python3.4 compat
> * add unit test that compares expected output to actual output
>

Sorry to be unclear; if there is existing test coverage via emerge I think
that is fine; I was just worried there were none.

I need to learn to check travis-ci for these pullreqs.


>
>  bin/filter-bash-environment.py|  47 +--
>  pym/portage/tests/bin/test_filter_bash_env.py | 113
> ++
>  2 files changed, 135 insertions(+), 25 deletions(-)
>  create mode 100644 pym/portage/tests/bin/test_filter_bash_env.py
>
> diff --git a/bin/filter-bash-environment.py b/bin/filter-bash-environment.
> py
> index a4cdc5429..668aa7452 100755
> --- a/bin/filter-bash-environment.py
> +++ b/bin/filter-bash-environment.py
> @@ -2,21 +2,19 @@
>  # Copyright 1999-2014 Gentoo Foundation
>  # Distributed under the terms of the GNU General Public License v2
>
> -import codecs
> -import io
>  import os
>  import re
>  import sys
>
> -here_doc_re = re.compile(r'.*\s<<[-]?(\w+)$')
> -func_start_re = re.compile(r'^[-\w]+\s*\(\)\s*$')
> -func_end_re = re.compile(r'^\}$')
> +here_doc_re = re.compile(br'.*\s<<[-]?(\w+)$')
> +func_start_re = re.compile(br'^[-\w]+\s*\(\)\s*$')
> +func_end_re = re.compile(br'^\}$')
>
> -var_assign_re = re.compile(r'(^|^declare\s+-\
> S+\s+|^declare\s+|^export\s+)([^=\s]+)=("|\')?.*$')
> -close_quote_re = re.compile(r'(\\"|"|\')\s*$')
> -readonly_re = re.compile(r'^declare\s+-(\S*)r(\S*)\s+')
> +var_assign_re = re.compile(br'(^|^declare\s+-\
> S+\s+|^declare\s+|^export\s+)([^=\s]+)=("|\')?.*$')
> +close_quote_re = re.compile(br'(\\"|"|\')\s*$')
> +readonly_re = re.compile(br'^declare\s+-(\S*)r(\S*)\s+')
>  # declare without assignment
> -var_declare_re = re.compile(r'^declare(\s+-\S+)?\s+([^=\s]+)\s*$')
> +var_declare_re = re.compile(br'^declare(\s+-\S+)?\s+([^=\s]+)\s*$')
>
>  def have_end_quote(quote, line):
> """
> @@ -32,16 +30,16 @@ def have_end_quote(quote, line):
>  def filter_declare_readonly_opt(line):
> readonly_match = readonly_re.match(line)
> if readonly_match is not None:
> -   declare_opts = ''
> +   declare_opts = b''
> for i in (1, 2):
> group = readonly_match.group(i)
> if group is not None:
> declare_opts += group
> if declare_opts:
> -   line = 'declare -%s %s' % \
> -   (declare_opts, line[readonly_match.end():])
> +   line = b'declare -' + declare_opts + \
> +   b' ' + line[readonly_match.end():]
> else:
> -   line = 'declare ' + line[readonly_match.end():]
> +   line = b'declare ' + line[readonly_match.end():]
> return line
>
>  def filter_bash_environment(pattern, file_in, file_out):
> @@ -57,7 +55,7 @@ def filter_bash_environment(pattern, file_in, file_out):
> for line in file_in:
> if multi_line_quote is not None:
> if not multi_line_quote_filter:
> -   file_out.write(line.replace("\1", ""))
> +   file_out.write(line.replace(b"\1", b""))
> if have_end_quote(multi_line_quote, line):
> multi_line_quote = None
> multi_line_quote_filter = None
> @@ -78,7 +76,7 @@ def filter_bash_environment(pattern, file_in, file_out):
> multi_line_quote_filter =
> filter_this
> if not filter_this:
> line = filter_declare_readonly_opt(
> line)
> -   file_out.write(line.replace("\1",
> ""))
> +   file_out.write(line.replace(b"\1",
> b""))
> continue
> else:
> declare_match = var_declare_re.match(line)
> @@ -98,7 +96,7 @@ def filter_bash_environment(pattern, file_in, file_out):
> continue
> here_doc = here_doc_re.match(line)
> if here_doc is not None:
> -   here_doc_delim = re.compile("^%s$" %
> here_doc.group(1))
> +   here_doc_delim = re.compile(b'^%s' +
> 

Re: [gentoo-portage-dev] [PATCH] filter-bash-environment.py: use buffered input, raw bytes (bug 647654)

2018-02-14 Thread Alec Warner
On Wed, Feb 14, 2018 at 3:38 PM, Zac Medico  wrote:

> Use sys.stdin.buffer instead of sys.stdin.buffer.raw, for buffered input.
> Also use raw bytes instead of unicode strings, in order to avoid making
> assumptions about character encodings, and also to avoid overhead from
> unicode decoding/encoding.
>

Maybe a functional test?

take real environment, check it into source code, run filter over it to get
output, run bash -n on output?

Better ideas?

-A


>
> Bug: https://bugs.gentoo.org/647654
> ---
>  bin/filter-bash-environment.py | 45 --
> 
>  1 file changed, 21 insertions(+), 24 deletions(-)
>
> diff --git a/bin/filter-bash-environment.py b/bin/filter-bash-environment.
> py
> index a4cdc5429..91c194b95 100755
> --- a/bin/filter-bash-environment.py
> +++ b/bin/filter-bash-environment.py
> @@ -2,21 +2,19 @@
>  # Copyright 1999-2014 Gentoo Foundation
>  # Distributed under the terms of the GNU General Public License v2
>
> -import codecs
> -import io
>  import os
>  import re
>  import sys
>
> -here_doc_re = re.compile(r'.*\s<<[-]?(\w+)$')
> -func_start_re = re.compile(r'^[-\w]+\s*\(\)\s*$')
> -func_end_re = re.compile(r'^\}$')
> +here_doc_re = re.compile(br'.*\s<<[-]?(\w+)$')
> +func_start_re = re.compile(br'^[-\w]+\s*\(\)\s*$')
> +func_end_re = re.compile(br'^\}$')
>
> -var_assign_re = re.compile(r'(^|^declare\s+-\
> S+\s+|^declare\s+|^export\s+)([^=\s]+)=("|\')?.*$')
> -close_quote_re = re.compile(r'(\\"|"|\')\s*$')
> -readonly_re = re.compile(r'^declare\s+-(\S*)r(\S*)\s+')
> +var_assign_re = re.compile(br'(^|^declare\s+-\
> S+\s+|^declare\s+|^export\s+)([^=\s]+)=("|\')?.*$')
> +close_quote_re = re.compile(br'(\\"|"|\')\s*$')
> +readonly_re = re.compile(br'^declare\s+-(\S*)r(\S*)\s+')
>  # declare without assignment
> -var_declare_re = re.compile(r'^declare(\s+-\S+)?\s+([^=\s]+)\s*$')
> +var_declare_re = re.compile(br'^declare(\s+-\S+)?\s+([^=\s]+)\s*$')
>
>  def have_end_quote(quote, line):
> """
> @@ -32,16 +30,16 @@ def have_end_quote(quote, line):
>  def filter_declare_readonly_opt(line):
> readonly_match = readonly_re.match(line)
> if readonly_match is not None:
> -   declare_opts = ''
> +   declare_opts = b''
> for i in (1, 2):
> group = readonly_match.group(i)
> if group is not None:
> declare_opts += group
> if declare_opts:
> -   line = 'declare -%s %s' % \
> +   line = b'declare -%s %s' % \
> (declare_opts, line[readonly_match.end():])
> else:
> -   line = 'declare ' + line[readonly_match.end():]
> +   line = b'declare ' + line[readonly_match.end():]
> return line
>
>  def filter_bash_environment(pattern, file_in, file_out):
> @@ -57,7 +55,7 @@ def filter_bash_environment(pattern, file_in, file_out):
> for line in file_in:
> if multi_line_quote is not None:
> if not multi_line_quote_filter:
> -   file_out.write(line.replace("\1", ""))
> +   file_out.write(line.replace(b"\1", b""))
> if have_end_quote(multi_line_quote, line):
> multi_line_quote = None
> multi_line_quote_filter = None
> @@ -78,7 +76,7 @@ def filter_bash_environment(pattern, file_in, file_out):
> multi_line_quote_filter =
> filter_this
> if not filter_this:
> line = filter_declare_readonly_opt(
> line)
> -   file_out.write(line.replace("\1",
> ""))
> +   file_out.write(line.replace(b"\1",
> b""))
> continue
> else:
> declare_match = var_declare_re.match(line)
> @@ -98,7 +96,7 @@ def filter_bash_environment(pattern, file_in, file_out):
> continue
> here_doc = here_doc_re.match(line)
> if here_doc is not None:
> -   here_doc_delim = re.compile("^%s$" %
> here_doc.group(1))
> +   here_doc_delim = re.compile(b"^%s$" %
> here_doc.group(1))
> file_out.write(line)
> continue
> # Note: here-documents are handled before functions since
> otherwise
> @@ -141,18 +139,17 @@ if __name__ == "__main__":
> file_in = sys.stdin
> file_out = sys.stdout
> if sys.hexversion >= 0x300:
> -   file_in = codecs.iterdecode(sys.stdin.buffer.raw,
> -   'utf_8', errors='replace')
> -   file_out = 

Re: [gentoo-portage-dev] [PATCH] emerge: add --changed-deps-report option (bug 645780)

2018-01-28 Thread Alec Warner
On Sun, Jan 28, 2018 at 8:10 PM, Zac Medico  wrote:

> On 01/28/2018 04:17 PM, Michael Orlitzky wrote:
> > Since ::gentoo is the only repository governed by the PMS, can't we make
> > repoman do this? The problem with requiring "repoman --force" for an
> > in-place dependency change is that repoman generally won't have access
> > to the unedited ebuild; but for ::gentoo, we can probably hack something
> > together in git. Have it source the old and new ebuilds, and compare
> > their dependency lists.
>
> Yes I'm in favor of that, but it doesn't accomplish my current goal, as
> I'll explain below.
>
> > It won't work outside of a git repo, but it will work in the one place
> > that really matters.
>
> My goal for this patch is to provide a safety net for users, such that
> they'll be automatically notified that the --changed-deps option is
> available when needed to solve problems involving stale dependencies.
> The reason behind this goal is that git commits flow directly to rsync
> without any restriction, which makes users vulnerable to stale
> dependencies, in the form of failed dependency calculations.
>

What I would go along with is:

1) If we can determine the depgraph failed and
2) We can determine that dependencies changed.
3) Show a NOTE telling users about --changed-deps=y

But I'm not sure the patch does this today? I think it will trigger too
aggressively. But ultimately I'm not against helping users work around this
sort of breakage.


> This goal is currently my highest priority, since I want to protect
> users from having difficulty with problems triggered by the new
> --dynamic-deps=n default [1].
>
> [1] https://bugs.gentoo.org/645550
> --
> Thanks,
> Zac
>
>


Re: [gentoo-portage-dev] [PATCH] emerge: add --changed-deps-report option (bug 645780)

2018-01-28 Thread Alec Warner
On Sun, Jan 28, 2018 at 1:09 PM, Zac Medico <zmed...@gentoo.org> wrote:

> On 01/28/2018 09:35 AM, Alec Warner wrote:
> >
> >
> > On Sun, Jan 28, 2018 at 9:51 AM, Zac Medico <zmed...@gentoo.org
> > <mailto:zmed...@gentoo.org>> wrote:
> >
> > The --dynamic-deps=n default causes confusion for users that are
> > accustomed to dynamic deps, therefore add a --changed-deps-report
> > option that is enabled by default (if --usepkgonly is not enabled).
> >
> > The --quiet option will suppress the report if none of the packages
> > having changed dependencies are in the dependency graph, since they
> > are harmless in that case. If any of these packages *are* in the
> > dependency graph, then --quiet suppresses the big NOTE section of
> > the report, but the HINT section is still displayed since it might
> > help users resolve problems that are solved by --changed-deps.
> >
> > Example output is as follows:
> >
> > !!! Detected ebuild dependency change(s) without revision bump:
> >
> > net-misc/openssh-7.5_p1-r3::gentoo
> > sys-fs/udisks-2.7.5::gentoo
> >
> > NOTE: For the ebuild(s) listed above, a new revision may be
> > warranted if there
> >   has been a dependency change with significant consequences.
> > Refer to the
> >   following page of the Gentoo Development Guide for examples of
> >   circumstances that may qualify:
>
>
> >
> > https://devmanual.gentoo.org/general-concepts/ebuild-revisions/
> > <https://devmanual.gentoo.org/general-concepts/ebuild-revisions/>
>
>
> >   If circumstances qualify, please report a bug which specifies
> > the current
> >   version of the ebuild listed above. Changes to ebuilds from
> > the 'gentoo'
> >   repository (ending with '::gentoo') can be browsed in GitWeb:
> >
> >   https://gitweb.gentoo.org/repo/gentoo.git/
> > <https://gitweb.gentoo.org/repo/gentoo.git/>
> >
> >   Use Gentoo's Bugzilla to report bugs only for the 'gentoo'
> > repository:
> >
> >   https://bugs.gentoo.org/
> >
> >   In order to suppress reports about dependency changes, add
> >   --changed-deps-report=n to the EMERGE_DEFAULT_OPTS variable in
> >   '/etc/portage/make.conf'.
> >
> > HINT: In order to avoid problems involving changed dependencies, use
> the
> >   --changed-deps option to automatically trigger rebuilds when
> > changed
> >   dependencies are detected. Refer to the emerge man page for
> more
> >   information about this option.
> >
> > Bug: https://bugs.gentoo.org/645780
> >
> >
> > I can't really support sending this large report to users.
> >
> > 1) Its fairly common practice today.
> > 2) All users will get the report.
> > 3) A subset of them will file bugs about the report.
> > 4) Devs make a decision about revbumping vs not; there doesn't seem to
> > be a way for devs to say "no this change is intentional, stop nagging
> > users."
>
> I think in practice we need to revbump for most changes. If the changes
> weren't worth propagating, then we wouldn't make them.
>
> > Ultimately I'm unsure what we are trying to accomplish here. Do we think
> > Devs are doing 4 on accident?
>
> It doesn't matter whether it's intentional or an accident. The problem
> is that users need to learn to react appropriately, and I think
> --change-deps-report is probably the most efficient way to train them.
>

I guess I'm less convinced users can / should do it.

Like they have to read the ebuild-revisions guide in the devmanual and
decide if the revbump is required or not?
Do we think users are able to do this?

Should we consider only showing reports to developers (e.g. putting it
behind a 'dev' FEATURE?)


>
> > If so, why can't we give them tools to find it ahead of time (repoman
> > et. al.) or tools to find it post-commit (tinderbox or similar; but
> > these are single-sourced reports.)
>
> Sure that can be done, but it will take some work. Honestly I think the
> devs can get it right without all that, they just need some
> reinforcement that has been absent up until now.
>
> Meanwhile, users need to be trained to cope with whatever comes their way.
>
> > I also feel like we are pushing action onto users here. If we agree with
> > the premise that this is a bug (and devs should always bump) then we
> > don't need 

Re: [gentoo-portage-dev] [PATCH] emerge: add --changed-deps-report option (bug 645780)

2018-01-28 Thread Alec Warner
On Sun, Jan 28, 2018 at 9:51 AM, Zac Medico  wrote:

> The --dynamic-deps=n default causes confusion for users that are
> accustomed to dynamic deps, therefore add a --changed-deps-report
> option that is enabled by default (if --usepkgonly is not enabled).
>
> The --quiet option will suppress the report if none of the packages
> having changed dependencies are in the dependency graph, since they
> are harmless in that case. If any of these packages *are* in the
> dependency graph, then --quiet suppresses the big NOTE section of
> the report, but the HINT section is still displayed since it might
> help users resolve problems that are solved by --changed-deps.
>
> Example output is as follows:
>
> !!! Detected ebuild dependency change(s) without revision bump:
>
> net-misc/openssh-7.5_p1-r3::gentoo
> sys-fs/udisks-2.7.5::gentoo
>
> NOTE: For the ebuild(s) listed above, a new revision may be warranted if
> there
>   has been a dependency change with significant consequences. Refer to
> the
>   following page of the Gentoo Development Guide for examples of
>   circumstances that may qualify:
>
>   https://devmanual.gentoo.org/general-concepts/ebuild-revisions/
>
>   If circumstances qualify, please report a bug which specifies the
> current
>   version of the ebuild listed above. Changes to ebuilds from the
> 'gentoo'
>   repository (ending with '::gentoo') can be browsed in GitWeb:
>
>   https://gitweb.gentoo.org/repo/gentoo.git/
>
>   Use Gentoo's Bugzilla to report bugs only for the 'gentoo'
> repository:
>
>   https://bugs.gentoo.org/
>
>   In order to suppress reports about dependency changes, add
>   --changed-deps-report=n to the EMERGE_DEFAULT_OPTS variable in
>   '/etc/portage/make.conf'.
>
> HINT: In order to avoid problems involving changed dependencies, use the
>   --changed-deps option to automatically trigger rebuilds when changed
>   dependencies are detected. Refer to the emerge man page for more
>   information about this option.
>
> Bug: https://bugs.gentoo.org/645780


I can't really support sending this large report to users.

1) Its fairly common practice today.
2) All users will get the report.
3) A subset of them will file bugs about the report.
4) Devs make a decision about revbumping vs not; there doesn't seem to be a
way for devs to say "no this change is intentional, stop nagging users."

Ultimately I'm unsure what we are trying to accomplish here. Do we think
Devs are doing 4 on accident?
If so, why can't we give them tools to find it ahead of time (repoman et.
al.) or tools to find it post-commit (tinderbox or similar; but these are
single-sourced reports.)

I also feel like we are pushing action onto users here. If we agree with
the premise that this is a bug (and devs should always bump) then we don't
need users to take any action;
we could build automation that sends these 'reports'. Its not like the user
is going to add anything interesting to the report. Making them do it by
hand just introduces transcription errors in filing. We just need to get
their permission to file the reports automatically when portage finds them.

-A


>
> ---
>  man/emerge.1  |  10 
>  pym/_emerge/create_depgraph_params.py |   5 ++
>  pym/_emerge/depgraph.py   | 103 ++
> +++-
>  pym/_emerge/main.py   |  13 +
>  4 files changed, 128 insertions(+), 3 deletions(-)
>
> diff --git a/man/emerge.1 b/man/emerge.1
> index 3c81b9c9f..9de1b7f74 100644
> --- a/man/emerge.1
> +++ b/man/emerge.1
> @@ -465,6 +465,16 @@ option also implies the \fB\-\-selective\fR option.
> Behavior with
>  respect to changed build\-time dependencies is controlled by the
>  \fB\-\-with\-bdeps\fR option.
>  .TP
> +.BR "\-\-changed\-deps\-report [ y | n ]"
> +Tells emerge to report ebuilds for which the ebuild dependencies have
> +changed since the installed instance was built. Behavior with respect to
> +changed build\-time dependencies is controlled by the
> +\fB\-\-with\-bdeps\fR option. This option is enabled automatically for
> +a dependency calculation if the cost of report generation is relatively
> +insignificant (any calculation exclusively involving binary packages is
> +exempt). The \fIEMERGE_DEFAULT_OPTS\fR variable may be used to disable
> +this option by default.
> +.TP
>  .BR \-\-changed\-use ", " \-U
>  Tells emerge to include installed packages where USE flags have
>  changed since installation. This option also implies the
> diff --git a/pym/_emerge/create_depgraph_params.py
> b/pym/_emerge/create_depgraph_params.py
> index cdea029ba..7d01610bb 100644
> --- a/pym/_emerge/create_depgraph_params.py
> +++ b/pym/_emerge/create_depgraph_params.py
> @@ -126,6 +126,11 @@ def create_depgraph_params(myopts, myaction):
> if changed_deps is not None:
> myparams['changed_deps'] = changed_deps
>
> +   

Re: [gentoo-portage-dev] Plan for initial integration of gemato with portage

2018-01-24 Thread Alec Warner
On Wed, Jan 24, 2018 at 2:58 PM, Michał Górny <mgo...@gentoo.org> wrote:

> W dniu śro, 24.01.2018 o godzinie 12∶54 -0500, użytkownik Alec Warner
> napisał:
> > On Wed, Jan 24, 2018 at 3:56 AM, Michał Górny <mgo...@gentoo.org> wrote:
> >
> > > Hi, everyone.
> > >
> > > Since the initial review of my patch lost focus, and lacked sufficient
> > > context, here's the plan that I'd like to follow in order to initially
> > > integrate gemato with portage and give our users secure checkouts by
> > > default.
> > >
> > > 1. Add postsync hook to Portage git. Eventually, it will be replaced by
> > > direct Portage support.
> > >
> > > 2. Add IUSE=+rsync-verify to portage- that controls installing the
> > > hook. This will give users the ability to easily disable it without
> jumping
> > > through cross package hoops.
> > >
> >
> > I think it makes sense to avoid installing the hook through this means
> > (e.g. I don't want it, so I set USE=-rsync-verify)
> >
> > I think its a bit trickier to control the hook's behavior. For instance:
> >
> > 1) I install portage[rsync-verify]. This installs the hook.
> > 2) I end up not liking the hook, I install portage[-rsync-verify]
> > 3) Does the hook get config-protected here?
>
> Keeping config-protected files applies only if the file were modified.
> In this case it just gets unmerged. I've just verified that.
>
> > > 3. Submit a news item for review that will explain how to initially
> verify
> > > the keys on existing installations.
> > >
> > > The news item would be published when the hook hits a release.
> > >
> > > What do you think? If you agree, then I'll start writing the news item.
> > >
> >
> > The other part of the user story I don't understand is what actions
> should
> > users take when verification legit fails?
> >
> > 1) file a bug?
> > 2) re-sync their tree?
> > 3) something else?
>
> I'd say one re-sync would be reasonable in case it could have been some
> rsync glitch. Beyond that, it would definitely be worth reporting...
> except the user will probably hit another mirror.
>

This sgtm.

If the number of reports is high we can build more complex reporting
infrastructure (e.g. to locate bad mirrors.)

-A


>
> --
> Best regards,
> Michał Górny
>
>
>


Re: [gentoo-portage-dev] Plan for initial integration of gemato with portage

2018-01-24 Thread Alec Warner
On Wed, Jan 24, 2018 at 3:56 AM, Michał Górny  wrote:

> Hi, everyone.
>
> Since the initial review of my patch lost focus, and lacked sufficient
> context, here's the plan that I'd like to follow in order to initially
> integrate gemato with portage and give our users secure checkouts by
> default.
>
> 1. Add postsync hook to Portage git. Eventually, it will be replaced by
> direct Portage support.
>

> 2. Add IUSE=+rsync-verify to portage- that controls installing the
> hook. This will give users the ability to easily disable it without jumping
> through cross package hoops.
>

I think it makes sense to avoid installing the hook through this means
(e.g. I don't want it, so I set USE=-rsync-verify)

I think its a bit trickier to control the hook's behavior. For instance:

1) I install portage[rsync-verify]. This installs the hook.
2) I end up not liking the hook, I install portage[-rsync-verify]
3) Does the hook get config-protected here?
4) I run emerge --depclean # gemato is un-merged here.

Now the hook fails because gemato is gone but the hook remains due to
config-protect?

5) I run etc-update and the hook is deleted; I don't recall etc-update
deleting files...

So basically I'm curious how the hook works w/config-protect; but
admittedly its been a while since I worried about such things so I could be
wrong.

Other considerations include:

1) Hooks don't matter anyway, so we may not care if it breaks.
2) Verification doesn't matter currently (portage will install software
regardless), so we may not care if it doesn't function.


>
> 3. Submit a news item for review that will explain how to initially verify
> the keys on existing installations.
>
> The news item would be published when the hook hits a release.
>
> What do you think? If you agree, then I'll start writing the news item.
>

The other part of the user story I don't understand is what actions should
users take when verification legit fails?

1) file a bug?
2) re-sync their tree?
3) something else?


--
> Best regards,
> Michał Górny (by phone)
>
>


Re: [gentoo-portage-dev] [PATCH 8/8] eshowkw: Always group Prefix keywords last

2018-01-23 Thread Alec Warner
On Tue, Jan 23, 2018 at 4:48 PM, Michał Górny  wrote:

> Always group all Prefix keywords after other types of keywords. This
> not only ensures that fbsd sorts first but more importantly stabilizes
> the LHS output between regular and -P variant -- that is, -P always adds
> additional keywords at the end.
> ---
>  pym/gentoolkit/eshowkw/keywords_header.py | 16 ++--
>  1 file changed, 10 insertions(+), 6 deletions(-)
>
> diff --git a/pym/gentoolkit/eshowkw/keywords_header.py
> b/pym/gentoolkit/eshowkw/keywords_header.py
> index 41b8ba4..1b64bfd 100644
> --- a/pym/gentoolkit/eshowkw/keywords_header.py
> +++ b/pym/gentoolkit/eshowkw/keywords_header.py
> @@ -142,12 +142,16 @@ class keywords_header:
> break
>
> # sort by, in order (to match Bugzilla):
> -   # 1. arch, then ~arch
> -   # 2. profile stability
> -   # 3. short keywords, then long (prefix, fbsd)
> -   # 4. keyword name in reverse component order
> -   normal.sort(key=lambda kw: (kw in self.__TESTING_KW_ARCHS,
> -   levels.get(kw, 99), kw.count('-'),
> list(reversed(kw.split('-')
> +   # 1. non-prefix, then prefix (stable output between -P and
> not)
> +   # 2. arch, then ~arch
> +   # 3. profile stability
> +   # 4. short keywords, then long (prefix, fbsd)
> +   # 5. keyword name in reverse component order
> +   normal.sort(key=lambda kw: (self.__isPrefix(kw),
> +   kw in self.__TESTING_KW_ARCHS,
> +   levels.get(kw, 99),
> +   kw.count('-'),
> +   list(reversed(kw.split('-')
>

I'm a bit sad about this lambda because its ended up a bit long.

What are your thoughts on splitting it out?

return normal
>
> def __readAdditionalFields(self):
> --
> 2.16.1
>
>
>


Re: [gentoo-portage-dev] [PATCH] dep_zapdeps: fix virtual/rust handling (bug 645416)

2018-01-23 Thread Alec Warner
I'm starting to be a bit sad about this for a couple of reasons.

1) dep_zapdeps is pretty big at this point, and every tweak for || () just
grows it. Do we have any idea:
  a) When dep_zapdeps will be 'done'?
  b) Plans for splitting it into more manageable pieces?
2) In general I worry that the logic for || () will essentially never be
complete, and we will be forced to tweak the implementation forever as
cases arise where the algorithm fails to compute the 'right' answer.
I'm a bit curious what 'right' even is anymore. The devmanual doesn't
really describe how any-of dependencies are resolved and so the changes
seem rather arbitrary. I am not worried about specific changes (like this
one) but the fact that we continually apply changes to make the 'right'
thing happen is the larger concern.

-A

On Tue, Jan 23, 2018 at 2:17 AM, Zac Medico  wrote:

> Fix the code from bug 643974 to set the installed_downgrade flag only
> if the selected newer package is either installed or in the graph. This
> is currently needed for appropriate handling of virtual/rust-1.19.0,
> since there's an upgrade to dev-lang/rust-1.23.0 available which may
> not be desired since it would mean that dev-lang/rust-bin-1.19.0 has
> to be installed in order to satisfy virtual/rust-1.19.0:
>
>  || ( =dev-lang/rust-1.19.0* =dev-lang/rust-bin-1.19.0* )
>
> So, the rust-bin choice is desirable only if the rust-1.23.0 package is
> already installed or in the graph.
>
> Fixes: 86ba22da7a2f ("dep_zapdeps: install new package, allow upgrade (bug
> 643974)")
> Bug: https://bugs.gentoo.org/645416
> ---
>  pym/portage/dep/dep_check.py   | 10 ++--
>  .../tests/resolver/test_or_upgrade_installed.py| 59
> ++
>  2 files changed, 66 insertions(+), 3 deletions(-)
>
> diff --git a/pym/portage/dep/dep_check.py b/pym/portage/dep/dep_check.py
> index c56f545ec..7e5a3186e 100644
> --- a/pym/portage/dep/dep_check.py
> +++ b/pym/portage/dep/dep_check.py
> @@ -463,11 +463,15 @@ def dep_zapdeps(unreduced, reduced, myroot,
> use_binaries=0, trees=None):
> avail_pkg = avail_pkg_use
> avail_slot = Atom("%s:%s" %
> (atom.cp, avail_pkg.slot))
>
> -   if downgrade_probe is not None:
> +   if downgrade_probe is not None and graph is not
> None:
> highest_in_slot =
> mydbapi_match_pkgs(avail_slot)
> +   highest_in_slot = (highest_in_slot[-1]
> +   if highest_in_slot else None)
> if (avail_pkg and highest_in_slot and
> -   avail_pkg < highest_in_slot[-1] and
> -   not downgrade_probe(avail_pkg)):
> +   avail_pkg < highest_in_slot and
> +   not downgrade_probe(avail_pkg) and
> +   (highest_in_slot.installed or
> +   highest_in_slot in graph)):
> installed_downgrade = True
>
> slot_map[avail_slot] = avail_pkg
> diff --git a/pym/portage/tests/resolver/test_or_upgrade_installed.py
> b/pym/portage/tests/resolver/test_or_upgrade_installed.py
> index 6e01d321d..7018e08de 100644
> --- a/pym/portage/tests/resolver/test_or_upgrade_installed.py
> +++ b/pym/portage/tests/resolver/test_or_upgrade_installed.py
> @@ -99,3 +99,62 @@ class OrUpgradeInstalledTestCase(TestCase):
> finally:
> playground.debug = False
> playground.cleanup()
> +
> +   def testVirtualRust(self):
> +   ebuilds = {
> +   'dev-lang/rust-1.19.0': {},
> +   'dev-lang/rust-1.23.0': {},
> +   'dev-lang/rust-bin-1.19.0': {},
> +   'virtual/rust-1.19.0': {
> +   'RDEPEND': '|| ( =dev-lang/rust-1.19.0*
> =dev-lang/rust-bin-1.19.0* )'
> +   },
> +   }
> +
> +   installed = {
> +   'dev-lang/rust-1.19.0': {},
> +   'virtual/rust-1.19.0': {
> +   'RDEPEND': '|| ( =dev-lang/rust-1.19.0*
> =dev-lang/rust-bin-1.19.0* )'
> +   },
> +   }
> +
> +   world = ['virtual/rust']
> +
> +   test_cases = (
> +   # Test bug 645416, where rust-bin-1.19.0 was
> pulled in
> +   # inappropriately due to the rust-1.23.0 update
> being
> +   # available.
> +   ResolverPlaygroundTestCase(
> +   ['virtual/rust'],
> +   options={'--update': True, '--deep': True},
> +

Re: [gentoo-portage-dev] [PATCH] dep_zapdeps: exclude virtuals from new_slot_count (bug 645190)

2018-01-20 Thread Alec Warner
On Sat, Jan 20, 2018 at 7:41 PM, Zac Medico <zmed...@gentoo.org> wrote:

> On 01/20/2018 04:23 PM, Alec Warner wrote:
> > On Sat, Jan 20, 2018 at 7:10 PM, Zac Medico <zmed...@gentoo.org
> > <mailto:zmed...@gentoo.org>> wrote:
> >
> > diff --git a/pym/portage/dep/dep_check.py
> b/pym/portage/dep/dep_check.py
> > index 7cf338819..c56f545ec 100644
> > --- a/pym/portage/dep/dep_check.py
> > +++ b/pym/portage/dep/dep_check.py
> > @@ -499,7 +499,8 @@ def dep_zapdeps(unreduced, reduced, myroot,
> > use_binaries=0, trees=None):
> > cp_map[avail_pkg.cp] = avail_pkg
> >
> > new_slot_count = (len(slot_map) if graph_db is None
> else
> > -   sum(not graph_db.match_pkgs(slot_atom) for
> > slot_atom in slot_map))
> > +   sum(not graph_db.match_pkgs(slot_atom) for
> > slot_atom in slot_map
> > +   if not slot_atom.cp.startswith("virtual/")))
> >
> >
> > I see this logic all over dep_zapdeps. But AFAIK its already using
> > instances of Atom here.
> > Can't we encode the cost inside of Atom, so that we don't need to carve
> > out that virtual atoms are 0 cost?
> >
> > Then we could write something like:
> >
> > new_slot_count = len(slot_map) if graph_db is None else
> >   sum(slot_atom.cost() for slot_atom in slot_map if not
> > graph_db.match_pkgs(slot_atom))
> >
> > Then virtuals would just have a cost() of 0. And others could have other
> > costs (or just 1.)
>
> That seems like a very practical approach. However, the cost is actually
> a property of the matched package rather than the atom itself. It's only
> because of GLEP 37 that we can treat the "virtual" category specially,
> but GLEP 37 actually says nothing about the "virtual" category! There's
> also a java-virtuals category!


> This mess was the motivation for my PROPERTIES=virtual suggestion:
>
> https://archives.gentoo.org/gentoo-dev/message/
> 9d449a18a96a25a547fcfd40544085cf
>
> If we implement something like PROPERTIES=virtual, then cost becomes a
> property of the package instance rather than the atom.
>

So I have two goals with my comments:

1) Avoid code repetition. If we are checking "if
foo.startswith('virtual/')" a bunch of places; it might be useful to factor
that out into either a helper:

def is_virtual(atom):
  return atom.startswith('virtual/')

Or attach this directly to the Atom class so callers can just do
atom.is_virtual()

I can kind of see why the latter might be icky (because its not necessarily
an intrinsic part of the Atom interface.)
I'm not sure its necessary to wait for PROPERTIES=virtual support to make
this change; its simply refactoring existing logic.

2) Avoid 'needing to know about virtuals'. I concede this is likely
difficult with the current state of things. the dbapi.match calls don't
return Packages; so we can't
rely on packages to communicate state. This ends up with code complexity at
call sites (e.g. if not match(atom) and isVirtual(atom) # action). I
concede this isn't
worth fixing right now.

-A


--
> Thanks,
> Zac
>
>


Re: [gentoo-portage-dev] [PATCH] dep_zapdeps: exclude virtuals from new_slot_count (bug 645190)

2018-01-20 Thread Alec Warner
On Sat, Jan 20, 2018 at 7:10 PM, Zac Medico  wrote:

> Fix new_slot_count to exclude virtual packages, since they are considered
> to have zero-cost. This solves an issue where the catalyst stage1 build
> would unexpectedly pull in static-dev to satisfy virtual/dev-manager,
> but eudev is the preferred choice.
>

> Bug: https://bugs.gentoo.org/645190
> Fixes: 9fdaf9bdbdf5 ("dep_check: use DNF to optimize overlapping virtual
> || deps (bug 632026)")
> Reported-by: Ben Kohler 
> ---
>  pym/portage/dep/dep_check.py   |  3 +-
>  .../resolver/test_virtual_minimize_children.py | 61
> ++
>  2 files changed, 63 insertions(+), 1 deletion(-)
>
> diff --git a/pym/portage/dep/dep_check.py b/pym/portage/dep/dep_check.py
> index 7cf338819..c56f545ec 100644
> --- a/pym/portage/dep/dep_check.py
> +++ b/pym/portage/dep/dep_check.py
> @@ -499,7 +499,8 @@ def dep_zapdeps(unreduced, reduced, myroot,
> use_binaries=0, trees=None):
> cp_map[avail_pkg.cp] = avail_pkg
>
> new_slot_count = (len(slot_map) if graph_db is None else
> -   sum(not graph_db.match_pkgs(slot_atom) for
> slot_atom in slot_map))
> +   sum(not graph_db.match_pkgs(slot_atom) for
> slot_atom in slot_map
> +   if not slot_atom.cp.startswith("virtual/")))
>

I see this logic all over dep_zapdeps. But AFAIK its already using
instances of Atom here.
Can't we encode the cost inside of Atom, so that we don't need to carve out
that virtual atoms are 0 cost?

Then we could write something like:

new_slot_count = len(slot_map) if graph_db is None else
  sum(slot_atom.cost() for slot_atom in slot_map if not
graph_db.match_pkgs(slot_atom))

Then virtuals would just have a cost() of 0. And others could have other
costs (or just 1.)


>
> this_choice = _dep_choice(atoms=atoms, slot_map=slot_map,
> cp_map=cp_map, all_available=all_available,
> diff --git a/pym/portage/tests/resolver/test_virtual_minimize_children.py
> b/pym/portage/tests/resolver/test_virtual_minimize_children.py
> index 6eb0409f2..287445e58 100644
> --- a/pym/portage/tests/resolver/test_virtual_minimize_children.py
> +++ b/pym/portage/tests/resolver/test_virtual_minimize_children.py
> @@ -226,3 +226,64 @@ class VirtualMinimizeChildrenTestCase(TestCase):
> finally:
> playground.debug = False
> playground.cleanup()
> +
> +   def testVirtualDevManager(self):
> +   ebuilds = {
> +   'sys-fs/eudev-3.1.5': {},
> +   'sys-fs/static-dev-0.1': {},
> +   'sys-fs/udev-233': {},
> +   'virtual/dev-manager-0': {
> +   'RDEPEND': '''
> +   || (
> +   virtual/udev
> +   sys-fs/static-dev
> +   )'''
> +   },
> +   'virtual/udev-0': {
> +   'RDEPEND': '''
> +   || (
> +   >=sys-fs/eudev-2.1.1
> +   >=sys-fs/udev-217
> +   )'''
> +   },
> +   }
> +
> +   test_cases = (
> +   # Test bug 645190, where static-dev was pulled in
> instead
> +   # of eudev.
> +   ResolverPlaygroundTestCase(
> +   [
> +   'virtual/dev-manager',
> +   ],
> +   success=True,
> +   mergelist=(
> +   'sys-fs/eudev-3.1.5',
> +   'virtual/udev-0',
> +   'virtual/dev-manager-0',
> +   ),
> +   ),
> +   # Test static-dev preference.
> +   ResolverPlaygroundTestCase(
> +   [
> +   'sys-fs/static-dev',
> +   'virtual/dev-manager',
> +   ],
> +   all_permutations=True,
> +   success=True,
> +   mergelist=(
> +   'sys-fs/static-dev-0.1',
> +   'virtual/dev-manager-0',
> +   ),
> +   ),
> +   )
> +
> +   playground = ResolverPlayground(debug=False,
> ebuilds=ebuilds)
> +
> +  

Re: [gentoo-portage-dev] [PATCH] ebuild: allow RESTRICT=network-sandbox in ebuilds

2018-01-17 Thread Alec Warner
On Wed, Jan 17, 2018 at 1:33 PM, Mike Frysinger  wrote:

> On 16 Jan 2018 23:32, Mike Gilbert wrote:
> > On Tue, Jan 16, 2018 at 4:46 PM, Mike Frysinger wrote:
> > > Some ebuilds are a bit hard to fix their use of the network in src
> > > phases, so allow them to disable things.  This allows us to turn off
> > > access by default and for the vast majority while we work out how to
> > > fix the few broken packages.
> >
> > If we are going to allow network sandboxing to be disabled in
> > individual ebuilds, we should also allow the other sandboxes to be
> > disabled for the same reasons. sys-apps/sandbox has been notoriously
> > buggy, for example.
>
> that sandbox can already be disabled dynamically in ebuilds as needed.
> i don't really see a reason for it to be a RESTRICT value.
>

It used to be in RESTRICT even; it was dropped in 2015.

https://archives.gentoo.org/gentoo-pms/message/05f8faf4f1477b4406619b92bb7722b7

I don't mind if portage acts on other tokens (allowed in the spec). I do
think we should file a patch to PMS.
If they accept it, other PMs can implement this feature in a future EAPI
(and in portage, it will just work in all EAPIs)

But if we never tell them about the tokens, they can never implement them.

-A

-mike
>


Re: [gentoo-portage-dev] [PATCH v2] misc: Distribute a repo.postsync.d hook to run gemato verification

2018-01-17 Thread Alec Warner
On Wed, Jan 17, 2018 at 10:25 AM, Michał Górny <mgo...@gentoo.org> wrote:

> W dniu wto, 16.01.2018 o godzinie 11∶32 -0800, użytkownik Zac Medico
> napisał:
> > On 01/16/2018 10:39 AM, Michał Górny wrote:
> > > W dniu wto, 16.01.2018 o godzinie 12∶44 -0500, użytkownik Alec Warner
> > > napisał:
> > > > On Tue, Jan 16, 2018 at 11:43 AM, Michał Górny <mgo...@gentoo.org>
> wrote:
> > > >
> > > > > Include a repo.postsync.d hook to verify the rsync checkout using
> > > > > gemato. Given that not all people will want to have it enabled
> > > > > unconditionally, no setup.py rules are included -- instead, the
> file
> > > > > would be installed conditionally by the ebuild.
> > > > >
> > > > > [v2: included link to the wiki page]
> > > > > ---
> > > > >  MANIFEST.in   |  2 +-
> > > > >  misc/repo.postsync.d/00gemato | 18 ++
> > > > >  2 files changed, 19 insertions(+), 1 deletion(-)
> > > > >  create mode 100644 misc/repo.postsync.d/00gemato
> > > > >
> > > > > diff --git a/MANIFEST.in b/MANIFEST.in
> > > > > index 4f6cac162..edc6704e7 100644
> > > > > --- a/MANIFEST.in
> > > > > +++ b/MANIFEST.in
> > > > > @@ -14,4 +14,4 @@ include cnf/make.conf.example.*
> > > > >  include .portage_not_installed
> > > > >
> > > > >  # extra scripts
> > > > > -include misc/*
> > > > > +graft misc
> > > > > diff --git a/misc/repo.postsync.d/00gemato b/misc/repo.postsync.d/
> 00gemato
> > > > > new file mode 100644
> > > > > index 0..f2af50925
> > > > > --- /dev/null
> > > > > +++ b/misc/repo.postsync.d/00gemato
> > > > > @@ -0,0 +1,18 @@
> > > > > +#!/bin/bash
> > > > > +# repo.postsync.d hook to verify ::gentoo checkout using gemato
> > > > > +
> > > > > +name=${1}
> > > > > +url=${2}
> > > > > +path=${3}
> > > > > +
> > > > > +# keyring installed by gentoo-keys
> > > > > +openpgp_key=/var/lib/gentoo/gkeys/keyrings/gentoo/release/
> pubring.gpg
> > > > >
> > > >
> > > > This seems a bit leaky to me.
> > > >
> > > > Possible to get gentoo-keys to print it?
> > > >
> > > > e.g:
> > > >
> > > > openpgp_key=$(gentoo-keys --print-key-path)
> > >
> > > But app-crypt/gentoo-keys doesn't include that executable, and it has
> > > no dependency on app-crypt/gkeys. I'd rather not introduce an
> artificial
> > > dependency here.
> >
> > I suppose we could using a separate ebuild to install this hook, so that
> > we can update it separately from portage if necessary. The hook can
> > still live in the portage repository (like emerge-delta-webrsync which
> > is also installed by a separate ebuild).
>
> I don't see a strong reason to add yet another rebuild for a single file
> that is going to be updated really rarely. However, if we're going to do
> it that way, then there's no point in putting it in Portage repository.
>
> However, this 'update it separately from portage' reminds me of repoman
> that frequently gets seriously outdated and/or incompatible with Portage
> because of independent release cycle...
>

I'll rephrase my objection.

I don't care what you do as long as Zac (the person releasing portage)
agrees with whatever
requirements you need. If we need 3 releases in a row because the hook is
buggy, as long as
Zac is happy with that I'm happy with that.

What I don't want to see is surprise when the hook is cut and suddenly its
buggy and we need new
cuts and Zac is not around, or HEAD is broken, or some other problem.

Looking at the release history, multiple cuts in O(few) days is fairly
common (11/20, 11/21, 12/10, 12/15)
so this seems low risk to me; but AFAIK Zac is usually driving these
changes himself so its a bit more obvious
what is going on. Or just allow Michał to cut his own portage releases when
he needs hook updates.

-A


>
> --
> Best regards,
> Michał Górny
>
>
>


Re: [gentoo-portage-dev] [PATCH v2] misc: Distribute a repo.postsync.d hook to run gemato verification

2018-01-16 Thread Alec Warner
On Tue, Jan 16, 2018 at 11:43 AM, Michał Górny  wrote:

> Include a repo.postsync.d hook to verify the rsync checkout using
> gemato. Given that not all people will want to have it enabled
> unconditionally, no setup.py rules are included -- instead, the file
> would be installed conditionally by the ebuild.
>
> [v2: included link to the wiki page]
> ---
>  MANIFEST.in   |  2 +-
>  misc/repo.postsync.d/00gemato | 18 ++
>  2 files changed, 19 insertions(+), 1 deletion(-)
>  create mode 100644 misc/repo.postsync.d/00gemato
>
> diff --git a/MANIFEST.in b/MANIFEST.in
> index 4f6cac162..edc6704e7 100644
> --- a/MANIFEST.in
> +++ b/MANIFEST.in
> @@ -14,4 +14,4 @@ include cnf/make.conf.example.*
>  include .portage_not_installed
>
>  # extra scripts
> -include misc/*
> +graft misc
> diff --git a/misc/repo.postsync.d/00gemato b/misc/repo.postsync.d/00gemato
> new file mode 100644
> index 0..f2af50925
> --- /dev/null
> +++ b/misc/repo.postsync.d/00gemato
> @@ -0,0 +1,18 @@
> +#!/bin/bash
> +# repo.postsync.d hook to verify ::gentoo checkout using gemato
> +
> +name=${1}
> +url=${2}
> +path=${3}
> +
> +# keyring installed by gentoo-keys
> +openpgp_key=/var/lib/gentoo/gkeys/keyrings/gentoo/release/pubring.gpg
>

This seems a bit leaky to me.

Possible to get gentoo-keys to print it?

e.g:

openpgp_key=$(gentoo-keys --print-key-path)

Or its fine if we think it won't move around. I guess that is mostly up to
dol-sen? :)

-A


> +
> +# apply only to ::gentoo, when synced over rsync.
> +if [[ ${name} == gentoo && ${url} == rsync://* ]]; then
> +   if ! gemato verify -K "${openpgp_key}" -s "${path}"; then
> +   eerror "For troubleshooting the verification failures,
> please see:"
> +   eerror "  https://wiki.gentoo.org/wiki/
> Project:Portage/Repository_Verification"
> +   return 1
> +   fi
> +fi
> --
> 2.16.0.rc2
>
>
>


Re: [gentoo-portage-dev] [PATCH] dep_zapdeps: install new package, allow upgrade (bug 643974)

2018-01-10 Thread Alec Warner
The code lgtm.

Slightly unsure if we need an entirely new test, as opposed to just
generalizing the test cases (like you did the code). But either way is
probably fine.

-A

On Tue, Jan 9, 2018 at 2:55 PM, Zac Medico  wrote:

> Prefer to install a new package in order to allow upgrade of an
> installed package. This generalizes the code from bug 635540 so
> that it both allows desirable upgrades and prevents unwanted
> downgrades.
>
> Fixes: 7c58e3737616 ("dep_zapdeps: install new package, avoid downgrade
> (bug 635540)")
> Bug: https://bugs.gentoo.org/643974
> ---
>  pym/portage/dep/dep_check.py   | 11 ++-
>  .../tests/resolver/test_or_upgrade_installed.py| 94
> ++
>  2 files changed, 99 insertions(+), 6 deletions(-)
>  create mode 100644 pym/portage/tests/resolver/
> test_or_upgrade_installed.py
>
> diff --git a/pym/portage/dep/dep_check.py b/pym/portage/dep/dep_check.py
> index 2bb9dc339..291626f56 100644
> --- a/pym/portage/dep/dep_check.py
> +++ b/pym/portage/dep/dep_check.py
> @@ -366,10 +366,8 @@ def dep_zapdeps(unreduced, reduced, myroot,
> use_binaries=0, trees=None):
> want_update_pkg = trees[myroot].get("want_update_pkg")
> downgrade_probe = trees[myroot].get("downgrade_probe")
> vardb = None
> -   vardb_match_pkgs = None
> if "vartree" in trees[myroot]:
> vardb = trees[myroot]["vartree"].dbapi
> -   vardb_match_pkgs = getattr(vardb, 'match_pkgs', None)
> if use_binaries:
> mydbapi = trees[myroot]["bintree"].dbapi
> else:
> @@ -465,10 +463,11 @@ def dep_zapdeps(unreduced, reduced, myroot,
> use_binaries=0, trees=None):
> avail_pkg = avail_pkg_use
> avail_slot = Atom("%s:%s" %
> (atom.cp, avail_pkg.slot))
>
> -   if vardb_match_pkgs is not None and
> downgrade_probe is not None:
> -   inst_pkg = vardb_match_pkgs(avail_slot)
> -   if (inst_pkg and avail_pkg < inst_pkg[-1]
> and
> -   not downgrade_probe(inst_pkg[-1]))
> :
> +   if downgrade_probe is not None:
> +   highest_in_slot =
> mydbapi_match_pkgs(avail_slot)
> +   if (avail_pkg and highest_in_slot and
> +   avail_pkg < highest_in_slot[-1] and
> +   not downgrade_probe(avail_pkg)):
> installed_downgrade = True
>
> slot_map[avail_slot] = avail_pkg
> diff --git a/pym/portage/tests/resolver/test_or_upgrade_installed.py
> b/pym/portage/tests/resolver/test_or_upgrade_installed.py
> new file mode 100644
> index 0..70703a614
> --- /dev/null
> +++ b/pym/portage/tests/resolver/test_or_upgrade_installed.py
> @@ -0,0 +1,94 @@
> +# Copyright 2018 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +from portage.tests import TestCase
> +from portage.tests.resolver.ResolverPlayground import (
> +   ResolverPlayground,
> +   ResolverPlaygroundTestCase,
> +)
> +
> +class OrUpgradeInstalledTestCase(TestCase):
> +
> +   def testOrUpgradeInstalled(self):
> +   ebuilds = {
> +   'net-misc/foo-1': {
> +   'EAPI': '6',
> +   'RDEPEND': '|| ( sys-libs/glibc[rpc(-)]
> net-libs/libtirpc )'
> +   },
> +   'net-libs/libtirpc-1': {
> +   'EAPI': '6',
> +   },
> +   'sys-libs/glibc-2.26': {
> +   'EAPI': '6',
> +   'IUSE': ''
> +   },
> +   'sys-libs/glibc-2.24': {
> +   'EAPI': '6',
> +   'IUSE': '+rpc'
> +   },
> +   }
> +
> +   installed = {
> +   'sys-libs/glibc-2.24': {
> +   'EAPI': '6',
> +   'IUSE': '+rpc',
> +   'USE': 'rpc',
> +   },
> +   }
> +
> +   world = ['sys-libs/glibc']
> +
> +   test_cases = (
> +   # Test bug 643974, where we need to install
> libtirpc
> +   # in order to upgrade glibc.
> +   ResolverPlaygroundTestCase(
> +   ['net-misc/foo', '@world'],
> +   options={'--update': True, '--deep': True},
> +   success=True,
> +   mergelist=['net-libs/libtirpc-1',
> 'sys-libs/glibc-2.26', 'net-misc/foo-1']
> +   

Re: [gentoo-portage-dev] [PATCH] install-qa-check: Do not install empty directories

2018-01-10 Thread Alec Warner
Please link to documentation on keepdir (e.g.
https://devmanual.gentoo.org/function-reference/install-functions/index.html)
in the helper?

I'm not sure there is better keepdir document?

Also looking at https://devmanual.gentoo.org/eclass-reference/ebuild/

It says "keepdir functions the same as dodir" but this has not been true
for a while?

-A



On Wed, Jan 10, 2018 at 11:48 AM, Michał Górny  wrote:

> Remove empty directories in install-qa-check phase in order to prevent
> Portage from installing them, and therefore from developers relying
> on them being installed.
>
> The PMS specifies the behavior upon merging empty directories
> as undefined, and specifically prohibits ebuilds from attempting
> to install empty directories. However, ebuilds occasionally still fall
> into the trap of relying on 'dodir' preserving the directory. Make
> the Portage behavior more strict in order to prevent that.
>
> Use 'install-qa-check.d' machinery for this as this provides an easy way
> for users to restore the old behavior (by overriding the check) if they
> need it for their private ebuilds. It also makes it possible to extend
> the check with some QA warnings, if we figure out how to do them.
>
> Currently no QA warnings are emitted as we do not want to pursue
> upstream build systems that create empty directories but the packages
> themselves do not rely on them being installed, e.g. when some files
> are installed into the directory conditionally.
> ---
>  bin/install-qa-check.d/95empty-dirs | 21 +
>  1 file changed, 21 insertions(+)
>  create mode 100644 bin/install-qa-check.d/95empty-dirs
>
> diff --git a/bin/install-qa-check.d/95empty-dirs b/bin/install-qa-check.d/
> 95empty-dirs
> new file mode 100644
> index 0..6b8790f59
> --- /dev/null
> +++ b/bin/install-qa-check.d/95empty-dirs
> @@ -0,0 +1,21 @@
> +# Remove empty directories installed by ebuild.
> +
> +# Rationale: PMS prohibits ebuilds from installing empty directories.
> +# Cleaning them up from the installation image provides an easy way
> +# to make sure that ebuilds are not relying on it while making it easy
> +# for users to override this if they need to.
> +#
> +# Technically, we could emit QA warnings here. However, we do not want
> +# to pursue every upstream build system that creates a directory
> +# and does not install any file into it (think of files installed
> +# conditionally), as long as the package functions correctly without
> +# the directory being actually installed.
> +
> +strip_empty_dirs() {
> +   find "${ED}" -mindepth 1 -type d -empty -delete
> +}
> +
> +strip_empty_dirs
> +: # guarantee successful exit
> +
> +# vim:ft=sh
> --
> 2.16.0.rc1
>
>
>


Re: [gentoo-portage-dev] [PATCH v2] bin/doins.py: implement install -p option (bug 642632)

2017-12-31 Thread Alec Warner
On Fri, Dec 29, 2017 at 7:19 PM, Zac Medico  wrote:

> Bug: https://bugs.gentoo.org/642632
> ---
> [PATCH v2] fix to copy atime, and split out _set_timestamps function
>
>  bin/doins.py| 28 +---
>  pym/portage/tests/bin/test_doins.py |  6 --
>  2 files changed, 29 insertions(+), 5 deletions(-)
>
> diff --git a/bin/doins.py b/bin/doins.py
> index 92e450979..0d03d8fb2 100644
> --- a/bin/doins.py
> +++ b/bin/doins.py
> @@ -107,6 +107,7 @@ def _parse_install_options(
> parser.add_argument('-g', '--group', default=-1, type=_parse_group)
> parser.add_argument('-o', '--owner', default=-1, type=_parse_user)
> parser.add_argument('-m', '--mode', default=0o755,
> type=_parse_mode)
> +   parser.add_argument('-p', '--preserve-timestamps',
> action='store_true')
> split_options = shlex.split(options)
> namespace, remaining = parser.parse_known_args(split_options)
> # Because parsing '--mode' option is partially supported. If
> unknown
> @@ -139,6 +140,24 @@ def _set_attributes(options, path):
> os.chmod(path, options.mode)
>
>
> +def _set_timestamps(source_stat, dest):
> +   """Apply timestamps from source_stat to dest.
> +
> +   Args:
> +   source_stat: stat result for the source file.
> +   dest: path to the dest file.
> +   """
> +   os.utime(dest, (source_stat.st_atime, source_stat.st_mtime))
> +
> +
> +if sys.version_info >= (3, 3):
> +   def _set_timestamps_ns(source_stat, dest):
> +   os.utime(dest, ns=(source_stat.st_atime_ns,
> source_stat.st_mtime_ns))
> +
> +   _set_timestamps_ns.__doc__ = _set_timestamps.__doc__
> +   _set_timestamps = _set_timestamps_ns
> +
> +
>  class _InsInProcessInstallRunner(object):
> """Implements `install` command behavior running in a process."""
>
> @@ -168,7 +187,8 @@ class _InsInProcessInstallRunner(object):
> True on success, otherwise False.
> """
> dest = os.path.join(dest_dir, os.path.basename(source))
> -   if not self._is_install_allowed(source, dest):
>

I'm going to be api picky here (so forgive me).

Previously is_install_allowed seemed to take great care with which stat was
used. But now callers have to just know to use os.stat (as opposed to
lstat, or other stats)?

Should we update the is_install_allowed comment (perhaps to say that
source_stat prefers os.stat?)

Part of me just wishes the stat was cached at a different layer (so that
code like this wasn't required just to save the syscall ;(

-A

+   sstat = os.stat(source)
> +   if not self._is_install_allowed(source, sstat, dest):
> return False
>
> # To emulate the `install` command, remove the dest file in
> @@ -187,6 +207,8 @@ class _InsInProcessInstallRunner(object):
> movefile._copyxattr(
> source, dest,
> exclude=self._xattr_exclude)
> +   if self._parsed_options.preserve_timestamps:
> +   _set_timestamps(sstat, dest)
> except Exception:
> logging.exception(
> 'Failed to copy file: '
> @@ -195,13 +217,14 @@ class _InsInProcessInstallRunner(object):
> return False
> return True
>
> -   def _is_install_allowed(self, source, dest):
> +   def _is_install_allowed(self, source, source_stat, dest):
> """Returns if installing source into dest should work.
>
> This is to keep compatibility with the `install` command.
>
> Args:
> source: path to the source file.
> +   source_stat: stat result for the source file.
> dest: path to the dest file.
>
> Returns:
> @@ -210,7 +233,6 @@ class _InsInProcessInstallRunner(object):
> # To match `install` command, use stat() for source, while
> # lstat() for dest. Raise an exception if stat(source)
> fails,
> # intentionally.
> -   source_stat = os.stat(source)
> try:
> dest_lstat = os.lstat(dest)
> except OSError as e:
> diff --git a/pym/portage/tests/bin/test_doins.py
> b/pym/portage/tests/bin/test_doins.py
> index 14d7adfa6..9b6c55d85 100644
> --- a/pym/portage/tests/bin/test_doins.py
> +++ b/pym/portage/tests/bin/test_doins.py
> @@ -38,13 +38,15 @@ class DoIns(setup_env.BinTestCase):
> self.init()
> try:
> env = setup_env.env
> -   env['INSOPTIONS'] = '-m0644'
> +   env['INSOPTIONS'] = '-pm0644'
> with 

Re: [gentoo-portage-dev] [PATCH] dep_check: use DNF to optimize overlapping virtual || deps (bug 632026)

2017-11-13 Thread Alec Warner
On Sun, Nov 12, 2017 at 6:48 AM, Zac Medico  wrote:

> Deps like these:
>
>   || ( foo bar ) || ( bar baz )
>
> Translate to disjunctive normal form (DNF):
>
>   || ( ( foo bar ) ( foo baz ) ( bar bar ) ( bar baz ) )
>
> Using DNF, if none of the packages are currently installed,
> then the ( bar bar ) choice will be automatically preferred
> since it is satisfied by the fewest number of packages.
> If the ( foo baz ) choice is already satisfied, then that
> choice will be preferred instead.
>
> Since DNF results in exponential explosion of the formula,
> only use DNF for the parts of the dependencies that have
> overlapping atoms.
>
> In order to simplify the implementation of the dnf_convert
> function, this patch also fixes _expand_new_virtuals to
> normalize results in the same way as use_reduce (with no
> redundant nested lists).
>
> Bug: https://bugs.gentoo.org/632026
> ---
>  pym/portage/dep/_dnf.py|  97 
>  pym/portage/dep/dep_check.py   | 127
> +++--
>  pym/portage/tests/dep/test_dnf_convert.py  |  48 
>  pym/portage/tests/dep/test_overlap_dnf.py  |  28 +
>  .../resolver/test_virtual_minimize_children.py |  92 +++
>  5 files changed, 385 insertions(+), 7 deletions(-)
>  create mode 100644 pym/portage/dep/_dnf.py
>  create mode 100644 pym/portage/tests/dep/test_dnf_convert.py
>  create mode 100644 pym/portage/tests/dep/test_overlap_dnf.py
>  create mode 100644 pym/portage/tests/resolver/test_virtual_minimize_
> children.py
>
> diff --git a/pym/portage/dep/_dnf.py b/pym/portage/dep/_dnf.py
> new file mode 100644
> index 0..503b209c2
> --- /dev/null
> +++ b/pym/portage/dep/_dnf.py
> @@ -0,0 +1,97 @@
> +# Copyright 2017 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +from __future__ import unicode_literals
> +
> +import itertools
> +
> +
> +def dnf_convert(dep_struct):
> +   """
> +   Convert dep_struct to disjunctive normal form (DNF), where
> dep_struct
> +   is either a conjunction or disjunction of the form produced by
> +   use_reduce(opconvert=True).
> +   """
> +   # Normalize input to have a top-level conjunction.
> +   if isinstance(dep_struct, list):
> +   if dep_struct and dep_struct[0] == '||':
> +   dep_struct = [dep_struct]
> +   else:
> +   dep_struct = [dep_struct]
> +
> +   conjunction = []
> +   disjunctions = []
> +
> +   for x in dep_struct:
> +   if isinstance (x, list):
> +   assert x
>


I'm not a huge fan of asserts, but if we use them can we use them in the
expr, message form?

assert x, "Assertion failed, wanted x in list form in dep: %s" % dep_struct

or whatever.


> +   if x[0] == '||':
> +   if any(isinstance(element, list) for
> element in x):
> +   x_dnf = ['||']
> +   for element in x[1:]:
> +   if isinstance(element,
> list):
> +   # Due to
> normalization, a disjunction must not be
> +   # nested directly
> in another disjunction, so this
> +   # must be a
> conjunction.
> +   assert element and
> element[0] != '||'
> +   element =
> dnf_convert(element)
> +   if
> contains_disjunction(element):
> +   assert
> (len(element) == 1 and
> +
>  element[0] and element[0][0] == '||')
> +
>  x_dnf.extend(element[0][1:])
> +   else:
> +
>  x_dnf.append(element)
> +   else:
> +
>  x_dnf.append(element)
> +   x = x_dnf
> +   disjunctions.append(x)
> +   else:
> +   for x in dnf_convert(x):
> +   if isinstance (x, list):
> +   # Due to normalization, a
> conjunction must not be
> +   # nested directly in
> another conjunction, so this
> +   # must be a disjunction.
> +   assert x and x[0] == '||'
> +   disjunctions.append(x)
> +   else:
> +   conjunction.append(x)
> +   else:
> +

Re: [gentoo-portage-dev] [PATCH] ebuild.sh: Completely ban external commands in global scope

2017-09-08 Thread Alec Warner
Must be old age setting in :(

Thanks,

-A

On Fri, Sep 8, 2017 at 2:54 PM, Michał Górny <mgo...@gentoo.org> wrote:

> W dniu pią, 08.09.2017 o godzinie 14∶48 -0400, użytkownik Alec Warner
> napisał:
> > Why PATH=/dev/null vs export PATH=""
>
> +  # note: we can't use empty because it implies current directory
>
> >
> > On Thu, Sep 7, 2017 at 3:36 AM, Michał Górny <mgo...@gentoo.org> wrote:
> >
> > > Dnia 31 sierpnia 2017 22:45:42 CEST, "Michał Górny" <mgo...@gentoo.org
> >
> > > napisał(a):
> > > > Set PATH to /dev/null when sourcing the ebuild for dependency
> > > > resolution
> > > > in order to prevent shell from finding external commands via PATH
> > > > lookup. While this does not prevent executing programs via full path,
> > > > it
> > > > should catch the majority of accidental uses.
> > > >
> > > > Closes: https://github.com/gentoo/portage/pull/199
> > > >
> > > > // Note: this can't be merged right now since we still have ebuilds
> > > > // calling external commands; see:
> > > > // https://bugs.gentoo.org/show_bug.cgi?id=629222
> > >
> > > Update: gentoo is green now
> > >
> > > > ---
> > > > bin/ebuild.sh | 6 +-
> > > > bin/isolated-functions.sh | 4 
> > > > 2 files changed, 9 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/bin/ebuild.sh b/bin/ebuild.sh
> > > > index c23561651..94a44d534 100755
> > > > --- a/bin/ebuild.sh
> > > > +++ b/bin/ebuild.sh
> > > > @@ -80,8 +80,12 @@ else
> > > >   done
> > > >   unset funcs x
> > > >
> > > > +  # prevent the shell from finding external executables
> > > > +  # note: we can't use empty because it implies current
> directory
> > > > +  _PORTAGE_ORIG_PATH=${PATH}
> > > > +  export PATH=/dev/null
> > > >   command_not_found_handle() {
> > > > -  die "Command not found while sourcing ebuild: ${*}"
> > > > +  die "External commands disallowed while sourcing
> ebuild:
> > >
> > > ${*}"
> > > >   }
> > > > fi
> > > >
> > > > diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
> > > > index e320f7132..b28e44f18 100644
> > > > --- a/bin/isolated-functions.sh
> > > > +++ b/bin/isolated-functions.sh
> > > > @@ -121,6 +121,10 @@ __helpers_die() {
> > > > }
> > > >
> > > > die() {
> > > > +  # restore PATH since die calls basename & sed
> > > > +  # TODO: make it pure bash
> > > > +  [[ -n ${_PORTAGE_ORIG_PATH} ]] && PATH=${_PORTAGE_ORIG_PATH}
> > > > +
> > > >   set +x # tracing only produces useless noise here
> > > >   local IFS=$' \t\n'
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Michał Górny (by phone)
> > >
> > >
>
> --
> Best regards,
> Michał Górny
>
>
>


Re: [gentoo-portage-dev] [PATCH] ebuild.sh: Completely ban external commands in global scope

2017-09-08 Thread Alec Warner
Why PATH=/dev/null vs export PATH=""

On Thu, Sep 7, 2017 at 3:36 AM, Michał Górny  wrote:

> Dnia 31 sierpnia 2017 22:45:42 CEST, "Michał Górny" 
> napisał(a):
> >Set PATH to /dev/null when sourcing the ebuild for dependency
> >resolution
> >in order to prevent shell from finding external commands via PATH
> >lookup. While this does not prevent executing programs via full path,
> >it
> >should catch the majority of accidental uses.
> >
> >Closes: https://github.com/gentoo/portage/pull/199
> >
> >// Note: this can't be merged right now since we still have ebuilds
> >// calling external commands; see:
> >// https://bugs.gentoo.org/show_bug.cgi?id=629222
>
> Update: gentoo is green now
>
> >---
> > bin/ebuild.sh | 6 +-
> > bin/isolated-functions.sh | 4 
> > 2 files changed, 9 insertions(+), 1 deletion(-)
> >
> >diff --git a/bin/ebuild.sh b/bin/ebuild.sh
> >index c23561651..94a44d534 100755
> >--- a/bin/ebuild.sh
> >+++ b/bin/ebuild.sh
> >@@ -80,8 +80,12 @@ else
> >   done
> >   unset funcs x
> >
> >+  # prevent the shell from finding external executables
> >+  # note: we can't use empty because it implies current directory
> >+  _PORTAGE_ORIG_PATH=${PATH}
> >+  export PATH=/dev/null
> >   command_not_found_handle() {
> >-  die "Command not found while sourcing ebuild: ${*}"
> >+  die "External commands disallowed while sourcing ebuild:
> ${*}"
> >   }
> > fi
> >
> >diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
> >index e320f7132..b28e44f18 100644
> >--- a/bin/isolated-functions.sh
> >+++ b/bin/isolated-functions.sh
> >@@ -121,6 +121,10 @@ __helpers_die() {
> > }
> >
> > die() {
> >+  # restore PATH since die calls basename & sed
> >+  # TODO: make it pure bash
> >+  [[ -n ${_PORTAGE_ORIG_PATH} ]] && PATH=${_PORTAGE_ORIG_PATH}
> >+
> >   set +x # tracing only produces useless noise here
> >   local IFS=$' \t\n'
> >
>
>
> --
> Best regards,
> Michał Górny (by phone)
>
>


Re: [gentoo-portage-dev] [PATCH v2] repoman commit: Support --bug (-b) and --closes (-c) for git footer

2017-08-05 Thread Alec Warner
On Fri, Aug 4, 2017 at 6:37 PM, Michał Górny  wrote:

> Support two new options: --bug (-b) and --closes (-c) to add a plain
> 'Bug' reference and a 'Closes' footer for a GitHub pull request. Both
> options can be specified multiple times, resulting in multiple footer
> tags being written.
>
> The --bug option accepts either a Gentoo Bugzilla bug number or an URL
> to any bug tracker. In the latter case, it performs two trivial
> transformations automatically: replaces long 'show_bug.cgi' Bugzilla
> URLs with the short 'https://bugs.gentoo.org/NN', and forces
> https:// for a few known services.
>
> The --closes option accepts either a GitHub Gentoo repository pull
> request number or an URL to any pull request (or bug) that uses
> the 'Closes' tag. In the latter case, https:// is forced for a few known
> services.
>
> Changes in v2: use urlparse instead of regexps
>
> ---
>  repoman/pym/repoman/actions.py   | 45 ++
> ++
>  repoman/pym/repoman/argparser.py | 16 +-
>  2 files changed, 60 insertions(+), 1 deletion(-)
>
> diff --git a/repoman/pym/repoman/actions.py b/repoman/pym/repoman/actions.
> py
> index 00bb5b2ca..8299ed0fe 100644
> --- a/repoman/pym/repoman/actions.py
> +++ b/repoman/pym/repoman/actions.py
> @@ -14,6 +14,11 @@ import tempfile
>  import time
>  from itertools import chain
>
> +try:
> +   from urllib.parse import parse_qs, urlsplit, urlunsplit
> +except ImportError:
> +   from urlparse import parse_qs, urlsplit, urlunsplit
> +
>  from _emerge.UserQuery import UserQuery
>
>  from repoman._portage import portage
> @@ -324,6 +329,13 @@ class Actions(object):
> return (changes.new, changes.changed, changes.removed,
> changes.no_expansion, changes.expansion)
>
> +   https_bugtrackers = frozenset([
> +   'bitbucket.org',
> +   'bugs.gentoo.org',
> +   'github.com',
> +   'gitlab.com',
> +   ])
> +
> def get_commit_footer(self):
> portage_version = getattr(portage, "VERSION", None)
> gpg_key = self.repoman_settings.get("PORTAGE_GPG_KEY", "")
> @@ -345,9 +357,42 @@ class Actions(object):
>
> # Common part of commit footer
> commit_footer = "\n"
> +   for bug in self.options.bug:
> +   # case 1: pure number NN
> +   if bug.isdigit():
> +   bug = 'https://bugs.gentoo.org/%s' %
> (bug, )
> +   else:
> +   purl = urlsplit(bug)
> +   qs = parse_qs(purl.query)
> +   # case 2: long Gentoo bugzilla URL to
> shorten
> +   if (purl.netloc == 'bugs.gentoo.org' and
> +   purl.path ==
> '/show_bug.cgi' and
> +   tuple(qs.keys()) ==
> ('id',)):
> +   bug = urlunsplit(('https',
> purl.netloc,
> +   qs['id'][-1], '',
> purl.fragment))
> +   # case 3: bug tracker w/ http -> https
> +   elif (purl.scheme == 'http' and
> +   purl.netloc in
> self.https_bugtrackers):
> +   bug = urlunsplit(('https',) +
> purl[1:])
> +   commit_footer += "Bug: %s\n" % (bug, )
> +
> +   for closes in self.options.closes:
> +   # case 1: pure number 
> +   if closes.isdigit():
> +   closes = 'https://github.com/gentoo/
> gentoo/pull/%s' % (closes, )
> +   else:
> +   purl = urlsplit(closes)
> +   # case 2: bug tracker w/ http -> https
> +   if purl.netloc in self.https_bugtrackers:
> +   closes = urlunsplit(('https',) +
> purl[1:])
> +   commit_footer += "Closes: %s\n" % (closes, )
> +
> if dco_sob:
> commit_footer += "Signed-off-by: %s\n" % (dco_sob,
> )
>
> +   print(commit_footer)
> +   raise SystemExit(666)
>

Debug lines?

-A


> +
> # Use new footer only for git (see bug #438364).
> if self.vcs_settings.vcs in ["git"]:
> commit_footer += "Package-Manager: Portage-%s,
> Repoman-%s" % (
> diff --git a/repoman/pym/repoman/argparser.py b/repoman/pym/repoman/
> argparser.py
> index 2d56a87e6..f32972288 100644
> --- a/repoman/pym/repoman/argparser.py
> +++ b/repoman/pym/repoman/argparser.py
> @@ -1,5 +1,5 @@
>  # repoman: Argument parser
> -# Copyright 2007-2014 Gentoo Foundation
> +# 

Re: [gentoo-portage-dev] [PATCH] repoman commit: Support --bug (-b) and --closes (-c) for git footer

2017-08-03 Thread Alec Warner
On Thu, Aug 3, 2017 at 12:15 PM, Michał Górny <mgo...@gentoo.org> wrote:

> On czw, 2017-08-03 at 11:18 -0400, Alec Warner wrote:
> > On Thu, Aug 3, 2017 at 10:18 AM, Michał Górny <mgo...@gentoo.org> wrote:
> >
> > > Support two new options: --bug (-b) and --closes (-c) to add a plain
> > > 'Bug' reference and a 'Closes' footer for a GitHub pull request. Both
> > > options can be specified multiple times, resulting in multiple footer
> > > tags being written.
> > >
> > > The --bug option accepts either a Gentoo Bugzilla bug number or an URL
> > > to any bug tracker. In the latter case, it performs two trivial
> > > transformations automatically: replaces long 'show_bug.cgi' Bugzilla
> > > URLs with the short 'https://bugs.gentoo.org/NN', and forces
> > > https:// for a few known services.
> > >
> > > The --closes option accepts either a GitHub Gentoo repository pull
> > > request number or an URL to any pull request (or bug) that uses
> > > the 'Closes' tag. In the latter case, https:// is forced for a few
> known
> > > services.
> > > ---
> > >  repoman/pym/repoman/actions.py   | 29 +
> > >  repoman/pym/repoman/argparser.py | 16 +++-
> > >  2 files changed, 44 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/repoman/pym/repoman/actions.py
> b/repoman/pym/repoman/actions.
> > > py
> > > index 00bb5b2ca..869ca7031 100644
> > > --- a/repoman/pym/repoman/actions.py
> > > +++ b/repoman/pym/repoman/actions.py
> > > @@ -324,6 +324,11 @@ class Actions(object):
> > > return (changes.new, changes.changed, changes.removed,
> > > changes.no_expansion,
> changes.expansion)
> > >
> > > +   long_bugzilla_url_re = re.compile(
> > > +   r'https?://bugs\.gentoo\.org/
> > > show_bug\.cgi\?id=(\d+(#.*)?)')
> > > +   http_bugtracker_url_re = re.compile(
> > > +   r'http://(bugs\.gentoo\.org|
> bitbucket\.org|git(hub|lab)\.
> > > com)/')
> > > +
> > >
> >
> > Would you consider using the urlparse module here, instead of regexp?
>
> Sure. Just for the second case, or for both?
>

I was actually more concerned about parsing the id query string than the
hostname; so I'd say both; but I leave it to your judgement.

-A


>
> --
> Best regards,
> Michał Górny
>


Re: [gentoo-portage-dev] [PATCH] repoman commit: Support --bug (-b) and --closes (-c) for git footer

2017-08-03 Thread Alec Warner
On Thu, Aug 3, 2017 at 10:18 AM, Michał Górny  wrote:

> Support two new options: --bug (-b) and --closes (-c) to add a plain
> 'Bug' reference and a 'Closes' footer for a GitHub pull request. Both
> options can be specified multiple times, resulting in multiple footer
> tags being written.
>
> The --bug option accepts either a Gentoo Bugzilla bug number or an URL
> to any bug tracker. In the latter case, it performs two trivial
> transformations automatically: replaces long 'show_bug.cgi' Bugzilla
> URLs with the short 'https://bugs.gentoo.org/NN', and forces
> https:// for a few known services.
>
> The --closes option accepts either a GitHub Gentoo repository pull
> request number or an URL to any pull request (or bug) that uses
> the 'Closes' tag. In the latter case, https:// is forced for a few known
> services.
> ---
>  repoman/pym/repoman/actions.py   | 29 +
>  repoman/pym/repoman/argparser.py | 16 +++-
>  2 files changed, 44 insertions(+), 1 deletion(-)
>
> diff --git a/repoman/pym/repoman/actions.py b/repoman/pym/repoman/actions.
> py
> index 00bb5b2ca..869ca7031 100644
> --- a/repoman/pym/repoman/actions.py
> +++ b/repoman/pym/repoman/actions.py
> @@ -324,6 +324,11 @@ class Actions(object):
> return (changes.new, changes.changed, changes.removed,
> changes.no_expansion, changes.expansion)
>
> +   long_bugzilla_url_re = re.compile(
> +   r'https?://bugs\.gentoo\.org/
> show_bug\.cgi\?id=(\d+(#.*)?)')
> +   http_bugtracker_url_re = re.compile(
> +   r'http://(bugs\.gentoo\.org|bitbucket\.org|git(hub|lab)\.
> com)/')
> +
>

Would you consider using the urlparse module here, instead of regexp?


> def get_commit_footer(self):
> portage_version = getattr(portage, "VERSION", None)
> gpg_key = self.repoman_settings.get("PORTAGE_GPG_KEY", "")
> @@ -345,6 +350,30 @@ class Actions(object):
>
> # Common part of commit footer
> commit_footer = "\n"
> +   for bug in self.options.bug:
> +   # case 1: pure number NN
> +   if bug.isdigit():
> +   bug = 'https://bugs.gentoo.org/%s' %
> (bug, )
> +   else:
> +   # case 2: long Gentoo bugzilla URL to
> shorten
> +   m = self.long_bugzilla_url_re.match(bug)
> +   if m is not None:
> +   bug = 'https://bugs.gentoo.org/%s'
> % (m.group(1), )
> +   # case 3: bug tracker w/ http -> https
> +   m = self.http_bugtracker_url_re.match(bug)
> +   if m is not None:
> +   bug = bug.replace('http', 'https',
> 1)
> +   commit_footer += "Bug: %s\n" % (bug, )
> +   for closes in self.options.closes:
> +   # case 1: pure number 
> +   if closes.isdigit():
> +   closes = 'https://github.com/gentoo/
> gentoo/pull/%s' % (closes, )
> +   else:
> +   # case 2: bug tracker w/ http -> https
> +   m = self.http_bugtracker_url_re.
> match(closes)
> +   if m is not None:
> +   closes = closes.replace('http',
> 'https', 1)
> +   commit_footer += "Closes: %s\n" % (closes, )
> if dco_sob:
> commit_footer += "Signed-off-by: %s\n" % (dco_sob,
> )
>
> diff --git a/repoman/pym/repoman/argparser.py b/repoman/pym/repoman/
> argparser.py
> index 2d56a87e6..f32972288 100644
> --- a/repoman/pym/repoman/argparser.py
> +++ b/repoman/pym/repoman/argparser.py
> @@ -1,5 +1,5 @@
>  # repoman: Argument parser
> -# Copyright 2007-2014 Gentoo Foundation
> +# Copyright 2007-2017 Gentoo Foundation
>  # Distributed under the terms of the GNU General Public License v2
>
>  """This module contains functions used in Repoman to parse CLI
> arguments."""
> @@ -58,6 +58,20 @@ def parse_args(argv, qahelp, repoman_default_opts):
> help='Request a confirmation before commiting')
>
> parser.add_argument(
> +   '-b', '--bug', dest='bug', action='append',
> metavar='',
> +   default=[],
> +   help=(
> +   'Mention a Gentoo or upstream bug in the commit
> footer; '
> +   'takes either Gentoo bug number or full bug URL'))
> +
> +   parser.add_argument(
> +   '-c', '--closes', dest='closes', action='append',
> metavar='',
> +   default=[],
> +   help=(
> +   'Adds a Closes footer to close GitHub 

Re: [gentoo-portage-dev] Portage and Update Security

2015-03-14 Thread Alec Warner
On Tue, Mar 10, 2015 at 2:15 PM, Vladimir Diaz vladimir.v.d...@gmail.com
wrote:

 Hi,

 I am a developer in the Secure Systems Lab at NYU.  Our lab has
 collaborated with popular software update systems in the open-source
 community, including APT, yum, and YaST, to address security problems.
 More recently, we have been working on a flexible security framework
 co-developed with the Tor project that can be easily added to software
 updaters to transparently solve many of the known security flaws we have
 uncovered in software updaters.  We would like to work with The Portage
 Development Project to better secure the Portage distribution system.


I'm not familiar with your work on APT, do you have a link?


 TUF
 https://github.com/theupdateframework/tuf#a-framework-for-securing-software-update-systems
 (The Update Framework) is a library that can be added to an existing
 software update system and is designed to update files in a more secure
 manner.  Many software updaters verify software updates with cryptographic
 signatures and hash functions, but they typically fail to protect against
 malicious attacks that target the metadata and update files presented to
 clients.  A rollback attack is one such example, where an attacker tricks a
 client into installing older files than those the client has already seen
 (these older files may be vulnerable versions that have since been fixed).
 A full list of attacks and weaknesses the framework is designed to address
 is provided here
 https://github.com/theupdateframework/tuf/blob/develop/SECURITY.md#security
 .

 Our website http://theupdateframework.com/index.html includes more
 information about TUF, including: papers
 https://github.com/theupdateframework/tuf/tree/develop/docs/papers and
 a specification
 https://github.com/theupdateframework/tuf/blob/develop/docs/tuf-spec.txt.
 If you want to see how an existing project integrates TUF, there is a
 standards track proposal
 https://github.com/pypa/interoperability-peps/blob/master/pep-0458-tuf-online-keys.rst#abstract
 to the Python community that you can review.  A more rigorous proposal that
 requires more administrative work on the repository, but provides more
 security protections, is also available
 https://www.python.org/dev/peps/pep-0480/.

 We were thinking of submitting a pull request that shows how such an
 integration would work.  So there hopefully won't be much leg work on your
 end apart from deciding how the system should be configured (key storage,
 roles, etc.).


 Would a pull request be of interest?  Is there anything you'd like us to
 say more about?


I guess I am less concerned with adding support to portage (which as you
note, is likely fairly straightforward) vs actually generating, publishing,
and signing the metadata; which you would have convince the infrastructure
team to do.


 Thanks,
 Vlad

 P.S.
 There are Informational http://wiki.gentoo.org/wiki/GLEP:57 and Standards
 Track http://wiki.gentoo.org/wiki/GLEP:58 GLEPs that reference our work
 and the security issues that our project addresses, but there hasn't been
 much recent activity on these proposals.


FWIW, I would rather adopt the standard than continue with a gentoo
specific thing; but I'm not the guy who is going to implement it. I would
recommend talking to the GLEP author (robb...@gentoo.org)

-A




 --
 vladimir.v.d...@gmail.com
 PGP fingerprint = ACCF 9DCA 73B9 862F 93C5  6608 63F8 90AA 1D25 3935
 --



Re: [gentoo-portage-dev] Re: running ebuild in src tree

2015-03-14 Thread Alec Warner
On Thu, Mar 12, 2015 at 10:26 AM, Joakim Tjernlund 
joakim.tjernl...@transmode.se wrote:

 On Thu, 2015-03-12 at 01:27 +, Duncan wrote:
  Zac Medico posted on Wed, 11 Mar 2015 12:03:10 -0700 as excerpted:
 
   On 03/11/2015 11:56 AM, Joakim Tjernlund wrote:
On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote:
 On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
  When developing code it would be really nice if one could run
 your ebuild on that src tree as is(no
  fetch, unpack etc.)

 The existing convention is to create an ebuild with version 
 and use one of the live vcs eclasses
 such as git-r3 to pull the live sources in the src_unpack
 function. In a future EAPI, we plan to add
 some features related to this [1].
   
I think you misunderstand, [1] is not what I want to do(I think):
   
Got my src working copy and made a few modds, not commitet yet. Now
 I just want build/test etc. before
committing and to do that I just run
 mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my
code is built which I already have in mytree.
  
   Well, you can create a - ebuild that copies your sources from
 $directory to $WORKDIR. Maybe use an
   environment to configure whether it pulls from a local directory or a
 vcs repository.
 
  @ Joakim T:
 
  FWIW, a commonly recommended user-level portage optimization is to point
  $PORTAGE_TMPDIR at a tmpfs mount.  As long as you have sufficient memory,
  that lets all building take place in the tmpfs and thus in memory,
 eliminating many read-accesses and
  most/all write accesses to permanent  storage during the build and
 (fake-)install phases.
 
  In addition to speeding up emerge itself, this reduces build-time I/O,
 which often becomes the bottleneck
  on which other processes may be  waiting as well, thus allowing other
 processes more efficient access to
  permanent storage while emerge is ongoing.  Between this and setting
 PORTAGE_NICENESS=20, emerge is /much/
  better behaved during builds,  interrupting other processes much less
 and thus letting you carry on with
  other things while emerge is doing its thing, with far less interruption.
  =:^)
 
  For instance, here I have /tmp as a tmpfs mount (with /var/tmp being a
 bind-mount of the same tmpfs), and
  in make.conf, have the line:
 
  PORTAGE_TMPDIR=/tmp
 
  Emerge then creates /tmp/portage, and within it, creates the usual cat/
 pkg/ build trees, etc, as it
  emerges various packages.
 
 
  Obviously, your sources in permanent storage are going to be cache-copied
  into memory as you do the build anyway, and pointing PORTAGE_TMPDIR at
 tmpfs then becomes a copy to
  (tmpfs) memory only.  While that doesn't  technically eliminate the
 copies (since the read into tmpfs will
  cache  and you'll have the tmpfs copy as well), it DOES mean most of the
 work  beyond the initial read into
  memory will be memory-only, so you DO  eliminate the permanent-storage
 copies.
 
  Is that sufficiently close to what you were looking to do?  Beyond that,
  as Zac suggests, just have the ebuild grab the sources from wherever you
  put them as your src_unpack, as at that point it'll be a copy to tmpfs,
 and thus take essentially the same
  time (or even less since it'll avoid  the build-time writes to permanent
 storage) as doing the in-place
  build  directly.  Plus, creating a tmpfs mount if necessary, and
 setting  PORTAGE_TMPDIR, is easy, and
  you'll dramatically speed-up normal builds  as well. =:^)
 

 No, there can be no copy of sources for what I want. It just gets in the
 way having to do that.
 Hacks like this seems to work:


I wouldn't really call it a hack either, but whichever ;)


 post_src_compile() {
 # make it compile every time
 rm -f ${PORTAGE_BUILDDIR}/.compiled
 }

 post_src_install() {
 # make it install every time
 rm -f ${PORTAGE_BUILDDIR}/.installed
 }

 #hmm, doesn't seem to be an post_package function
 #post_package() {
 #   rm -f ${PORTAGE_BUILDDIR}/.packaged
 #}

 src_unpack() {
 #dir need to exist
 mkdir -p ${S} || die
 }
 src_compile() {
 EBUILDDIR=$(dirname ${FILESDIR})
 MYTRUNK=${EBUILDDIR}/../../..
 # add sandbox exceptions
 addwrite ${MYTRUNK}

 cd ${MYTRUNK} || die
 cd ${PN}
 .
 }

 I'm a bit curious what sort of workflow you are trying to achieve here is.
Which build artifacts are you looking for; the actual built binaries or the
built package that is merged into the livefs?

If the artifacts are packages, then I think this approach makes some
sense...but otherwise I'd just go into trunk and run the build process
(what does the ebuild gain you here?)

-A


Re: [gentoo-portage-dev] [PATCH] emerge: Don't die when the user has an invalid locale setting

2014-08-05 Thread Alec Warner
On Mon, Aug 4, 2014 at 12:45 PM, Mike Gilbert flop...@gentoo.org wrote:

 ---
  pym/_emerge/main.py | 5 -
  1 file changed, 4 insertions(+), 1 deletion(-)

 diff --git a/pym/_emerge/main.py b/pym/_emerge/main.py
 index 1a920f7..722da84 100644
 --- a/pym/_emerge/main.py
 +++ b/pym/_emerge/main.py
 @@ -999,7 +999,10 @@ def emerge_main(args=None):
 args = portage._decode_argv(args)

 # Use system locale.
 -   locale.setlocale(locale.LC_ALL, '')
 +   try:
 +   locale.setlocale(locale.LC_ALL, '')
 +   except locale.Error as e:
 +   print(e)


Portage has its own output functions (writemsg?)
You need to decide if this should be printed even if -q is passed, should
it be colorized, etc...

-A



 # Disable color until we're sure that it should be enabled (after
 # EMERGE_DEFAULT_OPTS has been parsed).
 --
 2.0.4





Re: [gentoo-portage-dev] [PATCH] Support the 'packages' profile file as a directory.

2014-05-12 Thread Alec Warner
And the manpages..?

-A


On Mon, May 12, 2014 at 3:14 AM, Alexander Berntsen berna...@gentoo.orgwrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 Thanks! Committed as b4d8e300c04b768be7cd5c64116d6cc0453219b4.

 - --
 Alexander
 berna...@gentoo.org
 https://secure.plaimi.net/~alexander
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2.0.22 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

 iF4EAREIAAYFAlNwnxsACgkQRtClrXBQc7XBngD6A6FkUivoQAtXUaVOVLu+aLzX
 nUI7vZQOA4CvGIW5qK4BAIRvWMC3PQM1VrdoWDjR7Hh+euhCrRDn9RyG+JgEfkOz
 =LhmL
 -END PGP SIGNATURE-




Re: [gentoo-portage-dev] [PATCH 1/2] Use a localized size formatting function and ISO/IEC prefixes.

2014-03-30 Thread Alec Warner
On Sun, Mar 30, 2014 at 12:56 AM, Michał Górny mgo...@gentoo.org wrote:

 Dnia 2014-03-30, o godz. 00:03:02
 Alexander Berntsen berna...@gentoo.org napisał(a):

  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA256
 
  Please use a max 50 char commit message headline, without the period.
 
  On 29/03/14 19:45, Michał Górny wrote:
   +import locale +import math
  Why not explicitly import what you are going to use? (On the form of
  from foo import bar, baz.)

 I feel like one of the points in using Python is to explicitly name
 functions (and modules) which you use, rather than importing everything
 in a single namespace and having people scroll up and down to see where
 functions come from.


I agree with mgorny here. Tons of existing portage code violates this rule,
and it bugs me a lot. I hope that in the future we can move to imports
based on modules (and for things that are really long, explicit assignment,
perhaps to a local.)

-A


 --
 Best regards,
 Michał Górny



Re: [gentoo-portage-dev] [PATCH 3/3] Add an emaint module that can scan for failed merges and that can fix failed merges.

2014-03-15 Thread Alec Warner
On Fri, Mar 14, 2014 at 10:35 PM, Pavel Kazakov nullishz...@gentoo.orgwrote:

 On 03/13/14 15:09, Alec Warner wrote:
 
 
 
  On Wed, Mar 12, 2014 at 11:10 PM, Pavel Kazakov
  nullishz...@gentoo.org mailto:nullishz...@gentoo.org wrote:
 
  ---
   pym/portage/emaint/main.py|   6 +-
 

 ...

  Perhaps add an __iter__ function to TrackingFile, that implicitly
  calls load?
 
  Then this can be:
  for pkg, mtime in self._tracking_file:
 
  ?
 
  Just a thought.

 That's actually a neat idea. Going to do it.

 
 
  +   for pkg, mtime in self._tracking_file.load().items():
  +   if pkg not in failed_pkgs:
  +   failed_pkgs[pkg] = mtime
  +   return failed_pkgs
  +
  +
  +   def _remove_failed_dirs(self, failed_pkgs):
  +   
  +   Remove the directories of packages that failed to
  merge.
  +
  +   @param failed_pkgs: failed packages whose
  directories to remove
  +   @type failed_pkg: dict
  +   
  +   for failed_pkg in failed_pkgs:
  +   pkg_path = os.path.join(self._vardb_path,
  failed_pkg)
  +   # delete failed merge directory if it
  exists (it might not exist
  +   # if loaded from tracking file)
  +   if os.path.exists(pkg_path):
  +   shutil.rmtree(pkg_path)
  +   # TODO: try removing package contents to
  prevent orphaned
  +   # files
 
 
  I don't get this TODO, can you elaborate?

 Right now the current code deletes only the failed merge directory (e.g.
 /var/db/pkg/app-misc/-MERGING-lsx-0.1/).

 In /var/db/pkg/app-misc/-MERGING-lsx-0.1/, there should be a file called
 CONTENTS that records the package contents (e.g. obj
 /usr/bin/lsx-suckless 478333aa1cb7761bc8e3a400cbd976ab 1394688298).
 However, during a failed merge, the CONTENTS file may not be there or
 may be corrupt, so my code doesn't handle removing the contents right
 now, only the failed merge directory.

 Any suggestions for better wording?


an upcase CONTENTS, or just talk about vdb or metadata specifically.



 
 
  +
  +
  +   def _get_pkg_atoms(self, failed_pkgs, pkg_atoms, pkg_dne):
  +   
  +   Get the package atoms for the specified failed
  packages.
  +
  +   @param failed_pkgs: failed packages to iterate
  +   @type failed_pkgs: dict
  +   @param pkg_atoms: append package atoms to this set
  +   @type pkg_atoms: set
  +   @param pkg_dne: append any packages atoms that are
  invalid to this set
  +   @type pkg_dne: set
 
 
  Not following what dne means.

 How about pkg_invalid_entries? And I can change the comment to '@param
 pkg_invalid_entries: append any packages that are invalid to this set'

 After looking at the function more, I think it might make sense to
 instead return pkg_atoms and pkg_invalid_entries as a tuple, instead of
 having the calling code pass them in.


sgtm


 
 
  +   
  +
  +   emerge_config = load_emerge_config()
  +   portdb =
  emerge_config.target_config.trees['porttree'].dbapi
  +   for failed_pkg in failed_pkgs:
  +   # validate pkg name
  +   pkg_name = '%s' %
  failed_pkg.replace(MERGING_IDENTIFIER, '')
  +   pkg_atom = '=%s' % pkg_name
  +
  +   if not isvalidatom(pkg_atom):
  +   pkg_dne.append( '%s' is an
  invalid package atom. % pkg_atom)
  +   if not portdb.cpv_exists(pkg_name):
  +   pkg_dne.append( '%s' does not
  exist in the portage tree.
  +   % pkg_name)
  +   pkg_atoms.add(pkg_atom)
  +
  +
  +   def _emerge_pkg_atoms(self, module_output, pkg_atoms):
  +   
  +   Emerge the specified packages atoms.
  +
  +   @param module_output: output will be written to
  +   @type module_output: Class
  +   @param pkg_atoms: packages atoms to emerge
  +   @type pkg_atoms: set
  +   @rtype: list
  +   @return: List of results
  +   
  +   # TODO: rewrite code to use portage's APIs instead
  of a subprocess
  +   env = {
  +   FEATURES : -collision-detect
  -protect-owned,
  +   PATH : os.environ[PATH

Re: [gentoo-portage-dev] [PATCH 3/3] Add an emaint module that can scan for failed merges and that can fix failed merges.

2014-03-13 Thread Alec Warner
On Wed, Mar 12, 2014 at 11:10 PM, Pavel Kazakov nullishz...@gentoo.orgwrote:

 ---
  pym/portage/emaint/main.py|   6 +-
  pym/portage/emaint/modules/merges/__init__.py |  30 +++
  pym/portage/emaint/modules/merges/merges.py   | 281
 ++
  3 files changed, 315 insertions(+), 2 deletions(-)
  create mode 100644 pym/portage/emaint/modules/merges/__init__.py
  create mode 100644 pym/portage/emaint/modules/merges/merges.py

 diff --git a/pym/portage/emaint/main.py b/pym/portage/emaint/main.py
 index 6a17027..646883d 100644
 --- a/pym/portage/emaint/main.py
 +++ b/pym/portage/emaint/main.py
 @@ -98,10 +98,11 @@ def module_opts(module_controller, module):
  class TaskHandler(object):
 Handles the running of the tasks it is given

 -   def __init__(self, show_progress_bar=True, verbose=True,
 callback=None):
 +   def __init__(self, show_progress_bar=True, verbose=True,
 callback=None, module_output=None):
 self.show_progress_bar = show_progress_bar
 self.verbose = verbose
 self.callback = callback
 +   self.module_output = module_output
 self.isatty = os.environ.get('TERM') != 'dumb' and
 sys.stdout.isatty()
 self.progress_bar = ProgressBar(self.isatty,
 title=Emaint, max_desc_length=27)

 @@ -124,6 +125,7 @@ class TaskHandler(object):
 onProgress = None
 kwargs = {
 'onProgress': onProgress,
 +   'module_output': self.module_output,
 # pass in a copy of the options so a
 module can not pollute or change
 # them for other tasks if there is more to
 do.
 'options': options.copy()
 @@ -219,5 +221,5 @@ def emaint_main(myargv):
 # need to pass the parser options dict to the modules
 # so they are available if needed.
 task_opts = options.__dict__
 -   taskmaster = TaskHandler(callback=print_results)
 +   taskmaster = TaskHandler(callback=print_results,
 module_output=sys.stdout)
 taskmaster.run_tasks(tasks, func, status, options=task_opts)
 diff --git a/pym/portage/emaint/modules/merges/__init__.py
 b/pym/portage/emaint/modules/merges/__init__.py
 new file mode 100644
 index 000..96ee71b
 --- /dev/null
 +++ b/pym/portage/emaint/modules/merges/__init__.py
 @@ -0,0 +1,30 @@
 +# Copyright 2005-2014 Gentoo Foundation
 +# Distributed under the terms of the GNU General Public License v2
 +
 +Scan for failed merges and fix them.
 +
 +
 +module_spec = {
 +   'name': 'merges',
 +   'description': __doc__,
 +   'provides': {
 +   'merges': {
 +   'name': merges,
 +   'class': MergesHandler,
 +   'description': __doc__,
 +   'functions': ['check', 'fix', 'purge'],
 +   'func_desc': {
 +   'purge': {
 +   'short': '-P', 'long':
 '--purge-tracker',
 +   'help': 'Removes the list of
 previously failed merges.' +
 +   ' WARNING: Only
 use this option if you plan on' +
 +   ' manually fixing
 them or do not want them'
 +   ' re-installed.',
 +   'status': Removing %s,
 +   'action': 'store_true',
 +   'func': 'purge'
 +   }
 +   }
 +   }
 +   }
 +}
 diff --git a/pym/portage/emaint/modules/merges/merges.py
 b/pym/portage/emaint/modules/merges/merges.py
 new file mode 100644
 index 000..a99dad4
 --- /dev/null
 +++ b/pym/portage/emaint/modules/merges/merges.py
 @@ -0,0 +1,281 @@
 +# Copyright 2005-2014 Gentoo Foundation
 +# Distributed under the terms of the GNU General Public License v2
 +
 +from _emerge.actions import load_emerge_config
 +
 +import portage
 +from portage import os, _unicode_encode
 +from portage.const import MERGING_IDENTIFIER, PORTAGE_BIN_PATH,
 PRIVATE_PATH, \
 +   VDB_PATH
 +from portage.dep import isvalidatom
 +


import shutil
import subprocess
import sys
import time

Alphabetical order.


 +import time
 +import shutil
 +import sys
 +import subprocess
 +
 +class TrackingFile(object):
 +   File for keeping track of failed merges.
 +
 +
 +   def __init__(self, tracking_path):
 +   
 +   Create a TrackingFile object.
 +
 +   @param tracking_path: file path used to keep track of
 failed merges
 +   @type tracking_path: String
 +   
 +   self._tracking_path = _unicode_encode(tracking_path)
 

Re: [gentoo-portage-dev] [PATCH 3/3] Add an emaint module that can scan for failed merges and that can fix failed merges.

2014-03-13 Thread Alec Warner
On Thu, Mar 13, 2014 at 3:09 PM, Alec Warner anta...@gentoo.org wrote:




 On Wed, Mar 12, 2014 at 11:10 PM, Pavel Kazakov nullishz...@gentoo.orgwrote:

 ---
  pym/portage/emaint/main.py|   6 +-
  pym/portage/emaint/modules/merges/__init__.py |  30 +++
  pym/portage/emaint/modules/merges/merges.py   | 281
 ++
  3 files changed, 315 insertions(+), 2 deletions(-)
  create mode 100644 pym/portage/emaint/modules/merges/__init__.py
  create mode 100644 pym/portage/emaint/modules/merges/merges.py

 diff --git a/pym/portage/emaint/main.py b/pym/portage/emaint/main.py
 index 6a17027..646883d 100644
 --- a/pym/portage/emaint/main.py
 +++ b/pym/portage/emaint/main.py
 @@ -98,10 +98,11 @@ def module_opts(module_controller, module):
  class TaskHandler(object):
 Handles the running of the tasks it is given

 -   def __init__(self, show_progress_bar=True, verbose=True,
 callback=None):
 +   def __init__(self, show_progress_bar=True, verbose=True,
 callback=None, module_output=None):
 self.show_progress_bar = show_progress_bar
 self.verbose = verbose
 self.callback = callback
 +   self.module_output = module_output
 self.isatty = os.environ.get('TERM') != 'dumb' and
 sys.stdout.isatty()
 self.progress_bar = ProgressBar(self.isatty,
 title=Emaint, max_desc_length=27)

 @@ -124,6 +125,7 @@ class TaskHandler(object):
 onProgress = None
 kwargs = {
 'onProgress': onProgress,
 +   'module_output': self.module_output,
 # pass in a copy of the options so a
 module can not pollute or change
 # them for other tasks if there is more
 to do.
 'options': options.copy()
 @@ -219,5 +221,5 @@ def emaint_main(myargv):
 # need to pass the parser options dict to the modules
 # so they are available if needed.
 task_opts = options.__dict__
 -   taskmaster = TaskHandler(callback=print_results)
 +   taskmaster = TaskHandler(callback=print_results,
 module_output=sys.stdout)
 taskmaster.run_tasks(tasks, func, status, options=task_opts)
 diff --git a/pym/portage/emaint/modules/merges/__init__.py
 b/pym/portage/emaint/modules/merges/__init__.py
 new file mode 100644
 index 000..96ee71b
 --- /dev/null
 +++ b/pym/portage/emaint/modules/merges/__init__.py
 @@ -0,0 +1,30 @@
 +# Copyright 2005-2014 Gentoo Foundation
 +# Distributed under the terms of the GNU General Public License v2
 +
 +Scan for failed merges and fix them.
 +
 +
 +module_spec = {
 +   'name': 'merges',
 +   'description': __doc__,
 +   'provides': {
 +   'merges': {
 +   'name': merges,
 +   'class': MergesHandler,
 +   'description': __doc__,
 +   'functions': ['check', 'fix', 'purge'],
 +   'func_desc': {
 +   'purge': {
 +   'short': '-P', 'long':
 '--purge-tracker',
 +   'help': 'Removes the list of
 previously failed merges.' +
 +   ' WARNING: Only
 use this option if you plan on' +
 +   ' manually fixing
 them or do not want them'
 +   ' re-installed.',
 +   'status': Removing %s,
 +   'action': 'store_true',
 +   'func': 'purge'
 +   }
 +   }
 +   }
 +   }
 +}
 diff --git a/pym/portage/emaint/modules/merges/merges.py
 b/pym/portage/emaint/modules/merges/merges.py
 new file mode 100644
 index 000..a99dad4
 --- /dev/null
 +++ b/pym/portage/emaint/modules/merges/merges.py
 @@ -0,0 +1,281 @@
 +# Copyright 2005-2014 Gentoo Foundation
 +# Distributed under the terms of the GNU General Public License v2
 +
 +from _emerge.actions import load_emerge_config
 +
 +import portage
 +from portage import os, _unicode_encode
 +from portage.const import MERGING_IDENTIFIER, PORTAGE_BIN_PATH,
 PRIVATE_PATH, \
 +   VDB_PATH
 +from portage.dep import isvalidatom
 +


 import shutil
 import subprocess
 import sys
 import time

 Alphabetical order.


  +import time
 +import shutil
 +import sys
 +import subprocess
 +
 +class TrackingFile(object):
 +   File for keeping track of failed merges.
 +
 +
 +   def __init__(self, tracking_path):
 +   
 +   Create a TrackingFile object.
 +
 +   @param tracking_path: file path used to keep track of
 failed merges
 +   @type tracking_path: String

Re: [gentoo-portage-dev] [PATCH] Add an emaint module that can scan for failed merges and that can fix failed merges.

2014-02-19 Thread Alec Warner
On Wed, Feb 19, 2014 at 12:31 PM, Pavel Kazakov nullishz...@gentoo.orgwrote:

 ---
  pym/portage/emaint/modules/merges/__init__.py | 20 
  pym/portage/emaint/modules/merges/merges.py   | 70
 +++
  2 files changed, 90 insertions(+)
  create mode 100644 pym/portage/emaint/modules/merges/__init__.py
  create mode 100644 pym/portage/emaint/modules/merges/merges.py

 diff --git a/pym/portage/emaint/modules/merges/__init__.py
 b/pym/portage/emaint/modules/merges/__init__.py
 new file mode 100644
 index 000..2cd79af
 --- /dev/null
 +++ b/pym/portage/emaint/modules/merges/__init__.py
 @@ -0,0 +1,20 @@
 +# Copyright 2005-2014 Gentoo Foundation
 +# Distributed under the terms of the GNU General Public License v2
 +
 +Scan for failed merges and fix them.
 +


Scan for failed merges fix them.


 +
 +
 +module_spec = {
 +   'name': 'merges',
 +   'description': __doc__,
 +   'provides':{


'module1' ?


 +   'module1': {
 +   'name': merges,
 +   'class': MergesHandler,
 +   'description': __doc__,
 +   'functions': ['check', 'fix',],
 +   'func_desc': {}
 +   }
 +   }
 +   }
 diff --git a/pym/portage/emaint/modules/merges/merges.py
 b/pym/portage/emaint/modules/merges/merges.py
 new file mode 100644
 index 000..b243082
 --- /dev/null
 +++ b/pym/portage/emaint/modules/merges/merges.py
 @@ -0,0 +1,70 @@
 +# Copyright 2005-2014 Gentoo Foundation
 +# Distributed under the terms of the GNU General Public License v2
 +
 +import portage
 +from portage import os
 +from portage.const import PRIVATE_PATH, VDB_PATH
 +from portage.util import writemsg
 +
 +import shutil
 +import sys
 +
 +if sys.hexversion = 0x300:
 +   # pylint: disable=W0622
 +   long = int


What is this little guy? Can we just do this in a library someplace?


 +
 +class MergesHandler(object):
 +
 +   short_desc = Remove failed merges
 +


 @staticmethod decorator?

 +   def name():
 +   return merges
 +   name = staticmethod(name)
 +
 +   def __init__(self):


Generally you want to be able to change these later, so you might do
something like:

def __init__(self, eroot=None, vardb=None, vardb_path=None):
  self._eroot = error or portage.settings['EROOT']
  ... and so forth.

  Also..why can't self._vardb_path be queried from the vardb?


 +   self._eroot = portage.settings['EROOT']
 +   self._vardb = portage.db[self._eroot][vartree].dbapi
 +   self._vardb_path = os.path.join(self._eroot, VDB_PATH) +
 os.path.sep
 +
 +   def can_progressbar(self, func):
 +   return True
 +
 +   def _failed_packages(self, onProgress):
 +   for cat in os.listdir(self._vardb_path):

  os.listdir(os.path.join(...)) ?

 +   pkgs = os.listdir(self._vardb_path + cat)
 +   maxval = len(pkgs)
 +   for i, pkg in enumerate(pkgs):
 +   if onProgress:
 +   onProgress(maxval, i+1)
 +
 +   if '-MERGING-' in pkg:
 +   yield cat + os.path.sep + pkg
 +
 +   def check(self, **kwargs):
 +   onProgress = kwargs.get('onProgress', None)
 +   failed_pkgs = []
 +   for pkg in self._failed_packages(onProgress):
 +   failed_pkgs.append(pkg)
 +
 +   errors = ['%s' failed to merge. % x for x in failed_pkgs]
 +   return errors
 +
 +   def fix(self, **kwargs):
 +   onProgress = kwargs.get('onProgress', None)
 +   tracking_path = os.path.join(self._eroot, PRIVATE_PATH,
 'failed-merges');
 +   try:
 +   with open(tracking_path, 'w') as tracking_file:


is this unicode safe?


 +   for failed_pkg in
 self._failed_packages(onProgress):
 +
 tracking_file.write(failed_pkg + '\n')
 +   pkg_path =
 self._vardb_path + failed_pkg


os.path.join(...)


 +   # Delete failed merge
 directory
 +   # XXX: Would be a good
 idea to attempt try removing
 +   # package contents to
 prevent orphaned files


# XXX is terrible style. I realize a bunch of code does that, and it is
stupid.
# use
# TODO: foo




 +   shutil.rmtree(pkg_path)
 +   # Re-emerge package

+   pkg_name = '=' +
 failed_pkg.replace('-MERGING-', '')
 +
 features='FEATURES=-collision-detect -protect-owned'
 +   emerge_cmd=emerge
 --verbose --oneshot 

Re: [gentoo-portage-dev] [PATCH v2] Add --output-style option to repoman

2014-02-13 Thread Alec Warner
On Thu, Feb 13, 2014 at 7:42 AM, Brian Dolbec dol...@gentoo.org wrote:

 On Thu, 13 Feb 2014 03:19:35 -0500
 Mike Frysinger vap...@gentoo.org wrote:

  On Monday, February 10, 2014 20:22:36 Chris Reffett wrote:
   This patch adds a --output-style option to repoman, which gives the
   user a choice of output formats for the repoman checks. Choices are
   default (current style) and column (a greppable format), but it
   should be easy to add more. Fixes bug 481584.
 
  i'd expect a proper structured output would make sense to include in
  the default set.  like JSON.  just create a dict and send it to
  json.dump().

 He is working on more changes to repoman and the output. So, if you
 can, Chris, then do it, add a json option.


 
   v2: Fix docstring to be complete and in the standard format, make
   use of default choices in --output-style wrt comments by antarus
   and dol-sen
 
  erm, i thought the previous docstring was correct.  it followed
  PEP257 while this new one is like javadoc or something.
 

 It is the existing format that has been around in portage for years.
 There is even a page for it:

 http://www.gentoo.org/proj/en/portage/doc/policies/docstring-spec.xml

 It is also the style that epydoc recognizes.

   -utilities.format_qa_output(f, stats, fails, dofull, dofail,
   options, qawarnings)
   +if options.output_style == 'column':
   +   utilities.format_qa_output_column(f, stats, fails, dofull,
   dofail, options, qawarnings)
   +else:
   +   utilities.format_qa_output(f, stats, fails, dofull,
   dofail, options, qawarnings)
 
  use a func pointer instead.
  format_outputs = {
'column': utilities.format_qa_output_column,
'default': utilities.format_qa_output,
  }
  format_output = format_outputs.get(options.output_style,
format_outputs['default'])
  format_output(f, stats, fails, dofull, dofail, options, qawarnings)
 

 yeah, make it so.  Good spot, Mike


 Since Mike was too slow in replying, make another commit to change
 it.

   +   formatter.add_literal_data(NumberOf  + category
   +  )
 
  prefer to use % rather than + like so:
'NumberOf %s ' % category
 
   +   formatter.add_literal_data(%s % number)
 

 well actually, for simple additions like that, string1 + string2, it is
 actually faster.
 But for multiple additions,  %s is much better, faster.  Also if the
 string is translated, then use %s regardless.  That way the %s can be
 moved around for the translation.


In general we prefer % for readability purposes, not because it is faster.

foo = Bar + foo +   + baz + , + goat

foo = Bar %s %s, %s % (foo, baz, goat)

I think this case could go either way, because even with %, NumberOf%s is
not much of an improvement.
The code is littered with the former though, and it makes it really
annoying to read ;)

-A




  str(number)
  -mike



 --
 Brian Dolbec dolsen




Re: [gentoo-portage-dev] [PATCH] Add --output-style option to repoman

2014-02-10 Thread Alec Warner
On Mon, Feb 10, 2014 at 2:57 PM, Chris Reffett creff...@gentoo.org wrote:

 This patch adds a --output-style option to repoman, which gives the user
 a choice of output formats for the repoman checks. Choices are default
 (current style) and column (a greppable format), but it should be easy
 to add more. Fixes bug 481584.
 ---
  bin/repoman  | 18 +-
  pym/repoman/utilities.py | 38 ++
  2 files changed, 55 insertions(+), 1 deletion(-)

 diff --git a/bin/repoman b/bin/repoman
 index 3504b6b..957ee08 100755
 --- a/bin/repoman
 +++ b/bin/repoman
 @@ -144,9 +144,17 @@ def ParseArgs(argv, qahelp):
 'scan' : 'Scan directory tree for QA issues'
 }

 +   output_choices = {
 +   'default' : 'The normal output format',
 +   'column' : 'Columnar output suitable for use with grep'
 +   }
 +
 mode_keys = list(modes)
 mode_keys.sort()


output_keys = sorted(output_choices)

?


 +   output_keys = list(output_choices)
 +   output_keys.sort
 +
 parser = ArgumentParser(usage=repoman [options] [mode],
 description=Modes: %s %  | .join(mode_keys),
 epilog=For more help consult the man page.)
 @@ -228,6 +236,9 @@ def ParseArgs(argv, qahelp):
 parser.add_argument('--without-mask', dest='without_mask',
 action='store_true',
 default=False, help='behave as if no package.mask entries
 exist (not allowed with commit mode)')

 +   parser.add_argument('--output-style', dest='output_style',
 choices=output_keys,
 +   help='select output type')
 +
 parser.add_argument('--mode', dest='mode', choices=mode_keys,
 help='specify which mode repoman will run in
 (default=full)')

 @@ -256,6 +267,8 @@ def ParseArgs(argv, qahelp):
 if opts.mode == 'ci':
 opts.mode = 'commit'  # backwards compat shortcut

 +   if not opts.output_style:
 +   opts.output_style = 'default' #default to the standard
 output type
 # Use the verbosity and quiet options to fiddle with the loglevel
 appropriately
 for val in range(opts.verbosity):
 logger = logging.getLogger()
 @@ -2422,7 +2435,10 @@ console_writer.style_listener =
 style_file.new_styles

  f = formatter.AbstractFormatter(console_writer)

 -utilities.format_qa_output(f, stats, fails, dofull, dofail, options,
 qawarnings)
 +if options.output_style == 'column':
 +   utilities.format_qa_output_column(f, stats, fails, dofull, dofail,
 options, qawarnings)
 +else:
 +   utilities.format_qa_output(f, stats, fails, dofull, dofail,
 options, qawarnings)

  style_file.flush()
  del console_writer, f, style_file
 diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
 index 3ec3a4a..713f208 100644
 --- a/pym/repoman/utilities.py
 +++ b/pym/repoman/utilities.py
 @@ -330,6 +330,44 @@ def format_qa_output(formatter, stats, fails, dofull,
 dofail, options, qawarning
 formatter.add_line_break()


 +def format_qa_output_column(formatter, stats, fails, dofull, dofail,
 options, qawarnings):
 +   Helper function that formats output in a machine-parseable
 column format
 +
 +   Args:
 +   formatter - a subclass of Formatter
 +   stats - a dict of qa status items
 +   fails - a dict of qa status failures
 +   dofull - boolean to print full results or a summary
 +   dofail - boolean to decide if failure was hard or soft


missing options and qawarnings ?


 +
 +   Returns:
 +   None (modifies formatter)
 +   
 +   full = options.mode == 'full'
 +   for category, number in stats.items():
 +   # we only want key value pairs where value  0
 +   if number  1:
 +   continue
 +
 +   formatter.add_literal_data(NumberOf  + category +  )
 +   if category in qawarnings:
 +   formatter.push_style(WARN)
 +   else:
 +   formatter.push_style(BAD)
 +   formatter.add_literal_data(%s % number)
 +   formatter.pop_style()
 +   formatter.add_line_break()
 +   if not dofull:
 +   if not full and dofail and category in qawarnings:
 +   # warnings are considered noise when there
 are failures
 +   continue
 +   fails_list = fails[category]
 +   if not full and len(fails_list)  12:
 +   fails_list = fails_list[:12]
 +   for failure in fails_list:
 +   formatter.add_literal_data(category +  
 + failure)
 +   formatter.add_line_break()
 +
  def editor_is_executable(editor):
 
 Given an EDITOR 

Re: [gentoo-portage-dev] [RFC] Location of Portage bash API (outside of ebuilds)

2014-02-01 Thread Alec Warner
On Sat, Feb 1, 2014 at 12:08 PM, Arfrever Frehtes Taifersar Arahesis 
arfrever@gmail.com wrote:

 bin/isolated-functions.sh contains at least 1 useful function, which could
 be exposed for external consumers
 (without __ prefix), but must have private name (with __ prefix) when
 bin/isolated-functions.sh is used in
 ebuild environment.

 Possible solutions:
 1. Make /usr/lib/portage/bin/isolated-functions.sh magically define
 non-private variants of useful functions
when run in non-ebuild environment.
 2. Provide /usr/bin/portage.bash, which would source isolated-functions.sh
 (and maybe other files) and define
non-private variants of useful functions. /usr/bin/portage.bash would
 be easier sourceable due to PATH.
 3. Provide /usr/lib/portage.bash, which would source isolated-functions.sh
 (and maybe other files) and define
non-private variants of useful functions.
 4. Another location...

 (I would probably prefer solution #2.)

 I think 1 isn't very unix-y, 2-3 are fine. If we expect users to run it,
it goes in bin, if not, then lib.

-A



 --
 Arfrever Frehtes Taifersar Arahesis



Re: [gentoo-portage-dev] [PATCH 3/3] pym/portage/package/ebuild/fetch.py: Factor out _get_uris

2014-01-19 Thread Alec Warner
On Sat, Jan 18, 2014 at 7:07 PM, W. Trevor King wk...@tremily.us wrote:

 The current fetch() function is quite long, which makes it hard to
 know what I can change without adverse side effects.  By pulling this
 logic out of the main function, we get clearer logic in fetch() and
 more explicit input for the config extraction.

 This block was especially complicated, so I also created the helper
 functions _get_file_uri_tuples and _expand_mirror.  I'd prefer if
 _expand_mirror iterated through URIs instead of (group, URI) tuples,
 but we need a distinct marker for third-party URIs to build
 third_party_mirror_uris which is used to build primaryuri_dict which
 is used way down in fetch():

   if checksum_failure_count == \
   checksum_failure_primaryuri:
   # Switch to primaryuri mode in order
   # to increase the probablility of
   # of success.
   primaryuris = \
   primaryuri_dict.get(myfile)
   if primaryuris:
   uri_list.extend(
   reversed(primaryuris))

 I don't know if this is useful enough to motivate the uglier
 _expandmirror return values, but I'll kick that can down the road for
 now.
 ---
  pym/portage/package/ebuild/fetch.py | 197
 +---
  1 file changed, 117 insertions(+), 80 deletions(-)

 diff --git a/pym/portage/package/ebuild/fetch.py
 b/pym/portage/package/ebuild/fetch.py
 index bd572fa..a617945 100644
 --- a/pym/portage/package/ebuild/fetch.py
 +++ b/pym/portage/package/ebuild/fetch.py
 @@ -15,9 +15,9 @@ import sys
  import tempfile

  try:
 -   from urllib.parse import urlparse
 +   from urllib.parse import urlparse, urlunparse
  except ImportError:
 -   from urlparse import urlparse
 +   from urlparse import urlparse, urlunparse

  import portage
  portage.proxy.lazyimport.lazyimport(globals(),
 @@ -297,6 +297,118 @@ def _get_fetch_resume_size(settings, default='350K'):
 return v


 +def _get_file_uri_tuples(uris):
 +   Return a list of (filename, uri) tuples
 +   


As mike noted on another thread:

Return a list of (filename, uri) tuples.

+   file_uri_tuples = []
 +   # Check for 'items' attribute since OrderedDict is not a dict.
 +   if hasattr(uris, 'items'):
 +   for filename, uri_set in uris.items():
 +   for uri in uri_set:
 +   file_uri_tuples.append((filename, uri))
 +   if not uri_set:
 +   file_uri_tuples.append((filename, None))
 +   else:
 +   for uri in uris:
 +   if urlparse(uri).scheme:
 +   file_uri_tuples.append(
 +   (os.path.basename(myuri), myuri))
 +   else:
 +   file_uri_tuples.append(
 +   (os.path.basename(myuri), None))
 +   return file_uri_tuples
 +
 +
 +def _expand_mirror(uri, custom_mirrors=(), third_party_mirrors=()):
 +   Replace the 'mirror://' scheme in the uri


Sentence should end in a period.


 +
 +   Returns an iterable listing expanded (group, URI) tuples,
 +   where the group is either 'custom' or 'third-party'.
 +
 +   parsed = urlparse(uri)
 +   mirror = parsed.netloc
 +   path = parsed.path
 +   if path:
 +   # Try user-defined mirrors first
 +   if mirror in custom_mirrors:
 +   for cmirr in custom_mirrors[mirror]:
 +   m_uri = urlparse(cmirr)
 +   yield ('custom', urlunparse((
 +   cmirr.scheme, cmirr.netloc, path) +
 +   parsed[3:]))
 +
 +   # now try the official mirrors
 +   if mirror in third_party_mirrors:
 +   uris = []
 +   for locmirr in third_party_mirrors[mirror]:
 +   m_uri = urlparse(cmirr)
 +   uris.append(urlunparse((
 +   cmirr.scheme, cmirr.netloc, path) +
 +   parsed[3:]))
 +   random.shuffle(uris)
 +   for uri in uris:
 +   yield ('third-party', uri)
 +   else:
 +   writemsg(_(Invalid mirror definition in SRC_URI:\n),
 +noiselevel=-1)
 +   writemsg(  %s\n % (uri), noiselevel=-1)


Is this a py3k thing? You should be able to write:

writemsg(  %s\n % uri, noiselevel=-1)

The tuple is only needed for multiple arguments in py2k.



+
 +
 +def _get_uris(uris, settings, custom_mirrors=(), locations=()):


I want you to be careful here. This is bad style when you are constructing
mutable objects in parameter defaults:

 def func(a=[]):
...   a.append(5)
...
 del func
 def func(a=[]):
...   

Re: [gentoo-portage-dev] [PATCH v2 1/3] pym/portage/package/ebuild/fetch.py: Factor out _get_checksum_failure_max_tries

2014-01-19 Thread Alec Warner
On Sun, Jan 19, 2014 at 2:44 PM, Alec Warner anta...@gentoo.org wrote:

 On Sun, Jan 19, 2014 at 2:14 PM, W. Trevor King wk...@tremily.us wrote:

 The current fetch() function is quite long, which makes it hard to
 know what I can change without adverse side effects.  By pulling this
 logic out of the main function, we get clearer logic in fetch() and
 more explicit input for the config extraction.
 ---
  pym/portage/package/ebuild/fetch.py | 59
 +
  1 file changed, 34 insertions(+), 25 deletions(-)

 diff --git a/pym/portage/package/ebuild/fetch.py
 b/pym/portage/package/ebuild/fetch.py
 index 5316f03..6f46572 100644
 --- a/pym/portage/package/ebuild/fetch.py
 +++ b/pym/portage/package/ebuild/fetch.py
 @@ -240,6 +240,38 @@ _size_suffix_map = {
 'Y' : 80,
  }

 +
 +def _get_checksum_failure_max_tries(settings, default=5):
 +   
 +   Get the maximum number of failed download attempts
 +
 +   Generally, downloading the same file repeatedly from
 +   every single available mirror is a waste of bandwidth
 +   and time, so there needs to be a cap.
 +   
 +   v = default
 +   try:
 +   v = int(settings.get(PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS,
 +   default))
 +   except (ValueError, OverflowError):
 +   writemsg(_(!!! Variable
 PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS
 +contains non-integer value: '%s'\n) % \
 +   settings[PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS],
 +   noiselevel=-1)
 +   writemsg(_(!!! Using PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS 
 +   default value: %s\n) % default,
 +   noiselevel=-1)
 +   v = default
 +   if v  1:
 +   writemsg(_(!!! Variable
 PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS
 +contains value less than 1: '%s'\n) % v,
 noiselevel=-1)
 +   writemsg(_(!!! Using PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS 
 +   default value: %s\n) % default,
 +   noiselevel=-1)
 +   v = default
 +   return v
 +
 +


 This function and the next function you wrote are identical. How about
 making a single function?

 def getValueFromSettings(settings, key, default, validator):
   try:
 return validator(settings.get(key, default))
   except ValidationError as e:
 # writemsg the exception, it will contain the juicy bits.)

 def getIntValueFromSettings(settings, key, default):
   def validator(v):
 try:
   v = int(v)
except (ValueError, OverflowError) as e:
   raise ValidationError( bla bla bla not an int, we picked the
 default.)
 if v  1:
   v = default
   raise ValidationError(bla bla bla less than one we used the
 defualt.)

   return getValueFromSettings(settings, key, default, validator)


Note, don't actually use these function names, they are terrible.

-A



 Then we are not necessarily writing a bunch of the same functions over and
 over. The defaults we can stick in a config file, or in python, or at the
 call site.

 -A








  def fetch(myuris, mysettings, listonly=0, fetchonly=0,
 locks_in_subdir=.locks, use_locks=1, try_mirrors=1,
 digests=None,
 allow_missing_digests=True):
 @@ -263,31 +295,8 @@ def fetch(myuris, mysettings, listonly=0,
 fetchonly=0,
 print(_( \mirror\ mode desired and
 \mirror\ restriction found; skipping fetch.))
 return 1

 -   # Generally, downloading the same file repeatedly from
 -   # every single available mirror is a waste of bandwidth
 -   # and time, so there needs to be a cap.
 -   checksum_failure_max_tries = 5
 -   v = checksum_failure_max_tries
 -   try:
 -   v =
 int(mysettings.get(PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS,
 -   checksum_failure_max_tries))
 -   except (ValueError, OverflowError):
 -   writemsg(_(!!! Variable
 PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS
 -contains non-integer value: '%s'\n) % \
 -   mysettings[PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS],
 noiselevel=-1)
 -   writemsg(_(!!! Using PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS 
 -   default value: %s\n) %
 checksum_failure_max_tries,
 -   noiselevel=-1)
 -   v = checksum_failure_max_tries
 -   if v  1:
 -   writemsg(_(!!! Variable
 PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS
 -contains value less than 1: '%s'\n) % v,
 noiselevel=-1)
 -   writemsg(_(!!! Using PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS 
 -   default value: %s\n) %
 checksum_failure_max_tries,
 -   noiselevel=-1)
 -   v = checksum_failure_max_tries
 -   checksum_failure_max_tries = v
 -   del v
 +   checksum_failure_max_tries

Re: [gentoo-portage-dev] [PATCH 0/3] Initial fetch() refactoring

2014-01-19 Thread Alec Warner
On Sun, Jan 19, 2014 at 2:45 PM, Alexander Berntsen alexan...@plaimi.netwrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 On 19/01/14 22:22, Sebastian Luther wrote:
  The usual doc string style used in portage is:
 
   text 
 
  Please use that for new functions. Also make sure you don't use
  spaces to indent the last .
 
 As mentioned by Mike in another thread, we should use PEP 257[0]. I
 will convert old code to conform to this... sometime... soon... (I
 promise!)

 So if new patches could just do that right away, that would be neat.


Does pylint or pyflakes point out if you mess it up?

Automation for the win.

-A



 [0]  http://www.python.org/dev/peps/pep-0257/
 - --
 Alexander
 alexan...@plaimi.net
 http://plaimi.net/~alexander
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2.0.22 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

 iF4EAREIAAYFAlLcVaAACgkQRtClrXBQc7XjTwD/UsCQo+SJdiCX/QsMC8jFDPic
 k0jEAN6DA5xML6/nJYQA/36FszMfVMZ/vzg5VF9FS6BDWwGm/Qy2whyqLiF3FOrX
 =+5Yy
 -END PGP SIGNATURE-




Re: [gentoo-portage-dev] [PATCH 0/3] Initial fetch() refactoring

2014-01-19 Thread Alec Warner
On Sun, Jan 19, 2014 at 2:50 PM, Alexander Berntsen alexan...@plaimi.netwrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 On 19/01/14 23:46, Alec Warner wrote:
  Does pylint or pyflakes point out if you mess it up?
 Not very successfully in my experience.


I'm very against add a bunch of extra rules that have to be enforced by
hand. I want to make it easy to contribute, not more difficult. If bob can
run a tool that tells him all the things that are wrong with his patch,
that avoids us having like 1/3rd of the conversations on list ;)

-A


 - --
 Alexander
 alexan...@plaimi.net
 http://plaimi.net/~alexander
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2.0.22 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

 iF4EAREIAAYFAlLcVrsACgkQRtClrXBQc7XZcQD9GD8lV8HpZbQjpqu8ggNh4vOo
 /fvFsCI2PaWiEsC4ISUA/iYRiXAYD7SEF4a3R/t6vOdmEQSHyvZcxH9NDwI4/biQ
 =RMJ0
 -END PGP SIGNATURE-




Re: [gentoo-portage-dev] [PATCH 0/3] Initial fetch() refactoring

2014-01-19 Thread Alec Warner
On Sun, Jan 19, 2014 at 3:51 PM, Alexander Berntsen alexan...@plaimi.netwrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 On 19/01/14 23:54, Alec Warner wrote:
  I'm very against add a bunch of extra rules that have to be
  enforced by hand. I want to make it easy to contribute, not more
  difficult. If bob can run a tool that tells him all the things
  that are wrong with his patch, that avoids us having like 1/3rd of
  the conversations on list ;)
 Feel free to write a tool for this, or to contribute to any of the
 numerous linters and/or editor plug-ins. It would be much appreciated.


I already prefer pylint, and I think it does cover most of what I want. I
am working a pylintrc patch.


 As for the difficulty of PEP 257... I have higher hopes for Portage
 contributors than getting stuck at that. If I write a patch that makes
 most of the docstrings follow it, then they can infer 99% of the
 extra rules by just looking at the other functions and methods. If
 they fail to comply, we can just mention it. If the docstring
 formatting is the biggest issue with their patch, I doubt they'll have
 a hard time fixing it.


I'm not saying its hard, I'm saying it is a giant waste of time for the
list to tell people 'hey your docstrings are wrong' when they can just run
a tool to do it ;)

-A




 - --
 Alexander
 alexan...@plaimi.net
 http://plaimi.net/~alexander
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2.0.22 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

 iF4EAREIAAYFAlLcZPQACgkQRtClrXBQc7XizAD/Y/Gxc7N6VkgNFWRgP5lmQ84r
 UwSne2xaqJYphY9x1TcBAIRpjBHB580edLz/8zpT14lqhW3oOmeMz0pNMB8ssW5d
 =+zp5
 -END PGP SIGNATURE-




Re: [gentoo-portage-dev] [PATCH 1/3 v2] Have repoman check if the packages to unpack rare archive formats from SRC_URI are present in DEPEND (bug #205909).

2014-01-19 Thread Alec Warner
On Sun, Jan 19, 2014 at 6:23 PM, Tom Wijsman tom...@gentoo.org wrote:

 On Sun, 19 Jan 2014 04:38:31 -0500
 Mike Frysinger vap...@gentoo.org wrote:

  On Friday 17 January 2014 18:03:57 Tom Wijsman wrote:
   ---
 
  please shorten your commit summary and move more content to the body

 Right, I'm used to writing this on the command line; I'll try to look
 into setting up an editor or a GUI (gitg, ...) to do my commit messages.

   +getNonSystemArchiveDepends.archivers = {
 
  it is super weird to attach to the object like this.  some might even
  say wrong.

 It a function, this makes it a static function variable.

 What is weird here? It works properly, what would be wrong?


It is certainly weird (as we discussed on IRC.) I've never seen anyone do
it in any codebase I liked.

One of the problems is that it isn't immutable, so that earlier callers can
mess with later callers. That is not possible in vapier's proposal, as the
attributes are hidden in the function code and are not visible to callers.

 move it into the class definition.
  def getNonSystemArchiveDepends(fetchlist, eapi):
...
 
ARCHIVERS = {
...
}

 That makes it a non-static function variable? This is a regression.


I guess I am not seeing why it must be a static function variable. Can you
explain?



   +   re.compile('.*\.7[zZ]$'):app-arch/p7zip,
 
  regexes should always use raw strings.  there should also be a space
  after the colon in dicts.  so you want:
 
re.compile(r'.*\.7[zZ]$'): 'app-arch/p7zip',

 Raw strings are a solution when \ forms a problem, which is not the
 case here. The colon has no space after it in the rest of repoman near
 it, hence I left it out; is there a coding standard we're trying to
 respect here? Can you refer it to me?


For the colon's in dicts thing:

http://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Whitespace



   +   re.compile('.*\.lzma$'):app-arch/lzma-utils,
  xz-utils, not lzma-utils

 http://dev.gentoo.org/~ulm/pms/5/pms.html

 Please see unpack in PMS 11.3.3.13 Misc Commands, quote:

 lzma-compressed files (*.lzma). Ebuilds must ensure that LZMA Utils is
 installed.

 The TODO comment that had a PMS reference was near it before, but with
 the refactor it has since moved; I'll put back the reference to it.

   +   re.compile('.*\.(bz2?|tbz2)$'):app-arch/bzip2,
   +
   re.compile('.*\.(tar(\.(bz2?|gz|Z))?|tbz2|t[bg]z)?$'):app-arch/tar,
   +   re.compile('.*\.(gz|tar\.Z|t[bg]z|[zZ])$'):app-arch/gzip,
   +   re.compile('.*\.tar.xz$'):app-arch/tar,
 
  requiring people list gzip, tar, and bzip2 is a significant policy
  change (which i'm inclined to say is wrong).  it needs discussion on
  the dev mailing list first.

 Please see unpack in PMS 11.3.3.13 Misc Commands, example quote:

 gzip-compressed tar files (*.tar.gz, *.tgz, *.tar.Z, *.tbz). Ebuilds
 must ensure that GNU gzip and GNU tar are installed.

 To spare on mail length, I don't quote the rest of that enumeration.


The @system set in gentoo will ensure these are installed. I understand the
wording of PMS (as the dependencies should be expressed somewhere) but in
general we prefer to do that in @system. For the same reason all packages
do not depend on glibc, or the compiler, or anything else.



   +def checkArchiveDepends(atoms, catpkg, relative_path, \
   +   system_set_atoms, needed_unpack_depends, stats, fails):
 
  you don't need the \ there because you have paren already to gather
  things.
 
   +   for entry in needed_unpack_depends[catpkg]:
   +   if entry not in [atom.cp for atom in atoms if
   atom != ||]:
   +   stats['unpack.' + mytype + '.missing'] += 1
   +   fails['unpack.' + mytype +
   '.missing'].append( \
   +   relative_path + : %s is missing
   in %s % \
   +   (entry, mytype))
 
  i know existing style doesn't follow it, but imo we should avoid
  string concatenation.  it makes the code harder to read imo.
 
key = 'unpack.%s.missing' % mytype
stats[key] += 1
fails[key].append(...)
 
   +def eapi_has_xz_utils(eapi):
   +   return eapi not in (0, 1, 2)
 
  i would use eapi_supports_xz_archives unless there's a pre-existing
  style -mike

 Good idea, I'll take these last ones into account in the next version.

 Thank you everyone for the reviews so far.

 --
 With kind regards,

 Tom Wijsman (TomWij)
 Gentoo Developer

 E-mail address  : tom...@gentoo.org
 GPG Public Key  : 6D34E57D
 GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D



Re: [gentoo-portage-dev] [PATCH] Add repoman check to warn if src_prepare/src_configure are used in EAPI 0/1

2014-01-19 Thread Alec Warner
On Sun, Jan 19, 2014 at 9:47 PM, Mike Frysinger vap...@gentoo.org wrote:

 On Sunday 19 January 2014 11:59:36 Mike Gilbert wrote:
  On Sun, Jan 19, 2014 at 4:44 AM, Mike Frysinger vap...@gentoo.org
 wrote:
   Chromium OS for a long time was restricted to EAPI 4 for two reasons --
   it had an old portage version (and upgrading to a newer one regressed
   performance significantly, so we held off until we could figure out
 why)
 
  I am curious to know more about the performance regression if you can
  share. Is that something that got fixed, or did you disable some
  features (like the slot-operator stuff)?

 we finally tracked it down (was due to new the new FEATURES=merge-sync
 option.
 when you're installing 200 to 500 binary packages (with like 32 in
 parallel),
 that can easily choke your throughput.  some systems saw really excessive
 latencies (which i would guess was due to their drive taking longer to
 process
 dirty blocks).  but it took us some time to figure that out as we were
 making a
 large version jump and we didn't have too much time to dedicate to
 tracking it
 down.  lots o bugs to fix.
 -mike


Are you still doing crazy crap like disabling all of the portage locking? ;p

-A


Re: [gentoo-portage-dev] Signing off patches

2014-01-16 Thread Alec Warner
On Thu, Jan 16, 2014 at 5:20 AM, Alexander Berntsen alexan...@plaimi.netwrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 We have quite a few dedicated developers now. To ensure that good
 taste is exercised, and that best practices are followed, patches
 should be signed.


I'm confused, are you proposing all patches have all of these fields? Or we
should simply cherry-pick the fields we think are useful?



 My proposals:
 Signed-off-by: Wrote (a substantial portion of) the patch
 Reviewed-by: Reviewed the patch thoroughly
 Tested-by:  Tested the patch thoroughly
 Acked-by: Approved the concept but did not read the patch in detail
 (typically used by the maintainer of a specific portion, or our lead)
 Suggested-by: Designed the implementation
 Reported-by: Reported the bug/feature request

 These suggestions all stem from the Linux project.
 - --
 Alexander
 alexan...@plaimi.net
 http://plaimi.net/~alexander
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2.0.22 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

 iF4EAREIAAYFAlLX3JsACgkQRtClrXBQc7VhEQD9FKmFbyf9zxl+hLylkhQN/kv6
 o3DSM4xBr9fH4+1eokYA/3MbFLtDpli311d6ZqGD17kGLfz5wNj+5kPRATbiC256
 =cJNe
 -END PGP SIGNATURE-




Re: [gentoo-portage-dev] [PATCH 1/3] Have repoman check if the packages to unpack rare archive formats from SRC_URI are present in DEPEND (bug #205909).

2014-01-15 Thread Alec Warner
On Wed, Jan 15, 2014 at 4:07 PM, Tom Wijsman tom...@gentoo.org wrote:

 ---
  bin/repoman   | 53 +
  man/repoman.1 |  4 
  2 files changed, 57 insertions(+)


I urge you to not author new checks like this. /usr/bin/repoman is already
a mess.

Write these checks as functions, put them in a different file, and just
call them from the giant messy loop. At least that way the checks are self
contained (great for avoiding things like variable re-use or shadowing).

-A



 diff --git a/bin/repoman b/bin/repoman
 index d1542e9..9b703dc 100755
 --- a/bin/repoman
 +++ b/bin/repoman
 @@ -36,6 +36,9 @@ pym_path =
 osp.join(osp.dirname(osp.dirname(osp.realpath(__file__))), pym)
  sys.path.insert(0, pym_path)
  import portage
  portage._internal_caller = True
 +
 +from portage._sets.profiles import PackagesSystemSet
 +system_set_atoms = PackagesSystemSet(portage.settings.profiles).getAtoms()
  portage._disable_legacy_globals()

  try:
 @@ -300,6 +303,7 @@ qahelp = {
 inherit.missing: Ebuild uses functions from an eclass but does
 not inherit it,
 inherit.unused: Ebuild inherits an eclass but does not use it,
 java.eclassesnotused: With virtual/jdk in DEPEND you must
 inherit a java eclass,
 +   unpack.DEPEND.missing: A rare archive format was used in
 SRC_URI, but its package to unpack it is missing in DEPEND.,
 wxwidgets.eclassnotused: Ebuild DEPENDs on x11-libs/wxGTK
 without inheriting wxwidgets.eclass,
 KEYWORDS.dropped: Ebuilds that appear to have dropped KEYWORDS
 for some arch,
 KEYWORDS.missing: Ebuilds that have a missing or empty KEYWORDS
 variable,
 @@ -399,6 +403,7 @@ qawarnings = set((
  metadata.warning,
  portage.internal,
  repo.eapi.deprecated,
 +unpack.DEPEND.missing,
  usage.obsolete,
  upstream.workaround,
  LIVEVCS.stable,
 @@ -479,6 +484,25 @@ ruby_deprecated = frozenset([
 ruby_targets_ree18,
  ])

 +# TODO: Add functionality to support checking for deb2targz on platforms
 where
 +#   GNU binutils is absent; see PMS 5, section 11.3.3.13.
 +archive_formats = {
 +   \.7[zZ]:app-arch/p7zip,
 +   \.(bz2?|tbz2):app-arch/bzip2,
 +   \.jar:app-arch/unzip,
 +   \.(LH[aA]|lha|lzh):app-arch/lha,
 +   \.lzma:app-arch/lzma-utils,
 +   \.(rar|RAR):app-arch/unrar,
 +   \.(tar(\.(bz2?|gz|Z))?|tbz2|t[bg]z)?:app-arch/tar,
 +   \.(gz|tar\.Z|t[bg]z|[zZ]):app-arch/gzip,
 +   \.(zip|ZIP):app-arch/unzip,
 +}
 +
 +archive_formats_eapi_3_to_5 = {
 +   \.tar.xz:app-arch/tar,
 +   \.xz:app-arch/xz-utils,
 +}
 +
  metadata_xml_encoding = 'UTF-8'
  metadata_xml_declaration = '?xml version=1.0 encoding=%s?' % \
 (metadata_xml_encoding,)
 @@ -1559,6 +1583,7 @@ for x in effective_scanlist:
 fetchlist_dict = portage.FetchlistDict(checkdir, repoman_settings,
 portdb)
 myfiles_all = []
 src_uri_error = False
 +   needed_unpack_depends = {}
 for mykey in fetchlist_dict:
 try:
 myfiles_all.extend(fetchlist_dict[mykey])
 @@ -1573,7 +1598,22 @@ for x in effective_scanlist:
 stats[SRC_URI.syntax] += 1
 fails[SRC_URI.syntax].append(
 %s.ebuild SRC_URI: %s % (mykey,
 e))
 +
 +   # Compare each SRC_URI entry against archive_formats; if
 one of the
 +   # extensions match, we remember which archive depends are
 needed to
 +   # check them later on.
 +   needed_unpack_depends[mykey] = []
 +   for file_extension in archive_formats or \
 +   ((re.match('[345]$', eapi) is not None) \
 +   and file_extension in
 archive_formats_eapi_3_to_5):
 +   for entry in fetchlist_dict[mykey]:
 +   if re.match('.*%s$' % file_extension,
 entry) is not None:
 +   format =
 archive_formats[file_extension]
 +
 +   if format not in
 needed_unpack_depends[mykey]:
 +
 needed_unpack_depends[mykey].append(format)
 del fetchlist_dict
 +
 if not src_uri_error:
 # This test can produce false positives if SRC_URI could
 not
 # be parsed for one or more ebuilds. There's no point in
 @@ -2010,6 +2050,17 @@ for x in effective_scanlist:
 atoms = None
 badsyntax.append(str(e))

 +   if atoms and mytype == 'DEPEND':
 +   # We check whether the needed archive
 dependencies are present
 +   # in DEPEND, which were determined from
 SRC_URI.
 +   for entry in needed_unpack_depends[catdir
 + '/' + y]:
 +   if entry not in system_set_atoms
 and entry \
 +  

Re: [gentoo-portage-dev] New proposed modular sync system

2014-01-09 Thread Alec Warner
On Thu, Jan 9, 2014 at 9:33 AM, Brian Dolbec dol...@gentoo.org wrote:

 First off, I know many of you think portage needs to do everything on
 it's own.  NO outside dependencies.

 BUT!  Please hear me out.

 While working on many of gentoo's tools gentoolkit's equery, eclean,
 enalyze, Layman, mirrorselect, etc..  I have found that there was a lot
 of needless code duplication.  Both within the apps themselves and
 copying/re-inventing the nearly identical code in other apps.

 I want to move many of gentoo's core applications to using some more
 shared libraries.  But that in itself is not a topic to discuss in this
 thread.  Please, do not sidetrack this thread.  The above is only for
 background info leading to this proposed change.  This proposal follows
 along that thinking.

 Many years ago after re-writing ecleans code I began re-writing
 portage's emaint code so that it could be imported and used internally
 in eclean to re-generate the Packages binpkg index file.  In that code I
 found a todo which said make a plug-in system so we don't have to keep
 modifying this code  or something along those lines.  So I did.  It
 took 4 years+, rebasing many times and migrating vcs's, but Zac finally
 merged it.  It has been used in portage for more than a year. That
 plug-in system I designed to be very flexible and basic.  It can be used
 for many areas of portage to simplify it's code and the maintenance
 required to keep current with changes in repository types, and transfer
 methods.

 I have started a Proposals sub-page under the Portage project page in
 the wiki.  It has a link to a diagram I made showing how the plug-in
 system is laid out.  This thread will be used to discuss the proposal
 and the details needed for the module_spec dictionary that is used to
 provide the manager with the details of what a module provides, options,
 etc..

 For a working example of the system in action, just try out emaint -h,
 and some other actions it provides.  Then move one of it's modules out
 of the the emaint/modules/ directory and re-try emaint -h.  You will see
 the options for that module and it's actions are no longer available.
 No code editing was required, no errors occurred,...  Put it back,
 retry...

 The modular sync system I propose would re-use that plug-in manager to
 interface the sync manager code with the modules that perform the sync
 actions.  This sytem is easily extended with new modules for different
 transfer methods or VCS specific modules.

 This also makes it possible for other gentoo applications like layman to
 install a layman specific module which syncs all layman overlays that it
 installs.  All this done with out effort on the part of the portage team
 either for maintenance or code changes to support it's addition.  The
 same can be said for Funtoo, if our git module is not suiotable, they
 just install their own custom sync module.  Done :)

 These add-on sync modules can also be installed via their own ebuild.

 For example:  Zac has for years had a custom script that syncs the
 portage tree and makes a squashfs version of it which he uses for his
 PORTDIR. it is faster than a normal one.  He, or someone else could make
 a app-portage/squashfs-sync ebuild that installs a module for that
 purpose.  The user then just needs to edit his repos.conf file and
 change the sync-type to squashfs.  Then emerge --sync will do it
 automatically without user intervention and auxiliary scripts run
 manually or cron job.

 This system would also make it possible to modify emerge-s cli to accept
 target repos to sync. and not any others also installed.  Similarly to
 layman -s some-overlay, emerge --sync some-overlay


I think the opposition to this idea primarily falls on reliability. There
have been a number of hacks to portage over the years to keep it
functioning during upgrades of itself, even down to the non-standard place
where its .py files are stored (/usr/lib/portage/pym?) So the real question
is when we move to a plug-in architecture, how do we add, remove, upgrade
the plugins without breaking the actual package manager?

-A




 P.S. sorry for such a long email
 --
 Brian Dolbec dol...@gentoo.org



Re: [gentoo-portage-dev] [PATCH] econf: update configure/config.{sub,guess} atomically to avoid races

2013-12-17 Thread Alec Warner
LGTM


On Tue, Dec 17, 2013 at 4:26 PM, Brian Dolbec dol...@gentoo.org wrote:

 On Tue, 2013-12-17 at 18:28 -0500, Mike Frysinger wrote:
  Use $BASHPID which will be unique even in subshells.
 
  URL: https://bugs.gentoo.org/487478
  ---
   bin/phase-helpers.sh | 17 +++--
   1 file changed, 11 insertions(+), 6 deletions(-)
 
  diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
  index ec48c94..1a7ae03 100644
  --- a/bin/phase-helpers.sh
  +++ b/bin/phase-helpers.sh
  @@ -469,6 +469,7 @@ unpack() {
 
   econf() {
local x
  + local pid=${BASHPID}
 
if ! ___eapi_has_prefix_variables; then
local EPREFIX=
  @@ -501,18 +502,22 @@ econf() {
if [[ -n $CONFIG_SHELL  \
$(head -n1 $ECONF_SOURCE/configure) =~
 ^'#!'[[:space:]]*/bin/sh([[:space:]]|$) ]] ; then
# preserve timestamp, see bug #440304
  - touch -r $ECONF_SOURCE/configure
 $ECONF_SOURCE/configure._portage_tmp_.$$ || die
  - sed -e
 1s:^#![[:space:]]*/bin/sh:#!$CONFIG_SHELL: -i $ECONF_SOURCE/configure
 || \
  - die Substition of shebang in
 '$ECONF_SOURCE/configure' failed
  - touch -r
 $ECONF_SOURCE/configure._portage_tmp_.$$ $ECONF_SOURCE/configure || die
  - rm -f $ECONF_SOURCE/configure._portage_tmp_.$$
  + touch -r ${ECONF_SOURCE}/configure
 ${ECONF_SOURCE}/configure._portage_tmp_.${pid} || die
  + sed -i \
  + -e
 1s:^#![[:space:]]*/bin/sh:#!$CONFIG_SHELL: \
  + ${ECONF_SOURCE}/configure \
  + || die Substition of shebang in
 '${ECONF_SOURCE}/configure' failed
  + touch -r
 ${ECONF_SOURCE}/configure._portage_tmp_.${pid}
 ${ECONF_SOURCE}/configure || die
  + rm -f
 ${ECONF_SOURCE}/configure._portage_tmp_.${pid}
fi
if [ -e ${EPREFIX}/usr/share/gnuconfig/ ]; then
find ${WORKDIR} -type f '(' \
-name config.guess -o -name config.sub ')' -print0
 | \
while read -r -d $'\0' x ; do
__vecho  * econf: updating
 ${x/${WORKDIR}\/} with ${EPREFIX}/usr/share/gnuconfig/${x##*/}
  - cp -f
 ${EPREFIX}/usr/share/gnuconfig/${x##*/} ${x}
  + # Make sure we do this atomically incase
 we're run in parallel. #487478
  + cp -f
 ${EPREFIX}/usr/share/gnuconfig/${x##*/} ${x}.${pid}
  + mv -f ${x}.${pid} ${x}
done
fi
 


 Sorry, my bash skills are not enough to review this stuff. Others will
 have to reply :)
 --
 Brian Dolbec dol...@gentoo.org



Re: [gentoo-portage-dev] [PATCH] econf: update configure/config.{sub,guess} atomically to avoid races

2013-12-17 Thread Alec Warner
On Tue, Dec 17, 2013 at 5:41 PM, Greg Turner g...@malth.us wrote:

 On Tue, Dec 17, 2013 at 3:28 PM, Mike Frysinger vap...@gentoo.org wrote:
  +   sed -i \
  +   -e
 1s:^#![[:space:]]*/bin/sh:#!$CONFIG_SHELL: \
  +   ${ECONF_SOURCE}/configure \
  +   || die Substition of shebang in
 '${ECONF_SOURCE}/configure' failed

 Shouldn't we take the same copy, ${pid}-suffix, move approach, here
 that we've done with config.{sub,guess}?  Otherwise, what's to stop
 these sed's from trampling each other's work-in-progress (the fact
 that ebuilds don't crash left and right does suggest that some
 mechanism actually does prevent this, already -- but it's a mystery to
 me, and I worry it could be some kind of linux-specific filesystem
 quirk).


Sed is already atomic

antarus@goats5 /tmp/test $ cat foo
Debian Rocks!
antarus@goats5 /tmp/test $ strace -e trace=file sed -i -e
's/Debian/Gentoo/g' foo
execve(/bin/sed, [sed, -i, -e, s/Debian/Gentoo/g, foo], [/* 39
vars */]) = 0
access(/etc/ld.so.nohwcap, F_OK)  = -1 ENOENT (No such file or
directory)
access(/etc/ld.so.preload, R_OK)  = -1 ENOENT (No such file or
directory)
open(/etc/ld.so.cache, O_RDONLY|O_CLOEXEC) = 3
access(/etc/ld.so.nohwcap, F_OK)  = -1 ENOENT (No such file or
directory)
open(/lib/x86_64-linux-gnu/libselinux.so.1, O_RDONLY|O_CLOEXEC) = 3
access(/etc/ld.so.nohwcap, F_OK)  = -1 ENOENT (No such file or
directory)
open(/lib/x86_64-linux-gnu/libc.so.6, O_RDONLY|O_CLOEXEC) = 3
access(/etc/ld.so.nohwcap, F_OK)  = -1 ENOENT (No such file or
directory)
open(/lib/x86_64-linux-gnu/libdl.so.2, O_RDONLY|O_CLOEXEC) = 3
statfs(/selinux, {f_type=EXT2_SUPER_MAGIC, f_bsize=4096,
f_blocks=5419717, f_bfree=920598, f_bavail=645285, f_files=1379040,
f_ffree=885151, f_fsid={-495840576, 2082046975}, f_namelen=255,
f_frsize=4096}) = 0
open(/proc/filesystems, O_RDONLY) = 3
open(/usr/lib/locale/locale-archive, O_RDONLY|O_CLOEXEC) = 3
open(//lib/charset.alias, O_RDONLY)   = -1 ENOENT (No such file or
directory)
open(/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache, O_RDONLY) = 3
open(foo, O_RDONLY)   = 3
open(/proc/filesystems, O_RDONLY) = 4
open(./sedfDxBxl, O_RDWR|O_CREAT|O_EXCL, 0600) = 4
rename(./sedfDxBxl, foo)= 0

-A


 -gmt




Re: [gentoo-portage-dev] [PATCH] econf: update configure/config.{sub,guess} atomically to avoid races

2013-12-17 Thread Alec Warner
On Tue, Dec 17, 2013 at 6:53 PM, Greg Turner g...@malth.us wrote:

 On Tue, Dec 17, 2013 at 5:58 PM, Alec Warner anta...@gentoo.org wrote:
  Sed is already atomic
 
  antarus@goats5 /tmp/test $ cat foo
  Debian Rocks!
  antarus@goats5 /tmp/test $ strace -e trace=file sed -i -e
  's/Debian/Gentoo/g' foo
 [snip]
  open(foo, O_RDONLY)   = 3
  open(/proc/filesystems, O_RDONLY) = 4
  open(./sedfDxBxl, O_RDWR|O_CREAT|O_EXCL, 0600) = 4
  rename(./sedfDxBxl, foo)= 0

 Nice demonstration!  Been wondering about that...


I originally opened the source code and tried to find the actual rename
code...but gave up after about 2 minutes figuring strace would be quicker.
I was right ;)

-A



 -gmt




Re: [gentoo-portage-dev] Try to specify how to get that a USE flag is present in current ebuild

2012-09-23 Thread Alec Warner
On Sat, Sep 22, 2012 at 7:22 PM, Pacho Ramos pa...@gentoo.org wrote:
 El sáb, 22-09-2012 a las 13:54 -0400, Mike Frysinger escribió:
 On Friday 21 September 2012 15:08:20 Pacho Ramos wrote:
  In that one, we try to use the following:
  has vala ${IUSE//+/}  ! use vala  return 0

 inherit eutils
 use_if_iuse vala
 -mike

 I am aware of that one also, but Ciaran also wants to forbid it for the
 same reason :S

Well I assume Ciaran wants to forbid it because he is attempting to
write a PMS compliant PM; but in order to use these ebuilds properly
he has to emulate the unspecified behavior that the ebuilds rely on
upon. His claim is that the council is supposed to forbid this
behavior (presumably to make his job less horrible) but I don't see
them beating down your door to change it (and the behavior is not
new.)

-A



Re: [gentoo-portage-dev] tools-portage packages

2012-07-18 Thread Alec Warner
On Tue, Jul 17, 2012 at 10:48 PM, Paul Varner fuzzy...@gentoo.org wrote:
 All:

 (Apologies if you get this twice.  I never saw my initial send of the
 message hit the mailing list)

 Sending this here before I send to the gentoo-dev list.  Below is the
 list of packages that are managed by the tools-portage herd and their
 maintainer (if any).  I know we have several packages in the herd that
 are not being maintained and I would like to either get a maintainer or
 last rite the package.  I've included comments on what I know about the
 packages.

 $ equery meta -m $(fgrep tools-portage /usr/portage/*/*/metadata.xml |
 awk -F'/' '{printf(%s/%s\n, $4, $5)}')

 * app-portage/deltup [gentoo]
 None specified

 -- I don't know anything about this package.

 * app-portage/eclass-manpages [gentoo]
 vap...@gentoo.org

 -- Actively maintained by Mike.

 * app-portage/elogv [gentoo]
 fuzzy...@gentoo.org (Paul Varner)
 sp...@gentoo.org (Sebastian Pipping)

 -- Being maintained more by Sebastian than myself.

 * app-portage/elogviewer [gentoo]
 fuzzy...@gentoo.org (Paul Varner)

 -- Effectively unmaintained, I can't find the upstream repository and I
 haven't done anything with it recently.

 * app-portage/epm [gentoo]
 None specified

 -- Old perl script that mimics rpm.  I've done some ebuild cleanup, but
 otherwise unmaintained.

 * app-portage/esearch [gentoo]
 None specified

I used to maintain this; but I don't see a compelling reason to use it
over eix, so I recommend removal.

-A


 -- Actively maintained by myself and Brian.  Upstream repository is on
 Github

 * app-portage/etc-proposals [gentoo]
 dol...@gentoo.org (Brian Dolbec)

 -- Actively maintained by Brian

 * app-portage/euses [gentoo]
 j...@gentoo.org (Jeroen Roovers)

 -- Actively maintained by Jeroen

 * app-portage/genlop [gentoo]
 None specified

 -- Not actively being maintained although fixes have been made in the
 upstream gentoo-perl repository on Github. it might be appropriate to
 transfer this to the gentoo-perl herd.

 * app-portage/gentoolkit-dev [gentoo]
 None specified

 -- Actively maintained by Christian.

 * app-portage/gentoolkit [gentoo]
 None specified

 -- Actively maintained by Paul and Brian

 * app-portage/gpytage [gentoo]
 dol...@gentoo.org (Brian Dolbec)

 -- Brian took over maintenance. Not sure of active status

 * app-portage/layman [gentoo]
 dol...@gentoo.org (Brian Dolbec)

 -- Brian is actively maintaining

 * app-portage/maintainer-helper [local]
 None specified

 -- Not actively maintained, it really needs a maintainer or be dropped
 from the tree.

 * app-portage/mirrorselect [gentoo]
 None specified

 -- Actively maintained by various people.

 * app-portage/porthole [gentoo]
 dol...@gentoo.org (Brian Dolbec)
 Upstream Maintainer (please CC on bugs)

 -- Actively maintained by Brian

 * app-portage/splat [gentoo]
 None specified

 -- I don't know anything about this package.  Upstream URL is dead.

 * app-portage/udept [local]
 fuzzy...@gentoo.org

 -- Effectively last rited, due to dead upstream.  Still in tree in hard
 masked status due to no replacement.  However, in the time that it has
 been hard masked, there has not been anyone to step up and either fork
 or create a replacement.

 * app-portage/ufed [gentoo]
 None specified

 -- Not being maintained since Harald retired and needs a maintainer.
 Repository is on overlays.g.o

 * dev-util/bdelta [gentoo]
 sly...@gentoo.org (Sergei Trofimovich)
 Primary Maintainer

 -- I don't know anything about this package.  Based on the ChangeLog,
 Sergei is actively maintaining it.




Re: [gentoo-portage-dev] tools-portage packages

2012-07-18 Thread Alec Warner
On Wed, Jul 18, 2012 at 8:09 PM, Christian Ruppert id...@gentoo.org wrote:
 On 07/17/12 at 03:48PM -0500, Paul Varner wrote:
 All:

 (Apologies if you get this twice.  I never saw my initial send of the
 message hit the mailing list)

 Sending this here before I send to the gentoo-dev list.  Below is the
 list of packages that are managed by the tools-portage herd and their
 maintainer (if any).  I know we have several packages in the herd that
 are not being maintained and I would like to either get a maintainer or
 last rite the package.  I've included comments on what I know about the
 packages.

 $ equery meta -m $(fgrep tools-portage /usr/portage/*/*/metadata.xml |
 awk -F'/' '{printf(%s/%s\n, $4, $5)}')

 * app-portage/deltup [gentoo]
 None specified

 -- I don't know anything about this package.

 * app-portage/eclass-manpages [gentoo]
 vap...@gentoo.org

 -- Actively maintained by Mike.

 * app-portage/elogv [gentoo]
 fuzzy...@gentoo.org (Paul Varner)
 sp...@gentoo.org (Sebastian Pipping)

 -- Being maintained more by Sebastian than myself.

 * app-portage/elogviewer [gentoo]
 fuzzy...@gentoo.org (Paul Varner)

 -- Effectively unmaintained, I can't find the upstream repository and I
 haven't done anything with it recently.

 * app-portage/epm [gentoo]
 None specified

 -- Old perl script that mimics rpm.  I've done some ebuild cleanup, but
 otherwise unmaintained.

 * app-portage/esearch [gentoo]
 None specified

 -- Actively maintained by myself and Brian.  Upstream repository is on
 Github

 * app-portage/etc-proposals [gentoo]
 dol...@gentoo.org (Brian Dolbec)

 -- Actively maintained by Brian

 * app-portage/euses [gentoo]
 j...@gentoo.org (Jeroen Roovers)

 -- Actively maintained by Jeroen

 * app-portage/genlop [gentoo]
 None specified

 -- Not actively being maintained although fixes have been made in the
 upstream gentoo-perl repository on Github. it might be appropriate to
 transfer this to the gentoo-perl herd.

 * app-portage/gentoolkit-dev [gentoo]
 None specified

 -- Actively maintained by Christian.

 I'll keep maintaining it so far, mostly echangelog.


 * app-portage/gentoolkit [gentoo]
 None specified

 -- Actively maintained by Paul and Brian

 * app-portage/gpytage [gentoo]
 dol...@gentoo.org (Brian Dolbec)

 -- Brian took over maintenance. Not sure of active status

 * app-portage/layman [gentoo]
 dol...@gentoo.org (Brian Dolbec)

 -- Brian is actively maintaining

 * app-portage/maintainer-helper [local]
 None specified

 -- Not actively maintained, it really needs a maintainer or be dropped
 from the tree.

 * app-portage/mirrorselect [gentoo]
 None specified

 -- Actively maintained by various people.

 I guess I will still try to fix bugs (mostly major ones) when there are some.

I will fix bugs in the script if someone does the ebuild work or proxies for me.

-A



 * app-portage/porthole [gentoo]
 dol...@gentoo.org (Brian Dolbec)
 Upstream Maintainer (please CC on bugs)

 -- Actively maintained by Brian

 * app-portage/splat [gentoo]
 None specified

 -- I don't know anything about this package.  Upstream URL is dead.

 * app-portage/udept [local]
 fuzzy...@gentoo.org

 -- Effectively last rited, due to dead upstream.  Still in tree in hard
 masked status due to no replacement.  However, in the time that it has
 been hard masked, there has not been anyone to step up and either fork
 or create a replacement.

 * app-portage/ufed [gentoo]
 None specified

 -- Not being maintained since Harald retired and needs a maintainer.
 Repository is on overlays.g.o

 * dev-util/bdelta [gentoo]
 sly...@gentoo.org (Sergei Trofimovich)
 Primary Maintainer

 -- I don't know anything about this package.  Based on the ChangeLog,
 Sergei is actively maintaining it.


 --
 Regards,
 Christian Ruppert
 Gentoo Linux developer, Bugzilla administrator and Infrastructure member
 Fingerprint: EEB1 C341 7C84 B274 6C59 F243 5EAB 0C62 B427 ABC8



Re: [gentoo-portage-dev] [PATCH] portageq: add colormap helper

2012-03-10 Thread Alec Warner
On Sat, Mar 10, 2012 at 8:15 PM, Mike Frysinger vap...@gentoo.org wrote:
 Signed-off-by: Mike Frysinger vap...@gentoo.org
 ---
  bin/isolated-functions.sh              |    2 +-
  bin/portageq                           |    8 
  pym/portage/output.py                  |    6 ++
  pym/portage/package/ebuild/doebuild.py |    8 ++--
  4 files changed, 17 insertions(+), 7 deletions(-)

 diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
 index 9321ad5..98be41e 100644
 --- a/bin/isolated-functions.sh
 +++ b/bin/isolated-functions.sh
 @@ -431,8 +431,8 @@ set_colors() {
                BAD=$'\e[31;01m'
                HILITE=$'\e[36;01m'
                BRACKET=$'\e[34;01m'
 +               NORMAL=$'\e[0m'
        fi
 -       NORMAL=$'\e[0m'
  }

  RC_ENDCOL=yes
 diff --git a/bin/portageq b/bin/portageq
 index 5ecbb21..fcdb9d9 100755
 --- a/bin/portageq
 +++ b/bin/portageq
 @@ -44,6 +44,7 @@ del pym_path
  from portage import os
  from portage.eapi import eapi_has_repo_deps
  from portage.util import writemsg, writemsg_stdout
 +from portage.output import colormap
  portage.proxy.lazyimport.lazyimport(globals(),
        'subprocess',
        '_emerge.Package:Package',
 @@ -685,6 +686,13 @@ def distdir(argv):
        print(portage.settings[DISTDIR])


 +def colormap(argv):
 +       
 +       Display the color.map as environment variables.
 +       
 +       print(portage.output.colormap())
 +
 +
  def envvar(argv):
        variable+
        Returns a specific environment variable as exists prior to ebuild.sh.
 diff --git a/pym/portage/output.py b/pym/portage/output.py
 index 43d7503..98bec81 100644
 --- a/pym/portage/output.py
 +++ b/pym/portage/output.py
 @@ -325,6 +325,12 @@ def style_to_ansi_code(style):
                ret += codes.get(attr_name, attr_name)
        return ret

 +def colormap():
 +       mycolors = []
 +       for c in (GOOD, WARN, BAD, HILITE, BRACKET, NORMAL):
 +               mycolors.append(%s=$'%s' % (c, style_to_ansi_code(c)))
 +       return \n.join(mycolors)
 +
  def colorize(color_key, text):
        global havecolor
        if havecolor:
 diff --git a/pym/portage/package/ebuild/doebuild.py 
 b/pym/portage/package/ebuild/doebuild.py
 index c45aa03..4ff3eea 100644
 --- a/pym/portage/package/ebuild/doebuild.py
 +++ b/pym/portage/package/ebuild/doebuild.py
 @@ -50,7 +50,7 @@ from portage.exception import DigestException, 
 FileNotFound, \
        IncorrectParameter, InvalidDependString, PermissionDenied, \
        UnsupportedAPIException
  from portage.localization import _
 -from portage.output import style_to_ansi_code
 +from portage.output import colormap

I assume style_to_ansi_code is unused?

  from portage.package.ebuild.prepare_build_dirs import prepare_build_dirs
  from portage.util import apply_recursive_permissions, \
        apply_secpass_permissions, noiselimit, normalize_path, \
 @@ -300,11 +300,7 @@ def doebuild_environment(myebuild, mydo, myroot=None, 
 settings=None,
                mysettings[PORTAGE_CONFIGROOT], EBUILD_SH_ENV_DIR)

        # Allow color.map to control colors associated with einfo, ewarn, 
 etc...
 -       mycolors = []
 -       for c in (GOOD, WARN, BAD, HILITE, BRACKET):
 -               mycolors.append(%s=$'%s' % \
 -                       (c, style_to_ansi_code(c)))
 -       mysettings[PORTAGE_COLORMAP] = \n.join(mycolors)
 +       mysettings[PORTAGE_COLORMAP] = colormap()

        if COLUMNS not in mysettings:
                # Set COLUMNS, in order to prevent unnecessary stty calls
 --
 1.7.8.5





Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

2011-10-02 Thread Alec Warner
On Sun, Oct 2, 2011 at 1:39 PM, Zac Medico zmed...@gentoo.org wrote:
 On 10/02/2011 05:46 AM, Robin H. Johnson wrote:
 On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote:
 If we control these hashes via metadata/layout.conf, then we can toggle
 it atomically for all commiters. Otherwise, we'll have an annoying
 period of time where different committers are committing different sets
 of hashes, depending on their portage version.
 How do you suggest doing it via layout.conf? I've kept SHA256 in both
 sets for now, but if you could enforce new signatures including both
 WHIRLPOOL and SHA256, that would be great.

 How about if we put something like this in
 gentoo-x86/metadata/layout.conf now:

Reminds me, I was going to do an analysis on -commit mails to track
portage versions; I'll do that now.


   manifest2-sha1 = true
   manifest2-whirlpool = false

 Then we'll patch portage so that by default it will disable SHA1 and
 enable WHIRLPOOL, and the above settings will override the defaults.
 After the patched portage is marked stable in a month or so, we'll send
 an announcement to gentoo-announce, and remove the above settings from
 layout.conf.
 --
 Thanks,
 Zac





Re: [gentoo-portage-dev] [PATCH 1/4] Manifest2 hash: Whirlpool

2011-10-01 Thread Alec Warner
On Sat, Oct 1, 2011 at 11:08 AM, Mike Frysinger vap...@gentoo.org wrote:
 On Thursday, September 29, 2011 21:27:39 Robin H. Johnson wrote:
 Provide public-domain implementation of the Whirlpool hash algorithm to
 be used as new Manifest2 hash.

 Signed-off-by: Robin H. Johnson robb...@gentoo.org
 ---
  pym/portage/checksum.py       |    4 +
  pym/portage/util/whirlpool.py |  788
 + 2 files changed, 792
 insertions(+), 0 deletions(-)
  create mode 100644 pym/portage/util/whirlpool.py

 shouldn't we add pycryptoplus to the tree and depend on that rather than
 copying  pasting their code into portage ?
 -mike

It is ironic with all the gentoo devs complaining about bundled libs.
I think part of the problem is the maintenance issue (upgrades from
EAPI0, extra deps, etc..)






Re: [gentoo-portage-dev] portage docbook documentation - why not asciidoc ?

2010-11-27 Thread Alec Warner
On Sat, Nov 27, 2010 at 6:53 AM, Zac Medico zmed...@gentoo.org wrote:
 On 11/27/2010 01:25 AM, Sebastian Pipping wrote:
 On 11/26/10 21:18, Zac Medico wrote:
 The asciidoc format seems nice, but personally, I think I prefer docbook
 since the SGML/XML approach seems more flexible and extensible.

 Are you making use of that flexibility anywhere at the moment?  If not
 is such use planned for something specific?

 Honestly, I don't know.

 In case DocBook is keeping contributions down than cutting away certain
 flexibility to increase contributions could be a good trade-off, too.

 I'm not sure that docbook represents a significant barrier in this
 respect. It's hard to speculate. Maybe if we had a survey sampling the
 opinions of a broad spectrum of open-source developers, then we'd have
 more to go on.

In general I don't see docbook as a problem.  If I file a portage bug
and say something needs to be changed or documented then attaching
plain text additions is usually sufficient for inclusion.  The bigger
problem portage faces is that very few people actually know enough
about how it works and this leaves only a few people who can actually
write (or verify) that documentation is correct.


 As it is, I'm fairly comfortable with docbook. Now you want me to learn
 a new format, with possible drawbacks, in order to try and draw in some
 contributions that may never happen?

Here is where I would look for data.  Are there pending contributions?
 I have not seen anyone on this list specifically say I'd write some
documentation if it was not in docbook.  I have had issues (years
ago) getting vapier to write documentation; but I assumed that was
because he hated writing docs and he contributed documentation after a
while.

 --
 Thanks,
 Zac





Re: [gentoo-portage-dev] portage docbook documentation - why not asciidoc ?

2010-11-27 Thread Alec Warner
On Sat, Nov 27, 2010 at 1:50 PM, Mike Frysinger vap...@gentoo.org wrote:
 On Saturday, November 27, 2010 16:39:04 Alec Warner wrote:
 On Sat, Nov 27, 2010 at 6:53 AM, Zac Medico wrote:
  As it is, I'm fairly comfortable with docbook. Now you want me to learn
  a new format, with possible drawbacks, in order to try and draw in some
  contributions that may never happen?

 Here is where I would look for data.  Are there pending contributions?
  I have not seen anyone on this list specifically say I'd write some
 documentation if it was not in docbook.  I have had issues (years
 ago) getting vapier to write documentation; but I assumed that was
 because he hated writing docs and he contributed documentation after a
 while.

 not sure where you're going with this, but i'm the one who started the doc/
 dir and introduced the docbook system.  obviously i have no problem with it
 and would rather see real examples of people who are being impeded by it
 rather than yet another omg we need to switch to this awesome new format
 because it is so awesome.
 -mike


I'm saying you should write more docs!  But we agree on the other point ;)



Re: [gentoo-portage-dev] A question regarding portage GFS2

2010-03-15 Thread Alec Warner
On Mon, Mar 15, 2010 at 2:44 AM, Mikko Rinne mikko.ri...@tnnet.fi wrote:
 Hey,

 It's my first time I'm mailing to this list so be warned.

 I have a problem with GFS2. In the past there's been sys-fs/gfs2 ebuild
 available in the portage, but for now it's been removed as one of it's
 dependencies (openais) was removed.

You may get more relevant advice from emailing gentoo-user, as this
list is primarily focussed on development of sys-apps/portage.


 Normally I wouldn't mind about it, but as I had the original sys-fs/gfs2
 already built in the system and after running emerge --sync  emerge
 --update, I found out that it actually managed to broke down mkfs.gfs2,
 which is pretty essential part when you have GFS2 set up. The whole error
 is; mkfs.gfs2: error while loading shared libraries: libvolume_id.so.0:
 cannot open shared object file: No such file or directory

 So the question remains; As GFS2 was removed as sys-fs/gfs2, has it been
 moved to any other main streamline package? If so, where. Basicly all I need
 atm is the mkfs.gfs2, but any info about this is much appreciated. Thanks.

You are free to go to http://sources.gentoo.org, click on 'gentoo-x86'
and find the ebuild for sys-fs/gfs2 and use it (and its associated
files) to build a new mkfs.gfs2.
You should probably put this ebuild in an overlay

-A




 Best Regards,

 --
 Mikko Rinne
 Technical department
 +358-10-3091334

 TNNet Oy
 http://www.tnnet.fi
 http://www.avatar.fi





[gentoo-portage-dev] Fwd: Random patch

2009-10-19 Thread Alec Warner
Ugh I hate this list name ;p


-- Forwarded message --
From: Alec Warner anta...@gentoo.org
Date: Mon, Oct 19, 2009 at 12:38 AM
Subject: Random patch
To: gentoo-dev-port...@gentoo.org


Get rid of if True...True is always true in CPython.
replace icky:
 'foo' + \
 'bar'

constructs with implicit string concat:

('foo'
 'bar)


repoman-string-concat-plus-if-true-fixes.patch
Description: Binary data


Re: [gentoo-portage-dev] portageq doesn't clean-up generated tmp/ and var/

2009-07-07 Thread Alec Warner
On Tue, Jul 7, 2009 at 1:40 AM, Amit Dor-Shiferami...@oversi.com wrote:
 amit0 ~ # ls -la /var/db/pkg/var
 ls: cannot access /var/db/pkg/var: No such file or directory
 amit0 ~ # ls -la /var/db/pkg/tmp
 ls: cannot access /var/db/pkg/tmp: No such file or directory
 amit0 ~ # portageq match /var/db/pkg portage

Just FYI, you are probably misinterpreted what root means here;
which is why you are confused later on.

 amit0 ~ # find /var/db/pkg/var /var/db/pkg/tmp
 /var/db/pkg/var
 /var/db/pkg/var/tmp
 /var/db/pkg/var/lib
 /var/db/pkg/var/lib/portage
 /var/db/pkg/var/cache
 /var/db/pkg/var/cache/edb
 /var/db/pkg/tmp

Of course it wouldn't; those directories are not at all designed to be
temporary.

/var/cache/edb is a metadata cache.  /var/lib/portage holds cached
config entries, --resume stuff, world files, and the preserved libs
registry.  Deleting those every time portageq runs would be crazy.

 amit0 ~ # emerge -Op portage
 !!! Invalid db entry: /var/db/pkg/var/tmp
 !!! Invalid db entry: /var/db/pkg/var/lib
 !!! Invalid db entry: /var/db/pkg/var/cache

 These are the packages that would be merged, in order:

 !!! Invalid db entry: /var/db/pkg/var/tmp
 !!! Invalid db entry: /var/db/pkg/var/lib
 !!! Invalid db entry: /var/db/pkg/var/cache
 [ebuild U ] sys-apps/portage-2.1.6.11 [2.1.6.7]
 !!! Invalid db entry: /var/db/pkg/var/tmp
 !!! Invalid db entry: /var/db/pkg/var/lib
 !!! Invalid db entry: /var/db/pkg/var/cache

   match root atom
  Returns a \n separated list of category/package-version.
  When given an empty string, all installed packages will
  be listed.

root is:
   ROOT = [path]
  Use  ROOT  to specify the target root filesystem to be
used for merging packages or ebuilds. This variable can be set via
  the --root option or in make.conf(5) (the command line
overrides other settings).
  Defaults to /.

You probably want portageq has_version / 'sys-apps/portage' ?

maybe Zac can make has_version take atoms?







Re: [gentoo-portage-dev] Re: Conflicting RDEPENDS

2009-06-07 Thread Alec Warner
On Wed, Jun 3, 2009 at 4:05 AM, Marijn Schouten (hkBst)hk...@gentoo.org wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Alec Warner wrote:
 On Sun, May 31, 2009 at 4:21 AM, Marijn Schouten (hkBst)
 hk...@gentoo.org wrote:
 Duncan wrote:
 Patrick Börjesson psychoti...@lavabit.com posted
 20090529201741.gb11...@nexon.nexus, excerpted below, on  Fri, 29 May 2009
 22:17:41 +0200:

 Why exactly would you want to use --oneshot for a leaf package that is
 not depended on by any other package in the world set? If spam IS
 depended on by any other package (recursively) in the world set, it will
 be pulled in by --complete-graph, but that's not the case here if i
 understand it correctly, thus it's a package that you explicitly wanted
 installed, thus it belongs in the world set, and you should thus not use
 --oneshot for it.
 I use -1 by default, here (via scriptlet), mainly so I don't have to
 worry about cluttering up my world file while emerging individual
 packages, just as I always use -NuD with my @system and @world runs.

 But for leaf packages, it serves as a sort of test install as well.
 Since I always do revdep-rebuild -p and emerge --depclean -p after every
 update (typically 2-3 times a week), then rebuild and clean as I need to,
 keeping the trial merges on the depclean list for a few days keeps me
 aware of them.  If I know it's something I want to keep, I run a
 different scriptlet without the -1, but that's not often once a system is
 up and running with the normal working set merged.  Meanwhile, I
 ultimately either emerge -C (or let depclean handle it) the trialware,
 or emerge --noreplace, thus adding it to world.

 But experimental installs and their deps typically sit in the --depclean
 list for anything from a few minutes to a few days, until I decide
 whether I want to keep or remove them.

 If he was testing how the switches under discussion here worked and has a
 similar policy, I could easily see him using -1 by habit, even if he
 didn't explicitly reason that it was a test and therefore something he
 didn't want in @world.
 I think this is an interesting use-case. It would be very simple to handle 
 it by
 introducing an additional file that the package manager would use to record 
 the
 packages that are installed on trial-basis. This would make it possible to
 include these packages in dep-calculations, while still distinguishing them 
 from
 packages that are in @world. Of course you can also fake it by creating a 
 local
 virtual/trialware package (or possibly a @trialware group) of which you edit 
 the
 deps, but this would be less convenient. For my personal workflow using -1 
 for
 trials is working well enough, atm.

 Why is a custom set less convenient?

 Well, instead of emerge --trialware package you would first have to edit 
 your
 @trialware set and then emerge @trialware. The same goes for when you want 
 to
 remove some trialware.
 Perhaps some generalization of --trialware along the lines of
 - --add-to-set=trialware could be fleshed out as a useful extension of 
 portage.

I like sandwiches too, so perhaps we can have a
--sudo_make_me_a_sandwich option to emerge?

But seriously, this is linux.  If users want do deal with a set of
packages that are like trialware then they should use the sets
functionality that emerge already ships with.  emerge
--add-to-set=blah might be passable but IMSHO emerge has plenty of
options already and users can easy write their own wrappers for this
kind of thing.  Emerge doesn't need every tiny feature built into it.


 Marijn

 - --
 If you cannot read my mind, then listen to what I say.

 Marijn Schouten (hkBst), Gentoo Lisp project, Gentoo ML
 http://www.gentoo.org/proj/en/lisp/, #gentoo-{lisp,ml} on FreeNode
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2.0.11 (GNU/Linux)
 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

 iEYEARECAAYFAkomWO8ACgkQp/VmCx0OL2wGHACfdlOdzvfLM3aUafiuOVQTlRnz
 vvMAoMMeLUxnM2i8fpJhClxbsIqwMf3Z
 =HSIG
 -END PGP SIGNATURE-





  1   2   >