[gentoo-portage-dev] Re: [PATCH] emerge: add --autounmask-keep-keywords option (bug 622480)
Zac Medico posted on Sun, 13 Aug 2017 19:27:53 -0700 as excerpted: > On 08/13/2017 07:00 PM, M. J. Everitt wrote: >> Interesting .. I'm sure I shied away from that option for some reason >> ... wonder if zmedico can shed some light on the difference between the >> new options and the old, apart from some added flexibility ... > > The --autounmask-keep-keywords option allows you to adjust the the way > that decisions are made during dependency resolution. > Changes in decision making behavior have a large impact on the resulting > dependency calculation. It can mean the difference between a successful > calculation, and one that produces useless results. > > The --autounmask-write=n option has no influence on the decisions made, > The --autounmask-keep-keywords option gives finer-grained control. This > finer-grained control is only useful in cases where --autounmask=n would > prevent useful configuration changes from being made. Thanks. Clear and succinct. That improved my own understanding as well, particularly the distinction between changing the decisions made, and not changing them, but simply preventing them being automatically written, allowing the user direct control over what's actually written and how. =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Re: [gentoo-portage-dev] [PATCH] Rewrite doins in python (bug 624526)
On 08/14/2017 12:49 PM, M. J. Everitt wrote: > On 14/08/17 08:39, Zac Medico wrote: >> From: Hidehiko Abe >> >> doins is written in bash. However, specifically in case that >> too many files are installed, it is very slow. >> This CL rewrites the script in python for performance. >> >> BUG=chromium:712659 >> TEST=time (./setup_board --forace && \ >> ./build_package --withdev && \ >> ./build_image --noenable_rootfs_verification test) >> ===Before=== >> real21m35.445s >> user93m40.588s >> sys 21m31.224s >> >> ===After=== >> real17m30.106s >> user94m1.812s >> sys 20m13.468s >> > I know I'm gonna get chewed out on this one, but here goes anyway ... > > Surely for a package like chromium, who's build time is already in the > 'hours' range anyway, surely a couple of minutes gain for the install > phase is neither here nor there?! If there were some genuine filesystem > iop gains/etc for this change, I think I'd likely support it further .. It's going to reduce time, power consumption, and heat generation for all portage users. Also, we can use portage.util.file_copy to optimize it further with zero-copy, reflink, and sparse file support. > On this basis, what do the performance differences look like on an > 'average' package Well, it's very inefficient to fork/exec the install command for many files as the existing bash implementation does. The performance difference is related to the number of files. > .. and are there any regressions in this regard?! It's supposed to fallback to calling the install command if there are any unrecognized options, so the intention is for 100% compatibility. > I take issue with the copyright assignment, as I believe the legal .. > err .. IANAL devs are campaigning for full rights to be owned and > enforced by Gentoo Inc LLC in the US .. even if they have no idea what > that means or does .. :] Since it's a BSD-style license, we can copy the code into our project as long as we retain the copyright notice. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] Rewrite doins in python (bug 624526)
On 14/08/17 08:39, Zac Medico wrote: > From: Hidehiko Abe > > doins is written in bash. However, specifically in case that > too many files are installed, it is very slow. > This CL rewrites the script in python for performance. > > BUG=chromium:712659 > TEST=time (./setup_board --forace && \ > ./build_package --withdev && \ > ./build_image --noenable_rootfs_verification test) > ===Before=== > real21m35.445s > user93m40.588s > sys 21m31.224s > > ===After=== > real17m30.106s > user94m1.812s > sys 20m13.468s > I know I'm gonna get chewed out on this one, but here goes anyway ... Surely for a package like chromium, who's build time is already in the 'hours' range anyway, surely a couple of minutes gain for the install phase is neither here nor there?! If there were some genuine filesystem iop gains/etc for this change, I think I'd likely support it further .. On this basis, what do the performance differences look like on an 'average' package .. and are there any regressions in this regard?! I take issue with the copyright assignment, as I believe the legal .. err .. IANAL devs are campaigning for full rights to be owned and enforced by Gentoo Inc LLC in the US .. even if they have no idea what that means or does .. :] MJE signature.asc Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH] Rewrite doins in python (bug 624526)
From: Hidehiko Abe
doins is written in bash. However, specifically in case that
too many files are installed, it is very slow.
This CL rewrites the script in python for performance.
BUG=chromium:712659
TEST=time (./setup_board --forace && \
./build_package --withdev && \
./build_image --noenable_rootfs_verification test)
===Before===
real21m35.445s
user93m40.588s
sys 21m31.224s
===After===
real17m30.106s
user94m1.812s
sys 20m13.468s
Change-Id: Ib10f623961ba316753d58397cff5e72fbc343339
Reviewed-on: https://chromium-review.googlesource.com/559225
X-Chromium-Bug: 712659
X-Chromium-Bug-url: https://bugs.chromium.org/p/chromium/issues/detail?id=712659
X-Gentoo-Bug: 624526
X-Gentoo-Bug-url: https://bugs.gentoo.org/624526
---
bin/doins.py | 341 +++
bin/ebuild-helpers/doins | 123 ++---
2 files changed, 353 insertions(+), 111 deletions(-)
create mode 100644 bin/doins.py
diff --git a/bin/doins.py b/bin/doins.py
new file mode 100644
index 0..4a17287ca
--- /dev/null
+++ b/bin/doins.py
@@ -0,0 +1,341 @@
+#!/usr/bin/python -b
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+from __future__ import print_function
+
+import argparse
+import errno
+import grp
+import logging
+import os
+import pwd
+import re
+import shlex
+import shutil
+import stat
+import subprocess
+import sys
+
+from portage.util import movefile
+
+# Change back to original cwd _after_ all imports (bug #469338).
+# See also bin/ebuild-helpers/doins, which sets the cwd.
+os.chdir(os.environ["__PORTAGE_HELPER_CWD"])
+
+_PORTAGE_ACTUAL_DISTDIR = (
+ (os.environb if sys.version_info.major >= 3 else os.environ).get(
+ b'PORTAGE_ACTUAL_DISTDIR', b'') + b'/')
+
+
+def _parse_install_options(
+ options, inprocess_runner_class, subprocess_runner_class):
+ """Parses command line arguments for install command."""
+ parser = argparse.ArgumentParser()
+ parser.add_argument('-g', '--group', default=-1, type=_parse_group)
+ parser.add_argument('-o', '--owner', default=-1, type=_parse_user)
+ # "install"'s --mode option is complicated. So here is partially
+ # supported.
+ parser.add_argument('-m', '--mode', type=_parse_mode)
+ split_options = shlex.split(options)
+ namespace, remaining = parser.parse_known_args(split_options)
+ if remaining:
+ print('Unknown install options: %s, %r' % (options, remaining),
+ file=sys.stderr)
+ if os.environ.get('DOINSSTRICTOPTION', '') == '1':
+ sys.exit(1)
+ print('Continue with falling back to \'install\' '
+ 'command execution, which can be slower.',
+ file=sys.stderr)
+ return subprocess_runner_class(split_options)
+ return inprocess_runner_class(namespace)
+
+
+def _parse_group(group):
+ """Parses gid."""
+ g = grp.getgrnam(group)
+ if g:
+ return g.gr_gid
+ return int(group)
+
+
+def _parse_user(user):
+ """Parses uid."""
+ u = pwd.getpwnam(user)
+ if u:
+ return u.pw_uid
+ return int(user)
+
+
+def _parse_mode(mode):
+ # "install"'s --mode option is complicated. So here is partially
+ # supported.
+ # In Python 3, the prefix of octal int must be '0o' rather than '0'.
+ # So, set base explicitly in that case.
+ return int(mode, 8 if re.search(r'^0[0-7]*$', mode) else 0)
+
+
+def _set_attributes(options, path):
+ """Sets attributes the file/dir at given |path|.
+
+ Args:
+ options: object which has |owner|, |group| and |mode| fields.
+ |owner| is int value representing uid. Similary |group|
+ represents gid.
+ If -1 is set, just unchanged.
+ |mode| is the bits of permissions.
+ path: File/directory path.
+ """
+ if options.owner != -1 or options.group != -1:
+ os.lchown(path, options.owner, options.group)
+ if options.mode is not None:
+ os.chmod(path, options.mode)
+
+
+class _InsInProcessInstallRunner(object):
+ def __init__(self, parsed_options):
+ self._parsed_options = parsed_options
+ self._copy_xattr = (
+ 'xattr' in os.environ.get('FEATURES', '').split())
+ if self._copy_xattr:
+ self._xattr_exclude = os.environ.get(
+ 'PORTAGE_XATTR_EXCLUDE',
+ 'security.* system.nfs4_acl')
+
+ def run(self, source, dest_dir):
+ """Installs a file at |source| into |dest_dir| in process."""
+ dest = os.path.join(
