Re: [gentoo-dev] Dealing with XDG directories in ebuild environment
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 What's the point of having nonempty XDG_ variables in ebuilds? Use of these variables is scoped to user applications that use these in runtime, therefore I see no business for them in package (de)installation and it should be ok for portage to unset the variables and if application is using these during build/install (in system context) it's a problem with the application. - -- Jan Matějka| Gentoo Developer https://gentoo.org | Gentoo Linux GPG: A33E F5BC A9F6 DAFD 2021 6FB6 3EBF D45B EEB6 CA8B -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBCgAGBQJS6MKtAAoJEIN+7RD5ejah6GsH+wcuWYyl3Ki3J/U0KT8QTrOx s5mixaNjrZ6sveU/3dUtCvKz/l9ZyR4YmKcQ8/Syv/UmoGQRdwYs+AgFsmfPPx6Z N46KK0pg6KAgVpHjJtJ1ZAbKO0tu39dz7c+GimnUYTqjUdrNtSHu4AUiaEQKCBft DTzD68LGE7lthoXtz1Y5OPjD/U0PXpOowcLo98RSgQnGOggdjBxPNBE05WWVWxM6 9XIKFNdca9sFGBuJQwoXiPNTXxnzizTXnukUgzvuctMg5uPPq2hfKVyktCilslGA X+JHZ2jT5bzJgeHbJK+6zEoPfjgxrY4fVg2DmLfUmnt7gP8N3wU/6MKILt7CpdM= =aSXC -END PGP SIGNATURE-
Re: [gentoo-dev] Dealing with XDG directories in ebuild environment
On Wed, 2014-01-29 at 09:58 +0100, Jan Matejka wrote: What's the point of having nonempty XDG_ variables in ebuilds? One big reason is FEATURES=test. Test suites for freedesktop-compliant programs that actually run the program are likely to fail if XDG_* directories are resolved as something unwriteable. Also, many gnome-related packages have a build system that won't work if XDG_* variables are wrong. Python packages that use sphinx to generate API docs at build time will hit a sandbox violation if XDG_CONFIG_HOME is unwriteable, see https://bugs.gentoo.org/show_bug.cgi?id=499068 I would be astonished if some packages for kde, xfce, and other desktop environments are not affected as well.
Re: [gentoo-dev] Dealing with XDG directories in ebuild environment
On Wed, Jan 29, 2014 at 1:37 AM, Alexandre Rostovtsev
tetrom...@gentoo.org wrote:
[Replying again since my mailer messed up my original message.]
On Tue, 2014-01-28 at 12:03 -0500, Mike Gilbert wrote:
Option 3: Unset the variables
This should cause applications to default to locations under ${HOME}.
This could be done in global scope (unless I am overlooking something
in PMS), and so it would not require consumers to invoke anything
explicitly.
Only those applications that properly comply with standards :)
For instance, glib did not start respecting ${HOME} until version 2.36
if I remember right; before that, unset XDG_* variables would cause
g_get_user_cache_dir() etc. to fall back to something under /root/ no
matter where ${HOME} pointed. Which is the main reason why
gnome2_environment_reset() was created.
Ah, thank you for that insight.
Ok, so I'm thinking maybe it would be best to combine option 1 and
option 3. We would unset the variables in global scope, and then have
a function that ebuilds or other eclasses could call to basically do
exactly what gnome2_reset_environment does and reset them to some path
under $T.
I'll work on a prototype for that.
[gentoo-dev] [gentoo-project] New developer: Alexander Berntsen (bernalex)
Hello everyone, please join me in giving Alexander (bernalex) a very warm welcome. When asking for his reason to be become a member of the dev team, he said among other things I would be proud to be a Gentoo dev. I like it very much, that one can still be proud to be part of this team and community. Let's spread this spirit. He is joining us as staffer (for now, let's see what the future brings) to work on portage (the PM) development, where he already contributed code. He also has been actively helping with Gentoo Haskell, which will be another region of interest for him. And finally he also is in contact with the licensing team. One of the questions we asked him and where we got some impressive answer, was What other projects have you contributed to, if any? Some of the more mentionable ones follows. bweakfwu F-Droid GNU LibreJS i3 Libregamewiki LIMBS OFF ndla OpenRC quizbot Portage Project Sunrise PRISM break reactive-banana So it seems he loves to contribute to voluntary project, which we greatly appreciate. This also resembles in the lines he wrote about himself. I'm a free software hacker wizard who lives in Norway with my cohabitant who occasionally bakes me muffins. I have used Gentoo for many years, and am particularly interested in Portage. My main interests are hacking, free software, free culture, computer games, interactive fiction, role-playing games, fantasy literature, science fiction cinema, anime, music, and cosmology. Feel very welcome, Alexander, Justin P.s. And don't worry, Diego already kindly asked him to not port portage to haskell ;). signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Dealing with XDG directories in ebuild environment
Le mardi 28 janvier 2014 à 13:11 +, Steven J. Long a écrit : Alec Warner wrote: Sorry, I work on Portage. What I'm saying is that We are free to change the behavior of *portage* now; rather than waiting for a new EAPI. If an ebuild needs to define EAPI=eapi-next to 'correctly' use XDG_*, well that is someone else's can of worms. Agreed: portage can clear those vars from the env as mgorny stated on the bug, and an xdg.eclass (or w/e) can setup good defaults for packages which need them. Presumably it'd be inherited by gnome and kde eclasses, for example, so most people wouldn't even see it. Exactly. -- Gilles Dartiguelongue e...@gentoo.org Gentoo
[gentoo-dev] Catalyst news item
Hi. I would like to commit the following news item in the next few days. diff --git a/2014/2014-01-31-catalyst-head-changes/2014-01-31-catalyst-head-changes.en.txt b/2014/2014-01-31-catalyst-head-changes/2014-01-31-catalyst-head-changes.en.txt new file mode 100644 index 000..7c5736e --- /dev/null +++ b/2014/2014-01-31-catalyst-head-changes/2014-01-31-catalyst-head-changes.en.txt @@ -0,0 +1,22 @@ +Title: Catalyst head changes +Author: Jorge Manuel B. S. Vicetto jmbsvice...@gentoo.org +Content-Type: text/plain +Posted: 2014-01-31 +Revision: 1 +News-Item-Format: 1.0 +Display-If-Installed: dev-util/catalyst + +After many years of stalled development, the catalyst repository is +going to have major changes introduced to master in the next few days. +The work done in the rewrite branch[1] by Brian Dolbec, is finally +going to be merged into master through the pending branch[2]. +Anyone using catalyst to produce stages is advised to use the latest +release (currently 2.0.16). If you need to track the stable branch, +please use the catalyst 2.0. ebuild that tracks the 2.X branch. +Anyone wanting to help with catalyst development and testing is +encouraged to use the version and report issues to the catalyst +team, pending the understanding that master may be broken during the +next few months. + + [1] - http://git.overlays.gentoo.org/gitweb/?p=proj/catalyst.git;a=shortlog;h=refs/heads/rewrite-on-master + [2] - http://git.overlays.gentoo.org/gitweb/?p=proj/catalyst.git;a=shortlog;h=refs/heads/pending Regards, Jorge Manuel B. S. Vicetto Gentoo Developer
Re: [gentoo-dev] Catalyst news item
On Thu, Jan 30, 2014 at 02:14:53AM -0100, Jorge Manuel B. S. Vicetto wrote: +After many years of stalled development, the catalyst repository is +going to have major changes introduced to master in the next few days. “the next few days” sounds a little optimistic to me ;). “next few months”, which you use later, seems more accurate here too. +report issues to the catalyst team, This reads “catalyst@” to me, which is fine if that's what you indend. However, you may want to suggest gentoo-catalyst@ instead, if you want a wider net of possible responders. Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Catalyst news item
On Wed, 29 Jan 2014, W. Trevor King wrote: On Thu, Jan 30, 2014 at 02:14:53AM -0100, Jorge Manuel B. S. Vicetto wrote: +After many years of stalled development, the catalyst repository is +going to have major changes introduced to master in the next few days. ?the next few days? sounds a little optimistic to me ;). ?next few months?, which you use later, seems more accurate here too. In the next few days, master is going to be open again for work. It doesn't mean we're going to get immediate commits or that it will break the next day. In the next few months, no one should expect master to work / build all the time. +report issues to the catalyst team, This reads ?catalyst@? to me, which is fine if that's what you indend. However, you may want to suggest gentoo-catalyst@ instead, if you want a wider net of possible responders. I didn't use an email as I didn't mean to imply that the reports should be done by email. I expect we will prefer using bugzilla or poking us in #gentoo-releng, at least for quick status inquiries / updates. Cheers, Trevor Regards, Jorge.
Re: [gentoo-dev] Catalyst news item
On Wed, Jan 29, 2014 at 7:14 PM, Jorge Manuel B. S. Vicetto jmbsvice...@gentoo.org wrote: +Display-If-Installed: dev-util/catalyst Display-If-Installed: =dev-util/catalyst-
Re: [gentoo-dev] Catalyst news item
On Wed, 29 Jan 2014, Matt Turner wrote: On Wed, Jan 29, 2014 at 7:14 PM, Jorge Manuel B. S. Vicetto jmbsvice...@gentoo.org wrote: +Display-If-Installed: dev-util/catalyst Display-If-Installed: =dev-util/catalyst- Matt, my plan was to show this to anyone using catalyst. I believe this news item is relevant and interesting for anyone using catalyst, but if others disagree, I can restrict it to only those using catalyst-. Regards, Jorge Manuel B. S. Vicetto Gentoo Developer
[gentoo-dev] New eclass: xdg-basedir
Here's a simple eclass that pretty much just extracts the XDG stuff from
the gnome2_environment_reset function.
One difference: it creates 3 of the 4 directories under ${HOME} instead
of ${T}, just to mimic the default behavior in the XDG basedir spec a
bit more closely.
Please give a look.
# Copyright 1999-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
# @ECLASS: xdg-basedir
# @MAINTAINER:
# freedesktop-b...@gentoo.org
# @BLURB: Set up directories according to the XDG Base Directory Specification.
# @EXAMPLE:
# @CODE
# inherit xdg-basedir
#
# src_prepare() {
# xdg-basedir-setup
# }
# @CODE
# Unset these globally to avoid leaking values from the calling environment.
unset XDG_DATA_HOME XDG_CONFIG_HOME XDG_CACHE_HOME XDG_RUNTIME_DIR
# @FUNCTION: xdg-basedir-setup
# @DESCRIPTION:
# Creates sensible locations for the following environment variables:
# XDG_DATA_HOME
# XDG_CONFIG_HOME
# XDG_CACHE_HOME
# XDG_RUNTIME_DIR
xdg-basedir-setup() {
export XDG_DATA_HOME=${HOME}/.local/share
export XDG_CONFIG_HOME=${HOME}/.config
export XDG_CACHE_HOME=${HOME}/.cache
export XDG_RUNTIME_DIR=${T}/run
mkdir -p ${XDG_DATA_HOME} ${XDG_CONFIG_HOME} ${XDG_CACHE_HOME} ||
die
mkdir -p -m 0700 ${XDG_RUNTIME_DIR} || die
}
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-dev] Catalyst news item
On Wed, Jan 29, 2014 at 7:36 PM, Jorge Manuel B. S. Vicetto jmbsvice...@gentoo.org wrote: my plan was to show this to anyone using catalyst. I believe this news item is relevant and interesting for anyone using catalyst, but if others disagree, I can restrict it to only those using catalyst-. The new is that the master branch may be unstable, which is only relevant to people using the master branch. *shrug*
[gentoo-dev] January 2014 QA Policy Updates
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello all, The new QA team has completed its first meeting, and so I would like to announce policy changes agreed upon during the meeting which are relevant to the developer community. In the future, when a meeting results in changed or amended policy, we will notify the community via - -dev and -dev-announce, so there will probably be a summary email like this coming out about once a month. Changes to policy from this meeting: - -USE-controlled optional RDEPENDs policy clarified to say that those dependencies are not allowed, but QA will grant exceptions for certain circumstances (such as a package not working unless one of a set of optional deps is installed) - -The QA team policymaking workflow will look like the following: * User/dev/team member brings us an issue * Team investigates * Team discusses at the next meeting ** If the person is violating policy, we inform them of that and request that they stop violating policy ** If the existing policy is unclear, we update it ** If there is no existing policy, make one If we think a developer's actions are causing problems, we may ask them to stop/undo pending discussion by the QA team at the next meeting. - -Rules for the QA team editing peoples' packages: *For trivial fixes, such as repoman errors, we fix the issue and send the developer a friendly reminder *For large but non-critical fixes, we open a bug, wait 2 weeks, and if it is not fixed within that time frame we make the change. *For critical fixes, such as a problem that breaks the tree or a package, we fix the issue and send the developer a notice about our change - -The QA team will communicate changes to policy via emails to gentoo-dev and gentoo-dev-announce and by updating the QA policy page on the Gentoo Wiki. For anyone interested, the summary of the meeting can be found at https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Meeting_Summaries and the current set of QA policies can be found at https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Policies. If you have any questions or concerns about these policies, feel free to discuss them with us in #gentoo-qa or by emailing q...@gentoo.org. Chris Reffett Gentoo QA Lead -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iKYEARECAGYFAlLp51VfFIAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3Bl bnBncC5maWZ0aGhvcnNlbWFuLm5ldEM2NzU5RjUyMDczREJDQkVDQTBDRkE1NERC Nzk1QThBNDI2MTgzNTQACgkQ23laikJhg1R6zwCfXY0q7Ig3d40Xq2hScLcT4Hm6 zE8AoJfIWsuV9yAKdsuxwB6JSDr8KbZY =sheY -END PGP SIGNATURE-
[gentoo-portage-dev] [PATCH 00/10] First steps to get rid of backtracking
Hi all, as you may have noticed, emerge can in some cases take ages ( 5-10 minutes) to resolve dependencies these days. This happens when lots of backtracking is required to solve slot conflicts and/or to schedule slot operator rebuilds. The problem is that the current backtracking implementation has to rebuild the entire dependency graph from scratch each time it backtracks. This series of patches is a first step into fixing this problem. What these patches do: Patch 1 --- Adds a new data structure called package_tracker. This data structure is meant to replace several of the depgraph data structures. It has some new features not present in the existing data structures. 1) It can properly deal with several packages in the same slot. 2) It supports removal of previously added packages. 3) It tracks package conflicts automatically. 4) It has a more general concept of conflicts. Not all of the features are used for now, but they will make future changes easier. Patch 2-5 - These patches replace the old data structures with the package tracker Patch 6-8 - These patches fix several issues with emerge output. 6 and 8 are somewhat independent of the other patches in this series. Patch 7 introduces a new function used in Patch 10. Patch 9 --- New function for patch 10. Patch 10 This patch builds on top of all the previous patches. It introduces two new functions to the depgraph class. A function to remove packages that have previously been pulled in and a function to solve simple slot conflicts without backtracking. This should resolve most no parents that aren't satisfied by other packages in this slot slot conflicts. You may find these patches on github here: https://github.com/few/fews-portage-branch/tree/package_tracker Some numbers My system has quite a number of conflicts that give emerge a hard time resolving dependencies. -uDN world Without the patches: * 11 unsolved slot conflicts * 2 unsolved blockers * takes 5 backtracking steps and then fails With the patches: * no unsolved slot conflicts or blockers * takes 7 backtracking steps and then succeeds In this case it actually became slower, but was finally able to find a solution. -e world Without the patches: * 5 unsolved slot conflicts * 1 unsolved blocker * takes 23 backtracking steps and then fails With the patches: * 1 unsolved slot conflict (From a quick look it looks like there really is no solution.) * takes 13 backtracking steps and then fails In this case it's a lot faster, but still unacceptably slow. The result is improved by solving 4 out of 5 conflicts.
[gentoo-portage-dev] [PATCH 03/10] Replace mydbapi with _package_tracker
---
pym/_emerge/depgraph.py | 211 +++-
1 file changed, 101 insertions(+), 110 deletions(-)
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index fd59dda..9d234c2 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -344,7 +344,6 @@ class _dynamic_depgraph_config(object):
self._allow_backtracking = allow_backtracking
# Maps nodes to the reasons they were selected for
reinstallation.
self._reinstall_nodes = {}
- self.mydbapi = {}
# Contains a filtered view of preferred packages that are
selected
# from available repositories.
self._filtered_trees = {}
@@ -440,7 +439,6 @@ class _dynamic_depgraph_config(object):
# have after new packages have been installed.
fakedb = PackageTrackerDbapiWrapper(myroot,
self._package_tracker)
- self.mydbapi[myroot] = fakedb
def graph_tree():
pass
graph_tree.dbapi = fakedb
@@ -558,8 +556,6 @@ class depgraph(object):
if preload_installed_pkgs:
vardb = fake_vartree.dbapi
- fakedb = self._dynamic_config._graph_trees[
- myroot][vartree].dbapi
if not dynamic_deps:
for pkg in vardb:
@@ -724,25 +720,23 @@ class depgraph(object):
for pkg in list(self._dynamic_config.ignored_binaries):
- selected_pkg = self._dynamic_config.mydbapi[pkg.root
- ].match_pkgs(pkg.slot_atom)
+ selected_pkg = list()
- if not selected_pkg:
- continue
+ for selected_pkg in
self._dynamic_config._package_tracker.match(
+ pkg.root, pkg.slot_atom):
- selected_pkg = selected_pkg[-1]
- if selected_pkg pkg:
- self._dynamic_config.ignored_binaries.pop(pkg)
- continue
+ if selected_pkg pkg:
+
self._dynamic_config.ignored_binaries.pop(pkg)
+ break
- if selected_pkg.installed and \
- selected_pkg.cpv == pkg.cpv and \
- selected_pkg.build_time == pkg.build_time:
- # We don't care about ignored binaries when an
- # identical installed instance is selected to
- # fill the slot.
- self._dynamic_config.ignored_binaries.pop(pkg)
- continue
+ if selected_pkg.installed and \
+ selected_pkg.cpv == pkg.cpv and \
+ selected_pkg.build_time ==
pkg.build_time:
+ # We don't care about ignored binaries
when an
+ # identical installed instance is
selected to
+ # fill the slot.
+
self._dynamic_config.ignored_binaries.pop(pkg)
+ break
if not self._dynamic_config.ignored_binaries:
return
@@ -788,20 +782,25 @@ class depgraph(object):
# Exclude installed here since we only
# want to show available updates.
continue
- chosen_pkg = self._dynamic_config.mydbapi[pkg.root
- ].match_pkgs(pkg.slot_atom)
- if not chosen_pkg or chosen_pkg[-1] = pkg:
- continue
- k = (pkg.root, pkg.slot_atom)
- if k in missed_updates:
- other_pkg, mask_type, parent_atoms =
missed_updates[k]
- if other_pkg pkg:
- continue
- for mask_type, parent_atoms in mask_reasons.items():
- if not parent_atoms:
- continue
- missed_updates[k] = (pkg, mask_type,
parent_atoms)
- break
+ missed_update = True
+ any_selected = False
+ for chosen_pkg in
self._dynamic_config._package_tracker.match(
+
[gentoo-portage-dev] [PATCH 10/10] Solve some slot conflicts without backtracking
---
pym/_emerge/depgraph.py| 345 -
pym/portage/tests/resolver/test_backtracking.py| 13 +-
pym/portage/tests/resolver/test_blocker.py | 48 +++
pym/portage/tests/resolver/test_slot_collisions.py | 75 -
4 files changed, 457 insertions(+), 24 deletions(-)
create mode 100644 pym/portage/tests/resolver/test_blocker.py
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index 1bb086b..b5fecec 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -3,6 +3,7 @@
from __future__ import print_function, unicode_literals
+import collections
import errno
import io
import logging
@@ -78,7 +79,7 @@ from _emerge.resolver.backtracking import Backtracker,
BacktrackParameter
from _emerge.resolver.package_tracker import PackageTracker,
PackageTrackerDbapiWrapper
from _emerge.resolver.slot_collision import slot_conflict_handler
from _emerge.resolver.circular_dependency import circular_dependency_handler
-from _emerge.resolver.output import Display
+from _emerge.resolver.output import Display, format_unmatched_atom
if sys.hexversion = 0x300:
basestring = str
@@ -423,6 +424,8 @@ class _dynamic_depgraph_config(object):
self._complete_mode = False
self._slot_operator_deps = {}
self._package_tracker = PackageTracker()
+ # Track missed updates caused by solved conflicts.
+ self._conflict_missed_update = collections.defaultdict(dict)
for myroot in depgraph._frozen_config.trees:
self.sets[myroot] = _depgraph_sets()
@@ -769,7 +772,8 @@ class depgraph(object):
# missed update from each SLOT.
missed_updates = {}
for pkg, mask_reasons in \
- self._dynamic_config._runtime_pkg_mask.items():
+ chain(self._dynamic_config._runtime_pkg_mask.items(),
+
self._dynamic_config._conflict_missed_update.items()):
if pkg.installed:
# Exclude installed here since we only
# want to show available updates.
@@ -779,7 +783,8 @@ class depgraph(object):
for chosen_pkg in
self._dynamic_config._package_tracker.match(
pkg.root, pkg.slot_atom):
any_selected = True
- if chosen_pkg = pkg:
+ if chosen_pkg pkg or (not
chosen_pkg.installed and \
+ chosen_pkg.version == pkg.version):
missed_update = False
break
if any_selected and missed_update:
@@ -869,7 +874,7 @@ class depgraph(object):
self._show_merge_list()
msg = []
- msg.append(\nWARNING: One or more updates have been + \
+ msg.append(\nWARNING: One or more updates/rebuilds have been
+ \
skipped due to a dependency conflict:\n\n)
indent =
@@ -879,22 +884,29 @@ class depgraph(object):
msg.append( for %s % (pkg.root,))
msg.append(\n\n)
- for parent, atom in parent_atoms:
- msg.append(indent)
- msg.append(str(pkg))
+ msg.append(indent)
+ msg.append(str(pkg))
+ msg.append( conflicts with\n)
- msg.append( conflicts with\n)
- msg.append(2*indent)
+ for parent, atom in parent_atoms:
if isinstance(parent,
(PackageArg, AtomArg)):
# For PackageArg and AtomArg types, it's
# redundant to display the atom
attribute.
+ msg.append(2*indent)
msg.append(str(parent))
+ msg.append(\n)
else:
# Display the specific atom from SetArg
or
# Package types.
- msg.append(%s required by %s % (atom,
parent))
- msg.append(\n)
+ atom, marker = format_unmatched_atom(
+ pkg, atom,
self._pkg_use_enabled)
+
+ msg.append(2*indent)
+ msg.append(%s required by %s\n %
(atom, parent))
+
[gentoo-portage-dev] [PATCH 04/10] Replace _slot_collision_info with _package_tracker
--- pym/_emerge/depgraph.py| 59 -- pym/_emerge/resolver/slot_collision.py | 22 ++--- 2 files changed, 31 insertions(+), 50 deletions(-) diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py index 9d234c2..484ac14 100644 --- a/pym/_emerge/depgraph.py +++ b/pym/_emerge/depgraph.py @@ -378,11 +378,6 @@ class _dynamic_depgraph_config(object): # This use used to check if we have accounted for blockers # relevant to a package. self._traversed_pkg_deps = set() - # This should be ordered such that the backtracker will - # attempt to solve conflicts which occurred earlier first, - # since an earlier conflict can be the cause of a conflict - # which occurs later. - self._slot_collision_info = OrderedDict() # Slot collision nodes are not allowed to block other packages since # blocker validation is only able to account for one package per slot. self._slot_collision_nodes = set() @@ -915,7 +910,7 @@ class depgraph(object): cases. - if not self._dynamic_config._slot_collision_info: + if not any(self._dynamic_config._package_tracker.slot_conflicts()): return self._show_merge_list() @@ -971,16 +966,18 @@ class depgraph(object): is called, so that all relevant reverse dependencies are available for use in backtracking decisions. - for (slot_atom, root), slot_nodes in \ - self._dynamic_config._slot_collision_info.items(): - self._process_slot_conflict(root, slot_atom, slot_nodes) + for conflict in self._dynamic_config._package_tracker.slot_conflicts(): + self._process_slot_conflict(conflict) - def _process_slot_conflict(self, root, slot_atom, slot_nodes): + def _process_slot_conflict(self, conflict): Process slot conflict data to identify specific atoms which lead to conflict. These atoms only match a subset of the packages that have been pulled into a given slot. + root = conflict.root + slot_atom = conflict.atom + slot_nodes = conflict.pkgs debug = --debug in self._frozen_config.myopts @@ -1999,7 +1996,6 @@ class depgraph(object): existing_node, existing_node_matches = \ self._check_slot_conflict(pkg, dep.atom) - slot_collision = False if existing_node: if existing_node_matches: # The existing node can be reused. @@ -2032,19 +2028,7 @@ class depgraph(object): modified_use=self._pkg_use_enabled(existing_node))), level=logging.DEBUG, noiselevel=-1) - slot_collision = True - - if slot_collision: - # Now add this node to the graph so that self.display() - # can show use flags and --tree portage.output. This node is - # only being partially added to the graph. It must not be - # allowed to interfere with the other nodes that have been - # added. - # Even though the graph is now invalid, continue to process - # dependencies so that things like --fetchonly can still - # function despite collisions. - pass - elif not previously_added: + if not previously_added: self._dynamic_config._package_tracker.add_pkg(pkg) self._dynamic_config._filtered_trees[pkg.root][porttree].dbapi._clear_cache() self._dynamic_config._highest_pkg_cache.clear() @@ -2156,14 +2140,6 @@ class depgraph(object): def _add_slot_conflict(self, pkg): self._dynamic_config._slot_collision_nodes.add(pkg) - slot_key = (pkg.slot_atom, pkg.root) - slot_nodes = self._dynamic_config._slot_collision_info.get(slot_key) - if slot_nodes is None: - slot_nodes = set() - slot_nodes.update(self._dynamic_config._package_tracker.match( - pkg.root, pkg.slot_atom, installed=False)) -
[gentoo-portage-dev] [PATCH 08/10] Some small output fixes for the slot conflict handler
* unmatched atom printing now uses ^^ markers * unmatched atom printing properly supports sub-slots * Fix spurious no parents message caused by AtomArg parents --- pym/_emerge/resolver/slot_collision.py | 119 ++--- 1 file changed, 96 insertions(+), 23 deletions(-) diff --git a/pym/_emerge/resolver/slot_collision.py b/pym/_emerge/resolver/slot_collision.py index ca3fb74..c5936ad 100644 --- a/pym/_emerge/resolver/slot_collision.py +++ b/pym/_emerge/resolver/slot_collision.py @@ -1,4 +1,4 @@ -# Copyright 2010-2013 Gentoo Foundation +# Copyright 2010-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 from __future__ import print_function, unicode_literals @@ -273,12 +273,14 @@ class slot_conflict_handler(object): for ppkg, atom in parent_atoms: atom_set = InternalPackageSet(initial_atoms=(atom,)) atom_without_use_set = InternalPackageSet(initial_atoms=(atom.without_use,)) + atom_without_use_and_slot_set = InternalPackageSet(initial_atoms=( + atom.without_use.without_slot,)) for other_pkg in pkgs: if other_pkg == pkg: continue - if not atom_without_use_set.findAtomForPackage(other_pkg, \ + if not atom_without_use_and_slot_set.findAtomForPackage(other_pkg, \ modified_use=_pkg_use_enabled(other_pkg)): if atom.operator is not None: # The version range does not match. @@ -295,9 +297,11 @@ class slot_conflict_handler(object): atoms.add((ppkg, atom, other_pkg)) num_all_specific_atoms += 1 collision_reasons[key] = atoms - else: - # The sub_slot does not match. - key = (sub-slot, atom.sub_slot) + + elif not atom_without_use_set.findAtomForPackage(other_pkg, \ + modified_use=_pkg_use_enabled(other_pkg)): + # The slot and/or sub_slot does not match. + key = (slot, (atom.slot, atom.sub_slot, atom.slot_operator)) atoms = collision_reasons.get(key, set()) atoms.add((ppkg, atom, other_pkg)) num_all_specific_atoms += 1 @@ -342,6 +346,11 @@ class slot_conflict_handler(object): atoms.add((ppkg, atom, other_pkg)) collision_reasons[(use, flag)] = atoms num_all_specific_atoms += 1 + elif isinstance(ppkg, AtomArg) and other_pkg.installed: + parent_atoms = collision_reasons.get((AtomArg, None), set()) + parent_atoms.add((ppkg, atom)) + collision_reasons[(AtomArg, None)] = parent_atoms + num_all_specific_atoms += 1 msg.append( pulled in by\n) @@ -372,9 +381,11 @@ class slot_conflict_handler(object): if not verboseconflicts: selected_for_display.update( best_matches.values()) - elif type == sub-slot: +
[gentoo-portage-dev] [PATCH 09/10] Add digraph.discard
--- pym/portage/util/digraph.py | 12 +++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/pym/portage/util/digraph.py b/pym/portage/util/digraph.py index fc1fb86..4a9cb43 100644 --- a/pym/portage/util/digraph.py +++ b/pym/portage/util/digraph.py @@ -1,4 +1,4 @@ -# Copyright 2010-2013 Gentoo Foundation +# Copyright 2010-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 from __future__ import unicode_literals @@ -47,6 +47,16 @@ class digraph(object): priorities.append(priority) priorities.sort() + def discard(self, node): + + Like remove(), except it doesn't raises KeyError if the + node doesn't exist. + + try: + self.remove(node) + except KeyError: + pass + def remove(self, node): Removes the specified node from the digraph, also removing and ties to other nodes in the digraph. Raises KeyError if the -- 1.8.3.2
[gentoo-portage-dev] [PATCH 07/10] format_unmatched_atom: Pretty printing for unmatched atoms
This is a split out from the slot conflict handler to be used in other places. --- pym/_emerge/resolver/output.py | 109 +++-- 1 file changed, 106 insertions(+), 3 deletions(-) diff --git a/pym/_emerge/resolver/output.py b/pym/_emerge/resolver/output.py index 3e8552f..5f550be 100644 --- a/pym/_emerge/resolver/output.py +++ b/pym/_emerge/resolver/output.py @@ -1,4 +1,4 @@ -# Copyright 2010-2013 Gentoo Foundation +# Copyright 2010-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 Resolver output display operation. @@ -7,7 +7,7 @@ from __future__ import unicode_literals __all__ = ( - Display, + Display, format_unmatched_atom, ) import sys @@ -23,8 +23,9 @@ from portage.package.ebuild._spawn_nofetch import spawn_nofetch from portage.output import ( blue, colorize, create_color_func, darkblue, darkgreen, green, nc_len, teal) bad = create_color_func(BAD) +from portage._sets.base import InternalPackageSet from portage.util import writemsg_stdout -from portage.versions import best +from portage.versions import best, cpv_getversion from _emerge.Blocker import Blocker from _emerge.create_world_atom import create_world_atom @@ -916,3 +917,105 @@ class Display(object): self.print_changelog() return os.EX_OK + + +def format_unmatched_atom(pkg, atom, pkg_use_enabled): + + Returns two strings. The first string contains the + 'atom' with parts of the atom colored, which 'pkg' + doesn't match. The second string has the same number + of characters as the first one, but consists of only + white space or ^. The ^ characters have the same position + as the colored parts of the first string. + + # Things to check: + # 1. Version + # 2. cp + # 3. slot/sub_slot + # 4. repository + # 5. USE + + highlight = set() + + def perform_coloring(): + atom_str = + marker_str = + for ii, x in enumerate(atom): + if ii in highlight: + atom_str += colorize(BAD, x) + marker_str += ^ + else: + atom_str += x + marker_str += + return atom_str, marker_str + + if atom.cp != pkg.cp: + # Highlight the cp part only. + ii = atom.find(atom.cp) + highlight.update(range(ii, ii + len(atom.cp))) + return perform_coloring() + + version_atom = atom.without_repo.without_slot.without_use + version_atom_set = InternalPackageSet(initial_atoms=(version_atom,)) + highlight_version = not bool(version_atom_set.findAtomForPackage(pkg, + modified_use=pkg_use_enabled(pkg))) + + highlight_slot = False + if (atom.slot and atom.slot != pkg.slot) or \ + (atom.sub_slot and atom.sub_slot != pkg.sub_slot): + highlight_slot = True + + if highlight_version: + op = atom.operator + ver = None + if atom.cp != atom.cpv: + ver = cpv_getversion(atom.cpv) + + if op == =*: + op = = + ver += * + + if op is not None: + highlight.update(range(len(op))) + + if ver is not None: + start = atom.rfind(ver) + end = start + len(ver) + highlight.update(range(start, end)) + + if highlight_slot: + slot_str = : + atom.slot + if atom.sub_slot: + slot_str += / + atom.sub_slot + if atom.slot_operator: + slot_str += atom.slot_operator + start = atom.find(slot_str) + end = start + len(slot_str) + highlight.update(range(start, end)) + + highlight_use = set() + if atom.use: + use_atom = %s[%s] % (atom.cp, str(atom.use)) + use_atom_set = InternalPackageSet(initial_atoms=(use_atom,)) + if not use_atom_set.findAtomForPackage(pkg, \ + modified_use=pkg_use_enabled(pkg)): + missing_iuse = pkg.iuse.get_missing_iuse( + atom.unevaluated_atom.use.required) + if missing_iuse: + highlight_use = set(missing_iuse) + else: + #Use conditionals not met. + violated_atom = atom.violated_conditionals( + pkg_use_enabled(pkg), pkg.iuse.is_valid_flag) + if violated_atom.use is not None: +
[gentoo-portage-dev] [PATCH 05/10] Replace _slot_collision_nodes with _package_tracker
---
pym/_emerge/depgraph.py | 16 +++-
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index 484ac14..1bb086b 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -378,9 +378,6 @@ class _dynamic_depgraph_config(object):
# This use used to check if we have accounted for blockers
# relevant to a package.
self._traversed_pkg_deps = set()
- # Slot collision nodes are not allowed to block other packages
since
- # blocker validation is only able to account for one package
per slot.
- self._slot_collision_nodes = set()
self._parent_atoms = {}
self._slot_conflict_handler = None
self._circular_dependency_handler = None
@@ -1799,11 +1796,16 @@ class depgraph(object):
buildpkgonly = --buildpkgonly in self._frozen_config.myopts
nodeps = --nodeps in self._frozen_config.myopts
if dep.blocker:
+
+ # Slot collision nodes are not allowed to block other
packages since
+ # blocker validation is only able to account for one
package per slot.
+ is_slot_conflict_parent = any(dep.parent in
conflict.pkgs[1:] for conflict in \
+
self._dynamic_config._package_tracker.slot_conflicts())
if not buildpkgonly and \
not nodeps and \
not dep.collapsed_priority.ignored and \
not dep.collapsed_priority.optional and \
- dep.parent not in
self._dynamic_config._slot_collision_nodes:
+ not is_slot_conflict_parent:
if dep.parent.onlydeps:
# It's safe to ignore blockers if the
# parent is an --onlydeps node.
@@ -2019,7 +2021,6 @@ class depgraph(object):
level=logging.DEBUG, noiselevel=-1)
else:
- self._add_slot_conflict(pkg)
if debug:
writemsg_level(
%s%s %s\n % (Slot
Conflict:.ljust(15),
@@ -2138,9 +2139,6 @@ class depgraph(object):
self._dynamic_config._slot_operator_deps[slot_key] =
slot_info
slot_info.append(dep)
- def _add_slot_conflict(self, pkg):
- self._dynamic_config._slot_collision_nodes.add(pkg)
-
def _add_pkg_deps(self, pkg, allow_unsatisfied=False):
myroot = pkg.root
@@ -6019,7 +6017,7 @@ class depgraph(object):
if complete not in self._dynamic_config.myparams and \
self._dynamic_config._allow_backtracking and \
- self._dynamic_config._slot_collision_nodes and \
+
any(self._dynamic_config._package_tracker.slot_conflicts()) and \
not self._accept_blocker_conflicts():
self._dynamic_config.myparams[complete] = True
--
1.8.3.2
[gentoo-portage-dev] [PATCH 01/10] Add resolver/package_tracker
--- pym/_emerge/resolver/package_tracker.py | 310 1 file changed, 310 insertions(+) create mode 100644 pym/_emerge/resolver/package_tracker.py diff --git a/pym/_emerge/resolver/package_tracker.py b/pym/_emerge/resolver/package_tracker.py new file mode 100644 index 000..4aee4ea --- /dev/null +++ b/pym/_emerge/resolver/package_tracker.py @@ -0,0 +1,310 @@ +# Copyright 2014 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 + +from __future__ import print_function + +import collections + +import portage +portage.proxy.lazyimport.lazyimport(globals(), + 'portage:OrderedDict', + 'portage.dep:Atom,match_from_list', + 'portage.util:cmp_sort_key', + 'portage.versions:vercmp', +) + +_PackageConflict = collections.namedtuple(_PackageConflict, [root, pkgs, atom, description]) + +class PackageConflict(_PackageConflict): + + Class to track the reason for a conflict and the conflicting packages. + + def __iter__(self): + return iter(self.pkgs) + + def __contains__(self, pkg): + return pkg in self.pkgs + + def __len__(self): + return len(self.pkgs) + + +class PackageTracker(object): + + This class tracks packages which are currently + installed and packages which have been pulled into + the dependency graph. + + It automatically tracks conflicts between packages. + + Possible conflicts: + 1) Packages that share the same SLOT. + 2) Packages with the same cpv. + Not yet implemented: + 3) Packages that block each other. + + + def __init__(self): + # Mapping from package keys to set of packages. + self._cp_pkg_map = collections.defaultdict(list) + self._cp_vdb_pkg_map = collections.defaultdict(list) + # List of package keys that may contain conflicts. + # The insetation order must be preserved. + self._multi_pkgs = [] + + # Cache for result of conflicts(). + self._conflicts_cache = None + + # Records for each pulled package which installed package + # are replaced. + self._replacing = collections.defaultdict(list) + # Records which pulled packages replace this package. + self._replaced_by = collections.defaultdict(list) + + self._match_cache = collections.defaultdict(dict) + + def add_pkg(self, pkg): + + Add a new package to the tracker. Records conflicts as necessary. + + cp_key = pkg.root, pkg.cp + + try: + if any(other is pkg for other in self._cp_pkg_map[cp_key]): + return + except KeyError: + self._cp_pkg_map[cp_key] = [pkg] + else: + self._cp_pkg_map[cp_key].append(pkg) + if len(self._cp_pkg_map[cp_key]) == 2: + self._multi_pkgs.append(cp_key) + self._conflicts_cache = None + + self._replacing[pkg] = [] + for installed in self._cp_vdb_pkg_map[cp_key]: + if installed.slot_atom == pkg.slot_atom or \ + installed.cpv == pkg.cpv: + self._replacing[pkg].append(installed) + self._replaced_by[installed].append(pkg) + + self._match_cache.pop((pkg.root, pkg.cp)) + + def add_installed_pkg(self, installed): + + Add an installed package during vdb load. These packages + are not returned by matched_pull as long as add_pkg hasn't + been called with them. They are only returned by match_final. + + cp_key = installed.root, installed.cp + try: + if any(other is installed for other in self._cp_vdb_pkg_map[cp_key]): + return + except KeyError: + self._cp_vdb_pkg_map[cp_key] = [installed] + else: + self._cp_vdb_pkg_map[cp_key].append(installed) + + for pkg in self._cp_pkg_map[cp_key]: + if installed.slot_atom == pkg.slot_atom or \ + installed.cpv == pkg.cpv: + self._replacing[pkg].append(installed) + self._replaced_by[installed].append(pkg) + + def remove_pkg(self, pkg): + + Removes the package from the tracker. + Raises KeyError if it isn't present. + + cp_key = pkg.root, pkg.cp + try: +
Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000
Il 29/01/2014 17:33, Anthony G. Basile ha scritto: On 01/27/2014 09:02 AM, viv...@gmail.com wrote: On 01/26/14 23:53, Anthony G. Basile wrote: Hi everyone, A while back, I wrote a python wrapper for install to preserve xattrs. Its installed in LIBDIR/portage/bin/install.py. It is *painfully* slow. For a package like moodle with 16650 .php files, none of which probably need any xattr's set, it takes about 30 mins to install. I rewrote the wrapper in C. Replacing the python wrapper with the C wrapper, the same example reduces from about 30 mins to 2 mins. Mike and I did some back and forth about how best to write it. The latest version is pretty much done at https://bugs.gentoo.org/show_bug.cgi?id=465000#c56 We need to get that integrated into portage. 1) I'm not 100% sure how to do that. 2) We may want to install it at /usr/bin/install-xattr because I'm sure it will be useful for more than just portage. Comments? patch install from coreutils (and then upstream changes) is not an option? they already support selinux contexts anyway install-xattr could be useful and /usr/bin would be a good option IMHO Been there and I even had a patch ready. Upstream answer was '\0'. The only people who engaged the discussion were gentoo devs. Would patching coreutils have been the better approach? Some people might argue install and cp and mv etc should just copy contents to keep these utilities as simple as possible. Although, as you say, install can copy selinux contexts, and cp can copy xattr attributes. So what's the problem with extending installs functionality to include arbitrary xattr attributes? Anyhow, seeing as upstream is uninterested, I prefer this wrapper to maintaining a local patch against coreutils. thanks for answering, so yes the wrapper seem the only sensible thing, good luck, since there are no other questions in 3 days the plan must be perfect ;) on a totally unrelated and ipotetical note kde is discussing a metadata engine that use xattr instead of databases, so this could become important even for desktops
