[gentoo-portage-dev] [PATCH 00/10] First steps to get rid of backtracking
Hi all, as you may have noticed, emerge can in some cases take ages ( 5-10 minutes) to resolve dependencies these days. This happens when lots of backtracking is required to solve slot conflicts and/or to schedule slot operator rebuilds. The problem is that the current backtracking implementation has to rebuild the entire dependency graph from scratch each time it backtracks. This series of patches is a first step into fixing this problem. What these patches do: Patch 1 --- Adds a new data structure called package_tracker. This data structure is meant to replace several of the depgraph data structures. It has some new features not present in the existing data structures. 1) It can properly deal with several packages in the same slot. 2) It supports removal of previously added packages. 3) It tracks package conflicts automatically. 4) It has a more general concept of conflicts. Not all of the features are used for now, but they will make future changes easier. Patch 2-5 - These patches replace the old data structures with the package tracker Patch 6-8 - These patches fix several issues with emerge output. 6 and 8 are somewhat independent of the other patches in this series. Patch 7 introduces a new function used in Patch 10. Patch 9 --- New function for patch 10. Patch 10 This patch builds on top of all the previous patches. It introduces two new functions to the depgraph class. A function to remove packages that have previously been pulled in and a function to solve simple slot conflicts without backtracking. This should resolve most no parents that aren't satisfied by other packages in this slot slot conflicts. You may find these patches on github here: https://github.com/few/fews-portage-branch/tree/package_tracker Some numbers My system has quite a number of conflicts that give emerge a hard time resolving dependencies. -uDN world Without the patches: * 11 unsolved slot conflicts * 2 unsolved blockers * takes 5 backtracking steps and then fails With the patches: * no unsolved slot conflicts or blockers * takes 7 backtracking steps and then succeeds In this case it actually became slower, but was finally able to find a solution. -e world Without the patches: * 5 unsolved slot conflicts * 1 unsolved blocker * takes 23 backtracking steps and then fails With the patches: * 1 unsolved slot conflict (From a quick look it looks like there really is no solution.) * takes 13 backtracking steps and then fails In this case it's a lot faster, but still unacceptably slow. The result is improved by solving 4 out of 5 conflicts.
[gentoo-portage-dev] [PATCH 03/10] Replace mydbapi with _package_tracker
---
pym/_emerge/depgraph.py | 211 +++-
1 file changed, 101 insertions(+), 110 deletions(-)
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index fd59dda..9d234c2 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -344,7 +344,6 @@ class _dynamic_depgraph_config(object):
self._allow_backtracking = allow_backtracking
# Maps nodes to the reasons they were selected for
reinstallation.
self._reinstall_nodes = {}
- self.mydbapi = {}
# Contains a filtered view of preferred packages that are
selected
# from available repositories.
self._filtered_trees = {}
@@ -440,7 +439,6 @@ class _dynamic_depgraph_config(object):
# have after new packages have been installed.
fakedb = PackageTrackerDbapiWrapper(myroot,
self._package_tracker)
- self.mydbapi[myroot] = fakedb
def graph_tree():
pass
graph_tree.dbapi = fakedb
@@ -558,8 +556,6 @@ class depgraph(object):
if preload_installed_pkgs:
vardb = fake_vartree.dbapi
- fakedb = self._dynamic_config._graph_trees[
- myroot][vartree].dbapi
if not dynamic_deps:
for pkg in vardb:
@@ -724,25 +720,23 @@ class depgraph(object):
for pkg in list(self._dynamic_config.ignored_binaries):
- selected_pkg = self._dynamic_config.mydbapi[pkg.root
- ].match_pkgs(pkg.slot_atom)
+ selected_pkg = list()
- if not selected_pkg:
- continue
+ for selected_pkg in
self._dynamic_config._package_tracker.match(
+ pkg.root, pkg.slot_atom):
- selected_pkg = selected_pkg[-1]
- if selected_pkg pkg:
- self._dynamic_config.ignored_binaries.pop(pkg)
- continue
+ if selected_pkg pkg:
+
self._dynamic_config.ignored_binaries.pop(pkg)
+ break
- if selected_pkg.installed and \
- selected_pkg.cpv == pkg.cpv and \
- selected_pkg.build_time == pkg.build_time:
- # We don't care about ignored binaries when an
- # identical installed instance is selected to
- # fill the slot.
- self._dynamic_config.ignored_binaries.pop(pkg)
- continue
+ if selected_pkg.installed and \
+ selected_pkg.cpv == pkg.cpv and \
+ selected_pkg.build_time ==
pkg.build_time:
+ # We don't care about ignored binaries
when an
+ # identical installed instance is
selected to
+ # fill the slot.
+
self._dynamic_config.ignored_binaries.pop(pkg)
+ break
if not self._dynamic_config.ignored_binaries:
return
@@ -788,20 +782,25 @@ class depgraph(object):
# Exclude installed here since we only
# want to show available updates.
continue
- chosen_pkg = self._dynamic_config.mydbapi[pkg.root
- ].match_pkgs(pkg.slot_atom)
- if not chosen_pkg or chosen_pkg[-1] = pkg:
- continue
- k = (pkg.root, pkg.slot_atom)
- if k in missed_updates:
- other_pkg, mask_type, parent_atoms =
missed_updates[k]
- if other_pkg pkg:
- continue
- for mask_type, parent_atoms in mask_reasons.items():
- if not parent_atoms:
- continue
- missed_updates[k] = (pkg, mask_type,
parent_atoms)
- break
+ missed_update = True
+ any_selected = False
+ for chosen_pkg in
self._dynamic_config._package_tracker.match(
+
[gentoo-portage-dev] [PATCH 10/10] Solve some slot conflicts without backtracking
---
pym/_emerge/depgraph.py| 345 -
pym/portage/tests/resolver/test_backtracking.py| 13 +-
pym/portage/tests/resolver/test_blocker.py | 48 +++
pym/portage/tests/resolver/test_slot_collisions.py | 75 -
4 files changed, 457 insertions(+), 24 deletions(-)
create mode 100644 pym/portage/tests/resolver/test_blocker.py
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index 1bb086b..b5fecec 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -3,6 +3,7 @@
from __future__ import print_function, unicode_literals
+import collections
import errno
import io
import logging
@@ -78,7 +79,7 @@ from _emerge.resolver.backtracking import Backtracker,
BacktrackParameter
from _emerge.resolver.package_tracker import PackageTracker,
PackageTrackerDbapiWrapper
from _emerge.resolver.slot_collision import slot_conflict_handler
from _emerge.resolver.circular_dependency import circular_dependency_handler
-from _emerge.resolver.output import Display
+from _emerge.resolver.output import Display, format_unmatched_atom
if sys.hexversion = 0x300:
basestring = str
@@ -423,6 +424,8 @@ class _dynamic_depgraph_config(object):
self._complete_mode = False
self._slot_operator_deps = {}
self._package_tracker = PackageTracker()
+ # Track missed updates caused by solved conflicts.
+ self._conflict_missed_update = collections.defaultdict(dict)
for myroot in depgraph._frozen_config.trees:
self.sets[myroot] = _depgraph_sets()
@@ -769,7 +772,8 @@ class depgraph(object):
# missed update from each SLOT.
missed_updates = {}
for pkg, mask_reasons in \
- self._dynamic_config._runtime_pkg_mask.items():
+ chain(self._dynamic_config._runtime_pkg_mask.items(),
+
self._dynamic_config._conflict_missed_update.items()):
if pkg.installed:
# Exclude installed here since we only
# want to show available updates.
@@ -779,7 +783,8 @@ class depgraph(object):
for chosen_pkg in
self._dynamic_config._package_tracker.match(
pkg.root, pkg.slot_atom):
any_selected = True
- if chosen_pkg = pkg:
+ if chosen_pkg pkg or (not
chosen_pkg.installed and \
+ chosen_pkg.version == pkg.version):
missed_update = False
break
if any_selected and missed_update:
@@ -869,7 +874,7 @@ class depgraph(object):
self._show_merge_list()
msg = []
- msg.append(\nWARNING: One or more updates have been + \
+ msg.append(\nWARNING: One or more updates/rebuilds have been
+ \
skipped due to a dependency conflict:\n\n)
indent =
@@ -879,22 +884,29 @@ class depgraph(object):
msg.append( for %s % (pkg.root,))
msg.append(\n\n)
- for parent, atom in parent_atoms:
- msg.append(indent)
- msg.append(str(pkg))
+ msg.append(indent)
+ msg.append(str(pkg))
+ msg.append( conflicts with\n)
- msg.append( conflicts with\n)
- msg.append(2*indent)
+ for parent, atom in parent_atoms:
if isinstance(parent,
(PackageArg, AtomArg)):
# For PackageArg and AtomArg types, it's
# redundant to display the atom
attribute.
+ msg.append(2*indent)
msg.append(str(parent))
+ msg.append(\n)
else:
# Display the specific atom from SetArg
or
# Package types.
- msg.append(%s required by %s % (atom,
parent))
- msg.append(\n)
+ atom, marker = format_unmatched_atom(
+ pkg, atom,
self._pkg_use_enabled)
+
+ msg.append(2*indent)
+ msg.append(%s required by %s\n %
(atom, parent))
+
[gentoo-portage-dev] [PATCH 04/10] Replace _slot_collision_info with _package_tracker
--- pym/_emerge/depgraph.py| 59 -- pym/_emerge/resolver/slot_collision.py | 22 ++--- 2 files changed, 31 insertions(+), 50 deletions(-) diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py index 9d234c2..484ac14 100644 --- a/pym/_emerge/depgraph.py +++ b/pym/_emerge/depgraph.py @@ -378,11 +378,6 @@ class _dynamic_depgraph_config(object): # This use used to check if we have accounted for blockers # relevant to a package. self._traversed_pkg_deps = set() - # This should be ordered such that the backtracker will - # attempt to solve conflicts which occurred earlier first, - # since an earlier conflict can be the cause of a conflict - # which occurs later. - self._slot_collision_info = OrderedDict() # Slot collision nodes are not allowed to block other packages since # blocker validation is only able to account for one package per slot. self._slot_collision_nodes = set() @@ -915,7 +910,7 @@ class depgraph(object): cases. - if not self._dynamic_config._slot_collision_info: + if not any(self._dynamic_config._package_tracker.slot_conflicts()): return self._show_merge_list() @@ -971,16 +966,18 @@ class depgraph(object): is called, so that all relevant reverse dependencies are available for use in backtracking decisions. - for (slot_atom, root), slot_nodes in \ - self._dynamic_config._slot_collision_info.items(): - self._process_slot_conflict(root, slot_atom, slot_nodes) + for conflict in self._dynamic_config._package_tracker.slot_conflicts(): + self._process_slot_conflict(conflict) - def _process_slot_conflict(self, root, slot_atom, slot_nodes): + def _process_slot_conflict(self, conflict): Process slot conflict data to identify specific atoms which lead to conflict. These atoms only match a subset of the packages that have been pulled into a given slot. + root = conflict.root + slot_atom = conflict.atom + slot_nodes = conflict.pkgs debug = --debug in self._frozen_config.myopts @@ -1999,7 +1996,6 @@ class depgraph(object): existing_node, existing_node_matches = \ self._check_slot_conflict(pkg, dep.atom) - slot_collision = False if existing_node: if existing_node_matches: # The existing node can be reused. @@ -2032,19 +2028,7 @@ class depgraph(object): modified_use=self._pkg_use_enabled(existing_node))), level=logging.DEBUG, noiselevel=-1) - slot_collision = True - - if slot_collision: - # Now add this node to the graph so that self.display() - # can show use flags and --tree portage.output. This node is - # only being partially added to the graph. It must not be - # allowed to interfere with the other nodes that have been - # added. - # Even though the graph is now invalid, continue to process - # dependencies so that things like --fetchonly can still - # function despite collisions. - pass - elif not previously_added: + if not previously_added: self._dynamic_config._package_tracker.add_pkg(pkg) self._dynamic_config._filtered_trees[pkg.root][porttree].dbapi._clear_cache() self._dynamic_config._highest_pkg_cache.clear() @@ -2156,14 +2140,6 @@ class depgraph(object): def _add_slot_conflict(self, pkg): self._dynamic_config._slot_collision_nodes.add(pkg) - slot_key = (pkg.slot_atom, pkg.root) - slot_nodes = self._dynamic_config._slot_collision_info.get(slot_key) - if slot_nodes is None: - slot_nodes = set() - slot_nodes.update(self._dynamic_config._package_tracker.match( - pkg.root, pkg.slot_atom, installed=False)) -
[gentoo-portage-dev] [PATCH 08/10] Some small output fixes for the slot conflict handler
* unmatched atom printing now uses ^^ markers * unmatched atom printing properly supports sub-slots * Fix spurious no parents message caused by AtomArg parents --- pym/_emerge/resolver/slot_collision.py | 119 ++--- 1 file changed, 96 insertions(+), 23 deletions(-) diff --git a/pym/_emerge/resolver/slot_collision.py b/pym/_emerge/resolver/slot_collision.py index ca3fb74..c5936ad 100644 --- a/pym/_emerge/resolver/slot_collision.py +++ b/pym/_emerge/resolver/slot_collision.py @@ -1,4 +1,4 @@ -# Copyright 2010-2013 Gentoo Foundation +# Copyright 2010-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 from __future__ import print_function, unicode_literals @@ -273,12 +273,14 @@ class slot_conflict_handler(object): for ppkg, atom in parent_atoms: atom_set = InternalPackageSet(initial_atoms=(atom,)) atom_without_use_set = InternalPackageSet(initial_atoms=(atom.without_use,)) + atom_without_use_and_slot_set = InternalPackageSet(initial_atoms=( + atom.without_use.without_slot,)) for other_pkg in pkgs: if other_pkg == pkg: continue - if not atom_without_use_set.findAtomForPackage(other_pkg, \ + if not atom_without_use_and_slot_set.findAtomForPackage(other_pkg, \ modified_use=_pkg_use_enabled(other_pkg)): if atom.operator is not None: # The version range does not match. @@ -295,9 +297,11 @@ class slot_conflict_handler(object): atoms.add((ppkg, atom, other_pkg)) num_all_specific_atoms += 1 collision_reasons[key] = atoms - else: - # The sub_slot does not match. - key = (sub-slot, atom.sub_slot) + + elif not atom_without_use_set.findAtomForPackage(other_pkg, \ + modified_use=_pkg_use_enabled(other_pkg)): + # The slot and/or sub_slot does not match. + key = (slot, (atom.slot, atom.sub_slot, atom.slot_operator)) atoms = collision_reasons.get(key, set()) atoms.add((ppkg, atom, other_pkg)) num_all_specific_atoms += 1 @@ -342,6 +346,11 @@ class slot_conflict_handler(object): atoms.add((ppkg, atom, other_pkg)) collision_reasons[(use, flag)] = atoms num_all_specific_atoms += 1 + elif isinstance(ppkg, AtomArg) and other_pkg.installed: + parent_atoms = collision_reasons.get((AtomArg, None), set()) + parent_atoms.add((ppkg, atom)) + collision_reasons[(AtomArg, None)] = parent_atoms + num_all_specific_atoms += 1 msg.append( pulled in by\n) @@ -372,9 +381,11 @@ class slot_conflict_handler(object): if not verboseconflicts: selected_for_display.update( best_matches.values()) - elif type == sub-slot: +
[gentoo-portage-dev] [PATCH 09/10] Add digraph.discard
--- pym/portage/util/digraph.py | 12 +++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/pym/portage/util/digraph.py b/pym/portage/util/digraph.py index fc1fb86..4a9cb43 100644 --- a/pym/portage/util/digraph.py +++ b/pym/portage/util/digraph.py @@ -1,4 +1,4 @@ -# Copyright 2010-2013 Gentoo Foundation +# Copyright 2010-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 from __future__ import unicode_literals @@ -47,6 +47,16 @@ class digraph(object): priorities.append(priority) priorities.sort() + def discard(self, node): + + Like remove(), except it doesn't raises KeyError if the + node doesn't exist. + + try: + self.remove(node) + except KeyError: + pass + def remove(self, node): Removes the specified node from the digraph, also removing and ties to other nodes in the digraph. Raises KeyError if the -- 1.8.3.2
[gentoo-portage-dev] [PATCH 07/10] format_unmatched_atom: Pretty printing for unmatched atoms
This is a split out from the slot conflict handler to be used in other places. --- pym/_emerge/resolver/output.py | 109 +++-- 1 file changed, 106 insertions(+), 3 deletions(-) diff --git a/pym/_emerge/resolver/output.py b/pym/_emerge/resolver/output.py index 3e8552f..5f550be 100644 --- a/pym/_emerge/resolver/output.py +++ b/pym/_emerge/resolver/output.py @@ -1,4 +1,4 @@ -# Copyright 2010-2013 Gentoo Foundation +# Copyright 2010-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 Resolver output display operation. @@ -7,7 +7,7 @@ from __future__ import unicode_literals __all__ = ( - Display, + Display, format_unmatched_atom, ) import sys @@ -23,8 +23,9 @@ from portage.package.ebuild._spawn_nofetch import spawn_nofetch from portage.output import ( blue, colorize, create_color_func, darkblue, darkgreen, green, nc_len, teal) bad = create_color_func(BAD) +from portage._sets.base import InternalPackageSet from portage.util import writemsg_stdout -from portage.versions import best +from portage.versions import best, cpv_getversion from _emerge.Blocker import Blocker from _emerge.create_world_atom import create_world_atom @@ -916,3 +917,105 @@ class Display(object): self.print_changelog() return os.EX_OK + + +def format_unmatched_atom(pkg, atom, pkg_use_enabled): + + Returns two strings. The first string contains the + 'atom' with parts of the atom colored, which 'pkg' + doesn't match. The second string has the same number + of characters as the first one, but consists of only + white space or ^. The ^ characters have the same position + as the colored parts of the first string. + + # Things to check: + # 1. Version + # 2. cp + # 3. slot/sub_slot + # 4. repository + # 5. USE + + highlight = set() + + def perform_coloring(): + atom_str = + marker_str = + for ii, x in enumerate(atom): + if ii in highlight: + atom_str += colorize(BAD, x) + marker_str += ^ + else: + atom_str += x + marker_str += + return atom_str, marker_str + + if atom.cp != pkg.cp: + # Highlight the cp part only. + ii = atom.find(atom.cp) + highlight.update(range(ii, ii + len(atom.cp))) + return perform_coloring() + + version_atom = atom.without_repo.without_slot.without_use + version_atom_set = InternalPackageSet(initial_atoms=(version_atom,)) + highlight_version = not bool(version_atom_set.findAtomForPackage(pkg, + modified_use=pkg_use_enabled(pkg))) + + highlight_slot = False + if (atom.slot and atom.slot != pkg.slot) or \ + (atom.sub_slot and atom.sub_slot != pkg.sub_slot): + highlight_slot = True + + if highlight_version: + op = atom.operator + ver = None + if atom.cp != atom.cpv: + ver = cpv_getversion(atom.cpv) + + if op == =*: + op = = + ver += * + + if op is not None: + highlight.update(range(len(op))) + + if ver is not None: + start = atom.rfind(ver) + end = start + len(ver) + highlight.update(range(start, end)) + + if highlight_slot: + slot_str = : + atom.slot + if atom.sub_slot: + slot_str += / + atom.sub_slot + if atom.slot_operator: + slot_str += atom.slot_operator + start = atom.find(slot_str) + end = start + len(slot_str) + highlight.update(range(start, end)) + + highlight_use = set() + if atom.use: + use_atom = %s[%s] % (atom.cp, str(atom.use)) + use_atom_set = InternalPackageSet(initial_atoms=(use_atom,)) + if not use_atom_set.findAtomForPackage(pkg, \ + modified_use=pkg_use_enabled(pkg)): + missing_iuse = pkg.iuse.get_missing_iuse( + atom.unevaluated_atom.use.required) + if missing_iuse: + highlight_use = set(missing_iuse) + else: + #Use conditionals not met. + violated_atom = atom.violated_conditionals( + pkg_use_enabled(pkg), pkg.iuse.is_valid_flag) + if violated_atom.use is not None: +
[gentoo-portage-dev] [PATCH 05/10] Replace _slot_collision_nodes with _package_tracker
---
pym/_emerge/depgraph.py | 16 +++-
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index 484ac14..1bb086b 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -378,9 +378,6 @@ class _dynamic_depgraph_config(object):
# This use used to check if we have accounted for blockers
# relevant to a package.
self._traversed_pkg_deps = set()
- # Slot collision nodes are not allowed to block other packages
since
- # blocker validation is only able to account for one package
per slot.
- self._slot_collision_nodes = set()
self._parent_atoms = {}
self._slot_conflict_handler = None
self._circular_dependency_handler = None
@@ -1799,11 +1796,16 @@ class depgraph(object):
buildpkgonly = --buildpkgonly in self._frozen_config.myopts
nodeps = --nodeps in self._frozen_config.myopts
if dep.blocker:
+
+ # Slot collision nodes are not allowed to block other
packages since
+ # blocker validation is only able to account for one
package per slot.
+ is_slot_conflict_parent = any(dep.parent in
conflict.pkgs[1:] for conflict in \
+
self._dynamic_config._package_tracker.slot_conflicts())
if not buildpkgonly and \
not nodeps and \
not dep.collapsed_priority.ignored and \
not dep.collapsed_priority.optional and \
- dep.parent not in
self._dynamic_config._slot_collision_nodes:
+ not is_slot_conflict_parent:
if dep.parent.onlydeps:
# It's safe to ignore blockers if the
# parent is an --onlydeps node.
@@ -2019,7 +2021,6 @@ class depgraph(object):
level=logging.DEBUG, noiselevel=-1)
else:
- self._add_slot_conflict(pkg)
if debug:
writemsg_level(
%s%s %s\n % (Slot
Conflict:.ljust(15),
@@ -2138,9 +2139,6 @@ class depgraph(object):
self._dynamic_config._slot_operator_deps[slot_key] =
slot_info
slot_info.append(dep)
- def _add_slot_conflict(self, pkg):
- self._dynamic_config._slot_collision_nodes.add(pkg)
-
def _add_pkg_deps(self, pkg, allow_unsatisfied=False):
myroot = pkg.root
@@ -6019,7 +6017,7 @@ class depgraph(object):
if complete not in self._dynamic_config.myparams and \
self._dynamic_config._allow_backtracking and \
- self._dynamic_config._slot_collision_nodes and \
+
any(self._dynamic_config._package_tracker.slot_conflicts()) and \
not self._accept_blocker_conflicts():
self._dynamic_config.myparams[complete] = True
--
1.8.3.2
[gentoo-portage-dev] [PATCH 01/10] Add resolver/package_tracker
--- pym/_emerge/resolver/package_tracker.py | 310 1 file changed, 310 insertions(+) create mode 100644 pym/_emerge/resolver/package_tracker.py diff --git a/pym/_emerge/resolver/package_tracker.py b/pym/_emerge/resolver/package_tracker.py new file mode 100644 index 000..4aee4ea --- /dev/null +++ b/pym/_emerge/resolver/package_tracker.py @@ -0,0 +1,310 @@ +# Copyright 2014 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 + +from __future__ import print_function + +import collections + +import portage +portage.proxy.lazyimport.lazyimport(globals(), + 'portage:OrderedDict', + 'portage.dep:Atom,match_from_list', + 'portage.util:cmp_sort_key', + 'portage.versions:vercmp', +) + +_PackageConflict = collections.namedtuple(_PackageConflict, [root, pkgs, atom, description]) + +class PackageConflict(_PackageConflict): + + Class to track the reason for a conflict and the conflicting packages. + + def __iter__(self): + return iter(self.pkgs) + + def __contains__(self, pkg): + return pkg in self.pkgs + + def __len__(self): + return len(self.pkgs) + + +class PackageTracker(object): + + This class tracks packages which are currently + installed and packages which have been pulled into + the dependency graph. + + It automatically tracks conflicts between packages. + + Possible conflicts: + 1) Packages that share the same SLOT. + 2) Packages with the same cpv. + Not yet implemented: + 3) Packages that block each other. + + + def __init__(self): + # Mapping from package keys to set of packages. + self._cp_pkg_map = collections.defaultdict(list) + self._cp_vdb_pkg_map = collections.defaultdict(list) + # List of package keys that may contain conflicts. + # The insetation order must be preserved. + self._multi_pkgs = [] + + # Cache for result of conflicts(). + self._conflicts_cache = None + + # Records for each pulled package which installed package + # are replaced. + self._replacing = collections.defaultdict(list) + # Records which pulled packages replace this package. + self._replaced_by = collections.defaultdict(list) + + self._match_cache = collections.defaultdict(dict) + + def add_pkg(self, pkg): + + Add a new package to the tracker. Records conflicts as necessary. + + cp_key = pkg.root, pkg.cp + + try: + if any(other is pkg for other in self._cp_pkg_map[cp_key]): + return + except KeyError: + self._cp_pkg_map[cp_key] = [pkg] + else: + self._cp_pkg_map[cp_key].append(pkg) + if len(self._cp_pkg_map[cp_key]) == 2: + self._multi_pkgs.append(cp_key) + self._conflicts_cache = None + + self._replacing[pkg] = [] + for installed in self._cp_vdb_pkg_map[cp_key]: + if installed.slot_atom == pkg.slot_atom or \ + installed.cpv == pkg.cpv: + self._replacing[pkg].append(installed) + self._replaced_by[installed].append(pkg) + + self._match_cache.pop((pkg.root, pkg.cp)) + + def add_installed_pkg(self, installed): + + Add an installed package during vdb load. These packages + are not returned by matched_pull as long as add_pkg hasn't + been called with them. They are only returned by match_final. + + cp_key = installed.root, installed.cp + try: + if any(other is installed for other in self._cp_vdb_pkg_map[cp_key]): + return + except KeyError: + self._cp_vdb_pkg_map[cp_key] = [installed] + else: + self._cp_vdb_pkg_map[cp_key].append(installed) + + for pkg in self._cp_pkg_map[cp_key]: + if installed.slot_atom == pkg.slot_atom or \ + installed.cpv == pkg.cpv: + self._replacing[pkg].append(installed) + self._replaced_by[installed].append(pkg) + + def remove_pkg(self, pkg): + + Removes the package from the tracker. + Raises KeyError if it isn't present. + + cp_key = pkg.root, pkg.cp + try: +
Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000
Il 29/01/2014 17:33, Anthony G. Basile ha scritto: On 01/27/2014 09:02 AM, viv...@gmail.com wrote: On 01/26/14 23:53, Anthony G. Basile wrote: Hi everyone, A while back, I wrote a python wrapper for install to preserve xattrs. Its installed in LIBDIR/portage/bin/install.py. It is *painfully* slow. For a package like moodle with 16650 .php files, none of which probably need any xattr's set, it takes about 30 mins to install. I rewrote the wrapper in C. Replacing the python wrapper with the C wrapper, the same example reduces from about 30 mins to 2 mins. Mike and I did some back and forth about how best to write it. The latest version is pretty much done at https://bugs.gentoo.org/show_bug.cgi?id=465000#c56 We need to get that integrated into portage. 1) I'm not 100% sure how to do that. 2) We may want to install it at /usr/bin/install-xattr because I'm sure it will be useful for more than just portage. Comments? patch install from coreutils (and then upstream changes) is not an option? they already support selinux contexts anyway install-xattr could be useful and /usr/bin would be a good option IMHO Been there and I even had a patch ready. Upstream answer was '\0'. The only people who engaged the discussion were gentoo devs. Would patching coreutils have been the better approach? Some people might argue install and cp and mv etc should just copy contents to keep these utilities as simple as possible. Although, as you say, install can copy selinux contexts, and cp can copy xattr attributes. So what's the problem with extending installs functionality to include arbitrary xattr attributes? Anyhow, seeing as upstream is uninterested, I prefer this wrapper to maintaining a local patch against coreutils. thanks for answering, so yes the wrapper seem the only sensible thing, good luck, since there are no other questions in 3 days the plan must be perfect ;) on a totally unrelated and ipotetical note kde is discussing a metadata engine that use xattr instead of databases, so this could become important even for desktops
