Re: [gentoo-portage-dev] Pending changes to Portage ebuild behavior
Am 28.07.2014 17:03, schrieb Michał Górny: Hello, everyone. New install --- 1. no /usr/lib/portage/pym (it's not really necessary with python_targets), We need to make that emerge -C prevents the removal of the last python version for which portage is installed then. That may already work. 2. all python modules bytecode in site-packages, 3. emerge and other tools load portage from site-packages [proper bytecode used], Other tools already do that. Emerge is a different thing. It's not a tool based on portage. emerge and portage are two large, interconnected things. (run git grep _emerge in pym/portage) 4. but python modules need to be able to locate /usr/lib/portage/bin somehow. What about hardcoding this path at install time? Looking at pkgcore might give you some hints.
Re: [gentoo-portage-dev] support multiple eclass variants (aka unstable eclasses)
Am 08.07.2014 18:58, schrieb Michael Haubenwallner: Hello fellow Portage developers, attached portage patch draft aims to allow for easy distributing eclasses to be tested by multiple tinderboxes on various architectures, without being active for normal installs. What does the patch allow you to do, that you can't do right now? (i.e. put an eclass with the same name in an repository and use eclass-overrides to force its use in another repo?) Usage is to have in make.conf (or profile): PORTAGE_ECLASS_VARIANTS=testing and for the eclass: to live in tree-or-overlay/eclass/testing/ Thoughts? (even for var-naming, manpage yes/no/wording, ...) There are lots of places to update (including python code). See git grep eclass. Thank you! /haubi/
Re: [gentoo-portage-dev] [PATCH] Support the 'packages' profile file as a directory.
Am 12.05.2014 12:14, schrieb Alexander Berntsen: Thanks! Committed as b4d8e300c04b768be7cd5c64116d6cc0453219b4. This change requires a new profile format (you'd want to call it portage-3 probably). The change then needs to depend on it. - Sebastian
Re: [gentoo-portage-dev] [PATCH] Support the 'packages' profile file as a directory.
Am 12.05.2014 21:43, schrieb Alexander Berntsen: The patch fixes a more immediate problem. Per the Portage man page, any file in /etc/portage/make.profile/ can be a directory. Without this patch, Portage crashes. Please cite the part of the man page that says this. I only see (in man portage): 'Any file in this directory [...] that begins with package. or use. can be more than just a flat file.' The packages file does not start with package..
Re: [gentoo-portage-dev] plugin-sync progress report
Am 15.03.2014 21:32, schrieb Brian Dolbec: I've started working on the repository/config.py changes needed for the plugin-sync system. First: The following class performed checks on the repos.conf entries for the sync variables regardless if they were being used or not. [...] Second: - Should all the repos.conf entires to be synced be validated prior to syncing and fail if there are any errors? - Or, just call each sync module's sync() and let each fail individually and continue syncing remaining repos? Maybe that's just me, but I don't like things that sometimes work and sometimes not. Having some emerge invocations fail with broken config and some not, seems rather ugly to me. (This also implies that syncing a repo with working config is not possible as long as there are repos with broken config.) Third: - I would like to add a new config item for all repos. It is a variable to define which sources of sync operations which the repo will be allowed to be synced from. It could be a space separated list which could be used to prevent it from being synced via certain commands. What exactly does this variable mean? What are emaint or layman doing here?
[gentoo-portage-dev] [PATCH] Introduce depgraph._extend_slot_operator_conflicts
This function allows emerge to solve more slot conflicts without
backtracking.
It does this by creating more conflicts for packages with built slot
operator dependencies. This gives more freedom to
depgraph._solve_non_slot_operator_slot_conflicts, which is then able
to solve conflicts it wouldn't have otherwise.
---
pym/_emerge/depgraph.py| 483 +++--
pym/_emerge/resolver/output.py | 5 +-
2 files changed, 415 insertions(+), 73 deletions(-)
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index 835bd6b..970a9c7 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -427,6 +427,12 @@ class _dynamic_depgraph_config(object):
# Track missed updates caused by solved conflicts.
self._conflict_missed_update = collections.defaultdict(dict)
+ # Rebuilds caused by slot conflicts.
+ self._slot_conflict_rebuilds = {}
+ # Rebuilds caused by missed slot operator updates or
+ # slot conflicts.
+ self._forced_rebuilds = None
+
for myroot in depgraph._frozen_config.trees:
self.sets[myroot] = _depgraph_sets()
vardb =
depgraph._frozen_config.trees[myroot][vartree].dbapi
@@ -614,6 +620,9 @@ class depgraph(object):
Fill self._forced_rebuilds with packages that cause rebuilds.
+ if self._dynamic_config._forced_rebuilds is not None:
+ return
+
debug = --debug in self._frozen_config.myopts
# Get all atoms that might have caused a forced rebuild.
@@ -687,19 +696,23 @@ class depgraph(object):
writemsg_level(\n\n,
level=logging.DEBUG, noiselevel=-1)
- self._forced_rebuilds = forced_rebuilds
+ for child, parents in
self._dynamic_config._slot_conflict_rebuilds.items():
+ forced_rebuilds.setdefault(child.root,
{}).setdefault(child, set()).update(parents)
+
+ self._dynamic_config._forced_rebuilds = forced_rebuilds
def _show_abi_rebuild_info(self):
- if not self._forced_rebuilds:
+ forced_rebuilds = self._dynamic_config._forced_rebuilds
+ if not forced_rebuilds:
return
writemsg_stdout(\nThe following packages are causing
rebuilds:\n\n, noiselevel=-1)
- for root in self._forced_rebuilds:
- for child in self._forced_rebuilds[root]:
+ for root in forced_rebuilds:
+ for child in forced_rebuilds[root]:
writemsg_stdout( %s causes rebuilds for:\n %
(child,), noiselevel=-1)
- for parent in
self._forced_rebuilds[root][child]:
+ for parent in forced_rebuilds[root][child]:
writemsg_stdout(%s\n % (parent,),
noiselevel=-1)
def _show_ignored_binaries(self):
@@ -968,16 +981,16 @@ class depgraph(object):
writemsg(line + '\n', noiselevel=-1)
writemsg('\n', noiselevel=-1)
- def _solve_non_slot_operator_slot_conflicts(self):
+
+ def _extend_slot_operator_conflicts(self):
- This function solves slot conflicts which can
- be solved by simply choosing one of the conflicting
- and removing all the other ones.
- It is able to solve somewhat more complex cases where
- conflicts can only be solved simultaniously.
+ This function searches for packages that cause
+ slot conflicts by dependening on conflict packages
+ with built slot operator deps. If such a package
+ is found an alternative package is pulled in with
+ the hope that the alternative package would not
+ cuase the slot conflict.
- debug = --debug in self._frozen_config.myopts
-
# List all conflicts. Ignore those that involve slot operator
rebuilds
# as the logic there needs special slot conflict behavior which
isn't
# provided by this function.
@@ -990,9 +1003,133 @@ class depgraph(object):
if not conflicts:
return
- # Get a set of all conflicting packages.
+ # Compute a mapping from parent packages to hard
+ # required conflict packages.
+ conflict_parents = collections.defaultdict(list)
+ for conflict in conflicts:
+ all_parent_atoms = set()
+ for pkg in conflict:
+ all_parent_atoms.update(
+
Re: [gentoo-portage-dev] [PATCH] Add a test case for --newuse, --changed-use, and --binpkg-respect-use.
Am 24.02.2014 20:03, schrieb David James: It's the same patch -- I just accidentally sent it twice. So consider this a bump, then. Sorry, I saw your first mail, but somehow forgot about it. Committed now.
Re: [gentoo-portage-dev] [PATCH 1/2] As per bug #472104 add option --verbose-slot-rebuilds=y|n
Am 23.02.2014 09:07, schrieb Brian Dolbec: --- pym/_emerge/actions.py | 5 + pym/_emerge/depgraph.py | 5 +++-- pym/_emerge/main.py | 5 + pym/portage/package/ebuild/config.py | 3 +++ 4 files changed, 16 insertions(+), 2 deletions(-) diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py index 95c5c14..a3f7813 100644 --- a/pym/_emerge/actions.py +++ b/pym/_emerge/actions.py @@ -2923,6 +2923,11 @@ def adjust_config(myopts, settings): settings[PORTAGE_VERBOSE] = 1 settings.backup_changes(PORTAGE_VERBOSE) + if (--verbose-slot-rebuilds in myopts and + myopts[--verbose-slot-rebuilds] not in (y,True)): + settings[VERBOSE_SLOT_REBUILDS] = 0 + settings.backup_changes(VERBOSE_SLOT_REBUILDS) Why do you put that into the config class? In the resolver you should just do: if --verbose-slot-rebuilds in self._frozen_config.myopts: Otherwise looks good.
Re: [gentoo-portage-dev] [As per bug #472104 add option --verbose-slot-rebuilds
Am 23.02.2014 21:02, schrieb Brian Dolbec: Hopefully there isn't anything else Seabastian finds in this one. Thank you Sebastian for your review and fixes :) Looks good now. Brian Dolbec (2): As per bug #472104 add option --verbose-slot-rebuilds=y|n Whitespace cleanup pym/_emerge/actions.py | 4 ++-- pym/_emerge/depgraph.py | 5 +++-- pym/_emerge/main.py | 11 --- 3 files changed, 13 insertions(+), 7 deletions(-)
Re: [gentoo-portage-dev] [PATCH] portdbapi: Add cache to avoid repeated failing os.access calls
Am 18.02.2014 22:35, schrieb David James: + cache_key = (mycpv, mytree, myrepo) + try: + return self._findname2_cache[cache_key] + except KeyError: + self._findname2_cache[cache_key] = (None, 0) To me, it seems potentially error-prone to cache a (potentially) incorrect value and then correct it later. It is. The problem are all these returns. The whole thing would need to be reorganized to fix this. I'd rather go for a memoize decorator and leave that thing alone. If I just could find a usable one. Would it be possible to refactor your patch so that we only cache the value when we know the final answer? + except TypeError: In what circumstances does it happen that mytree / myrepo are unhashable types? Can you add a comment to explain this? That's my mistake. I was under the impression that mytree is actually mytreeS and would accept a list. I'll remove the except TypeError: in a future version of the patch.
Re: [gentoo-portage-dev] [PATCH] Implement --newrepo flag.
Am 17.02.2014 22:44, schrieb David James: From a theoretical perspective, the reason why I implemented this is because ebuilds are the source of truth. According to the ebuilds, the owner repo of dev-libs/C-1 is repo1, so this means that, with --newrepo, any binpkg for dev-libs/C-1 from repo2 would be invalid. Actually this test passes even without --new-repo. So yes, your interpretation is what's used by portage. Patch is committed. This functionality is useful in the case where an ebuild changes repos. In this case, with --newrepo, you would want the package to be rebuilt and any binaries to be invalidated, and our tests validate this. Cheers, David
[gentoo-portage-dev] [PATCH] portdbapi: Add cache to avoid repeated failing os.access calls
---
pym/portage/dbapi/porttree.py | 13 +
1 file changed, 13 insertions(+)
diff --git a/pym/portage/dbapi/porttree.py b/pym/portage/dbapi/porttree.py
index 590e3c5..2dc406b 100644
--- a/pym/portage/dbapi/porttree.py
+++ b/pym/portage/dbapi/porttree.py
@@ -254,6 +254,7 @@ class portdbapi(dbapi):
self._aux_cache = {}
self._broken_ebuilds = set()
+ self._findname2_cache = {}
@property
def _event_loop(self):
@@ -372,6 +373,14 @@ class portdbapi(dbapi):
the file we wanted.
If myrepo is not None it will find packages from this
repository(overlay)
+ cache_key = (mycpv, mytree, myrepo)
+ try:
+ return self._findname2_cache[cache_key]
+ except KeyError:
+ self._findname2_cache[cache_key] = (None, 0)
+ except TypeError:
+ cache_key = None
+
if not mycpv:
return (None, 0)
@@ -383,6 +392,8 @@ class portdbapi(dbapi):
mysplit = mycpv.split(/)
psplit = pkgsplit(mysplit[1])
if psplit is None or len(mysplit) != 2:
+ if cache_key:
+ del self._findname2_cache[cache_key]
raise InvalidPackageName(mycpv)
# For optimal performace in this hot spot, we do manual unicode
@@ -402,6 +413,8 @@ class portdbapi(dbapi):
filename = x + _os.sep + relative_path
if _os.access(_unicode_encode(filename,
encoding=encoding, errors=errors), _os.R_OK):
+ if cache_key:
+ self._findname2_cache[cache_key] =
(filename, x)
return (filename, x)
return (None, 0)
--
1.8.3.2
[gentoo-portage-dev] Avoid repeated failing os.access
Hi all, people have several times complained about useless access calls in strace logs. The problem is that a successful call in portdbapi.findname2 is cached by aux_get, but a failed is not. This patch works around this by adding a cache for findname2. This gives a speed up of about 5% compared to current master. The real problem is that the resolver calls aux_get for a given cpv for each existing repo. That's because portdbapi.cp_list returns only cpvs without telling the caller to which repo it belongs. Fixing that properly would require quite some work on the portdbapi class. There was the complaint that instead of writing the caching myself I should be using a memoization decorator. Unfortunately nobody could suggest a good implementation with proper license to use. If someone knows one, let me know. - Sebastian
Re: [gentoo-portage-dev] [PATCH] Implement --newrepo flag.
Am 17.02.2014 17:39, schrieb Brian Dolbec: Mike, if you can confirm the tests pass. Go ahead and merge it. Please give me some more time to review it. I'll commit it if it lgtm.
Re: [gentoo-portage-dev] [PATCH] Implement --newrepo flag.
Am 17.02.2014 18:52, schrieb David James:
@@ -96,6 +105,68 @@ class MultirepoTestCase(TestCase):
snip
+
+ #--newrepo: pick ebuild if binpkg/ebuild have different
repo
+ ResolverPlaygroundTestCase(
+ [dev-libs/C],
+ options = {--usepkg: True, --newrepo: True,
--selective: True},
+ success = True,
+ check_repo_names = True,
+ mergelist = [dev-libs/C-1::repo1]),
Why should it take the ebuild from the repo with lower priority?
I thought the intend was that the package should be reinstalled if it is
now pulled from a repo with higher priority than the repo from which the
installed package came.
Re: [gentoo-portage-dev] [PATCH] Remove bogus countdown during clean list printing (bug 501296)
Am 17.02.2014 08:56, schrieb Mike Frysinger: On Friday, February 14, 2014 22:22:03 Sebastian Luther wrote: Before this patch a fake unmerge countdown would be shown after an @system package was printed. If the users aborts at this point, it looked as if some packages were missing from the clean list compared to the --pretend output. Acked-by: Mike Frysinger vap...@gentoo.org -mike committed
Re: [gentoo-portage-dev] [PATCH] Always warn about unknown mirrors (bug 501352)
Am 17.02.2014 17:21, schrieb Brian Dolbec: On Sat, 15 Feb 2014 13:40:23 +0100 Sebastian Luther sebastianlut...@gmx.de wrote: --- pym/portage/package/ebuild/fetch.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) Yeah, looks fine. Merge it :) Sorry for the delay reviewing. committed
Re: [gentoo-portage-dev] [RFC] Inherit updates settings from master repositories by default
Am 17.02.2014 09:06, schrieb Mike Frysinger: On Wednesday, February 05, 2014 19:11:12 Sebastian Luther wrote: Am 05.02.2014 09:03, schrieb Mike Frysinger: On Saturday, February 01, 2014 20:38:05 Arfrever Frehtes Taifersar Arahesis wrote: this i'm not so sure about. when you have a local overlay, portage complains when there are no masters which means most people have just blindly added masters = gentoo. but if they have packages in there using the old name (on purpose), then updates will constantly tromp on that. The old behavior was to always apply the updates from ::gentoo as long as the repo didn't have its own updates. This means it doesn't matter if the repo sets the masters = gentoo as long as it doesn't contain updates. ok, i thought it always ignored the gentoo updates dir At least that's what Arfrever wrote in the first mail as 'old behavior'. I assume that's correct. at least, there should be one of: - one-time automatic migration of existing layout.conf files where we set updates-master = for them. How do you know if it's the user's repo or a layman repo, where layout.conf is manged by other people? you ask layman. this isn't difficult. layman was just an example. The user could as well have cloned a repo by hand. When you said one-time automatic migration I thought of an update during portage installation (like it's done for other things). Do you have something else in mind? - a warning phase where we complain if the field isn't set, and we default to current behavior. once some time has elapsed, we stop warning and we change the default. Be sure to only hit users which are really affected by the change (i.e. repos with existing updates and master repos which contain updates, which affect packages in the repo). doing it based on the current set of affected packages doens't make sense then the set of possible updates is constantly changing It's true that it may change, but this should only happen very seldom and only during the transition period. The idea is that if you don't restrict the warning like that, you're going to create a lot of noise. -mike
Re: [gentoo-portage-dev] [PATCH] Implement --newrepo flag.
Am 14.02.2014 21:43, schrieb David James: Uploaded here: https://chromium-review.googlesource.com/#/c/186651/ You should be able to cherry-pick using this command: git fetch https://chromium.googlesource.com/chromiumos/third_party/portage_tool refs/changes/51/186651/1 git cherry-pick FETCH_HEAD The patch has conflicts in depgraph.py. I haven't added automated tests yet. Cheers, David
Re: [gentoo-portage-dev] [RFC] Inherit updates settings from master repositories by default
Am 01.02.2014 20:38, schrieb Arfrever Frehtes Taifersar Arahesis:
Handling of inheritance of updates settings
(${repository_location}/profiles/updates/*) is broken in Portage
and triggers DeprecationWarnings.
Current behavior: - When profiles/updates directory exists in
repository X, then only settings from this directory are used for
packages installed from repository X. - When profiles/updates
directory does not exist in repository X, then only settings from
profiles/updates directory of repository set in (scheduled for
deprecation) 'main-repo' attribute of 'DEFAULT' section in
repos.conf are used for packages installed from repository X,
regardless of value of 'masters' attribute in metadata/layout.conf
of repository X.
Suggested behavior: - Support for new 'updates-masters' attribute
in metadata/layout.conf. If this attribute is not set explicitly,
then it defaults to value of 'masters' attribute. - Settings from
profiles/updates directories in repository X and repositories set
in 'updates-masters' attribute in metadata/layout.conf of
repository X are used for packages installed from repository X.
(If a repository containing profiles/updates directory wants to
still not inherit updates settings from master repositories, then
it will need 'updates-masters =' in metadata/layout.conf.)
+1
The old behavior doesn't make any sense and the new one is in line
what's done for every other thing inherited from the masters.
-- Arfrever Frehtes Taifersar Arahesis
Re: [gentoo-portage-dev] New proposed modular sync system
Am 31.01.2014 08:27, schrieb Brian Dolbec: 18. Modify emerge-webrsync cli to use the new webrsync module I'd just remove emerge-webrsync and ask the user to use emerge --sync instead. Some people are not able to use a normal rsync tree due to firewall issues and bandwith restrictions. I didn't mean to remove the functionality provided by emerge-webrsync. Since in point 17 you want to move the functionality into a module, the user should be able to set sync-type=web-rsync in repos.conf and have emerge --sync sync it. Next up for discussion is what additional functionality should a module supply? I think it should provide a list or dictionary of the repos.conf variables it needs or uses. I also think that the module should provide a function for validating the conf variables meet some minimum standard/obvious error checking. Why should it provide that list at all, if it has a function to validate its input? The calling code should just check sync-type, select the appropriate module and pass everything to it. I am thinking more about an emaint module for checking repos.conf entries and help generation. The sync module would provide the validation function and the info the emaint module would use to list variable requirements. Yes, the plugin module should provide it's own man page, but maybe it doesn't or the user just wants a quick list of the variable requirements. emaint --list --variables syncs rsync: ... git: ... gpg-sig-uri squashfs: sync-uri gpg-sig-uri optional, blah blah,... ... I guess I'm just not familiar enough with your approach to see all the things you want to do with it.
[gentoo-portage-dev] [PATCH 00/10] First steps to get rid of backtracking
Hi all, as you may have noticed, emerge can in some cases take ages ( 5-10 minutes) to resolve dependencies these days. This happens when lots of backtracking is required to solve slot conflicts and/or to schedule slot operator rebuilds. The problem is that the current backtracking implementation has to rebuild the entire dependency graph from scratch each time it backtracks. This series of patches is a first step into fixing this problem. What these patches do: Patch 1 --- Adds a new data structure called package_tracker. This data structure is meant to replace several of the depgraph data structures. It has some new features not present in the existing data structures. 1) It can properly deal with several packages in the same slot. 2) It supports removal of previously added packages. 3) It tracks package conflicts automatically. 4) It has a more general concept of conflicts. Not all of the features are used for now, but they will make future changes easier. Patch 2-5 - These patches replace the old data structures with the package tracker Patch 6-8 - These patches fix several issues with emerge output. 6 and 8 are somewhat independent of the other patches in this series. Patch 7 introduces a new function used in Patch 10. Patch 9 --- New function for patch 10. Patch 10 This patch builds on top of all the previous patches. It introduces two new functions to the depgraph class. A function to remove packages that have previously been pulled in and a function to solve simple slot conflicts without backtracking. This should resolve most no parents that aren't satisfied by other packages in this slot slot conflicts. You may find these patches on github here: https://github.com/few/fews-portage-branch/tree/package_tracker Some numbers My system has quite a number of conflicts that give emerge a hard time resolving dependencies. -uDN world Without the patches: * 11 unsolved slot conflicts * 2 unsolved blockers * takes 5 backtracking steps and then fails With the patches: * no unsolved slot conflicts or blockers * takes 7 backtracking steps and then succeeds In this case it actually became slower, but was finally able to find a solution. -e world Without the patches: * 5 unsolved slot conflicts * 1 unsolved blocker * takes 23 backtracking steps and then fails With the patches: * 1 unsolved slot conflict (From a quick look it looks like there really is no solution.) * takes 13 backtracking steps and then fails In this case it's a lot faster, but still unacceptably slow. The result is improved by solving 4 out of 5 conflicts.
[gentoo-portage-dev] [PATCH 03/10] Replace mydbapi with _package_tracker
---
pym/_emerge/depgraph.py | 211 +++-
1 file changed, 101 insertions(+), 110 deletions(-)
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index fd59dda..9d234c2 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -344,7 +344,6 @@ class _dynamic_depgraph_config(object):
self._allow_backtracking = allow_backtracking
# Maps nodes to the reasons they were selected for
reinstallation.
self._reinstall_nodes = {}
- self.mydbapi = {}
# Contains a filtered view of preferred packages that are
selected
# from available repositories.
self._filtered_trees = {}
@@ -440,7 +439,6 @@ class _dynamic_depgraph_config(object):
# have after new packages have been installed.
fakedb = PackageTrackerDbapiWrapper(myroot,
self._package_tracker)
- self.mydbapi[myroot] = fakedb
def graph_tree():
pass
graph_tree.dbapi = fakedb
@@ -558,8 +556,6 @@ class depgraph(object):
if preload_installed_pkgs:
vardb = fake_vartree.dbapi
- fakedb = self._dynamic_config._graph_trees[
- myroot][vartree].dbapi
if not dynamic_deps:
for pkg in vardb:
@@ -724,25 +720,23 @@ class depgraph(object):
for pkg in list(self._dynamic_config.ignored_binaries):
- selected_pkg = self._dynamic_config.mydbapi[pkg.root
- ].match_pkgs(pkg.slot_atom)
+ selected_pkg = list()
- if not selected_pkg:
- continue
+ for selected_pkg in
self._dynamic_config._package_tracker.match(
+ pkg.root, pkg.slot_atom):
- selected_pkg = selected_pkg[-1]
- if selected_pkg pkg:
- self._dynamic_config.ignored_binaries.pop(pkg)
- continue
+ if selected_pkg pkg:
+
self._dynamic_config.ignored_binaries.pop(pkg)
+ break
- if selected_pkg.installed and \
- selected_pkg.cpv == pkg.cpv and \
- selected_pkg.build_time == pkg.build_time:
- # We don't care about ignored binaries when an
- # identical installed instance is selected to
- # fill the slot.
- self._dynamic_config.ignored_binaries.pop(pkg)
- continue
+ if selected_pkg.installed and \
+ selected_pkg.cpv == pkg.cpv and \
+ selected_pkg.build_time ==
pkg.build_time:
+ # We don't care about ignored binaries
when an
+ # identical installed instance is
selected to
+ # fill the slot.
+
self._dynamic_config.ignored_binaries.pop(pkg)
+ break
if not self._dynamic_config.ignored_binaries:
return
@@ -788,20 +782,25 @@ class depgraph(object):
# Exclude installed here since we only
# want to show available updates.
continue
- chosen_pkg = self._dynamic_config.mydbapi[pkg.root
- ].match_pkgs(pkg.slot_atom)
- if not chosen_pkg or chosen_pkg[-1] = pkg:
- continue
- k = (pkg.root, pkg.slot_atom)
- if k in missed_updates:
- other_pkg, mask_type, parent_atoms =
missed_updates[k]
- if other_pkg pkg:
- continue
- for mask_type, parent_atoms in mask_reasons.items():
- if not parent_atoms:
- continue
- missed_updates[k] = (pkg, mask_type,
parent_atoms)
- break
+ missed_update = True
+ any_selected = False
+ for chosen_pkg in
self._dynamic_config._package_tracker.match(
+
[gentoo-portage-dev] [PATCH 10/10] Solve some slot conflicts without backtracking
---
pym/_emerge/depgraph.py| 345 -
pym/portage/tests/resolver/test_backtracking.py| 13 +-
pym/portage/tests/resolver/test_blocker.py | 48 +++
pym/portage/tests/resolver/test_slot_collisions.py | 75 -
4 files changed, 457 insertions(+), 24 deletions(-)
create mode 100644 pym/portage/tests/resolver/test_blocker.py
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index 1bb086b..b5fecec 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -3,6 +3,7 @@
from __future__ import print_function, unicode_literals
+import collections
import errno
import io
import logging
@@ -78,7 +79,7 @@ from _emerge.resolver.backtracking import Backtracker,
BacktrackParameter
from _emerge.resolver.package_tracker import PackageTracker,
PackageTrackerDbapiWrapper
from _emerge.resolver.slot_collision import slot_conflict_handler
from _emerge.resolver.circular_dependency import circular_dependency_handler
-from _emerge.resolver.output import Display
+from _emerge.resolver.output import Display, format_unmatched_atom
if sys.hexversion = 0x300:
basestring = str
@@ -423,6 +424,8 @@ class _dynamic_depgraph_config(object):
self._complete_mode = False
self._slot_operator_deps = {}
self._package_tracker = PackageTracker()
+ # Track missed updates caused by solved conflicts.
+ self._conflict_missed_update = collections.defaultdict(dict)
for myroot in depgraph._frozen_config.trees:
self.sets[myroot] = _depgraph_sets()
@@ -769,7 +772,8 @@ class depgraph(object):
# missed update from each SLOT.
missed_updates = {}
for pkg, mask_reasons in \
- self._dynamic_config._runtime_pkg_mask.items():
+ chain(self._dynamic_config._runtime_pkg_mask.items(),
+
self._dynamic_config._conflict_missed_update.items()):
if pkg.installed:
# Exclude installed here since we only
# want to show available updates.
@@ -779,7 +783,8 @@ class depgraph(object):
for chosen_pkg in
self._dynamic_config._package_tracker.match(
pkg.root, pkg.slot_atom):
any_selected = True
- if chosen_pkg = pkg:
+ if chosen_pkg pkg or (not
chosen_pkg.installed and \
+ chosen_pkg.version == pkg.version):
missed_update = False
break
if any_selected and missed_update:
@@ -869,7 +874,7 @@ class depgraph(object):
self._show_merge_list()
msg = []
- msg.append(\nWARNING: One or more updates have been + \
+ msg.append(\nWARNING: One or more updates/rebuilds have been
+ \
skipped due to a dependency conflict:\n\n)
indent =
@@ -879,22 +884,29 @@ class depgraph(object):
msg.append( for %s % (pkg.root,))
msg.append(\n\n)
- for parent, atom in parent_atoms:
- msg.append(indent)
- msg.append(str(pkg))
+ msg.append(indent)
+ msg.append(str(pkg))
+ msg.append( conflicts with\n)
- msg.append( conflicts with\n)
- msg.append(2*indent)
+ for parent, atom in parent_atoms:
if isinstance(parent,
(PackageArg, AtomArg)):
# For PackageArg and AtomArg types, it's
# redundant to display the atom
attribute.
+ msg.append(2*indent)
msg.append(str(parent))
+ msg.append(\n)
else:
# Display the specific atom from SetArg
or
# Package types.
- msg.append(%s required by %s % (atom,
parent))
- msg.append(\n)
+ atom, marker = format_unmatched_atom(
+ pkg, atom,
self._pkg_use_enabled)
+
+ msg.append(2*indent)
+ msg.append(%s required by %s\n %
(atom, parent))
+
[gentoo-portage-dev] [PATCH 04/10] Replace _slot_collision_info with _package_tracker
--- pym/_emerge/depgraph.py| 59 -- pym/_emerge/resolver/slot_collision.py | 22 ++--- 2 files changed, 31 insertions(+), 50 deletions(-) diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py index 9d234c2..484ac14 100644 --- a/pym/_emerge/depgraph.py +++ b/pym/_emerge/depgraph.py @@ -378,11 +378,6 @@ class _dynamic_depgraph_config(object): # This use used to check if we have accounted for blockers # relevant to a package. self._traversed_pkg_deps = set() - # This should be ordered such that the backtracker will - # attempt to solve conflicts which occurred earlier first, - # since an earlier conflict can be the cause of a conflict - # which occurs later. - self._slot_collision_info = OrderedDict() # Slot collision nodes are not allowed to block other packages since # blocker validation is only able to account for one package per slot. self._slot_collision_nodes = set() @@ -915,7 +910,7 @@ class depgraph(object): cases. - if not self._dynamic_config._slot_collision_info: + if not any(self._dynamic_config._package_tracker.slot_conflicts()): return self._show_merge_list() @@ -971,16 +966,18 @@ class depgraph(object): is called, so that all relevant reverse dependencies are available for use in backtracking decisions. - for (slot_atom, root), slot_nodes in \ - self._dynamic_config._slot_collision_info.items(): - self._process_slot_conflict(root, slot_atom, slot_nodes) + for conflict in self._dynamic_config._package_tracker.slot_conflicts(): + self._process_slot_conflict(conflict) - def _process_slot_conflict(self, root, slot_atom, slot_nodes): + def _process_slot_conflict(self, conflict): Process slot conflict data to identify specific atoms which lead to conflict. These atoms only match a subset of the packages that have been pulled into a given slot. + root = conflict.root + slot_atom = conflict.atom + slot_nodes = conflict.pkgs debug = --debug in self._frozen_config.myopts @@ -1999,7 +1996,6 @@ class depgraph(object): existing_node, existing_node_matches = \ self._check_slot_conflict(pkg, dep.atom) - slot_collision = False if existing_node: if existing_node_matches: # The existing node can be reused. @@ -2032,19 +2028,7 @@ class depgraph(object): modified_use=self._pkg_use_enabled(existing_node))), level=logging.DEBUG, noiselevel=-1) - slot_collision = True - - if slot_collision: - # Now add this node to the graph so that self.display() - # can show use flags and --tree portage.output. This node is - # only being partially added to the graph. It must not be - # allowed to interfere with the other nodes that have been - # added. - # Even though the graph is now invalid, continue to process - # dependencies so that things like --fetchonly can still - # function despite collisions. - pass - elif not previously_added: + if not previously_added: self._dynamic_config._package_tracker.add_pkg(pkg) self._dynamic_config._filtered_trees[pkg.root][porttree].dbapi._clear_cache() self._dynamic_config._highest_pkg_cache.clear() @@ -2156,14 +2140,6 @@ class depgraph(object): def _add_slot_conflict(self, pkg): self._dynamic_config._slot_collision_nodes.add(pkg) - slot_key = (pkg.slot_atom, pkg.root) - slot_nodes = self._dynamic_config._slot_collision_info.get(slot_key) - if slot_nodes is None: - slot_nodes = set() - slot_nodes.update(self._dynamic_config._package_tracker.match( - pkg.root, pkg.slot_atom, installed=False)) -
[gentoo-portage-dev] [PATCH 08/10] Some small output fixes for the slot conflict handler
* unmatched atom printing now uses ^^ markers * unmatched atom printing properly supports sub-slots * Fix spurious no parents message caused by AtomArg parents --- pym/_emerge/resolver/slot_collision.py | 119 ++--- 1 file changed, 96 insertions(+), 23 deletions(-) diff --git a/pym/_emerge/resolver/slot_collision.py b/pym/_emerge/resolver/slot_collision.py index ca3fb74..c5936ad 100644 --- a/pym/_emerge/resolver/slot_collision.py +++ b/pym/_emerge/resolver/slot_collision.py @@ -1,4 +1,4 @@ -# Copyright 2010-2013 Gentoo Foundation +# Copyright 2010-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 from __future__ import print_function, unicode_literals @@ -273,12 +273,14 @@ class slot_conflict_handler(object): for ppkg, atom in parent_atoms: atom_set = InternalPackageSet(initial_atoms=(atom,)) atom_without_use_set = InternalPackageSet(initial_atoms=(atom.without_use,)) + atom_without_use_and_slot_set = InternalPackageSet(initial_atoms=( + atom.without_use.without_slot,)) for other_pkg in pkgs: if other_pkg == pkg: continue - if not atom_without_use_set.findAtomForPackage(other_pkg, \ + if not atom_without_use_and_slot_set.findAtomForPackage(other_pkg, \ modified_use=_pkg_use_enabled(other_pkg)): if atom.operator is not None: # The version range does not match. @@ -295,9 +297,11 @@ class slot_conflict_handler(object): atoms.add((ppkg, atom, other_pkg)) num_all_specific_atoms += 1 collision_reasons[key] = atoms - else: - # The sub_slot does not match. - key = (sub-slot, atom.sub_slot) + + elif not atom_without_use_set.findAtomForPackage(other_pkg, \ + modified_use=_pkg_use_enabled(other_pkg)): + # The slot and/or sub_slot does not match. + key = (slot, (atom.slot, atom.sub_slot, atom.slot_operator)) atoms = collision_reasons.get(key, set()) atoms.add((ppkg, atom, other_pkg)) num_all_specific_atoms += 1 @@ -342,6 +346,11 @@ class slot_conflict_handler(object): atoms.add((ppkg, atom, other_pkg)) collision_reasons[(use, flag)] = atoms num_all_specific_atoms += 1 + elif isinstance(ppkg, AtomArg) and other_pkg.installed: + parent_atoms = collision_reasons.get((AtomArg, None), set()) + parent_atoms.add((ppkg, atom)) + collision_reasons[(AtomArg, None)] = parent_atoms + num_all_specific_atoms += 1 msg.append( pulled in by\n) @@ -372,9 +381,11 @@ class slot_conflict_handler(object): if not verboseconflicts: selected_for_display.update( best_matches.values()) - elif type == sub-slot: +
[gentoo-portage-dev] [PATCH 09/10] Add digraph.discard
--- pym/portage/util/digraph.py | 12 +++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/pym/portage/util/digraph.py b/pym/portage/util/digraph.py index fc1fb86..4a9cb43 100644 --- a/pym/portage/util/digraph.py +++ b/pym/portage/util/digraph.py @@ -1,4 +1,4 @@ -# Copyright 2010-2013 Gentoo Foundation +# Copyright 2010-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 from __future__ import unicode_literals @@ -47,6 +47,16 @@ class digraph(object): priorities.append(priority) priorities.sort() + def discard(self, node): + + Like remove(), except it doesn't raises KeyError if the + node doesn't exist. + + try: + self.remove(node) + except KeyError: + pass + def remove(self, node): Removes the specified node from the digraph, also removing and ties to other nodes in the digraph. Raises KeyError if the -- 1.8.3.2
[gentoo-portage-dev] [PATCH 07/10] format_unmatched_atom: Pretty printing for unmatched atoms
This is a split out from the slot conflict handler to be used in other places. --- pym/_emerge/resolver/output.py | 109 +++-- 1 file changed, 106 insertions(+), 3 deletions(-) diff --git a/pym/_emerge/resolver/output.py b/pym/_emerge/resolver/output.py index 3e8552f..5f550be 100644 --- a/pym/_emerge/resolver/output.py +++ b/pym/_emerge/resolver/output.py @@ -1,4 +1,4 @@ -# Copyright 2010-2013 Gentoo Foundation +# Copyright 2010-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 Resolver output display operation. @@ -7,7 +7,7 @@ from __future__ import unicode_literals __all__ = ( - Display, + Display, format_unmatched_atom, ) import sys @@ -23,8 +23,9 @@ from portage.package.ebuild._spawn_nofetch import spawn_nofetch from portage.output import ( blue, colorize, create_color_func, darkblue, darkgreen, green, nc_len, teal) bad = create_color_func(BAD) +from portage._sets.base import InternalPackageSet from portage.util import writemsg_stdout -from portage.versions import best +from portage.versions import best, cpv_getversion from _emerge.Blocker import Blocker from _emerge.create_world_atom import create_world_atom @@ -916,3 +917,105 @@ class Display(object): self.print_changelog() return os.EX_OK + + +def format_unmatched_atom(pkg, atom, pkg_use_enabled): + + Returns two strings. The first string contains the + 'atom' with parts of the atom colored, which 'pkg' + doesn't match. The second string has the same number + of characters as the first one, but consists of only + white space or ^. The ^ characters have the same position + as the colored parts of the first string. + + # Things to check: + # 1. Version + # 2. cp + # 3. slot/sub_slot + # 4. repository + # 5. USE + + highlight = set() + + def perform_coloring(): + atom_str = + marker_str = + for ii, x in enumerate(atom): + if ii in highlight: + atom_str += colorize(BAD, x) + marker_str += ^ + else: + atom_str += x + marker_str += + return atom_str, marker_str + + if atom.cp != pkg.cp: + # Highlight the cp part only. + ii = atom.find(atom.cp) + highlight.update(range(ii, ii + len(atom.cp))) + return perform_coloring() + + version_atom = atom.without_repo.without_slot.without_use + version_atom_set = InternalPackageSet(initial_atoms=(version_atom,)) + highlight_version = not bool(version_atom_set.findAtomForPackage(pkg, + modified_use=pkg_use_enabled(pkg))) + + highlight_slot = False + if (atom.slot and atom.slot != pkg.slot) or \ + (atom.sub_slot and atom.sub_slot != pkg.sub_slot): + highlight_slot = True + + if highlight_version: + op = atom.operator + ver = None + if atom.cp != atom.cpv: + ver = cpv_getversion(atom.cpv) + + if op == =*: + op = = + ver += * + + if op is not None: + highlight.update(range(len(op))) + + if ver is not None: + start = atom.rfind(ver) + end = start + len(ver) + highlight.update(range(start, end)) + + if highlight_slot: + slot_str = : + atom.slot + if atom.sub_slot: + slot_str += / + atom.sub_slot + if atom.slot_operator: + slot_str += atom.slot_operator + start = atom.find(slot_str) + end = start + len(slot_str) + highlight.update(range(start, end)) + + highlight_use = set() + if atom.use: + use_atom = %s[%s] % (atom.cp, str(atom.use)) + use_atom_set = InternalPackageSet(initial_atoms=(use_atom,)) + if not use_atom_set.findAtomForPackage(pkg, \ + modified_use=pkg_use_enabled(pkg)): + missing_iuse = pkg.iuse.get_missing_iuse( + atom.unevaluated_atom.use.required) + if missing_iuse: + highlight_use = set(missing_iuse) + else: + #Use conditionals not met. + violated_atom = atom.violated_conditionals( + pkg_use_enabled(pkg), pkg.iuse.is_valid_flag) + if violated_atom.use is not None: +
[gentoo-portage-dev] [PATCH 05/10] Replace _slot_collision_nodes with _package_tracker
---
pym/_emerge/depgraph.py | 16 +++-
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index 484ac14..1bb086b 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -378,9 +378,6 @@ class _dynamic_depgraph_config(object):
# This use used to check if we have accounted for blockers
# relevant to a package.
self._traversed_pkg_deps = set()
- # Slot collision nodes are not allowed to block other packages
since
- # blocker validation is only able to account for one package
per slot.
- self._slot_collision_nodes = set()
self._parent_atoms = {}
self._slot_conflict_handler = None
self._circular_dependency_handler = None
@@ -1799,11 +1796,16 @@ class depgraph(object):
buildpkgonly = --buildpkgonly in self._frozen_config.myopts
nodeps = --nodeps in self._frozen_config.myopts
if dep.blocker:
+
+ # Slot collision nodes are not allowed to block other
packages since
+ # blocker validation is only able to account for one
package per slot.
+ is_slot_conflict_parent = any(dep.parent in
conflict.pkgs[1:] for conflict in \
+
self._dynamic_config._package_tracker.slot_conflicts())
if not buildpkgonly and \
not nodeps and \
not dep.collapsed_priority.ignored and \
not dep.collapsed_priority.optional and \
- dep.parent not in
self._dynamic_config._slot_collision_nodes:
+ not is_slot_conflict_parent:
if dep.parent.onlydeps:
# It's safe to ignore blockers if the
# parent is an --onlydeps node.
@@ -2019,7 +2021,6 @@ class depgraph(object):
level=logging.DEBUG, noiselevel=-1)
else:
- self._add_slot_conflict(pkg)
if debug:
writemsg_level(
%s%s %s\n % (Slot
Conflict:.ljust(15),
@@ -2138,9 +2139,6 @@ class depgraph(object):
self._dynamic_config._slot_operator_deps[slot_key] =
slot_info
slot_info.append(dep)
- def _add_slot_conflict(self, pkg):
- self._dynamic_config._slot_collision_nodes.add(pkg)
-
def _add_pkg_deps(self, pkg, allow_unsatisfied=False):
myroot = pkg.root
@@ -6019,7 +6017,7 @@ class depgraph(object):
if complete not in self._dynamic_config.myparams and \
self._dynamic_config._allow_backtracking and \
- self._dynamic_config._slot_collision_nodes and \
+
any(self._dynamic_config._package_tracker.slot_conflicts()) and \
not self._accept_blocker_conflicts():
self._dynamic_config.myparams[complete] = True
--
1.8.3.2
[gentoo-portage-dev] [PATCH 01/10] Add resolver/package_tracker
--- pym/_emerge/resolver/package_tracker.py | 310 1 file changed, 310 insertions(+) create mode 100644 pym/_emerge/resolver/package_tracker.py diff --git a/pym/_emerge/resolver/package_tracker.py b/pym/_emerge/resolver/package_tracker.py new file mode 100644 index 000..4aee4ea --- /dev/null +++ b/pym/_emerge/resolver/package_tracker.py @@ -0,0 +1,310 @@ +# Copyright 2014 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 + +from __future__ import print_function + +import collections + +import portage +portage.proxy.lazyimport.lazyimport(globals(), + 'portage:OrderedDict', + 'portage.dep:Atom,match_from_list', + 'portage.util:cmp_sort_key', + 'portage.versions:vercmp', +) + +_PackageConflict = collections.namedtuple(_PackageConflict, [root, pkgs, atom, description]) + +class PackageConflict(_PackageConflict): + + Class to track the reason for a conflict and the conflicting packages. + + def __iter__(self): + return iter(self.pkgs) + + def __contains__(self, pkg): + return pkg in self.pkgs + + def __len__(self): + return len(self.pkgs) + + +class PackageTracker(object): + + This class tracks packages which are currently + installed and packages which have been pulled into + the dependency graph. + + It automatically tracks conflicts between packages. + + Possible conflicts: + 1) Packages that share the same SLOT. + 2) Packages with the same cpv. + Not yet implemented: + 3) Packages that block each other. + + + def __init__(self): + # Mapping from package keys to set of packages. + self._cp_pkg_map = collections.defaultdict(list) + self._cp_vdb_pkg_map = collections.defaultdict(list) + # List of package keys that may contain conflicts. + # The insetation order must be preserved. + self._multi_pkgs = [] + + # Cache for result of conflicts(). + self._conflicts_cache = None + + # Records for each pulled package which installed package + # are replaced. + self._replacing = collections.defaultdict(list) + # Records which pulled packages replace this package. + self._replaced_by = collections.defaultdict(list) + + self._match_cache = collections.defaultdict(dict) + + def add_pkg(self, pkg): + + Add a new package to the tracker. Records conflicts as necessary. + + cp_key = pkg.root, pkg.cp + + try: + if any(other is pkg for other in self._cp_pkg_map[cp_key]): + return + except KeyError: + self._cp_pkg_map[cp_key] = [pkg] + else: + self._cp_pkg_map[cp_key].append(pkg) + if len(self._cp_pkg_map[cp_key]) == 2: + self._multi_pkgs.append(cp_key) + self._conflicts_cache = None + + self._replacing[pkg] = [] + for installed in self._cp_vdb_pkg_map[cp_key]: + if installed.slot_atom == pkg.slot_atom or \ + installed.cpv == pkg.cpv: + self._replacing[pkg].append(installed) + self._replaced_by[installed].append(pkg) + + self._match_cache.pop((pkg.root, pkg.cp)) + + def add_installed_pkg(self, installed): + + Add an installed package during vdb load. These packages + are not returned by matched_pull as long as add_pkg hasn't + been called with them. They are only returned by match_final. + + cp_key = installed.root, installed.cp + try: + if any(other is installed for other in self._cp_vdb_pkg_map[cp_key]): + return + except KeyError: + self._cp_vdb_pkg_map[cp_key] = [installed] + else: + self._cp_vdb_pkg_map[cp_key].append(installed) + + for pkg in self._cp_pkg_map[cp_key]: + if installed.slot_atom == pkg.slot_atom or \ + installed.cpv == pkg.cpv: + self._replacing[pkg].append(installed) + self._replaced_by[installed].append(pkg) + + def remove_pkg(self, pkg): + + Removes the package from the tracker. + Raises KeyError if it isn't present. + + cp_key = pkg.root, pkg.cp + try: +
[gentoo-portage-dev] layout.conf: What's our opinion?
Hi all,
${repository}/metadata/layout.conf is a file that allows a repository
maintainer to adjust the package manager's behavior for a repository.
I guess the best known is the 'masters' key, but there are lots of other
things by now (see man portage).
Currently layout.conf is not under PMS control. This basically means
that every PM (or version thereof) may support different keys and assign
different meanings to them. Portage's behavior for unknown keys in
layout.conf is to ignore them without a warning.
The bad thing about this is that some layout.conf keys portage currently
supports, may render the repository unusable for a PM if it doesn't
support them.
To avoid this type of breakage in other areas (ebuilds, dependency
resolution, ...) PMS has been created. Since the council demands PMS to
be followed, I would expect that they also want the general idea of not
breaking things randomly to be followed.
This brings us to reason that made me write that mail. Some days ago
Arfrever committed some additions to layout.conf [1], for which he
apparently had the ack from Zac from some months ago [2].
After discussing this one IRC I came to the conclusion that we just
disagree on how we should handle additions to layout.conf.
Basically it's either
1) We add things as we see fit. or
2) We should only add things if absolutely necessary..
I obviously would prefer 2) to follow the things shouldn't break
randomly route.
So what's your opinion? Should we go for 1) or 2) or something else?
- Sebastian
[1]
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=4c409a049c394389b1de398db511380e2fed0437
[2] http://dpaste.com/1560782/
[gentoo-portage-dev] [PATCH] Implement FEATURES=mirror for emerge (bug 498498)
This was only implemented for the ebuild command before. --- pym/_emerge/Scheduler.py | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/pym/_emerge/Scheduler.py b/pym/_emerge/Scheduler.py index d663e97..2fd4d7e 100644 --- a/pym/_emerge/Scheduler.py +++ b/pym/_emerge/Scheduler.py @@ -165,6 +165,8 @@ class Scheduler(PollScheduler): self._build_opts.buildpkg_exclude = InternalPackageSet( \ initial_atoms= .join(myopts.get(--buildpkg-exclude, [])).split(), \ allow_wildcard=True, allow_repo=True) + if mirror in self.settings.features: + self._build_opts.fetch_all_uri = True self._binpkg_opts = self._binpkg_opts_class() for k in self._binpkg_opts.__slots__: @@ -752,11 +754,11 @@ class Scheduler(PollScheduler): pass elif pkg.type_name == ebuild: - prefetcher = EbuildFetcher(background=True, config_pool=self._ConfigPool(pkg.root, self._allocate_config, self._deallocate_config), - fetchonly=1, logfile=self._fetch_log, + fetchonly=1, fetchall=self._build_opts.fetch_all_uri, + logfile=self._fetch_log, pkg=pkg, prefetch=True, scheduler=self._sched_iface) elif pkg.type_name == binary and \ -- 1.8.3.2
Re: [gentoo-portage-dev] [PATCH] Implement FEATURES=mirror for emerge (bug 498498)
The removed white space was unintentional. Please remove that change before committing that patch.
Re: [gentoo-portage-dev] [PATCH 0/3] Initial fetch() refactoring
Am 19.01.2014 04:07, schrieb W. Trevor King: I felt like I should actually make a useful contribution to balance the less useful commit-message discussion ;). Browsing through the open Portage bugs, #175612 looked interesting. After I looked at pym/portage/package/ebuild/fetch.py, I decided to take a step back and just try and refactor fetch(), which was pushing 1k lines. Here are three paches pulling fairly self-contained blocks out of fetch(). I thought I'd float them to the list to see if this was a productive avenue, or if this is going to be too much work to review. I tried to avoid making too many changes other than the function-extraction, but in some places I couldn't help myself ;). Good idea. The patches aren't particularly well tested yet. I ran the test suite and got some errors, but they seemed to be related to my non-root invocation, and not due to these changes themselves. I thought I'd post my work so far, before digging deeper into the test suite. The tests can be run as non-root user. Make sure all the files in the git repo are +rw for that user. You should check the result using pyflakes. Do you have these patches in the publicly accessible git repo? Cheers, Trevor [1]: https://bugs.gentoo.org/show_bug.cgi?id=175612 W. Trevor King (3): pym/portage/package/ebuild/fetch.py: Factor out _get_checksum_failure_max_tries pym/portage/package/ebuild/fetch.py: Factor out _get_fetch_resume_size pym/portage/package/ebuild/fetch.py: Factor out _get_uris pym/portage/package/ebuild/fetch.py | 305 +--- 1 file changed, 177 insertions(+), 128 deletions(-)
Re: [gentoo-portage-dev] [PATCH 3/3] pym/portage/package/ebuild/fetch.py: Factor out _get_uris
Am 19.01.2014 04:07, schrieb W. Trevor King: +def _get_uris(uris, settings, custom_mirrors=(), locations=()): missing doc string
Re: [gentoo-portage-dev] [PATCH 0/3] Initial fetch() refactoring
Am 19.01.2014 04:07, schrieb W. Trevor King: The patches aren't particularly well tested yet. I ran the test suite and got some errors, but they seemed to be related to my non-root invocation, and not due to these changes themselves. I thought I'd post my work so far, before digging deeper into the test suite. Since I doubt there's currently any test or any way to write one, I'd suggest to try and run emerge and analyze coverage with dev-python/coverage or a similar tool.
Re: [gentoo-portage-dev] [PATCH 1/3] Have repoman check if the packages to unpack rare archive formats from SRC_URI are present in DEPEND (bug #205909).
Am 16.01.2014 01:07, schrieb Tom Wijsman:
+
+archive_formats_eapi_3_to_5 = {
+ \.tar.xz:app-arch/tar,
+ \.xz:app-arch/xz-utils,
Shouldn't .tar.xz require both tar and xz-utils? Other entries may
have the same problem.
Re: [gentoo-portage-dev] [PATCH 1/3] Have repoman check if the packages to unpack rare archive formats from SRC_URI are present in DEPEND (bug #205909).
Am 16.01.2014 22:40, schrieb Tom Wijsman:
On Thu, 16 Jan 2014 08:03:03 +0100 Sebastian Luther
sebastianlut...@gmx.de wrote:
Am 16.01.2014 01:07, schrieb Tom Wijsman:
--- bin/repoman | 53
+
man/repoman.1 | 4 2 files changed, 57 insertions(+)
diff --git a/bin/repoman b/bin/repoman index d1542e9..9b703dc
100755 --- a/bin/repoman +++ b/bin/repoman @@ -36,6 +36,9 @@
pym_path =
osp.join(osp.dirname(osp.dirname(osp.realpath(__file__))),
pym) sys.path.insert(0, pym_path) import portage
portage._internal_caller = True + +from portage._sets.profiles
import PackagesSystemSet +system_set_atoms =
PackagesSystemSet(portage.settings.profiles).getAtoms()
portage._disable_legacy_globals()
You should be using repoman_settings instead of
portage.settings.
If I understand correctly, that is this URL?
http://dev.gentoo.org/~zmedico/portage/doc/api/portage.repository.config-module.html
How do I get the @system set out of that?
portage.settings and repoman_settings are instances of
portage.package.ebuild.config.config. I just looked at the code and
think now that you should keep using PackagesSystemSet to get the
@system atoms. Just turn them into a set afterwards with set(atom.cp
for atom in system_set_atoms).
The reason to use atom.cp is that =cat/foo-1 could be part of that
set and then 'cat/foo in system_set_atoms' would return False.
Considering the later use
Which use?
The if entry not in system_set_atoms line. You're using __contains__
there (with the 'in'). You don't use the additional magic provided by
PackageSet (which is a super class of PackagesSystemSet).
you don't need PackagesSystemSet set here, just use a set.
Okay, thus I need to create some kind of set object here (I don't
see one in the list of
http://dev.gentoo.org/~zmedico/portage/doc/api/ though) and then
specify that it would be the @system set? Which class?
Whenever I say 'set' I mean python's builtin set.
And use atom.cp instead of the atoms.
So, if I understood correctly; using list comprehension, I
directly transform the getAtoms() to a list of atom.cp's... Okay,
good idea.
try: @@ -300,6 +303,7 @@ qahelp = { inherit.missing: Ebuild
uses functions from an eclass but does not inherit it,
inherit.unused: Ebuild inherits an eclass but does not use
it, java.eclassesnotused: With virtual/jdk in DEPEND you
must inherit a java eclass, + unpack.DEPEND.missing: A rare
archive format was used in SRC_URI, but its package to unpack
it is missing in DEPEND., wxwidgets.eclassnotused: Ebuild
DEPENDs on x11-libs/wxGTK without inheriting wxwidgets.eclass,
KEYWORDS.dropped: Ebuilds that appear to have dropped
KEYWORDS for some arch, KEYWORDS.missing: Ebuilds that have
a missing or empty KEYWORDS variable, @@ -399,6 +403,7 @@
qawarnings = set(( metadata.warning, portage.internal,
repo.eapi.deprecated, +unpack.DEPEND.missing,
usage.obsolete, upstream.workaround, LIVEVCS.stable, @@
-479,6 +484,25 @@ ruby_deprecated = frozenset([
ruby_targets_ree18, ])
+# TODO: Add functionality to support checking for deb2targz
on platforms where +# GNU binutils is absent; see PMS 5,
section 11.3.3.13. +archive_formats = { +
\.7[zZ]:app-arch/p7zip, +
\.(bz2?|tbz2):app-arch/bzip2, + \.jar:app-arch/unzip, +
\.(LH[aA]|lha|lzh):app-arch/lha, +
\.lzma:app-arch/lzma-utils, +
\.(rar|RAR):app-arch/unrar, +
\.(tar(\.(bz2?|gz|Z))?|tbz2|t[bg]z)?:app-arch/tar, +
\.(gz|tar\.Z|t[bg]z|[zZ]):app-arch/gzip, +
\.(zip|ZIP):app-arch/unzip, +} +
+archive_formats_eapi_3_to_5 = { + \.tar.xz:app-arch/tar, +
\.xz:app-arch/xz-utils, +} + metadata_xml_encoding =
'UTF-8' metadata_xml_declaration = '?xml version=1.0
encoding=%s?' % \ (metadata_xml_encoding,) @@ -1559,6
+1583,7 @@ for x in effective_scanlist: fetchlist_dict =
portage.FetchlistDict(checkdir, repoman_settings, portdb)
myfiles_all = [] src_uri_error = False +needed_unpack_depends
= {} for mykey in fetchlist_dict: try:
myfiles_all.extend(fetchlist_dict[mykey]) @@ -1573,7 +1598,22
@@ for x in effective_scanlist: stats[SRC_URI.syntax] += 1
fails[SRC_URI.syntax].append( %s.ebuild SRC_URI: %s %
(mykey, e)) + + # Compare each SRC_URI entry against
archive_formats; if one of the +# extensions match, we
remember which archive depends are needed to + # check them
later on. + needed_unpack_depends[mykey] = [] + for
file_extension in archive_formats or \ +
((re.match('[345]$',
eapi) is not None) \
Use portage.eapi for the line above.
Why? 'eapi' is the EAPI of the ebuild, what is wrong with that?
What I want you to do is to change (re.match('[345]$', eapi) into
something like: eapi_has_xz_unpack(eapi). the function
eapi_has_xz_unpack needs to be written. It should be part of portage.eapi.
You may have to add a new function to portage.eapi.
What would the purpose of that function
Re: [gentoo-portage-dev] [PATCH 1/3] Have repoman check if the packages to unpack rare archive formats from SRC_URI are present in DEPEND (bug #205909).
Am 18.01.2014 00:00, schrieb Tom Wijsman: On Fri, 17 Jan 2014 09:35:58 +0100 Sebastian Luther sebastianlut...@gmx.de wrote: The if entry not in system_set_atoms line. You're using __contains__ there (with the 'in'). You don't use the additional magic provided by PackageSet (which is a super class of PackagesSystemSet). This is __contains__, which does what I want as far as I can see: http://dev.gentoo.org/~zmedico/portage/doc/api/portage._sets.base-pysrc.html#PackageSet.__contains__ What is there wrong with this? There's nothing wrong with that. It's just that the builtin set type supports that too. As for the additional magic, do you mean containsCPV? Looking at it: http://dev.gentoo.org/~zmedico/portage/doc/api/portage._sets.base-pysrc.html#PackageSet.containsCPV It seems more complex than is necessary, is there a benefit to this? I thought more about iterAtomsForPackage, which is used in lots of places in the resolver. The comment was about the fact that you're using a more complicated selfmade class instead of a builtin type, even if the builtin type would suffice. The comment made more sense when I was still suggesting to not use PackageSystemSet at all.
Re: [gentoo-portage-dev] Helping out
Here is another one. This is the first patch mentioned here that touches
the dependency resolver. I'd be nice if we had more people with the
ability to work on it. So if you're interested in that, take a look.
Bug 498122 - portage-2.2.8 takes nearly twice as long to calculate
dependencies for world update
The problem with the patch mentioned in that bug is, that from there on
metadata access (specifically USE needs to be computed) is required to
search for slot operator rebuilds.
I expect that the metadata access can be avoided in most cases. To fix
this bug, I suggest to implement a check that decides if the metadata
access is required.
Here are some hints.
Take a look at the first block of the patch. It's about computing the
'atoms' variable. This variable is a set of all atoms in all *DEPEND
strings of some package ('replacement_parent').
This list is later searched for atoms for which atom.cp is equal to
dep.atom.cp (and which aren't blockers).
Before this patch the whole content of the *DEPEND string was added to
'atoms'. An example:
DEPEND=x? ( cat/foo ) would always result in atoms = {cat/foo}, even
if the use flag 'x' was disabled for the package.
The leads to problems in the following case:
DEPEND=x? ( =cat/foo-1 ) !x? ( =cat/foo-2 ).
The code later in the function tries to find a package that matches all
atoms in 'atoms'. Obviously there cannot be any package that satisfies
both =cat/foo-1 and =cat/foo-2 at the same time.
And this is not necessary at all here, because of the way the DEPEND is
written. After evaluating the use conditionals there is always one atom
left. But to figure this out, you need to know USE (the set of enabled
use flags).
Since computing USE is expansive, we'd like to avoid computing it.
Consider the following example:
We're searching for atoms with atom.cp == cat/foo:
DEPEND=cat/foo x? ( cat/bar )
No use conditional influences the set of atoms in this DEPEND for which
atom.cp == cat/foo. There's no point in evaluation them and in
consequence there's no point of computing USE.
I propose to implement a function* that maps (package, cp) to the set of
atoms in the *DEPEND of package for which atom.cp == cp (and which
aren't blockers).
This function should:
* First decide if evaluating USE is necessary or not
* Then evaluate the conditionals if required
* Compute and return the set of atoms with atom.cp == cp.
This function should cache its results. For the case without USE, the
cache should be part of self._frozen_config (contains stuff that never
changes). For the case that needs USE, it should be in
self._dynamic_config (contains stuff that may change between
backtracking steps).
For the implementation of the check you'll want to look at the functions
in portage.dep, specifically paren_reduce (ignore the deprecation
warning, we may have to remove that).
Feel free to ask questions here or on IRC.
hf :)
Re: [gentoo-portage-dev] [PATCH] Document bugzilla workflow
Am 15.01.2014 17:20, schrieb Tom Wijsman: On Wed, 15 Jan 2014 07:29:19 +0100 Sebastian Luther sebastianlut...@gmx.de wrote: Am 15.01.2014 04:11, schrieb Tom Wijsman: I send the first mail with this wording 8 days ago. Enough time to comment on it. I'd prefer to discuss it on the list. Yes, but not all comments were discussed yet, therefore (dis)agreement on them is missing; and this last thing rather became a topic of discussion due to the work clashes that we saw happen twice. I'd say the clashes occurred because nobody mentioned at all what they are working on. Since people started using IN_PROGRESS to mean I'm working on it, this shouldn't happen again. Yes, I see some commit messages not refer to bugs which is something we will want to avoid; think this might need to go into the commit policy. There's nothing wrong with fixing/implementing something that nobody filed a bug about. The way it was is to not care about them at all. There was no agreement on the the other thread if these things should be used or not. So I left it vague so everyone could use it, but they are not forced to. Hmm, could this result in conflicting usage of these? Maybe, but I'd first see if the usage patterns converge to something that makes everyone happy. +There are a number of bugs named [TRACKER] * that collect bugs +for specific topics. Confirmed bugs should be marked as blocking +these tracker bugs if appropriate. For clarity, it should be mentioned that this does not mean to block the tracker for the next version; this could be misinterpreted. Considering that the tracker gets renamed, I'm not sure what you mean here. As you are confused yourself by misinterpreting what you have written, you demonstrate the case for the need of clarity here; this is not about the next version tracker or it being renamed at all, it's about all other trackers that are not version trackers. The part of the policy quoted here doesn't make that clear, it had me puzzling for a moment too when I first read that; I think you were puzzled too now... Sorry, I failed to properly read what you quoted. I think once you know that these other trackers exist, it's clear. If you want something added there, that's fine with me too. Sebastian
Re: [gentoo-portage-dev] [PATCH] Document bugzilla workflow
Am 15.01.2014 19:41, schrieb Tom Wijsman: Yes, I see some commit messages not refer to bugs which is something we will want to avoid; think this might need to go into the commit policy. There's nothing wrong with fixing/implementing something that nobody filed a bug about. Sorry, consider the common case where a bug was filed but the commit message does not refer to that bug. Also note that I am trying to refer to the ChangeLog of Portage itself, not that of the ebuild; thus I mean the commit messages for the Portage repo, not to the Portage tree. Not sure if we're talking about the same things. 1) If you fix something that has a bug, you should refer to that in the git commit message. 2) There's nothing wrong with a git commit message that does not refer to a bug, if there is no bug filed. The whole point of documenting it in a workflow is to make it converge; Not the converge I meant. What I meant was to allow people to test different styles and hope that the one that works best will be adopted by everyone at some point. Once that happens you can document that style. if you instead leave things unclear or undocumented, you have no guaranteed convergence and might even see a disconvergence. Yes, maybe. One then needs to see if that is a problem and if it is then force everyone to use one style. It's already making people unhappy right now; because as it is documented now, it is turned from the meaningful experience that the previous Portage team had before to something that is meaningless. It is a regression in checking the list of bugs that block the tracker, as the states of the bugs no longer have a value as it is documented now. Previously the bug state was not used at all. There is no regression.
Re: [gentoo-portage-dev] [PATCH 1/3] Have repoman check if the packages to unpack rare archive formats from SRC_URI are present in DEPEND (bug #205909).
Am 16.01.2014 01:07, schrieb Tom Wijsman:
---
bin/repoman | 53 +
man/repoman.1 | 4
2 files changed, 57 insertions(+)
diff --git a/bin/repoman b/bin/repoman
index d1542e9..9b703dc 100755
--- a/bin/repoman
+++ b/bin/repoman
@@ -36,6 +36,9 @@ pym_path =
osp.join(osp.dirname(osp.dirname(osp.realpath(__file__))), pym)
sys.path.insert(0, pym_path)
import portage
portage._internal_caller = True
+
+from portage._sets.profiles import PackagesSystemSet
+system_set_atoms = PackagesSystemSet(portage.settings.profiles).getAtoms()
portage._disable_legacy_globals()
You should be using repoman_settings instead of portage.settings.
Considering the later use, you don't need PackagesSystemSet set here,
just use a set. And use atom.cp instead of the atoms.
try:
@@ -300,6 +303,7 @@ qahelp = {
inherit.missing: Ebuild uses functions from an eclass but does not
inherit it,
inherit.unused: Ebuild inherits an eclass but does not use it,
java.eclassesnotused: With virtual/jdk in DEPEND you must inherit a
java eclass,
+ unpack.DEPEND.missing: A rare archive format was used in SRC_URI,
but its package to unpack it is missing in DEPEND.,
wxwidgets.eclassnotused: Ebuild DEPENDs on x11-libs/wxGTK without
inheriting wxwidgets.eclass,
KEYWORDS.dropped: Ebuilds that appear to have dropped KEYWORDS for
some arch,
KEYWORDS.missing: Ebuilds that have a missing or empty KEYWORDS
variable,
@@ -399,6 +403,7 @@ qawarnings = set((
metadata.warning,
portage.internal,
repo.eapi.deprecated,
+unpack.DEPEND.missing,
usage.obsolete,
upstream.workaround,
LIVEVCS.stable,
@@ -479,6 +484,25 @@ ruby_deprecated = frozenset([
ruby_targets_ree18,
])
+# TODO: Add functionality to support checking for deb2targz on platforms
where
+# GNU binutils is absent; see PMS 5, section 11.3.3.13.
+archive_formats = {
+ \.7[zZ]:app-arch/p7zip,
+ \.(bz2?|tbz2):app-arch/bzip2,
+ \.jar:app-arch/unzip,
+ \.(LH[aA]|lha|lzh):app-arch/lha,
+ \.lzma:app-arch/lzma-utils,
+ \.(rar|RAR):app-arch/unrar,
+ \.(tar(\.(bz2?|gz|Z))?|tbz2|t[bg]z)?:app-arch/tar,
+ \.(gz|tar\.Z|t[bg]z|[zZ]):app-arch/gzip,
+ \.(zip|ZIP):app-arch/unzip,
+}
+
+archive_formats_eapi_3_to_5 = {
+ \.tar.xz:app-arch/tar,
+ \.xz:app-arch/xz-utils,
+}
+
metadata_xml_encoding = 'UTF-8'
metadata_xml_declaration = '?xml version=1.0 encoding=%s?' % \
(metadata_xml_encoding,)
@@ -1559,6 +1583,7 @@ for x in effective_scanlist:
fetchlist_dict = portage.FetchlistDict(checkdir, repoman_settings,
portdb)
myfiles_all = []
src_uri_error = False
+ needed_unpack_depends = {}
for mykey in fetchlist_dict:
try:
myfiles_all.extend(fetchlist_dict[mykey])
@@ -1573,7 +1598,22 @@ for x in effective_scanlist:
stats[SRC_URI.syntax] += 1
fails[SRC_URI.syntax].append(
%s.ebuild SRC_URI: %s % (mykey, e))
+
+ # Compare each SRC_URI entry against archive_formats; if one of
the
+ # extensions match, we remember which archive depends are
needed to
+ # check them later on.
+ needed_unpack_depends[mykey] = []
+ for file_extension in archive_formats or \
+ ((re.match('[345]$', eapi) is not None) \
Use portage.eapi for the line above. You may have to add a new function
to portage.eapi.
+ and file_extension in
archive_formats_eapi_3_to_5):
+ for entry in fetchlist_dict[mykey]:
+ if re.match('.*%s$' % file_extension, entry) is
not None:
+ format = archive_formats[file_extension]
As these regex are used frequently, they should be compiled using
re.compile.
+
+ if format not in
needed_unpack_depends[mykey]:
+
needed_unpack_depends[mykey].append(format)
I'd make needed_unpack_depends[mykey] a set. Then you can just add()
instead of checking and appending.
del fetchlist_dict
+
if not src_uri_error:
# This test can produce false positives if SRC_URI could not
# be parsed for one or more ebuilds. There's no point in
@@ -2010,6 +2050,17 @@ for x in effective_scanlist:
atoms = None
badsyntax.append(str(e))
+ if atoms and mytype == 'DEPEND':
Use if atoms and buildtime: here.
+ # We check whether the needed archive
dependencies are present
+ # in DEPEND, which were determined from SRC_URI.
+ for entry in
Re: [gentoo-portage-dev] [PATCH] Document bugzilla workflow
Am 15.01.2014 04:11, schrieb Tom Wijsman: On Wed, 8 Jan 2014 19:25:19 +0100 sebastianlut...@gmx.de wrote: From: Sebastian Luther sebastianlut...@gmx.de Can you fix your git config too? See vapier's feedback on creffet. I'll take a look. +Bugzilla + More discussion is needed before we should add this; at least I think it should be brought up during the meeting this Sunday, because we've barely had feedback and at least one suggestion doesn't appear discussed and/or incorporated. I send the first mail with this wording 8 days ago. Enough time to comment on it. I'd prefer to discuss it on the list. I've commented on the suggestion by dol-sen which I like the most; especially the assignment part, I think it also contains some other neat additions. Besides that, I'll comment my thoughts on some of the other parts here: +There always exists a tracker bug, named: +[Tracker] sys-apps/portage-next version. + +This bug is renamed from X.Y.Z to X.Y.Z+1 after a release, until +it gets closed when Y changes and a new one is opened. While this spares out on tracker bugs, we lose the ability to track which bugs were fixed in which version, That's not true. Just look at the tracker for 2.2.9. Between the renames of the tracker you'll see which bug has been marked as blocking. These are the fixed ones. unless we enforce that all bug numbers get to be listed in the ChangeLog; do we have a policy for that? The numbers go into the ebuild changelog, but I don't think that's written down somewhere (/me looks at dol-sen). +Whenever a commit for a specific bug is made to the git repo, +the corresponding bug gets changed in the following ways: +* InVCS is added to Keywords +* The bug is marked as blocking the tracker for the next version +* A comment is added saying: This is fixed in git: url to commit +(note that the bug stays open) +1 +After a release all open bugs blocking the tracker are closed +with the comment This is fixed in version.. +1 +For individual open bugs it is encouraged to set UNCONFIRMED, +CONFIRMED or IN_PROGESS as appropriate. What is as appropriate? As I mentioned, this needs more discussion; please keep this the way it was, it makes the tracker bug more useful. The way it was is to not care about them at all. There was no agreement on the the other thread if these things should be used or not. So I left it vague so everyone could use it, but they are not forced to. +There are a number of bugs named [TRACKER] * that collect bugs +for specific topics. Confirmed bugs should be marked as blocking +these tracker bugs if appropriate. For clarity, it should be mentioned that this does not mean to block the tracker for the next version; this could be misinterpreted. Considering that the tracker gets renamed, I'm not sure what you mean here. +It is encouraged to set the alias field for frequently used bugs. Yes, but please set it to something specific enough; I'm tired of searching for a random word and get into one or another old bug. Most likely candidates are the trackers.
Re: [gentoo-portage-dev] New proposed modular sync system
Am 09.01.2014 18:33, schrieb Brian Dolbec: introduction history of the plugin system I fully agree with idea behind the plugin system. That is, to keep things that are separate, separate. It probably wouldn't have occurred to me to implement it that way (i.e. with auto-detection) but that's just fine. I don't see a problem with reusing that system here. I have started a Proposals sub-page under the Portage project page in the wiki. It has a link to a diagram I made showing how the plug-in system is laid out. This thread will be used to discuss the proposal and the details needed for the module_spec dictionary that is used to provide the manager with the details of what a module provides, options, etc.. The layout makes sense. Except the problems I see with where the modules are installed (see later). Not sure about module_spec yet. [...] stuff about ebuilds installing plugins While I see a valid use case here (especially with your squashfs example), I wonder how that is supposed to work. 1) Where should the ebuild install the plugin? 2) How does portage find those plugins. 3) How does portage's behavior to run with the currently selected python version, mix with the fact that ebuilds usually install only for some set of python versions? (especially python 2 vs 3). This system would also make it possible to modify emerge-s cli to accept target repos to sync. and not any others also installed. Similarly to layman -s some-overlay, emerge --sync some-overlay emerge --sync some-overlay already works. P.S. sorry for such a long email
Re: [gentoo-portage-dev] Helping out
Am 06.01.2014 21:46, schrieb Jesus Rivero (Neurogeek): Hi all, WRT dolsen's email to -dev: ...Those interested in joining, please subscribe to the gentoo-portage-dev list and send an email stating what you can offer and any ideas you have to improve portage's code base. I'd like to help out. I'm somewhat familiar with portage internals and I know Python. Let me know what do you need from me. I tried to compile a list of bugs for people to get started. If you aren't interested in any of those, feel free to roam the bugs assigned to dev-portage. There are more than enough. Bug 490358 - sys-apps/portage: when using git repo directly, version should include git sha1 (edit) Bug 490896 - make.conf `source`: support expanding variables Bug 489088 - sys-apps/portage-2.2.7: FEATURES='cgroup ipc-sandbox network-sandbox' do not fail gracefully Somewhat larger project that needs discussion with the team: Bug 478384 - sys-apps/portage: repos.conf: support tarball (emerge-webrsync) Split this bug in small self contained tasks and file a bug for each task: Bug 480190 - implement GLEP-58 to 61 Cheers, -- Jesus Rivero (Neurogeek) Gentoo Developer
Re: [gentoo-portage-dev] Bugzilla workflow
Here's a new version of the workflow that tries to incorporate all the comments. This should go into the DEVELOPING file once everyone agrees. Bugzilla There always exists a tracker bug, named: [Tracker] sys-apps/portage-next version. This bug is renamed from X.Y.Z to X.Y.Z+1 after a release, until it gets closed when Y changes and a new one is opened. Whenever a commit for a specific bug is made to the git repo, the corresponding bug gets changed in the following ways: * InVCS is added to Keywords * The bug is marked as blocking the tracker for the next version * A comment is added saying: This is fixed in git: url to commit (note that the bug stays open) After a release all open bugs blocking the tracker are closed with the comment This is fixed in version.. For individual open bugs it is encouraged to set UNCONFIRMED, CONFIRMED or IN_PROGESS as appropriate. There are a number of bugs named [TRACKER] * that collect bugs for specific topics. Confirmed bugs should be marked as blocking these tracker bugs if appropriate. It is encouraged to set the alias field for frequently used bugs.
Re: [gentoo-portage-dev] [PATCH 1/2] ResolverPlayground: Write layout.conf if needed
Am 06.01.2014 22:38, schrieb Mike Frysinger:
On Monday 06 January 2014 16:04:16 sebastianlut...@gmx.de wrote:
+f = open(layout_conf_path, w) +
f.write(masters =
test_repo\n) + f.close()
please use with in your code with open(layout_conf_path, 'w') as
f: f.write('masters = test_repo\n') -mike
Arfrever pointed this out on IRC and I already send a new version of
this patch that uses with.
Re: [gentoo-portage-dev] [PATCH 3/3] mkrelease: add a --runtests shortcut
Am 05.01.2014 16:50, schrieb Mike Frysinger:
This automates the release+test cycle a bit so people don't have to do
it all by hand.
---
DEVELOPING | 7 ++-
mkrelease.sh | 15 +--
2 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/DEVELOPING b/DEVELOPING
index 7aac81b..b704a50 100644
--- a/DEVELOPING
+++ b/DEVELOPING
@@ -167,11 +167,8 @@ Releases
First create a git tag for this release:
git tag v2.2.8
-Then create the tarball:
- ./mkrelease.sh --changelog-rev v2.2.7 --tag 2.2.8
-
-Unpack the tarball and run tests:
- ./runtests.sh --python-versions=supported
+Then create the tarball and run the tests:
+ ./mkrelease.sh --changelog-rev v2.2.7 --tag --runtests 2.2.8
Shouldn't this be:
./mkrelease.sh --changelog-rev v2.2.7 --tag 2.2.8 --runtests 2.2.8
?
Make sure you have all supported python versions installed first
(see PYTHON_SUPPORTED_VERSIONS in runtests.sh).
diff --git a/mkrelease.sh b/mkrelease.sh
index 50bdb3c..334b4fb 100755
--- a/mkrelease.sh
+++ b/mkrelease.sh
@@ -8,9 +8,10 @@ BRANCH=${BRANCH:-master}
USE_TAG=false
CHANGELOG_REVISION=
UPLOAD_LOCATION=
+RUNTESTS=false
usage() {
- echo Usage: ${0##*/} [--changelog-rev tree-ish] [-t|--tag]
[-u|--upload location] version
+ echo Usage: ${0##*/} [--changelog-rev tree-ish] [-t|--tag]
[-u|--upload location] [--runtests] version
exit ${1:-0}
}
@@ -19,7 +20,7 @@ die() {
usage 1
}
-ARGS=$(getopt -o htu: --long help,changelog-rev:,tag,upload: \
+ARGS=$(getopt -o htu: --long help,changelog-rev:,runtests,tag,upload: \
-n ${0##*/} -- $@)
[ $? != 0 ] die initialization error
@@ -42,6 +43,10 @@ while true; do
-h|--help)
usage
;;
+ --runtests)
+ RUNTESTS=true
+ shift
+ ;;
--)
shift
break
@@ -78,6 +83,12 @@ cp -a ${SOURCE_DIR}/{bin,cnf,doc,man,misc,pym}
${RELEASE_DIR}/ || die direc
cp
${SOURCE_DIR}/{DEVELOPING,LICENSE,Makefile,NEWS,README,RELEASE-NOTES,TEST-NOTES}
\
${RELEASE_DIR}/ || die file copy failed
+if [[ ${RUNTESTS} == true ]] ; then
+ pushd ${SOURCE_DIR} /dev/null
+ ./runtests.sh --python-versions=supported || die tests failed
+ popd /dev/null
+fi
+
rm -rf ${SOURCE_DIR} || die directory cleanup failed
echo Setting portage.VERSION
Re: [gentoo-portage-dev] [PATCH 3/3] mkrelease: add a --runtests shortcut
Am 05.01.2014 20:34, schrieb Mike Frysinger: On Sunday 05 January 2014 13:04:04 Sebastian Luther wrote: Am 05.01.2014 16:50, schrieb Mike Frysinger: +Then create the tarball and run the tests: + ./mkrelease.sh --changelog-rev v2.2.7 --tag --runtests 2.2.8 Shouldn't this be: ./mkrelease.sh --changelog-rev v2.2.7 --tag 2.2.8 --runtests 2.2.8 nope. the --tag option doesn't take any arguments. the old mkrelease would silently ignore extra args, but the updated one i pushed will now fail on this. Ok. also, please snip any content you're not replying to. makes it harder to scan for responses when there's a lot of untouched content. -mike Ok.
Re: [gentoo-portage-dev] [PATCH] Document layout.conf more (bug 395359)
Am 05.01.2014 20:31, schrieb Mike Frysinger: extending the example isn't the way to go. see the commit i pushed. if you've got details that i missed, then rebase on top of that and send an update. -mike Yes, your's looks better. One thing about the 'profile-formats' key. portage-1-compat is not a valid value (see _valid_profile_formats variable), but pms is. From the use in other places it seems that portage-1-compat is like portage-1, but warns for incompatibilities with pms. Sebastian
Re: [gentoo-portage-dev] New portage version with latest fixes in git is needed
Am 04.01.2014 16:03, schrieb Mike Frysinger: On Saturday 04 January 2014 09:56:28 Brian Dolbec wrote: On Sat, 2014-01-04 at 15:28 +0100, Pacho Ramos wrote: I think this was pointed some days ago, but would be nice (and needed) to get a newer portage version including fixes from git, some of them important like: https://bugs.gentoo.org/show_bug.cgi?id=490362 Not sure if I can contribute with something on this, but if you thing so, please let me know Yes, I planned to get it out this weekend, if all goes well. There are one or two more patches to apply. Then I'll start packaging it. Is everyone else in agreement? we should document the release process. ideally, it should be as simple as: make dist PV=2.1.8 and even: make distcheck PV=2.1.8 -mike There is mkrelease.sh in the portage repo. The following commands creates a portage-2.2.7.tar.bz2 that's almost identical to the one released by Zac (Changelog differs): $ ./mkrelease.sh --changelog-rev v2.2.6 -t 2.2.7 2.2.7 The first argument states where the Changelog entries should begin (tag v2.2.6 in this case). The second argument is the tag to create the tarball from. The third argument is the new version. I don't know how Zac choose the --changelog-rev parameter. That's why the Changelog is different. For 2.2.8 the process should look like: * Create a tag named v2.2.8. * ./mkrelease.sh --changelog-rev -t 2.2.8 2.2.8 * Upload the tarball somewhere * Copy the ebuild (adjust SRC_URI) Sebastian
Re: [gentoo-portage-dev] New portage version with latest fixes in git is needed
Am 04.01.2014 15:56, schrieb Brian Dolbec: On Sat, 2014-01-04 at 15:28 +0100, Pacho Ramos wrote: Hello I think this was pointed some days ago, but would be nice (and needed) to get a newer portage version including fixes from git, some of them important like: https://bugs.gentoo.org/show_bug.cgi?id=490362 Not sure if I can contribute with something on this, but if you thing so, please let me know Thanks! Yes, I planned to get it out this weekend, if all goes well. There are one or two more patches to apply. Then I'll start packaging it. I'd say, apply only the really necessary patches (i.e. the python-2.6 compat patch) and commit the remaining patches after the release. Is everyone else in agreement? Go for it. Sebastian
Re: [gentoo-portage-dev] [PATCH] Make use of optparse to fix argument parsing for Python 2.6 in bin/chpathtool.py.
Am 29.12.2013 03:21, schrieb Tom Wijsman:
---
bin/chpathtool.py | 45 -
1 file changed, 36 insertions(+), 9 deletions(-)
diff --git a/bin/chpathtool.py b/bin/chpathtool.py
index aa3b7d4..0cb5d64 100755
--- a/bin/chpathtool.py
+++ b/bin/chpathtool.py
@@ -13,6 +13,12 @@ import sys
from portage.util._argparse import ArgumentParser
+# Argument parsing compatibility for Python 2.6 using optparse.
+if sys.hexversion 0x207:
+ from optparse import OptionParser
+
+from optparse import OptionError
+
CONTENT_ENCODING = 'utf_8'
FS_ENCODING = 'utf_8'
@@ -147,15 +153,36 @@ def chpath_inplace_symlink(filename, st, old, new):
def main(argv):
parser = ArgumentParser(description=__doc__)
- parser.add_argument('location', default=None,
- help='root directory (e.g. $D)')
- parser.add_argument('old', default=None,
- help='original build prefix (e.g. /)')
- parser.add_argument('new', default=None,
- help='new install prefix (e.g. $EPREFIX)')
- opts = parser.parse_args(argv)
-
- location, old, new = opts.location, opts.old, opts.new
+ try:
+ parser.add_argument('location', default=None,
+ help='root directory (e.g. $D)')
+ parser.add_argument('old', default=None,
+ help='original build prefix (e.g. /)')
+ parser.add_argument('new', default=None,
+ help='new install prefix (e.g. $EPREFIX)')
+ opts = parser.parse_args(argv)
+
+ location, old, new = opts.location, opts.old, opts.new
+ except OptionError:
+ # Argument parsing compatibility for Python 2.6 using optparse.
+ if sys.hexversion 0x207:
+ parser = OptionParser(description=__doc__,
+ usage=usage: %prog [-h] location old new\n\n
+ \
+ location: root directory (e.g. $D)\n + \
+ old: original build prefix (e.g. /)\n
+ \
+ new: new install prefix (e.g.
$EPREFIX))
+
+ (opts, args) = parser.parse_args()
+
+ if len(args) != 3:
+ parser.print_usage()
+ print(%s: error: expected 3 arguments, got %i
+ % (__file__, len(args)))
+ return
+
+ location, old, new = args[0:3]
+ else:
+ raise
is_text_file = IsTextFile()
Patch looks good.
While I do not really like the approach, I didn't see a better way.
Tweaking portage.util._argparse.ArgumentParser would just mean to
reinvent argparse.
Sebastian
Re: [gentoo-portage-dev] [PATCH] QA warning for files in /var/{cache,lib,lock,run}/ or /run/ (bug 493154)
Am 11.12.2013 08:57, schrieb Mike Frysinger:
On Thursday 05 December 2013 15:57:17 sebastianlut...@gmx.de
wrote:
--- a/bin/misc-functions.sh +++ b/bin/misc-functions.sh @@ -242,6
+242,11 @@ install_qa_check() { [[ -d ${ED}/$x ]] f+= $x\n
done
+# It's ok create these directories, but not to install into
them. (bug 493154)
our style uses #12345 rather than (bug 12345) # It's ok create
these directories, but not to install into them. #493154
+for x in var/cache var/lib var/lock var/run run ; do + [[ -d
${ED}/$x ]] [[ $(find ${ED}/${x} -prune -empty) = ]]
f+= $x\n
the -d check doesn't handle symlinks correctly ... baselayout would
install /var/run - /run, and the -d would deref that to the real
symlink. however, `find` will not descend into the symlink, so we
end up being saved by that.
we have -z for detecting empty output rather than comparing to an
empty string
non-builtin vars should use braces, so that'd be ${x}. i know some
of the code in here doesn't follow that, but we should be marching
in that direction with new code.
My bash knowledge is rather limited. So, thanks for your comments.
if [[ -n $f ]] ; then eqawarn QA Notice: This ebuild installs
into the following
deprecated directories:
this warning is incorrect for these paths. we need to create a new
warning that clearly explains what is going on.
Right, thanks for fixing.
i'll send out a new version. -mike
FWIW, new patch looks fine.
Sebastian
Re: [gentoo-portage-dev] Releasing portage-2.2.8
Am 09.12.2013 16:44, schrieb Brian Dolbec: On Mon, 2013-12-09 at 14:56 +0100, Sebastian Luther wrote: Hi all, I'd like to release a new portage version. You may find the list of fixed bugs on the tracker [1] (those not marked as fixed). Most notable are the slot operator related fixes and the fix for binary packages with respect to use dependencies on USE_EXPAND values. I don't think there's anything that should go into the NEWS file. Any objections? What is then needed is someone to create the tarball using mkrelease.sh and upload it somewhere. The ebuild needs an updated SRC_URI. Any Volunteers? [1] https://bugs.gentoo.org/484436 Yeah, we also need to update the ebuild. 2.2.7 was still using the now deprecated EAPI 2 which was getting the attention from the stabilizing devs. I think it is time to update the ebuild eapi. We can always keep 2.2.7 around for an upgrade path from older eapi's. I haven't looked at the upgrade path for quite a while, but it was usually a pain when portage changed its EAPI or python deps. While it's true that leaving old ebuilds available helps with that, it's still annoying. So if we do this, we should have a real good reason. The council voted that EAPI 1 and 2 are discouraged. I'd argue that minimizing upgrade hassle is more worth than not using a discouraged EAPI. To summarize: leave it as is. Mike, what about Sebastian's last patch about the /var/run and compny. The email I fail to find at the moment. I know there was some additional comments on the bug about /var/lib and one other directory being excluded.
Re: [gentoo-portage-dev] [PATCH] Remove ban on installing *.a and *.la files in / (bug 492542)
Note that I don't have an opinion if that makes sense or not. If you don't agree, then I'd suggest to start a discussion on bugzi [1]. [1] https://bugs.gentoo.org/492542
Re: [gentoo-portage-dev] [PATCH] make.conf.5: Document PYTHON_TARGETS, bug #493180
Should we really document specific USE_EXPAND variables? What about all the others? USE_EXPAND=ABI_MIPS ABI_X86 ALSA_CARDS APACHE2_MODULES APACHE2_MPMS CALLIGRA_FEATURES CAMERAS COLLECTD_PLUGINS CROSSCOMPILE_OPTS CURL_SSL DRACUT_MODULES DVB_CARDS ELIBC ENLIGHTENMENT_MODULES FCDSL_CARDS FFTOOLS FOO2ZJS_DEVICES FRITZCAPI_CARDS GPSD_PROTOCOLS GRUB_PLATFORMS INPUT_DEVICES KERNEL LCD_DEVICES LIBREOFFICE_EXTENSIONS LINGUAS LIRC_DEVICES MONKEYD_PLUGINS NETBEANS_MODULES NGINX_MODULES_HTTP NGINX_MODULES_MAIL OFED_DRIVERS OFFICE_IMPLEMENTATION OPENMPI_FABRICS OPENMPI_OFED_FEATURES OPENMPI_RM PHP_TARGETS PYTHON_SINGLE_TARGET PYTHON_TARGETS QEMU_SOFTMMU_TARGETS QEMU_USER_TARGETS RUBY_TARGETS SANE_BACKENDS USERLAND VIDEO_CARDS VOICEMAIL_STORAGE XFCE_PLUGINS XTABLES_ADDONS
Re: [gentoo-portage-dev] [PATCH] make.conf.5: Document PYTHON_TARGETS, bug #493180
Am 03.12.2013 11:08, schrieb Alexander Berntsen: On 03/12/13 10:46, Sebastian Luther wrote: Should we really document specific USE_EXPAND variables? What about all the others? I think so, yes. [...] I can do this over the weekend. The problem with all these variables is that we don't maintain them. If they disappear, change meaning or new ones are added we constantly have to fix our documentation. The python team's documentation [1] is just fine imo. Maybe this one needs to be more user visible? Maybe by linking it into the handbook (if it isn't already)? [1] http://www.gentoo.org/proj/en/Python/python-r1/user-guide.xml
Re: [gentoo-portage-dev] [PATCH] Fix unnecessary rebuild (bug 487198)
Whoever commits this: Please give the test case a better name like testSlotConflictMixedDependencies.
Re: [gentoo-portage-dev] [PATCH 2/2] Add git-r3 to the list of live eclasses
See bug 489662. https://bugs.gentoo.org/489662
Re: [gentoo-portage-dev] [PATCH] Another slot operator bug (bug 486580, try 2)
Please ignore this one. I reorganized these large duplicated if's as suggested by dol-sen on IRC in a new version of this patch.
Re: [gentoo-portage-dev] [PATCH] egencache: --write-timestamp to create metadata/timestamp.chk
Am 27.11.2013 22:20, schrieb Mike Frysinger:
On Wednesday 27 November 2013 05:31:17 sebastianlut...@gmx.de
wrote:
+if options.write_timestamp: + timestamp_path =
os.path.join(repo_path, metadata, timestamp.chk) + try: +
timestampfile = open(timestamp_path, w) + try: +
timestampfile.write( +
time.strftime(%a, %d %b %Y %H:%M:%S
+,
time.gmtime()))
+timestampfile.write(\n) +
finally: +
timestampfile.close()
use a with block, and merge those writes
with open(timestamp_path, 'w') as f: f.write(time.strftime('%a, %d
%b %Y %H:%M:%S +\n', time.gmtime())) -mike
What if the f.write raises an exception?
[gentoo-portage-dev] Wanted: committer for the portage git repo
Hi everyone, since zmedico seems to be unavailable for now, I started to fix some bugs (see below). The problem is that I need someone with commit rights to the portage git repo, with the time to ack and commit my stuff. Does anyone volunteer to play this role? What is rather critical for now are the patches in the slot_operator_fixes branch in my github repo [1]. They fix: Bug 486580 - sys-apps/portage: seems to rebuild python-exec revdeps 'too early' Bug 491518 - sys-apps/portage-2.2.7 tries to force rebuild llvm on every other package merge Bug 489704 - emerge keeps rebuilding (flags rR) the same files every time it runs Bug 472104 - emerge: Show more info for slot rebuilds might help with: Bug 468486 - 2.2.0_alpha173 - emerge --deep is extremely slow Bug 492564 - portage tries to upgrade more than requested [1] https://github.com/few/fews-portage-branch/tree/slot_operator_fixes Sebastian
Re: [gentoo-dev] Re: RFC: A tiny news item for migrating to libjpeg-turbo
Am 23.04.2012 19:49, schrieb Pacho Ramos: El lun, 23-04-2012 a las 10:17 -0400, Mike Gilbert escribió: On Mon, Apr 23, 2012 at 8:28 AM, Duncan 1i5t5.dun...@cox.net wrote: Samuli Suominen posted on Mon, 23 Apr 2012 14:22:53 +0300 as excerpted: Title: The default JPEG implementation [...] All users are recommended to migrate: # emerge -C media-libs/jpeg:0 # emerge -1 media-libs/libjpeg-turbo That of course leaves the system without a jpeg library between the jpeg unmerge and the completion of the libjpeg-turbo merge. If the build process fails for some reason... There's no way to use portage's automatic block-resolving ability here to avoid that, I take it? This works for me. floppym@naomi ~ % emerge -pv1 -j1 libjpeg-turbo These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild N ] media-libs/libjpeg-turbo-1.2.0-r1 USE=-java -static-libs 0 kB [uninstall ] media-libs/jpeg-8d USE=-static-libs [blocks b ] media-libs/jpeg:0 (media-libs/jpeg:0 is blocking media-libs/libjpeg-turbo-1.2.0-r1) I guess it will work when jpeg is not in world file... maybe people should be told to drop it and, then, let emerge do all the work automatically. There is: # emerge --deselect media-libs/jpeg The problem is that this would also remove things like media-libs/jpeg:62 from the world file.
Re: [gentoo-dev] Re: [gentoo-commits] gentoo-x86 commit in net-im/qutecom: metadata.xml ChangeLog qutecom-2.2_p20110210.ebuild
Original-Nachricht Datum: Fri, 14 Oct 2011 02:01:00 -0400 Von: Mike Frysinger vap...@gentoo.org An: gentoo-dev@lists.gentoo.org Betreff: Re: [gentoo-dev] Re: [gentoo-commits] gentoo-x86 commit in net-im/qutecom: metadata.xml ChangeLog qutecom-2.2_p20110210.ebuild On Thursday 13 October 2011 14:15:54 Sebastian Luther wrote: WARNING: One or more updates have been skipped due to a dependency conflict: dev-python/numpy:0 (dev-python/numpy-1.6.0::gentoo, ebuild scheduled for merge) conflicts with ~dev-python/numpy-1.5.1 required by (sci-mathematics/sage-4.7.1-r2::sage-on-gentoo, installed) dev-python/pexpect:0 (dev-python/pexpect-2.4-r1::sage-on-gentoo, ebuild scheduled for merge) conflicts with ~dev-python/pexpect-2.0 required by (sci-mathematics/sage-4.7.1-r2::sage-on-gentoo, installed) Fact is that sci-mathematics/sage can't be made work without those deps. Fact is that I want this package and couldn't care less if I have the latest version of these other two packages. If in turn I cared for the other two packages, then I would have to remove sage. It's a choice but nothing else. it's a crap choice. users shouldn't have to select from diff sets of packages because some are too broken to work properly. it's a bug and needs to be fixed. Sure, it would be better if it could be fixed. But that's far out of reach at this point (and maybe forever because of the complexity of this thing). People already have to do random choices for some packages, where completely unrelated packages block each other because of file collisions. and it shouldn't require twisting of arms to make people fix their broken packages. also, sci-mathematics/sage is a poor example here. it isn't in the main tree. If you want an example from the tree, use geany and geany-plugins. if people want to add poor packages to their overlays, they're free to. There are two different aspects here. Having strange deps may make it look poor for the packager, but this does not mean that the package is poor from a user pov. After all the primary point of packages being in the tree is be used by the users and not for the packager's fun of maintaining them (even though it helps if it is fun). I agree that those deps should be avoid if possible, but killing an otherwise working package because of them hurts the user more than it helps. -mike Sebastian -- NEU: FreePhone - 0ct/min Handyspartarif mit Geld-zurück-Garantie! Jetzt informieren: http://www.gmx.net/de/go/freephone
Re: [gentoo-dev] Re: [gentoo-commits] gentoo-x86 commit in net-im/qutecom: metadata.xml ChangeLog qutecom-2.2_p20110210.ebuild
Am 13.10.2011 15:13, schrieb Ciaran McCreesh: On Thu, 13 Oct 2011 12:23:07 +0200 Chí-Thanh Christopher Nguyễn chith...@gentoo.org wrote: So qutecom is not broken and needs not be removed as long as linux-headers-2.6.38 is in the tree. Dependencies using , =, =, ~ or =* are broken, except in certain special situations inside ||. Why exactly are they broken? Sure, it will force the user to choose between installing this and having the latest version of the package installed. But I don't see this being a problem. As an example I always get this on world updates: WARNING: One or more updates have been skipped due to a dependency conflict: dev-python/numpy:0 (dev-python/numpy-1.6.0::gentoo, ebuild scheduled for merge) conflicts with ~dev-python/numpy-1.5.1 required by (sci-mathematics/sage-4.7.1-r2::sage-on-gentoo, installed) dev-python/pexpect:0 (dev-python/pexpect-2.4-r1::sage-on-gentoo, ebuild scheduled for merge) conflicts with ~dev-python/pexpect-2.0 required by (sci-mathematics/sage-4.7.1-r2::sage-on-gentoo, installed) Fact is that sci-mathematics/sage can't be made work without those deps. Fact is that I want this package and couldn't care less if I have the latest version of these other two packages. If in turn I cared for the other two packages, then I would have to remove sage. It's a choice but nothing else. And before some asks, no portage wouldn't break this dep these days: # emerge -1u --ignore-default-opts -vtp pexpect These are the packages that would be merged, in reverse order: Calculating dependencies... done! Total: 0 packages, Size of downloads: 0 kB WARNING: One or more updates have been skipped due to a dependency conflict: dev-python/pexpect:0 (dev-python/pexpect-2.4-r1::sage-on-gentoo, ebuild scheduled for merge) conflicts with ~dev-python/pexpect-2.0 required by (sci-mathematics/sage-notebook-0.8.19-r1::sage-on-gentoo, installed) Sebastian
Re: [gentoo-dev] What are || ( ) dependencies?
Am 17.12.2010 16:25, schrieb Ciaran McCreesh: So would anyone be especially opposed to making best leftmost an explicit requirement, enforced by repoman where possible (at least for the = / case)? Why can't the PM handle = / cases itself? Sebastian
Re: [gentoo-dev] What are || ( ) dependencies?
Am 17.12.2010 17:37, schrieb Ciaran McCreesh: On Fri, 17 Dec 2010 17:27:05 +0100 Sebastian Luther sebastianlut...@gmx.de wrote: Am 17.12.2010 16:25, schrieb Ciaran McCreesh: So would anyone be especially opposed to making best leftmost an explicit requirement, enforced by repoman where possible (at least for the = / case)? Why can't the PM handle = / cases itself? Because things are almost never as simple as 'just' = / . You can add in clever trickery to deal with very specific cases, but the second someone throws things off by adding in a use dependency or a third package, things get weird. I thought we were talking about the simplest case here, that is a list of atoms for the same cat/pkg. Consider a variation on the original case: || ( a-2 =a-2[x] ) where the user has specified -x for a. What should happen then? What about || ( a-2[x] b =a-2 ) ? Should that be rewritten in the same way? What about || ( a-2[x] ( =a-2 b ) ) ? Should the package mangler be clever enough to figure that one out too? What if b isn't already installed there? What would repoman enforce here? Which is really the problem: clever heuristics get extremely complicated very quickly, and they're never enough. Agreed. Sebastian
Re: [gentoo-portage-dev] [RFC] Store [,R,P]DEPEND with unevaluated use conditionals in vdb
Am 24.04.2010 13:32, schrieb Gentoo: On Fri, 2010-04-23 at 22:31 -0700, Zac Medico wrote: On 04/23/2010 05:43 AM, Sebastian Luther wrote: Someone might come up with some logic to detect new use flags in *DEPEND, but this looks like a hack to me. It doesn't seem too bad to me. It doesn't work, because it's not guaranteed, that only use flag from IUSE are used in use conditionals. That means you can't do it reliably without the unevaluated value. The clean solution is to store the unevaluated string. Do you want to do this for $PKGDIR/Packages as well? We've always evaluated USE conditionals in there since we were copying the behavior of the older genpkgindex tool and that's how it behaved. We should do it there too for the same reason as for storing it in the vdb. Never heard of that tool, but anyone handling portage's binpkgs should use the portage api which provides an easy way to evaluate the use conditionals. Also note that if we want to rely on having unevaluated strings then we'll probably want to try to get alternative package managers to behave the same way (maybe specify it in PMS). The vdb isn't covered by PMS. Paludis stores the unevaluated value, pkgcore stores the evaluated value. Question is: Does anyone have a good argument to not use the old behavior again? space and ease of parsing for minimal pkg mergers. Minimal mergers have to handle it anyway, since this has been the old behavior until some weeks ago. For portage API users, the difference is a (rather long) one liner. What do you mean with space? Sebastian [1] commit e6be6590e99522f9be69e2af8eff87919d9bf31f on 2010-02-14 I think we'll have to handle the evaluated strings anyway since this code has already been released and stabilized in portage-2.1.8.x, and USE conditionals have been evaluate in $PKGDIR/Packages for even longer. Because of this, I see little or no benefit in changing it back to unevaluated strings at this point. Good. Thanks for not reverting back to those old behaviors. And the new use case isn't of any relevance? As a compromise: What about storing both values?
[gentoo-portage-dev] [RFC] Store [,R,P]DEPEND with unevaluated use conditionals in vdb
Hi everyone! Some weeks ago [1] portage changed its behavior regarding what gets stored as *DEPEND in the vdb. The curret behavior is to store the *DEPEND variables with evaluated use conditionals. That is, if the package has DEPEND=foo? ( cat/pkg2 ), either DEPEND=cat/pkg2 or DEPEND= end up in the vdb, depending on the state of the use flag 'foo'. The old behavior was to store the unevaluated string. In the example above DEPEND=foo? ( cat/pkg2 ) would have been stored in the vdb. I propose to revert back to the original behavior. What is the proposed change good for? I see two reason. The first one more ideological and reads: Don't throw away information for no good reason. I say for no good reason, because the evaluated value is easy to compute for users of the portage api, where the unevaluated value is lost. The second reason is that I want it to detect cases where the *DEPEND variables have been changed without rev bump. To do this I want to compare the *DEPEND string of the ebuild and the installed package. Why can't you use the evaluated *DEPEND string? Form time to time ebuilds get use flags added to IUSE without rev bump. If the new use flags are used in *DEPEND variables and I evaluate the use conditionals based on the enabled use flags of the installed package, I'll always treat the newly added use flag as disabled. The problem is that this might be wrong. Consider this example: installed package: IUSE= RDEPEND= ebuild: IUSE=some_feature RDEPEND=some_feature? ( cat/pkg2 ) Assume the RDEPEND of the installed package is wrong, that is it should have had cat/pkg2 in RDEPEND. After the maintainer noticed that, he added a flag to disable the feature that needs cat/pkg2. Since some_feature will be treated as disabled when the use conditionals in the ebuild's RDEPEND are evaluated, RDEPEND will look unchanged. That means portage won't pick up the change when it should. Someone might come up with some logic to detect new use flags in *DEPEND, but this looks like a hack to me. The clean solution is to store the unevaluated string. Question is: Does anyone have a good argument to not use the old behavior again? Sebastian [1] commit e6be6590e99522f9be69e2af8eff87919d9bf31f on 2010-02-14
