Re: [gentoo-portage-dev] erroneous behavior in 2-style USE dependencies?
Le 17/06/2020 à 08:15, Michał Górny a écrit : On Tue, 2020-06-16 at 23:07 +, Michael Lienhardt wrote: Dear all, My bad for not noticing it sooner, but when there is a dependency like ">=sys-fs/udev-208-r1:0/0[static-libs?]" (that occurs in virtual/libgudev-215-r3), since 'static-libs' is not a use flags of sys-fs/udev-242, that cpv is silently not considered during dependency solving by emerge. However, the PMS states: - it is an error for a use dependency to be applied to an ebuild which does not have the flag in question in IUSE_REFERENCEABLE This is a bit like undefined behavior. PMS says such a thing shouldn't happen (i.e. the ebuild is wrong) but does not force specific error handling. You could reject the ebuild entirely or reject dependencies that don't have the flag in question (even if it's disabled). You could also pretend it's 'static-libs(-)?' but that would be bad if the default was supposed to be '(+)'. Indeed. It's true that when I read "error" I understand "something that never occur in a distributed gentoo repository". This is related to the tool I'm working on: should my tool allow this behavior, or fail like it is currently doing (I guess the former)? That depends on what the tool is doing. I suppose you probably don't need very strict behavior there. Indeed, I don't need a strict behavior, but since I saw an ambiguity between the PMS and emerge, I went to check with you if this ambiguity was intentional, and found out along the way how to deal with this situation in my tool. My tool is still the SAT-based dependency analysis you don't really believe in :p. It's going forward slowly but (among other things) thanks to the help of you all, I got a paper accepted to a top software engineering conference. So, thanks! Of course, my final goal is to have the tool fully functional (it will the subject of two other papers -- there's a lot to say on how to deal with gentoo). Since the current gentoo repo includes "erroneous" 2-style USE dependency, I will change my tool's current behavior (raising an error) to the "no match with warning" that Zac proposed (which seems the safest approach to take in the current situation). Best, Michael
Re: [gentoo-portage-dev] erroneous behavior in 2-style USE dependencies?
Le 17/06/2020 à 10:35, Ulrich Mueller a écrit : On Wed, 17 Jun 2020, Michael Lienhardt wrote: But maybe, "error" here in the PMS mean "the cpvs without the use flag does not match that dependency and a warning should be raised to improve compatibility in the future". In that case, it would be clearer for me to change 'error' in the PMS into something like "results in a no match, IMHO we cannot assume that. If the flag is not in the dependency's IUSE_EFFECTIVE then behaviour is undefined. Fair enough. However, currently the PMS says it is an error, not an undefined behavior. but should be avoided". That way, it is explicitly stated that missing use flags for a 2-style USE dependency is accepted (which is the current behavior of emerge) but frown upon, without forcing any specific error handling, like Michał accurately pointed out. The real problem is that we don't have a good procedure for removing flags from ebuilds with reverse (2-style) use dependencies. (And even with 4-style use dependencies the problem remains that one cannot know in advance whether removal of the flag implies that the feature is now unconditionally enabled, or that it is disabled.) Indeed. This is outside the scope of my original question, but intuitively, I would imagine that the devs should know why they remove a use flag. It's just an idea, but I see two possibilities. 1. either the change is temporary: in that case, they could keep the use flag and set its value in REQUIRED_USE during that period 2. either the change is durable: in that case, it is still possible to keep the use flag (while still setting its value in REQUIRED_USE) during a period of time during which it is possible to update the dependencies toward that package (the use flag would become deprecated before being removed). That way, enforcing the error described in the PMS would be telling to the devs that they didn't update their dependencies during the transition period. Best, Michael
Re: [gentoo-portage-dev] erroneous behavior in 2-style USE dependencies?
On 6/17/20 1:25 AM, Zac Medico wrote: > On 6/16/20 7:47 PM, Michael Lienhardt wrote: >> >> >> On 6/16/20 11:59 PM, Zac Medico wrote: >>> On 6/16/20 6:38 PM, Michael Lienhardt wrote: >>>> With the first version of DEPEND, emerge succeed: >>>> # emerge -pv app-misc/dummy-master >>>> >>>> These are the packages that would be merged, in order: >>>> >>>> Calculating dependencies... done! >>>> [ebuild N ] app-misc/dummy-slave-2::gentoo USE="-static-libs" 0 KiB >>>> [ebuild N ] app-misc/dummy-master-1::gentoo USE="-static-libs" 0 KiB >>> >>> This success is expected, yes? Do you suggest to change the behavior >>> somehow? >> >> The way I interpret the PMS, this success is not expected: >> the atom ">=app-misc/dummy-slave-1" matches the cpv >> "app-misc/dummy-slave-1" which does not contains the use flag 'static-libs', >> and thus I expected a 'missing use flag' error. > > For this calculation, it would be a waste of time to read the IUSE > metadata for app-misc/dummy-slave-1, since app-misc/dummy-slave-2 is the > highest available version. Good point. I changed the version of app-misc/dummy-slave-1 into app-misc/dummy-slave-3 (so now the higher version is the one without the 'static-libs' use flag), and still no error/warning message. > I hope that PMS does not specify that package managers must read IUSE > metadata for irrelevant package versions! I think there is indeed an ambiguity there: Section 8.3 of the PMS states when a cpv matches an atom, but does not say which cpvs should be tested against an atom during dependency analysis. This is where my interpretation was maybe wrong: when I see "error" in Section 8.3.4 I understand "all cpv matching an atom with this 2-style USE dependency *must* have the use flag declared, otherwise the .ebuild should be considered erroneous" (I have a strong notion of error). I thus thought that all .ebuilds in the distributed repos were checked (before distribution -- not by emerge) against that error. But maybe, "error" here in the PMS mean "the cpvs without the use flag does not match that dependency and a warning should be raised to improve compatibility in the future". In that case, it would be clearer for me to change 'error' in the PMS into something like "results in a no match, but should be avoided". That way, it is explicitly stated that missing use flags for a 2-style USE dependency is accepted (which is the current behavior of emerge) but frown upon, without forcing any specific error handling, like Michał accurately pointed out. >> I'm not suggesting to change the behavior of emerge, I'm saying that: >> - the way I read the PMS, I expect behavior A, but in practice, I see >> behavior B. >> - what does the portage devs / PMS gurus think about that? >> - is my understanding of the PMS wrong, and it actually says "behavior B >> is expected"? >> - if yes, where did I fail in my understanding? >> - if no, should emerge or the PMS be updated so they both describe the >> same behavior? >> - I will implement your ruling in my tool, which I try to match as closely >> as possible to the PMS > > In this context I think the spirit of what PMS says is that the package > manager should emit some kind of warning message if it finds missing > IUSE. Now, in the example above, if the package manager has no reason to > examine the IUSE metadata of app-misc/dummy-slave-1 because > app-misc/dummy-slave-2 is the highest available version, then there's no > opportunity for it to emit a warning message. >From what I've seen now, emerge considers a 2-style USE dependency error as a >"no match without warning message", and fails with my second version of DEPEND >because there are no .ebuild matching the dependency: the error message it >issues only describes why there is no solution, it is not a warning about an >erroneous dependency. Best, Michael
Re: [gentoo-portage-dev] erroneous behavior in 2-style USE dependencies?
On 6/16/20 11:59 PM, Zac Medico wrote: > On 6/16/20 6:38 PM, Michael Lienhardt wrote: >> With the first version of DEPEND, emerge succeed: >> # emerge -pv app-misc/dummy-master >> >> These are the packages that would be merged, in order: >> >> Calculating dependencies... done! >> [ebuild N ] app-misc/dummy-slave-2::gentoo USE="-static-libs" 0 KiB >> [ebuild N ] app-misc/dummy-master-1::gentoo USE="-static-libs" 0 KiB > > This success is expected, yes? Do you suggest to change the behavior > somehow? The way I interpret the PMS, this success is not expected: the atom ">=app-misc/dummy-slave-1" matches the cpv "app-misc/dummy-slave-1" which does not contains the use flag 'static-libs', and thus I expected a 'missing use flag' error. I'm not suggesting to change the behavior of emerge, I'm saying that: - the way I read the PMS, I expect behavior A, but in practice, I see behavior B. - what does the portage devs / PMS gurus think about that? - is my understanding of the PMS wrong, and it actually says "behavior B is expected"? - if yes, where did I fail in my understanding? - if no, should emerge or the PMS be updated so they both describe the same behavior? - I will implement your ruling in my tool, which I try to match as closely as possible to the PMS >> With the second version of DEPEND, emerge fails: >> # emerge -pv app-misc/dummy-master >> >> These are the packages that would be merged, in order: >> >> Calculating dependencies... done! >> >> emerge: there are no ebuilds built with USE flags to satisfy >> "=app-misc/dummy-slave-1[static-libs?]". >> !!! One of the following packages is required to complete your request: >> - app-misc/dummy-slave-1::gentoo (Missing IUSE: static-libs) >> (dependency required by "app-misc/dummy-master-1::gentoo" [ebuild]) >> (dependency required by "app-misc/dummy-master" [argument]) > > This failure is expected, yes? Do you suggest to change the behavior > somehow? The way I interpret the PMS, this failure is expected. I'm sorry if I'm not always clear, I try to be, and many thanks to take the time to answer my (unexpected and strange) questions. Best, Michael
Re: [gentoo-portage-dev] erroneous behavior in 2-style USE dependencies?
On 6/16/20 9:31 PM, Zac Medico wrote:
> On 6/16/20 11:07 PM, Michael Lienhardt wrote:
>> I'm sorry, my client didn't allow to send plain text email anymore...
>>
>> So, here is my original email.
>>
>> Dear all,
>>
>> My bad for not noticing it sooner, but when there is a dependency like
>> ">=sys-fs/udev-208-r1:0/0[static-libs?]" (that occurs in
>> virtual/libgudev-215-r3),
>> since 'static-libs' is not a use flags of sys-fs/udev-242, that cpv is
>> silently not considered during dependency solving by emerge.
>> However, the PMS states:
>> - it is an error for a use dependency to be applied to an ebuild which does
>> not have the flag in question in IUSE_REFERENCEABLE
>> - For EAPIs listed in table 5.4 as not supporting profile defined IUSE
>> injection, IUSE_REFERENCEABLE is equal to the calculated IUSE value. For
>> EAPIs where profile defined IUSE injection is supported, IUSE_REFERENCEABLE
>> is equal to IUSE_EFFECTIVE
>> And 'static-libs' is not in the IUSE_EFFECTIVE of sys-fs/udev-242 (that
>> ebuild has EAPI=6).
>> So it seems to me that this current behavior of emerge should be considered
>> an error, no? Or the PMS should be updated?
>>
>> This is related to the tool I'm working on: should my tool allow this
>> behavior, or fail like it is currently doing (I guess the former)?
> It's valid as a 4-style dependency with use-dep-defaults.
I know. Except that in virtual/libgudev-215-r3.ebuild we have in the DEPEND:
>=sys-fs/udev-208-r1:0/0[${MULTILIB_USEDEP},gudev(-),introspection(-)?,static-libs?]
It is a 2-style dependency (following the PMS).
I checked with 3 dummy packages
1. app-misc/dummy-master-1
EAPI=6
SLOT=0
KEYWORDS="amd64"
IUSE="static-libs"
DEPEND=">=app-misc/dummy-slave-1[static-libs?]"
#DEPEND="=app-misc/dummy-slave-1[static-libs?]"
2. app-misc/dummy-slave-1
EAPI=6
SLOT=0
KEYWORDS="amd64"
IUSE=""
DEPEND=""
3. app-misc/dummy-slave-2
EAPI=6
SLOT=0
KEYWORDS="amd64"
IUSE="static-libs"
DEPEND=""
With the first version of DEPEND, emerge succeed:
# emerge -pv app-misc/dummy-master
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild N ] app-misc/dummy-slave-2::gentoo USE="-static-libs" 0 KiB
[ebuild N ] app-misc/dummy-master-1::gentoo USE="-static-libs" 0 KiB
With the second version of DEPEND, emerge fails:
# emerge -pv app-misc/dummy-master
These are the packages that would be merged, in order:
Calculating dependencies... done!
emerge: there are no ebuilds built with USE flags to satisfy
"=app-misc/dummy-slave-1[static-libs?]".
!!! One of the following packages is required to complete your request:
- app-misc/dummy-slave-1::gentoo (Missing IUSE: static-libs)
(dependency required by "app-misc/dummy-master-1::gentoo" [ebuild])
(dependency required by "app-misc/dummy-master" [argument])
Michael
Re: [gentoo-portage-dev] erroneous behavior in 2-style USE dependencies?
I'm sorry, my client didn't allow to send plain text email anymore... So, here is my original email. Dear all, My bad for not noticing it sooner, but when there is a dependency like ">=sys-fs/udev-208-r1:0/0[static-libs?]" (that occurs in virtual/libgudev-215-r3), since 'static-libs' is not a use flags of sys-fs/udev-242, that cpv is silently not considered during dependency solving by emerge. However, the PMS states: - it is an error for a use dependency to be applied to an ebuild which does not have the flag in question in IUSE_REFERENCEABLE - For EAPIs listed in table 5.4 as not supporting profile defined IUSE injection, IUSE_REFERENCEABLE is equal to the calculated IUSE value. For EAPIs where profile defined IUSE injection is supported, IUSE_REFERENCEABLE is equal to IUSE_EFFECTIVE And 'static-libs' is not in the IUSE_EFFECTIVE of sys-fs/udev-242 (that ebuild has EAPI=6). So it seems to me that this current behavior of emerge should be considered an error, no? Or the PMS should be updated? This is related to the tool I'm working on: should my tool allow this behavior, or fail like it is currently doing (I guess the former)? Best, Michael On 6/16/20 7:42 PM, Brian Dolbec wrote: > > Please do NOT send html emails. text only please >
Re: [gentoo-portage-dev] precisions on installed packages' dependencies
- Alec Warner a écrit : > Sorry I'm being overly academic. My concern earlier is that you mentioned a > goal of "never breaking installed packages' which I found to be a fairly > audacious goal. The idea is that we should build tools that achieve this > practically (e.g. via heuristics such as := slot operators) while > understanding that the complexities of application deploys are legion and > the tool will never handle them all. So the goal of never breaking them is > more an idealistic goal rather than a practical reality. I agree. I started this discussion because I thought that the content of the /var/db/pkg/* folder was not enough to keep the dependencies. Then Zack and you showed me that I only saw the tip of the iceberg (in my defense, I first wanted to only keep the package's abstract dependencies, not the ones of the actual code. And then the discussion was really interesting). My experience in dependency management is limited to an extended model of debian packages, so my question were (and will be) naive. I understand that with the current status of Portage: - we can consider that the dependencies specified in a package ensure that it can be installed and run - however, package update and package removal is not guaranteed to work. Slot operators is an integrated way to capture some update behaviors, but not all. In general, a dedicated method (like for perl) can be needed. I do believe that never breaking dependencies (at the code level) is a nice idealistic goal. It might not always be possible to achieve it, but you did talk of much work done to do it where it is possible. And, to come back to my previous question, I imagine that the slot operator is an ad-hoc but very useful to avoid dependency breakage when possible. However, this operator looks strange to me: my (naive) intuition to express a trigger for package recompilation would be the other way around (i.e., it is the package that is updated that says what changes, and so, which other packages must be recompiled); like you illustrated with perl, an external tool (possibly different for each package) that gives which packages must be recompiled due to a specific package update. This is why I asked why is the slot operator better as a recompilation trigger compared to other approaches? Is it because it only requires for the developer to add a "=" sign (simplicity is very important)? Or for another reason? Many thanks! Michael
Re: [gentoo-portage-dev] precisions on installed packages' dependencies
- Alec Warner a écrit : > On Tue, Mar 24, 2020 at 11:31 AM wrote: > > However, I still doubt that only storing the soname dependencies is enough. > > Consider package A (that cannot be recompiled) that depends on package B > > which provides lib L.so. > > B is recompiled with different use flags, which put different > > functionalities in L.so. > > The dependencies of A are still satisfied (B is installed, L.so is > > available), but since the content of L.so changed, A cannot execute anymore. > > Hypothetically, can this scenario occur? > > Can this scenario occur in practice? > > Is there a way in emerge/portage to avoid it? > > You have far more experience than me on this, and it would be nice for me > > to know what I'm up against. > > A lot of this has to do with the specifics of how package managers manage > system state, as well as various quirks of subsets of the tree. For > example, a perl upgrade (X->Y) will often break perl modules who expect > perl-X, but get perl-Y. So one fix is to try to keep perl-X installed (so > we SLOT perl and have N perls installed.) Then you need to decide which > version of perl to build things against (X or Y, or both?) We took this > tactic in the python ecosystem; but perl is not slotted in Gentoo, and so > upgrading perl breaks all perl modules. There is a tool > (gentoo-perl-cleaner) that will walk the deptree and fix all of these > broken packages that you run after an upgrade. > > I'm not sure it's strictly avoidable. You could build perl-Y, then rebuild > all perl-modules against perl-Y, then merge the entire result to the > livefs. This will reduce the breakage time but likely not eliminate it; > plus it seems hard to implement in practice without modern filesystem tools > (overlayfs, btrfs, zfs or similar tech to make it atomic.) It also doesn't > account for executing code. What happens to perl-X code that is executing > when you unmerge perl-X? The short answer is that code might break and > 'proper' management means you should restart services after an upgrade > (something Gentoo doesn't typically do; but is common in Debian for > example.) Many thanks for this answer. To sum up what I understood, the problem is not really the dependencies, but which recompilation (and service restart) are triggered with an update. In gentoo, there is the ":=" slot operator (and others similar) in dependencies that trigger the recompilation when the dependency's slot change, but this is the only existing mechanism. And this is why every time perl changes, the compilation of its modules is not triggered and they are most probably broken. Correct? And then, in this context, keeping the installed packages' dependencies consistent is up to debate: packages will get broken in any case... It is clearly impossible to have a tool that automatically detect all implementation dependency breakage. Again, there's something I probably don't see: why was slot operators chosen (among other possibilities) as a mechanism to trigger recompilation? I'm very grateful to you all for the time you take to read and answer my questions. Best, Michael
Re: [gentoo-portage-dev] precisions on installed packages' dependencies
- Zac Medico a écrit : > > The goal of my tool is to have correct manipulation of package > > dependencies, and in particular here, I focus on the packages that are > > installed but not in the portage tree/a local overlay anymore (the problem > > does not occur for other packages). > > It seems that installed packages do not store which are the actual cpv they > > depend on. Correct? > > Right, because unfortunately that's something that changes over time. > > Also, we may not be able to pin it down at any given moment if we have > inconsistent || preferences as described here: > > https://archives.gentoo.org/gentoo-dev/message/550d3859dea6d0fb0b39064628992634 Hmm, I think I see what you mean. Storing the cpvs that was used during solving the package's dependencies would be too restrictive, since two different packages could provide the exact same functionalities/libraries. And so, during a system update, only looking at the cpv dependencies would trigger useless recompilation of the packages that depend on the updated packages. Correct? Btw, my tool's solver does not have the problem discussed in the thread you're mentioning: atom order in lists has no influence in my solver. Would fixing the inconsistent || preferences make storing cpvs for installed packages more realistic? > > Also, I wanted to use the ebuild tool to install/uninstall package, which > > is not possible with this solution apparently. > > Why not? Would the preserve-libs feature solve your problem? ... I'm sorry, I wasn't aware of this feature. It would definitively solve the issue (except, as described in the bug 459038, when external tools remove libs). This discussion is very interesting! If I take this double layer of dependencies, I have to check how this influences the theory underlying my tool. However, I still doubt that only storing the soname dependencies is enough. Consider package A (that cannot be recompiled) that depends on package B which provides lib L.so. B is recompiled with different use flags, which put different functionalities in L.so. The dependencies of A are still satisfied (B is installed, L.so is available), but since the content of L.so changed, A cannot execute anymore. Hypothetically, can this scenario occur? Can this scenario occur in practice? Is there a way in emerge/portage to avoid it? > Well, there are a lot of upgrades that can't be performed without > temporarily breaking something, so the "forbid broken dependencies" idea > doesn't sound feasible to me. Could you tell me about several instances of such needed dependency breakage? You have far more experience than me on this, and it would be nice for me to know what I'm up against. Many thanks! Michael
Re : Re: [gentoo-portage-dev] precisions on installed packages' dependencies
- Zac Medico a écrit : > > 3. before removing a library, "ebuild unmerge" always checks if it is used > > by another package: this means that installed packages' dependencies are > > never broken. > > That's true if the package is removed via emerge --depclean, but emerge > --unmerge does not account for dependencies. > > Also, it's possible for dependencies of installed packages to be > temporarily broken by upgrades. In cases like this, the breakage will > eventually be resolved by a rebuild (which occurs automatically for slot > operator := deps), upgraded, or by emerge --depclean (which removes > unneeded packages). Many thanks for your answers. They made me realize that the problem I'm facing is a bit more tricky than I first quickly though. I'll try to explain the problem, tell me if I'm not clear somewhere. The goal of my tool is to have correct manipulation of package dependencies, and in particular here, I focus on the packages that are installed but not in the portage tree/a local overlay anymore (the problem does not occur for other packages). It seems that installed packages do not store which are the actual cpv they depend on. Correct? Hence, when an installed package cannot be updated/recompiled because it is not in the tree anymore, like you said, its dependencies can be broken (due to the package it depends on being updated). Currently, this issue is circumvented (only using depclean) by keeping the libs: the package's dependencies are broken, but it's ok because it can still run (which, in the end of the day, is what we want). However, from your answer, it seems that this fix is not entirely integrated in the emerge/portage toolchain (like you said, emerge --unmerge removes everything, and emerge -u removes the old libs). To sum up, the problem I'm facing is that with the current way installed packages are managed, we can break dependencies (and the only way to fix them is to remove the installed package with the broken dependencies, that can never be installed again). Hence, for my tool, I have two solutions for that problem: either I forbid for dependencies to ever be broken, or I allow it. Solution 1: forbid broken dependencies. This requires to extend the information stored on installed package with the list of the actual cpvs they depend (or at least the cp+slot, which is enough to get back the cpvs). That way, I can say in the solver "if you want to keep that package, you need to keep these packages as they currently are". However, I have no idea on how to do that, and doing this only for my tool would mean that one cannot switch between emerge (quick) and my tool (correct), which is a feature I think is essential. Do you think adding this new information to installed packages could be integrated into emerge/portage itself? I could work on it (expect question ^^), test it on my prototype, and do a pull request when everything's working. Solution 2: allow broken dependencies. Here, the idea is to use the same fix as is currently done with depclean, but in my tool's planner (i.e., the part that install/unistall the packages) directly. That way, I say in the solver "that installed package has no dependency", but when I upgrade/remove packages, I say "Oh but wait, that other package still need these libs, let's keep them". This solution may not require any change in portage/emerge, but I have no idea on how to know which libs are needed by a package, and how to track these libs owners without looking at every installed package's files (which are stored in the CONTENT file, if I'm not mistaken). Also, I wanted to use the ebuild tool to install/uninstall package, which is not possible with this solution apparently. In case I need to implement this, could you give me some clue on how to achieve it? Among these two solutions, I prefer the first one: we stay at the level of package dependencies (and it looks simpler to implement). However, it is maybe easier/better to use the second approach, I don't know. Do you have some suggestions? Thanks! Michael
[gentoo-portage-dev] precisions on installed packages' dependencies
Dear all, Still in the process of improving my solver (and make it a usable tool), I need to have a better idea on how installed packages should be managed. I didn't find anything on that topic in the PMS (if I've missed it, I'm sorry). Could you confirm/correct my following understanding: 1. installed packages that are still in the portage tree can be unmerged/updated without any restriction (as specified in their .ebuild) 2. installed packages that are not in the portage tree can only be kept as is or unmerged 3. before removing a library, "ebuild unmerge" always checks if it is used by another package: this means that installed packages' dependencies are never broken. Many thanks! Michael
[gentoo-portage-dev] Constraint-Based Dependency Solver: initial results
Dear all, It's possible that the goal of my current work and its possible benefit for the gentoo community are not very clear. In my experience, emerge is a very good tool to install new packages whose use flags have already been configured. However, when the packages are not correctly configured, emerge can guess, without guarantee, some use flags to select/unselect; and unmerging a package seems to be the user's entire responsability. My work has several goals: - provide a correct and complete implementation of a dependency solver for portage that can help debug emerge - since the solver is correct and complete, it would provide a "safe" implementation of unmerge, where missing dependencies would be satisfied by the installation of new packages - provide a rather small code base that is easy to debug and on which it is easy to prototype some tools, like REQUIRED_USE checks, or interactive use flag configuration - be usable, both in term of memory usage and computation time. I finished implementing the solver and did some preliminary benchs (1000 tests where I checked if a random set of packages -- different for each test -- could be installed), including comparison with emerge in "search" modality, i.e., with the options --autounmask y --autounmask-continue y --autounmask-backtrack y Since this is preliminary, I wanted to talk to you about it but I don't have every bugs identified yet. In average, my solver is a bit less than 10 times slower than emerge (not very good, but not bad as it is still far better than a brute force approach, which is more than 100 times slower and takes 3Gb of memory). It is important to note that my solver is not suited for end user usage yet (the answer it gives is correct, but random -- it includes useless packages, useless package removal and old versions), but it is the first tool that I know of that can correctly and completely (modulo bugs ^^) check emerge's behavior. A first result: in more than 25% of the tests that can be installed, emerge fails. Most of these failures come from the fact that even in "search" modality, emerge stops when it sees a REQUIRED_USE that is not correctly configured (I'll post a bug report about it and about the SLOT heuristics in a few days). I still need to look at the other failures to see what caused them. Additionally, it seems that when I do "emerge package1 package2", emerge first solves the dependencies of package1, and then of package2. It seems that when resolving the depenendencies of package2, emerge can end up with a conflict between the choices it made for solving the dependencies of package1 and the requirements of package2. I say "seems" because my tests were automatically done with a rather long list of packages (so other reasons for the observed failures are possible). I'll try to pin down the actual emerge behavior and possibly file a bug report. I am currently porting the tests on docker (to have a safe testing environment) and once that's done, I will be able to give actual bug reports. In the future, I have the following plan: - cleanup the output of the solver, so it wouldn't be random and be usable instead (this is "just" a technical and boring algorithm to implement, but time consuming) - install the configuration computed by my solver (I am still unsure that emerge --nodeps would be flexible enough, and maybe I will have to implement my own planner) - find a correct abstraction of packages, similar to the SLOT heuristics ( https://gitweb.gentoo.org/proj/portage.git/commit/?id=a9064d08ef4c92a5d0d1bfb3dc8a01b7850812b0 ), to improve efficiency - design an interactive package configuration algorithm + UI that would happen during the dependency solving process and really help the user configuring what he wants and let the solver do the rest - start reading portage's bug tracker and continue reading its code - extend pdepa with other kind of relevant analysis All comments/suggestions are welcomed. Best, Michael
Re : Re: [gentoo-portage-dev] Constraint-Based Dependency Solver for Portage: again
- Zac Medico a écrit : > It's capable of considering older versions, but maybe there's some > deficiency in the algorithm. We should analyze a specific example in > order to understand the behavior. > > Maybe you're referring to this code which forces the highest version in > the event of a conflict: > > https://gitweb.gentoo.org/proj/portage.git/commit/?id=a9064d08ef4c92a5d0d1bfb3dc8a01b7850812b0 > Yes, this seems to be the cause of the problem, thank you. For testing, I created two ebuilds (and tested with "emerge -pv --autounmask-backtrack y net-misc/pdepa"): ## net-misc/pdepa-1.0 EAPI=6 KEYWORDS="amd64" SLOT="1" IUSE="feature" REQUIRED_USE="" DEPEND="" ## net-misc/pdepa-2.0 EAPI=6 KEYWORDS="amd64" SLOT="1" IUSE="feature" REQUIRED_USE="^^ ( feature )" # feature is not set => not installable DEPEND="" Like you said, due to SLOT conflict, only net-misc/pdepa-2.0 is tried, and emerge fails. Changing the SLOT of net-misc/pdepa-2.0 to 2 "solves the issue": emerge succeeds and propose to install net-misc/pdepa-1.0 However, this solution is only local: if in net-misc/pdepa-2.0 I replace REQUIRED_USE="" # now the package has no configuration problem DEPEND="!virtual/libc" # but it conflicts with an important library then emerge fails again, saying that virtual/libc blocks net-misc/pdepa-2.0 I don't know how many actual packages cannot be installed due to this optimization. I noticed this behavior in a previous version of the portage tree, when trying to install sys-auth/polkit without configuring it: - old version: REQUIRED_USE="??( systemd consolekit )" - more recent version REQUIRED_USE="^^ ( consolekit, elogind systemd )" In practice however, this was not a problem, as some dependencies of polkit deep in the tree did require one of ( systemd consolekit ) to be set. If you want, I can implement a test that check if this optimization is a problem in practice. Checking the issue shouldn't be too difficult (I think I just need to check that all REQUIRED_USE and *DEPEND are equivalent for a SLOT). Best, Michael
[gentoo-portage-dev] Constraint-Based Dependency Solver for Portage: again
Dear all, A few months ago, I got back to my constraint-based dependency solver for portage, that I had to leave for a while. Thanks to Zac Medico, it is now based on portage itself to query the portage tree, and so the code is far simpler (and far less buggy). It is on github: https://github.com/gzoumix/pdepa I still have some work to do on the implementation, and with some colleagues, we are planning to publish it in a conference, with the related theory. However, to have relevant information to publish, I need your help, if you could answer some questions that will come up during testing. For instance, in all my tests, emerge (during its dependency resolution) always replaces atoms with the latest version of the pc that matches it, even with all possible backtracking options being selected (I noticed this behavior because emerge failed installing a package such that the latest corresponding cpv could be installed, while the previous version could be). Is it really the default behavior of emerge, and if yes, is there a way to make emerge consider all matching cpv for an atom? Thank you! Michael
Re: [gentoo-portage-dev] [PATCH 1/2] portage.package.ebuild.config: Rename iuse_implicit -> iuse_effective
Many thanks. I should definitively read this document, that is far more precise that anything I have found on the wiki or on devmanual. Il 06/02/2018 00:05, Zac Medico ha scritto: On 02/05/2018 02:46 PM, Michael Lienhardt wrote: Is the IUSE_IMPLICIT variable in the make.defaults also changed into IUSE_EFFECTIVE? I'm sorry if this question was already discussed/answered somewhere else. The IUSE_EFFECTIVE variable is generated from IUSE_IMPLICIT and some other variables. It's documented in the "USE and IUSE handling" section here: https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-11900011.1.1
Re: [gentoo-portage-dev] [PATCH 1/2] portage.package.ebuild.config: Rename iuse_implicit -> iuse_effective
Is the IUSE_IMPLICIT variable in the make.defaults also changed into IUSE_EFFECTIVE? I'm sorry if this question was already discussed/answered somewhere else. Michael Il 04/02/2018 14:40, Michał Górny ha scritto: Rename the iuse_implicit variable used in USE_EXPAND handling to iuse_effective, since that is what is actually passed there. Correct naming makes figuring out what the function does much easier. --- pym/portage/package/ebuild/config.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py index 5624e86d3..35cf4f614 100644 --- a/pym/portage/package/ebuild/config.py +++ b/pym/portage/package/ebuild/config.py @@ -1307,13 +1307,13 @@ class config(object): """ def __init__(self, settings, unfiltered_use, - use, usemask, iuse_implicit, + use, usemask, iuse_effective, use_expand_split, use_expand_dict): self._settings = settings self._unfiltered_use = unfiltered_use self._use = use self._usemask = usemask - self._iuse_implicit = iuse_implicit + self._iuse_effective = iuse_effective self._use_expand_split = use_expand_split self._use_expand_dict = use_expand_dict @@ -1331,7 +1331,7 @@ class config(object): if has_wildcard: var_split = [ x for x in var_split if x != "*" ] has_iuse = set() - for x in self._iuse_implicit: + for x in self._iuse_effective: if x[:prefix_len] == prefix: has_iuse.add(x[prefix_len:]) if has_wildcard:
Re: Re : Re: [gentoo-portage-dev] Constraint-Based Dependency Solver for Portage: a prototype
Dear Alexander, Many thanks for the advices. I started few discussions on the forum and will go to FOSDEM. I'll see where it will go. Best, Michael Il 16/12/2017 14:39, Alexander Berntsen ha scritto: On 13/12/17 02:52, michael.lienha...@laposte.net wrote: But maybe there are things we can do to help start a dialog, like: - reaching in other mailing lists I don't think a post to gentoo-dev would be remiss in this case. - posting on a Gentoo forum Always useful, I'm told, though I don't venture there. But that way you're far more likely to engage *users*. - participating in a workshop/conference/other where we could directly meet and discuss with the community FOSDEM and Linux Days are probably the best choices. - or simply starting an informal discussion by email where instead of having to look into the Github repository, you could directly ask me If someone has the time, that'll probably naturally happen through the MLs. Christmas time tends to be peak bikeshedding hours at Gentoo, so maybe cross-post to -dev closer to the holidays?
Re : Re: [gentoo-portage-dev] Constraint-Based Dependency Solver for Portage: a prototype
Dear Alexander, Many thanks for your reply and your encouragements. The point that you raised is very interesting and was partially done in Debian (they defined a wrapper around apt-get instead of refactoring it): http://manpages.ubuntu.com/manpages/zesty/man8/apt-cudf-get.8.html Part of their work was formalized in coq and implemented in OCaml. In our case, we don't have any mechanized formalization of our model (maybe in the future). I too (and my colleagues) hope that someone on the team could have some time to look into our project. But maybe there are things we can do to help start a dialog, like: - reaching in other mailing lists - posting on a Gentoo forum - participating in a workshop/conference/other where we could directly meet and discuss with the community - or simply starting an informal discussion by email where instead of having to look into the Github repository, you could directly ask me Does anyone have suggestions on that topic? Again, many thanks. I really hope that with everyone's feedback, suggestions, and help, we could make something useful from this prototype. Michael Lienhardt PS: I forgot in my previous mail to talk about the other persons involved in this project: - Jacopo Mauro, Post-doc in UiO (Norway), developer of the solver backend - Simone Donetti, Engineer in Unito (Italy), he helped me perform some tests - Ferruccio Damiani (Unito), Einar Broch Johnsen and Ingrid Chieh Yu (UiO), our supervisors
[gentoo-portage-dev] Constraint-Based Dependency Solver for Portage: a prototype
Dear Portage developers, I am a Post-doc in formal methods and software engineering. With my colleagues, we are working on a formal model for software composition, and were looking for a concrete example of such model to motivate and guide our work. I knew portage from using gentoo since 2007, and knew that it is the perfect use case for us. The first result of our work is a prototype for a constraint-based dependency solver for Portage: like the emerge tool, it takes in parameter a list of atoms to install, and computes a full list of packages to install to satisfy the package dependency relation. Up to bugs, this tool is correct and complete: it will always find a solution if it exists, and always tell if there are none. For instance, it successfully computed that gnome-base/gnome cannot be installed by default (on a udev system), but found a solution that replaces sys-fs/eudev by sys-apps/systemd when we allow the tool to change the USE flag selection of the packages. With this prototype, we also compiled (90% of) a documentation on how portage manages package configuration (USE flags declaration, selection, masking, keywording, ...). Link to the prototype: https://github.com/HyVar/gentoo_to_mspl Link to the documentation: https://github.com/HyVar/gentoo_to_mspl/blob/master/PORTAGE.md We would really like to know your opinions, impressions and suggestions about this work. We would also like to know how useful this tool could be for the community: as for now, it is a prototype of a dependency solver (that would definitively need some work to be usable in production), but it also offers the possibility of any kind of formal analysis on the REQUIRED_USE and dependencies in packages, like the one described in https://bugs.gentoo.org/417753 For instance, our tool already checks for obvious reasons (inconsistent REQUIRED_USE or unmet dependencies) causing a package not to be installable. In particular, on the Portage version available in http://www.osboxes.org/gentoo/ , our tool identified 14 packages that could not be installed for these reasons (the full list in in post-scriptum). Additionally, our implementation is based on what I understood of the portage's documentation, which I compiled in the PORTAGE.md document: it would be very helpful if you could point error that I made or subtleties that I didn't understand or missed. Best Regards, Michael Lienhardt PS: list of uninstallable packages: dev-java/jruby-1.7.12 media-video/nvidia-settings-340.58 dev-ruby/bitescript-0.0.9 dev-java/spring-core-3.2.4 app-i18n/ibus-table-code-1.2.0.20100305 dev-ruby/weakling-0.0.4 sci-libs/ogdi-3.1.5-r1 dev-java/jcs-2.0 net-misc/asterisk-rate_engine-0.5.4 games-fps/doom3-mitm-20070129 app-office/impressive-0.10.5 dev-java/spring-aop-3.2.4 dev-ruby/duby-0.0.2-r1 dev-db/mycli-
