Re: [gentoo-dev] What are || ( ) dependencies?
On Sat, 18 Dec 2010 22:22:07 -0800 Zac Medico zmed...@gentoo.org wrote: What about if you decide upon a early on, and then later on something hard-depends upon b? Then you're collapsing the graph too early. =) (speaking as an utter novice) This is the same kind of case as in bug 264434. We solved it in portage by putting || and virtual dependencies on stack, and delaying their evaluation until as late as possible. You may be able to dream up some corner cases where this approach doesn't help, but in practice it seems to help more often than not. That's just another case where a fancy heuristic sometimes gives you better results, but in general doesn't solve the problem at all. If you've got two lots of undecided || ( ) deps, sooner or later you have to decide at least one lot, but you can't correctly make that decision until you've decided the other lot (which of course then can't be decided until you've decided the first...). We're all having to be way too clever here, and it isn't even helping. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] What are || ( ) dependencies?
On Fri, 17 Dec 2010 20:13:55 -0600 Donnie Berkholz dberkh...@gentoo.org wrote: What about if you decide upon a early on, and then later on something hard-depends upon b? Then you're collapsing the graph too early. =) (speaking as an utter novice) Yeah, but unfortunately, there's no way to figure out when too early is. What if it's one of a's dependencies that hard-depends upon b? Until you've decided upon something, you don't know what dependencies are going to be pulled in, so you're left having to make possibly incorrect decisions and then try to undo them later on if possible. Why is this a problem that needs to be resolved at the specification level rather than a difference between implementations? If a package manager is making strange choices, The problem's how you define strange choices. If dependencies aren't listed best-leftmost, every package manager makes strange choices for some combinations. Either this can be fixed by getting developers to always write things best-leftmost, or it can be fixed by mandating specific behaviour for all package managers for || ( ) deps. I'd much rather we did the former. I'd thought people already knew that this was typical behavior of an || group (as per the simple example in ebuild(5)), but you've said differently later in this thread. I certainly wouldn't mind documenting that left is best in cases where none are installed, since this has been expected behavior to those of us who do know. Well, we're running across a fair number of cases along the lines of the libX11 one https://bugs.gentoo.org/show_bug.cgi?id=348518 is what prompted the email -- it turns out vlc is by no means the only package doing this, though, which gives me two options for Paludis: add in a heuristic that gets that very specific case right (and update PMS requiring package manglers to do the same), or get people to list their deps the other way around. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] What are || ( ) dependencies?
On 12/17/2010 06:13 PM, Donnie Berkholz wrote: On 15:25 Fri 17 Dec , Ciaran McCreesh wrote: Things get messier when you've got || ( a b-2.1 ) and b-2.0 is installed and a is not. Should b be upgraded to 2.1, or should a be selected? It depends ... see later. What about if you decide upon a early on, and then later on something hard-depends upon b? Then you're collapsing the graph too early. =) (speaking as an utter novice) This is the same kind of case as in bug 264434. We solved it in portage by putting || and virtual dependencies on stack, and delaying their evaluation until as late as possible. You may be able to dream up some corner cases where this approach doesn't help, but in practice it seems to help more often than not. [1] http://bugs.gentoo.org/show_bug.cgi?id=264434 -- Thanks, Zac
[gentoo-dev] What are || ( ) dependencies?
How should a dependency like || ( a b c ) be interpreted? Traditionally, it's been described as something like: * if a matches an installed package, a * otherwise, if b matches an installed package, b * otherwise, if c matches an installed package, c * otherwise, if a is installable, a * otherwise, if b is installable, b * otherwise, if c is installable, c * otherwise, error Things get messier when you've got || ( a b-2.1 ) and b-2.0 is installed and a is not. Should b be upgraded to 2.1, or should a be selected? What about if you decide upon a early on, and then later on something hard-depends upon b? What about if you've got || ( a[foo] b ) and a[-foo] is installed? As a result of things like this, Portage has had various different sets of heuristics over time, and Paludis has had a different set. This is causing problems. Specifically, consider a dependency like the following, which is present in quite a few ebuilds: || ( x11-libs/libX11-1.3.99.901[xcb] =x11-libs/libX11-1.3.99.901 ) Paludis currently interprets this as I prefer 1.3.99.901, but will also accept =1.3.99.901. In particular, if 1.3.99.901[xcb] is already installed, libX11 won't be upgraded. Some Portage versions also do this, and others don't. There's one easy fix, which solves this and every other possible convoluted case (and some of those can be fairly horrible...): require ebuild developers to always list 'best' things leftmost. So if you're doing || ( =a-2 a-2 ) then you must put the = dep first (even if the = version is masked -- that's guaranteed to work). If the dependency is rewritten like this then all the ambiguity goes away: || ( =x11-libs/libX11-1.3.99.901 x11-libs/libX11-1.3.99.901[xcb] ) The other option is that we mandate a particular selection algorithm for || ( ) dependencies. This is a nuisance, for three reasons: * different Portage versions have done different things * it prevents the package mangler from doing something clever or offering additional features * every algorithm will do the wrong thing for certain combinations of dependencies if not given any preference information So would anyone be especially opposed to making best leftmost an explicit requirement, enforced by repoman where possible (at least for the = / case)? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] What are || ( ) dependencies?
Am 17.12.2010 16:25, schrieb Ciaran McCreesh: So would anyone be especially opposed to making best leftmost an explicit requirement, enforced by repoman where possible (at least for the = / case)? Why can't the PM handle = / cases itself? Sebastian
Re: [gentoo-dev] What are || ( ) dependencies?
On Fri, 17 Dec 2010 17:27:05 +0100 Sebastian Luther sebastianlut...@gmx.de wrote: Am 17.12.2010 16:25, schrieb Ciaran McCreesh: So would anyone be especially opposed to making best leftmost an explicit requirement, enforced by repoman where possible (at least for the = / case)? Why can't the PM handle = / cases itself? Because things are almost never as simple as 'just' = / . You can add in clever trickery to deal with very specific cases, but the second someone throws things off by adding in a use dependency or a third package, things get weird. Consider a variation on the original case: || ( a-2 =a-2[x] ) where the user has specified -x for a. What should happen then? What about || ( a-2[x] b =a-2 ) ? Should that be rewritten in the same way? What about || ( a-2[x] ( =a-2 b ) ) ? Should the package mangler be clever enough to figure that one out too? What if b isn't already installed there? Which is really the problem: clever heuristics get extremely complicated very quickly, and they're never enough. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] What are || ( ) dependencies?
On 12/17/10 4:25 PM, Ciaran McCreesh wrote: As a result of things like this, Portage has had various different sets of heuristics over time, and Paludis has had a different set. Generally it seems fine to have different heuristics (I'll comment on the specific problem below). Paludis currently interprets this as I prefer 1.3.99.901, but will also accept =1.3.99.901. In particular, if 1.3.99.901[xcb] is already installed, libX11 won't be upgraded. Some Portage versions also do this, and others don't. I don't understand why we can't upgrade libX11 in that case. Shouldn't emerge -uDNa world (or its Paludis equivalent) think like this: Okay, I have libX11 installed here, and a more recent version is available. The more recent version satisfies this || () dependency, so just update it. There's one easy fix, which solves this and every other possible convoluted case (and some of those can be fairly horrible...): require ebuild developers to always list 'best' things leftmost. Sounds reasonable. The other option is that we mandate a particular selection algorithm for || ( ) dependencies. Doesn't that somehow contradict the idea that || () lists equivalent dependencies? Maybe we should fix the heuristics. In this specific case, it seems reasonable to still upgrade libX11, right? So would anyone be especially opposed to making best leftmost an explicit requirement, enforced by repoman where possible (at least for the = / case)? I don't think that = / case is enforceable by repoman (i.e. that we always prefer the more recent version of a package). However, saying that the preferred dependency should be listed first sounds reasonable. Paweł signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] What are || ( ) dependencies?
Am 17.12.2010 17:37, schrieb Ciaran McCreesh: On Fri, 17 Dec 2010 17:27:05 +0100 Sebastian Luther sebastianlut...@gmx.de wrote: Am 17.12.2010 16:25, schrieb Ciaran McCreesh: So would anyone be especially opposed to making best leftmost an explicit requirement, enforced by repoman where possible (at least for the = / case)? Why can't the PM handle = / cases itself? Because things are almost never as simple as 'just' = / . You can add in clever trickery to deal with very specific cases, but the second someone throws things off by adding in a use dependency or a third package, things get weird. I thought we were talking about the simplest case here, that is a list of atoms for the same cat/pkg. Consider a variation on the original case: || ( a-2 =a-2[x] ) where the user has specified -x for a. What should happen then? What about || ( a-2[x] b =a-2 ) ? Should that be rewritten in the same way? What about || ( a-2[x] ( =a-2 b ) ) ? Should the package mangler be clever enough to figure that one out too? What if b isn't already installed there? What would repoman enforce here? Which is really the problem: clever heuristics get extremely complicated very quickly, and they're never enough. Agreed. Sebastian
Re: [gentoo-dev] What are || ( ) dependencies?
On Fri, 17 Dec 2010 17:49:22 +0100 Paweł Hajdan, Jr. phajdan...@gentoo.org wrote: Paludis currently interprets this as I prefer 1.3.99.901, but will also accept =1.3.99.901. In particular, if 1.3.99.901[xcb] is already installed, libX11 won't be upgraded. Some Portage versions also do this, and others don't. I don't understand why we can't upgrade libX11 in that case. Shouldn't emerge -uDNa world (or its Paludis equivalent) think like this: Okay, I have libX11 installed here, and a more recent version is available. The more recent version satisfies this || () dependency, so just update it. That's not really how the Paludis resolver thinks. Basically, when it encounters a || ( ) dependency, it selects one of the children based upon a scoring algorithm. At the time it encounters the || ( ) dependency, it doesn't know for sure that it's allowed to upgrade libX11, since a later ebuild might hard-dep upon the lower version. Now, I *could* make it treat the very specific case of || ( a-1[x] =a-1 ) as being a single dep spec like a[either 1[x] or =1], but that doesn't really help as soon as someone does this: || ( a-1[x] ( =a-1 b ) ) which looks a lot like something someone would do. In this specific case, it seems reasonable to still upgrade libX11, right? In this very specific case, rewriting || ( a[stuff] a[otherstuff] ) to be like a[stuff || otherstuff] would solve the problem. But even slightly altering the dependencies would break this. So the question is to what degree a package mangler is required to be clever. So would anyone be especially opposed to making best leftmost an explicit requirement, enforced by repoman where possible (at least for the = / case)? I don't think that = / case is enforceable by repoman (i.e. that we always prefer the more recent version of a package). It's at least detectable... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] What are || ( ) dependencies?
On Fri, 17 Dec 2010 17:56:21 +0100 Sebastian Luther sebastianlut...@gmx.de wrote: Why can't the PM handle = / cases itself? Because things are almost never as simple as 'just' = / . You can add in clever trickery to deal with very specific cases, but the second someone throws things off by adding in a use dependency or a third package, things get weird. I thought we were talking about the simplest case here, that is a list of atoms for the same cat/pkg. Here's the problem: if the package mangler gets this right (which afaik both Portage and Paludis do): || ( a-1 =a-1 ) you might naively expect it to get these right too: || ( a-1 ( =a-1 b ) ) || ( a-1 a-1[x] ) # where a[-x] is the current user configuration which it probably won't, and even if it does, it will get other minor variations on these themes wrong. What about || ( a-2[x] ( =a-2 b ) ) ? Should the package mangler be clever enough to figure that one out too? What if b isn't already installed there? What would repoman enforce here? For that case, it probably couldn't detect it. But getting repoman to yell at people for the simple cases at least would probably help people to learn that they should always go best-leftest. Part of the problem here is that developers don't seem to know about the whole leftmost thing, and the heuristics we've all put in mean they can quite often get away with not knowing it. It also doesn't help that it's not really documented anywhere. It's not in the devmanual, it's not in PMS (and it's hard to put it there, assuming we're not requiring a particular selection algorithm), and the stuff in ebuild(5) for || dependencies is just plain wrong. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] What are || ( ) dependencies?
On 12/17/10 18:09, Ciaran McCreesh wrote: On Fri, 17 Dec 2010 17:56:21 +0100 Sebastian Luther sebastianlut...@gmx.de wrote: Why can't the PM handle = / cases itself? Because things are almost never as simple as 'just' = / . You can add in clever trickery to deal with very specific cases, but the second someone throws things off by adding in a use dependency or a third package, things get weird. I thought we were talking about the simplest case here, that is a list of atoms for the same cat/pkg. Here's the problem: if the package mangler gets this right (which afaik both Portage and Paludis do): || ( a-1 =a-1 ) you might naively expect it to get these right too: || ( a-1 ( =a-1 b ) ) || ( a-1 a-1[x] ) # where a[-x] is the current user configuration which it probably won't, and even if it does, it will get other minor variations on these themes wrong. There are multiple valid solutions. The default heuristic for right seems to be highest version, but other solutions fulfill the dependency specification too. Part of the problem here is that developers don't seem to know about the whole leftmost thing, and the heuristics we've all put in mean they can quite often get away with not knowing it. I'm not sure if I want that enforced, but a suggestion that left-to-right order may be used by the package manager sounds like a reasonable idea. It also doesn't help that it's not really documented anywhere. It's not in the devmanual, it's not in PMS (and it's hard to put it there, assuming we're not requiring a particular selection algorithm), and the stuff in ebuild(5) for || dependencies is just plain wrong. Patches Welcome? ;)
Re: [gentoo-dev] What are || ( ) dependencies?
On 15:25 Fri 17 Dec , Ciaran McCreesh wrote: How should a dependency like || ( a b c ) be interpreted? Traditionally, it's been described as something like: * if a matches an installed package, a * otherwise, if b matches an installed package, b * otherwise, if c matches an installed package, c * otherwise, if a is installable, a * otherwise, if b is installable, b * otherwise, if c is installable, c * otherwise, error Things get messier when you've got || ( a b-2.1 ) and b-2.0 is installed and a is not. Should b be upgraded to 2.1, or should a be selected? It depends ... see later. What about if you decide upon a early on, and then later on something hard-depends upon b? Then you're collapsing the graph too early. =) (speaking as an utter novice) What about if you've got || ( a[foo] b ) and a[-foo] is installed? See later again.. As a result of things like this, Portage has had various different sets of heuristics over time, and Paludis has had a different set. This is causing problems. Specifically, consider a dependency like the following, which is present in quite a few ebuilds: || ( x11-libs/libX11-1.3.99.901[xcb] =x11-libs/libX11-1.3.99.901 ) Paludis currently interprets this as I prefer 1.3.99.901, but will also accept =1.3.99.901. In particular, if 1.3.99.901[xcb] is already installed, libX11 won't be upgraded. Some Portage versions also do this, and others don't. Why is this a problem that needs to be resolved at the specification level rather than a difference between implementations? If a package manager is making strange choices, There's one easy fix, which solves this and every other possible convoluted case (and some of those can be fairly horrible...): require ebuild developers to always list 'best' things leftmost. So if you're doing || ( =a-2 a-2 ) then you must put the = dep first (even if the = version is masked -- that's guaranteed to work). If the dependency is rewritten like this then all the ambiguity goes away: I'd thought people already knew that this was typical behavior of an || group (as per the simple example in ebuild(5)), but you've said differently later in this thread. I certainly wouldn't mind documenting that left is best in cases where none are installed, since this has been expected behavior to those of us who do know. However, that doesn't resolve the case where a package is installed but is either too old or has a mismatched USE flag to the dep. It's not clear to me how this proposal would deal with the system-dependent components. I think whether a deep upgrade is requested (or otherwise directly targeting the dependency for possible upgrading) should impact the choice between = and . If no upgrade is desired, allow the dep to be fulfilled; otherwise attempt an upgrade to the newest version and see if it matches. For the mismatched case, I'd attempt to flip the USE unless that would produce a conflict with another package; in that case, pick the leftmost other fulfiller. || ( =x11-libs/libX11-1.3.99.901 x11-libs/libX11-1.3.99.901[xcb] ) The other option is that we mandate a particular selection algorithm for || ( ) dependencies. This is a nuisance, for three reasons: * different Portage versions have done different things * it prevents the package mangler from doing something clever or offering additional features * every algorithm will do the wrong thing for certain combinations of dependencies if not given any preference information I think this last point is one of the strongest ones. Humans may need to decide what's best for any specific package's dependencies. So would anyone be especially opposed to making best leftmost an explicit requirement, enforced by repoman where possible (at least for the = / case)? Not in the entirely ambiguous case, but things are trickier when one of the packages is installed, even in nonmatching version/USE. -- Thanks, Donnie Donnie Berkholz Sr. Developer, Gentoo Linux Blog: http://dberkholz.wordpress.com pgpJzD1EkNdBv.pgp Description: PGP signature
