Re: [gentoo-dev] The fun of being a PDEPEND
On 08/10/2011 10:19 PM, Fabio Erculiani wrote: On Thu, Aug 11, 2011 at 1:31 AM, Zac Medico zmed...@gentoo.org wrote: On 08/10/2011 02:14 PM, Fabio Erculiani wrote: The problem here is that Portage enforces the same rule by trying to schedule the PDEPEND as soon as possible This behavior was introduced in order to solve bug 180045 [1]. We can accomplish similar results to the ASAP (as soon as possible) behavior in cases like this, if we create a virtual/meta-package that pulls in the circularly dependent packages. We also havo to update the reverse dependencies to refer to the virtual/meta-package. In case of dev-java/jdom and dev-java/jdom-jaxen my idea was to create virtual/jdom (having it to pull in both deps in RDEPEND) and update the reverse dependencies. Waiting to hear back from Java herd (Caster, actually). The ASAP behavior seems relatively optimal, which makes it difficult to argue that ebuild maintainers should have to go to the trouble of creating virtuals and updating reverse dependencies. It seems like your setting up an ongoing conflict with ebuild maintainers if you don't implement the ASAP behavior. Isn't it worth your trouble to implement the ASAP behavior, just to get them out of your hair? OTOH, I think that the gray area should be cleared out by clearly stating what is legal or not in an updated EAPI. Isn't that reasonable? It's already been allowed for years, so a new EAPI would only make sense if your taking away the ASAP behavior, which seems like a step backwards. Given the push-back that you're likely to get from ebuild developers over time, I think you're much better off if you just implement the ASAP behavior. -- Thanks, Zac
Re: [gentoo-dev] The fun of being a PDEPEND
On Wed, 10 Aug 2011 23:31:01 -0700 Zac Medico zmed...@gentoo.org wrote: The ASAP behavior seems relatively optimal Unfortunately, what it *seems* and what it *is* are two entirely different things, and the whole thing breaks down when post-dependent packages have dependencies themselves. If ASAP were specified, you'd be introducing a situation where the set of packages that can be relied upon by a particular package depends upon whether that package is being installed as a target or as a dependency of something else. Mandated ASAP simply isn't logically consistent. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] The fun of being a PDEPEND
On Thu, Aug 11, 2011 at 7:36 AM, Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Wed, 10 Aug 2011 23:14:22 +0200 Fabio Erculiani lx...@gentoo.org wrote: I've intermittently spent my last two days trying to figure out a weird bug on Entropy dependency resolution algorithm (which is actually just a simple topological sorting out of a digraph) You can't use a naive topological sort to do dependency resolution. RDEPEND-RDEPEND cycles are legal and common. Yes, they're legal and common but they can likely generate build failures. Entropy always tried to strictly follow PMS (bugs apart). Given the same document, about PDEPEND it says something like this: there is no guarantee that a PDEPEND gets installed at a certain, specified point during the schedule, if not specifically listed as RDEPEND by some package. The problem here is that Portage enforces the same rule by trying to schedule the PDEPEND as soon as possible, which leads ebuild writers to think to have gotten the deps order right. In simple words, it doesn't seem that Portage itself follows PMS or PMS is detailed enough. Portage *tries*. There's no guarantee that it will succeed. If you need a particular ordering, you can't use PDEPEND. If something's there only because of a PDEPEND, there are absolutely no guarantees whatsoever as to when it will get resolved; PDEPEND imposes absolutely no reliable conditions upon ordering. Any ebuild assuming otherwise should be fixed. Given your definition of PDEPEND, jdom stuff should get fixed. No matter what Portage/Paludis quirks do. Purely as a quality of implementation issue, scheduling a PDEPEND reasonably soon after (or even before) the package requiring it may be a good idea, simply because users may get confused if when they try to install a bunch of things and one of those things gets installed long before related packages. But you can't rely upon that happening. But since this can lead to failures, the correct behaviour must get defined by PMS. Otherwise everybody will go implementing schedulers as they like most. (Incidentally, one could also argue that package manglers should always try to come up with the most perverse legal ordering possible for any situation. That way bugs will be identified much more quickly. Part of the problem with Portage is that it has so many tricks and so little error checking that broken things quite often appear to work.) Bad design is bad design. And there is no excuse. My feeling is that since Portage is actually able to figure out the correct order by just using tricks like the ASAP, and even though PMS doesn't say anything about that, the issue is considered minor. I would rather see PMS clarify the correct behaviour of schedulers with respect to PDEPEND. -- Ciaran McCreesh -- Fabio Erculiani
Re: [gentoo-dev] The fun of being a PDEPEND
On Thu, Aug 11, 2011 at 8:31 AM, Zac Medico zmed...@gentoo.org wrote: The ASAP behavior seems relatively optimal, which makes it difficult to argue that ebuild maintainers should have to go to the trouble of creating virtuals and updating reverse dependencies. Yes it is and I agree, but the point here is that PMS doesn't say anything about it. It seems like your setting up an ongoing conflict with ebuild maintainers if you don't implement the ASAP behavior. Isn't it worth your trouble to implement the ASAP behavior, just to get them out of your hair? No it's not, but I would have the matter clarified first, and perhaps eventually fixed by updating PMS documentation. OTOH, I think that the gray area should be cleared out by clearly stating what is legal or not in an updated EAPI. Isn't that reasonable? It's already been allowed for years, so a new EAPI would only make sense if your taking away the ASAP behavior, which seems like a step backwards. Given the push-back that you're likely to get from ebuild developers over time, I think you're much better off if you just implement the ASAP behavior. I would rather want to see it becoming mandatory by PMS, also. But beside the ASAP, do you agree that there is still a dependency issue? -- Thanks, Zac -- Fabio Erculiani
Re: [gentoo-dev] The fun of being a PDEPEND
On Thu, 11 Aug 2011 08:54:54 +0200 Fabio Erculiani lx...@gentoo.org wrote: I've intermittently spent my last two days trying to figure out a weird bug on Entropy dependency resolution algorithm (which is actually just a simple topological sorting out of a digraph) You can't use a naive topological sort to do dependency resolution. RDEPEND-RDEPEND cycles are legal and common. Yes, they're legal and common but they can likely generate build failures. They're also necessary to support if you want to handle 'system' correctly. Something I made a while back (and being careful to keep X out of system, since usually it's in there): http://dev.exherbo.org/~ciaranm/resolution-neato.png All the orange packages depend upon all the other orange packages. Given your definition of PDEPEND, jdom stuff should get fixed. No matter what Portage/Paludis quirks do. Yup. Purely as a quality of implementation issue, scheduling a PDEPEND reasonably soon after (or even before) the package requiring it may be a good idea, simply because users may get confused if when they try to install a bunch of things and one of those things gets installed long before related packages. But you can't rely upon that happening. But since this can lead to failures, the correct behaviour must get defined by PMS. Otherwise everybody will go implementing schedulers as they like most. There is no single correct behaviour. Any ordering that meets the rules specified in PMS is correct. (Incidentally, one could also argue that package manglers should always try to come up with the most perverse legal ordering possible for any situation. That way bugs will be identified much more quickly. Part of the problem with Portage is that it has so many tricks and so little error checking that broken things quite often appear to work.) Bad design is bad design. And there is no excuse. My feeling is that since Portage is actually able to figure out the correct order by just using tricks like the ASAP, and even though PMS doesn't say anything about that, the issue is considered minor. I would rather see PMS clarify the correct behaviour of schedulers with respect to PDEPEND. Forced ASAP is logically inconsistent. ASAP only *sometimes* works, and it can't be relied upon. If a package only works with ASAP, it's broken, both by the spec and by Portage under certain circumstances. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] The fun of being a PDEPEND
On Thu, 11 Aug 2011 08:59:01 +0200 Fabio Erculiani lx...@gentoo.org wrote: On Thu, Aug 11, 2011 at 8:31 AM, Zac Medico zmed...@gentoo.org wrote: The ASAP behavior seems relatively optimal, which makes it difficult to argue that ebuild maintainers should have to go to the trouble of creating virtuals and updating reverse dependencies. Yes it is and I agree, but the point here is that PMS doesn't say anything about it. ...because there's no guarantee that it will work, and because if it's enforced rather than an optional extra that may be ignored, then there will be no way of implementing a correct dependency resolver. I would rather want to see it becoming mandatory by PMS, also. Not possible. ASAP is a heuristic, not a rule. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] The fun of being a PDEPEND
On Thu, Aug 11, 2011 at 08:54:54AM +0200, Fabio Erculiani wrote: On Thu, Aug 11, 2011 at 7:36 AM, Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: Purely as a quality of implementation issue, scheduling a PDEPEND reasonably soon after (or even before) the package requiring it may be a good idea, simply because users may get confused if when they try to install a bunch of things and one of those things gets installed long before related packages. But you can't rely upon that happening. But since this can lead to failures, the correct behaviour must get defined by PMS. Otherwise everybody will go implementing schedulers as they like most. The behaviour is already defined; PDEPEND will be installed at some point, not guranteed to be immediately, soon after, before, or as the very last step. There is *no* gurantee of ordering for a PDEPEND dep. There's nothing to fix here in PMS; the problem is in ebuilds incorrectly assuming ASAP behaviour. (Incidentally, one could also argue that package manglers should always try to come up with the most perverse legal ordering possible for any situation. That way bugs will be identified much more quickly. Part of the problem with Portage is that it has so many tricks and so little error checking that broken things quite often appear to work.) Bad design is bad design. And there is no excuse. My feeling is that since Portage is actually able to figure out the correct order by just using tricks like the ASAP, and even though PMS doesn't say anything about that, the issue is considered minor. I would rather see PMS clarify the correct behaviour of schedulers with respect to PDEPEND. With respect, incorrect interpretation of the spec is incorrect interpretation of the spec. It's entirely possible, even in ASAP PDEPEND merging, for the pdepend target to wind up very, very far away from what what actually dep'ed it. There is zero difference between that, and arbitrary placement- the ebuild must be written to either RDEP on it if it needs it for runtime, or to PDEP on it and not require it for initial usage. Basically, you're trying to make PDEPEND into a half assed RDEPEND- a try and schedule it ASAP, but you still can't rely on it being available; that's actually _worse_ than the current PDEPEND rules since it gives ebuild authors the notion they can most of the time rely on it being ordered prior. Fix the ebuilds; there isn't anything to fix in the spec here. ~harring
Re: [gentoo-dev] The fun of being a PDEPEND
The case which triggered my attention was actually
app-office/libreoffice with USE=java. pkg_setup (through
java-utils-2.eclass java-pkg_switch-vm [1]) expects to find a
functional JDK environment, even though jdom-jaxen is not required as
RDEPEND by anything inside java eclasses and libreoffice ebuild,
because in fact, it's only declared as PDEPEND (in order to have the
circularity sorted out).
Given this real-world example, the ASAP feature makes possible to have
jdom-jaxen pulled in, in time, even if it's not something that is
required to PMS-compliant PMs.
So? A few options:
1. implement the ASAP feature in Entropy and live with broken dependencies
2. live with broken dependencies
3. Fix the broken dependencies
4. Fix the broken dependencies and have PMS defining rules for
scheduling PDEPENDs.
[1] depend-java-query --get-vm ${JAVA_PKG_NV_DEPEND:-${DEPEND}}
expects to find a sane JDK environment.
--
Fabio Erculiani
http://lxnay.com
Re: [gentoo-dev] The fun of being a PDEPEND
On 08/10/2011 11:59 PM, Fabio Erculiani wrote: I would rather want to see it becoming mandatory by PMS, also. But beside the ASAP, do you agree that there is still a dependency issue? Well, let's just say that it's not practical for me to be bothered by such things. I can't control how ebuild developers write their dependencies, so I try to make the dependency resolver cope as well as possible with whatever input it's given. -- Thanks, Zac
Re: [gentoo-dev] The fun of being a PDEPEND
On 08/11/2011 12:13 AM, Fabio Erculiani wrote: 1. implement the ASAP feature in Entropy and live with broken dependencies 2. live with broken dependencies 3. Fix the broken dependencies 4. Fix the broken dependencies and have PMS defining rules for scheduling PDEPENDs. It's fair enough for you to call them broken dependencies. However, when writing broken dependencies it what comes natural to ebuild developers, is it practical to fight them even though you can write your dependency resolver to cope with it? -- Thanks, Zac
Re: [gentoo-dev] The fun of being a PDEPEND
On Thu, 11 Aug 2011 00:37:18 -0700 Zac Medico zmed...@gentoo.org wrote: It's fair enough for you to call them broken dependencies. However, when writing broken dependencies it what comes natural to ebuild developers, is it practical to fight them even though you can write your dependency resolver to cope with it? You can't write a dependency resolver to cope with it. You can write a dependency resolver that hides the problem some of the time. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] The fun of being a PDEPEND
On Thu, 11 Aug 2011, Fabio Erculiani wrote:
The case which triggered my attention was actually
app-office/libreoffice with USE=java. pkg_setup (through
java-utils-2.eclass java-pkg_switch-vm [1]) expects to find a
functional JDK environment,
Generally, you cannot rely on any dependency (outside of the system
set) being present in pkg_setup:
http://dev.gentoo.org/~ulm/pms/head/pms.html#x1-720008
Ulrich
[1] depend-java-query --get-vm ${JAVA_PKG_NV_DEPEND:-${DEPEND}}
expects to find a sane JDK environment.
Re: [gentoo-dev] The fun of being a PDEPEND
On Thu, Aug 11, 2011 at 10:24 AM, Ulrich Mueller u...@gentoo.org wrote: On Thu, 11 Aug 2011, Fabio Erculiani wrote: Generally, you cannot rely on any dependency (outside of the system set) being present in pkg_setup: http://dev.gentoo.org/~ulm/pms/head/pms.html#x1-720008 Ulrich You may want to comment on bug #378741 -- Fabio Erculiani http://lxnay.com
Re: [gentoo-dev] The fun of being a PDEPEND
On 08/10/2011 02:14 PM, Fabio Erculiani wrote: The problem here is that Portage enforces the same rule by trying to schedule the PDEPEND as soon as possible This behavior was introduced in order to solve bug 180045 [1]. We can accomplish similar results to the ASAP (as soon as possible) behavior in cases like this, if we create a virtual/meta-package that pulls in the circularly dependent packages. We also havo to update the reverse dependencies to refer to the virtual/meta-package. So, the ASAP behavior is really just a convenience so that ebuild maintainers don't have to go to the trouble of creating a virtual and updating all the reverse dependencies. [1] http://bugs.gentoo.org/show_bug.cgi?id=180045 -- Thanks, Zac
Re: [gentoo-dev] The fun of being a PDEPEND
On Thu, Aug 11, 2011 at 1:31 AM, Zac Medico zmed...@gentoo.org wrote: On 08/10/2011 02:14 PM, Fabio Erculiani wrote: The problem here is that Portage enforces the same rule by trying to schedule the PDEPEND as soon as possible This behavior was introduced in order to solve bug 180045 [1]. We can accomplish similar results to the ASAP (as soon as possible) behavior in cases like this, if we create a virtual/meta-package that pulls in the circularly dependent packages. We also havo to update the reverse dependencies to refer to the virtual/meta-package. In case of dev-java/jdom and dev-java/jdom-jaxen my idea was to create virtual/jdom (having it to pull in both deps in RDEPEND) and update the reverse dependencies. Waiting to hear back from Java herd (Caster, actually). OTOH, I think that the gray area should be cleared out by clearly stating what is legal or not in an updated EAPI. Isn't that reasonable? So, the ASAP behavior is really just a convenience so that ebuild maintainers don't have to go to the trouble of creating a virtual and updating all the reverse dependencies. [1] http://bugs.gentoo.org/show_bug.cgi?id=180045 -- Thanks, Zac -- Fabio Erculiani
Re: [gentoo-dev] The fun of being a PDEPEND
On Wed, 10 Aug 2011 23:14:22 +0200 Fabio Erculiani lx...@gentoo.org wrote: I've intermittently spent my last two days trying to figure out a weird bug on Entropy dependency resolution algorithm (which is actually just a simple topological sorting out of a digraph) You can't use a naive topological sort to do dependency resolution. RDEPEND-RDEPEND cycles are legal and common. Entropy always tried to strictly follow PMS (bugs apart). Given the same document, about PDEPEND it says something like this: there is no guarantee that a PDEPEND gets installed at a certain, specified point during the schedule, if not specifically listed as RDEPEND by some package. The problem here is that Portage enforces the same rule by trying to schedule the PDEPEND as soon as possible, which leads ebuild writers to think to have gotten the deps order right. In simple words, it doesn't seem that Portage itself follows PMS or PMS is detailed enough. Portage *tries*. There's no guarantee that it will succeed. If you need a particular ordering, you can't use PDEPEND. If something's there only because of a PDEPEND, there are absolutely no guarantees whatsoever as to when it will get resolved; PDEPEND imposes absolutely no reliable conditions upon ordering. Any ebuild assuming otherwise should be fixed. Purely as a quality of implementation issue, scheduling a PDEPEND reasonably soon after (or even before) the package requiring it may be a good idea, simply because users may get confused if when they try to install a bunch of things and one of those things gets installed long before related packages. But you can't rely upon that happening. (Incidentally, one could also argue that package manglers should always try to come up with the most perverse legal ordering possible for any situation. That way bugs will be identified much more quickly. Part of the problem with Portage is that it has so many tricks and so little error checking that broken things quite often appear to work.) -- Ciaran McCreesh signature.asc Description: PGP signature
