Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-31 Thread Vlastimil Babka
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Kevin F. Quinn wrote:
 On a general note, introducing dynamic dependencies into the depgraph
 worries me, although I'm not sure I can articulate why.

Can you articulate metadata cache? :)
- --
Vlastimil Babka (Caster)
Gentoo/Java
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFwJf8tbrAj05h3oQRAgmCAJ4gYozwZqYL8f2Qi3mLx+cE9G4shwCeNnhM
+n+IOpU9vuEw7g7xkSLHEi0=
=d+dS
-END PGP SIGNATURE-
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-31 Thread Stephen Bennett
On Tue, 30 Jan 2007 17:06:51 +0100
Marius Mauch [EMAIL PROTECTED] wrote:

 The idea is to add a special category (let's call it active for
 now) that has the following properties:
 - this category doesn't exist in portdir or vdb (= no ebuilds)
 - when portage ($pkgmanager) encounters a active/foo atom in a
 dependency string it executes some special code (e.g.
 $PORTDIR/scripts/active-check/foo =active/foo-1) to determine if
 that atom is satisfied

Given that in the general case the package manager can't change the
active provider and will have to bail with an appropriate message that
the user needs to change it themselves, the obvious solution to this is
the previously-discussed-somewhere-I-can't-remember ebuild function,
called on depgraph creation, to check that it will be able to compile
in the current system environment. It's considerably simpler and more
generally useful than subverting DEPEND to add weird special-case hacks
to it.
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-31 Thread Alec Warner
Stephen Bennett wrote:
 On Tue, 30 Jan 2007 17:06:51 +0100
 Marius Mauch [EMAIL PROTECTED] wrote:
 
 The idea is to add a special category (let's call it active for
 now) that has the following properties:
 - this category doesn't exist in portdir or vdb (= no ebuilds)
 - when portage ($pkgmanager) encounters a active/foo atom in a
 dependency string it executes some special code (e.g.
 $PORTDIR/scripts/active-check/foo =active/foo-1) to determine if
 that atom is satisfied
 
 Given that in the general case the package manager can't change the
 active provider and will have to bail with an appropriate message that
 the user needs to change it themselves, the obvious solution to this is
 the previously-discussed-somewhere-I-can't-remember ebuild function,
 called on depgraph creation, to check that it will be able to compile
 in the current system environment. It's considerably simpler and more
 generally useful than subverting DEPEND to add weird special-case hacks
 to it.

No one said subverting DEPEND was necessarily required.

This stuff is essentially another visibility filter.

Think for example along the ACCEPT_RESTRICT lines, but less fugly.

User has FEATURES=sandbox

ebuild has RESTRICT=sandbox

Ebuild is not visible because it is incompatable with current environment.

The only problem here being some visibility filters are 'soft' (like
sandbox and USE vars) and some filters are hard (dependencies).  USE
flags can often be flippy flopped to get a solution (as with sandbox,
which can be selectively turned off).
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-31 Thread Kevin F. Quinn
On Wed, 31 Jan 2007 12:27:10 -0500
Alec Warner [EMAIL PROTECTED] wrote:

  Hmm; one could get the same benefit by introducing a new interface
  (e.g. pkg_env_check()) which is defined to return true if the
  environment is ok, false otherwise (with some text to stdout,
  perhaps). The package manager would then run this function, after
  building the depgraph and finding the candidate packages to merge,
  for each candidate package - if any package fails is env_check,
  none of the packages get emerged.  Note this is then completely
  independent of depgraph creation.
  
  In the 'tr1' example, I'd imagine something like this:
  
  use.local.desc: boost: Use boost library for tr1 rather than gcc's.
  
  ebuild:
  
  ...
  inherit ... toolchain-funcs versionator ...
  ...
  DEPEND=... boost? ( dev-libs/boost ) ...
  ...
  pkg_env_check() {
  use boost  return 0
  version_is_at_least 4.1 $(gcc-version)  return 0
  echo Either USE boost, or switch to gcc later than 4.1
  }
  
  
  (with a default definition, pkg_env_check() { return 0; } )
 
 In an ideal system you'd want this stuff in the metadata cache so that
 the resolver can deal with it up front.

You're talking about the metadata on the host, rather than the stuff on
the rsync servers?  I'm not sure you could cache the results even on
the host - you would need to know what could affect the results so as
to know when the cached information is out of date and has to be
recalculated.  That would either have to checked on every emerge, or
made a separate switch (i.e. rely on the user to tell emerge when the
environment has changed).

  All resolution is a brute
 force metadata search; and assuming we had all the necessary data up
 front, we can optimize the search there (see pkgcore and restriction
 subsystem) versus IMHO doing a 'dumb' search and then running through
 a list of criteria for inclusion.
 
 This is the same reason why built_with_use in pkg_setup is really just
 use_deps; these metadata should be included during resolution, not
 after.

My concern about dynamic dependencies runs to use deps, as well :)
One could consider use-deps to be a special case of Marius' active
checks.  how pkg P was built isn't so different from slot S of P is
active in terms of dep-graph creation; both are asking about the
state of host  target systems, rather than the tree.

In the case of USE deps, things are saner because the data doesn't
change without the package manager knowing about it.  Effectively the
depgraph becomes static w.r.t. the tree + installation record (rather
than just static w.r.t. the tree).  With active checks implemented in
the depgraph, however, that is no longer the case - the depgraph can
change independently of the tree and the installation record.

 With that said, I'm not sure how easy it would be to rewrite that
 code; and this is simpler in that it's just a few bash functions as
 opposed to more resolver foo.

There's a lot to be said for keeping things simple, of course :)  It's
easy enough to mess up things like dep-graphs in any case - introducing
these sorts of dynamic dependencies can render it substantially more
complex.

Another way to look at it, is to consider how often this sort of thing
comes up.  My understanding is that it is relatively rare; across the
10,000+ packages in the tree only a handful use 'built_with_use' fex.
That makes a strong case for having a simple solution in the near term,
and re-visit if it becomes commonplace.

-- 
Kevin F. Quinn


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-31 Thread Alec Warner
Kevin F. Quinn wrote:
 On Wed, 31 Jan 2007 12:27:10 -0500
 Alec Warner [EMAIL PROTECTED] wrote:
 
 Hmm; one could get the same benefit by introducing a new interface
 (e.g. pkg_env_check()) which is defined to return true if the
 environment is ok, false otherwise (with some text to stdout,
 perhaps). The package manager would then run this function, after
 building the depgraph and finding the candidate packages to merge,
 for each candidate package - if any package fails is env_check,
 none of the packages get emerged.  Note this is then completely
 independent of depgraph creation.

 In the 'tr1' example, I'd imagine something like this:

 use.local.desc: boost: Use boost library for tr1 rather than gcc's.

 ebuild:

 ...
 inherit ... toolchain-funcs versionator ...
 ...
 DEPEND=... boost? ( dev-libs/boost ) ...
 ...
 pkg_env_check() {
 use boost  return 0
 version_is_at_least 4.1 $(gcc-version)  return 0
 echo Either USE boost, or switch to gcc later than 4.1
 }


 (with a default definition, pkg_env_check() { return 0; } )
 In an ideal system you'd want this stuff in the metadata cache so that
 the resolver can deal with it up front.
 
 You're talking about the metadata on the host, rather than the stuff on
 the rsync servers?  I'm not sure you could cache the results even on
 the host - you would need to know what could affect the results so as
 to know when the cached information is out of date and has to be
 recalculated.  That would either have to checked on every emerge, or
 made a separate switch (i.e. rely on the user to tell emerge when the
 environment has changed).

I am talking about the host yeah; cache was a bad term on my part;
obviously you cannot cache stuff like this.

 My concern about dynamic dependencies runs to use deps, as well :)
 One could consider use-deps to be a special case of Marius' active
 checks.  how pkg P was built isn't so different from slot S of P is
 active in terms of dep-graph creation; both are asking about the
 state of host  target systems, rather than the tree.
 
 In the case of USE deps, things are saner because the data doesn't
 change without the package manager knowing about it.  Effectively the
 depgraph becomes static w.r.t. the tree + installation record (rather
 than just static w.r.t. the tree).  With active checks implemented in
 the depgraph, however, that is no longer the case - the depgraph can
 change independently of the tree and the installation record.
 

I am unsure how fast these types of checks would or could be.  I mean we
can add arbitrary key caching and arbitrary key matching, but then that
grows the cache substantially and probably slows down dependency
resolution.  As you stated, some things just can't be cached properly
and still have value.

 With that said, I'm not sure how easy it would be to rewrite that
 code; and this is simpler in that it's just a few bash functions as
 opposed to more resolver foo.
 
 There's a lot to be said for keeping things simple, of course :)  It's
 easy enough to mess up things like dep-graphs in any case - introducing
 these sorts of dynamic dependencies can render it substantially more
 complex.

I think the complexity exists already though; currently we are just
hiding it, requiring people to find workarounds in an otherwise complex
playground of building packages.

 
 Another way to look at it, is to consider how often this sort of thing
 comes up.  My understanding is that it is relatively rare; across the
 10,000+ packages in the tree only a handful use 'built_with_use' fex.
 That makes a strong case for having a simple solution in the near term,
 and re-visit if it becomes commonplace.
 

I think your understanding is off then.  A quick grep (qrep -H
built_with_use | wc -l) gives me 814 calls to 'built_with_use' (qgrep is
in portage-utils).

If you grep through eclasses you will also see 53 separate calls; so you
can imagine what the real usage could be (definately at least 1/20th the
tree, not something I'd call minor).
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-31 Thread Marius Mauch
On Wed, 31 Jan 2007 17:47:26 +
Stephen Bennett [EMAIL PROTECTED] wrote:

 On Tue, 30 Jan 2007 17:06:51 +0100
 Marius Mauch [EMAIL PROTECTED] wrote:
 
  The idea is to add a special category (let's call it active for
  now) that has the following properties:
  - this category doesn't exist in portdir or vdb (= no ebuilds)
  - when portage ($pkgmanager) encounters a active/foo atom in a
  dependency string it executes some special code (e.g.
  $PORTDIR/scripts/active-check/foo =active/foo-1) to determine if
  that atom is satisfied
 
 Given that in the general case the package manager can't change the
 active provider and will have to bail with an appropriate message that
 the user needs to change it themselves, the obvious solution to this is
 the previously-discussed-somewhere-I-can't-remember ebuild function,
 called on depgraph creation, to check that it will be able to compile
 in the current system environment. It's considerably simpler and more
 generally useful than subverting DEPEND to add weird special-case hacks
 to it.

Yeah, thinking about it I have to agree tat generic dep_check() support (to 
give the thing a name) would be a better solution here.

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] [RFC] Depending on active version

2007-01-30 Thread Marius Mauch
Sometimes a package has to depend on a specific version of a slotted package 
being the active one to build correctly, like in the current tr1 discussion 
on -dev [1] or with packages that depend on the running kernel.

Currently this isn't really possible, however I while ago I got an idea how to 
solve this. Keep in mind this is just a rough idea and I'm pretty sure some 
people can/will point out why it is a stupid idea, but anyway:

The idea is to add a special category (let's call it active for now) that has 
the following properties:
- this category doesn't exist in portdir or vdb (= no ebuilds)
- when portage ($pkgmanager) encounters a active/foo atom in a dependency 
string it executes some special code (e.g. $PORTDIR/scripts/active-check/foo 
=active/foo-1) to determine if that atom is satisfied

(and yes, this kinda goes with multi-repo/multi-format support)

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-30 Thread Brian Harring
On Tue, Jan 30, 2007 at 05:06:51PM +0100, Marius Mauch wrote:
 Sometimes a package has to depend on a specific version of a 
 slotted package being the active one to build correctly, like in 
 the current tr1 discussion on -dev [1] or with packages that 
 depend on the running kernel.

tr1 is partially addressed via addition of a 'binding' modifier for 
rdeps, to state that ||() deps are locked down after compilation.

Doesn't gurantee the user doesn't pop back to 3.4 after compilation, 
but that's their own mess.

 The idea is to add a special category (let's call it active for 
 now) that has the following properties:
 - this category doesn't exist in portdir or vdb (= no ebuilds)
 - when portage ($pkgmanager) encounters a active/foo atom in a 
 dependency string it executes some special code (e.g. 
 $PORTDIR/scripts/active-check/foo =active/foo-1) to determine if 
 that atom is satisfied

Non deterministic resolution; previous steps in the graph can cause 
that value to flip to a different setting by the time the 'dep' is 
encountered.

That's ignoring the kick in the nads usage of this will due to 
resolution...

 (and yes, this kinda goes with multi-repo/multi-format support)

Don't really see how this enables multiple standalone repos in any 
sane way, so that one requires justification...

~harring


pgpetWeCOF0tF.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-30 Thread Alec Warner
Marius Mauch wrote:
 Sometimes a package has to depend on a specific version of a slotted package 
 being the active one to build correctly, like in the current tr1 
 discussion on -dev [1] or with packages that depend on the running kernel.
 
 Currently this isn't really possible, however I while ago I got an idea how 
 to solve this. Keep in mind this is just a rough idea and I'm pretty sure 
 some people can/will point out why it is a stupid idea, but anyway:
 
 The idea is to add a special category (let's call it active for now) that 
 has the following properties:
 - this category doesn't exist in portdir or vdb (= no ebuilds)
 - when portage ($pkgmanager) encounters a active/foo atom in a dependency 
 string it executes some special code (e.g. $PORTDIR/scripts/active-check/foo 
 =active/foo-1) to determine if that atom is satisfied
 
 (and yes, this kinda goes with multi-repo/multi-format support)
 
 Marius

I don't see why how this is any less complicated than just adding more
functionality to || deps (runtime versus compile time)
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-30 Thread Marius Mauch
On Tue, 30 Jan 2007 08:25:31 -0800
Brian Harring [EMAIL PROTECTED] wrote:

 On Tue, Jan 30, 2007 at 05:06:51PM +0100, Marius Mauch wrote:
  Sometimes a package has to depend on a specific version of a 
  slotted package being the active one to build correctly, like in 
  the current tr1 discussion on -dev [1] or with packages that 
  depend on the running kernel.
 
 tr1 is partially addressed via addition of a 'binding' modifier for 
 rdeps, to state that ||() deps are locked down after compilation.

And how would that solve the actual issue of expressing I need /usr/bin/gcc to 
run gcc-4.1 and not gcc-3.4?
The lockdown of || deps is a completely separate issue, unless I'm missing 
something.

  The idea is to add a special category (let's call it active for 
  now) that has the following properties:
  - this category doesn't exist in portdir or vdb (= no ebuilds)
  - when portage ($pkgmanager) encounters a active/foo atom in a 
  dependency string it executes some special code (e.g. 
  $PORTDIR/scripts/active-check/foo =active/foo-1) to determine if 
  that atom is satisfied
 
 Non deterministic resolution; previous steps in the graph can cause 
 that value to flip to a different setting by the time the 'dep' is 
 encountered.

Ok, that's a problem, though for the use cases at hand (gcc and kernel) it 
would be mostly irrelevant.

 That's ignoring the kick in the nads usage of this will due to 
 resolution...

Neglectable IMO, it's not such a common use case anyway, and I don't think I 
have to compare it to the current solution (die in setup or compile).

  (and yes, this kinda goes with multi-repo/multi-format support)
 
 Don't really see how this enables multiple standalone repos in any 
 sane way, so that one requires justification...

Where did I say anything about enabling? It would need more or less a separate 
repository (dbapi) instance, so it would require such support.

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list